2024-04-25T01:37:02.018 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T01:37:02.018 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342/orig.config.yaml 2024-04-25T01:37:02.072 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:37:02.166 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672342' kernel: kdb: 1 sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF82udbYFdRf9tjtMmjIylP+lAgqNC2Zzy5PaIYR33d1VbMz9xJLhyD3NL2ihqyfcEzpbnkGGJpvbOPd63OyZ1s= smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/v/iguqqCAkM4GT3humQA9fhNSYy03YKicJINuDvyy/svrxqMFd2Y7uQEVPThOrhWY1ZlTrZXJX6n3ZBOSIm8= smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAk8R7bNpgxgHx1j/zCAi+KwGKg9dU4qWHeaksTyaq0rSn/Zf/+1RAsa8NZ6Coutd4YHp+6krinPeNiN23lKOck= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T01:37:02.166 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-25T01:37:02.167 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-25T01:37:02.167 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T01:37:02.168 INFO:teuthology.task.internal:Checking packages... 2024-04-25T01:37:02.190 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-25T01:37:02.190 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T01:37:02.190 INFO:teuthology.packaging:ref: None 2024-04-25T01:37:02.190 INFO:teuthology.packaging:tag: None 2024-04-25T01:37:02.190 INFO:teuthology.packaging:branch: squid 2024-04-25T01:37:02.190 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:37:02.191 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-25T01:37:02.500 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-25T01:37:02.502 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T01:37:02.509 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T01:37:02.509 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T01:37:02.558 INFO:teuthology.task.internal:Saving configuration 2024-04-25T01:37:02.580 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T01:37:02.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T01:37:02.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:32:08.826590', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF82udbYFdRf9tjtMmjIylP+lAgqNC2Zzy5PaIYR33d1VbMz9xJLhyD3NL2ihqyfcEzpbnkGGJpvbOPd63OyZ1s='} 2024-04-25T01:37:02.652 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:32:08.828142', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/v/iguqqCAkM4GT3humQA9fhNSYy03YKicJINuDvyy/svrxqMFd2Y7uQEVPThOrhWY1ZlTrZXJX6n3ZBOSIm8='} 2024-04-25T01:37:02.673 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:32:08.829383', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAk8R7bNpgxgHx1j/zCAi+KwGKg9dU4qWHeaksTyaq0rSn/Zf/+1RAsa8NZ6Coutd4YHp+6krinPeNiN23lKOck='} 2024-04-25T01:37:02.673 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T01:37:02.680 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T01:37:02.680 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T01:37:02.680 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T01:37:02.680 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T01:37:02.828 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd903d15160>, signals=[15]) 2024-04-25T01:37:02.829 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T01:37:02.853 INFO:teuthology.task.internal:Opening connections... 2024-04-25T01:37:02.854 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T01:37:02.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:37:02.968 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-04-25T01:37:02.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:37:03.047 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-04-25T01:37:03.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:37:03.122 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T01:37:03.152 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-04-25T01:37:03.172 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-04-25T01:37:03.172 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-04-25T01:37:03.232 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-04-25T01:37:03.232 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-04-25T01:37:03.232 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-04-25T01:37:03.232 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:37:03.232 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-04-25T01:37:03.233 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:37:03.233 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:37:03.233 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-04-25T01:37:03.233 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:37:03.233 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:37:03.233 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:37:03.233 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:37:03.233 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:37:03.233 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:37:03.234 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-04-25T01:37:03.260 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-04-25T01:37:03.279 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-04-25T01:37:03.279 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-04-25T01:37:03.338 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-04-25T01:37:03.338 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2024-04-25T01:37:03.338 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-04-25T01:37:03.338 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:37:03.338 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2024-04-25T01:37:03.338 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:37:03.338 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:37:03.338 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-04-25T01:37:03.338 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:37:03.338 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:37:03.339 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:37:03.339 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:37:03.339 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:37:03.339 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:37:03.339 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-04-25T01:37:03.359 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-04-25T01:37:03.377 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-04-25T01:37:03.378 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-04-25T01:37:03.436 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-04-25T01:37:03.436 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-04-25T01:37:03.436 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-04-25T01:37:03.436 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:37:03.436 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-04-25T01:37:03.437 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:37:03.437 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:37:03.437 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-04-25T01:37:03.437 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:37:03.437 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:37:03.437 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:37:03.437 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:37:03.437 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:37:03.437 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:37:03.437 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-04-25T01:37:03.459 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T01:37:03.529 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T01:37:03.548 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T01:37:03.549 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:37:03.551 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:37:03.554 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:37:03.570 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T01:37:03.590 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T01:37:03.591 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:37:03.609 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:37:03.611 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:37:03.638 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T01:37:03.705 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T01:37:03.724 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-25T01:37:03.724 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-25T01:37:03.724 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:37:03.725 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:37:03.725 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:37:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:37:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:37:03.743 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-04-25T01:37:03.744 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-04-25T01:37:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:37:03.745 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-04-25T01:37:03.801 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:37:03.802 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-437.el9.x86_64 2024-04-25T01:37:03.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-04-25T01:37:03.803 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:37:03.804 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:37:03.804 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-437.el9.x86_64 2024-04-25T01:37:03.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2024-04-25T01:37:03.804 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.14.0-437.el9.x86_64 2024-04-25T01:37:03.804 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2024-04-25T01:37:05.972 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-25T01:37:06.839 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-04-25T01:37:08.648 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-25T01:37:09.036 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-04-25T01:37:10.737 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-04-25T01:37:13.607 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.2 MB 00:01 2024-04-25T01:37:15.349 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-25T01:37:15.880 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.2 MB 00:01 2024-04-25T01:37:16.140 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-04-25T01:37:17.719 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 21 MB 00:02 2024-04-25T01:37:17.721 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-04-25T01:37:19.758 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-25T01:37:21.134 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.2 MB 00:01 2024-04-25T01:37:22.642 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-04-25T01:37:22.985 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-04-25T01:37:24.610 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-25T01:37:24.840 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-25T01:37:24.990 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T01:37:24.990 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T01:37:25.071 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-25T01:37:25.072 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-25T01:37:25.072 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-25T01:37:25.174 DEBUG:teuthology.orchestra.run.smithi031:> echo no | sudo yum reinstall kernel || true 2024-04-25T01:37:25.693 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:37:22 AM UTC. 2024-04-25T01:37:25.816 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-04-25T01:37:25.828 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalling: 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 255 k 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 0 2024-04-25T01:37:25.829 INFO:teuthology.orchestra.run.smithi031.stderr:Operation aborted. 2024-04-25T01:37:25.880 INFO:teuthology.orchestra.run.smithi031.stdout:Is this ok [y/N]: 2024-04-25T01:37:25.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum reinstall -y kernel || true 2024-04-25T01:37:26.398 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 01:37:22 AM UTC. 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-04-25T01:37:26.519 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalling: 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 255 k 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 0 2024-04-25T01:37:26.520 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T01:37:26.663 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T01:37:26.745 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-25T01:37:26.746 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-25T01:37:26.747 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-25T01:37:26.873 DEBUG:teuthology.orchestra.run.smithi134:> echo no | sudo yum reinstall kernel || true 2024-04-25T01:37:27.094 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-437.el9.x86_64.rpm 863 kB/s | 255 kB 00:00 2024-04-25T01:37:27.094 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:37:27.094 INFO:teuthology.orchestra.run.smithi031.stdout:Total 444 kB/s | 255 kB 00:00 2024-04-25T01:37:27.115 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-25T01:37:27.328 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-25T01:37:27.328 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-25T01:37:27.333 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-25T01:37:27.334 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-25T01:37:27.372 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:37:24 AM UTC. 2024-04-25T01:37:27.462 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-25T01:37:27.516 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-25T01:37:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:37:27.528 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:37:27.530 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-25T01:37:27.530 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-25T01:37:27.530 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-25T01:37:27.531 INFO:teuthology.orchestra.run.smithi134.stdout:Reinstalling: 2024-04-25T01:37:27.531 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:37:27.531 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:37:27.531 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-25T01:37:27.531 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-25T01:37:27.531 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:37:27.531 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 255 k 2024-04-25T01:37:27.531 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 0 2024-04-25T01:37:27.532 INFO:teuthology.orchestra.run.smithi134.stderr:Operation aborted. 2024-04-25T01:37:27.592 INFO:teuthology.orchestra.run.smithi134.stdout:Is this ok [y/N]: 2024-04-25T01:37:27.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum reinstall -y kernel || true 2024-04-25T01:37:28.085 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 01:37:24 AM UTC. 2024-04-25T01:37:28.169 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:37:28.169 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:37:28.200 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-25T01:37:28.201 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-25T01:37:28.201 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-25T01:37:28.201 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-25T01:37:28.201 INFO:teuthology.orchestra.run.smithi134.stdout:Reinstalling: 2024-04-25T01:37:28.201 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:37:28.201 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:37:28.201 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-25T01:37:28.201 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-25T01:37:28.201 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:37:28.202 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 255 k 2024-04-25T01:37:28.202 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 0 2024-04-25T01:37:28.202 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-25T01:37:28.717 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.0 MB/s | 255 kB 00:00 2024-04-25T01:37:28.717 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:37:28.717 INFO:teuthology.orchestra.run.smithi134.stdout:Total 494 kB/s | 255 kB 00:00 2024-04-25T01:37:28.737 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-25T01:37:28.888 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:37:28.888 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:37:28.889 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalled: 2024-04-25T01:37:28.889 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T01:37:28.889 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:37:28.889 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-25T01:37:28.921 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-25T01:37:28.921 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-25T01:37:28.926 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-25T01:37:28.926 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-25T01:37:29.066 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T01:37:29.067 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-25T01:37:29.114 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T01:37:29.115 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T01:37:29.115 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T01:37:29.115 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T01:37:29.115 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T01:37:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:37:29.135 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:37:29.145 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2024-04-25T01:37:29.171 DEBUG:teuthology.parallel:result is None 2024-04-25T01:37:29.860 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-25T01:37:29.911 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:37:29.912 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:37:30.613 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:37:30.613 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:37:30.613 INFO:teuthology.orchestra.run.smithi134.stdout:Reinstalled: 2024-04-25T01:37:30.613 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T01:37:30.613 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:37:30.613 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-25T01:37:30.795 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T01:37:30.842 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T01:37:30.842 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T01:37:30.842 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T01:37:30.842 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-25T01:37:30.842 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T01:37:30.877 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-04-25T01:37:30.896 DEBUG:teuthology.parallel:result is None 2024-04-25T01:37:31.842 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T01:37:31.842 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T01:37:31.948 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-25T01:37:31.949 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-25T01:37:31.949 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-25T01:37:32.049 DEBUG:teuthology.orchestra.run.smithi040:> echo no | sudo yum reinstall kernel || true 2024-04-25T01:37:32.579 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:37:29 AM UTC. 2024-04-25T01:37:32.709 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-25T01:37:32.727 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-25T01:37:32.727 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-25T01:37:32.727 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-25T01:37:32.727 INFO:teuthology.orchestra.run.smithi040.stdout:Reinstalling: 2024-04-25T01:37:32.727 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:37:32.728 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:37:32.728 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-25T01:37:32.728 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-25T01:37:32.728 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:37:32.728 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 255 k 2024-04-25T01:37:32.728 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 0 2024-04-25T01:37:32.728 INFO:teuthology.orchestra.run.smithi040.stderr:Operation aborted. 2024-04-25T01:37:32.788 INFO:teuthology.orchestra.run.smithi040.stdout:Is this ok [y/N]: 2024-04-25T01:37:32.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum reinstall -y kernel || true 2024-04-25T01:37:33.303 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 01:37:29 AM UTC. 2024-04-25T01:37:33.423 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-25T01:37:33.424 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-25T01:37:33.424 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-04-25T01:37:33.424 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-25T01:37:33.425 INFO:teuthology.orchestra.run.smithi040.stdout:Reinstalling: 2024-04-25T01:37:33.425 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:37:33.425 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:37:33.425 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-25T01:37:33.425 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-04-25T01:37:33.425 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:37:33.425 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 255 k 2024-04-25T01:37:33.425 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 0 2024-04-25T01:37:33.425 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-25T01:37:33.862 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.6 MB/s | 255 kB 00:00 2024-04-25T01:37:33.863 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:37:33.863 INFO:teuthology.orchestra.run.smithi040.stdout:Total 582 kB/s | 255 kB 00:00 2024-04-25T01:37:33.878 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-25T01:37:34.075 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-25T01:37:34.075 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-25T01:37:34.080 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-25T01:37:34.081 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-25T01:37:34.219 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-25T01:37:34.297 INFO:teuthology.orchestra.run.smithi040.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:37:34.315 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:37:34.960 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:37:34.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:37:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:37:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:37:35.694 INFO:teuthology.orchestra.run.smithi040.stdout:Reinstalled: 2024-04-25T01:37:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T01:37:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:37:35.695 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-25T01:37:35.885 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T01:37:35.974 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T01:37:35.974 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T01:37:35.974 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T01:37:35.974 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-25T01:37:35.974 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T01:37:36.007 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2024-04-25T01:37:36.035 DEBUG:teuthology.parallel:result is None 2024-04-25T01:37:36.035 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T01:37:36.043 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T01:37:36.043 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:37:36.045 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:37:36.078 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:37:36.096 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T01:37:36.131 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T01:37:36.230 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T01:37:36.230 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:37:36.233 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:37:36.235 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:37:36.269 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T01:37:36.313 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T01:37:36.314 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:37:36.316 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:37:36.318 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:37:36.348 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:37:36.349 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:37:36.358 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:37:36.360 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:37:36.379 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:37:36.391 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:37:36.392 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T01:37:36.399 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T01:37:36.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:37:36.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:37:36.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:37:36.434 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T01:37:36.449 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T01:37:36.449 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:37:36.473 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:37:36.475 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:37:36.492 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:37:36.572 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:37:36.622 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:37:36.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:37:36.752 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:37:36.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:37:36.818 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:37:36.860 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:37:36.907 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:37:36.990 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:37:37.029 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-25T01:37:37.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:37:37.093 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:37:37.140 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:37:37.195 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:37:37.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:37:37.320 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T01:37:37.320 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:37:37.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-04-25T01:37:37.389 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-04-25T01:37:37.391 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-04-25T01:37:37.427 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:37:37.430 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:37:37.466 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:37:37.952 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T01:37:37.961 INFO:teuthology.task.internal:Starting timer... 2024-04-25T01:37:37.961 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T01:37:38.034 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T01:37:38.133 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T01:37:38.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-04-25T01:37:38.187 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-04-25T01:37:38.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-04-25T01:37:38.247 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-04-25T01:37:38.249 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-04-25T01:37:38.301 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-04-25T01:37:38.303 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T01:37:38.303 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-04-25T01:37:38.325 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-04-25T01:37:38.325 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-04-25T01:37:38.355 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-04-25T01:37:38.355 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-04-25T01:37:38.377 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-04-25T01:37:38.377 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2024-04-25T01:37:38.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:37:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:37:38.402 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:37:38.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:37:38.427 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:37:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:37:38.454 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T01:37:38.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:37:38.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:37:38.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:37:38.538 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T01:37:38.552 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T01:37:38.559 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T01:37:38.681 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T01:37:38.697 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T01:37:38.699 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi040.front.sepia.ceph.com,smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T01:41:12.013 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2024-04-25T01:41:12.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-25T01:41:12.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:41:12.091 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-04-25T01:41:12.170 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-25T01:41:12.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-25T01:41:12.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:41:12.246 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-04-25T01:41:12.331 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-25T01:41:12.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-25T01:41:12.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:41:12.407 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-04-25T01:41:12.486 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-25T01:41:12.486 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T01:41:12.497 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T01:41:12.498 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:41:12.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:41:12.500 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:41:12.500 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:41:12.503 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:41:12.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:41:12.532 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:41:12.533 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:41:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:41:12.548 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:41:12.555 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:41:12.569 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:41:12.574 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-04-25T01:41:12.574 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-04-25T01:41:12.585 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-04-25T01:41:12.586 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-04-25T01:41:12.592 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-04-25T01:41:12.599 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:41:12.600 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:41:12.602 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-04-25T01:41:12.614 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:41:12.614 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:41:12.616 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:41:12.628 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:41:12.668 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:41:12.671 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:41:12.671 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-04-25T01:41:12.671 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.671 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.672 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.672 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.675 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:41:12.678 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:41:12.678 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-25T01:41:12.678 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.679 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.679 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.679 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.682 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:41:12.686 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:41:12.686 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-04-25T01:41:12.686 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.686 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.686 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.686 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:41:12.687 INFO:teuthology.run_tasks:Running task install... 2024-04-25T01:41:12.698 DEBUG:teuthology.task.install:project ceph 2024-04-25T01:41:12.698 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-25T01:41:12.698 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-25T01:41:12.698 INFO:teuthology.task.install:Using flavor: default 2024-04-25T01:41:12.706 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T01:41:12.706 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T01:41:12.706 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:41:12.707 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:41:12.709 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:41:12.709 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:41:12.710 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:41:12.710 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:41:12.861 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:41:12.861 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:41:12.937 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:41:12.937 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:41:12.944 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:41:12.944 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:41:13.030 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:41:13.030 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:41:13.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:41:13.043 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:41:13.043 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T01:41:13.043 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:41:13.049 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:41:13.049 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-25T01:41:13.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:41:13.056 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:41:13.057 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:41:13.068 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:41:13.068 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:41:13.076 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:41:13.076 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:41:13.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:41:13.133 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:41:13.143 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:41:13.202 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:41:13.205 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:41:13.219 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:41:13.266 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-04-25T01:41:13.268 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-04-25T01:41:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-04-25T01:41:13.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-04-25T01:41:13.283 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2024-04-25T01:41:13.284 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2024-04-25T01:41:13.493 INFO:teuthology.orchestra.run.smithi134.stdout:48 files removed 2024-04-25T01:41:13.501 INFO:teuthology.orchestra.run.smithi031.stdout:48 files removed 2024-04-25T01:41:13.518 INFO:teuthology.orchestra.run.smithi040.stdout:48 files removed 2024-04-25T01:41:13.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:41:13.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:41:13.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:41:14.213 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 201 kB/s | 83 kB 00:00 2024-04-25T01:41:14.230 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-04-25T01:41:14.255 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 176 kB/s | 83 kB 00:00 2024-04-25T01:41:14.597 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 39 kB/s | 11 kB 00:00 2024-04-25T01:41:14.606 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-04-25T01:41:14.647 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-04-25T01:41:14.931 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2024-04-25T01:41:15.006 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-25T01:41:15.023 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-04-25T01:41:16.599 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.0 MB 00:01 2024-04-25T01:41:16.898 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.0 MB 00:01 2024-04-25T01:41:18.098 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 8.0 MB 00:03 2024-04-25T01:41:18.308 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-04-25T01:41:18.658 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-04-25T01:41:21.340 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 8.7 MB/s | 19 MB 00:02 2024-04-25T01:41:22.576 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T01:41:24.018 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-04-25T01:41:25.027 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.2 MB 00:02 2024-04-25T01:41:25.531 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-25T01:41:25.866 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 6.2 MB 00:00 2024-04-25T01:41:26.919 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-04-25T01:41:27.375 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 16 kB 00:00 2024-04-25T01:41:28.870 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-25T01:41:29.021 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-25T01:41:30.887 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-25T01:41:33.139 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:41:33.139 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:41:33.150 INFO:teuthology.orchestra.run.smithi040.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:41:33.151 INFO:teuthology.orchestra.run.smithi040.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:41:33.152 INFO:teuthology.orchestra.run.smithi040.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:41:33.153 INFO:teuthology.orchestra.run.smithi040.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:41:33.155 INFO:teuthology.orchestra.run.smithi040.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:41:33.157 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:41:33.158 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:41:33.222 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-25T01:41:33.231 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-25T01:41:33.231 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:41:33.232 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:41:33.233 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:41:33.233 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:41:33.233 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:41:33.233 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:41:33.233 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:41:33.233 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:41:33.233 INFO:teuthology.orchestra.run.smithi040.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:41:33.233 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:41:33.233 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:41:33.233 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:41:33.233 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:41:33.235 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:41:33.236 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:41:33.237 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:41:33.238 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:41:33.239 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:41:33.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:41:33.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:41:33.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:41:33.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:41:33.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:41:33.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:41:33.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:41:33.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:41:33.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:41:33.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:41:33.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:41:33.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:41:33.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:41:33.243 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout:Install 148 Packages 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 4 Packages 2024-04-25T01:41:33.244 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:41:33.245 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 220 M 2024-04-25T01:41:33.245 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-25T01:41:34.102 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-04-25T01:41:34.113 INFO:teuthology.orchestra.run.smithi040.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-04-25T01:41:34.122 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T01:41:34.313 INFO:teuthology.orchestra.run.smithi040.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 4.4 MB/s | 890 kB 00:00 2024-04-25T01:41:34.364 INFO:teuthology.orchestra.run.smithi040.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 2.8 MB/s | 145 kB 00:00 2024-04-25T01:41:34.673 INFO:teuthology.orchestra.run.smithi040.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 7.7 MB/s | 2.4 MB 00:00 2024-04-25T01:41:34.898 INFO:teuthology.orchestra.run.smithi040.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 6.6 MB/s | 1.5 MB 00:00 2024-04-25T01:41:35.090 INFO:teuthology.orchestra.run.smithi040.stdout:(6/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 4.7 MB/s | 5.4 MB 00:01 2024-04-25T01:41:35.550 INFO:teuthology.orchestra.run.smithi040.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 7.2 MB/s | 4.7 MB 00:00 2024-04-25T01:41:35.974 INFO:teuthology.orchestra.run.smithi040.stdout:(8/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 10 MB/s | 21 MB 00:02 2024-04-25T01:41:36.024 INFO:teuthology.orchestra.run.smithi040.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 507 kB/s | 25 kB 00:00 2024-04-25T01:41:36.203 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:41:36.203 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:41:36.214 INFO:teuthology.orchestra.run.smithi134.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:41:36.215 INFO:teuthology.orchestra.run.smithi134.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:41:36.216 INFO:teuthology.orchestra.run.smithi134.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:41:36.216 INFO:teuthology.orchestra.run.smithi134.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:41:36.218 INFO:teuthology.orchestra.run.smithi134.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:41:36.220 INFO:teuthology.orchestra.run.smithi134.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:41:36.221 INFO:teuthology.orchestra.run.smithi134.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:41:36.280 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-25T01:41:36.289 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-04-25T01:41:36.289 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-25T01:41:36.289 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-04-25T01:41:36.289 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-25T01:41:36.289 INFO:teuthology.orchestra.run.smithi134.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:41:36.289 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:41:36.289 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:41:36.290 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-25T01:41:36.292 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:41:36.293 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:41:36.294 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:41:36.295 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:41:36.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:41:36.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:41:36.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:41:36.299 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:41:36.300 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-25T01:41:36.301 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-04-25T01:41:36.301 INFO:teuthology.orchestra.run.smithi134.stdout:Install 148 Packages 2024-04-25T01:41:36.301 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 4 Packages 2024-04-25T01:41:36.301 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:41:36.301 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 220 M 2024-04-25T01:41:36.301 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-25T01:41:36.447 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:41:36.448 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:41:36.459 INFO:teuthology.orchestra.run.smithi031.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:41:36.460 INFO:teuthology.orchestra.run.smithi031.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:41:36.461 INFO:teuthology.orchestra.run.smithi031.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:41:36.462 INFO:teuthology.orchestra.run.smithi031.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:41:36.464 INFO:teuthology.orchestra.run.smithi031.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:41:36.466 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:41:36.467 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:41:36.532 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-25T01:41:36.541 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-04-25T01:41:36.541 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-25T01:41:36.541 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-04-25T01:41:36.541 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-25T01:41:36.541 INFO:teuthology.orchestra.run.smithi031.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:41:36.541 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:41:36.542 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:41:36.543 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:41:36.544 INFO:teuthology.orchestra.run.smithi031.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:41:36.545 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:41:36.546 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:41:36.547 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:41:36.548 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:41:36.549 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:41:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:41:36.551 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:41:36.552 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:41:36.553 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-25T01:41:36.553 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-04-25T01:41:36.553 INFO:teuthology.orchestra.run.smithi031.stdout:Install 148 Packages 2024-04-25T01:41:36.553 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 4 Packages 2024-04-25T01:41:36.553 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:41:36.553 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 220 M 2024-04-25T01:41:36.553 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-25T01:41:36.557 INFO:teuthology.orchestra.run.smithi040.stdout:(10/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 10 MB/s | 10 MB 00:01 2024-04-25T01:41:36.641 INFO:teuthology.orchestra.run.smithi040.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 2.6 MB/s | 222 kB 00:00 2024-04-25T01:41:36.699 INFO:teuthology.orchestra.run.smithi040.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 466 kB/s | 27 kB 00:00 2024-04-25T01:41:36.766 INFO:teuthology.orchestra.run.smithi040.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 475 kB/s | 32 kB 00:00 2024-04-25T01:41:36.934 INFO:teuthology.orchestra.run.smithi040.stdout:(14/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 9.1 MB/s | 17 MB 00:01 2024-04-25T01:41:36.967 INFO:teuthology.orchestra.run.smithi040.stdout:(15/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 3.6 MB/s | 745 kB 00:00 2024-04-25T01:41:37.026 INFO:teuthology.orchestra.run.smithi040.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 2.1 MB/s | 127 kB 00:00 2024-04-25T01:41:37.059 INFO:teuthology.orchestra.run.smithi040.stdout:(17/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 1.3 MB/s | 163 kB 00:00 2024-04-25T01:41:37.101 INFO:teuthology.orchestra.run.smithi040.stdout:(18/152): libradosstriper1-19.0.0-2445.gd2c45d9 6.5 MB/s | 501 kB 00:00 2024-04-25T01:41:37.112 INFO:teuthology.orchestra.run.smithi134.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-04-25T01:41:37.168 INFO:teuthology.orchestra.run.smithi040.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 677 kB/s | 45 kB 00:00 2024-04-25T01:41:37.243 INFO:teuthology.orchestra.run.smithi040.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.7 MB/s | 133 kB 00:00 2024-04-25T01:41:37.302 INFO:teuthology.orchestra.run.smithi040.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 2.7 MB/s | 162 kB 00:00 2024-04-25T01:41:37.377 INFO:teuthology.orchestra.run.smithi040.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 4.2 MB/s | 321 kB 00:00 2024-04-25T01:41:37.461 INFO:teuthology.orchestra.run.smithi040.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.5 MB/s | 299 kB 00:00 2024-04-25T01:41:37.536 INFO:teuthology.orchestra.run.smithi040.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 99 kB 00:00 2024-04-25T01:41:37.612 INFO:teuthology.orchestra.run.smithi040.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-25T01:41:37.721 INFO:teuthology.orchestra.run.smithi031.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-04-25T01:41:37.771 INFO:teuthology.orchestra.run.smithi040.stdout:(26/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 7.5 MB/s | 5.3 MB 00:00 2024-04-25T01:41:37.870 INFO:teuthology.orchestra.run.smithi134.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.1 MB/s | 890 kB 00:00 2024-04-25T01:41:37.879 INFO:teuthology.orchestra.run.smithi040.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 1.5 MB/s | 171 kB 00:00 2024-04-25T01:41:37.963 INFO:teuthology.orchestra.run.smithi040.stdout:(28/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 8.8 MB/s | 3.1 MB 00:00 2024-04-25T01:41:37.970 INFO:teuthology.orchestra.run.smithi134.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.4 MB/s | 145 kB 00:00 2024-04-25T01:41:37.988 INFO:teuthology.orchestra.run.smithi040.stdout:(29/152): ceph-grafana-dashboards-19.0.0-2445.g 293 kB/s | 32 kB 00:00 2024-04-25T01:41:38.014 INFO:teuthology.orchestra.run.smithi040.stdout:(30/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 2.7 MB/s | 141 kB 00:00 2024-04-25T01:41:38.464 INFO:teuthology.orchestra.run.smithi040.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 7.4 MB/s | 3.5 MB 00:00 2024-04-25T01:41:38.564 INFO:teuthology.orchestra.run.smithi040.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.4 MB/s | 249 kB 00:00 2024-04-25T01:41:38.665 INFO:teuthology.orchestra.run.smithi040.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 493 kB/s | 49 kB 00:00 2024-04-25T01:41:38.763 INFO:teuthology.orchestra.run.smithi031.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 854 kB/s | 890 kB 00:01 2024-04-25T01:41:38.773 INFO:teuthology.orchestra.run.smithi040.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 142 kB/s | 15 kB 00:00 2024-04-25T01:41:38.927 INFO:teuthology.orchestra.run.smithi040.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 1.7 MB/s | 264 kB 00:00 2024-04-25T01:41:39.047 INFO:teuthology.orchestra.run.smithi031.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 511 kB/s | 145 kB 00:00 2024-04-25T01:41:39.604 INFO:teuthology.orchestra.run.smithi134.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 1.4 MB/s | 2.4 MB 00:01 2024-04-25T01:41:39.999 INFO:teuthology.orchestra.run.smithi040.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 3.7 MB/s | 7.4 MB 00:01 2024-04-25T01:41:40.091 INFO:teuthology.orchestra.run.smithi040.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 200 kB/s | 18 kB 00:00 2024-04-25T01:41:40.182 INFO:teuthology.orchestra.run.smithi040.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 878 kB/s | 80 kB 00:00 2024-04-25T01:41:40.291 INFO:teuthology.orchestra.run.smithi040.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.2 MB/s | 803 kB 00:00 2024-04-25T01:41:40.341 INFO:teuthology.orchestra.run.smithi040.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-25T01:41:40.392 INFO:teuthology.orchestra.run.smithi040.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-25T01:41:40.475 INFO:teuthology.orchestra.run.smithi040.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-04-25T01:41:40.517 INFO:teuthology.orchestra.run.smithi040.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-25T01:41:40.551 INFO:teuthology.orchestra.run.smithi040.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 4.0 MB/s | 135 kB 00:00 2024-04-25T01:41:40.563 INFO:teuthology.orchestra.run.smithi031.stdout:(4/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 1.7 MB/s | 5.4 MB 00:03 2024-04-25T01:41:40.593 INFO:teuthology.orchestra.run.smithi040.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-25T01:41:40.643 INFO:teuthology.orchestra.run.smithi040.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T01:41:40.702 INFO:teuthology.orchestra.run.smithi040.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 12 MB/s | 689 kB 00:00 2024-04-25T01:41:40.760 INFO:teuthology.orchestra.run.smithi040.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 943 kB 00:00 2024-04-25T01:41:40.802 INFO:teuthology.orchestra.run.smithi040.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-04-25T01:41:40.836 INFO:teuthology.orchestra.run.smithi040.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 649 kB/s | 21 kB 00:00 2024-04-25T01:41:40.846 INFO:teuthology.orchestra.run.smithi134.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 1.2 MB/s | 1.5 MB 00:01 2024-04-25T01:41:40.886 INFO:teuthology.orchestra.run.smithi040.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T01:41:40.920 INFO:teuthology.orchestra.run.smithi040.stdout:(52/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 382 kB/s | 762 kB 00:01 2024-04-25T01:41:40.937 INFO:teuthology.orchestra.run.smithi040.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 651 kB/s | 33 kB 00:00 2024-04-25T01:41:40.979 INFO:teuthology.orchestra.run.smithi040.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 436 kB/s | 18 kB 00:00 2024-04-25T01:41:41.021 INFO:teuthology.orchestra.run.smithi040.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 269 kB/s | 11 kB 00:00 2024-04-25T01:41:41.063 INFO:teuthology.orchestra.run.smithi040.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 160 kB 00:00 2024-04-25T01:41:41.088 INFO:teuthology.orchestra.run.smithi134.stdout:(6/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 1.3 MB/s | 5.4 MB 00:04 2024-04-25T01:41:41.113 INFO:teuthology.orchestra.run.smithi040.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-04-25T01:41:41.172 INFO:teuthology.orchestra.run.smithi040.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-25T01:41:41.222 INFO:teuthology.orchestra.run.smithi040.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.9 MB/s | 3.0 MB 00:00 2024-04-25T01:41:41.325 INFO:teuthology.orchestra.run.smithi040.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 3.7 MB/s | 582 kB 00:00 2024-04-25T01:41:41.356 INFO:teuthology.orchestra.run.smithi040.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 190 kB/s | 25 kB 00:00 2024-04-25T01:41:41.381 INFO:teuthology.orchestra.run.smithi040.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 851 kB/s | 47 kB 00:00 2024-04-25T01:41:41.407 INFO:teuthology.orchestra.run.smithi040.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-25T01:41:41.429 INFO:teuthology.orchestra.run.smithi134.stdout:(7/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 4.5 MB/s | 21 MB 00:04 2024-04-25T01:41:41.432 INFO:teuthology.orchestra.run.smithi040.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-25T01:41:41.457 INFO:teuthology.orchestra.run.smithi040.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 704 kB/s | 35 kB 00:00 2024-04-25T01:41:41.589 INFO:teuthology.orchestra.run.smithi031.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 1.5 MB/s | 1.5 MB 00:01 2024-04-25T01:41:41.599 INFO:teuthology.orchestra.run.smithi040.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-04-25T01:41:41.666 INFO:teuthology.orchestra.run.smithi040.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-04-25T01:41:41.708 INFO:teuthology.orchestra.run.smithi040.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-25T01:41:41.750 INFO:teuthology.orchestra.run.smithi040.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-25T01:41:41.775 INFO:teuthology.orchestra.run.smithi040.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-04-25T01:41:41.851 INFO:teuthology.orchestra.run.smithi040.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-04-25T01:41:41.914 INFO:teuthology.orchestra.run.smithi031.stdout:(6/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 844 kB/s | 2.4 MB 00:02 2024-04-25T01:41:41.934 INFO:teuthology.orchestra.run.smithi040.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 31 MB/s | 5.8 MB 00:00 2024-04-25T01:41:41.959 INFO:teuthology.orchestra.run.smithi040.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 807 kB/s | 87 kB 00:00 2024-04-25T01:41:42.035 INFO:teuthology.orchestra.run.smithi040.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 3.2 MB/s | 244 kB 00:00 2024-04-25T01:41:42.110 INFO:teuthology.orchestra.run.smithi040.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 34 MB/s | 6.0 MB 00:00 2024-04-25T01:41:42.135 INFO:teuthology.orchestra.run.smithi040.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-25T01:41:42.152 INFO:teuthology.orchestra.run.smithi040.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 611 kB/s | 26 kB 00:00 2024-04-25T01:41:42.177 INFO:teuthology.orchestra.run.smithi040.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 458 kB/s | 19 kB 00:00 2024-04-25T01:41:42.203 INFO:teuthology.orchestra.run.smithi040.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-25T01:41:42.219 INFO:teuthology.orchestra.run.smithi040.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 831 kB/s | 35 kB 00:00 2024-04-25T01:41:42.287 INFO:teuthology.orchestra.run.smithi040.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-04-25T01:41:42.362 INFO:teuthology.orchestra.run.smithi040.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 5.1 MB 00:00 2024-04-25T01:41:42.396 INFO:teuthology.orchestra.run.smithi040.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-04-25T01:41:42.421 INFO:teuthology.orchestra.run.smithi040.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-25T01:41:42.438 INFO:teuthology.orchestra.run.smithi040.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-25T01:41:42.480 INFO:teuthology.orchestra.run.smithi040.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-25T01:41:42.505 INFO:teuthology.orchestra.run.smithi040.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-04-25T01:41:42.530 INFO:teuthology.orchestra.run.smithi040.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-25T01:41:42.556 INFO:teuthology.orchestra.run.smithi040.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T01:41:42.581 INFO:teuthology.orchestra.run.smithi040.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T01:41:42.631 INFO:teuthology.orchestra.run.smithi040.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 835 kB/s | 42 kB 00:00 2024-04-25T01:41:42.690 INFO:teuthology.orchestra.run.smithi040.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 757 kB/s | 44 kB 00:00 2024-04-25T01:41:42.749 INFO:teuthology.orchestra.run.smithi040.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-25T01:41:42.807 INFO:teuthology.orchestra.run.smithi040.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 615 kB/s | 36 kB 00:00 2024-04-25T01:41:42.966 INFO:teuthology.orchestra.run.smithi040.stdout:(95/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-04-25T01:41:43.021 INFO:teuthology.orchestra.run.smithi040.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 222 kB/s | 47 kB 00:00 2024-04-25T01:41:43.050 INFO:teuthology.orchestra.run.smithi040.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-04-25T01:41:43.075 INFO:teuthology.orchestra.run.smithi040.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-04-25T01:41:43.100 INFO:teuthology.orchestra.run.smithi040.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T01:41:43.126 INFO:teuthology.orchestra.run.smithi040.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T01:41:43.151 INFO:teuthology.orchestra.run.smithi040.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 892 kB/s | 45 kB 00:00 2024-04-25T01:41:43.418 INFO:teuthology.orchestra.run.smithi040.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-04-25T01:41:43.535 INFO:teuthology.orchestra.run.smithi040.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 4.6 MB/s | 548 kB 00:00 2024-04-25T01:41:43.602 INFO:teuthology.orchestra.run.smithi040.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 4.5 MB/s | 309 kB 00:00 2024-04-25T01:41:43.661 INFO:teuthology.orchestra.run.smithi040.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-04-25T01:41:43.688 INFO:teuthology.orchestra.run.smithi134.stdout:(8/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el9 4.5 MB/s | 10 MB 00:02 2024-04-25T01:41:43.771 INFO:teuthology.orchestra.run.smithi134.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 303 kB/s | 25 kB 00:00 2024-04-25T01:41:43.802 INFO:teuthology.orchestra.run.smithi040.stdout:(106/152): libarrow-9.0.0-10.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-04-25T01:41:43.911 INFO:teuthology.orchestra.run.smithi040.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-25T01:41:43.945 INFO:teuthology.orchestra.run.smithi040.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-04-25T01:41:43.978 INFO:teuthology.orchestra.run.smithi040.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-04-25T01:41:44.003 INFO:teuthology.orchestra.run.smithi040.stdout:(110/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.0 MB/s | 863 kB 00:00 2024-04-25T01:41:44.029 INFO:teuthology.orchestra.run.smithi040.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-25T01:41:44.081 INFO:teuthology.orchestra.run.smithi040.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 10 MB/s | 548 kB 00:00 2024-04-25T01:41:44.115 INFO:teuthology.orchestra.run.smithi040.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-04-25T01:41:44.156 INFO:teuthology.orchestra.run.smithi040.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 776 kB/s | 32 kB 00:00 2024-04-25T01:41:44.190 INFO:teuthology.orchestra.run.smithi040.stdout:(115/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 838 kB 00:00 2024-04-25T01:41:44.207 INFO:teuthology.orchestra.run.smithi040.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-25T01:41:44.232 INFO:teuthology.orchestra.run.smithi040.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-04-25T01:41:44.258 INFO:teuthology.orchestra.run.smithi040.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-04-25T01:41:44.283 INFO:teuthology.orchestra.run.smithi040.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-04-25T01:41:44.300 INFO:teuthology.orchestra.run.smithi040.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 844 kB/s | 35 kB 00:00 2024-04-25T01:41:44.317 INFO:teuthology.orchestra.run.smithi040.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 6.0 MB/s | 206 kB 00:00 2024-04-25T01:41:44.338 INFO:teuthology.orchestra.run.smithi134.stdout:(10/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x8 1.3 MB/s | 4.7 MB 00:03 2024-04-25T01:41:44.351 INFO:teuthology.orchestra.run.smithi040.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-04-25T01:41:44.401 INFO:teuthology.orchestra.run.smithi040.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-04-25T01:41:44.426 INFO:teuthology.orchestra.run.smithi040.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-25T01:41:44.451 INFO:teuthology.orchestra.run.smithi040.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-04-25T01:41:44.477 INFO:teuthology.orchestra.run.smithi040.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-25T01:41:44.502 INFO:teuthology.orchestra.run.smithi040.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 389 kB/s | 19 kB 00:00 2024-04-25T01:41:44.527 INFO:teuthology.orchestra.run.smithi040.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 394 kB/s | 20 kB 00:00 2024-04-25T01:41:44.538 INFO:teuthology.orchestra.run.smithi134.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 1.1 MB/s | 222 kB 00:00 2024-04-25T01:41:44.561 INFO:teuthology.orchestra.run.smithi040.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-04-25T01:41:44.586 INFO:teuthology.orchestra.run.smithi040.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 793 kB/s | 46 kB 00:00 2024-04-25T01:41:44.614 INFO:teuthology.orchestra.run.smithi031.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 1.6 MB/s | 4.7 MB 00:03 2024-04-25T01:41:44.639 INFO:teuthology.orchestra.run.smithi134.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 271 kB/s | 27 kB 00:00 2024-04-25T01:41:44.645 INFO:teuthology.orchestra.run.smithi040.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 944 kB/s | 79 kB 00:00 2024-04-25T01:41:44.670 INFO:teuthology.orchestra.run.smithi040.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 691 kB/s | 58 kB 00:00 2024-04-25T01:41:44.695 INFO:teuthology.orchestra.run.smithi040.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-25T01:41:44.720 INFO:teuthology.orchestra.run.smithi040.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 329 kB/s | 16 kB 00:00 2024-04-25T01:41:44.737 INFO:teuthology.orchestra.run.smithi040.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-04-25T01:41:44.762 INFO:teuthology.orchestra.run.smithi040.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 675 kB/s | 28 kB 00:00 2024-04-25T01:41:44.780 INFO:teuthology.orchestra.run.smithi134.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 224 kB/s | 32 kB 00:00 2024-04-25T01:41:44.788 INFO:teuthology.orchestra.run.smithi040.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-04-25T01:41:44.813 INFO:teuthology.orchestra.run.smithi040.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-25T01:41:44.838 INFO:teuthology.orchestra.run.smithi040.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-25T01:41:44.864 INFO:teuthology.orchestra.run.smithi040.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-04-25T01:41:44.889 INFO:teuthology.orchestra.run.smithi040.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-25T01:41:44.914 INFO:teuthology.orchestra.run.smithi040.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-25T01:41:44.940 INFO:teuthology.orchestra.run.smithi040.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-25T01:41:44.965 INFO:teuthology.orchestra.run.smithi040.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-04-25T01:41:44.990 INFO:teuthology.orchestra.run.smithi040.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 400 kB/s | 20 kB 00:00 2024-04-25T01:41:45.015 INFO:teuthology.orchestra.run.smithi040.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-25T01:41:45.066 INFO:teuthology.orchestra.run.smithi040.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-04-25T01:41:45.114 INFO:teuthology.orchestra.run.smithi031.stdout:(8/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 2.7 MB/s | 21 MB 00:07 2024-04-25T01:41:45.223 INFO:teuthology.orchestra.run.smithi031.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 233 kB/s | 25 kB 00:00 2024-04-25T01:41:45.298 INFO:teuthology.orchestra.run.smithi134.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 1.4 MB/s | 745 kB 00:00 2024-04-25T01:41:45.506 INFO:teuthology.orchestra.run.smithi134.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 785 kB/s | 163 kB 00:00 2024-04-25T01:41:45.615 INFO:teuthology.orchestra.run.smithi134.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.1 MB/s | 127 kB 00:00 2024-04-25T01:41:45.923 INFO:teuthology.orchestra.run.smithi134.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 1.6 MB/s | 501 kB 00:00 2024-04-25T01:41:45.966 INFO:teuthology.orchestra.run.smithi040.stdout:(148/152): ceph-test-19.0.0-2445.gd2c45d97.el9. 4.9 MB/s | 48 MB 00:09 2024-04-25T01:41:46.150 INFO:teuthology.orchestra.run.smithi040.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 7.4 MB/s | 1.3 MB 00:00 2024-04-25T01:41:46.292 INFO:teuthology.orchestra.run.smithi040.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 17 MB/s | 2.4 MB 00:00 2024-04-25T01:41:46.526 INFO:teuthology.orchestra.run.smithi040.stdout:(151/152): librados2-19.0.0-2445.gd2c45d97.el9. 2.2 MB/s | 3.4 MB 00:01 2024-04-25T01:41:48.732 INFO:teuthology.orchestra.run.smithi134.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 1.9 MB/s | 5.3 MB 00:02 2024-04-25T01:41:48.832 INFO:teuthology.orchestra.run.smithi134.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 453 kB/s | 45 kB 00:00 2024-04-25T01:41:48.934 INFO:teuthology.orchestra.run.smithi134.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.3 MB/s | 133 kB 00:00 2024-04-25T01:41:49.034 INFO:teuthology.orchestra.run.smithi134.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.6 MB/s | 162 kB 00:00 2024-04-25T01:41:49.243 INFO:teuthology.orchestra.run.smithi134.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 1.5 MB/s | 321 kB 00:00 2024-04-25T01:41:49.300 INFO:teuthology.orchestra.run.smithi040.stdout:(152/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 758 kB/s | 3.1 MB 00:04 2024-04-25T01:41:49.304 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:41:49.304 INFO:teuthology.orchestra.run.smithi040.stdout:Total 14 MB/s | 220 MB 00:16 2024-04-25T01:41:49.427 INFO:teuthology.orchestra.run.smithi134.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 1.6 MB/s | 299 kB 00:00 2024-04-25T01:41:49.527 INFO:teuthology.orchestra.run.smithi134.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 998 kB/s | 99 kB 00:00 2024-04-25T01:41:49.636 INFO:teuthology.orchestra.run.smithi134.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 787 kB/s | 85 kB 00:00 2024-04-25T01:41:50.364 INFO:teuthology.orchestra.run.smithi031.stdout:(10/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 1.8 MB/s | 10 MB 00:05 2024-04-25T01:41:50.413 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-25T01:41:50.548 INFO:teuthology.orchestra.run.smithi031.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 1.2 MB/s | 222 kB 00:00 2024-04-25T01:41:50.570 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-25T01:41:50.571 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-25T01:41:50.648 INFO:teuthology.orchestra.run.smithi031.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 271 kB/s | 27 kB 00:00 2024-04-25T01:41:50.748 INFO:teuthology.orchestra.run.smithi031.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 317 kB/s | 32 kB 00:00 2024-04-25T01:41:50.853 INFO:teuthology.orchestra.run.smithi134.stdout:(26/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 2.5 MB/s | 3.1 MB 00:01 2024-04-25T01:41:51.020 INFO:teuthology.orchestra.run.smithi134.stdout:(27/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 1.7 MB/s | 17 MB 00:09 2024-04-25T01:41:51.045 INFO:teuthology.orchestra.run.smithi134.stdout:(28/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 894 kB/s | 171 kB 00:00 2024-04-25T01:41:51.112 INFO:teuthology.orchestra.run.smithi134.stdout:(29/152): ceph-grafana-dashboards-19.0.0-2445.g 347 kB/s | 32 kB 00:00 2024-04-25T01:41:51.115 INFO:teuthology.orchestra.run.smithi031.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 2.0 MB/s | 745 kB 00:00 2024-04-25T01:41:51.145 INFO:teuthology.orchestra.run.smithi134.stdout:(30/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.4 MB/s | 141 kB 00:00 2024-04-25T01:41:51.299 INFO:teuthology.orchestra.run.smithi031.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 891 kB/s | 163 kB 00:00 2024-04-25T01:41:51.407 INFO:teuthology.orchestra.run.smithi031.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.1 MB/s | 127 kB 00:00 2024-04-25T01:41:51.749 INFO:teuthology.orchestra.run.smithi031.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 1.4 MB/s | 501 kB 00:00 2024-04-25T01:41:52.199 INFO:teuthology.orchestra.run.smithi031.stdout:(18/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 1.6 MB/s | 17 MB 00:10 2024-04-25T01:41:52.283 INFO:teuthology.orchestra.run.smithi031.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 544 kB/s | 45 kB 00:00 2024-04-25T01:41:52.358 INFO:teuthology.orchestra.run.smithi031.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.7 MB/s | 133 kB 00:00 2024-04-25T01:41:52.440 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-25T01:41:52.440 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-25T01:41:52.467 INFO:teuthology.orchestra.run.smithi031.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.5 MB/s | 162 kB 00:00 2024-04-25T01:41:52.629 INFO:teuthology.orchestra.run.smithi134.stdout:(31/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 5.5 MB/s | 48 MB 00:08 2024-04-25T01:41:52.642 INFO:teuthology.orchestra.run.smithi031.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 1.8 MB/s | 321 kB 00:00 2024-04-25T01:41:52.779 INFO:teuthology.orchestra.run.smithi134.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 1.6 MB/s | 249 kB 00:00 2024-04-25T01:41:52.792 INFO:teuthology.orchestra.run.smithi031.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 1.9 MB/s | 299 kB 00:00 2024-04-25T01:41:52.854 INFO:teuthology.orchestra.run.smithi134.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 660 kB/s | 49 kB 00:00 2024-04-25T01:41:52.876 INFO:teuthology.orchestra.run.smithi031.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.2 MB/s | 99 kB 00:00 2024-04-25T01:41:52.904 INFO:teuthology.orchestra.run.smithi134.stdout:(34/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 2.0 MB/s | 3.5 MB 00:01 2024-04-25T01:41:52.930 INFO:teuthology.orchestra.run.smithi134.stdout:(35/152): ceph-prometheus-alerts-19.0.0-2445.gd 202 kB/s | 15 kB 00:00 2024-04-25T01:41:52.943 INFO:teuthology.orchestra.run.smithi031.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.2 MB/s | 85 kB 00:00 2024-04-25T01:41:53.089 INFO:teuthology.orchestra.run.smithi134.stdout:(36/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 1.4 MB/s | 264 kB 00:00 2024-04-25T01:41:53.139 INFO:teuthology.orchestra.run.smithi134.stdout:(37/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 3.6 MB/s | 762 kB 00:00 2024-04-25T01:41:53.223 INFO:teuthology.orchestra.run.smithi134.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 3.6 MB/s | 7.4 MB 00:02 2024-04-25T01:41:53.248 INFO:teuthology.orchestra.run.smithi134.stdout:(39/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 115 kB/s | 18 kB 00:00 2024-04-25T01:41:53.315 INFO:teuthology.orchestra.run.smithi134.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 458 kB/s | 80 kB 00:00 2024-04-25T01:41:53.373 INFO:teuthology.orchestra.run.smithi134.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 195 kB 00:00 2024-04-25T01:41:53.399 INFO:teuthology.orchestra.run.smithi134.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-04-25T01:41:53.440 INFO:teuthology.orchestra.run.smithi134.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-04-25T01:41:53.466 INFO:teuthology.orchestra.run.smithi134.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-25T01:41:53.483 INFO:teuthology.orchestra.run.smithi134.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-25T01:41:53.508 INFO:teuthology.orchestra.run.smithi134.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-25T01:41:53.533 INFO:teuthology.orchestra.run.smithi134.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T01:41:53.567 INFO:teuthology.orchestra.run.smithi134.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 6.3 MB/s | 1.2 MB 00:00 2024-04-25T01:41:53.877 INFO:teuthology.orchestra.run.smithi031.stdout:(26/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 3.3 MB/s | 3.1 MB 00:00 2024-04-25T01:41:53.952 INFO:teuthology.orchestra.run.smithi031.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.2 MB/s | 171 kB 00:00 2024-04-25T01:41:54.001 INFO:teuthology.orchestra.run.smithi134.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 247 kB/s | 107 kB 00:00 2024-04-25T01:41:54.002 INFO:teuthology.orchestra.run.smithi031.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 639 kB/s | 32 kB 00:00 2024-04-25T01:41:54.069 INFO:teuthology.orchestra.run.smithi031.stdout:(29/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 2.3 MB/s | 5.3 MB 00:02 2024-04-25T01:41:54.094 INFO:teuthology.orchestra.run.smithi031.stdout:(30/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.5 MB/s | 141 kB 00:00 2024-04-25T01:41:54.109 INFO:teuthology.orchestra.run.smithi134.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 199 kB/s | 21 kB 00:00 2024-04-25T01:41:54.193 INFO:teuthology.orchestra.run.smithi134.stdout:(51/152): automake-1.16.2-8.el9.noarch.rpm 1.0 MB/s | 689 kB 00:00 2024-04-25T01:41:54.310 INFO:teuthology.orchestra.run.smithi134.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-04-25T01:41:54.335 INFO:teuthology.orchestra.run.smithi134.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 33 kB 00:00 2024-04-25T01:41:54.377 INFO:teuthology.orchestra.run.smithi134.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 313 kB 00:00 2024-04-25T01:41:54.428 INFO:teuthology.orchestra.run.smithi134.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 198 kB/s | 18 kB 00:00 2024-04-25T01:41:54.478 INFO:teuthology.orchestra.run.smithi134.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-04-25T01:41:54.536 INFO:teuthology.orchestra.run.smithi134.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-04-25T01:41:54.587 INFO:teuthology.orchestra.run.smithi134.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-04-25T01:41:54.962 INFO:teuthology.orchestra.run.smithi134.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 1.5 MB/s | 662 kB 00:00 2024-04-25T01:41:54.987 INFO:teuthology.orchestra.run.smithi134.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 1.4 MB/s | 582 kB 00:00 2024-04-25T01:41:55.062 INFO:teuthology.orchestra.run.smithi134.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-04-25T01:41:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 472 kB/s | 47 kB 00:00 2024-04-25T01:41:55.196 INFO:teuthology.orchestra.run.smithi134.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-25T01:41:55.246 INFO:teuthology.orchestra.run.smithi134.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-04-25T01:41:55.295 INFO:teuthology.orchestra.run.smithi031.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 2.9 MB/s | 3.5 MB 00:01 2024-04-25T01:41:55.305 INFO:teuthology.orchestra.run.smithi134.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 325 kB/s | 35 kB 00:00 2024-04-25T01:41:55.453 INFO:teuthology.orchestra.run.smithi031.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 1.5 MB/s | 249 kB 00:00 2024-04-25T01:41:55.520 INFO:teuthology.orchestra.run.smithi031.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 741 kB/s | 49 kB 00:00 2024-04-25T01:41:55.612 INFO:teuthology.orchestra.run.smithi031.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 167 kB/s | 15 kB 00:00 2024-04-25T01:41:55.695 INFO:teuthology.orchestra.run.smithi031.stdout:(35/152): ceph-mgr-diskprediction-local-19.0.0- 4.6 MB/s | 7.4 MB 00:01 2024-04-25T01:41:55.788 INFO:teuthology.orchestra.run.smithi031.stdout:(36/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 1.5 MB/s | 264 kB 00:00 2024-04-25T01:41:55.789 INFO:teuthology.orchestra.run.smithi134.stdout:(66/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.0 MB/s | 3.0 MB 00:01 2024-04-25T01:41:55.813 INFO:teuthology.orchestra.run.smithi031.stdout:(37/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 6.4 MB/s | 762 kB 00:00 2024-04-25T01:41:55.838 INFO:teuthology.orchestra.run.smithi031.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 364 kB/s | 18 kB 00:00 2024-04-25T01:41:55.947 INFO:teuthology.orchestra.run.smithi134.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-25T01:41:56.089 INFO:teuthology.orchestra.run.smithi134.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 6.3 MB/s | 911 kB 00:00 2024-04-25T01:41:56.105 INFO:teuthology.orchestra.run.smithi031.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 275 kB/s | 80 kB 00:00 2024-04-25T01:41:56.214 INFO:teuthology.orchestra.run.smithi134.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-04-25T01:41:56.230 INFO:teuthology.orchestra.run.smithi031.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 195 kB 00:00 2024-04-25T01:41:56.339 INFO:teuthology.orchestra.run.smithi031.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-04-25T01:41:56.364 INFO:teuthology.orchestra.run.smithi031.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-04-25T01:41:56.423 INFO:teuthology.orchestra.run.smithi031.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-25T01:41:56.465 INFO:teuthology.orchestra.run.smithi031.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 9.8 MB/s | 1.2 MB 00:00 2024-04-25T01:41:56.490 INFO:teuthology.orchestra.run.smithi031.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-25T01:41:56.524 INFO:teuthology.orchestra.run.smithi031.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-25T01:41:56.582 INFO:teuthology.orchestra.run.smithi031.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-04-25T01:41:56.614 INFO:teuthology.orchestra.run.smithi134.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-04-25T01:41:57.357 INFO:teuthology.orchestra.run.smithi031.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 827 kB/s | 689 kB 00:00 2024-04-25T01:41:57.458 INFO:teuthology.orchestra.run.smithi031.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.0 MB/s | 107 kB 00:00 2024-04-25T01:41:57.616 INFO:teuthology.orchestra.run.smithi031.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 136 kB/s | 21 kB 00:00 2024-04-25T01:41:57.657 INFO:teuthology.orchestra.run.smithi134.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 2.0 MB/s | 4.7 MB 00:02 2024-04-25T01:41:57.741 INFO:teuthology.orchestra.run.smithi031.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 2.5 MB/s | 313 kB 00:00 2024-04-25T01:41:57.782 INFO:teuthology.orchestra.run.smithi134.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 701 kB/s | 87 kB 00:00 2024-04-25T01:41:57.867 INFO:teuthology.orchestra.run.smithi031.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 262 kB/s | 33 kB 00:00 2024-04-25T01:41:58.298 INFO:teuthology.orchestra.run.smithi134.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 614 kB/s | 1.0 MB 00:01 2024-04-25T01:41:58.434 INFO:teuthology.orchestra.run.smithi031.stdout:(53/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 3.7 MB/s | 48 MB 00:13 2024-04-25T01:41:58.448 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-25T01:41:58.529 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:41:58.569 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:41:58.622 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:41:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:41:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:41:58.732 INFO:teuthology.orchestra.run.smithi134.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 564 kB/s | 244 kB 00:00 2024-04-25T01:41:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:41:58.734 INFO:teuthology.orchestra.run.smithi031.stdout:(54/152): bison-3.7.4-5.el9.x86_64.rpm 438 kB/s | 943 kB 00:02 2024-04-25T01:41:58.759 INFO:teuthology.orchestra.run.smithi031.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 56 kB/s | 18 kB 00:00 2024-04-25T01:41:58.851 INFO:teuthology.orchestra.run.smithi031.stdout:(56/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.0 MB/s | 3.0 MB 00:00 2024-04-25T01:41:58.876 INFO:teuthology.orchestra.run.smithi031.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 78 kB/s | 11 kB 00:00 2024-04-25T01:41:58.960 INFO:teuthology.orchestra.run.smithi031.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-04-25T01:41:59.077 INFO:teuthology.orchestra.run.smithi031.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-04-25T01:41:59.081 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:41:59.082 INFO:teuthology.orchestra.run.smithi134.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 709 kB/s | 248 kB 00:00 2024-04-25T01:41:59.093 INFO:teuthology.orchestra.run.smithi031.stdout:(60/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 479 kB/s | 160 kB 00:00 2024-04-25T01:41:59.178 INFO:teuthology.orchestra.run.smithi031.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-04-25T01:41:59.191 INFO:teuthology.orchestra.run.smithi134.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 237 kB/s | 26 kB 00:00 2024-04-25T01:41:59.198 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:41:59.202 INFO:teuthology.orchestra.run.smithi031.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 436 kB/s | 47 kB 00:00 2024-04-25T01:41:59.286 INFO:teuthology.orchestra.run.smithi031.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-25T01:41:59.291 INFO:teuthology.orchestra.run.smithi134.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 192 kB/s | 19 kB 00:00 2024-04-25T01:41:59.339 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:41:59.374 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:41:59.378 INFO:teuthology.orchestra.run.smithi031.stdout:(64/152): librdkafka-1.6.1-102.el9.x86_64.rpm 1.3 MB/s | 662 kB 00:00 2024-04-25T01:41:59.403 INFO:teuthology.orchestra.run.smithi031.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 940 kB/s | 188 kB 00:00 2024-04-25T01:41:59.420 INFO:teuthology.orchestra.run.smithi031.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 263 kB/s | 35 kB 00:00 2024-04-25T01:41:59.467 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:41:59.504 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:41:59.566 INFO:teuthology.orchestra.run.smithi134.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 626 kB/s | 172 kB 00:00 2024-04-25T01:41:59.664 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:41:59.683 INFO:teuthology.orchestra.run.smithi134.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 297 kB/s | 35 kB 00:00 2024-04-25T01:41:59.709 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:41:59.737 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:41:59.842 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:41:59.871 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:41:59.934 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:41:59.955 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:41:59.976 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:42:00.020 INFO:teuthology.orchestra.run.smithi031.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 2.6 MB/s | 1.6 MB 00:00 2024-04-25T01:42:00.162 INFO:teuthology.orchestra.run.smithi031.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 6.3 MB/s | 911 kB 00:00 2024-04-25T01:42:00.174 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:42:00.206 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:42:00.262 INFO:teuthology.orchestra.run.smithi031.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-04-25T01:42:00.318 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:42:00.422 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:42:00.463 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:42:00.499 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:42:00.575 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:42:00.614 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:42:00.658 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:42:00.726 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:42:00.799 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:42:00.850 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:42:00.879 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:42:00.923 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:42:01.001 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:42:01.050 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:42:01.094 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:42:01.122 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:42:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:42:01.245 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:42:01.275 INFO:teuthology.orchestra.run.smithi134.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 1.1 MB/s | 5.8 MB 00:05 2024-04-25T01:42:01.331 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:42:01.342 INFO:teuthology.orchestra.run.smithi134.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 1.7 MB/s | 6.0 MB 00:03 2024-04-25T01:42:01.459 INFO:teuthology.orchestra.run.smithi134.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 222 kB 00:00 2024-04-25T01:42:01.478 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:42:01.533 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:42:01.568 INFO:teuthology.orchestra.run.smithi134.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 712 kB/s | 77 kB 00:00 2024-04-25T01:42:01.643 INFO:teuthology.orchestra.run.smithi134.stdout:(84/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 682 kB/s | 250 kB 00:00 2024-04-25T01:42:01.676 INFO:teuthology.orchestra.run.smithi134.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 389 kB/s | 42 kB 00:00 2024-04-25T01:42:01.785 INFO:teuthology.orchestra.run.smithi134.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-04-25T01:42:01.894 INFO:teuthology.orchestra.run.smithi134.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-04-25T01:42:01.930 INFO:teuthology.orchestra.run.smithi031.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 1.9 MB/s | 4.7 MB 00:02 2024-04-25T01:42:02.019 INFO:teuthology.orchestra.run.smithi134.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 408 kB/s | 51 kB 00:00 2024-04-25T01:42:02.077 INFO:teuthology.orchestra.run.smithi134.stdout:(89/152): python3-protobuf-3.14.0-13.el9.noarch 620 kB/s | 269 kB 00:00 2024-04-25T01:42:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:42:02.136 INFO:teuthology.orchestra.run.smithi134.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 459 kB/s | 54 kB 00:00 2024-04-25T01:42:02.197 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:42:02.231 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:42:02.245 INFO:teuthology.orchestra.run.smithi134.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 384 kB/s | 42 kB 00:00 2024-04-25T01:42:02.274 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:42:02.311 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:42:02.353 INFO:teuthology.orchestra.run.smithi134.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 408 kB/s | 44 kB 00:00 2024-04-25T01:42:02.354 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:42:02.363 INFO:teuthology.orchestra.run.smithi031.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-04-25T01:42:02.470 INFO:teuthology.orchestra.run.smithi134.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-04-25T01:42:02.471 INFO:teuthology.orchestra.run.smithi031.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 808 kB/s | 87 kB 00:00 2024-04-25T01:42:02.516 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:42:02.573 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:42:02.579 INFO:teuthology.orchestra.run.smithi134.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 331 kB/s | 36 kB 00:00 2024-04-25T01:42:02.629 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:42:02.679 INFO:teuthology.orchestra.run.smithi134.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 474 kB/s | 47 kB 00:00 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:42:02.740 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:42:02.777 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:42:02.788 INFO:teuthology.orchestra.run.smithi134.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 305 kB 00:00 2024-04-25T01:42:02.825 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:42:02.859 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:42:02.896 INFO:teuthology.orchestra.run.smithi134.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 534 kB/s | 58 kB 00:00 2024-04-25T01:42:02.903 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:42:02.945 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:42:03.030 INFO:teuthology.orchestra.run.smithi134.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 703 kB/s | 94 kB 00:00 2024-04-25T01:42:03.088 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:42:03.133 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:42:03.147 INFO:teuthology.orchestra.run.smithi134.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 546 kB/s | 64 kB 00:00 2024-04-25T01:42:03.163 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:42:03.206 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:42:03.230 INFO:teuthology.orchestra.run.smithi031.stdout:(73/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 1.2 MB/s | 4.4 MB 00:03 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi134.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 359 kB/s | 45 kB 00:00 2024-04-25T01:42:03.303 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:42:03.338 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:42:03.397 INFO:teuthology.orchestra.run.smithi134.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-04-25T01:42:03.512 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:42:03.545 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:42:03.569 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:42:03.597 INFO:teuthology.orchestra.run.smithi031.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 666 kB/s | 244 kB 00:00 2024-04-25T01:42:03.623 INFO:teuthology.orchestra.run.smithi134.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-04-25T01:42:03.649 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:42:03.787 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:42:03.955 INFO:teuthology.orchestra.run.smithi031.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 692 kB/s | 248 kB 00:00 2024-04-25T01:42:03.978 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:42:04.072 INFO:teuthology.orchestra.run.smithi031.stdout:(76/152): postgresql-server-13.14-1.el9.x86_64. 1.5 MB/s | 5.8 MB 00:03 2024-04-25T01:42:04.108 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:42:04.173 INFO:teuthology.orchestra.run.smithi134.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 997 kB/s | 548 kB 00:00 2024-04-25T01:42:04.214 INFO:teuthology.orchestra.run.smithi031.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 99 kB/s | 26 kB 00:00 2024-04-25T01:42:04.239 INFO:teuthology.orchestra.run.smithi031.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 115 kB/s | 19 kB 00:00 2024-04-25T01:42:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:42:04.331 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi031.stdout:(79/152): python3-babel-2.9.1-2.el9.noarch.rpm 3.2 MB/s | 6.0 MB 00:01 2024-04-25T01:42:04.365 INFO:teuthology.orchestra.run.smithi031.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 278 kB/s | 35 kB 00:00 2024-04-25T01:42:04.371 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:42:04.400 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:42:04.434 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:42:04.468 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:42:04.512 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:42:04.515 INFO:teuthology.orchestra.run.smithi031.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 573 kB/s | 172 kB 00:00 2024-04-25T01:42:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:42:04.641 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:42:04.673 INFO:teuthology.orchestra.run.smithi134.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 618 kB/s | 309 kB 00:00 2024-04-25T01:42:04.692 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:42:04.724 INFO:teuthology.orchestra.run.smithi031.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 699 kB/s | 250 kB 00:00 2024-04-25T01:42:04.729 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:42:04.787 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:42:04.840 INFO:teuthology.orchestra.run.smithi134.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 117 kB/s | 19 kB 00:00 2024-04-25T01:42:04.874 INFO:teuthology.orchestra.run.smithi031.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 620 kB/s | 222 kB 00:00 2024-04-25T01:42:04.899 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:42:04.908 INFO:teuthology.orchestra.run.smithi031.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 420 kB/s | 77 kB 00:00 2024-04-25T01:42:04.958 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:42:05.058 INFO:teuthology.orchestra.run.smithi031.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 230 kB/s | 42 kB 00:00 2024-04-25T01:42:05.315 INFO:teuthology.orchestra.run.smithi134.stdout:(106/152): libarrow-9.0.0-10.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-04-25T01:42:05.324 INFO:teuthology.orchestra.run.smithi031.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 595 kB/s | 159 kB 00:00 2024-04-25T01:42:05.375 INFO:teuthology.orchestra.run.smithi031.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 577 kB/s | 269 kB 00:00 2024-04-25T01:42:05.482 INFO:teuthology.orchestra.run.smithi134.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 156 kB/s | 26 kB 00:00 2024-04-25T01:42:05.526 INFO:teuthology.orchestra.run.smithi031.stdout:(88/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.3 MB/s | 5.1 MB 00:01 2024-04-25T01:42:05.550 INFO:teuthology.orchestra.run.smithi031.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 291 kB/s | 51 kB 00:00 2024-04-25T01:42:05.626 INFO:teuthology.orchestra.run.smithi031.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 537 kB/s | 54 kB 00:00 2024-04-25T01:42:05.699 INFO:teuthology.orchestra.run.smithi134.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 224 kB/s | 49 kB 00:00 2024-04-25T01:42:05.726 INFO:teuthology.orchestra.run.smithi031.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 417 kB/s | 42 kB 00:00 2024-04-25T01:42:05.736 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:42:05.778 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:42:05.812 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:42:05.826 INFO:teuthology.orchestra.run.smithi031.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 442 kB/s | 44 kB 00:00 2024-04-25T01:42:05.851 INFO:teuthology.orchestra.run.smithi031.stdout:(93/152): python3-pyasn1-modules-0.4.8-6.el9.no 530 kB/s | 279 kB 00:00 2024-04-25T01:42:05.918 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:42:05.924 INFO:teuthology.orchestra.run.smithi134.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 299 kB/s | 67 kB 00:00 2024-04-25T01:42:05.935 INFO:teuthology.orchestra.run.smithi031.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-04-25T01:42:05.960 INFO:teuthology.orchestra.run.smithi031.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 330 kB/s | 36 kB 00:00 2024-04-25T01:42:05.971 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:42:06.000 INFO:teuthology.orchestra.run.smithi134.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-25T01:42:06.035 INFO:teuthology.orchestra.run.smithi031.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 473 kB/s | 47 kB 00:00 2024-04-25T01:42:06.063 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:42:06.106 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:42:06.136 INFO:teuthology.orchestra.run.smithi031.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 579 kB/s | 58 kB 00:00 2024-04-25T01:42:06.236 INFO:teuthology.orchestra.run.smithi031.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 937 kB/s | 94 kB 00:00 2024-04-25T01:42:06.292 INFO:teuthology.orchestra.run.smithi134.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-04-25T01:42:06.292 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:42:06.320 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:42:06.336 INFO:teuthology.orchestra.run.smithi031.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 638 kB/s | 64 kB 00:00 2024-04-25T01:42:06.367 INFO:teuthology.orchestra.run.smithi134.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-04-25T01:42:06.419 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:42:06.425 INFO:teuthology.orchestra.run.smithi134.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 746 kB/s | 43 kB 00:00 2024-04-25T01:42:06.437 INFO:teuthology.orchestra.run.smithi031.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 447 kB/s | 45 kB 00:00 2024-04-25T01:42:06.462 INFO:teuthology.orchestra.run.smithi031.stdout:(101/152): socat-1.7.4.1-5.el9.x86_64.rpm 608 kB/s | 305 kB 00:00 2024-04-25T01:42:06.484 INFO:teuthology.orchestra.run.smithi134.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 552 kB/s | 32 kB 00:00 2024-04-25T01:42:06.537 INFO:teuthology.orchestra.run.smithi031.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-04-25T01:42:06.551 INFO:teuthology.orchestra.run.smithi134.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 226 kB/s | 15 kB 00:00 2024-04-25T01:42:06.626 INFO:teuthology.orchestra.run.smithi134.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-25T01:42:06.702 INFO:teuthology.orchestra.run.smithi134.stdout:(117/152): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-25T01:42:06.869 INFO:teuthology.orchestra.run.smithi134.stdout:(118/152): python3-numpy-1.20.1-5.el9.x86_64.rp 726 kB/s | 5.1 MB 00:07 2024-04-25T01:42:06.969 INFO:teuthology.orchestra.run.smithi134.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.1 MB/s | 303 kB 00:00 2024-04-25T01:42:07.036 INFO:teuthology.orchestra.run.smithi134.stdout:(120/152): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-04-25T01:42:07.109 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:42:07.224 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:42:07.246 INFO:teuthology.orchestra.run.smithi031.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 774 kB/s | 548 kB 00:00 2024-04-25T01:42:07.306 INFO:teuthology.orchestra.run.smithi134.stdout:(121/152): python3-grpcio-1.46.7-10.el9.x86_64. 7.6 MB/s | 2.0 MB 00:00 2024-04-25T01:42:07.330 INFO:teuthology.orchestra.run.smithi134.stdout:(122/152): python3-colorama-0.4.4-12.el9.noarch 77 kB/s | 35 kB 00:00 2024-04-25T01:42:07.428 INFO:teuthology.orchestra.run.smithi134.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-25T01:42:07.456 INFO:teuthology.orchestra.run.smithi134.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 85 kB/s | 11 kB 00:00 2024-04-25T01:42:07.489 INFO:teuthology.orchestra.run.smithi031.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-04-25T01:42:07.588 INFO:teuthology.orchestra.run.smithi031.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 195 kB/s | 19 kB 00:00 2024-04-25T01:42:07.604 INFO:teuthology.orchestra.run.smithi134.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 101 kB/s | 18 kB 00:00 2024-04-25T01:42:07.629 INFO:teuthology.orchestra.run.smithi134.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 132 kB/s | 23 kB 00:00 2024-04-25T01:42:07.671 INFO:teuthology.orchestra.run.smithi134.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 290 kB/s | 19 kB 00:00 2024-04-25T01:42:07.705 INFO:teuthology.orchestra.run.smithi031.stdout:(106/152): protobuf-compiler-3.14.0-13.el9.x86_ 695 kB/s | 863 kB 00:01 2024-04-25T01:42:07.722 INFO:teuthology.orchestra.run.smithi134.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-04-25T01:42:07.772 INFO:teuthology.orchestra.run.smithi134.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-04-25T01:42:07.839 INFO:teuthology.orchestra.run.smithi134.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 396 kB/s | 46 kB 00:00 2024-04-25T01:42:07.872 INFO:teuthology.orchestra.run.smithi134.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 787 kB/s | 79 kB 00:00 2024-04-25T01:42:07.980 INFO:teuthology.orchestra.run.smithi031.stdout:(107/152): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-25T01:42:07.997 INFO:teuthology.orchestra.run.smithi134.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 364 kB/s | 58 kB 00:00 2024-04-25T01:42:08.064 INFO:teuthology.orchestra.run.smithi031.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 72 kB/s | 26 kB 00:00 2024-04-25T01:42:08.081 INFO:teuthology.orchestra.run.smithi031.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-04-25T01:42:08.123 INFO:teuthology.orchestra.run.smithi134.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-04-25T01:42:08.181 INFO:teuthology.orchestra.run.smithi031.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-25T01:42:08.248 INFO:teuthology.orchestra.run.smithi134.stdout:(134/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 719 kB/s | 90 kB 00:00 2024-04-25T01:42:08.256 INFO:teuthology.orchestra.run.smithi031.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 351 kB/s | 67 kB 00:00 2024-04-25T01:42:08.273 INFO:teuthology.orchestra.run.smithi134.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 60 kB/s | 16 kB 00:00 2024-04-25T01:42:08.298 INFO:teuthology.orchestra.run.smithi031.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-04-25T01:42:08.390 INFO:teuthology.orchestra.run.smithi031.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 475 kB/s | 43 kB 00:00 2024-04-25T01:42:08.482 INFO:teuthology.orchestra.run.smithi031.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 352 kB/s | 32 kB 00:00 2024-04-25T01:42:08.567 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:42:08.574 INFO:teuthology.orchestra.run.smithi031.stdout:(115/152): python3-asyncssh-2.13.2-5.el9.noarch 1.7 MB/s | 548 kB 00:00 2024-04-25T01:42:08.591 INFO:teuthology.orchestra.run.smithi031.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-04-25T01:42:08.649 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:42:08.665 INFO:teuthology.orchestra.run.smithi134.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 68 kB/s | 28 kB 00:00 2024-04-25T01:42:08.674 INFO:teuthology.orchestra.run.smithi031.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-25T01:42:08.677 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:42:08.699 INFO:teuthology.orchestra.run.smithi134.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 72 kB/s | 31 kB 00:00 2024-04-25T01:42:08.710 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:42:08.716 INFO:teuthology.orchestra.run.smithi031.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-25T01:42:08.775 INFO:teuthology.orchestra.run.smithi031.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-04-25T01:42:08.817 INFO:teuthology.orchestra.run.smithi031.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-04-25T01:42:08.875 INFO:teuthology.orchestra.run.smithi031.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-25T01:42:08.967 INFO:teuthology.orchestra.run.smithi031.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-25T01:42:09.039 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:42:09.067 INFO:teuthology.orchestra.run.smithi031.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-25T01:42:09.098 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:42:09.118 INFO:teuthology.orchestra.run.smithi031.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 6.8 MB/s | 2.0 MB 00:00 2024-04-25T01:42:09.168 INFO:teuthology.orchestra.run.smithi031.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-04-25T01:42:09.190 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:42:09.218 INFO:teuthology.orchestra.run.smithi031.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-25T01:42:09.220 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:42:09.257 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:42:09.269 INFO:teuthology.orchestra.run.smithi031.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-04-25T01:42:09.319 INFO:teuthology.orchestra.run.smithi031.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-25T01:42:09.386 INFO:teuthology.orchestra.run.smithi031.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-04-25T01:42:09.419 INFO:teuthology.orchestra.run.smithi031.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 461 kB/s | 46 kB 00:00 2024-04-25T01:42:09.487 INFO:teuthology.orchestra.run.smithi031.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 785 kB/s | 79 kB 00:00 2024-04-25T01:42:09.520 INFO:teuthology.orchestra.run.smithi031.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-04-25T01:42:09.587 INFO:teuthology.orchestra.run.smithi031.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-04-25T01:42:09.621 INFO:teuthology.orchestra.run.smithi031.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-25T01:42:09.688 INFO:teuthology.orchestra.run.smithi031.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 897 kB/s | 90 kB 00:00 2024-04-25T01:42:09.730 INFO:teuthology.orchestra.run.smithi031.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-04-25T01:42:09.754 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:42:09.785 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:42:09.788 INFO:teuthology.orchestra.run.smithi031.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-04-25T01:42:09.830 INFO:teuthology.orchestra.run.smithi031.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-25T01:42:09.864 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:42:09.888 INFO:teuthology.orchestra.run.smithi031.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 590 kB/s | 59 kB 00:00 2024-04-25T01:42:09.894 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:42:09.930 INFO:teuthology.orchestra.run.smithi031.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-04-25T01:42:09.989 INFO:teuthology.orchestra.run.smithi031.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-04-25T01:42:10.031 INFO:teuthology.orchestra.run.smithi031.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-25T01:42:10.090 INFO:teuthology.orchestra.run.smithi031.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 895 kB/s | 90 kB 00:00 2024-04-25T01:42:10.148 INFO:teuthology.orchestra.run.smithi031.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-25T01:42:10.190 INFO:teuthology.orchestra.run.smithi031.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-04-25T01:42:10.249 INFO:teuthology.orchestra.run.smithi031.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-25T01:42:10.324 INFO:teuthology.orchestra.run.smithi031.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-25T01:42:10.682 INFO:teuthology.orchestra.run.smithi031.stdout:(148/152): librados2-19.0.0-2445.gd2c45d97.el9. 7.8 MB/s | 3.4 MB 00:00 2024-04-25T01:42:10.949 INFO:teuthology.orchestra.run.smithi031.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 5.1 MB/s | 1.3 MB 00:00 2024-04-25T01:42:11.150 INFO:teuthology.orchestra.run.smithi031.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-04-25T01:42:11.200 INFO:teuthology.orchestra.run.smithi031.stdout:(151/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 3.6 MB/s | 3.1 MB 00:00 2024-04-25T01:42:11.641 INFO:teuthology.orchestra.run.smithi134.stdout:(138/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.7 MB/s | 16 MB 00:09 2024-04-25T01:42:11.891 INFO:teuthology.orchestra.run.smithi134.stdout:(139/152): python3-tempora-5.0.0-2.el9.noarch.r 143 kB/s | 36 kB 00:00 2024-04-25T01:42:12.008 INFO:teuthology.orchestra.run.smithi134.stdout:(140/152): python3-typing-extensions-4.4.0-2.el 436 kB/s | 51 kB 00:00 2024-04-25T01:42:12.183 INFO:teuthology.orchestra.run.smithi134.stdout:(141/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-04-25T01:42:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:42:12.308 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:42:12.434 INFO:teuthology.orchestra.run.smithi134.stdout:(142/152): python3-websocket-client-1.2.3-2.el9 359 kB/s | 90 kB 00:00 2024-04-25T01:42:12.784 INFO:teuthology.orchestra.run.smithi134.stdout:(143/152): python3-werkzeug-2.0.3-3.el9.1.noarc 1.2 MB/s | 427 kB 00:00 2024-04-25T01:42:12.851 INFO:teuthology.orchestra.run.smithi134.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 300 kB/s | 20 kB 00:00 2024-04-25T01:42:12.880 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:42:12.920 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:42:12.934 INFO:teuthology.orchestra.run.smithi134.stdout:(145/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-25T01:42:13.009 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:42:13.009 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:42:13.009 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:42:13.009 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:13.037 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:42:13.093 INFO:teuthology.orchestra.run.smithi134.stdout:(146/152): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-25T01:42:13.626 INFO:teuthology.orchestra.run.smithi134.stdout:(147/152): librados2-19.0.0-2445.gd2c45d97.el9. 6.4 MB/s | 3.4 MB 00:00 2024-04-25T01:42:13.818 INFO:teuthology.orchestra.run.smithi134.stdout:(148/152): python3-routes-2.5.1-5.el9.noarch.rp 37 kB/s | 188 kB 00:05 2024-04-25T01:42:13.885 INFO:teuthology.orchestra.run.smithi134.stdout:(149/152): python3-rsa-4.9-2.el9.noarch.rpm 11 kB/s | 59 kB 00:05 2024-04-25T01:42:13.994 INFO:teuthology.orchestra.run.smithi134.stdout:(150/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 8.5 MB/s | 3.1 MB 00:00 2024-04-25T01:42:14.036 INFO:teuthology.orchestra.run.smithi134.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 6.2 MB/s | 1.3 MB 00:00 2024-04-25T01:42:14.094 INFO:teuthology.orchestra.run.smithi134.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-04-25T01:42:14.100 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:42:14.100 INFO:teuthology.orchestra.run.smithi134.stdout:Total 5.8 MB/s | 220 MB 00:37 2024-04-25T01:42:15.206 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-25T01:42:15.352 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-25T01:42:15.352 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-25T01:42:17.122 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-25T01:42:17.122 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-25T01:42:23.030 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-25T01:42:23.123 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:42:23.161 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:42:23.223 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:42:23.257 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:42:23.283 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:42:23.333 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:42:23.706 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:42:23.767 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:42:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:42:23.910 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:42:24.001 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:42:24.030 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:42:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:42:24.244 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:42:24.272 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:42:24.375 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:42:24.398 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:42:24.461 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:42:24.482 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:42:24.505 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:42:24.649 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:42:24.682 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:42:24.788 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:42:24.893 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:42:24.939 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:42:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:42:24.950 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2024-04-25T01:42:24.951 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2024-04-25T01:42:24.951 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2024-04-25T01:42:24.951 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2024-04-25T01:42:24.951 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2024-04-25T01:42:24.951 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2024-04-25T01:42:24.951 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2024-04-25T01:42:24.951 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:24.982 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:42:25.047 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:42:25.090 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:42:25.118 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:42:25.133 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:42:25.134 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:42:25.134 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:25.134 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:42:25.134 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:25.165 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:42:25.194 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:42:25.262 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:42:25.316 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:42:25.346 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:42:25.381 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:42:25.484 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:42:25.519 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:42:25.548 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:42:25.618 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:42:25.662 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:42:25.736 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:42:25.872 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:42:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:42:26.452 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:42:26.536 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:42:26.565 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:42:26.599 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:42:26.637 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:42:26.671 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:42:26.678 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:42:26.698 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:42:26.699 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:26.699 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:42:26.699 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:42:26.699 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:42:26.699 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:26.842 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:42:26.890 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:42:26.930 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:42:26.944 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:42:26.953 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:42:26.953 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:26.953 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:42:26.954 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:42:26.954 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:42:26.954 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:27.009 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:42:27.040 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:42:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:42:27.123 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:42:27.159 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:42:27.202 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:42:27.253 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:42:27.388 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:42:27.414 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:42:27.438 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:42:27.438 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:27.438 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:42:27.438 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:42:27.438 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:42:27.438 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:27.441 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:42:27.472 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:42:27.513 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:42:27.554 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:42:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:42:27.612 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:42:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:42:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:42:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:42:27.690 INFO:teuthology.orchestra.run.smithi031.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 752 kB/s | 16 MB 00:22 2024-04-25T01:42:27.696 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:42:27.696 INFO:teuthology.orchestra.run.smithi031.stdout:Total 4.3 MB/s | 220 MB 00:51 2024-04-25T01:42:27.704 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:42:27.729 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:42:27.796 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:42:27.872 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:42:27.922 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:42:27.932 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:42:27.961 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:42:27.989 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:42:28.013 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:42:28.023 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:42:28.062 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:42:28.094 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:42:28.111 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:42:28.124 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:42:28.166 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:42:28.200 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:42:28.241 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:42:28.255 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:42:28.276 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:42:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:42:28.331 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:42:28.354 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:42:28.358 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:42:28.398 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:42:28.429 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:42:28.475 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:42:28.574 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:42:28.622 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:42:28.749 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:42:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:42:28.813 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:42:28.832 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-25T01:42:28.878 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:42:28.913 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:42:28.996 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-25T01:42:28.996 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-25T01:42:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:42:29.394 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:42:29.428 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:42:29.527 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:42:29.577 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:42:29.666 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:42:29.722 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:42:29.898 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:42:29.919 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:42:29.997 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:42:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:42:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:42:30.636 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:42:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:42:30.874 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-25T01:42:30.875 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-25T01:42:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:42:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:42:31.207 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:42:31.378 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:42:31.409 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:42:31.433 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:42:31.434 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:31.434 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:42:31.434 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:42:31.434 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:42:31.434 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:42:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:42:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:42:31.651 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:42:31.960 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:42:32.056 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:42:32.084 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:42:32.117 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:42:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:42:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:42:32.538 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:42:32.568 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:42:32.599 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:42:32.604 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:42:32.637 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:42:32.661 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:42:32.662 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:32.662 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:42:32.662 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:42:32.662 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:42:32.662 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:32.705 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:42:32.727 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:42:32.728 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:32.728 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:42:32.728 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:32.772 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:42:32.795 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:42:32.795 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:32.796 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:42:32.796 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:42:32.796 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:42:32.796 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:33.076 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:42:33.109 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:42:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:42:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:42:33.138 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:33.138 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:42:33.138 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:42:33.138 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:42:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:33.179 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:42:33.209 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:42:35.470 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:42:35.505 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:42:36.042 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:42:36.084 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:42:36.175 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:42:36.176 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:42:36.176 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:42:36.176 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:42:36.210 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:42:36.884 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-25T01:42:36.969 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:42:37.010 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:42:37.062 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:42:37.095 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:42:37.120 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:42:37.164 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:42:37.511 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:42:37.612 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:42:37.720 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:42:37.755 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:42:37.848 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:42:37.884 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:42:37.911 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:42:37.952 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:42:37.994 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:42:38.046 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:42:38.060 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:42:38.098 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:42:38.125 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:42:38.234 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:42:38.250 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:42:38.260 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:42:38.290 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:42:38.319 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:42:38.322 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:42:38.344 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:42:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:42:38.569 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:42:38.602 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:42:38.712 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:42:38.809 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:42:38.851 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:42:38.886 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:42:38.950 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:42:38.962 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:42:38.984 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:42:39.004 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:42:39.026 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:42:39.079 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:42:39.111 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:42:39.139 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:42:39.202 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:42:39.291 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:42:39.335 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:42:39.414 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:42:39.454 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:42:39.498 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:42:39.526 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:42:39.601 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:42:39.707 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:42:39.786 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:42:39.915 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:42:39.933 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:42:39.986 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:42:40.005 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:42:40.047 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:42:40.134 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:42:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:42:40.168 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:42:40.244 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:42:40.276 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:42:40.277 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:42:40.371 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:42:40.470 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:42:40.552 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:42:40.642 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:42:40.675 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:42:40.719 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:42:40.757 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:42:40.799 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:42:40.956 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:42:41.018 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:42:41.074 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:42:41.149 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:42:41.185 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:42:41.222 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:42:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:42:41.304 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:42:41.348 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:42:41.390 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:42:41.543 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:42:41.595 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:42:41.625 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:42:41.667 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:42:41.766 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:42:41.799 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:42:41.848 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:42:41.873 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:42:41.897 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:42:41.978 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:42:42.114 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:42:42.155 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:42:42.183 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:42:42.217 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:42:42.257 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:42:42.296 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:42:42.326 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:42:42.361 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:42:42.394 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:42:42.436 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:42:42.470 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:42:42.552 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:42:42.609 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:42:42.640 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:42:42.688 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:42:42.800 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:42:42.851 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:42:43.634 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:42:43.680 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:42:43.713 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:42:43.820 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:42:43.873 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:42:43.964 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:42:43.999 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:42:44.175 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:42:44.196 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:42:44.276 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:42:44.967 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:42:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:42:46.017 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:42:46.018 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:42:46.018 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:42:46.018 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:42:46.018 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:42:46.018 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:42:46.018 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:42:46.018 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:42:46.018 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:42:46.018 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:42:46.018 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:42:46.018 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:42:46.019 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:42:46.020 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:42:46.020 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:42:46.020 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:42:46.020 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:42:46.020 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:42:46.034 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:42:46.034 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:42:46.034 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:42:46.034 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:42:46.034 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:42:46.034 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:42:46.034 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:42:46.034 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:42:46.035 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:42:46.036 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:42:46.037 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:42:46.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:42:46.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:42:46.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:42:46.041 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:42:46.042 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:42:46.043 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:42:46.402 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:42:46.490 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:42:46.519 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:42:46.551 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:42:46.879 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:42:46.922 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:42:47.005 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:42:47.035 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:42:47.072 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:42:47.568 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:42:47.592 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:42:47.594 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:42:47.594 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-04-25T01:42:47.594 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-04-25T01:42:47.594 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-04-25T01:42:47.595 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-04-25T01:42:47.595 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-04-25T01:42:47.595 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-04-25T01:42:47.595 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-04-25T01:42:47.595 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:42:47.663 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:42:47.693 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:42:47.772 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:42:47.787 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:42:47.787 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:47.787 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:42:47.787 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:42:48.234 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:42:48.234 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:48.234 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.235 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:48.236 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:42:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.238 INFO:teuthology.orchestra.run.smithi040.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:42:48.239 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:42:48.240 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:42:48.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:42:48.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:42:48.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:48.244 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:42:48.245 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:42:48.245 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:42:48.245 INFO:teuthology.orchestra.run.smithi040.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:42:48.245 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:42:48.245 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:42:48.245 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:42:48.245 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-25T01:42:48.245 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-25T01:42:48.650 DEBUG:teuthology.parallel:result is None 2024-04-25T01:42:49.267 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:42:49.292 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:42:49.292 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:49.292 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:42:49.293 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:42:49.293 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:42:49.293 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:42:49.517 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:42:49.540 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:42:49.540 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:49.540 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:42:49.540 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:42:49.540 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:42:49.541 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:42:49.985 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:42:50.007 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:42:50.008 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:50.008 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:42:50.008 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:42:50.008 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:42:50.008 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:42:50.109 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:42:50.114 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:42:50.140 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:42:50.172 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:42:50.227 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:42:50.414 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:42:50.471 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:42:50.563 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:42:50.652 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:42:50.713 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:42:50.760 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:42:50.794 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:42:50.835 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:42:50.854 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:42:50.854 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:42:50.854 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:42:50.854 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:42:50.867 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:42:50.878 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:42:50.895 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:42:51.276 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:42:51.301 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:42:51.333 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:42:51.395 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:42:51.433 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:42:52.463 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:42:52.501 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:42:53.464 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:42:53.493 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:42:53.596 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:42:53.747 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:42:53.787 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:42:53.810 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:42:53.810 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:53.810 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:42:53.810 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:42:53.810 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:42:53.810 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:42:53.845 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:42:53.857 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:42:53.954 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:42:54.002 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:42:54.896 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:42:54.931 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:42:54.954 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:42:54.954 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:54.954 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:42:54.954 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:42:54.954 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:42:54.954 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:42:54.999 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:42:55.021 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:42:55.021 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:55.021 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:42:55.022 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:42:55.066 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:42:55.089 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:42:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:42:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:42:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:42:55.089 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:42:55.392 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:42:55.415 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:42:55.415 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:55.415 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:42:55.415 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:42:55.415 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:42:55.415 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:42:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:42:59.970 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:43:00.011 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:43:00.043 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:43:00.070 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:43:00.265 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:43:00.308 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:43:00.338 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:43:00.930 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:43:00.969 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:43:00.989 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:43:01.832 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:43:01.906 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:43:01.948 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:43:02.030 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:43:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:43:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:43:02.161 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:43:02.194 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:43:02.194 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:43:02.287 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:43:02.351 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:43:02.736 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:43:02.736 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-04-25T01:43:02.736 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-04-25T01:43:02.737 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-04-25T01:43:02.737 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-04-25T01:43:02.737 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-04-25T01:43:02.737 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-04-25T01:43:02.737 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-04-25T01:43:02.737 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:43:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:43:02.928 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:43:02.928 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:43:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:04.473 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:43:04.500 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:43:04.501 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:43:04.501 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:43:04.501 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:43:04.501 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:43:04.501 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:04.732 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:43:04.755 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:43:04.756 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:43:04.756 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:43:04.756 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:43:04.756 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:43:04.756 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:43:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:43:05.240 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:43:05.240 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:43:05.240 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:43:05.240 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:43:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:43:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:43:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:43:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:43:05.726 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:43:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:43:05.905 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:43:06.048 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:43:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:43:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:43:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:43:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:43:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:43:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:43:06.772 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:43:06.805 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:43:07.896 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:43:07.932 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:43:08.388 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:43:08.388 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:43:08.388 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:43:08.388 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:43:08.388 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:43:08.388 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:43:08.388 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:43:08.388 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:43:08.388 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:43:08.389 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:43:08.390 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:43:08.391 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:43:08.392 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:43:08.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:43:08.394 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:43:08.395 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:43:08.395 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:43:08.395 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:43:08.395 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:43:08.395 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:43:08.395 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:43:08.395 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:43:08.395 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:43:08.395 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:43:08.395 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:43:08.395 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:43:08.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:43:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:43:08.398 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:43:08.398 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:43:08.398 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:43:08.398 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:43:08.398 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:43:08.398 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:43:08.398 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:43:08.398 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:43:08.398 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:43:08.398 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:43:08.399 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:43:08.399 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:43:08.399 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:43:08.399 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:43:08.399 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:43:08.399 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:43:08.399 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:43:08.399 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:43:08.399 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:43:08.399 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:43:08.400 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:43:08.400 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:43:08.947 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:43:08.982 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:43:09.091 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:43:09.261 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:43:09.292 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:43:09.316 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:43:09.317 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:43:09.317 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:43:09.317 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:43:09.317 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:43:09.317 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:43:09.379 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:43:09.481 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:43:09.534 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:43:10.473 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:43:10.473 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:43:10.474 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-04-25T01:43:10.474 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.474 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.474 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:43:10.474 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:43:10.474 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-25T01:43:10.474 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:43:10.474 INFO:teuthology.orchestra.run.smithi134.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:43:10.474 INFO:teuthology.orchestra.run.smithi134.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:43:10.474 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:43:10.474 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:10.475 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:10.476 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.477 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.478 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:43:10.478 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:43:10.478 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:43:10.478 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:43:10.478 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:43:10.478 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.478 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.478 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:43:10.478 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.478 INFO:teuthology.orchestra.run.smithi134.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:43:10.478 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:43:10.479 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:43:10.480 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:43:10.481 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:43:10.482 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:43:10.482 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:43:10.482 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:43:10.482 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:43:10.482 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:43:10.482 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:43:10.482 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:43:10.482 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:43:10.482 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:43:10.482 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:43:10.483 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:43:10.484 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:43:10.485 INFO:teuthology.orchestra.run.smithi134.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:43:10.485 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:43:10.485 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:43:10.485 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:43:10.485 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:43:10.485 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:43:10.485 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:43:10.485 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:43:10.486 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.486 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.486 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:10.486 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:43:10.486 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:43:10.486 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:43:10.486 INFO:teuthology.orchestra.run.smithi134.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:43:10.486 INFO:teuthology.orchestra.run.smithi134.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:43:10.486 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:43:10.487 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:43:10.487 INFO:teuthology.orchestra.run.smithi134.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:43:10.487 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:43:10.487 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-25T01:43:10.512 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:43:10.536 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:43:10.536 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:43:10.536 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:43:10.536 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:43:10.536 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:43:10.536 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:10.579 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:43:10.602 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:43:10.602 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:43:10.602 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:43:10.603 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:10.655 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:43:10.678 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:43:10.678 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:43:10.678 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:43:10.679 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:43:10.679 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:43:10.679 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:10.910 DEBUG:teuthology.parallel:result is None 2024-04-25T01:43:10.997 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:43:11.021 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:43:11.021 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:43:11.021 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:43:11.021 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:43:11.021 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:43:11.021 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:43:16.168 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:43:16.213 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:43:16.243 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:43:16.840 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:43:16.881 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:43:16.902 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:43:17.813 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:43:17.912 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:43:17.955 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:43:18.041 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:43:18.083 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:43:18.083 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:43:18.168 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:43:18.201 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:43:18.201 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:43:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:43:18.369 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:43:24.100 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:43:24.101 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:43:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:43:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:43:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:43:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:43:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:43:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:43:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:43:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:43:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:43:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:43:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:43:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:43:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:43:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:43:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:43:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:43:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:43:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:43:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:43:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:43:26.140 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:43:26.141 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:26.142 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:43:26.143 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:43:26.144 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:43:26.145 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:43:26.146 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:43:26.147 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:43:26.148 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:43:26.149 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:43:26.150 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:43:26.151 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:43:26.151 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:43:26.151 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:43:26.151 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:43:26.151 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:43:26.151 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:43:26.151 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.151 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.151 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:43:26.151 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:43:26.151 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:43:26.152 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:43:26.152 INFO:teuthology.orchestra.run.smithi031.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:43:26.152 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:43:26.152 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:43:26.152 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:43:26.152 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:43:26.152 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-25T01:43:26.511 DEBUG:teuthology.parallel:result is None 2024-04-25T01:43:26.512 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:43:26.730 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:43:26.753 INFO:teuthology.orchestra.run.smithi031.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:43:26.753 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:43:26.754 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:43:26.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:43:26.971 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:43:26.997 INFO:teuthology.orchestra.run.smithi040.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:43:26.997 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:43:26.997 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:43:26.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:43:27.143 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:43:27.168 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:43:27.168 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:43:27.168 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:43:27.170 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T01:43:27.170 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:43:27.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:43:27.199 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-25T01:43:27.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:43:27.232 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T01:43:27.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:43:27.262 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T01:43:27.262 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:43:27.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:43:27.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:43:27.354 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-25T01:43:27.354 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:43:27.383 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:43:27.452 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T01:43:27.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:43:27.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:43:27.540 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T01:43:27.540 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:43:27.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:43:27.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:43:27.629 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-25T01:43:27.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:43:27.670 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:43:27.737 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T01:43:27.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:43:27.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:43:27.824 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T01:43:27.825 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:43:27.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:43:27.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:43:27.913 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-25T01:43:27.913 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:43:27.938 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:43:28.002 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T01:43:28.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:43:28.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:43:28.090 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T01:43:28.211 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-25T01:43:28.211 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:43:28.212 INFO:tasks.cephadm:Cluster fsid is 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:28.212 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T01:43:28.212 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.31', 'mon.b': '172.21.15.40', 'mon.c': '172.21.15.134'} 2024-04-25T01:43:28.212 INFO:tasks.cephadm:First mon is mon.a on smithi031 2024-04-25T01:43:28.213 INFO:tasks.cephadm:First mgr is x 2024-04-25T01:43:28.213 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T01:43:28.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname $(hostname -s) 2024-04-25T01:43:28.237 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname $(hostname -s) 2024-04-25T01:43:28.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo hostname $(hostname -s) 2024-04-25T01:43:28.286 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T01:43:28.286 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:43:28.428 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54987/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 22:37:46.422067', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54981/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 17:00:00.872117', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-25T01:43:28.645 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 d2c45d977142e024a59b68a8f1dce8f0b9840551 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:43:28.646 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm 2024-04-25T01:43:28.646 DEBUG:teuthology.orchestra.run.smithi031:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:43:29.237 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:43 /home/ubuntu/cephtest/cephadm 2024-04-25T01:43:29.238 DEBUG:teuthology.orchestra.run.smithi040:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:43:29.638 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:43 /home/ubuntu/cephtest/cephadm 2024-04-25T01:43:29.639 DEBUG:teuthology.orchestra.run.smithi134:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:43:30.133 INFO:teuthology.orchestra.run.smithi134.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:43 /home/ubuntu/cephtest/cephadm 2024-04-25T01:43:30.134 DEBUG:teuthology.orchestra.run.smithi031:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:43:30.157 DEBUG:teuthology.orchestra.run.smithi040:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:43:30.180 DEBUG:teuthology.orchestra.run.smithi134:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:43:30.258 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 on all hosts... 2024-04-25T01:43:30.258 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:43:30.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:43:30.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:43:31.755 INFO:teuthology.orchestra.run.smithi040.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:43:31.894 INFO:teuthology.orchestra.run.smithi134.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:43:31.963 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:43:59.589 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-25T01:43:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:43:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:43:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "repo_digests": [ 2024-04-25T01:43:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:43:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-04-25T01:43:59.590 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-25T01:43:59.846 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-25T01:43:59.846 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:43:59.847 INFO:teuthology.orchestra.run.smithi040.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:43:59.847 INFO:teuthology.orchestra.run.smithi040.stdout: "repo_digests": [ 2024-04-25T01:43:59.847 INFO:teuthology.orchestra.run.smithi040.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:43:59.847 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-25T01:43:59.847 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-25T01:44:08.250 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:44:08.251 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:44:08.251 INFO:teuthology.orchestra.run.smithi031.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:44:08.251 INFO:teuthology.orchestra.run.smithi031.stdout: "repo_digests": [ 2024-04-25T01:44:08.251 INFO:teuthology.orchestra.run.smithi031.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:44:08.251 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T01:44:08.251 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:44:08.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph 2024-04-25T01:44:08.316 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph 2024-04-25T01:44:08.349 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph 2024-04-25T01:44:08.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /etc/ceph 2024-04-25T01:44:08.405 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /etc/ceph 2024-04-25T01:44:08.434 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /etc/ceph 2024-04-25T01:44:08.461 INFO:tasks.cephadm:Writing seed config... 2024-04-25T01:44:08.463 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T01:44:08.463 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T01:44:08.463 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T01:44:08.463 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T01:44:08.463 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T01:44:08.463 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T01:44:08.463 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T01:44:08.463 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T01:44:08.463 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T01:44:08.463 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T01:44:08.463 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T01:44:08.464 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T01:44:08.465 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T01:44:08.466 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T01:44:08.466 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T01:44:08.466 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T01:44:08.466 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T01:44:08.466 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T01:44:08.466 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-25T01:44:08.467 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:44:08.467 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T01:44:08.482 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 36afbea4-02a5-11ef-bc93-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T01:44:08.483 DEBUG:teuthology.orchestra.run.smithi031:mon.a> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T01:44:08.525 DEBUG:teuthology.orchestra.run.smithi031:mgr.x> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T01:44:08.568 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T01:44:08.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 -v bootstrap --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.31 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:44:08.763 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:44:08.764 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551', '-v', 'bootstrap', '--fsid', '36afbea4-02a5-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.31', '--skip-admin-label'] 2024-04-25T01:44:08.764 INFO:teuthology.orchestra.run.smithi031.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T01:44:08.764 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying podman|docker is present... 2024-04-25T01:44:08.789 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T01:44:08.789 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying lvm2 is present... 2024-04-25T01:44:08.790 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying time synchronization is in place... 2024-04-25T01:44:08.797 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T01:44:08.797 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T01:44:08.803 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T01:44:08.803 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T01:44:08.810 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2024-04-25T01:44:08.816 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2024-04-25T01:44:08.816 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2024-04-25T01:44:08.816 INFO:teuthology.orchestra.run.smithi031.stdout:Repeating the final host check... 2024-04-25T01:44:08.840 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T01:44:08.840 INFO:teuthology.orchestra.run.smithi031.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T01:44:08.840 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl is present 2024-04-25T01:44:08.840 INFO:teuthology.orchestra.run.smithi031.stdout:lvcreate is present 2024-04-25T01:44:08.846 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T01:44:08.846 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T01:44:08.852 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T01:44:08.852 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T01:44:08.859 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2024-04-25T01:44:08.865 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2024-04-25T01:44:08.865 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2024-04-25T01:44:08.865 INFO:teuthology.orchestra.run.smithi031.stdout:Host looks OK 2024-04-25T01:44:08.865 INFO:teuthology.orchestra.run.smithi031.stdout:Cluster fsid: 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:08.865 INFO:teuthology.orchestra.run.smithi031.stdout:Acquiring lock 140216198909472 on /run/cephadm/36afbea4-02a5-11ef-bc93-c7b262605968.lock 2024-04-25T01:44:08.866 INFO:teuthology.orchestra.run.smithi031.stdout:Lock 140216198909472 acquired on /run/cephadm/36afbea4-02a5-11ef-bc93-c7b262605968.lock 2024-04-25T01:44:08.866 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 3300 ... 2024-04-25T01:44:08.866 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 6789 ... 2024-04-25T01:44:08.866 INFO:teuthology.orchestra.run.smithi031.stdout:Base mon IP(s) is [172.21.15.31:3300, 172.21.15.31:6789], mon addrv is [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-04-25T01:44:08.869 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.31 metric 100 2024-04-25T01:44:08.870 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.31 metric 100 2024-04-25T01:44:08.872 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T01:44:08.872 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-25T01:44:08.872 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-25T01:44:08.875 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T01:44:08.875 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T01:44:08.875 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T01:44:08.875 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-25T01:44:08.875 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:caba/64 scope link noprefixroute 2024-04-25T01:44:08.875 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T01:44:08.876 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2024-04-25T01:44:08.876 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2024-04-25T01:44:08.876 INFO:teuthology.orchestra.run.smithi031.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T01:44:08.876 INFO:teuthology.orchestra.run.smithi031.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T01:44:08.877 INFO:teuthology.orchestra.run.smithi031.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:44:09.343 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T01:44:09.343 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:44:09.343 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T01:44:09.343 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:58a99251f41e8a467bafa0ec849428709568906ca732f10972304f37056be6c5 2024-04-25T01:44:09.343 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T01:44:09.343 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying config sha256:f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T01:44:09.343 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T01:44:10.314 INFO:teuthology.orchestra.run.smithi031.stdout:ceph: stdout ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T01:44:10.314 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph version: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T01:44:10.314 INFO:teuthology.orchestra.run.smithi031.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T01:44:11.103 INFO:teuthology.orchestra.run.smithi031.stdout:stat: stdout 167 167 2024-04-25T01:44:11.103 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial keys... 2024-04-25T01:44:12.074 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQBrtSlmOKOyHBAAjPX/ndDZOuK4+y1HQrdqjA== 2024-04-25T01:44:12.928 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQBstSlmzCBzGBAAQdNghCrxsX8DcNrEAquqEA== 2024-04-25T01:44:13.871 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQBttSlm5QJrEBAA+u/JFaaLY2xpxfIPr8C8fA== 2024-04-25T01:44:13.872 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial monmap... 2024-04-25T01:44:14.728 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T01:44:14.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T01:44:14.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:14.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T01:44:14.729 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool for a [v2:172.21.15.31:3300,v1:172.21.15.31:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T01:44:14.729 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = quincy 2024-04-25T01:44:14.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: set fsid to 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:14.729 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T01:44:14.729 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:44:14.729 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mon... 2024-04-25T01:44:15.712 INFO:teuthology.orchestra.run.smithi031.stdout:create mon.a on 2024-04-25T01:44:15.940 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T01:44:16.083 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T01:44:16.275 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-36afbea4-02a5-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-36afbea4-02a5-11ef-bc93-c7b262605968.target. 2024-04-25T01:44:16.276 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-36afbea4-02a5-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-36afbea4-02a5-11ef-bc93-c7b262605968.target. 2024-04-25T01:44:16.581 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.a 2024-04-25T01:44:16.581 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T01:44:16.757 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-36afbea4-02a5-11ef-bc93-c7b262605968.target.wants/ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-36afbea4-02a5-11ef-bc93-c7b262605968@.service. 2024-04-25T01:44:17.383 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:17 smithi031 podman[28814]: 2024-04-25 01:44:16.994808466 +0000 UTC m=+0.018754505 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:44:18.228 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:17 smithi031 podman[28814]: 2024-04-25 01:44:17.857902324 +0000 UTC m=+0.881848365 container create cd198426406da004e0e3acb246e42513fb6b5042e3d9729905be0f4f35245928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T01:44:20.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:19 smithi031 podman[28814]: 2024-04-25 01:44:19.847817886 +0000 UTC m=+2.871763928 container init cd198426406da004e0e3acb246e42513fb6b5042e3d9729905be0f4f35245928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T01:44:20.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:19 smithi031 podman[28814]: 2024-04-25 01:44:19.852262964 +0000 UTC m=+2.876209004 container start cd198426406da004e0e3acb246e42513fb6b5042e3d9729905be0f4f35245928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T01:44:20.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:19 smithi031 ceph-mon[28833]: mkfs 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:20.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:19 smithi031 ceph-mon[28833]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:44:20.823 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:44:20.823 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-25T01:44:20.832 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:44:20.832 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T01:44:20.832 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-25T01:44:20.832 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T01:44:20.832 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon to start... 2024-04-25T01:44:20.833 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon... 2024-04-25T01:44:21.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 bash[28814]: cd198426406da004e0e3acb246e42513fb6b5042e3d9729905be0f4f35245928 2024-04-25T01:44:21.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 systemd[1]: Started Ceph mon.a for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:44:21.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 ceph-mon[28833]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:44:21.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 ceph-mon[28833]: monmap epoch 1 2024-04-25T01:44:21.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 ceph-mon[28833]: fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:21.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 ceph-mon[28833]: last_changed 2024-04-25T01:44:14.223274+0000 2024-04-25T01:44:21.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 ceph-mon[28833]: created 2024-04-25T01:44:14.223274+0000 2024-04-25T01:44:21.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 ceph-mon[28833]: min_mon_release 19 (squid) 2024-04-25T01:44:21.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 ceph-mon[28833]: election_strategy: 1 2024-04-25T01:44:21.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 ceph-mon[28833]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T01:44:21.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 ceph-mon[28833]: fsmap 2024-04-25T01:44:21.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 ceph-mon[28833]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T01:44:21.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:20 smithi031 ceph-mon[28833]: mgrmap e1: no daemons active 2024-04-25T01:44:23.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:23 smithi031 ceph-mon[28833]: from='client.? 172.21.15.31:0/293079853' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T01:44:29.432 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T01:44:29.432 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout id: 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:29.432 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout services: 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 3s) 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout data: 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.456+0000 7fb31a196700 1 Processor -- start 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.456+0000 7fb31a196700 1 -- start start 2024-04-25T01:44:29.433 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.456+0000 7fb31a196700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb314107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.456+0000 7fb31a196700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb314108530 con 0x7fb314107bd0 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.456+0000 7fb3137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb314107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.456+0000 7fb3137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb314107ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52050/0 (socket says 172.21.15.31:52050) 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.456+0000 7fb3137fe700 1 -- 172.21.15.31:0/3507474190 learned_addr learned my addr 172.21.15.31:0/3507474190 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.457+0000 7fb3137fe700 1 -- 172.21.15.31:0/3507474190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3141086b0 con 0x7fb314107bd0 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.457+0000 7fb3137fe700 1 --2- 172.21.15.31:0/3507474190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb314107ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb3040098a0 tx=0x7fb304009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4769baebbff3d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.457+0000 7fb3127fc700 1 -- 172.21.15.31:0/3507474190 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb304004160 con 0x7fb314107bd0 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.457+0000 7fb3127fc700 1 -- 172.21.15.31:0/3507474190 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb304004300 con 0x7fb314107bd0 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.457+0000 7fb3127fc700 1 -- 172.21.15.31:0/3507474190 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3040036a0 con 0x7fb314107bd0 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.458+0000 7fb31a196700 1 -- 172.21.15.31:0/3507474190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 msgr2=0x7fb314107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.458+0000 7fb31a196700 1 --2- 172.21.15.31:0/3507474190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb314107ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb3040098a0 tx=0x7fb304009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.458+0000 7fb31a196700 1 -- 172.21.15.31:0/3507474190 shutdown_connections 2024-04-25T01:44:29.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.458+0000 7fb31a196700 1 --2- 172.21.15.31:0/3507474190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb314107ff0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.458+0000 7fb31a196700 1 -- 172.21.15.31:0/3507474190 >> 172.21.15.31:0/3507474190 conn(0x7fb3141031f0 msgr2=0x7fb314105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.458+0000 7fb31a196700 1 -- 172.21.15.31:0/3507474190 shutdown_connections 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.458+0000 7fb31a196700 1 -- 172.21.15.31:0/3507474190 wait complete. 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.458+0000 7fb31a196700 1 Processor -- start 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.459+0000 7fb31a196700 1 -- start start 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.459+0000 7fb31a196700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb31419b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.459+0000 7fb31a196700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb314109080 con 0x7fb314107bd0 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.459+0000 7fb3137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb31419b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.459+0000 7fb3137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb31419b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52060/0 (socket says 172.21.15.31:52060) 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.459+0000 7fb3137fe700 1 -- 172.21.15.31:0/293079853 learned_addr learned my addr 172.21.15.31:0/293079853 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.459+0000 7fb3137fe700 1 -- 172.21.15.31:0/293079853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31419b760 con 0x7fb314107bd0 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.459+0000 7fb3137fe700 1 --2- 172.21.15.31:0/293079853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb31419b220 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb304000c00 tx=0x7fb304004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.460+0000 7fb310ff9700 1 -- 172.21.15.31:0/293079853 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb304003db0 con 0x7fb314107bd0 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.460+0000 7fb310ff9700 1 -- 172.21.15.31:0/293079853 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb304011630 con 0x7fb314107bd0 2024-04-25T01:44:29.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.460+0000 7fb310ff9700 1 -- 172.21.15.31:0/293079853 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb304011950 con 0x7fb314107bd0 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.460+0000 7fb31a196700 1 -- 172.21.15.31:0/293079853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb31419b9f0 con 0x7fb314107bd0 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.460+0000 7fb31a196700 1 -- 172.21.15.31:0/293079853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb31419e6e0 con 0x7fb314107bd0 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.460+0000 7fb310ff9700 1 -- 172.21.15.31:0/293079853 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb304011af0 con 0x7fb314107bd0 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.460+0000 7fb310ff9700 1 -- 172.21.15.31:0/293079853 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb30401c450 con 0x7fb314107bd0 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.461+0000 7fb31a196700 1 -- 172.21.15.31:0/293079853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb314063890 con 0x7fb314107bd0 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.463+0000 7fb310ff9700 1 -- 172.21.15.31:0/293079853 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb304032020 con 0x7fb314107bd0 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.536+0000 7fb31a196700 1 -- 172.21.15.31:0/293079853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fb31402dcb0 con 0x7fb314107bd0 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.536+0000 7fb310ff9700 1 -- 172.21.15.31:0/293079853 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+310 (secure 0 0 0) 0x7fb304004260 con 0x7fb314107bd0 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.537+0000 7fb31a196700 1 -- 172.21.15.31:0/293079853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 msgr2=0x7fb31419b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.537+0000 7fb31a196700 1 --2- 172.21.15.31:0/293079853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb31419b220 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb304000c00 tx=0x7fb304004750 comp rx=0 tx=0).stop 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.537+0000 7fb31a196700 1 -- 172.21.15.31:0/293079853 shutdown_connections 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.537+0000 7fb31a196700 1 --2- 172.21.15.31:0/293079853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb314107bd0 0x7fb31419b220 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.537+0000 7fb31a196700 1 -- 172.21.15.31:0/293079853 >> 172.21.15.31:0/293079853 conn(0x7fb3141031f0 msgr2=0x7fb314191e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:29.436 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.537+0000 7fb31a196700 1 -- 172.21.15.31:0/293079853 shutdown_connections 2024-04-25T01:44:29.437 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.537+0000 7fb31a196700 1 -- 172.21.15.31:0/293079853 wait complete. 2024-04-25T01:44:29.437 INFO:teuthology.orchestra.run.smithi031.stdout:mon is available 2024-04-25T01:44:29.437 INFO:teuthology.orchestra.run.smithi031.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T01:44:30.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:30 smithi031 ceph-mon[28833]: from='client.? 172.21.15.31:0/3062808991' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T01:44:30.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:30 smithi031 ceph-mon[28833]: from='client.? 172.21.15.31:0/3062808991' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T01:44:30.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T01:44:30.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.254+0000 7fdee08af700 1 Processor -- start 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.254+0000 7fdee08af700 1 -- start start 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.255+0000 7fdee08af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.255+0000 7fdee08af700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc108530 con 0x7fdedc107bd0 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.255+0000 7fdeda59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.255+0000 7fdeda59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc107ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52062/0 (socket says 172.21.15.31:52062) 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.255+0000 7fdeda59c700 1 -- 172.21.15.31:0/436709408 learned_addr learned my addr 172.21.15.31:0/436709408 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.255+0000 7fdeda59c700 1 -- 172.21.15.31:0/436709408 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc1086b0 con 0x7fdedc107bd0 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.256+0000 7fdeda59c700 1 --2- 172.21.15.31:0/436709408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc107ff0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdec40098a0 tx=0x7fdec4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236b24bd10c5fd43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.256+0000 7fded959a700 1 -- 172.21.15.31:0/436709408 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdec4004160 con 0x7fdedc107bd0 2024-04-25T01:44:30.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.256+0000 7fded959a700 1 -- 172.21.15.31:0/436709408 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec4004300 con 0x7fdedc107bd0 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.256+0000 7fded959a700 1 -- 172.21.15.31:0/436709408 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdec40036a0 con 0x7fdedc107bd0 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.256+0000 7fdee08af700 1 -- 172.21.15.31:0/436709408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 msgr2=0x7fdedc107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.256+0000 7fdee08af700 1 --2- 172.21.15.31:0/436709408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc107ff0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdec40098a0 tx=0x7fdec4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.256+0000 7fdee08af700 1 -- 172.21.15.31:0/436709408 shutdown_connections 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.256+0000 7fdee08af700 1 --2- 172.21.15.31:0/436709408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc107ff0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.256+0000 7fdee08af700 1 -- 172.21.15.31:0/436709408 >> 172.21.15.31:0/436709408 conn(0x7fdedc1031f0 msgr2=0x7fdedc105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.256+0000 7fdee08af700 1 -- 172.21.15.31:0/436709408 shutdown_connections 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.256+0000 7fdee08af700 1 -- 172.21.15.31:0/436709408 wait complete. 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.257+0000 7fdee08af700 1 Processor -- start 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.257+0000 7fdee08af700 1 -- start start 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.258+0000 7fdee08af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc19b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.258+0000 7fdee08af700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc109080 con 0x7fdedc107bd0 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.258+0000 7fdeda59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc19b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.258+0000 7fdeda59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc19b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52068/0 (socket says 172.21.15.31:52068) 2024-04-25T01:44:30.825 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.258+0000 7fdeda59c700 1 -- 172.21.15.31:0/3062808991 learned_addr learned my addr 172.21.15.31:0/3062808991 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.258+0000 7fdeda59c700 1 -- 172.21.15.31:0/3062808991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc19b760 con 0x7fdedc107bd0 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.258+0000 7fdeda59c700 1 --2- 172.21.15.31:0/3062808991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc19b220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdec4000c00 tx=0x7fdec4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.258+0000 7fded37fe700 1 -- 172.21.15.31:0/3062808991 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdec4003db0 con 0x7fdedc107bd0 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.258+0000 7fded37fe700 1 -- 172.21.15.31:0/3062808991 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec4011630 con 0x7fdedc107bd0 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.258+0000 7fded37fe700 1 -- 172.21.15.31:0/3062808991 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdec4011950 con 0x7fdedc107bd0 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.258+0000 7fdee08af700 1 -- 172.21.15.31:0/3062808991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdedc19b9f0 con 0x7fdedc107bd0 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.259+0000 7fdee08af700 1 -- 172.21.15.31:0/3062808991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdedc19e6e0 con 0x7fdedc107bd0 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.259+0000 7fded37fe700 1 -- 172.21.15.31:0/3062808991 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdec4011af0 con 0x7fdedc107bd0 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.259+0000 7fded37fe700 1 -- 172.21.15.31:0/3062808991 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdec401c450 con 0x7fdedc107bd0 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.260+0000 7fdee08af700 1 -- 172.21.15.31:0/3062808991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdedc063890 con 0x7fdedc107bd0 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.262+0000 7fded37fe700 1 -- 172.21.15.31:0/3062808991 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fdec4032020 con 0x7fdedc107bd0 2024-04-25T01:44:30.826 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.333+0000 7fdee08af700 1 -- 172.21.15.31:0/3062808991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fdedc02dcb0 con 0x7fdedc107bd0 2024-04-25T01:44:30.827 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.336+0000 7fded37fe700 1 -- 172.21.15.31:0/3062808991 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7fdec4004260 con 0x7fdedc107bd0 2024-04-25T01:44:30.827 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.337+0000 7fdee08af700 1 -- 172.21.15.31:0/3062808991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 msgr2=0x7fdedc19b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:30.827 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.337+0000 7fdee08af700 1 --2- 172.21.15.31:0/3062808991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc19b220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdec4000c00 tx=0x7fdec4004750 comp rx=0 tx=0).stop 2024-04-25T01:44:30.827 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.337+0000 7fded37fe700 1 -- 172.21.15.31:0/3062808991 <== mon.0 v2:172.21.15.31:3300/0 8 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdec401c6a0 con 0x7fdedc107bd0 2024-04-25T01:44:30.827 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.337+0000 7fdee08af700 1 -- 172.21.15.31:0/3062808991 shutdown_connections 2024-04-25T01:44:30.827 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.337+0000 7fdee08af700 1 --2- 172.21.15.31:0/3062808991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdedc107bd0 0x7fdedc19b220 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:30.827 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.337+0000 7fdee08af700 1 -- 172.21.15.31:0/3062808991 >> 172.21.15.31:0/3062808991 conn(0x7fdedc1031f0 msgr2=0x7fdedc191e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:30.827 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.337+0000 7fdee08af700 1 -- 172.21.15.31:0/3062808991 shutdown_connections 2024-04-25T01:44:30.827 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:30.337+0000 7fdee08af700 1 -- 172.21.15.31:0/3062808991 wait complete. 2024-04-25T01:44:30.827 INFO:teuthology.orchestra.run.smithi031.stdout:Generating new minimal ceph.conf... 2024-04-25T01:44:31.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:31 smithi031 ceph-mon[28833]: from='client.? 172.21.15.31:0/3392430234' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:44:31.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.268+0000 7f8e2da6e700 1 Processor -- start 2024-04-25T01:44:31.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.268+0000 7f8e2da6e700 1 -- start start 2024-04-25T01:44:31.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.268+0000 7f8e2da6e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28105bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:31.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.268+0000 7f8e2da6e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8e28106180 con 0x7f8e281057b0 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.268+0000 7f8e277fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28105bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.268+0000 7f8e277fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28105bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52084/0 (socket says 172.21.15.31:52084) 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.268+0000 7f8e277fe700 1 -- 172.21.15.31:0/840744874 learned_addr learned my addr 172.21.15.31:0/840744874 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.269+0000 7f8e277fe700 1 -- 172.21.15.31:0/840744874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e28106300 con 0x7f8e281057b0 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.269+0000 7f8e277fe700 1 --2- 172.21.15.31:0/840744874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28105bb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8e180098a0 tx=0x7f8e18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e160cedbafd9d744 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.269+0000 7f8e267fc700 1 -- 172.21.15.31:0/840744874 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e18004160 con 0x7f8e281057b0 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.269+0000 7f8e267fc700 1 -- 172.21.15.31:0/840744874 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8e18004300 con 0x7f8e281057b0 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.269+0000 7f8e267fc700 1 -- 172.21.15.31:0/840744874 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e18011a80 con 0x7f8e281057b0 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.270+0000 7f8e2da6e700 1 -- 172.21.15.31:0/840744874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 msgr2=0x7f8e28105bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.270+0000 7f8e2da6e700 1 --2- 172.21.15.31:0/840744874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28105bb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8e180098a0 tx=0x7f8e18009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.270+0000 7f8e2da6e700 1 -- 172.21.15.31:0/840744874 shutdown_connections 2024-04-25T01:44:31.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.270+0000 7f8e2da6e700 1 --2- 172.21.15.31:0/840744874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28105bb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.270+0000 7f8e2da6e700 1 -- 172.21.15.31:0/840744874 >> 172.21.15.31:0/840744874 conn(0x7f8e28101300 msgr2=0x7f8e28103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.270+0000 7f8e2da6e700 1 -- 172.21.15.31:0/840744874 shutdown_connections 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.270+0000 7f8e2da6e700 1 -- 172.21.15.31:0/840744874 wait complete. 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.270+0000 7f8e2da6e700 1 Processor -- start 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e2da6e700 1 -- start start 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e2da6e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28108b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e2da6e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2810bee0 con 0x7f8e281057b0 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e277fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28108b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e277fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28108b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52100/0 (socket says 172.21.15.31:52100) 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e277fe700 1 -- 172.21.15.31:0/3392430234 learned_addr learned my addr 172.21.15.31:0/3392430234 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e277fe700 1 -- 172.21.15.31:0/3392430234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e28109090 con 0x7f8e281057b0 2024-04-25T01:44:31.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e277fe700 1 --2- 172.21.15.31:0/3392430234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28108b50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e18000c00 tx=0x7f8e18011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e24ff9700 1 -- 172.21.15.31:0/3392430234 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e180039f0 con 0x7f8e281057b0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e24ff9700 1 -- 172.21.15.31:0/3392430234 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8e18003b90 con 0x7f8e281057b0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e2da6e700 1 -- 172.21.15.31:0/3392430234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e28107280 con 0x7f8e281057b0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.271+0000 7f8e24ff9700 1 -- 172.21.15.31:0/3392430234 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e1801a740 con 0x7f8e281057b0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.272+0000 7f8e2da6e700 1 -- 172.21.15.31:0/3392430234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e281077c0 con 0x7f8e281057b0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.272+0000 7f8e24ff9700 1 -- 172.21.15.31:0/3392430234 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8e1802d3f0 con 0x7f8e281057b0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.272+0000 7f8e24ff9700 1 -- 172.21.15.31:0/3392430234 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8e18019700 con 0x7f8e281057b0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.272+0000 7f8e2da6e700 1 -- 172.21.15.31:0/3392430234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e28063890 con 0x7f8e281057b0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.275+0000 7f8e24ff9700 1 -- 172.21.15.31:0/3392430234 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8e18024af0 con 0x7f8e281057b0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.345+0000 7f8e2da6e700 1 -- 172.21.15.31:0/3392430234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f8e28107a70 con 0x7f8e281057b0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.345+0000 7f8e24ff9700 1 -- 172.21.15.31:0/3392430234 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f8e18032030 con 0x7f8e281057b0 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.346+0000 7f8e2da6e700 1 -- 172.21.15.31:0/3392430234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 msgr2=0x7f8e28108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:31.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.346+0000 7f8e2da6e700 1 --2- 172.21.15.31:0/3392430234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28108b50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e18000c00 tx=0x7f8e18011b00 comp rx=0 tx=0).stop 2024-04-25T01:44:31.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.346+0000 7f8e2da6e700 1 -- 172.21.15.31:0/3392430234 shutdown_connections 2024-04-25T01:44:31.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.346+0000 7f8e2da6e700 1 --2- 172.21.15.31:0/3392430234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e281057b0 0x7f8e28108b50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:31.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.346+0000 7f8e2da6e700 1 -- 172.21.15.31:0/3392430234 >> 172.21.15.31:0/3392430234 conn(0x7f8e28101300 msgr2=0x7f8e28101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:31.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.346+0000 7f8e2da6e700 1 -- 172.21.15.31:0/3392430234 shutdown_connections 2024-04-25T01:44:31.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.346+0000 7f8e2da6e700 1 -- 172.21.15.31:0/3392430234 wait complete. 2024-04-25T01:44:31.891 INFO:teuthology.orchestra.run.smithi031.stdout:Restarting the monitor... 2024-04-25T01:44:32.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:31 smithi031 systemd[1]: Stopping Ceph mon.a for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:44:32.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:32 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a[28829]: 2024-04-25T01:44:32.154+0000 7fe38c9cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:44:32.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:32 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a[28829]: 2024-04-25T01:44:32.154+0000 7fe38c9cc700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T01:44:32.642 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:32 smithi031 podman[29078]: 2024-04-25 01:44:32.380068353 +0000 UTC m=+0.318967195 container died cd198426406da004e0e3acb246e42513fb6b5042e3d9729905be0f4f35245928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T01:44:32.978 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:32 smithi031 podman[29078]: 2024-04-25 01:44:32.741801837 +0000 UTC m=+0.680700679 container cleanup cd198426406da004e0e3acb246e42513fb6b5042e3d9729905be0f4f35245928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-25T01:44:32.979 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:32 smithi031 bash[29078]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a 2024-04-25T01:44:33.384 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 podman[29092]: 2024-04-25 01:44:33.000990749 +0000 UTC m=+0.611359772 container remove cd198426406da004e0e3acb246e42513fb6b5042e3d9729905be0f4f35245928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T01:44:33.384 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 systemd[1]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T01:44:33.384 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 systemd[1]: Stopped Ceph mon.a for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:44:33.384 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 systemd[1]: Starting Ceph mon.a for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:44:33.645 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 podman[29186]: 2024-04-25 01:44:33.39336086 +0000 UTC m=+0.116892221 container create 9b79660ad9d9b4bb16bd84f0ee4d8b683d2ae6bb527324878cb032828214574b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T01:44:33.645 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 podman[29186]: 2024-04-25 01:44:33.30281694 +0000 UTC m=+0.026348302 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:44:33.645 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 podman[29186]: 2024-04-25 01:44:33.587247047 +0000 UTC m=+0.310778409 container init 9b79660ad9d9b4bb16bd84f0ee4d8b683d2ae6bb527324878cb032828214574b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-25T01:44:33.645 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 podman[29186]: 2024-04-25 01:44:33.590122371 +0000 UTC m=+0.313653735 container start 9b79660ad9d9b4bb16bd84f0ee4d8b683d2ae6bb527324878cb032828214574b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: pidfile_write: ignore empty --pid-file 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: load: jerasure load: lrc 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: RocksDB version: 7.9.2 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Git sha 0 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: DB SUMMARY 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: DB Session ID: 9CP827LGVBFV2FLONXPE 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: CURRENT file: CURRENT 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T01:44:33.646 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105057 ; 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.error_if_exists: 0 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.create_if_missing: 0 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.paranoid_checks: 1 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.env: 0x5587263f8c60 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.info_log: 0x558727f95aa0 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.statistics: (nil) 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.use_fsync: 0 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_log_file_size: 0 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T01:44:33.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.allow_fallocate: 1 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.use_direct_reads: 0 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.db_log_dir: 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.wal_dir: 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T01:44:33.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T01:44:33.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T01:44:33.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T01:44:33.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T01:44:33.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.write_buffer_manager: 0x558727f681e0 2024-04-25T01:44:33.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T01:44:33.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T01:44:33.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T01:44:33.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T01:44:33.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T01:44:33.649 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.unordered_write: 0 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.row_cache: None 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.wal_filter: None 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.two_write_queues: 0 2024-04-25T01:44:33.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.wal_compression: 0 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.atomic_flush: 0 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.log_readahead_size: 0 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T01:44:33.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_background_jobs: 2 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_background_compactions: -1 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_subcompactions: 1 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_open_files: -1 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T01:44:33.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_background_flushes: -1 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Compression algorithms supported: 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: kZSTD supported: 0 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: kXpressCompression supported: 0 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: kZlibCompression supported: 1 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: kSnappyCompression supported: 1 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: kLZ4Compression supported: 1 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: kBZip2Compression supported: 0 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T01:44:33.654 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.merge_operator: 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_filter: None 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558727f95ba0) 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks: 1 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_top_level_index_and_filter: 1 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_type: 0 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_index_type: 0 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_shortening: 1 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout: checksum: 4 2024-04-25T01:44:33.655 INFO:journalctl@ceph.mon.a.smithi031.stdout: no_block_cache: 0 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache: 0x558727f99090 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_name: BinnedLRUCache 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_options: 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: capacity : 536870912 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_shard_bits : 4 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: strict_capacity_limit : 0 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: high_pri_pool_ratio: 0.000 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_compressed: (nil) 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: persistent_cache: (nil) 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size: 4096 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size_deviation: 10 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_restart_interval: 16 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_block_restart_interval: 1 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: metadata_block_size: 4096 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: partition_filters: 0 2024-04-25T01:44:33.656 INFO:journalctl@ceph.mon.a.smithi031.stdout: use_delta_encoding: 1 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout: filter_policy: bloomfilter 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout: whole_key_filtering: 1 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout: verify_compression: 0 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout: read_amp_bytes_per_bit: 0 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout: format_version: 5 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_index_compression: 1 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_align: 0 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout: max_auto_readahead_size: 262144 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout: prepopulate_block_cache: 0 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout: initial_auto_readahead_size: 8192 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compression: NoCompression 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T01:44:33.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.num_levels: 7 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:44:33.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T01:44:33.659 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T01:44:33.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.table_properties_collectors: 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.inplace_update_support: 0 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.bloom_locality: 0 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.max_successive_merges: 0 2024-04-25T01:44:33.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.ttl: 2592000 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.enable_blob_files: false 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.min_blob_size: 0 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T01:44:33.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f1d56c28-9fed-4a39-a084-01ecd15f644c 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714009473619022, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714009473620130, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100011, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 332, "table_properties": {"data_size": 97925, "index_size": 304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14193, "raw_average_key_size": 49, "raw_value_size": 90225, "raw_average_value_size": 314, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714009473, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f1d56c28-9fed-4a39-a084-01ecd15f644c", "db_session_id": "9CP827LGVBFV2FLONXPE", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714009473620215, "job": 1, "event": "recovery_finished"} 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55872808a000 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: DB pointer 0x558728074000 2024-04-25T01:44:33.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** DB Stats ** 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: L0 2/0 99.50 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:44:33.664 INFO:journalctl@ceph.mon.a.smithi031.stdout: Sum 2/0 99.50 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T01:44:33.665 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative compaction: 0.00 GB write, 30.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval compaction: 0.00 GB write, 30.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache BinnedLRUCache@0x558727f99090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: starting mon.a rank 0 at public addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] at bind addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: mon.a@-1(???) e1 preinit fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: mon.a@-1(???).mds e1 new map 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: mon.a@-1(???).mds e1 print_map 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout: e1 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:44:33.666 INFO:journalctl@ceph.mon.a.smithi031.stdout: legacy client fscid: -1 2024-04-25T01:44:33.667 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T01:44:33.667 INFO:journalctl@ceph.mon.a.smithi031.stdout: No filesystems configured 2024-04-25T01:44:33.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:44:33.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:44:33.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:44:33.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:44:33.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T01:44:33.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 bash[29186]: 9b79660ad9d9b4bb16bd84f0ee4d8b683d2ae6bb527324878cb032828214574b 2024-04-25T01:44:33.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 systemd[1]: Started Ceph mon.a for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:44:33.667 INFO:teuthology.orchestra.run.smithi031.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T01:44:33.903 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:44:33.903 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: monmap epoch 1 2024-04-25T01:44:33.903 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:33.903 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: last_changed 2024-04-25T01:44:14.223274+0000 2024-04-25T01:44:33.904 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: created 2024-04-25T01:44:14.223274+0000 2024-04-25T01:44:33.904 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: min_mon_release 19 (squid) 2024-04-25T01:44:33.904 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: election_strategy: 1 2024-04-25T01:44:33.904 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T01:44:33.904 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: fsmap 2024-04-25T01:44:33.904 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T01:44:33.904 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:33 smithi031 ceph-mon[29204]: mgrmap e1: no daemons active 2024-04-25T01:44:35.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.102+0000 7fe831778700 1 Processor -- start 2024-04-25T01:44:35.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.102+0000 7fe831778700 1 -- start start 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.103+0000 7fe831778700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.103+0000 7fe831778700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe82c108520 con 0x7fe82c107bc0 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.103+0000 7fe82affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.103+0000 7fe82affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c107fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46602/0 (socket says 172.21.15.31:46602) 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.103+0000 7fe82affd700 1 -- 172.21.15.31:0/716456702 learned_addr learned my addr 172.21.15.31:0/716456702 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.103+0000 7fe82affd700 1 -- 172.21.15.31:0/716456702 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82c1086a0 con 0x7fe82c107bc0 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.104+0000 7fe82affd700 1 --2- 172.21.15.31:0/716456702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c107fe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0098a0 tx=0x7fe81c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efb0446c7c1efbb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.104+0000 7fe829ffb700 1 -- 172.21.15.31:0/716456702 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe81c004160 con 0x7fe82c107bc0 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.104+0000 7fe829ffb700 1 -- 172.21.15.31:0/716456702 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe81c004300 con 0x7fe82c107bc0 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.104+0000 7fe829ffb700 1 -- 172.21.15.31:0/716456702 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe81c011a80 con 0x7fe82c107bc0 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.104+0000 7fe831778700 1 -- 172.21.15.31:0/716456702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 msgr2=0x7fe82c107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.104+0000 7fe831778700 1 --2- 172.21.15.31:0/716456702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c107fe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0098a0 tx=0x7fe81c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.104+0000 7fe831778700 1 -- 172.21.15.31:0/716456702 shutdown_connections 2024-04-25T01:44:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.104+0000 7fe831778700 1 --2- 172.21.15.31:0/716456702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c107fe0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.104+0000 7fe831778700 1 -- 172.21.15.31:0/716456702 >> 172.21.15.31:0/716456702 conn(0x7fe82c1031c0 msgr2=0x7fe82c105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.104+0000 7fe831778700 1 -- 172.21.15.31:0/716456702 shutdown_connections 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.104+0000 7fe831778700 1 -- 172.21.15.31:0/716456702 wait complete. 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.105+0000 7fe831778700 1 Processor -- start 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.105+0000 7fe831778700 1 -- start start 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.105+0000 7fe831778700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c19baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.105+0000 7fe831778700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe82c109bc0 con 0x7fe82c107bc0 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.105+0000 7fe82affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c19baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.105+0000 7fe82affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c19baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46608/0 (socket says 172.21.15.31:46608) 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.105+0000 7fe82affd700 1 -- 172.21.15.31:0/3382343444 learned_addr learned my addr 172.21.15.31:0/3382343444 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.106+0000 7fe82affd700 1 -- 172.21.15.31:0/3382343444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82c19c030 con 0x7fe82c107bc0 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.106+0000 7fe82affd700 1 --2- 172.21.15.31:0/3382343444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c19baf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe81c006af0 tx=0x7fe81c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.106+0000 7fe813fff700 1 -- 172.21.15.31:0/3382343444 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe81c003930 con 0x7fe82c107bc0 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.106+0000 7fe831778700 1 -- 172.21.15.31:0/3382343444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe82c19c2c0 con 0x7fe82c107bc0 2024-04-25T01:44:35.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.106+0000 7fe813fff700 1 -- 172.21.15.31:0/3382343444 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe81c003ad0 con 0x7fe82c107bc0 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.106+0000 7fe813fff700 1 -- 172.21.15.31:0/3382343444 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe81c0195e0 con 0x7fe82c107bc0 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.106+0000 7fe831778700 1 -- 172.21.15.31:0/3382343444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe82c19efb0 con 0x7fe82c107bc0 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.106+0000 7fe813fff700 1 -- 172.21.15.31:0/3382343444 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe81c019c90 con 0x7fe82c107bc0 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.107+0000 7fe813fff700 1 -- 172.21.15.31:0/3382343444 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe81c024730 con 0x7fe82c107bc0 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.107+0000 7fe831778700 1 -- 172.21.15.31:0/3382343444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe82c063890 con 0x7fe82c107bc0 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.110+0000 7fe813fff700 1 -- 172.21.15.31:0/3382343444 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe81c043dd0 con 0x7fe82c107bc0 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.180+0000 7fe831778700 1 -- 172.21.15.31:0/3382343444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fe82c19f180 con 0x7fe82c107bc0 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.184+0000 7fe813fff700 1 -- 172.21.15.31:0/3382343444 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fe81c032020 con 0x7fe82c107bc0 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.185+0000 7fe831778700 1 -- 172.21.15.31:0/3382343444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 msgr2=0x7fe82c19baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.185+0000 7fe831778700 1 --2- 172.21.15.31:0/3382343444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c19baf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe81c006af0 tx=0x7fe81c011b70 comp rx=0 tx=0).stop 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.185+0000 7fe831778700 1 -- 172.21.15.31:0/3382343444 shutdown_connections 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.185+0000 7fe831778700 1 --2- 172.21.15.31:0/3382343444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe82c107bc0 0x7fe82c19baf0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.185+0000 7fe831778700 1 -- 172.21.15.31:0/3382343444 >> 172.21.15.31:0/3382343444 conn(0x7fe82c1031c0 msgr2=0x7fe82c103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:35.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.185+0000 7fe831778700 1 -- 172.21.15.31:0/3382343444 shutdown_connections 2024-04-25T01:44:35.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:34.185+0000 7fe831778700 1 -- 172.21.15.31:0/3382343444 wait complete. 2024-04-25T01:44:35.055 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T01:44:35.076 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:44:35.076 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mgr... 2024-04-25T01:44:35.077 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T01:44:35.077 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T01:44:35.427 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.x 2024-04-25T01:44:35.427 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T01:44:35.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:35 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3382343444' entity='client.admin' 2024-04-25T01:44:35.660 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-36afbea4-02a5-11ef-bc93-c7b262605968.target.wants/ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-36afbea4-02a5-11ef-bc93-c7b262605968@.service. 2024-04-25T01:44:35.729 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:35 smithi031 systemd[1]: Starting Ceph mgr.x for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:44:36.177 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:35 smithi031 podman[29419]: 2024-04-25 01:44:35.811991216 +0000 UTC m=+0.018278472 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:44:36.177 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:35 smithi031 podman[29419]: 2024-04-25 01:44:35.917266067 +0000 UTC m=+0.123553318 container create 8a3e25b0c23cd91357918bafd5644422cde5dd1ca8c3a51c8f6badd470ce5958 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T01:44:36.177 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:36 smithi031 podman[29419]: 2024-04-25 01:44:36.111300459 +0000 UTC m=+0.317587711 container init 8a3e25b0c23cd91357918bafd5644422cde5dd1ca8c3a51c8f6badd470ce5958 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T01:44:36.177 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:36 smithi031 podman[29419]: 2024-04-25 01:44:36.114297408 +0000 UTC m=+0.320584668 container start 8a3e25b0c23cd91357918bafd5644422cde5dd1ca8c3a51c8f6badd470ce5958 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T01:44:36.177 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:36 smithi031 bash[29419]: 8a3e25b0c23cd91357918bafd5644422cde5dd1ca8c3a51c8f6badd470ce5958 2024-04-25T01:44:36.191 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:44:36.191 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-25T01:44:36.199 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:44:36.199 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T01:44:36.199 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-25T01:44:36.199 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T01:44:36.207 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:44:36.207 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-25T01:44:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:44:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T01:44:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-25T01:44:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T01:44:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr to start... 2024-04-25T01:44:36.215 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr... 2024-04-25T01:44:36.458 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:36 smithi031 systemd[1]: Started Ceph mgr.x for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:44:36.459 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:36 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:36.271+0000 7f1dcf3dd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:44:36.459 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:36 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:36.466+0000 7f1dcf3dd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:44:36.729 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:36 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:36.574+0000 7f1dcf3dd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:44:36.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:36 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2989129277' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:44:36.993 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:36 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:36.912+0000 7f1dcf3dd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:44:36.993 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:37 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:37.000+0000 7f1dcf3dd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:44:37.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:37.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:44:37.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:44:37.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:37.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:44:37.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:44:37.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:37.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:44:37.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:37.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:44:37.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:44:37.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:44:37.374 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:44:37.374 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:44:37.374 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:44:37.374 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:37.374 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:44:37.374 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:44:37.374 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:44:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:44:19.900022+0000", 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:44:37.376 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.666+0000 7fcc52b32700 1 Processor -- start 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.666+0000 7fcc52b32700 1 -- start start 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.666+0000 7fcc52b32700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c07c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.666+0000 7fcc52b32700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c07c8f0 con 0x7fcc4c07de70 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.666+0000 7fcc508ce700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c07c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.666+0000 7fcc508ce700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c07c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46654/0 (socket says 172.21.15.31:46654) 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.666+0000 7fcc508ce700 1 -- 172.21.15.31:0/3532380880 learned_addr learned my addr 172.21.15.31:0/3532380880 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.667+0000 7fcc508ce700 1 -- 172.21.15.31:0/3532380880 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4c07ca70 con 0x7fcc4c07de70 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.667+0000 7fcc508ce700 1 --2- 172.21.15.31:0/3532380880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c07c320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcc400098a0 tx=0x7fcc40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82aef479cdaf2936 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.667+0000 7fcc4b7fe700 1 -- 172.21.15.31:0/3532380880 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc40004160 con 0x7fcc4c07de70 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.667+0000 7fcc4b7fe700 1 -- 172.21.15.31:0/3532380880 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fcc40004300 con 0x7fcc4c07de70 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.667+0000 7fcc4b7fe700 1 -- 172.21.15.31:0/3532380880 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc40011a80 con 0x7fcc4c07de70 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.668+0000 7fcc52b32700 1 -- 172.21.15.31:0/3532380880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 msgr2=0x7fcc4c07c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.668+0000 7fcc52b32700 1 --2- 172.21.15.31:0/3532380880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c07c320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcc400098a0 tx=0x7fcc40009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.668+0000 7fcc52b32700 1 -- 172.21.15.31:0/3532380880 shutdown_connections 2024-04-25T01:44:37.377 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.668+0000 7fcc52b32700 1 --2- 172.21.15.31:0/3532380880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c07c320 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.668+0000 7fcc52b32700 1 -- 172.21.15.31:0/3532380880 >> 172.21.15.31:0/3532380880 conn(0x7fcc4c103110 msgr2=0x7fcc4c105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.668+0000 7fcc52b32700 1 -- 172.21.15.31:0/3532380880 shutdown_connections 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.668+0000 7fcc52b32700 1 -- 172.21.15.31:0/3532380880 wait complete. 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc52b32700 1 Processor -- start 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc52b32700 1 -- start start 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc52b32700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc52b32700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c10e100 con 0x7fcc4c07de70 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc508ce700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc508ce700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46668/0 (socket says 172.21.15.31:46668) 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc508ce700 1 -- 172.21.15.31:0/2989129277 learned_addr learned my addr 172.21.15.31:0/2989129277 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc508ce700 1 -- 172.21.15.31:0/2989129277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4c19e250 con 0x7fcc4c07de70 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc508ce700 1 --2- 172.21.15.31:0/2989129277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c19dd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcc40006af0 tx=0x7fcc40011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:37.378 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc49ffb700 1 -- 172.21.15.31:0/2989129277 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc40003930 con 0x7fcc4c07de70 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc49ffb700 1 -- 172.21.15.31:0/2989129277 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fcc40003ad0 con 0x7fcc4c07de70 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.669+0000 7fcc52b32700 1 -- 172.21.15.31:0/2989129277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4c19e4e0 con 0x7fcc4c07de70 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.670+0000 7fcc49ffb700 1 -- 172.21.15.31:0/2989129277 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc400195e0 con 0x7fcc4c07de70 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.670+0000 7fcc52b32700 1 -- 172.21.15.31:0/2989129277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc4c1a11c0 con 0x7fcc4c07de70 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.670+0000 7fcc49ffb700 1 -- 172.21.15.31:0/2989129277 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcc40019c20 con 0x7fcc4c07de70 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.670+0000 7fcc49ffb700 1 -- 172.21.15.31:0/2989129277 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcc40024730 con 0x7fcc4c07de70 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.671+0000 7fcc52b32700 1 -- 172.21.15.31:0/2989129277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc4c063890 con 0x7fcc4c07de70 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.673+0000 7fcc49ffb700 1 -- 172.21.15.31:0/2989129277 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fcc40043dd0 con 0x7fcc4c07de70 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.747+0000 7fcc52b32700 1 -- 172.21.15.31:0/2989129277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcc4c1a1390 con 0x7fcc4c07de70 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.748+0000 7fcc49ffb700 1 -- 172.21.15.31:0/2989129277 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fcc40032030 con 0x7fcc4c07de70 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.748+0000 7fcc52b32700 1 -- 172.21.15.31:0/2989129277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 msgr2=0x7fcc4c19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.749+0000 7fcc52b32700 1 --2- 172.21.15.31:0/2989129277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c19dd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcc40006af0 tx=0x7fcc40011b70 comp rx=0 tx=0).stop 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.749+0000 7fcc52b32700 1 -- 172.21.15.31:0/2989129277 shutdown_connections 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.749+0000 7fcc52b32700 1 --2- 172.21.15.31:0/2989129277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc4c07de70 0x7fcc4c19dd10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:37.379 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.749+0000 7fcc52b32700 1 -- 172.21.15.31:0/2989129277 >> 172.21.15.31:0/2989129277 conn(0x7fcc4c103110 msgr2=0x7fcc4c103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:37.380 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.749+0000 7fcc52b32700 1 -- 172.21.15.31:0/2989129277 shutdown_connections 2024-04-25T01:44:37.380 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.749+0000 7fcc52b32700 1 -- 172.21.15.31:0/2989129277 wait complete. 2024-04-25T01:44:37.380 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (1/15)... 2024-04-25T01:44:38.275 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:38 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:38.263+0000 7f1dcf3dd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:44:38.615 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:38 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:38.446+0000 7f1dcf3dd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:44:38.615 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:38 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:38.623+0000 7f1dcf3dd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:44:39.223 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:38 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:38.926+0000 7f1dcf3dd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:44:39.223 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:39 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:39.022+0000 7f1dcf3dd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:44:39.223 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:39 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:39.231+0000 7f1dcf3dd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:44:39.479 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:39 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:39.440+0000 7f1dcf3dd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:44:40.073 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:39 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3428660191' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:44:40.478 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:40 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:40.080+0000 7f1dcf3dd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:44:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:44:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:44:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:44:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:44:40.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:44:40.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:44:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:44:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:44:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:44:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:44:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:44:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:44:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:44:40.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:44:40.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:44:19.900022+0000", 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:40.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.813+0000 7ff7ef28e700 1 Processor -- start 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.813+0000 7ff7ef28e700 1 -- start start 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.813+0000 7ff7ef28e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e810aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.813+0000 7ff7ef28e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e807be90 con 0x7ff7e8108660 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.813+0000 7ff7ed02a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e810aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.813+0000 7ff7ed02a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e810aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46678/0 (socket says 172.21.15.31:46678) 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.813+0000 7ff7ed02a700 1 -- 172.21.15.31:0/2727610269 learned_addr learned my addr 172.21.15.31:0/2727610269 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.814+0000 7ff7ed02a700 1 -- 172.21.15.31:0/2727610269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e810af90 con 0x7ff7e8108660 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.814+0000 7ff7ed02a700 1 --2- 172.21.15.31:0/2727610269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e810aa50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff7e40098a0 tx=0x7ff7e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174919c2a35094e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.814+0000 7ff7dffff700 1 -- 172.21.15.31:0/2727610269 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7e4004160 con 0x7ff7e8108660 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.814+0000 7ff7dffff700 1 -- 172.21.15.31:0/2727610269 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff7e4004300 con 0x7ff7e8108660 2024-04-25T01:44:40.582 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.814+0000 7ff7dffff700 1 -- 172.21.15.31:0/2727610269 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7e4011a80 con 0x7ff7e8108660 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.815+0000 7ff7ef28e700 1 -- 172.21.15.31:0/2727610269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 msgr2=0x7ff7e810aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.815+0000 7ff7ef28e700 1 --2- 172.21.15.31:0/2727610269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e810aa50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff7e40098a0 tx=0x7ff7e4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.815+0000 7ff7ef28e700 1 -- 172.21.15.31:0/2727610269 shutdown_connections 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.815+0000 7ff7ef28e700 1 --2- 172.21.15.31:0/2727610269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e810aa50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.815+0000 7ff7ef28e700 1 -- 172.21.15.31:0/2727610269 >> 172.21.15.31:0/2727610269 conn(0x7ff7e8102030 msgr2=0x7ff7e8104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.815+0000 7ff7ef28e700 1 -- 172.21.15.31:0/2727610269 shutdown_connections 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.815+0000 7ff7ef28e700 1 -- 172.21.15.31:0/2727610269 wait complete. 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.816+0000 7ff7ef28e700 1 Processor -- start 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.816+0000 7ff7ef28e700 1 -- start start 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.816+0000 7ff7ef28e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e8107ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.816+0000 7ff7ef28e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e810bf50 con 0x7ff7e8108660 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.816+0000 7ff7ed02a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e8107ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.816+0000 7ff7ed02a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e8107ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46694/0 (socket says 172.21.15.31:46694) 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.816+0000 7ff7ed02a700 1 -- 172.21.15.31:0/3428660191 learned_addr learned my addr 172.21.15.31:0/3428660191 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:40.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.816+0000 7ff7ed02a700 1 -- 172.21.15.31:0/3428660191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e8108420 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.816+0000 7ff7ed02a700 1 --2- 172.21.15.31:0/3428660191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e8107ee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4006af0 tx=0x7ff7e4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.817+0000 7ff7de7fc700 1 -- 172.21.15.31:0/3428660191 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7e4003930 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.817+0000 7ff7de7fc700 1 -- 172.21.15.31:0/3428660191 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff7e4003ad0 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.817+0000 7ff7ef28e700 1 -- 172.21.15.31:0/3428660191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7e8106610 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.817+0000 7ff7de7fc700 1 -- 172.21.15.31:0/3428660191 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7e40195e0 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.817+0000 7ff7ef28e700 1 -- 172.21.15.31:0/3428660191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7e8106b50 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.817+0000 7ff7de7fc700 1 -- 172.21.15.31:0/3428660191 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff7e4019c40 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.817+0000 7ff7de7fc700 1 -- 172.21.15.31:0/3428660191 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff7e4024730 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.818+0000 7ff7ef28e700 1 -- 172.21.15.31:0/3428660191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7e8063890 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.820+0000 7ff7de7fc700 1 -- 172.21.15.31:0/3428660191 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff7e4043dd0 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.893+0000 7ff7ef28e700 1 -- 172.21.15.31:0/3428660191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff7e8107080 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.894+0000 7ff7de7fc700 1 -- 172.21.15.31:0/3428660191 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff7e4032020 con 0x7ff7e8108660 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.895+0000 7ff7ef28e700 1 -- 172.21.15.31:0/3428660191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 msgr2=0x7ff7e8107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.895+0000 7ff7ef28e700 1 --2- 172.21.15.31:0/3428660191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e8107ee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4006af0 tx=0x7ff7e4011b70 comp rx=0 tx=0).stop 2024-04-25T01:44:40.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.895+0000 7ff7ef28e700 1 -- 172.21.15.31:0/3428660191 shutdown_connections 2024-04-25T01:44:40.585 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.895+0000 7ff7ef28e700 1 --2- 172.21.15.31:0/3428660191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7e8108660 0x7ff7e8107ee0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:40.585 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.895+0000 7ff7ef28e700 1 -- 172.21.15.31:0/3428660191 >> 172.21.15.31:0/3428660191 conn(0x7ff7e8102030 msgr2=0x7ff7e8102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:40.585 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.895+0000 7ff7ef28e700 1 -- 172.21.15.31:0/3428660191 shutdown_connections 2024-04-25T01:44:40.585 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:39.895+0000 7ff7ef28e700 1 -- 172.21.15.31:0/3428660191 wait complete. 2024-04-25T01:44:40.585 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (2/15)... 2024-04-25T01:44:40.853 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:40 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:40.523+0000 7f1dcf3dd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:44:40.853 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:40 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:40.639+0000 7f1dcf3dd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:44:40.853 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:40 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:40.860+0000 7f1dcf3dd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:44:41.729 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:41 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:41.403+0000 7f1dcf3dd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:44:41.729 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:41 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:41.498+0000 7f1dcf3dd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:44:42.094 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:41 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:41.748+0000 7f1dcf3dd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:44:42.094 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:41 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:41.886+0000 7f1dcf3dd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:44:42.094 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:41 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:41.983+0000 7f1dcf3dd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:44:42.094 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:42 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:42.102+0000 7f1dcf3dd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:44:42.479 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:42 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:42.196+0000 7f1dcf3dd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:44:43.229 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:43 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1287800111' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:44:43.229 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:43 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:43.058+0000 7f1dcf3dd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:44:43.662 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:43 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:43.289+0000 7f1dcf3dd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:44:43.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:43.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:44:43.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:44:43.676 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:44:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:44:43.678 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:44:43.679 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:44:19.900022+0000", 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:44:43.680 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.993+0000 7f3620378700 1 Processor -- start 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.993+0000 7f3620378700 1 -- start start 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.993+0000 7f3620378700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f3618108440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.993+0000 7f3620378700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3618108a10 con 0x7f3618108020 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.993+0000 7f361e114700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f3618108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.993+0000 7f361e114700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f3618108440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43764/0 (socket says 172.21.15.31:43764) 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.993+0000 7f361e114700 1 -- 172.21.15.31:0/3432549775 learned_addr learned my addr 172.21.15.31:0/3432549775 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.994+0000 7f361e114700 1 -- 172.21.15.31:0/3432549775 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3618106c80 con 0x7f3618108020 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.994+0000 7f361e114700 1 --2- 172.21.15.31:0/3432549775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f3618108440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f36140098a0 tx=0x7f3614009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3c92bc729c5369 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.994+0000 7f361d112700 1 -- 172.21.15.31:0/3432549775 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3614004160 con 0x7f3618108020 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.994+0000 7f361d112700 1 -- 172.21.15.31:0/3432549775 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3614004300 con 0x7f3618108020 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.994+0000 7f361d112700 1 -- 172.21.15.31:0/3432549775 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3614011a80 con 0x7f3618108020 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.995+0000 7f3620378700 1 -- 172.21.15.31:0/3432549775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 msgr2=0x7f3618108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:43.681 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.995+0000 7f3620378700 1 --2- 172.21.15.31:0/3432549775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f3618108440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f36140098a0 tx=0x7f3614009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.995+0000 7f3620378700 1 -- 172.21.15.31:0/3432549775 shutdown_connections 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.995+0000 7f3620378700 1 --2- 172.21.15.31:0/3432549775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f3618108440 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.995+0000 7f3620378700 1 -- 172.21.15.31:0/3432549775 >> 172.21.15.31:0/3432549775 conn(0x7f3618101750 msgr2=0x7f3618103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.995+0000 7f3620378700 1 -- 172.21.15.31:0/3432549775 shutdown_connections 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.995+0000 7f3620378700 1 -- 172.21.15.31:0/3432549775 wait complete. 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.996+0000 7f3620378700 1 Processor -- start 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.996+0000 7f3620378700 1 -- start start 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.996+0000 7f3620378700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f36181065f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.996+0000 7f3620378700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f361810b040 con 0x7f3618108020 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.996+0000 7f361e114700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f36181065f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.996+0000 7f361e114700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f36181065f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43766/0 (socket says 172.21.15.31:43766) 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.996+0000 7f361e114700 1 -- 172.21.15.31:0/1287800111 learned_addr learned my addr 172.21.15.31:0/1287800111 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.996+0000 7f361e114700 1 -- 172.21.15.31:0/1287800111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361819e830 con 0x7f3618108020 2024-04-25T01:44:43.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.997+0000 7f361e114700 1 --2- 172.21.15.31:0/1287800111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f36181065f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3614000c00 tx=0x7f3614011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.997+0000 7f360f7fe700 1 -- 172.21.15.31:0/1287800111 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36140039f0 con 0x7f3618108020 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.997+0000 7f360f7fe700 1 -- 172.21.15.31:0/1287800111 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3614003b90 con 0x7f3618108020 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.997+0000 7f3620378700 1 -- 172.21.15.31:0/1287800111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361819ea00 con 0x7f3618108020 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.997+0000 7f360f7fe700 1 -- 172.21.15.31:0/1287800111 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f361401a740 con 0x7f3618108020 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.997+0000 7f3620378700 1 -- 172.21.15.31:0/1287800111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f361819eda0 con 0x7f3618108020 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.997+0000 7f360f7fe700 1 -- 172.21.15.31:0/1287800111 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f361402d3f0 con 0x7f3618108020 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.997+0000 7f360f7fe700 1 -- 172.21.15.31:0/1287800111 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3614019700 con 0x7f3618108020 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.998+0000 7f3620378700 1 -- 172.21.15.31:0/1287800111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3618063890 con 0x7f3618108020 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.000+0000 7f360f7fe700 1 -- 172.21.15.31:0/1287800111 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3614024af0 con 0x7f3618108020 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.074+0000 7f3620378700 1 -- 172.21.15.31:0/1287800111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f361819f0a0 con 0x7f3618108020 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.075+0000 7f360f7fe700 1 -- 172.21.15.31:0/1287800111 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f36140199a0 con 0x7f3618108020 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.076+0000 7f3620378700 1 -- 172.21.15.31:0/1287800111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 msgr2=0x7f36181065f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:43.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.076+0000 7f3620378700 1 --2- 172.21.15.31:0/1287800111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f36181065f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3614000c00 tx=0x7f3614011b00 comp rx=0 tx=0).stop 2024-04-25T01:44:43.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.076+0000 7f3620378700 1 -- 172.21.15.31:0/1287800111 shutdown_connections 2024-04-25T01:44:43.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.076+0000 7f3620378700 1 --2- 172.21.15.31:0/1287800111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3618108020 0x7f36181065f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:43.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.076+0000 7f3620378700 1 -- 172.21.15.31:0/1287800111 >> 172.21.15.31:0/1287800111 conn(0x7f3618101750 msgr2=0x7f3618102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:43.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.076+0000 7f3620378700 1 -- 172.21.15.31:0/1287800111 shutdown_connections 2024-04-25T01:44:43.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.076+0000 7f3620378700 1 -- 172.21.15.31:0/1287800111 wait complete. 2024-04-25T01:44:43.684 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (3/15)... 2024-04-25T01:44:44.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: Activating manager daemon x 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: mgrmap e2: x(active, starting, since 0.00421734s) 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: from='mgr.14100 172.21.15.31:0/1408271778' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: from='mgr.14100 172.21.15.31:0/1408271778' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: from='mgr.14100 172.21.15.31:0/1408271778' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: from='mgr.14100 172.21.15.31:0/1408271778' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: from='mgr.14100 172.21.15.31:0/1408271778' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: Manager daemon x is now available 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: from='mgr.14100 172.21.15.31:0/1408271778' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: from='mgr.14100 172.21.15.31:0/1408271778' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: from='mgr.14100 172.21.15.31:0/1408271778' entity='mgr.x' 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: from='mgr.14100 172.21.15.31:0/1408271778' entity='mgr.x' 2024-04-25T01:44:44.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:44 smithi031 ceph-mon[29204]: from='mgr.14100 172.21.15.31:0/1408271778' entity='mgr.x' 2024-04-25T01:44:45.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:45 smithi031 ceph-mon[29204]: mgrmap e3: x(active, since 1.00721s) 2024-04-25T01:44:46.414 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:46 smithi031 ceph-mon[29204]: mgrmap e4: x(active, since 2s) 2024-04-25T01:44:46.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:46.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:44:46.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:44:46.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:44:46.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:44:46.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:44:46.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:44:46.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:46.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:44:46.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:44:46.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:44:46.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:46.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:44:46.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:44:46.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:46.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-25T01:44:46.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:44:46.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:46.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:44:46.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:44:46.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:46.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:44:46.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:46.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:44:46.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:44:46.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:44:46.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:44:46.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:44:46.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:44:46.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:46.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:44:46.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:44:46.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:44:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:44:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:44:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:44:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:44:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:44:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:44:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:44:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:44:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:44:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:44:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:44:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:44:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:44:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:44:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:44:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:44:46.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:44:46.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:44:46.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:46.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:44:46.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:44:46.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:44:19.900022+0000", 2024-04-25T01:44:46.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:44:46.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:44:46.999 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:44:46.999 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:44:46.999 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.118+0000 7fceace49700 1 Processor -- start 2024-04-25T01:44:46.999 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.118+0000 7fceace49700 1 -- start start 2024-04-25T01:44:46.999 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.118+0000 7fceace49700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea807c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:46.999 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.118+0000 7fceace49700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcea807c840 con 0x7fcea807dd90 2024-04-25T01:44:46.999 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.119+0000 7fcea659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea807c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:47.000 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.119+0000 7fcea659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea807c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43832/0 (socket says 172.21.15.31:43832) 2024-04-25T01:44:47.000 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.119+0000 7fcea659c700 1 -- 172.21.15.31:0/578733913 learned_addr learned my addr 172.21.15.31:0/578733913 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:47.000 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.119+0000 7fcea659c700 1 -- 172.21.15.31:0/578733913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea807c9c0 con 0x7fcea807dd90 2024-04-25T01:44:47.000 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.119+0000 7fcea659c700 1 --2- 172.21.15.31:0/578733913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea807c300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce98009e80 tx=0x7fce9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726c4965fcde2719 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:47.000 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.119+0000 7fcea559a700 1 -- 172.21.15.31:0/578733913 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce98017070 con 0x7fcea807dd90 2024-04-25T01:44:47.000 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.119+0000 7fcea559a700 1 -- 172.21.15.31:0/578733913 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fce9800b8c0 con 0x7fcea807dd90 2024-04-25T01:44:47.000 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.119+0000 7fcea559a700 1 -- 172.21.15.31:0/578733913 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce98004520 con 0x7fcea807dd90 2024-04-25T01:44:47.001 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.120+0000 7fceace49700 1 -- 172.21.15.31:0/578733913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 msgr2=0x7fcea807c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:47.001 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.120+0000 7fceace49700 1 --2- 172.21.15.31:0/578733913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea807c300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce98009e80 tx=0x7fce9800b330 comp rx=0 tx=0).stop 2024-04-25T01:44:47.001 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.120+0000 7fceace49700 1 -- 172.21.15.31:0/578733913 shutdown_connections 2024-04-25T01:44:47.001 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.120+0000 7fceace49700 1 --2- 172.21.15.31:0/578733913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea807c300 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:47.001 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.120+0000 7fceace49700 1 -- 172.21.15.31:0/578733913 >> 172.21.15.31:0/578733913 conn(0x7fcea81031c0 msgr2=0x7fcea8105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:47.001 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.120+0000 7fceace49700 1 -- 172.21.15.31:0/578733913 shutdown_connections 2024-04-25T01:44:47.001 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.120+0000 7fceace49700 1 -- 172.21.15.31:0/578733913 wait complete. 2024-04-25T01:44:47.001 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.121+0000 7fceace49700 1 Processor -- start 2024-04-25T01:44:47.002 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.121+0000 7fceace49700 1 -- start start 2024-04-25T01:44:47.002 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.121+0000 7fceace49700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea819dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:47.002 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.121+0000 7fceace49700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcea810e050 con 0x7fcea807dd90 2024-04-25T01:44:47.002 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.121+0000 7fcea659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea819dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:47.002 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.121+0000 7fcea659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea819dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43834/0 (socket says 172.21.15.31:43834) 2024-04-25T01:44:47.002 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.121+0000 7fcea659c700 1 -- 172.21.15.31:0/566775485 learned_addr learned my addr 172.21.15.31:0/566775485 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:47.002 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.122+0000 7fcea659c700 1 -- 172.21.15.31:0/566775485 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea819e0f0 con 0x7fcea807dd90 2024-04-25T01:44:47.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.122+0000 7fcea659c700 1 --2- 172.21.15.31:0/566775485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea819dbb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fce98009e50 tx=0x7fce98004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:47.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.122+0000 7fce9f7fe700 1 -- 172.21.15.31:0/566775485 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce98020020 con 0x7fcea807dd90 2024-04-25T01:44:47.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.122+0000 7fce9f7fe700 1 -- 172.21.15.31:0/566775485 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fce9800bba0 con 0x7fcea807dd90 2024-04-25T01:44:47.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.122+0000 7fceace49700 1 -- 172.21.15.31:0/566775485 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea819e380 con 0x7fcea807dd90 2024-04-25T01:44:47.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.122+0000 7fce9f7fe700 1 -- 172.21.15.31:0/566775485 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce98017040 con 0x7fcea807dd90 2024-04-25T01:44:47.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.122+0000 7fceace49700 1 -- 172.21.15.31:0/566775485 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea81a1070 con 0x7fcea807dd90 2024-04-25T01:44:47.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.123+0000 7fce9f7fe700 1 -- 172.21.15.31:0/566775485 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fce98023020 con 0x7fcea807dd90 2024-04-25T01:44:47.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.123+0000 7fce9f7fe700 1 --2- 172.21.15.31:0/566775485 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fce94039000 0x7fce9403b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:47.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.123+0000 7fceace49700 1 -- 172.21.15.31:0/566775485 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea8063890 con 0x7fcea807dd90 2024-04-25T01:44:47.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.123+0000 7fce9f7fe700 1 -- 172.21.15.31:0/566775485 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fce9800f050 con 0x7fcea807dd90 2024-04-25T01:44:47.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.123+0000 7fcea5d9b700 1 --2- 172.21.15.31:0/566775485 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fce94039000 0x7fce9403b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:47.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.124+0000 7fcea5d9b700 1 --2- 172.21.15.31:0/566775485 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fce94039000 0x7fce9403b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fce90006fd0 tx=0x7fce90006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:47.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.129+0000 7fce9f7fe700 1 -- 172.21.15.31:0/566775485 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fce9802a020 con 0x7fcea807dd90 2024-04-25T01:44:47.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.422+0000 7fceace49700 1 -- 172.21.15.31:0/566775485 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcea81a1240 con 0x7fcea807dd90 2024-04-25T01:44:47.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.422+0000 7fce9f7fe700 1 -- 172.21.15.31:0/566775485 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fce9800bd50 con 0x7fcea807dd90 2024-04-25T01:44:47.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.425+0000 7fceace49700 1 -- 172.21.15.31:0/566775485 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fce94039000 msgr2=0x7fce9403b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:47.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.425+0000 7fceace49700 1 --2- 172.21.15.31:0/566775485 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fce94039000 0x7fce9403b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fce90006fd0 tx=0x7fce90006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:47.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.425+0000 7fceace49700 1 -- 172.21.15.31:0/566775485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 msgr2=0x7fcea819dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:47.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.425+0000 7fceace49700 1 --2- 172.21.15.31:0/566775485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea819dbb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fce98009e50 tx=0x7fce98004670 comp rx=0 tx=0).stop 2024-04-25T01:44:47.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.426+0000 7fceace49700 1 -- 172.21.15.31:0/566775485 shutdown_connections 2024-04-25T01:44:47.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.426+0000 7fceace49700 1 --2- 172.21.15.31:0/566775485 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fce94039000 0x7fce9403b4c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:47.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.426+0000 7fceace49700 1 --2- 172.21.15.31:0/566775485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea807dd90 0x7fcea819dbb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:47.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.426+0000 7fceace49700 1 -- 172.21.15.31:0/566775485 >> 172.21.15.31:0/566775485 conn(0x7fcea81031c0 msgr2=0x7fcea8103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:47.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.426+0000 7fceace49700 1 -- 172.21.15.31:0/566775485 shutdown_connections 2024-04-25T01:44:47.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.426+0000 7fceace49700 1 -- 172.21.15.31:0/566775485 wait complete. 2024-04-25T01:44:47.005 INFO:teuthology.orchestra.run.smithi031.stdout:mgr is available 2024-04-25T01:44:47.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:47 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/566775485' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:44:48.177 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T01:44:48.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.398+0000 7f6960626700 1 Processor -- start 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.398+0000 7f6960626700 1 -- start start 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.398+0000 7f6960626700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958105cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.398+0000 7f6960626700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6958106200 con 0x7f69581058a0 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.398+0000 7f695e3c2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958105cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.399+0000 7f695e3c2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958105cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43842/0 (socket says 172.21.15.31:43842) 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.399+0000 7f695e3c2700 1 -- 172.21.15.31:0/209884489 learned_addr learned my addr 172.21.15.31:0/209884489 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.399+0000 7f695e3c2700 1 -- 172.21.15.31:0/209884489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6958106380 con 0x7f69581058a0 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.399+0000 7f695e3c2700 1 --2- 172.21.15.31:0/209884489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958105cc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6954009e80 tx=0x7f695400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb66f4f25b1eb99c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:48.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.399+0000 7f695d3c0700 1 -- 172.21.15.31:0/209884489 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6954017070 con 0x7f69581058a0 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.399+0000 7f695d3c0700 1 -- 172.21.15.31:0/209884489 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f695400b8c0 con 0x7f69581058a0 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.399+0000 7f695d3c0700 1 -- 172.21.15.31:0/209884489 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6954004520 con 0x7f69581058a0 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.400+0000 7f6960626700 1 -- 172.21.15.31:0/209884489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 msgr2=0x7f6958105cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.400+0000 7f6960626700 1 --2- 172.21.15.31:0/209884489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958105cc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6954009e80 tx=0x7f695400b330 comp rx=0 tx=0).stop 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.400+0000 7f6960626700 1 -- 172.21.15.31:0/209884489 shutdown_connections 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.400+0000 7f6960626700 1 --2- 172.21.15.31:0/209884489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958105cc0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.400+0000 7f6960626700 1 -- 172.21.15.31:0/209884489 >> 172.21.15.31:0/209884489 conn(0x7f6958101300 msgr2=0x7f6958103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.400+0000 7f6960626700 1 -- 172.21.15.31:0/209884489 shutdown_connections 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.400+0000 7f6960626700 1 -- 172.21.15.31:0/209884489 wait complete. 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.400+0000 7f6960626700 1 Processor -- start 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.400+0000 7f6960626700 1 -- start start 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f6960626700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958108a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:48.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f6960626700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f695810bed0 con 0x7f69581058a0 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f695e3c2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958108a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f695e3c2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958108a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43858/0 (socket says 172.21.15.31:43858) 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f695e3c2700 1 -- 172.21.15.31:0/812890611 learned_addr learned my addr 172.21.15.31:0/812890611 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f695e3c2700 1 -- 172.21.15.31:0/812890611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6958108fb0 con 0x7f69581058a0 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f695e3c2700 1 --2- 172.21.15.31:0/812890611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958108a70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f695400b7b0 tx=0x7f6954004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f694f7fe700 1 -- 172.21.15.31:0/812890611 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6954020020 con 0x7f69581058a0 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f694f7fe700 1 -- 172.21.15.31:0/812890611 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f695400bcb0 con 0x7f69581058a0 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f6960626700 1 -- 172.21.15.31:0/812890611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69581071a0 con 0x7f69581058a0 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f694f7fe700 1 -- 172.21.15.31:0/812890611 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6954017040 con 0x7f69581058a0 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.401+0000 7f6960626700 1 -- 172.21.15.31:0/812890611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69581076e0 con 0x7f69581058a0 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.402+0000 7f694f7fe700 1 -- 172.21.15.31:0/812890611 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f6954023020 con 0x7f69581058a0 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.402+0000 7f6960626700 1 -- 172.21.15.31:0/812890611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6958063890 con 0x7f69581058a0 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.403+0000 7f694f7fe700 1 --2- 172.21.15.31:0/812890611 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f6944039050 0x7f694403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:48.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.403+0000 7f694f7fe700 1 -- 172.21.15.31:0/812890611 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f695400f050 con 0x7f69581058a0 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.403+0000 7f695dbc1700 1 --2- 172.21.15.31:0/812890611 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f6944039050 0x7f694403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.403+0000 7f695dbc1700 1 --2- 172.21.15.31:0/812890611 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f6944039050 0x7f694403b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6948006fd0 tx=0x7f6948006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.408+0000 7f694f7fe700 1 -- 172.21.15.31:0/812890611 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f695402a020 con 0x7f69581058a0 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.611+0000 7f6960626700 1 -- 172.21.15.31:0/812890611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f6958107ed0 con 0x7f69581058a0 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.612+0000 7f694f7fe700 1 -- 172.21.15.31:0/812890611 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f695402a200 con 0x7f69581058a0 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.614+0000 7f6960626700 1 -- 172.21.15.31:0/812890611 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f6944039050 msgr2=0x7f694403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.614+0000 7f6960626700 1 --2- 172.21.15.31:0/812890611 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f6944039050 0x7f694403b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6948006fd0 tx=0x7f6948006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.614+0000 7f6960626700 1 -- 172.21.15.31:0/812890611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 msgr2=0x7f6958108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.614+0000 7f6960626700 1 --2- 172.21.15.31:0/812890611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958108a70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f695400b7b0 tx=0x7f6954004670 comp rx=0 tx=0).stop 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.614+0000 7f6960626700 1 -- 172.21.15.31:0/812890611 shutdown_connections 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.614+0000 7f6960626700 1 --2- 172.21.15.31:0/812890611 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f6944039050 0x7f694403b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.614+0000 7f6960626700 1 --2- 172.21.15.31:0/812890611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69581058a0 0x7f6958108a70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.614+0000 7f6960626700 1 -- 172.21.15.31:0/812890611 >> 172.21.15.31:0/812890611 conn(0x7f6958101300 msgr2=0x7f6958101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:48.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.614+0000 7f6960626700 1 -- 172.21.15.31:0/812890611 shutdown_connections 2024-04-25T01:44:48.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.614+0000 7f6960626700 1 -- 172.21.15.31:0/812890611 wait complete. 2024-04-25T01:44:48.190 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling cephadm module... 2024-04-25T01:44:48.426 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:48 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/812890611' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T01:44:49.588 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:49 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1259168958' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T01:44:49.589 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: ignoring --setuser ceph since I am not root 2024-04-25T01:44:49.589 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: ignoring --setgroup ceph since I am not root 2024-04-25T01:44:49.589 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:49.482+0000 7f2424966200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:44:49.880 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:49.674+0000 7f2424966200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:44:49.880 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:49.780+0000 7f2424966200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:44:49.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.638+0000 7f805c88d700 1 Processor -- start 2024-04-25T01:44:49.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.638+0000 7f805c88d700 1 -- start start 2024-04-25T01:44:49.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.638+0000 7f805c88d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f8058108520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:49.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.638+0000 7f805c88d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8058108af0 con 0x7f8058108100 2024-04-25T01:44:49.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.638+0000 7f805659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f8058108520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:49.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.638+0000 7f805659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f8058108520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43870/0 (socket says 172.21.15.31:43870) 2024-04-25T01:44:49.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.638+0000 7f805659c700 1 -- 172.21.15.31:0/2479694765 learned_addr learned my addr 172.21.15.31:0/2479694765 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.638+0000 7f805659c700 1 -- 172.21.15.31:0/2479694765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8058105c00 con 0x7f8058108100 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.639+0000 7f805659c700 1 --2- 172.21.15.31:0/2479694765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f8058108520 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f80480098a0 tx=0x7f8048009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae805e982185e01f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.639+0000 7f8055d9b700 1 -- 172.21.15.31:0/2479694765 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8048004160 con 0x7f8058108100 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.639+0000 7f8055d9b700 1 -- 172.21.15.31:0/2479694765 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8048004300 con 0x7f8058108100 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.639+0000 7f8055d9b700 1 -- 172.21.15.31:0/2479694765 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8048011a80 con 0x7f8058108100 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.639+0000 7f805c88d700 1 -- 172.21.15.31:0/2479694765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 msgr2=0x7f8058108520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.639+0000 7f805c88d700 1 --2- 172.21.15.31:0/2479694765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f8058108520 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f80480098a0 tx=0x7f8048009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.639+0000 7f805c88d700 1 -- 172.21.15.31:0/2479694765 shutdown_connections 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.639+0000 7f805c88d700 1 --2- 172.21.15.31:0/2479694765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f8058108520 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.639+0000 7f805c88d700 1 -- 172.21.15.31:0/2479694765 >> 172.21.15.31:0/2479694765 conn(0x7f8058101780 msgr2=0x7f8058103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.639+0000 7f805c88d700 1 -- 172.21.15.31:0/2479694765 shutdown_connections 2024-04-25T01:44:49.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.640+0000 7f805c88d700 1 -- 172.21.15.31:0/2479694765 wait complete. 2024-04-25T01:44:49.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.640+0000 7f805c88d700 1 Processor -- start 2024-04-25T01:44:49.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.640+0000 7f805c88d700 1 -- start start 2024-04-25T01:44:49.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.640+0000 7f805c88d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f805819dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:49.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.640+0000 7f805c88d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8058106c20 con 0x7f8058108100 2024-04-25T01:44:49.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.641+0000 7f805659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f805819dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:49.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.641+0000 7f805659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f805819dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43872/0 (socket says 172.21.15.31:43872) 2024-04-25T01:44:49.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.641+0000 7f805659c700 1 -- 172.21.15.31:0/1259168958 learned_addr learned my addr 172.21.15.31:0/1259168958 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:49.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.641+0000 7f805659c700 1 -- 172.21.15.31:0/1259168958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805819e1d0 con 0x7f8058108100 2024-04-25T01:44:49.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.641+0000 7f805659c700 1 --2- 172.21.15.31:0/1259168958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f805819dc90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8048011a50 tx=0x7f8048011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:49.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.641+0000 7f804f7fe700 1 -- 172.21.15.31:0/1259168958 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8048003a80 con 0x7f8058108100 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.641+0000 7f804f7fe700 1 -- 172.21.15.31:0/1259168958 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8048003c20 con 0x7f8058108100 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.641+0000 7f804f7fe700 1 -- 172.21.15.31:0/1259168958 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f804801a650 con 0x7f8058108100 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.641+0000 7f805c88d700 1 -- 172.21.15.31:0/1259168958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f805819e460 con 0x7f8058108100 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.641+0000 7f805c88d700 1 -- 172.21.15.31:0/1259168958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80581a1150 con 0x7f8058108100 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.642+0000 7f804f7fe700 1 -- 172.21.15.31:0/1259168958 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f8048024ad0 con 0x7f8058108100 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.642+0000 7f804f7fe700 1 --2- 172.21.15.31:0/1259168958 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f8044039050 0x7f804403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.642+0000 7f805c88d700 1 -- 172.21.15.31:0/1259168958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8058063890 con 0x7f8058108100 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.643+0000 7f804f7fe700 1 -- 172.21.15.31:0/1259168958 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f804804d470 con 0x7f8058108100 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.643+0000 7f804ffff700 1 --2- 172.21.15.31:0/1259168958 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f8044039050 0x7f804403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.643+0000 7f804ffff700 1 --2- 172.21.15.31:0/1259168958 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f8044039050 0x7f804403b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8040006fd0 tx=0x7f8040006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.648+0000 7f804f7fe700 1 -- 172.21.15.31:0/1259168958 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f804800f2e0 con 0x7f8058108100 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:48.916+0000 7f805c88d700 1 -- 172.21.15.31:0/1259168958 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f80581a1320 con 0x7f8058108100 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.303+0000 7f804f7fe700 1 -- 172.21.15.31:0/1259168958 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f80581a1320 con 0x7f8058108100 2024-04-25T01:44:49.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.304+0000 7f804f7fe700 1 -- 172.21.15.31:0/1259168958 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f804802d3f0 con 0x7f8058108100 2024-04-25T01:44:49.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.307+0000 7f805c88d700 1 -- 172.21.15.31:0/1259168958 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f8044039050 msgr2=0x7f804403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:49.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.307+0000 7f805c88d700 1 --2- 172.21.15.31:0/1259168958 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f8044039050 0x7f804403b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8040006fd0 tx=0x7f8040006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:49.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.307+0000 7f805c88d700 1 -- 172.21.15.31:0/1259168958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 msgr2=0x7f805819dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:49.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.307+0000 7f805c88d700 1 --2- 172.21.15.31:0/1259168958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f805819dc90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8048011a50 tx=0x7f8048011b30 comp rx=0 tx=0).stop 2024-04-25T01:44:49.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.307+0000 7f805c88d700 1 -- 172.21.15.31:0/1259168958 shutdown_connections 2024-04-25T01:44:49.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.307+0000 7f805c88d700 1 --2- 172.21.15.31:0/1259168958 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f8044039050 0x7f804403b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:49.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.307+0000 7f805c88d700 1 --2- 172.21.15.31:0/1259168958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8058108100 0x7f805819dc90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:49.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.307+0000 7f805c88d700 1 -- 172.21.15.31:0/1259168958 >> 172.21.15.31:0/1259168958 conn(0x7f8058101780 msgr2=0x7f8058194350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:49.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.307+0000 7f805c88d700 1 -- 172.21.15.31:0/1259168958 shutdown_connections 2024-04-25T01:44:49.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.307+0000 7f805c88d700 1 -- 172.21.15.31:0/1259168958 wait complete. 2024-04-25T01:44:50.139 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:50 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:50.107+0000 7f2424966200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:44:50.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:50 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1259168958' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T01:44:50.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:50 smithi031 ceph-mon[29204]: mgrmap e5: x(active, since 6s) 2024-04-25T01:44:50.479 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:50 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:50.197+0000 7f2424966200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:44:51.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:44:51.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T01:44:51.236 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.344+0000 7f40c1df1700 1 Processor -- start 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.344+0000 7f40c1df1700 1 -- start start 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.344+0000 7f40c1df1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc10a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.344+0000 7f40c1df1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f40bc07bed0 con 0x7f40bc1085d0 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.344+0000 7f40bb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc10a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.344+0000 7f40bb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc10a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43900/0 (socket says 172.21.15.31:43900) 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.344+0000 7f40bb7fe700 1 -- 172.21.15.31:0/4140665291 learned_addr learned my addr 172.21.15.31:0/4140665291 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.344+0000 7f40bb7fe700 1 -- 172.21.15.31:0/4140665291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40bc10af00 con 0x7f40bc1085d0 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.345+0000 7f40bb7fe700 1 --2- 172.21.15.31:0/4140665291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc10a9c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0098a0 tx=0x7f40ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f0c5bf36cd42c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.345+0000 7f40baffd700 1 -- 172.21.15.31:0/4140665291 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac004160 con 0x7f40bc1085d0 2024-04-25T01:44:51.237 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.345+0000 7f40baffd700 1 -- 172.21.15.31:0/4140665291 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f40ac004300 con 0x7f40bc1085d0 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.345+0000 7f40baffd700 1 -- 172.21.15.31:0/4140665291 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac011a80 con 0x7f40bc1085d0 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.345+0000 7f40c1df1700 1 -- 172.21.15.31:0/4140665291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 msgr2=0x7f40bc10a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.345+0000 7f40c1df1700 1 --2- 172.21.15.31:0/4140665291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc10a9c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0098a0 tx=0x7f40ac009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.345+0000 7f40c1df1700 1 -- 172.21.15.31:0/4140665291 shutdown_connections 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.345+0000 7f40c1df1700 1 --2- 172.21.15.31:0/4140665291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc10a9c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.345+0000 7f40c1df1700 1 -- 172.21.15.31:0/4140665291 >> 172.21.15.31:0/4140665291 conn(0x7f40bc101fe0 msgr2=0x7f40bc104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.346+0000 7f40c1df1700 1 -- 172.21.15.31:0/4140665291 shutdown_connections 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.346+0000 7f40c1df1700 1 -- 172.21.15.31:0/4140665291 wait complete. 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.346+0000 7f40c1df1700 1 Processor -- start 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.346+0000 7f40c1df1700 1 -- start start 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.346+0000 7f40c1df1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc107e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.346+0000 7f40c1df1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f40bc10bed0 con 0x7f40bc1085d0 2024-04-25T01:44:51.238 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.347+0000 7f40bb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc107e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.347+0000 7f40bb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc107e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43910/0 (socket says 172.21.15.31:43910) 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.347+0000 7f40bb7fe700 1 -- 172.21.15.31:0/516646247 learned_addr learned my addr 172.21.15.31:0/516646247 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.347+0000 7f40bb7fe700 1 -- 172.21.15.31:0/516646247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40bc108390 con 0x7f40bc1085d0 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.347+0000 7f40bb7fe700 1 --2- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc107e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f40ac006af0 tx=0x7f40ac011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.347+0000 7f40b97fa700 1 -- 172.21.15.31:0/516646247 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac0039e0 con 0x7f40bc1085d0 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.347+0000 7f40b97fa700 1 -- 172.21.15.31:0/516646247 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f40ac003b80 con 0x7f40bc1085d0 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.347+0000 7f40b97fa700 1 -- 172.21.15.31:0/516646247 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac0196a0 con 0x7f40bc1085d0 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.347+0000 7f40c1df1700 1 -- 172.21.15.31:0/516646247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40bc106580 con 0x7f40bc1085d0 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.347+0000 7f40c1df1700 1 -- 172.21.15.31:0/516646247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40bc106ac0 con 0x7f40bc1085d0 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.348+0000 7f40c1df1700 1 -- 172.21.15.31:0/516646247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40bc063890 con 0x7f40bc1085d0 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.349+0000 7f40b97fa700 1 -- 172.21.15.31:0/516646247 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f40ac0199a0 con 0x7f40bc1085d0 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.349+0000 7f40b97fa700 1 --2- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f40a8039000 0x7f40a803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.349+0000 7f40b97fa700 1 -- 172.21.15.31:0/516646247 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f40ac04d840 con 0x7f40bc1085d0 2024-04-25T01:44:51.239 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.349+0000 7f40b2ffd700 1 -- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f40a8039000 msgr2=0x7f40a803b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3153405418 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.349+0000 7f40b2ffd700 1 --2- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f40a8039000 0x7f40a803b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.354+0000 7f40b97fa700 1 -- 172.21.15.31:0/516646247 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f40ac0042a0 con 0x7f40bc1085d0 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.550+0000 7f40b2ffd700 1 -- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f40a8039000 msgr2=0x7f40a803b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3153405418 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.550+0000 7f40b2ffd700 1 --2- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f40a8039000 0x7f40a803b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.667+0000 7f40c1df1700 1 -- 172.21.15.31:0/516646247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f40bc1073f0 con 0x7f40bc1085d0 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.667+0000 7f40b97fa700 1 -- 172.21.15.31:0/516646247 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f40ac04d260 con 0x7f40bc1085d0 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.669+0000 7f40c1df1700 1 -- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f40a8039000 msgr2=0x7f40a803b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.669+0000 7f40c1df1700 1 --2- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f40a8039000 0x7f40a803b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.669+0000 7f40c1df1700 1 -- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 msgr2=0x7f40bc107e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.669+0000 7f40c1df1700 1 --2- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc107e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f40ac006af0 tx=0x7f40ac011c40 comp rx=0 tx=0).stop 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.669+0000 7f40c1df1700 1 -- 172.21.15.31:0/516646247 shutdown_connections 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.669+0000 7f40c1df1700 1 --2- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7f40a8039000 0x7f40a803b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.669+0000 7f40c1df1700 1 --2- 172.21.15.31:0/516646247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40bc1085d0 0x7f40bc107e50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:51.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.669+0000 7f40c1df1700 1 -- 172.21.15.31:0/516646247 >> 172.21.15.31:0/516646247 conn(0x7f40bc101fe0 msgr2=0x7f40bc102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:51.241 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.670+0000 7f40c1df1700 1 -- 172.21.15.31:0/516646247 shutdown_connections 2024-04-25T01:44:51.241 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:50.670+0000 7f40c1df1700 1 -- 172.21.15.31:0/516646247 wait complete. 2024-04-25T01:44:51.241 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2024-04-25T01:44:51.241 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 5... 2024-04-25T01:44:51.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:51 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/516646247' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T01:44:51.564 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:51 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:51.459+0000 7f2424966200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:44:51.873 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:51 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:51.640+0000 7f2424966200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:44:51.873 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:51 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:51.817+0000 7f2424966200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:44:52.207 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:52.119+0000 7f2424966200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:44:52.207 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:52.215+0000 7f2424966200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:44:52.479 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:52.425+0000 7f2424966200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:44:52.979 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:52.625+0000 7f2424966200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:44:53.479 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:53 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:53.254+0000 7f2424966200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:44:53.979 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:53 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:53.694+0000 7f2424966200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:44:53.979 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:53 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:53.794+0000 7f2424966200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:44:54.479 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:54 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:54.003+0000 7f2424966200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:44:54.886 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:54 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:54.546+0000 7f2424966200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:44:54.886 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:54 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:54.641+0000 7f2424966200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:44:55.229 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:54 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:54.893+0000 7f2424966200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:44:55.229 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:55 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:55.033+0000 7f2424966200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:44:55.229 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:55 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:55.131+0000 7f2424966200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:44:55.729 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:55 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:55.251+0000 7f2424966200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:44:55.729 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:55 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:55.346+0000 7f2424966200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:44:56.457 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:56.188+0000 7f2424966200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:44:56.457 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:44:56.411+0000 7f2424966200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:44:56.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: Active manager daemon x restarted 2024-04-25T01:44:56.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: Activating manager daemon x 2024-04-25T01:44:56.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T01:44:56.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: mgrmap e6: x(active, starting, since 0.00355889s) 2024-04-25T01:44:56.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:44:56.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:44:56.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:44:56.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:44:56.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:44:56.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: Manager daemon x is now available 2024-04-25T01:44:56.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:44:56.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:44:56.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:44:56.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:44:56.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:56 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:44:57.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:57 smithi031 ceph-mon[29204]: Found migration_current of "None". Setting to last migration. 2024-04-25T01:44:57.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:57 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:44:57.717 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:57 smithi031 ceph-mon[29204]: mgrmap e7: x(active, since 1.00526s) 2024-04-25T01:44:57.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:44:57.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T01:44:57.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.675+0000 7fe7043a9700 1 Processor -- start 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.675+0000 7fe7043a9700 1 -- start start 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.675+0000 7fe7043a9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.675+0000 7fe7043a9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe6fc1085c0 con 0x7fe6fc107bd0 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.676+0000 7fe702145700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.676+0000 7fe702145700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc107ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43922/0 (socket says 172.21.15.31:43922) 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.676+0000 7fe702145700 1 -- 172.21.15.31:0/2664867686 learned_addr learned my addr 172.21.15.31:0/2664867686 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.676+0000 7fe702145700 1 -- 172.21.15.31:0/2664867686 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6fc108d80 con 0x7fe6fc107bd0 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.676+0000 7fe702145700 1 --2- 172.21.15.31:0/2664867686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc107ff0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80098a0 tx=0x7fe6f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd24cdf682157c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.676+0000 7fe701143700 1 -- 172.21.15.31:0/2664867686 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6f8004160 con 0x7fe6fc107bd0 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.676+0000 7fe701143700 1 -- 172.21.15.31:0/2664867686 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe6f8004300 con 0x7fe6fc107bd0 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.676+0000 7fe701143700 1 -- 172.21.15.31:0/2664867686 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6f8011a80 con 0x7fe6fc107bd0 2024-04-25T01:44:57.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.677+0000 7fe7043a9700 1 -- 172.21.15.31:0/2664867686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 msgr2=0x7fe6fc107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.677+0000 7fe7043a9700 1 --2- 172.21.15.31:0/2664867686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc107ff0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80098a0 tx=0x7fe6f8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.677+0000 7fe7043a9700 1 -- 172.21.15.31:0/2664867686 shutdown_connections 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.677+0000 7fe7043a9700 1 --2- 172.21.15.31:0/2664867686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc107ff0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.677+0000 7fe7043a9700 1 -- 172.21.15.31:0/2664867686 >> 172.21.15.31:0/2664867686 conn(0x7fe6fc103130 msgr2=0x7fe6fc1055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.677+0000 7fe7043a9700 1 -- 172.21.15.31:0/2664867686 shutdown_connections 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.677+0000 7fe7043a9700 1 -- 172.21.15.31:0/2664867686 wait complete. 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.678+0000 7fe7043a9700 1 Processor -- start 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.678+0000 7fe7043a9700 1 -- start start 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.678+0000 7fe7043a9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc07db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.678+0000 7fe7043a9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe6fc10c130 con 0x7fe6fc107bd0 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.678+0000 7fe702145700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc07db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.678+0000 7fe702145700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc07db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43934/0 (socket says 172.21.15.31:43934) 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.678+0000 7fe702145700 1 -- 172.21.15.31:0/473190884 learned_addr learned my addr 172.21.15.31:0/473190884 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.678+0000 7fe702145700 1 -- 172.21.15.31:0/473190884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6fc07e0d0 con 0x7fe6fc107bd0 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.679+0000 7fe702145700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc07db90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8000c00 tx=0x7fe6f8004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.679+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6f80039c0 con 0x7fe6fc107bd0 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.679+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe6f8003b60 con 0x7fe6fc107bd0 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.679+0000 7fe7043a9700 1 -- 172.21.15.31:0/473190884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6fc07c2c0 con 0x7fe6fc107bd0 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.679+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6f801a6f0 con 0x7fe6fc107bd0 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.679+0000 7fe7043a9700 1 -- 172.21.15.31:0/473190884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6fc07c800 con 0x7fe6fc107bd0 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.680+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fe6f801a890 con 0x7fe6fc107bd0 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.680+0000 7fe6ef7fe700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 0x7fe6e803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.680+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 --> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe6e803bc70 con 0x7fe6e80390a0 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.680+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe6f804d280 con 0x7fe6fc107bd0 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.680+0000 7fe701944700 1 -- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 msgr2=0x7fe6e803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3153405418 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.680+0000 7fe701944700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 0x7fe6e803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.881+0000 7fe701944700 1 -- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 msgr2=0x7fe6e803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3153405418 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.881+0000 7fe701944700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 0x7fe6e803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:44:57.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:52.282+0000 7fe701944700 1 -- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 msgr2=0x7fe6e803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3153405418 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:52.282+0000 7fe701944700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 0x7fe6e803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.083+0000 7fe701944700 1 -- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 msgr2=0x7fe6e803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3153405418 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.083+0000 7fe701944700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 0x7fe6e803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:54.684+0000 7fe701944700 1 -- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 msgr2=0x7fe6e803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3153405418 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:54.684+0000 7fe701944700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 0x7fe6e803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:56.415+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7fe6f801ab80 con 0x7fe6fc107bd0 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:56.415+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 msgr2=0x7fe6e803b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:56.415+0000 7fe6ef7fe700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 0x7fe6e803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.416+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fe6f804ed00 con 0x7fe6fc107bd0 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.416+0000 7fe6ef7fe700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe6e803c9d0 0x7fe6e803edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.416+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 --> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe6e803bc70 con 0x7fe6e803c9d0 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.417+0000 7fe701944700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe6e803c9d0 0x7fe6e803edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.417+0000 7fe701944700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe6e803c9d0 0x7fe6e803edc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0003a10 tx=0x7fe6f00092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.418+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 <== mgr.14120 v2:172.21.15.31:6800/3170545029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fe6e803bc70 con 0x7fe6e803c9d0 2024-04-25T01:44:57.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 -- 172.21.15.31:0/473190884 --> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe6fc063890 con 0x7fe6e803c9d0 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe6ef7fe700 1 -- 172.21.15.31:0/473190884 <== mgr.14120 v2:172.21.15.31:6800/3170545029 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fe6fc063890 con 0x7fe6e803c9d0 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 -- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe6e803c9d0 msgr2=0x7fe6e803edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe6e803c9d0 0x7fe6e803edc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0003a10 tx=0x7fe6f00092b0 comp rx=0 tx=0).stop 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 -- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 msgr2=0x7fe6fc07db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc07db90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8000c00 tx=0x7fe6f8004290 comp rx=0 tx=0).stop 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 -- 172.21.15.31:0/473190884 shutdown_connections 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe6e803c9d0 0x7fe6e803edc0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6fc107bd0 0x7fe6fc07db90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 --2- 172.21.15.31:0/473190884 >> [v2:172.21.15.31:6800/3153405418,v1:172.21.15.31:6801/3153405418] conn(0x7fe6e80390a0 0x7fe6e803b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 -- 172.21.15.31:0/473190884 >> 172.21.15.31:0/473190884 conn(0x7fe6fc103130 msgr2=0x7fe6fc103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 -- 172.21.15.31:0/473190884 shutdown_connections 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:57.425+0000 7fe7043a9700 1 -- 172.21.15.31:0/473190884 wait complete. 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 5 is available 2024-04-25T01:44:57.981 INFO:teuthology.orchestra.run.smithi031.stdout:Setting orchestrator backend to cephadm... 2024-04-25T01:44:58.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:58 smithi031 ceph-mon[29204]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T01:44:58.635 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:58 smithi031 ceph-mon[29204]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T01:44:59.164 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.403+0000 7f2db4360700 1 Processor -- start 2024-04-25T01:44:59.164 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.403+0000 7f2db4360700 1 -- start start 2024-04-25T01:44:59.164 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.404+0000 7f2db4360700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac106e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:59.164 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.404+0000 7f2db4360700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac107450 con 0x7f2dac106a60 2024-04-25T01:44:59.164 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.404+0000 7f2db20fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:59.164 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.404+0000 7f2db20fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35890/0 (socket says 172.21.15.31:35890) 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.404+0000 7f2db20fc700 1 -- 172.21.15.31:0/799454219 learned_addr learned my addr 172.21.15.31:0/799454219 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.404+0000 7f2db20fc700 1 -- 172.21.15.31:0/799454219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac107c60 con 0x7f2dac106a60 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.404+0000 7f2db20fc700 1 --2- 172.21.15.31:0/799454219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac106e80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0098a0 tx=0x7f2d9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83727e5015bdc580 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.404+0000 7f2db10fa700 1 -- 172.21.15.31:0/799454219 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d9c004160 con 0x7f2dac106a60 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.404+0000 7f2db10fa700 1 -- 172.21.15.31:0/799454219 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2d9c004300 con 0x7f2dac106a60 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.404+0000 7f2db10fa700 1 -- 172.21.15.31:0/799454219 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d9c011a80 con 0x7f2dac106a60 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.405+0000 7f2db4360700 1 -- 172.21.15.31:0/799454219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 msgr2=0x7f2dac106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.405+0000 7f2db4360700 1 --2- 172.21.15.31:0/799454219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac106e80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0098a0 tx=0x7f2d9c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.405+0000 7f2db4360700 1 -- 172.21.15.31:0/799454219 shutdown_connections 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.405+0000 7f2db4360700 1 --2- 172.21.15.31:0/799454219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac106e80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.405+0000 7f2db4360700 1 -- 172.21.15.31:0/799454219 >> 172.21.15.31:0/799454219 conn(0x7f2dac101fe0 msgr2=0x7f2dac104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.405+0000 7f2db4360700 1 -- 172.21.15.31:0/799454219 shutdown_connections 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.405+0000 7f2db4360700 1 -- 172.21.15.31:0/799454219 wait complete. 2024-04-25T01:44:59.165 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.406+0000 7f2db4360700 1 Processor -- start 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.406+0000 7f2db4360700 1 -- start start 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.406+0000 7f2db4360700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac07bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.406+0000 7f2db4360700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac10b010 con 0x7f2dac106a60 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.406+0000 7f2db20fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac07bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.406+0000 7f2db20fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac07bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35898/0 (socket says 172.21.15.31:35898) 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.406+0000 7f2db20fc700 1 -- 172.21.15.31:0/464850779 learned_addr learned my addr 172.21.15.31:0/464850779 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.406+0000 7f2db20fc700 1 -- 172.21.15.31:0/464850779 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac19e7e0 con 0x7f2dac106a60 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.407+0000 7f2db20fc700 1 --2- 172.21.15.31:0/464850779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac07bad0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c000c00 tx=0x7f2d9c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.407+0000 7f2da37fe700 1 -- 172.21.15.31:0/464850779 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d9c0039f0 con 0x7f2dac106a60 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.407+0000 7f2da37fe700 1 -- 172.21.15.31:0/464850779 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2d9c003b90 con 0x7f2dac106a60 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.407+0000 7f2db4360700 1 -- 172.21.15.31:0/464850779 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dac19e9b0 con 0x7f2dac106a60 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.407+0000 7f2da37fe700 1 -- 172.21.15.31:0/464850779 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d9c01a740 con 0x7f2dac106a60 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.407+0000 7f2db4360700 1 -- 172.21.15.31:0/464850779 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dac19edd0 con 0x7f2dac106a60 2024-04-25T01:44:59.166 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.408+0000 7f2db4360700 1 -- 172.21.15.31:0/464850779 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dac063890 con 0x7f2dac106a60 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.408+0000 7f2da37fe700 1 -- 172.21.15.31:0/464850779 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f2d9c024af0 con 0x7f2dac106a60 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.409+0000 7f2da37fe700 1 --2- 172.21.15.31:0/464850779 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f2d98038f80 0x7f2d9803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.409+0000 7f2da37fe700 1 -- 172.21.15.31:0/464850779 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2d9c04d4f0 con 0x7f2dac106a60 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.409+0000 7f2db18fb700 1 --2- 172.21.15.31:0/464850779 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f2d98038f80 0x7f2d9803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.409+0000 7f2db18fb700 1 --2- 172.21.15.31:0/464850779 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f2d98038f80 0x7f2d9803b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2da8006fd0 tx=0x7f2da8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.414+0000 7f2da37fe700 1 -- 172.21.15.31:0/464850779 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2d9c02c560 con 0x7f2dac106a60 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.468+0000 7f2da37fe700 1 -- 172.21.15.31:0/464850779 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f2d9c0247c0 con 0x7f2dac106a60 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.642+0000 7f2db4360700 1 -- 172.21.15.31:0/464850779 --> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f2dac1032c0 con 0x7f2d98038f80 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.651+0000 7f2da37fe700 1 -- 172.21.15.31:0/464850779 <== mgr.14120 v2:172.21.15.31:6800/3170545029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2dac1032c0 con 0x7f2d98038f80 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.653+0000 7f2db4360700 1 -- 172.21.15.31:0/464850779 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f2d98038f80 msgr2=0x7f2d9803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.653+0000 7f2db4360700 1 --2- 172.21.15.31:0/464850779 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f2d98038f80 0x7f2d9803b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2da8006fd0 tx=0x7f2da8006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.653+0000 7f2db4360700 1 -- 172.21.15.31:0/464850779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 msgr2=0x7f2dac07bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.653+0000 7f2db4360700 1 --2- 172.21.15.31:0/464850779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac07bad0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c000c00 tx=0x7f2d9c011b00 comp rx=0 tx=0).stop 2024-04-25T01:44:59.167 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.653+0000 7f2db4360700 1 -- 172.21.15.31:0/464850779 shutdown_connections 2024-04-25T01:44:59.168 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.653+0000 7f2db4360700 1 --2- 172.21.15.31:0/464850779 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f2d98038f80 0x7f2d9803b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:59.168 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.653+0000 7f2db4360700 1 --2- 172.21.15.31:0/464850779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dac106a60 0x7f2dac07bad0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:59.168 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.653+0000 7f2db4360700 1 -- 172.21.15.31:0/464850779 >> 172.21.15.31:0/464850779 conn(0x7f2dac101fe0 msgr2=0x7f2dac102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:59.168 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.654+0000 7f2db4360700 1 -- 172.21.15.31:0/464850779 shutdown_connections 2024-04-25T01:44:59.168 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:58.654+0000 7f2db4360700 1 -- 172.21.15.31:0/464850779 wait complete. 2024-04-25T01:44:59.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:59 smithi031 ceph-mon[29204]: mgrmap e8: x(active, since 2s) 2024-04-25T01:44:59.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:59 smithi031 ceph-mon[29204]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:44:59.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:59 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:44:59.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:59 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:44:59.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:59 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:44:59.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:44:59 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:45:00.386 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T01:45:00.386 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.594+0000 7f9e9ab06700 1 Processor -- start 2024-04-25T01:45:00.386 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.594+0000 7f9e9ab06700 1 -- start start 2024-04-25T01:45:00.386 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.594+0000 7f9e9ab06700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e94108470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:00.386 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.594+0000 7f9e9ab06700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9e94108a40 con 0x7f9e94108050 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.594+0000 7f9e988a2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e94108470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.594+0000 7f9e988a2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e94108470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35910/0 (socket says 172.21.15.31:35910) 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.594+0000 7f9e988a2700 1 -- 172.21.15.31:0/269593753 learned_addr learned my addr 172.21.15.31:0/269593753 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.594+0000 7f9e988a2700 1 -- 172.21.15.31:0/269593753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e94106c60 con 0x7f9e94108050 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.595+0000 7f9e988a2700 1 --2- 172.21.15.31:0/269593753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e94108470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e880098a0 tx=0x7f9e88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4b2da1321648d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.595+0000 7f9e937fe700 1 -- 172.21.15.31:0/269593753 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e88004160 con 0x7f9e94108050 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.595+0000 7f9e937fe700 1 -- 172.21.15.31:0/269593753 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f9e88004300 con 0x7f9e94108050 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.595+0000 7f9e937fe700 1 -- 172.21.15.31:0/269593753 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e88011a80 con 0x7f9e94108050 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.595+0000 7f9e9ab06700 1 -- 172.21.15.31:0/269593753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 msgr2=0x7f9e94108470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.595+0000 7f9e9ab06700 1 --2- 172.21.15.31:0/269593753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e94108470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e880098a0 tx=0x7f9e88009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.596+0000 7f9e9ab06700 1 -- 172.21.15.31:0/269593753 shutdown_connections 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.596+0000 7f9e9ab06700 1 --2- 172.21.15.31:0/269593753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e94108470 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.596+0000 7f9e9ab06700 1 -- 172.21.15.31:0/269593753 >> 172.21.15.31:0/269593753 conn(0x7f9e94101750 msgr2=0x7f9e94103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:00.387 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.596+0000 7f9e9ab06700 1 -- 172.21.15.31:0/269593753 shutdown_connections 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.596+0000 7f9e9ab06700 1 -- 172.21.15.31:0/269593753 wait complete. 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.596+0000 7f9e9ab06700 1 Processor -- start 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.596+0000 7f9e9ab06700 1 -- start start 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.597+0000 7f9e9ab06700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e941065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.597+0000 7f9e9ab06700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9410b020 con 0x7f9e94108050 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.597+0000 7f9e988a2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e941065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.597+0000 7f9e988a2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e941065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35918/0 (socket says 172.21.15.31:35918) 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.597+0000 7f9e988a2700 1 -- 172.21.15.31:0/2332430453 learned_addr learned my addr 172.21.15.31:0/2332430453 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.597+0000 7f9e988a2700 1 -- 172.21.15.31:0/2332430453 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9419e850 con 0x7f9e94108050 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.597+0000 7f9e988a2700 1 --2- 172.21.15.31:0/2332430453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e941065d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e88006af0 tx=0x7f9e88011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.597+0000 7f9e91ffb700 1 -- 172.21.15.31:0/2332430453 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e880039e0 con 0x7f9e94108050 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.597+0000 7f9e91ffb700 1 -- 172.21.15.31:0/2332430453 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f9e88003b80 con 0x7f9e94108050 2024-04-25T01:45:00.388 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.597+0000 7f9e9ab06700 1 -- 172.21.15.31:0/2332430453 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9419ea20 con 0x7f9e94108050 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.597+0000 7f9e91ffb700 1 -- 172.21.15.31:0/2332430453 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e880196a0 con 0x7f9e94108050 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.598+0000 7f9e9ab06700 1 -- 172.21.15.31:0/2332430453 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e9419edc0 con 0x7f9e94108050 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.599+0000 7f9e91ffb700 1 -- 172.21.15.31:0/2332430453 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f9e8801ab50 con 0x7f9e94108050 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.599+0000 7f9e9ab06700 1 -- 172.21.15.31:0/2332430453 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e94063890 con 0x7f9e94108050 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.599+0000 7f9e91ffb700 1 --2- 172.21.15.31:0/2332430453 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f9e84039050 0x7f9e8403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.599+0000 7f9e91ffb700 1 -- 172.21.15.31:0/2332430453 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9e8804d520 con 0x7f9e94108050 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.599+0000 7f9e93fff700 1 --2- 172.21.15.31:0/2332430453 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f9e84039050 0x7f9e8403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.600+0000 7f9e93fff700 1 --2- 172.21.15.31:0/2332430453 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f9e84039050 0x7f9e8403b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e80006fd0 tx=0x7f9e80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.604+0000 7f9e91ffb700 1 -- 172.21.15.31:0/2332430453 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9e8800b3e0 con 0x7f9e94108050 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.816+0000 7f9e9ab06700 1 -- 172.21.15.31:0/2332430453 --> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f9e94102a40 con 0x7f9e84039050 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.817+0000 7f9e91ffb700 1 -- 172.21.15.31:0/2332430453 <== mgr.14120 v2:172.21.15.31:6800/3170545029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f9e94102a40 con 0x7f9e84039050 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.819+0000 7f9e9ab06700 1 -- 172.21.15.31:0/2332430453 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f9e84039050 msgr2=0x7f9e8403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.819+0000 7f9e9ab06700 1 --2- 172.21.15.31:0/2332430453 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f9e84039050 0x7f9e8403b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e80006fd0 tx=0x7f9e80006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:00.389 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.819+0000 7f9e9ab06700 1 -- 172.21.15.31:0/2332430453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 msgr2=0x7f9e941065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:00.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.819+0000 7f9e9ab06700 1 --2- 172.21.15.31:0/2332430453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e941065d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e88006af0 tx=0x7f9e88011c40 comp rx=0 tx=0).stop 2024-04-25T01:45:00.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.820+0000 7f9e9ab06700 1 -- 172.21.15.31:0/2332430453 shutdown_connections 2024-04-25T01:45:00.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.820+0000 7f9e9ab06700 1 --2- 172.21.15.31:0/2332430453 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f9e84039050 0x7f9e8403b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:00.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.820+0000 7f9e9ab06700 1 --2- 172.21.15.31:0/2332430453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e94108050 0x7f9e941065d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:00.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.820+0000 7f9e9ab06700 1 -- 172.21.15.31:0/2332430453 >> 172.21.15.31:0/2332430453 conn(0x7f9e94101750 msgr2=0x7f9e94102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:00.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.820+0000 7f9e9ab06700 1 -- 172.21.15.31:0/2332430453 shutdown_connections 2024-04-25T01:45:00.390 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:59.820+0000 7f9e9ab06700 1 -- 172.21.15.31:0/2332430453 wait complete. 2024-04-25T01:45:00.390 INFO:teuthology.orchestra.run.smithi031.stdout:Generating ssh key... 2024-04-25T01:45:00.707 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:00 smithi031 ceph-mon[29204]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:00.707 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:00 smithi031 ceph-mon[29204]: [25/Apr/2024:01:45:00] ENGINE Bus STARTING 2024-04-25T01:45:01.467 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: Generating public/private rsa key pair. 2024-04-25T01:45:01.467 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: Your identification has been saved in /tmp/tmpn6t262ux/key. 2024-04-25T01:45:01.467 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: Your public key has been saved in /tmp/tmpn6t262ux/key.pub. 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: The key fingerprint is: 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: SHA256:a47lENyPSzErcf7zV1Noi3C2Sgi5D7rHZw6x8nYLRnM ceph-36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: The key's randomart image is: 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: +---[RSA 3072]----+ 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: | | 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: | | 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: | . . | 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: | .o. . o o .| 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: | BoE. + + ..| 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: | .oX.B. o ...| 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: | .oBoO... ..| 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: | .++%=oo . | 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: | .+o=*..o.. | 2024-04-25T01:45:01.468 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: +----[SHA256]-----+ 2024-04-25T01:45:01.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-mon[29204]: [25/Apr/2024:01:45:00] ENGINE Serving on http://172.21.15.31:8765 2024-04-25T01:45:01.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-mon[29204]: [25/Apr/2024:01:45:00] ENGINE Serving on https://172.21.15.31:7150 2024-04-25T01:45:01.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-mon[29204]: [25/Apr/2024:01:45:00] ENGINE Bus STARTED 2024-04-25T01:45:01.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:01.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-mon[29204]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:01.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-mon[29204]: Generating ssh key... 2024-04-25T01:45:01.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:45:01.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:01 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:45:01.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.814+0000 7f96ddba4700 1 Processor -- start 2024-04-25T01:45:01.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.814+0000 7f96ddba4700 1 -- start start 2024-04-25T01:45:01.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.814+0000 7f96ddba4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d8107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:01.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.814+0000 7f96ddba4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f96d8108530 con 0x7f96d8107bd0 2024-04-25T01:45:01.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.815+0000 7f96d77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d8107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:01.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.815+0000 7f96d77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d8107ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35920/0 (socket says 172.21.15.31:35920) 2024-04-25T01:45:01.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.815+0000 7f96d77fe700 1 -- 172.21.15.31:0/2698094004 learned_addr learned my addr 172.21.15.31:0/2698094004 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:01.818 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.815+0000 7f96d77fe700 1 -- 172.21.15.31:0/2698094004 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d81086b0 con 0x7f96d8107bd0 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.815+0000 7f96d77fe700 1 --2- 172.21.15.31:0/2698094004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d8107ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f96c80098a0 tx=0x7f96c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa82e12e434087 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.815+0000 7f96d67fc700 1 -- 172.21.15.31:0/2698094004 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96c8004160 con 0x7f96d8107bd0 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.815+0000 7f96d67fc700 1 -- 172.21.15.31:0/2698094004 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f96c8004300 con 0x7f96d8107bd0 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.815+0000 7f96d67fc700 1 -- 172.21.15.31:0/2698094004 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96c8011a80 con 0x7f96d8107bd0 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.816+0000 7f96ddba4700 1 -- 172.21.15.31:0/2698094004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 msgr2=0x7f96d8107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.816+0000 7f96ddba4700 1 --2- 172.21.15.31:0/2698094004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d8107ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f96c80098a0 tx=0x7f96c8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.816+0000 7f96ddba4700 1 -- 172.21.15.31:0/2698094004 shutdown_connections 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.816+0000 7f96ddba4700 1 --2- 172.21.15.31:0/2698094004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d8107ff0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.816+0000 7f96ddba4700 1 -- 172.21.15.31:0/2698094004 >> 172.21.15.31:0/2698094004 conn(0x7f96d81031f0 msgr2=0x7f96d8105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.816+0000 7f96ddba4700 1 -- 172.21.15.31:0/2698094004 shutdown_connections 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.816+0000 7f96ddba4700 1 -- 172.21.15.31:0/2698094004 wait complete. 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.817+0000 7f96ddba4700 1 Processor -- start 2024-04-25T01:45:01.819 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.817+0000 7f96ddba4700 1 -- start start 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.817+0000 7f96ddba4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d807db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.817+0000 7f96ddba4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f96d8109bd0 con 0x7f96d8107bd0 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.817+0000 7f96d77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d807db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.817+0000 7f96d77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d807db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35932/0 (socket says 172.21.15.31:35932) 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.817+0000 7f96d77fe700 1 -- 172.21.15.31:0/295586966 learned_addr learned my addr 172.21.15.31:0/295586966 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.817+0000 7f96d77fe700 1 -- 172.21.15.31:0/295586966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d807e0d0 con 0x7f96d8107bd0 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.817+0000 7f96d77fe700 1 --2- 172.21.15.31:0/295586966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d807db90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f96c8011a50 tx=0x7f96c8011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.818+0000 7f96d4ff9700 1 -- 172.21.15.31:0/295586966 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96c8003a80 con 0x7f96d8107bd0 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.818+0000 7f96d4ff9700 1 -- 172.21.15.31:0/295586966 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f96c8003c20 con 0x7f96d8107bd0 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.818+0000 7f96ddba4700 1 -- 172.21.15.31:0/295586966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d807c2c0 con 0x7f96d8107bd0 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.818+0000 7f96d4ff9700 1 -- 172.21.15.31:0/295586966 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96c801a650 con 0x7f96d8107bd0 2024-04-25T01:45:01.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.818+0000 7f96ddba4700 1 -- 172.21.15.31:0/295586966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d807c800 con 0x7f96d8107bd0 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.819+0000 7f96d4ff9700 1 -- 172.21.15.31:0/295586966 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f96c801a900 con 0x7f96d8107bd0 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.819+0000 7f96ddba4700 1 -- 172.21.15.31:0/295586966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d8063890 con 0x7f96d8107bd0 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.819+0000 7f96d4ff9700 1 --2- 172.21.15.31:0/295586966 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f96c00390a0 0x7f96c003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.819+0000 7f96d4ff9700 1 -- 172.21.15.31:0/295586966 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f96c804d670 con 0x7f96d8107bd0 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.819+0000 7f96d6ffd700 1 --2- 172.21.15.31:0/295586966 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f96c00390a0 0x7f96c003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.819+0000 7f96d6ffd700 1 --2- 172.21.15.31:0/295586966 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f96c00390a0 0x7f96c003b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f96cc006fd0 tx=0x7f96cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:00.824+0000 7f96d4ff9700 1 -- 172.21.15.31:0/295586966 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f96c8019c70 con 0x7f96d8107bd0 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.030+0000 7f96ddba4700 1 -- 172.21.15.31:0/295586966 --> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f96d8104410 con 0x7f96c00390a0 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.245+0000 7f96d4ff9700 1 -- 172.21.15.31:0/295586966 <== mgr.14120 v2:172.21.15.31:6800/3170545029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f96d8104410 con 0x7f96c00390a0 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.247+0000 7f96ddba4700 1 -- 172.21.15.31:0/295586966 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f96c00390a0 msgr2=0x7f96c003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.247+0000 7f96ddba4700 1 --2- 172.21.15.31:0/295586966 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f96c00390a0 0x7f96c003b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f96cc006fd0 tx=0x7f96cc006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.247+0000 7f96ddba4700 1 -- 172.21.15.31:0/295586966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 msgr2=0x7f96d807db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.247+0000 7f96ddba4700 1 --2- 172.21.15.31:0/295586966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d807db90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f96c8011a50 tx=0x7f96c8011b30 comp rx=0 tx=0).stop 2024-04-25T01:45:01.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.247+0000 7f96ddba4700 1 -- 172.21.15.31:0/295586966 shutdown_connections 2024-04-25T01:45:01.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.247+0000 7f96ddba4700 1 --2- 172.21.15.31:0/295586966 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f96c00390a0 0x7f96c003b560 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:01.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.248+0000 7f96ddba4700 1 --2- 172.21.15.31:0/295586966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d8107bd0 0x7f96d807db90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:01.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.248+0000 7f96ddba4700 1 -- 172.21.15.31:0/295586966 >> 172.21.15.31:0/295586966 conn(0x7f96d81031f0 msgr2=0x7f96d8103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:01.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.248+0000 7f96ddba4700 1 -- 172.21.15.31:0/295586966 shutdown_connections 2024-04-25T01:45:01.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:01.248+0000 7f96ddba4700 1 -- 172.21.15.31:0/295586966 wait complete. 2024-04-25T01:45:03.010 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC75hcTaPfNof6hT475tM8Eur1tKkeGsBmM7b+kF8p0dxuMwUe50pyovyw03S1ySRdHWLtVdI4lAH+/04sjLDftySfLeo5oMiSvUO46knjK6tFnuFizYcEFl+09bInPxttROpvPylzycg0wjWk8ks1+bCrMGVWzsmQZHjGt+0eQ5JmdZc8NL/RuwVNCmCLN10NTIMvT5Z9LL5ylBylxeJuBFh9yQY7Zlam/7oOyjxmEd35/JGH0a/QXX69QVNeVy8YOfPSAutTQ8y/vG404ai4jjdPLD72YDi2UerHx9SLPzCRuBrrUdhq2KFIbIBiTLCaVzVdSETYRJtTZwAfqgS/XsnCj/vyD9NznJCD87gEYY2iHzDw0SEU+/RzHOP4cYlF/+VnRw8GchNbvsT65qpo5GhcSAj+roU33uIQMZdmzTTUFyBfmnHEC4qM1hJnuvr6pc9+ZjiAAF+k9sG/QlOKyvmp2HrfuMZIwESSNcu6b3MKVvZT+8ZYyqb3iATyuCLU= ceph-36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:03.010 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.216+0000 7fe1d7a66700 1 Processor -- start 2024-04-25T01:45:03.010 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.217+0000 7fe1d7a66700 1 -- start start 2024-04-25T01:45:03.010 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.217+0000 7fe1d7a66700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d0107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:03.010 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.217+0000 7fe1d7a66700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d0108530 con 0x7fe1d0107bd0 2024-04-25T01:45:03.010 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.217+0000 7fe1d5802700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d0107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:03.010 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.217+0000 7fe1d5802700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d0107ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35942/0 (socket says 172.21.15.31:35942) 2024-04-25T01:45:03.010 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.217+0000 7fe1d5802700 1 -- 172.21.15.31:0/700665056 learned_addr learned my addr 172.21.15.31:0/700665056 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:03.010 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.218+0000 7fe1d5802700 1 -- 172.21.15.31:0/700665056 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d01086b0 con 0x7fe1d0107bd0 2024-04-25T01:45:03.010 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.218+0000 7fe1d5802700 1 --2- 172.21.15.31:0/700665056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d0107ff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc0098a0 tx=0x7fe1cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f868dbcb77a04e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.218+0000 7fe1d4800700 1 -- 172.21.15.31:0/700665056 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1cc004160 con 0x7fe1d0107bd0 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.218+0000 7fe1d4800700 1 -- 172.21.15.31:0/700665056 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe1cc004300 con 0x7fe1d0107bd0 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.218+0000 7fe1d4800700 1 -- 172.21.15.31:0/700665056 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1cc011a80 con 0x7fe1d0107bd0 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.219+0000 7fe1d7a66700 1 -- 172.21.15.31:0/700665056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 msgr2=0x7fe1d0107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.219+0000 7fe1d7a66700 1 --2- 172.21.15.31:0/700665056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d0107ff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc0098a0 tx=0x7fe1cc009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.219+0000 7fe1d7a66700 1 -- 172.21.15.31:0/700665056 shutdown_connections 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.219+0000 7fe1d7a66700 1 --2- 172.21.15.31:0/700665056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d0107ff0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.219+0000 7fe1d7a66700 1 -- 172.21.15.31:0/700665056 >> 172.21.15.31:0/700665056 conn(0x7fe1d01031f0 msgr2=0x7fe1d0105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.219+0000 7fe1d7a66700 1 -- 172.21.15.31:0/700665056 shutdown_connections 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.219+0000 7fe1d7a66700 1 -- 172.21.15.31:0/700665056 wait complete. 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.219+0000 7fe1d7a66700 1 Processor -- start 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.219+0000 7fe1d7a66700 1 -- start start 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1d7a66700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:03.011 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1d7a66700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d0109bd0 con 0x7fe1d0107bd0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1d5802700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1d5802700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d007db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35948/0 (socket says 172.21.15.31:35948) 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1d5802700 1 -- 172.21.15.31:0/263850650 learned_addr learned my addr 172.21.15.31:0/263850650 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1d5802700 1 -- 172.21.15.31:0/263850650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d007e0d0 con 0x7fe1d0107bd0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1d5802700 1 --2- 172.21.15.31:0/263850650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d007db90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006af0 tx=0x7fe1cc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1c2ffd700 1 -- 172.21.15.31:0/263850650 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1cc0039e0 con 0x7fe1d0107bd0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1c2ffd700 1 -- 172.21.15.31:0/263850650 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe1cc003b80 con 0x7fe1d0107bd0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1d7a66700 1 -- 172.21.15.31:0/263850650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d007c2c0 con 0x7fe1d0107bd0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1c2ffd700 1 -- 172.21.15.31:0/263850650 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1cc0196a0 con 0x7fe1d0107bd0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.220+0000 7fe1d7a66700 1 -- 172.21.15.31:0/263850650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d007c800 con 0x7fe1d0107bd0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.221+0000 7fe1c2ffd700 1 -- 172.21.15.31:0/263850650 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fe1cc019950 con 0x7fe1d0107bd0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.221+0000 7fe1d7a66700 1 -- 172.21.15.31:0/263850650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d0063890 con 0x7fe1d0107bd0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.221+0000 7fe1c2ffd700 1 --2- 172.21.15.31:0/263850650 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe1bc039050 0x7fe1bc03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.221+0000 7fe1c2ffd700 1 -- 172.21.15.31:0/263850650 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe1cc04d6f0 con 0x7fe1d0107bd0 2024-04-25T01:45:03.012 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.222+0000 7fe1d5001700 1 --2- 172.21.15.31:0/263850650 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe1bc039050 0x7fe1bc03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.222+0000 7fe1d5001700 1 --2- 172.21.15.31:0/263850650 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe1bc039050 0x7fe1bc03b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4006fd0 tx=0x7fe1c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.227+0000 7fe1c2ffd700 1 -- 172.21.15.31:0/263850650 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe1cc023c90 con 0x7fe1d0107bd0 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.433+0000 7fe1d7a66700 1 -- 172.21.15.31:0/263850650 --> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fe1d007d060 con 0x7fe1bc039050 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.434+0000 7fe1c2ffd700 1 -- 172.21.15.31:0/263850650 <== mgr.14120 v2:172.21.15.31:6800/3170545029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fe1d007d060 con 0x7fe1bc039050 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.436+0000 7fe1d7a66700 1 -- 172.21.15.31:0/263850650 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe1bc039050 msgr2=0x7fe1bc03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.436+0000 7fe1d7a66700 1 --2- 172.21.15.31:0/263850650 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe1bc039050 0x7fe1bc03b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4006fd0 tx=0x7fe1c4006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.436+0000 7fe1d7a66700 1 -- 172.21.15.31:0/263850650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 msgr2=0x7fe1d007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.436+0000 7fe1d7a66700 1 --2- 172.21.15.31:0/263850650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d007db90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006af0 tx=0x7fe1cc011c40 comp rx=0 tx=0).stop 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.436+0000 7fe1d7a66700 1 -- 172.21.15.31:0/263850650 shutdown_connections 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.436+0000 7fe1d7a66700 1 --2- 172.21.15.31:0/263850650 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe1bc039050 0x7fe1bc03b510 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.436+0000 7fe1d7a66700 1 --2- 172.21.15.31:0/263850650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1d0107bd0 0x7fe1d007db90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.436+0000 7fe1d7a66700 1 -- 172.21.15.31:0/263850650 >> 172.21.15.31:0/263850650 conn(0x7fe1d01031f0 msgr2=0x7fe1d0103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.436+0000 7fe1d7a66700 1 -- 172.21.15.31:0/263850650 shutdown_connections 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:02.436+0000 7fe1d7a66700 1 -- 172.21.15.31:0/263850650 wait complete. 2024-04-25T01:45:03.013 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:45:03.014 INFO:teuthology.orchestra.run.smithi031.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T01:45:03.014 INFO:teuthology.orchestra.run.smithi031.stdout:Adding host smithi031... 2024-04-25T01:45:03.645 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:03 smithi031 ceph-mon[29204]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:04.512 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:04 smithi031 ceph-mon[29204]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:04.513 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:04 smithi031 ceph-mon[29204]: Deploying cephadm binary to smithi031 2024-04-25T01:45:08.458 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Added host 'smithi031' with addr '172.21.15.31' 2024-04-25T01:45:08.458 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.432+0000 7efc729a6700 1 Processor -- start 2024-04-25T01:45:08.458 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.432+0000 7efc729a6700 1 -- start start 2024-04-25T01:45:08.458 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.432+0000 7efc729a6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:08.458 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.432+0000 7efc729a6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efc6c108530 con 0x7efc6c107bd0 2024-04-25T01:45:08.458 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.432+0000 7efc6bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:08.458 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.432+0000 7efc6bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c107ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40704/0 (socket says 172.21.15.31:40704) 2024-04-25T01:45:08.458 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.432+0000 7efc6bfff700 1 -- 172.21.15.31:0/3459303280 learned_addr learned my addr 172.21.15.31:0/3459303280 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.432+0000 7efc6bfff700 1 -- 172.21.15.31:0/3459303280 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6c1086b0 con 0x7efc6c107bd0 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.433+0000 7efc6bfff700 1 --2- 172.21.15.31:0/3459303280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c107ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0098a0 tx=0x7efc5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=382bf216b9967ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.433+0000 7efc6affd700 1 -- 172.21.15.31:0/3459303280 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc5c004160 con 0x7efc6c107bd0 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.433+0000 7efc6affd700 1 -- 172.21.15.31:0/3459303280 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7efc5c004300 con 0x7efc6c107bd0 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.433+0000 7efc6affd700 1 -- 172.21.15.31:0/3459303280 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc5c011a80 con 0x7efc6c107bd0 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.433+0000 7efc729a6700 1 -- 172.21.15.31:0/3459303280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 msgr2=0x7efc6c107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.433+0000 7efc729a6700 1 --2- 172.21.15.31:0/3459303280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c107ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0098a0 tx=0x7efc5c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.433+0000 7efc729a6700 1 -- 172.21.15.31:0/3459303280 shutdown_connections 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.433+0000 7efc729a6700 1 --2- 172.21.15.31:0/3459303280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c107ff0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.433+0000 7efc729a6700 1 -- 172.21.15.31:0/3459303280 >> 172.21.15.31:0/3459303280 conn(0x7efc6c1031f0 msgr2=0x7efc6c105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.434+0000 7efc729a6700 1 -- 172.21.15.31:0/3459303280 shutdown_connections 2024-04-25T01:45:08.459 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.434+0000 7efc729a6700 1 -- 172.21.15.31:0/3459303280 wait complete. 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.434+0000 7efc729a6700 1 Processor -- start 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.434+0000 7efc729a6700 1 -- start start 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc729a6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c19ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc729a6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efc6c109bd0 con 0x7efc6c107bd0 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc6bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c19ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc6bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c19ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40706/0 (socket says 172.21.15.31:40706) 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc6bfff700 1 -- 172.21.15.31:0/2551037230 learned_addr learned my addr 172.21.15.31:0/2551037230 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc6bfff700 1 -- 172.21.15.31:0/2551037230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6c19bfa0 con 0x7efc6c107bd0 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc6bfff700 1 --2- 172.21.15.31:0/2551037230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c19ba60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efc5c011a50 tx=0x7efc5c011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc697fa700 1 -- 172.21.15.31:0/2551037230 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc5c003a80 con 0x7efc6c107bd0 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc697fa700 1 -- 172.21.15.31:0/2551037230 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7efc5c003c20 con 0x7efc6c107bd0 2024-04-25T01:45:08.460 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc697fa700 1 -- 172.21.15.31:0/2551037230 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc5c01a650 con 0x7efc6c107bd0 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc729a6700 1 -- 172.21.15.31:0/2551037230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc6c19c230 con 0x7efc6c107bd0 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.435+0000 7efc729a6700 1 -- 172.21.15.31:0/2551037230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc6c19ef20 con 0x7efc6c107bd0 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.436+0000 7efc697fa700 1 -- 172.21.15.31:0/2551037230 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7efc5c01a8c0 con 0x7efc6c107bd0 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.436+0000 7efc697fa700 1 --2- 172.21.15.31:0/2551037230 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7efc54039050 0x7efc5403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.436+0000 7efc729a6700 1 -- 172.21.15.31:0/2551037230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc6c063890 con 0x7efc6c107bd0 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.437+0000 7efc697fa700 1 -- 172.21.15.31:0/2551037230 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efc5c04d6d0 con 0x7efc6c107bd0 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.437+0000 7efc6b7fe700 1 --2- 172.21.15.31:0/2551037230 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7efc54039050 0x7efc5403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.437+0000 7efc6b7fe700 1 --2- 172.21.15.31:0/2551037230 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7efc54039050 0x7efc5403b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efc60006fd0 tx=0x7efc60006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.442+0000 7efc697fa700 1 -- 172.21.15.31:0/2551037230 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efc5c022340 con 0x7efc6c107bd0 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:03.652+0000 7efc729a6700 1 -- 172.21.15.31:0/2551037230 --> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}) -- 0x7efc6c104410 con 0x7efc54039050 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:07.860+0000 7efc697fa700 1 -- 172.21.15.31:0/2551037230 <== mgr.14120 v2:172.21.15.31:6800/3170545029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7efc6c104410 con 0x7efc54039050 2024-04-25T01:45:08.461 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:07.863+0000 7efc729a6700 1 -- 172.21.15.31:0/2551037230 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7efc54039050 msgr2=0x7efc5403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:08.462 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:07.863+0000 7efc729a6700 1 --2- 172.21.15.31:0/2551037230 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7efc54039050 0x7efc5403b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efc60006fd0 tx=0x7efc60006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:08.462 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:07.863+0000 7efc729a6700 1 -- 172.21.15.31:0/2551037230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 msgr2=0x7efc6c19ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:08.462 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:07.863+0000 7efc729a6700 1 --2- 172.21.15.31:0/2551037230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c19ba60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efc5c011a50 tx=0x7efc5c011b30 comp rx=0 tx=0).stop 2024-04-25T01:45:08.462 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:07.863+0000 7efc729a6700 1 -- 172.21.15.31:0/2551037230 shutdown_connections 2024-04-25T01:45:08.462 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:07.863+0000 7efc729a6700 1 --2- 172.21.15.31:0/2551037230 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7efc54039050 0x7efc5403b510 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:08.462 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:07.863+0000 7efc729a6700 1 --2- 172.21.15.31:0/2551037230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6c107bd0 0x7efc6c19ba60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:08.462 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:07.863+0000 7efc729a6700 1 -- 172.21.15.31:0/2551037230 >> 172.21.15.31:0/2551037230 conn(0x7efc6c1031f0 msgr2=0x7efc6c103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:08.462 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:07.863+0000 7efc729a6700 1 -- 172.21.15.31:0/2551037230 shutdown_connections 2024-04-25T01:45:08.462 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:07.863+0000 7efc729a6700 1 -- 172.21.15.31:0/2551037230 wait complete. 2024-04-25T01:45:08.462 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mon service... 2024-04-25T01:45:09.146 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:08 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:45:09.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:08 smithi031 ceph-mon[29204]: Added host smithi031 2024-04-25T01:45:09.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:08 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:10.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:10 smithi031 ceph-mon[29204]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:10.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:10 smithi031 ceph-mon[29204]: Saving service mon spec with placement count:5 2024-04-25T01:45:10.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:10 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:45:10.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T01:45:10.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.923+0000 7febcd68c700 1 Processor -- start 2024-04-25T01:45:10.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.923+0000 7febcd68c700 1 -- start start 2024-04-25T01:45:10.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.923+0000 7febcd68c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc8108440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:10.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.923+0000 7febcd68c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7febc8108a10 con 0x7febc8108020 2024-04-25T01:45:10.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.924+0000 7febc6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc8108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:10.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.924+0000 7febc6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc8108440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40708/0 (socket says 172.21.15.31:40708) 2024-04-25T01:45:10.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.924+0000 7febc6ffd700 1 -- 172.21.15.31:0/4285980520 learned_addr learned my addr 172.21.15.31:0/4285980520 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:10.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.924+0000 7febc6ffd700 1 -- 172.21.15.31:0/4285980520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc8108b90 con 0x7febc8108020 2024-04-25T01:45:10.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.924+0000 7febc6ffd700 1 --2- 172.21.15.31:0/4285980520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc8108440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7febc00098a0 tx=0x7febc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7097a1cca3a22659 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:10.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.924+0000 7febc5ffb700 1 -- 172.21.15.31:0/4285980520 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febc0004160 con 0x7febc8108020 2024-04-25T01:45:10.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.924+0000 7febc5ffb700 1 -- 172.21.15.31:0/4285980520 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7febc0004300 con 0x7febc8108020 2024-04-25T01:45:10.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.924+0000 7febc5ffb700 1 -- 172.21.15.31:0/4285980520 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febc0011a80 con 0x7febc8108020 2024-04-25T01:45:10.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.925+0000 7febcd68c700 1 -- 172.21.15.31:0/4285980520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 msgr2=0x7febc8108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:10.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.925+0000 7febcd68c700 1 --2- 172.21.15.31:0/4285980520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc8108440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7febc00098a0 tx=0x7febc0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:10.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.925+0000 7febcd68c700 1 -- 172.21.15.31:0/4285980520 shutdown_connections 2024-04-25T01:45:10.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.925+0000 7febcd68c700 1 --2- 172.21.15.31:0/4285980520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc8108440 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:10.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.925+0000 7febcd68c700 1 -- 172.21.15.31:0/4285980520 >> 172.21.15.31:0/4285980520 conn(0x7febc8101750 msgr2=0x7febc8103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:10.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.925+0000 7febcd68c700 1 -- 172.21.15.31:0/4285980520 shutdown_connections 2024-04-25T01:45:10.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.925+0000 7febcd68c700 1 -- 172.21.15.31:0/4285980520 wait complete. 2024-04-25T01:45:10.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.925+0000 7febcd68c700 1 Processor -- start 2024-04-25T01:45:10.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.926+0000 7febcd68c700 1 -- start start 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.926+0000 7febcd68c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc819dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.926+0000 7febcd68c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7febc81068c0 con 0x7febc8108020 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.926+0000 7febc6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc819dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.926+0000 7febc6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc819dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40720/0 (socket says 172.21.15.31:40720) 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.926+0000 7febc6ffd700 1 -- 172.21.15.31:0/3584090479 learned_addr learned my addr 172.21.15.31:0/3584090479 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.926+0000 7febc6ffd700 1 -- 172.21.15.31:0/3584090479 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc819e120 con 0x7febc8108020 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.926+0000 7febc6ffd700 1 --2- 172.21.15.31:0/3584090479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc819dbe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febc0006af0 tx=0x7febc0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.926+0000 7febbffff700 1 -- 172.21.15.31:0/3584090479 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febc00039e0 con 0x7febc8108020 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.927+0000 7febbffff700 1 -- 172.21.15.31:0/3584090479 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7febc0003b80 con 0x7febc8108020 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.927+0000 7febcd68c700 1 -- 172.21.15.31:0/3584090479 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febc819e410 con 0x7febc8108020 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.927+0000 7febcd68c700 1 -- 172.21.15.31:0/3584090479 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febc81a10a0 con 0x7febc8108020 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.927+0000 7febbffff700 1 -- 172.21.15.31:0/3584090479 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febc00196a0 con 0x7febc8108020 2024-04-25T01:45:10.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.927+0000 7febbffff700 1 -- 172.21.15.31:0/3584090479 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7febc0019990 con 0x7febc8108020 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.927+0000 7febcd68c700 1 -- 172.21.15.31:0/3584090479 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feba8005360 con 0x7febc8108020 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.928+0000 7febbffff700 1 --2- 172.21.15.31:0/3584090479 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7febb4039050 0x7febb403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.928+0000 7febbffff700 1 -- 172.21.15.31:0/3584090479 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7febc004d330 con 0x7febc8108020 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.928+0000 7febc67fc700 1 --2- 172.21.15.31:0/3584090479 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7febb4039050 0x7febb403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.928+0000 7febc67fc700 1 --2- 172.21.15.31:0/3584090479 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7febb4039050 0x7febb403b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7febb0006fd0 tx=0x7febb0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:08.933+0000 7febbffff700 1 -- 172.21.15.31:0/3584090479 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7febc0024c70 con 0x7febc8108020 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.153+0000 7febcd68c700 1 -- 172.21.15.31:0/3584090479 --> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7feba8000bf0 con 0x7febb4039050 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.158+0000 7febbffff700 1 -- 172.21.15.31:0/3584090479 <== mgr.14120 v2:172.21.15.31:6800/3170545029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7feba8000bf0 con 0x7febb4039050 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.160+0000 7febcd68c700 1 -- 172.21.15.31:0/3584090479 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7febb4039050 msgr2=0x7febb403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.160+0000 7febcd68c700 1 --2- 172.21.15.31:0/3584090479 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7febb4039050 0x7febb403b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7febb0006fd0 tx=0x7febb0006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.160+0000 7febcd68c700 1 -- 172.21.15.31:0/3584090479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 msgr2=0x7febc819dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.160+0000 7febcd68c700 1 --2- 172.21.15.31:0/3584090479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc819dbe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febc0006af0 tx=0x7febc0011c40 comp rx=0 tx=0).stop 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.161+0000 7febcd68c700 1 -- 172.21.15.31:0/3584090479 shutdown_connections 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.161+0000 7febcd68c700 1 --2- 172.21.15.31:0/3584090479 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7febb4039050 0x7febb403b510 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:10.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.161+0000 7febcd68c700 1 --2- 172.21.15.31:0/3584090479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8108020 0x7febc819dbe0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:10.728 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.161+0000 7febcd68c700 1 -- 172.21.15.31:0/3584090479 >> 172.21.15.31:0/3584090479 conn(0x7febc8101750 msgr2=0x7febc8194200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:10.728 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.161+0000 7febcd68c700 1 -- 172.21.15.31:0/3584090479 shutdown_connections 2024-04-25T01:45:10.728 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:09.161+0000 7febcd68c700 1 -- 172.21.15.31:0/3584090479 wait complete. 2024-04-25T01:45:10.728 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mgr service... 2024-04-25T01:45:11.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:11 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:45:11.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:11 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:45:11.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T01:45:11.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.153+0000 7fa4500ae700 1 Processor -- start 2024-04-25T01:45:11.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.154+0000 7fa4500ae700 1 -- start start 2024-04-25T01:45:11.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.154+0000 7fa4500ae700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa448106260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:11.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.154+0000 7fa4500ae700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa448106830 con 0x7fa448105e40 2024-04-25T01:45:11.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.154+0000 7fa44de4a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa448106260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.154+0000 7fa44de4a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa448106260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40724/0 (socket says 172.21.15.31:40724) 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.154+0000 7fa44de4a700 1 -- 172.21.15.31:0/320225635 learned_addr learned my addr 172.21.15.31:0/320225635 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.154+0000 7fa44de4a700 1 -- 172.21.15.31:0/320225635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa448108dd0 con 0x7fa448105e40 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.154+0000 7fa44de4a700 1 --2- 172.21.15.31:0/320225635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa448106260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4380098a0 tx=0x7fa438009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=944cea2cba131bac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.155+0000 7fa44ce48700 1 -- 172.21.15.31:0/320225635 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa438004160 con 0x7fa448105e40 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.155+0000 7fa44ce48700 1 -- 172.21.15.31:0/320225635 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa438004300 con 0x7fa448105e40 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.155+0000 7fa44ce48700 1 -- 172.21.15.31:0/320225635 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa438011a50 con 0x7fa448105e40 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.155+0000 7fa4500ae700 1 -- 172.21.15.31:0/320225635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 msgr2=0x7fa448106260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.155+0000 7fa4500ae700 1 --2- 172.21.15.31:0/320225635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa448106260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4380098a0 tx=0x7fa438009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.155+0000 7fa4500ae700 1 -- 172.21.15.31:0/320225635 shutdown_connections 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.155+0000 7fa4500ae700 1 --2- 172.21.15.31:0/320225635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa448106260 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.155+0000 7fa4500ae700 1 -- 172.21.15.31:0/320225635 >> 172.21.15.31:0/320225635 conn(0x7fa448101750 msgr2=0x7fa448103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:11.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.155+0000 7fa4500ae700 1 -- 172.21.15.31:0/320225635 shutdown_connections 2024-04-25T01:45:11.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.156+0000 7fa4500ae700 1 -- 172.21.15.31:0/320225635 wait complete. 2024-04-25T01:45:11.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.156+0000 7fa4500ae700 1 Processor -- start 2024-04-25T01:45:11.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.156+0000 7fa4500ae700 1 -- start start 2024-04-25T01:45:11.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.156+0000 7fa4500ae700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa44819ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:11.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.156+0000 7fa4500ae700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa44810c180 con 0x7fa448105e40 2024-04-25T01:45:11.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.156+0000 7fa44de4a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa44819ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:11.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.156+0000 7fa44de4a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa44819ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40738/0 (socket says 172.21.15.31:40738) 2024-04-25T01:45:11.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.156+0000 7fa44de4a700 1 -- 172.21.15.31:0/3287388583 learned_addr learned my addr 172.21.15.31:0/3287388583 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:11.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.157+0000 7fa44de4a700 1 -- 172.21.15.31:0/3287388583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44819bfd0 con 0x7fa448105e40 2024-04-25T01:45:11.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.157+0000 7fa44de4a700 1 --2- 172.21.15.31:0/3287388583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa44819ba90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa438006b20 tx=0x7fa438011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:11.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.157+0000 7fa43effd700 1 -- 172.21.15.31:0/3287388583 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4380039e0 con 0x7fa448105e40 2024-04-25T01:45:11.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.157+0000 7fa4500ae700 1 -- 172.21.15.31:0/3287388583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa44819c260 con 0x7fa448105e40 2024-04-25T01:45:11.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.157+0000 7fa43effd700 1 -- 172.21.15.31:0/3287388583 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa43801a8e0 con 0x7fa448105e40 2024-04-25T01:45:11.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.157+0000 7fa43effd700 1 -- 172.21.15.31:0/3287388583 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4380247e0 con 0x7fa448105e40 2024-04-25T01:45:11.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.157+0000 7fa4500ae700 1 -- 172.21.15.31:0/3287388583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa44819ef50 con 0x7fa448105e40 2024-04-25T01:45:11.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.158+0000 7fa43effd700 1 -- 172.21.15.31:0/3287388583 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fa438022020 con 0x7fa448105e40 2024-04-25T01:45:11.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.158+0000 7fa4500ae700 1 -- 172.21.15.31:0/3287388583 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa448063890 con 0x7fa448105e40 2024-04-25T01:45:11.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.158+0000 7fa43effd700 1 --2- 172.21.15.31:0/3287388583 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fa4340390a0 0x7fa43403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:11.983 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.158+0000 7fa43effd700 1 -- 172.21.15.31:0/3287388583 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa43804d5a0 con 0x7fa448105e40 2024-04-25T01:45:11.983 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.158+0000 7fa44d649700 1 --2- 172.21.15.31:0/3287388583 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fa4340390a0 0x7fa43403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:11.983 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.159+0000 7fa44d649700 1 --2- 172.21.15.31:0/3287388583 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fa4340390a0 0x7fa43403b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa444006fd0 tx=0x7fa444006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:11.983 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.163+0000 7fa43effd700 1 -- 172.21.15.31:0/3287388583 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa43802cdf0 con 0x7fa448105e40 2024-04-25T01:45:11.983 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.384+0000 7fa4500ae700 1 -- 172.21.15.31:0/3287388583 --> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fa448102910 con 0x7fa4340390a0 2024-04-25T01:45:11.983 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.393+0000 7fa43effd700 1 -- 172.21.15.31:0/3287388583 <== mgr.14120 v2:172.21.15.31:6800/3170545029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa448102910 con 0x7fa4340390a0 2024-04-25T01:45:11.983 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.395+0000 7fa4500ae700 1 -- 172.21.15.31:0/3287388583 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fa4340390a0 msgr2=0x7fa43403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:11.983 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.395+0000 7fa4500ae700 1 --2- 172.21.15.31:0/3287388583 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fa4340390a0 0x7fa43403b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa444006fd0 tx=0x7fa444006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:11.983 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.395+0000 7fa4500ae700 1 -- 172.21.15.31:0/3287388583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 msgr2=0x7fa44819ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:11.984 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.395+0000 7fa4500ae700 1 --2- 172.21.15.31:0/3287388583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa44819ba90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa438006b20 tx=0x7fa438011ad0 comp rx=0 tx=0).stop 2024-04-25T01:45:11.984 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.395+0000 7fa4500ae700 1 -- 172.21.15.31:0/3287388583 shutdown_connections 2024-04-25T01:45:11.984 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.395+0000 7fa4500ae700 1 --2- 172.21.15.31:0/3287388583 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fa4340390a0 0x7fa43403b560 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:11.984 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.395+0000 7fa4500ae700 1 --2- 172.21.15.31:0/3287388583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448105e40 0x7fa44819ba90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:11.984 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.395+0000 7fa4500ae700 1 -- 172.21.15.31:0/3287388583 >> 172.21.15.31:0/3287388583 conn(0x7fa448101750 msgr2=0x7fa448102200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:11.984 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.396+0000 7fa4500ae700 1 -- 172.21.15.31:0/3287388583 shutdown_connections 2024-04-25T01:45:11.984 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:11.396+0000 7fa4500ae700 1 -- 172.21.15.31:0/3287388583 wait complete. 2024-04-25T01:45:12.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:12 smithi031 ceph-mon[29204]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:12.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:12 smithi031 ceph-mon[29204]: Saving service mgr spec with placement count:2 2024-04-25T01:45:12.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:12 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:45:13.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:13 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1645658965' entity='client.admin' 2024-04-25T01:45:13.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:13 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:45:13.853 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.436+0000 7ff64935b700 1 Processor -- start 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.436+0000 7ff64935b700 1 -- start start 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.436+0000 7ff64935b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64407c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.436+0000 7ff64935b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff64407e190 con 0x7ff64407dd70 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.436+0000 7ff642ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64407c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.436+0000 7ff642ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64407c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40744/0 (socket says 172.21.15.31:40744) 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.436+0000 7ff642ffd700 1 -- 172.21.15.31:0/101639482 learned_addr learned my addr 172.21.15.31:0/101639482 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.437+0000 7ff642ffd700 1 -- 172.21.15.31:0/101639482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64407c820 con 0x7ff64407dd70 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.437+0000 7ff642ffd700 1 --2- 172.21.15.31:0/101639482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64407c2e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff6340098a0 tx=0x7ff634009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbbab8f610ac89a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.437+0000 7ff641ffb700 1 -- 172.21.15.31:0/101639482 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff634004160 con 0x7ff64407dd70 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.437+0000 7ff641ffb700 1 -- 172.21.15.31:0/101639482 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff634004300 con 0x7ff64407dd70 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.437+0000 7ff641ffb700 1 -- 172.21.15.31:0/101639482 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff634011a50 con 0x7ff64407dd70 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.438+0000 7ff64935b700 1 -- 172.21.15.31:0/101639482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 msgr2=0x7ff64407c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:13.854 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.438+0000 7ff64935b700 1 --2- 172.21.15.31:0/101639482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64407c2e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff6340098a0 tx=0x7ff634009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.438+0000 7ff64935b700 1 -- 172.21.15.31:0/101639482 shutdown_connections 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.438+0000 7ff64935b700 1 --2- 172.21.15.31:0/101639482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64407c2e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.438+0000 7ff64935b700 1 -- 172.21.15.31:0/101639482 >> 172.21.15.31:0/101639482 conn(0x7ff6441031f0 msgr2=0x7ff644105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.438+0000 7ff64935b700 1 -- 172.21.15.31:0/101639482 shutdown_connections 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.438+0000 7ff64935b700 1 -- 172.21.15.31:0/101639482 wait complete. 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.438+0000 7ff64935b700 1 Processor -- start 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.439+0000 7ff64935b700 1 -- start start 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.439+0000 7ff64935b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64419dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.439+0000 7ff64935b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff64410e060 con 0x7ff64407dd70 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.439+0000 7ff642ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64419dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.439+0000 7ff642ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64419dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40752/0 (socket says 172.21.15.31:40752) 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.439+0000 7ff642ffd700 1 -- 172.21.15.31:0/1645658965 learned_addr learned my addr 172.21.15.31:0/1645658965 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:13.855 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.439+0000 7ff642ffd700 1 -- 172.21.15.31:0/1645658965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64419e150 con 0x7ff64407dd70 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.439+0000 7ff642ffd700 1 --2- 172.21.15.31:0/1645658965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64419dc10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff634011a20 tx=0x7ff634011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.439+0000 7ff62bfff700 1 -- 172.21.15.31:0/1645658965 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff634003b70 con 0x7ff64407dd70 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.440+0000 7ff62bfff700 1 -- 172.21.15.31:0/1645658965 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff63401a8e0 con 0x7ff64407dd70 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.440+0000 7ff62bfff700 1 -- 172.21.15.31:0/1645658965 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff634024990 con 0x7ff64407dd70 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.440+0000 7ff64935b700 1 -- 172.21.15.31:0/1645658965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64419e3e0 con 0x7ff64407dd70 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.440+0000 7ff64935b700 1 -- 172.21.15.31:0/1645658965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6441a10d0 con 0x7ff64407dd70 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.440+0000 7ff62bfff700 1 -- 172.21.15.31:0/1645658965 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7ff634022020 con 0x7ff64407dd70 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.441+0000 7ff62bfff700 1 --2- 172.21.15.31:0/1645658965 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7ff62c039050 0x7ff62c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.441+0000 7ff62bfff700 1 -- 172.21.15.31:0/1645658965 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff63404e6d0 con 0x7ff64407dd70 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.441+0000 7ff6427fc700 1 --2- 172.21.15.31:0/1645658965 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7ff62c039050 0x7ff62c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.441+0000 7ff64935b700 1 -- 172.21.15.31:0/1645658965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff644063890 con 0x7ff64407dd70 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.441+0000 7ff6427fc700 1 --2- 172.21.15.31:0/1645658965 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7ff62c039050 0x7ff62c03b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff638006fd0 tx=0x7ff638006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.446+0000 7ff62bfff700 1 -- 172.21.15.31:0/1645658965 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff634020c10 con 0x7ff64407dd70 2024-04-25T01:45:13.856 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.664+0000 7ff64935b700 1 -- 172.21.15.31:0/1645658965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7ff6441a1570 con 0x7ff64407dd70 2024-04-25T01:45:13.857 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.667+0000 7ff62bfff700 1 -- 172.21.15.31:0/1645658965 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7ff63400b5a0 con 0x7ff64407dd70 2024-04-25T01:45:13.857 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.670+0000 7ff64935b700 1 -- 172.21.15.31:0/1645658965 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7ff62c039050 msgr2=0x7ff62c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:13.857 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.670+0000 7ff64935b700 1 --2- 172.21.15.31:0/1645658965 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7ff62c039050 0x7ff62c03b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff638006fd0 tx=0x7ff638006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:13.857 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.670+0000 7ff64935b700 1 -- 172.21.15.31:0/1645658965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 msgr2=0x7ff64419dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:13.857 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.670+0000 7ff64935b700 1 --2- 172.21.15.31:0/1645658965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64419dc10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff634011a20 tx=0x7ff634011b00 comp rx=0 tx=0).stop 2024-04-25T01:45:13.857 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.670+0000 7ff64935b700 1 -- 172.21.15.31:0/1645658965 shutdown_connections 2024-04-25T01:45:13.857 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.670+0000 7ff64935b700 1 --2- 172.21.15.31:0/1645658965 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7ff62c039050 0x7ff62c03b510 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:13.857 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.670+0000 7ff64935b700 1 --2- 172.21.15.31:0/1645658965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff64407dd70 0x7ff64419dc10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:13.857 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.670+0000 7ff64935b700 1 -- 172.21.15.31:0/1645658965 >> 172.21.15.31:0/1645658965 conn(0x7ff6441031f0 msgr2=0x7ff644103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:13.857 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.670+0000 7ff64935b700 1 -- 172.21.15.31:0/1645658965 shutdown_connections 2024-04-25T01:45:13.857 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:12.670+0000 7ff64935b700 1 -- 172.21.15.31:0/1645658965 wait complete. 2024-04-25T01:45:15.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.313+0000 7fe5f8b5b700 1 Processor -- start 2024-04-25T01:45:15.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.313+0000 7fe5f8b5b700 1 -- start start 2024-04-25T01:45:15.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.314+0000 7fe5f8b5b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f4107fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:15.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.314+0000 7fe5f8b5b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41085a0 con 0x7fe5f4107bb0 2024-04-25T01:45:15.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.314+0000 7fe5f259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f4107fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:15.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.314+0000 7fe5f259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f4107fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60682/0 (socket says 172.21.15.31:60682) 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.314+0000 7fe5f259c700 1 -- 172.21.15.31:0/3304583238 learned_addr learned my addr 172.21.15.31:0/3304583238 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.314+0000 7fe5f259c700 1 -- 172.21.15.31:0/3304583238 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f4108d60 con 0x7fe5f4107bb0 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.314+0000 7fe5f259c700 1 --2- 172.21.15.31:0/3304583238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f4107fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40098a0 tx=0x7fe5e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff03c7fca67b179 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.314+0000 7fe5f159a700 1 -- 172.21.15.31:0/3304583238 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e4004160 con 0x7fe5f4107bb0 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.315+0000 7fe5f159a700 1 -- 172.21.15.31:0/3304583238 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5e4004300 con 0x7fe5f4107bb0 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.315+0000 7fe5f159a700 1 -- 172.21.15.31:0/3304583238 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e4011a50 con 0x7fe5f4107bb0 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.315+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/3304583238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 msgr2=0x7fe5f4107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.315+0000 7fe5f8b5b700 1 --2- 172.21.15.31:0/3304583238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f4107fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40098a0 tx=0x7fe5e4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.315+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/3304583238 shutdown_connections 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.315+0000 7fe5f8b5b700 1 --2- 172.21.15.31:0/3304583238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f4107fd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.315+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/3304583238 >> 172.21.15.31:0/3304583238 conn(0x7fe5f4103150 msgr2=0x7fe5f4105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.315+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/3304583238 shutdown_connections 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.315+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/3304583238 wait complete. 2024-04-25T01:45:15.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.316+0000 7fe5f8b5b700 1 Processor -- start 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.316+0000 7fe5f8b5b700 1 -- start start 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.316+0000 7fe5f8b5b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f419ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.316+0000 7fe5f8b5b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f410c110 con 0x7fe5f4107bb0 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.316+0000 7fe5f259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f419ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.316+0000 7fe5f259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f419ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60696/0 (socket says 172.21.15.31:60696) 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.317+0000 7fe5f259c700 1 -- 172.21.15.31:0/4252144153 learned_addr learned my addr 172.21.15.31:0/4252144153 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.317+0000 7fe5f259c700 1 -- 172.21.15.31:0/4252144153 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f419bfc0 con 0x7fe5f4107bb0 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.317+0000 7fe5f259c700 1 --2- 172.21.15.31:0/4252144153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f419ba80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4011710 tx=0x7fe5e4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.317+0000 7fe5e37fe700 1 -- 172.21.15.31:0/4252144153 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e40039c0 con 0x7fe5f4107bb0 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.317+0000 7fe5e37fe700 1 -- 172.21.15.31:0/4252144153 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5e4003b60 con 0x7fe5f4107bb0 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.317+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/4252144153 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f419c250 con 0x7fe5f4107bb0 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.317+0000 7fe5e37fe700 1 -- 172.21.15.31:0/4252144153 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e401a690 con 0x7fe5f4107bb0 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.317+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/4252144153 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f419ef40 con 0x7fe5f4107bb0 2024-04-25T01:45:15.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.318+0000 7fe5e37fe700 1 -- 172.21.15.31:0/4252144153 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fe5e401a940 con 0x7fe5f4107bb0 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.318+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/4252144153 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f4063890 con 0x7fe5f4107bb0 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.319+0000 7fe5e37fe700 1 --2- 172.21.15.31:0/4252144153 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe5dc039050 0x7fe5dc03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.319+0000 7fe5e37fe700 1 -- 172.21.15.31:0/4252144153 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe5e404d5f0 con 0x7fe5f4107bb0 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.319+0000 7fe5f1d9b700 1 --2- 172.21.15.31:0/4252144153 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe5dc039050 0x7fe5dc03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.319+0000 7fe5f1d9b700 1 --2- 172.21.15.31:0/4252144153 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe5dc039050 0x7fe5dc03b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8006fd0 tx=0x7fe5e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.324+0000 7fe5e37fe700 1 -- 172.21.15.31:0/4252144153 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe5e4019c70 con 0x7fe5f4107bb0 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.531+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/4252144153 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fe5f419f560 con 0x7fe5f4107bb0 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.534+0000 7fe5e37fe700 1 -- 172.21.15.31:0/4252144153 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fe5e4020240 con 0x7fe5f4107bb0 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.536+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/4252144153 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe5dc039050 msgr2=0x7fe5dc03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.536+0000 7fe5f8b5b700 1 --2- 172.21.15.31:0/4252144153 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe5dc039050 0x7fe5dc03b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8006fd0 tx=0x7fe5e8006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.536+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/4252144153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 msgr2=0x7fe5f419ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.536+0000 7fe5f8b5b700 1 --2- 172.21.15.31:0/4252144153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f419ba80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4011710 tx=0x7fe5e4011ad0 comp rx=0 tx=0).stop 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.536+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/4252144153 shutdown_connections 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.536+0000 7fe5f8b5b700 1 --2- 172.21.15.31:0/4252144153 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fe5dc039050 0x7fe5dc03b510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:15.104 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.536+0000 7fe5f8b5b700 1 --2- 172.21.15.31:0/4252144153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5f4107bb0 0x7fe5f419ba80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:15.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.536+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/4252144153 >> 172.21.15.31:0/4252144153 conn(0x7fe5f4103150 msgr2=0x7fe5f4103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:15.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.538+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/4252144153 shutdown_connections 2024-04-25T01:45:15.105 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:14.538+0000 7fe5f8b5b700 1 -- 172.21.15.31:0/4252144153 wait complete. 2024-04-25T01:45:15.105 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling the dashboard module... 2024-04-25T01:45:15.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:15 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/4252144153' entity='client.admin' 2024-04-25T01:45:15.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:15 smithi031 ceph-mon[29204]: from='mgr.14120 172.21.15.31:0/4191077064' entity='mgr.x' 2024-04-25T01:45:16.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:16 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3351086604' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T01:45:16.711 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:16 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: ignoring --setuser ceph since I am not root 2024-04-25T01:45:16.711 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:16 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: ignoring --setgroup ceph since I am not root 2024-04-25T01:45:16.980 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:16 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:16.727+0000 7f83a5a14200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:45:16.980 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:16 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:16.917+0000 7f83a5a14200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:45:17.338 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:17 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:17.024+0000 7f83a5a14200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:45:17.338 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:17 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:17.345+0000 7f83a5a14200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:45:17.363 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.515+0000 7fb0e77bb700 1 Processor -- start 2024-04-25T01:45:17.363 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.515+0000 7fb0e77bb700 1 -- start start 2024-04-25T01:45:17.363 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.515+0000 7fb0e77bb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e0106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:17.363 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.515+0000 7fb0e77bb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e01074a0 con 0x7fb0e0106ab0 2024-04-25T01:45:17.363 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.516+0000 7fb0e5557700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e0106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:17.363 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.516+0000 7fb0e5557700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e0106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60704/0 (socket says 172.21.15.31:60704) 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.516+0000 7fb0e5557700 1 -- 172.21.15.31:0/439794052 learned_addr learned my addr 172.21.15.31:0/439794052 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.516+0000 7fb0e5557700 1 -- 172.21.15.31:0/439794052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e0107cb0 con 0x7fb0e0106ab0 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.516+0000 7fb0e5557700 1 --2- 172.21.15.31:0/439794052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e0106ed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80098a0 tx=0x7fb0d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e806b47f0050e9d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.516+0000 7fb0dffff700 1 -- 172.21.15.31:0/439794052 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d8004160 con 0x7fb0e0106ab0 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.516+0000 7fb0dffff700 1 -- 172.21.15.31:0/439794052 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0d8004300 con 0x7fb0e0106ab0 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.516+0000 7fb0dffff700 1 -- 172.21.15.31:0/439794052 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d8011a50 con 0x7fb0e0106ab0 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.517+0000 7fb0e77bb700 1 -- 172.21.15.31:0/439794052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 msgr2=0x7fb0e0106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.517+0000 7fb0e77bb700 1 --2- 172.21.15.31:0/439794052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e0106ed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80098a0 tx=0x7fb0d8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.517+0000 7fb0e77bb700 1 -- 172.21.15.31:0/439794052 shutdown_connections 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.517+0000 7fb0e77bb700 1 --2- 172.21.15.31:0/439794052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e0106ed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.517+0000 7fb0e77bb700 1 -- 172.21.15.31:0/439794052 >> 172.21.15.31:0/439794052 conn(0x7fb0e0102030 msgr2=0x7fb0e0104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.517+0000 7fb0e77bb700 1 -- 172.21.15.31:0/439794052 shutdown_connections 2024-04-25T01:45:17.364 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.517+0000 7fb0e77bb700 1 -- 172.21.15.31:0/439794052 wait complete. 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0e77bb700 1 Processor -- start 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0e77bb700 1 -- start start 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0e77bb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e007ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0e77bb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e010b060 con 0x7fb0e0106ab0 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0e5557700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e007ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0e5557700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e007ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60720/0 (socket says 172.21.15.31:60720) 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0e5557700 1 -- 172.21.15.31:0/3351086604 learned_addr learned my addr 172.21.15.31:0/3351086604 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0e5557700 1 -- 172.21.15.31:0/3351086604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e019e820 con 0x7fb0e0106ab0 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0e5557700 1 --2- 172.21.15.31:0/3351086604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e007ba70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8011710 tx=0x7fb0d8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0de7fc700 1 -- 172.21.15.31:0/3351086604 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d80039c0 con 0x7fb0e0106ab0 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0de7fc700 1 -- 172.21.15.31:0/3351086604 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0d8003b60 con 0x7fb0e0106ab0 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0de7fc700 1 -- 172.21.15.31:0/3351086604 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d801a690 con 0x7fb0e0106ab0 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.518+0000 7fb0e77bb700 1 -- 172.21.15.31:0/3351086604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e019e9f0 con 0x7fb0e0106ab0 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.519+0000 7fb0e77bb700 1 -- 172.21.15.31:0/3351086604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e019ed90 con 0x7fb0e0106ab0 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.519+0000 7fb0de7fc700 1 -- 172.21.15.31:0/3351086604 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb0d8024430 con 0x7fb0e0106ab0 2024-04-25T01:45:17.365 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.519+0000 7fb0e77bb700 1 -- 172.21.15.31:0/3351086604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e0063890 con 0x7fb0e0106ab0 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.520+0000 7fb0de7fc700 1 --2- 172.21.15.31:0/3351086604 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fb0c8039050 0x7fb0c803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.520+0000 7fb0de7fc700 1 -- 172.21.15.31:0/3351086604 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb0d804d660 con 0x7fb0e0106ab0 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.520+0000 7fb0e4d56700 1 --2- 172.21.15.31:0/3351086604 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fb0c8039050 0x7fb0c803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.520+0000 7fb0e4d56700 1 --2- 172.21.15.31:0/3351086604 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fb0c8039050 0x7fb0c803b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4006fd0 tx=0x7fb0d4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.525+0000 7fb0de7fc700 1 -- 172.21.15.31:0/3351086604 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb0d800f2e0 con 0x7fb0e0106ab0 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:15.809+0000 7fb0e77bb700 1 -- 172.21.15.31:0/3351086604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fb0e019f1f0 con 0x7fb0e0106ab0 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.537+0000 7fb0de7fc700 1 -- 172.21.15.31:0/3351086604 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fb0d8024720 con 0x7fb0e0106ab0 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.539+0000 7fb0de7fc700 1 -- 172.21.15.31:0/3351086604 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fb0d802d770 con 0x7fb0e0106ab0 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.541+0000 7fb0e77bb700 1 -- 172.21.15.31:0/3351086604 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fb0c8039050 msgr2=0x7fb0c803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.541+0000 7fb0e77bb700 1 --2- 172.21.15.31:0/3351086604 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fb0c8039050 0x7fb0c803b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4006fd0 tx=0x7fb0d4006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.541+0000 7fb0e77bb700 1 -- 172.21.15.31:0/3351086604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 msgr2=0x7fb0e007ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.541+0000 7fb0e77bb700 1 --2- 172.21.15.31:0/3351086604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e007ba70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8011710 tx=0x7fb0d8011ad0 comp rx=0 tx=0).stop 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.541+0000 7fb0e77bb700 1 -- 172.21.15.31:0/3351086604 shutdown_connections 2024-04-25T01:45:17.366 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.541+0000 7fb0e77bb700 1 --2- 172.21.15.31:0/3351086604 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fb0c8039050 0x7fb0c803b510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:17.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.541+0000 7fb0e77bb700 1 --2- 172.21.15.31:0/3351086604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0e0106ab0 0x7fb0e007ba70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:17.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.541+0000 7fb0e77bb700 1 -- 172.21.15.31:0/3351086604 >> 172.21.15.31:0/3351086604 conn(0x7fb0e0102030 msgr2=0x7fb0e0102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:17.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.541+0000 7fb0e77bb700 1 -- 172.21.15.31:0/3351086604 shutdown_connections 2024-04-25T01:45:17.367 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:16.542+0000 7fb0e77bb700 1 -- 172.21.15.31:0/3351086604 wait complete. 2024-04-25T01:45:17.667 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:17 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:17.437+0000 7f83a5a14200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:45:17.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:17 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3351086604' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T01:45:17.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:17 smithi031 ceph-mon[29204]: mgrmap e9: x(active, since 20s) 2024-04-25T01:45:18.722 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:45:18.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T01:45:18.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:45:18.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T01:45:18.730 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T01:45:18.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:45:18.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.787+0000 7fcb1d932700 1 Processor -- start 2024-04-25T01:45:18.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.787+0000 7fcb1d932700 1 -- start start 2024-04-25T01:45:18.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.787+0000 7fcb1d932700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1810aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:18.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.787+0000 7fcb1d932700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1807be90 con 0x7fcb18108610 2024-04-25T01:45:18.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.788+0000 7fcb16ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1810aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:18.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.788+0000 7fcb16ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1810aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60752/0 (socket says 172.21.15.31:60752) 2024-04-25T01:45:18.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.788+0000 7fcb16ffd700 1 -- 172.21.15.31:0/186822944 learned_addr learned my addr 172.21.15.31:0/186822944 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:18.731 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.788+0000 7fcb16ffd700 1 -- 172.21.15.31:0/186822944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1810af40 con 0x7fcb18108610 2024-04-25T01:45:18.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.788+0000 7fcb16ffd700 1 --2- 172.21.15.31:0/186822944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1810aa00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0098a0 tx=0x7fcb0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4baa715e17493362 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:18.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.788+0000 7fcb15ffb700 1 -- 172.21.15.31:0/186822944 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb0c004160 con 0x7fcb18108610 2024-04-25T01:45:18.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.788+0000 7fcb15ffb700 1 -- 172.21.15.31:0/186822944 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb0c004300 con 0x7fcb18108610 2024-04-25T01:45:18.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.788+0000 7fcb15ffb700 1 -- 172.21.15.31:0/186822944 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb0c011a50 con 0x7fcb18108610 2024-04-25T01:45:18.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.789+0000 7fcb1d932700 1 -- 172.21.15.31:0/186822944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 msgr2=0x7fcb1810aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:18.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.789+0000 7fcb1d932700 1 --2- 172.21.15.31:0/186822944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1810aa00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0098a0 tx=0x7fcb0c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:18.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.789+0000 7fcb1d932700 1 -- 172.21.15.31:0/186822944 shutdown_connections 2024-04-25T01:45:18.732 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.789+0000 7fcb1d932700 1 --2- 172.21.15.31:0/186822944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1810aa00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.789+0000 7fcb1d932700 1 -- 172.21.15.31:0/186822944 >> 172.21.15.31:0/186822944 conn(0x7fcb18101fe0 msgr2=0x7fcb18104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.789+0000 7fcb1d932700 1 -- 172.21.15.31:0/186822944 shutdown_connections 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.789+0000 7fcb1d932700 1 -- 172.21.15.31:0/186822944 wait complete. 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.790+0000 7fcb1d932700 1 Processor -- start 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.790+0000 7fcb1d932700 1 -- start start 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.790+0000 7fcb1d932700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1819ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.790+0000 7fcb1d932700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1810bf40 con 0x7fcb18108610 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.790+0000 7fcb16ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1819ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.791+0000 7fcb16ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1819ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60754/0 (socket says 172.21.15.31:60754) 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.791+0000 7fcb16ffd700 1 -- 172.21.15.31:0/2755938804 learned_addr learned my addr 172.21.15.31:0/2755938804 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.791+0000 7fcb16ffd700 1 -- 172.21.15.31:0/2755938804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1819bf70 con 0x7fcb18108610 2024-04-25T01:45:18.733 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.791+0000 7fcb16ffd700 1 --2- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1819ba30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c011a20 tx=0x7fcb0c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.791+0000 7fcb1c930700 1 -- 172.21.15.31:0/2755938804 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb0c003b70 con 0x7fcb18108610 2024-04-25T01:45:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.791+0000 7fcb1c930700 1 -- 172.21.15.31:0/2755938804 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb0c01a8e0 con 0x7fcb18108610 2024-04-25T01:45:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.791+0000 7fcb1d932700 1 -- 172.21.15.31:0/2755938804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb1819c200 con 0x7fcb18108610 2024-04-25T01:45:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.791+0000 7fcb1c930700 1 -- 172.21.15.31:0/2755938804 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb0c024990 con 0x7fcb18108610 2024-04-25T01:45:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.791+0000 7fcb1d932700 1 -- 172.21.15.31:0/2755938804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb1819eef0 con 0x7fcb18108610 2024-04-25T01:45:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.792+0000 7fcb1c930700 1 -- 172.21.15.31:0/2755938804 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fcb0c022020 con 0x7fcb18108610 2024-04-25T01:45:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.792+0000 7fcb1d932700 1 -- 172.21.15.31:0/2755938804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb18063890 con 0x7fcb18108610 2024-04-25T01:45:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.792+0000 7fcb1c930700 1 --2- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fcb00039050 0x7fcb0003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.792+0000 7fcb1c930700 1 -- 172.21.15.31:0/2755938804 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcb0c04d560 con 0x7fcb18108610 2024-04-25T01:45:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.792+0000 7fcb167fc700 1 -- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fcb00039050 msgr2=0x7fcb0003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3170545029 2024-04-25T01:45:18.734 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.793+0000 7fcb167fc700 1 --2- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fcb00039050 0x7fcb0003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.797+0000 7fcb1c930700 1 -- 172.21.15.31:0/2755938804 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcb0c00be30 con 0x7fcb18108610 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.994+0000 7fcb167fc700 1 -- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fcb00039050 msgr2=0x7fcb0003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3170545029 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:17.994+0000 7fcb167fc700 1 --2- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fcb00039050 0x7fcb0003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.109+0000 7fcb1d932700 1 -- 172.21.15.31:0/2755938804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fcb1819f4a0 con 0x7fcb18108610 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.109+0000 7fcb1c930700 1 -- 172.21.15.31:0/2755938804 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fcb0c02c510 con 0x7fcb18108610 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.111+0000 7fcb1d932700 1 -- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fcb00039050 msgr2=0x7fcb0003b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.111+0000 7fcb1d932700 1 --2- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fcb00039050 0x7fcb0003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.111+0000 7fcb1d932700 1 -- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 msgr2=0x7fcb1819ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.111+0000 7fcb1d932700 1 --2- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1819ba30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c011a20 tx=0x7fcb0c011b00 comp rx=0 tx=0).stop 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.111+0000 7fcb1d932700 1 -- 172.21.15.31:0/2755938804 shutdown_connections 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.111+0000 7fcb1d932700 1 --2- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7fcb00039050 0x7fcb0003b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.111+0000 7fcb1d932700 1 --2- 172.21.15.31:0/2755938804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb18108610 0x7fcb1819ba30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:18.736 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.111+0000 7fcb1d932700 1 -- 172.21.15.31:0/2755938804 >> 172.21.15.31:0/2755938804 conn(0x7fcb18101fe0 msgr2=0x7fcb18102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:18.736 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.112+0000 7fcb1d932700 1 -- 172.21.15.31:0/2755938804 shutdown_connections 2024-04-25T01:45:18.736 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:18.112+0000 7fcb1d932700 1 -- 172.21.15.31:0/2755938804 wait complete. 2024-04-25T01:45:18.736 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2024-04-25T01:45:18.736 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 9... 2024-04-25T01:45:18.845 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:18 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2755938804' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T01:45:18.846 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:18 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:18.696+0000 7f83a5a14200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:45:19.158 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:18 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:18.881+0000 7f83a5a14200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:45:19.158 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:19 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:19.060+0000 7f83a5a14200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:45:19.452 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:19 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:19.364+0000 7f83a5a14200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:45:19.452 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:19 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:19.460+0000 7f83a5a14200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:45:19.729 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:19 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:19.669+0000 7f83a5a14200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:45:20.229 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:19 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:19.869+0000 7f83a5a14200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:45:20.924 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:20 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:20.492+0000 7f83a5a14200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:45:21.229 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:20 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:20.932+0000 7f83a5a14200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:45:21.230 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:21 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:21.032+0000 7f83a5a14200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:45:21.729 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:21 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:21.241+0000 7f83a5a14200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:45:22.121 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:21 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:21.783+0000 7f83a5a14200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:45:22.121 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:21 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:21.878+0000 7f83a5a14200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:45:22.477 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:22 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:22.129+0000 7f83a5a14200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:45:22.477 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:22 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:22.267+0000 7f83a5a14200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:45:22.478 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:22 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:22.366+0000 7f83a5a14200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:45:22.478 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:22 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:22.485+0000 7f83a5a14200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:45:22.730 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:22 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:22.580+0000 7f83a5a14200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:45:23.691 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:23.420+0000 7f83a5a14200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:45:23.691 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:45:23.644+0000 7f83a5a14200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:45:23.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: Active manager daemon x restarted 2024-04-25T01:45:23.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: Activating manager daemon x 2024-04-25T01:45:23.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T01:45:23.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: mgrmap e10: x(active, starting, since 0.00352551s) 2024-04-25T01:45:23.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:45:23.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:45:23.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:45:23.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:45:23.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:45:23.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: Manager daemon x is now available 2024-04-25T01:45:23.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:23.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:45:24.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:45:24.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:24 smithi031 ceph-mon[29204]: [25/Apr/2024:01:45:24] ENGINE Bus STARTING 2024-04-25T01:45:24.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:24 smithi031 ceph-mon[29204]: [25/Apr/2024:01:45:24] ENGINE Serving on http://172.21.15.31:8765 2024-04-25T01:45:24.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:24 smithi031 ceph-mon[29204]: [25/Apr/2024:01:45:24] ENGINE Serving on https://172.21.15.31:7150 2024-04-25T01:45:24.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:24 smithi031 ceph-mon[29204]: [25/Apr/2024:01:45:24] ENGINE Bus STARTED 2024-04-25T01:45:24.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:24 smithi031 ceph-mon[29204]: mgrmap e11: x(active, since 1.00594s) 2024-04-25T01:45:25.474 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:45:25.474 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.196+0000 7f7d8d39f700 1 Processor -- start 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.196+0000 7f7d8d39f700 1 -- start start 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.196+0000 7f7d8d39f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d881062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.196+0000 7f7d8d39f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d88106880 con 0x7f7d88105e90 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.196+0000 7f7d86ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d881062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.196+0000 7f7d86ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d881062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60758/0 (socket says 172.21.15.31:60758) 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.196+0000 7f7d86ffd700 1 -- 172.21.15.31:0/3649151190 learned_addr learned my addr 172.21.15.31:0/3649151190 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.197+0000 7f7d86ffd700 1 -- 172.21.15.31:0/3649151190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d88108e20 con 0x7f7d88105e90 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.197+0000 7f7d86ffd700 1 --2- 172.21.15.31:0/3649151190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d881062b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7d780098a0 tx=0x7f7d78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21977e543b702609 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.197+0000 7f7d85ffb700 1 -- 172.21.15.31:0/3649151190 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d78004160 con 0x7f7d88105e90 2024-04-25T01:45:25.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.197+0000 7f7d85ffb700 1 -- 172.21.15.31:0/3649151190 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d78004300 con 0x7f7d88105e90 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.197+0000 7f7d85ffb700 1 -- 172.21.15.31:0/3649151190 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d78011a50 con 0x7f7d88105e90 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.198+0000 7f7d8d39f700 1 -- 172.21.15.31:0/3649151190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 msgr2=0x7f7d881062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.198+0000 7f7d8d39f700 1 --2- 172.21.15.31:0/3649151190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d881062b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7d780098a0 tx=0x7f7d78009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.198+0000 7f7d8d39f700 1 -- 172.21.15.31:0/3649151190 shutdown_connections 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.198+0000 7f7d8d39f700 1 --2- 172.21.15.31:0/3649151190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d881062b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.198+0000 7f7d8d39f700 1 -- 172.21.15.31:0/3649151190 >> 172.21.15.31:0/3649151190 conn(0x7f7d881017a0 msgr2=0x7f7d88103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.198+0000 7f7d8d39f700 1 -- 172.21.15.31:0/3649151190 shutdown_connections 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.198+0000 7f7d8d39f700 1 -- 172.21.15.31:0/3649151190 wait complete. 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.198+0000 7f7d8d39f700 1 Processor -- start 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.198+0000 7f7d8d39f700 1 -- start start 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.199+0000 7f7d8d39f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d8819bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.199+0000 7f7d8d39f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8810c1d0 con 0x7f7d88105e90 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.199+0000 7f7d86ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d8819bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.199+0000 7f7d86ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d8819bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60764/0 (socket says 172.21.15.31:60764) 2024-04-25T01:45:25.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.199+0000 7f7d86ffd700 1 -- 172.21.15.31:0/1221174652 learned_addr learned my addr 172.21.15.31:0/1221174652 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.199+0000 7f7d86ffd700 1 -- 172.21.15.31:0/1221174652 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8819c000 con 0x7f7d88105e90 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.199+0000 7f7d86ffd700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d8819bac0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7d78006b20 tx=0x7f7d78011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.199+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d780039e0 con 0x7f7d88105e90 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.199+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d7801a8e0 con 0x7f7d88105e90 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.199+0000 7f7d8d39f700 1 -- 172.21.15.31:0/1221174652 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8819c290 con 0x7f7d88105e90 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.199+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d780247e0 con 0x7f7d88105e90 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.200+0000 7f7d8d39f700 1 -- 172.21.15.31:0/1221174652 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d8819ef80 con 0x7f7d88105e90 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.200+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f7d78022020 con 0x7f7d88105e90 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.201+0000 7f7d7ffff700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 0x7f7d7403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.201+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 --> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7d7403bc70 con 0x7f7d740390a0 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.201+0000 7f7d867fc700 1 -- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 msgr2=0x7f7d7403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3170545029 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.201+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7d7804d4a0 con 0x7f7d88105e90 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.201+0000 7f7d867fc700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 0x7f7d7403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:45:25.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.402+0000 7f7d867fc700 1 -- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 msgr2=0x7f7d7403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3170545029 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.402+0000 7f7d867fc700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 0x7f7d7403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.803+0000 7f7d867fc700 1 -- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 msgr2=0x7f7d7403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3170545029 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:19.803+0000 7f7d867fc700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 0x7f7d7403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:20.604+0000 7f7d867fc700 1 -- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 msgr2=0x7f7d7403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3170545029 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:20.604+0000 7f7d867fc700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 0x7f7d7403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:22.205+0000 7f7d867fc700 1 -- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 msgr2=0x7f7d7403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3170545029 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:22.205+0000 7f7d867fc700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 0x7f7d7403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:23.648+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f7d7802d720 con 0x7f7d88105e90 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:23.648+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 msgr2=0x7f7d7403b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:23.648+0000 7f7d7ffff700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 0x7f7d7403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.649+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f7d7804f820 con 0x7f7d88105e90 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.650+0000 7f7d7ffff700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d7403cbb0 0x7f7d7403efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.650+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7d78001d30 con 0x7f7d7403cbb0 2024-04-25T01:45:25.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.650+0000 7f7d867fc700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d7403cbb0 0x7f7d7403efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.651+0000 7f7d867fc700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d7403cbb0 0x7f7d7403efa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7d70003a10 tx=0x7f7d700092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.651+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f7d78001d30 con 0x7f7d7403cbb0 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.658+0000 7f7d8d39f700 1 -- 172.21.15.31:0/1221174652 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f7d88063890 con 0x7f7d7403cbb0 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.658+0000 7f7d7ffff700 1 -- 172.21.15.31:0/1221174652 <== mgr.14152 v2:172.21.15.31:6800/2306032998 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f7d88063890 con 0x7f7d7403cbb0 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.658+0000 7f7d8d39f700 1 -- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d7403cbb0 msgr2=0x7f7d7403efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.658+0000 7f7d8d39f700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d7403cbb0 0x7f7d7403efa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7d70003a10 tx=0x7f7d700092b0 comp rx=0 tx=0).stop 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.658+0000 7f7d8d39f700 1 -- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 msgr2=0x7f7d8819bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.658+0000 7f7d8d39f700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d8819bac0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7d78006b20 tx=0x7f7d78011ad0 comp rx=0 tx=0).stop 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.659+0000 7f7d8d39f700 1 -- 172.21.15.31:0/1221174652 shutdown_connections 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.659+0000 7f7d8d39f700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d7403cbb0 0x7f7d7403efa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.659+0000 7f7d8d39f700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88105e90 0x7f7d8819bac0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.659+0000 7f7d8d39f700 1 --2- 172.21.15.31:0/1221174652 >> [v2:172.21.15.31:6800/3170545029,v1:172.21.15.31:6801/3170545029] conn(0x7f7d740390a0 0x7f7d7403b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.659+0000 7f7d8d39f700 1 -- 172.21.15.31:0/1221174652 >> 172.21.15.31:0/1221174652 conn(0x7f7d881017a0 msgr2=0x7f7d88102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:25.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.659+0000 7f7d8d39f700 1 -- 172.21.15.31:0/1221174652 shutdown_connections 2024-04-25T01:45:25.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:24.659+0000 7f7d8d39f700 1 -- 172.21.15.31:0/1221174652 wait complete. 2024-04-25T01:45:25.480 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 9 is available 2024-04-25T01:45:25.480 INFO:teuthology.orchestra.run.smithi031.stdout:Generating a dashboard self-signed certificate... 2024-04-25T01:45:26.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:26 smithi031 ceph-mon[29204]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T01:45:26.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:26 smithi031 ceph-mon[29204]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T01:45:26.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:27.037 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.914+0000 7f15a9692700 1 Processor -- start 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.914+0000 7f15a9692700 1 -- start start 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.914+0000 7f15a9692700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a407c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.914+0000 7f15a9692700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f15a407c910 con 0x7f15a407de90 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.914+0000 7f15a2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a407c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.914+0000 7f15a2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a407c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49204/0 (socket says 172.21.15.31:49204) 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.914+0000 7f15a2ffd700 1 -- 172.21.15.31:0/2217472455 learned_addr learned my addr 172.21.15.31:0/2217472455 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.915+0000 7f15a2ffd700 1 -- 172.21.15.31:0/2217472455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a407ca90 con 0x7f15a407de90 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.915+0000 7f15a2ffd700 1 --2- 172.21.15.31:0/2217472455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a407c340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f15940098a0 tx=0x7f1594009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98b7c94e31603c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.915+0000 7f15a1ffb700 1 -- 172.21.15.31:0/2217472455 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1594004160 con 0x7f15a407de90 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.915+0000 7f15a1ffb700 1 -- 172.21.15.31:0/2217472455 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1594004300 con 0x7f15a407de90 2024-04-25T01:45:27.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.915+0000 7f15a1ffb700 1 -- 172.21.15.31:0/2217472455 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1594011a50 con 0x7f15a407de90 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.916+0000 7f15a9692700 1 -- 172.21.15.31:0/2217472455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 msgr2=0x7f15a407c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.916+0000 7f15a9692700 1 --2- 172.21.15.31:0/2217472455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a407c340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f15940098a0 tx=0x7f1594009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.916+0000 7f15a9692700 1 -- 172.21.15.31:0/2217472455 shutdown_connections 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.916+0000 7f15a9692700 1 --2- 172.21.15.31:0/2217472455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a407c340 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.916+0000 7f15a9692700 1 -- 172.21.15.31:0/2217472455 >> 172.21.15.31:0/2217472455 conn(0x7f15a4103110 msgr2=0x7f15a4105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.916+0000 7f15a9692700 1 -- 172.21.15.31:0/2217472455 shutdown_connections 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.916+0000 7f15a9692700 1 -- 172.21.15.31:0/2217472455 wait complete. 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.916+0000 7f15a9692700 1 Processor -- start 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.917+0000 7f15a9692700 1 -- start start 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.917+0000 7f15a9692700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a419dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.917+0000 7f15a9692700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f15a410e130 con 0x7f15a407de90 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.917+0000 7f15a2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a419dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.917+0000 7f15a2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a419dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49208/0 (socket says 172.21.15.31:49208) 2024-04-25T01:45:27.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.917+0000 7f15a2ffd700 1 -- 172.21.15.31:0/1185562064 learned_addr learned my addr 172.21.15.31:0/1185562064 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.917+0000 7f15a2ffd700 1 -- 172.21.15.31:0/1185562064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a419e1c0 con 0x7f15a407de90 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.917+0000 7f15a2ffd700 1 --2- 172.21.15.31:0/1185562064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a419dc80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1594011a20 tx=0x7f1594011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.917+0000 7f158bfff700 1 -- 172.21.15.31:0/1185562064 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15940039c0 con 0x7f15a407de90 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.918+0000 7f158bfff700 1 -- 172.21.15.31:0/1185562064 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1594003b60 con 0x7f15a407de90 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.918+0000 7f15a9692700 1 -- 172.21.15.31:0/1185562064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a419e450 con 0x7f15a407de90 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.918+0000 7f158bfff700 1 -- 172.21.15.31:0/1185562064 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f159401a6b0 con 0x7f15a407de90 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.918+0000 7f15a9692700 1 -- 172.21.15.31:0/1185562064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a41a1140 con 0x7f15a407de90 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.919+0000 7f158bfff700 1 -- 172.21.15.31:0/1185562064 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f159401a980 con 0x7f15a407de90 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.919+0000 7f15a9692700 1 -- 172.21.15.31:0/1185562064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15a4063890 con 0x7f15a407de90 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.919+0000 7f158bfff700 1 --2- 172.21.15.31:0/1185562064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f158c038f90 0x7f158c03b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.919+0000 7f158bfff700 1 -- 172.21.15.31:0/1185562064 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f159404d5f0 con 0x7f15a407de90 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.919+0000 7f15a27fc700 1 --2- 172.21.15.31:0/1185562064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f158c038f90 0x7f158c03b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.919+0000 7f15a27fc700 1 --2- 172.21.15.31:0/1185562064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f158c038f90 0x7f158c03b450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1598006fd0 tx=0x7f1598006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:27.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:25.924+0000 7f158bfff700 1 -- 172.21.15.31:0/1185562064 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f159401ad70 con 0x7f15a407de90 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.134+0000 7f15a9692700 1 -- 172.21.15.31:0/1185562064 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f15a4104360 con 0x7f158c038f90 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.166+0000 7f158bfff700 1 -- 172.21.15.31:0/1185562064 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f159402d910 con 0x7f15a407de90 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.490+0000 7f158bfff700 1 -- 172.21.15.31:0/1185562064 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f15a4104360 con 0x7f158c038f90 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.492+0000 7f15a9692700 1 -- 172.21.15.31:0/1185562064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f158c038f90 msgr2=0x7f158c03b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.492+0000 7f15a9692700 1 --2- 172.21.15.31:0/1185562064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f158c038f90 0x7f158c03b450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1598006fd0 tx=0x7f1598006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.492+0000 7f15a9692700 1 -- 172.21.15.31:0/1185562064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 msgr2=0x7f15a419dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.492+0000 7f15a9692700 1 --2- 172.21.15.31:0/1185562064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a419dc80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1594011a20 tx=0x7f1594011b00 comp rx=0 tx=0).stop 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.492+0000 7f15a9692700 1 -- 172.21.15.31:0/1185562064 shutdown_connections 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.492+0000 7f15a9692700 1 --2- 172.21.15.31:0/1185562064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f158c038f90 0x7f158c03b450 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.492+0000 7f15a9692700 1 --2- 172.21.15.31:0/1185562064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15a407de90 0x7f15a419dc80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.492+0000 7f15a9692700 1 -- 172.21.15.31:0/1185562064 >> 172.21.15.31:0/1185562064 conn(0x7f15a4103110 msgr2=0x7f15a4103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.493+0000 7f15a9692700 1 -- 172.21.15.31:0/1185562064 shutdown_connections 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:26.493+0000 7f15a9692700 1 -- 172.21.15.31:0/1185562064 wait complete. 2024-04-25T01:45:27.041 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial admin user... 2024-04-25T01:45:27.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:27 smithi031 ceph-mon[29204]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:27.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:27 smithi031 ceph-mon[29204]: mgrmap e12: x(active, since 2s) 2024-04-25T01:45:27.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:27 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:27.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:27 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:27.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:27 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:29.143 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:28 smithi031 ceph-mon[29204]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:29.143 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:28 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:29.152 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$bmfAMI99mKtU6g9sBVHH/Ouv9XHK1AZ478TbOmf0pIIdlCkgl/8NC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714009527, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.466+0000 7f1d81a43700 1 Processor -- start 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.466+0000 7f1d81a43700 1 -- start start 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.466+0000 7f1d81a43700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.466+0000 7f1d81a43700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c108530 con 0x7f1d7c107bd0 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.466+0000 7f1d7affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.466+0000 7f1d7affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c107ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49212/0 (socket says 172.21.15.31:49212) 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.466+0000 7f1d7affd700 1 -- 172.21.15.31:0/1923163468 learned_addr learned my addr 172.21.15.31:0/1923163468 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.466+0000 7f1d7affd700 1 -- 172.21.15.31:0/1923163468 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7c1086b0 con 0x7f1d7c107bd0 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.467+0000 7f1d7affd700 1 --2- 172.21.15.31:0/1923163468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c107ff0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0098a0 tx=0x7f1d6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b890c49f6e9c610 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.467+0000 7f1d79ffb700 1 -- 172.21.15.31:0/1923163468 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d6c004160 con 0x7f1d7c107bd0 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.467+0000 7f1d79ffb700 1 -- 172.21.15.31:0/1923163468 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d6c004300 con 0x7f1d7c107bd0 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.467+0000 7f1d79ffb700 1 -- 172.21.15.31:0/1923163468 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d6c011a50 con 0x7f1d7c107bd0 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.467+0000 7f1d81a43700 1 -- 172.21.15.31:0/1923163468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 msgr2=0x7f1d7c107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:29.153 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.467+0000 7f1d81a43700 1 --2- 172.21.15.31:0/1923163468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c107ff0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0098a0 tx=0x7f1d6c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.467+0000 7f1d81a43700 1 -- 172.21.15.31:0/1923163468 shutdown_connections 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.467+0000 7f1d81a43700 1 --2- 172.21.15.31:0/1923163468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c107ff0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.467+0000 7f1d81a43700 1 -- 172.21.15.31:0/1923163468 >> 172.21.15.31:0/1923163468 conn(0x7f1d7c1031f0 msgr2=0x7f1d7c105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.468+0000 7f1d81a43700 1 -- 172.21.15.31:0/1923163468 shutdown_connections 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.468+0000 7f1d81a43700 1 -- 172.21.15.31:0/1923163468 wait complete. 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.468+0000 7f1d81a43700 1 Processor -- start 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.468+0000 7f1d81a43700 1 -- start start 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.468+0000 7f1d81a43700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c19b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.468+0000 7f1d81a43700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c109c10 con 0x7f1d7c107bd0 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.469+0000 7f1d7affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c19b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.469+0000 7f1d7affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c19b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49220/0 (socket says 172.21.15.31:49220) 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.469+0000 7f1d7affd700 1 -- 172.21.15.31:0/2921966819 learned_addr learned my addr 172.21.15.31:0/2921966819 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.469+0000 7f1d7affd700 1 -- 172.21.15.31:0/2921966819 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7c19bf30 con 0x7f1d7c107bd0 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.469+0000 7f1d7affd700 1 --2- 172.21.15.31:0/2921966819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c19b9f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c006b20 tx=0x7f1d6c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:29.154 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.469+0000 7f1d80a41700 1 -- 172.21.15.31:0/2921966819 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d6c003b70 con 0x7f1d7c107bd0 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.469+0000 7f1d80a41700 1 -- 172.21.15.31:0/2921966819 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d6c01a8e0 con 0x7f1d7c107bd0 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.469+0000 7f1d81a43700 1 -- 172.21.15.31:0/2921966819 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7c19c1c0 con 0x7f1d7c107bd0 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.469+0000 7f1d80a41700 1 -- 172.21.15.31:0/2921966819 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d6c0249b0 con 0x7f1d7c107bd0 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.469+0000 7f1d81a43700 1 -- 172.21.15.31:0/2921966819 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d7c19eeb0 con 0x7f1d7c107bd0 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.470+0000 7f1d80a41700 1 -- 172.21.15.31:0/2921966819 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f1d6c022020 con 0x7f1d7c107bd0 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.470+0000 7f1d81a43700 1 -- 172.21.15.31:0/2921966819 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d7c063890 con 0x7f1d7c107bd0 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.470+0000 7f1d80a41700 1 --2- 172.21.15.31:0/2921966819 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1d64039100 0x7f1d6403b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.470+0000 7f1d80a41700 1 -- 172.21.15.31:0/2921966819 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1d6c04d910 con 0x7f1d7c107bd0 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.470+0000 7f1d7a7fc700 1 --2- 172.21.15.31:0/2921966819 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1d64039100 0x7f1d6403b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.471+0000 7f1d7a7fc700 1 --2- 172.21.15.31:0/2921966819 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1d64039100 0x7f1d6403b5c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d70006fd0 tx=0x7f1d70006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.475+0000 7f1d80a41700 1 -- 172.21.15.31:0/2921966819 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1d6c00b3e0 con 0x7f1d7c107bd0 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.711+0000 7f1d81a43700 1 -- 172.21.15.31:0/2921966819 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f1d7c19c6a0 con 0x7f1d64039100 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.958+0000 7f1d80a41700 1 -- 172.21.15.31:0/2921966819 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f1d7c19c6a0 con 0x7f1d64039100 2024-04-25T01:45:29.155 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.960+0000 7f1d81a43700 1 -- 172.21.15.31:0/2921966819 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1d64039100 msgr2=0x7f1d6403b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:29.156 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.960+0000 7f1d81a43700 1 --2- 172.21.15.31:0/2921966819 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1d64039100 0x7f1d6403b5c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d70006fd0 tx=0x7f1d70006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:29.156 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.960+0000 7f1d81a43700 1 -- 172.21.15.31:0/2921966819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 msgr2=0x7f1d7c19b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:29.156 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.960+0000 7f1d81a43700 1 --2- 172.21.15.31:0/2921966819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c19b9f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c006b20 tx=0x7f1d6c011ad0 comp rx=0 tx=0).stop 2024-04-25T01:45:29.156 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.961+0000 7f1d81a43700 1 -- 172.21.15.31:0/2921966819 shutdown_connections 2024-04-25T01:45:29.156 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.961+0000 7f1d81a43700 1 --2- 172.21.15.31:0/2921966819 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1d64039100 0x7f1d6403b5c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:29.156 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.961+0000 7f1d81a43700 1 --2- 172.21.15.31:0/2921966819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d7c107bd0 0x7f1d7c19b9f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:29.156 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.961+0000 7f1d81a43700 1 -- 172.21.15.31:0/2921966819 >> 172.21.15.31:0/2921966819 conn(0x7f1d7c1031f0 msgr2=0x7f1d7c103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:29.156 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.961+0000 7f1d81a43700 1 -- 172.21.15.31:0/2921966819 shutdown_connections 2024-04-25T01:45:29.156 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:27.961+0000 7f1d81a43700 1 -- 172.21.15.31:0/2921966819 wait complete. 2024-04-25T01:45:29.157 INFO:teuthology.orchestra.run.smithi031.stdout:Fetching dashboard port number... 2024-04-25T01:45:30.116 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:29 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/111067570' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T01:45:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T01:45:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.547+0000 7f6c142bd700 1 Processor -- start 2024-04-25T01:45:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.547+0000 7f6c142bd700 1 -- start start 2024-04-25T01:45:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.547+0000 7f6c142bd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c108450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.547+0000 7f6c142bd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c108a20 con 0x7f6c0c108030 2024-04-25T01:45:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.547+0000 7f6c12059700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c108450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.547+0000 7f6c12059700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c108450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49226/0 (socket says 172.21.15.31:49226) 2024-04-25T01:45:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.547+0000 7f6c12059700 1 -- 172.21.15.31:0/4223408579 learned_addr learned my addr 172.21.15.31:0/4223408579 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:30.392 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.548+0000 7f6c12059700 1 -- 172.21.15.31:0/4223408579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c106c40 con 0x7f6c0c108030 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.548+0000 7f6c12059700 1 --2- 172.21.15.31:0/4223408579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c108450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800bb40 tx=0x7f6c0800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef81d5caac1e4f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.548+0000 7f6c11057700 1 -- 172.21.15.31:0/4223408579 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c0800dbe0 con 0x7f6c0c108030 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.548+0000 7f6c11057700 1 -- 172.21.15.31:0/4223408579 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c080044e0 con 0x7f6c0c108030 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.548+0000 7f6c11057700 1 -- 172.21.15.31:0/4223408579 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c08010470 con 0x7f6c0c108030 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.548+0000 7f6c142bd700 1 -- 172.21.15.31:0/4223408579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 msgr2=0x7f6c0c108450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.548+0000 7f6c142bd700 1 --2- 172.21.15.31:0/4223408579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c108450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800bb40 tx=0x7f6c0800be50 comp rx=0 tx=0).stop 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.549+0000 7f6c142bd700 1 -- 172.21.15.31:0/4223408579 shutdown_connections 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.549+0000 7f6c142bd700 1 --2- 172.21.15.31:0/4223408579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c108450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.549+0000 7f6c142bd700 1 -- 172.21.15.31:0/4223408579 >> 172.21.15.31:0/4223408579 conn(0x7f6c0c101750 msgr2=0x7f6c0c103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.549+0000 7f6c142bd700 1 -- 172.21.15.31:0/4223408579 shutdown_connections 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.549+0000 7f6c142bd700 1 -- 172.21.15.31:0/4223408579 wait complete. 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.549+0000 7f6c142bd700 1 Processor -- start 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.549+0000 7f6c142bd700 1 -- start start 2024-04-25T01:45:30.393 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.549+0000 7f6c142bd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.549+0000 7f6c142bd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c105b30 con 0x7f6c0c108030 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.550+0000 7f6c12059700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.550+0000 7f6c12059700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49240/0 (socket says 172.21.15.31:49240) 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.550+0000 7f6c12059700 1 -- 172.21.15.31:0/111067570 learned_addr learned my addr 172.21.15.31:0/111067570 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.550+0000 7f6c12059700 1 -- 172.21.15.31:0/111067570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c199d90 con 0x7f6c0c108030 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.550+0000 7f6c12059700 1 --2- 172.21.15.31:0/111067570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c199850 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c08003bf0 tx=0x7f6c08003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.550+0000 7f6c037fe700 1 -- 172.21.15.31:0/111067570 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c08027070 con 0x7f6c0c108030 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.550+0000 7f6c037fe700 1 -- 172.21.15.31:0/111067570 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c08010bd0 con 0x7f6c0c108030 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.550+0000 7f6c142bd700 1 -- 172.21.15.31:0/111067570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c0c19a020 con 0x7f6c0c108030 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.550+0000 7f6c037fe700 1 -- 172.21.15.31:0/111067570 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c08022b30 con 0x7f6c0c108030 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.550+0000 7f6c142bd700 1 -- 172.21.15.31:0/111067570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c0c19a480 con 0x7f6c0c108030 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.551+0000 7f6c037fe700 1 -- 172.21.15.31:0/111067570 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f6c0802a070 con 0x7f6c0c108030 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.551+0000 7f6c142bd700 1 -- 172.21.15.31:0/111067570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c0c063890 con 0x7f6c0c108030 2024-04-25T01:45:30.394 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.551+0000 7f6c037fe700 1 --2- 172.21.15.31:0/111067570 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6bf80390f0 0x7f6bf803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.551+0000 7f6c037fe700 1 -- 172.21.15.31:0/111067570 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6c0801f070 con 0x7f6c0c108030 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.551+0000 7f6c11858700 1 --2- 172.21.15.31:0/111067570 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6bf80390f0 0x7f6bf803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.552+0000 7f6c11858700 1 --2- 172.21.15.31:0/111067570 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6bf80390f0 0x7f6bf803b5b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc006fd0 tx=0x7f6bfc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.556+0000 7f6c037fe700 1 -- 172.21.15.31:0/111067570 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6c08015070 con 0x7f6c0c108030 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.763+0000 7f6c142bd700 1 -- 172.21.15.31:0/111067570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f6c0c192db0 con 0x7f6c0c108030 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.764+0000 7f6c037fe700 1 -- 172.21.15.31:0/111067570 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f6c0c192db0 con 0x7f6c0c108030 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.766+0000 7f6c142bd700 1 -- 172.21.15.31:0/111067570 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6bf80390f0 msgr2=0x7f6bf803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.766+0000 7f6c142bd700 1 --2- 172.21.15.31:0/111067570 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6bf80390f0 0x7f6bf803b5b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc006fd0 tx=0x7f6bfc006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.766+0000 7f6c142bd700 1 -- 172.21.15.31:0/111067570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 msgr2=0x7f6c0c199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.766+0000 7f6c142bd700 1 --2- 172.21.15.31:0/111067570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c199850 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c08003bf0 tx=0x7f6c08003fb0 comp rx=0 tx=0).stop 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.766+0000 7f6c142bd700 1 -- 172.21.15.31:0/111067570 shutdown_connections 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.766+0000 7f6c142bd700 1 --2- 172.21.15.31:0/111067570 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6bf80390f0 0x7f6bf803b5b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.766+0000 7f6c142bd700 1 --2- 172.21.15.31:0/111067570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c0c108030 0x7f6c0c199850 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.766+0000 7f6c142bd700 1 -- 172.21.15.31:0/111067570 >> 172.21.15.31:0/111067570 conn(0x7f6c0c101750 msgr2=0x7f6c0c18fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:30.395 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.766+0000 7f6c142bd700 1 -- 172.21.15.31:0/111067570 shutdown_connections 2024-04-25T01:45:30.396 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:29.766+0000 7f6c142bd700 1 -- 172.21.15.31:0/111067570 wait complete. 2024-04-25T01:45:30.400 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:45:30.400 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-25T01:45:30.408 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:45:30.408 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T01:45:30.409 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-25T01:45:30.409 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T01:45:30.410 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph Dashboard is now available at: 2024-04-25T01:45:30.410 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:30.410 INFO:teuthology.orchestra.run.smithi031.stdout: URL: https://smithi031:8443/ 2024-04-25T01:45:30.410 INFO:teuthology.orchestra.run.smithi031.stdout: User: admin 2024-04-25T01:45:30.410 INFO:teuthology.orchestra.run.smithi031.stdout: Password: e57g7ck5oc 2024-04-25T01:45:30.410 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:30.411 INFO:teuthology.orchestra.run.smithi031.stdout:Saving cluster configuration to /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config directory 2024-04-25T01:45:30.411 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling autotune for osd_memory_target 2024-04-25T01:45:31.230 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:30 smithi031 ceph-mon[29204]: mgrmap e13: x(active, since 6s) 2024-04-25T01:45:31.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.782+0000 7f0e55cff700 1 Processor -- start 2024-04-25T01:45:31.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.782+0000 7f0e55cff700 1 -- start start 2024-04-25T01:45:31.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.783+0000 7f0e55cff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e50106e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:31.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.783+0000 7f0e55cff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0e50107450 con 0x7f0e50106a60 2024-04-25T01:45:31.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.783+0000 7f0e4f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e50106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:31.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.783+0000 7f0e4f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e50106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49252/0 (socket says 172.21.15.31:49252) 2024-04-25T01:45:31.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.783+0000 7f0e4f7fe700 1 -- 172.21.15.31:0/1063845001 learned_addr learned my addr 172.21.15.31:0/1063845001 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:31.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.783+0000 7f0e4f7fe700 1 -- 172.21.15.31:0/1063845001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e50107c60 con 0x7f0e50106a60 2024-04-25T01:45:31.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.783+0000 7f0e4f7fe700 1 --2- 172.21.15.31:0/1063845001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e50106e80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0e400098a0 tx=0x7f0e40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c819159aadebcb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.783+0000 7f0e4effd700 1 -- 172.21.15.31:0/1063845001 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e40004160 con 0x7f0e50106a60 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.783+0000 7f0e4effd700 1 -- 172.21.15.31:0/1063845001 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e40004300 con 0x7f0e50106a60 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.783+0000 7f0e4effd700 1 -- 172.21.15.31:0/1063845001 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e40011a50 con 0x7f0e50106a60 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.784+0000 7f0e55cff700 1 -- 172.21.15.31:0/1063845001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 msgr2=0x7f0e50106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.784+0000 7f0e55cff700 1 --2- 172.21.15.31:0/1063845001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e50106e80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0e400098a0 tx=0x7f0e40009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.784+0000 7f0e55cff700 1 -- 172.21.15.31:0/1063845001 shutdown_connections 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.784+0000 7f0e55cff700 1 --2- 172.21.15.31:0/1063845001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e50106e80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.784+0000 7f0e55cff700 1 -- 172.21.15.31:0/1063845001 >> 172.21.15.31:0/1063845001 conn(0x7f0e50101fe0 msgr2=0x7f0e50104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.784+0000 7f0e55cff700 1 -- 172.21.15.31:0/1063845001 shutdown_connections 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.784+0000 7f0e55cff700 1 -- 172.21.15.31:0/1063845001 wait complete. 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.785+0000 7f0e55cff700 1 Processor -- start 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.785+0000 7f0e55cff700 1 -- start start 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.785+0000 7f0e55cff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e5019ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:31.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.785+0000 7f0e55cff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5010b010 con 0x7f0e50106a60 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.785+0000 7f0e4f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e5019ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.785+0000 7f0e4f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e5019ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49260/0 (socket says 172.21.15.31:49260) 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.785+0000 7f0e4f7fe700 1 -- 172.21.15.31:0/3195768655 learned_addr learned my addr 172.21.15.31:0/3195768655 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.785+0000 7f0e4f7fe700 1 -- 172.21.15.31:0/3195768655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5019bfb0 con 0x7f0e50106a60 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.786+0000 7f0e4f7fe700 1 --2- 172.21.15.31:0/3195768655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e5019ba70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0e40006b20 tx=0x7f0e40011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.786+0000 7f0e4d7fa700 1 -- 172.21.15.31:0/3195768655 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e400039e0 con 0x7f0e50106a60 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.786+0000 7f0e55cff700 1 -- 172.21.15.31:0/3195768655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e5019c240 con 0x7f0e50106a60 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.786+0000 7f0e4d7fa700 1 -- 172.21.15.31:0/3195768655 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e4001a8e0 con 0x7f0e50106a60 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.786+0000 7f0e4d7fa700 1 -- 172.21.15.31:0/3195768655 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e400247e0 con 0x7f0e50106a60 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.786+0000 7f0e55cff700 1 -- 172.21.15.31:0/3195768655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e5019ef30 con 0x7f0e50106a60 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.787+0000 7f0e4d7fa700 1 -- 172.21.15.31:0/3195768655 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f0e40022020 con 0x7f0e50106a60 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.787+0000 7f0e55cff700 1 -- 172.21.15.31:0/3195768655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e50063890 con 0x7f0e50106a60 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.787+0000 7f0e4d7fa700 1 --2- 172.21.15.31:0/3195768655 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0e3c039140 0x7f0e3c03b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:31.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.787+0000 7f0e4d7fa700 1 -- 172.21.15.31:0/3195768655 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0e4004d620 con 0x7f0e50106a60 2024-04-25T01:45:31.593 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.787+0000 7f0e46ffd700 1 --2- 172.21.15.31:0/3195768655 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0e3c039140 0x7f0e3c03b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:31.593 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.787+0000 7f0e46ffd700 1 --2- 172.21.15.31:0/3195768655 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0e3c039140 0x7f0e3c03b600 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0e38006fd0 tx=0x7f0e38006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:31.593 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.792+0000 7f0e4d7fa700 1 -- 172.21.15.31:0/3195768655 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0e4000b3e0 con 0x7f0e50106a60 2024-04-25T01:45:31.593 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.996+0000 7f0e55cff700 1 -- 172.21.15.31:0/3195768655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f0e5002de30 con 0x7f0e50106a60 2024-04-25T01:45:31.593 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.996+0000 7f0e4d7fa700 1 -- 172.21.15.31:0/3195768655 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f0e400172b0 con 0x7f0e50106a60 2024-04-25T01:45:31.593 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.998+0000 7f0e55cff700 1 -- 172.21.15.31:0/3195768655 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0e3c039140 msgr2=0x7f0e3c03b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:31.593 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.998+0000 7f0e55cff700 1 --2- 172.21.15.31:0/3195768655 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0e3c039140 0x7f0e3c03b600 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0e38006fd0 tx=0x7f0e38006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:31.593 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.998+0000 7f0e55cff700 1 -- 172.21.15.31:0/3195768655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 msgr2=0x7f0e5019ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:31.593 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.998+0000 7f0e55cff700 1 --2- 172.21.15.31:0/3195768655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e5019ba70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0e40006b20 tx=0x7f0e40011ad0 comp rx=0 tx=0).stop 2024-04-25T01:45:31.594 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.999+0000 7f0e55cff700 1 -- 172.21.15.31:0/3195768655 shutdown_connections 2024-04-25T01:45:31.594 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.999+0000 7f0e55cff700 1 --2- 172.21.15.31:0/3195768655 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0e3c039140 0x7f0e3c03b600 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:31.594 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.999+0000 7f0e55cff700 1 --2- 172.21.15.31:0/3195768655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e50106a60 0x7f0e5019ba70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:31.594 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.999+0000 7f0e55cff700 1 -- 172.21.15.31:0/3195768655 >> 172.21.15.31:0/3195768655 conn(0x7f0e50101fe0 msgr2=0x7f0e50102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:31.594 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.999+0000 7f0e55cff700 1 -- 172.21.15.31:0/3195768655 shutdown_connections 2024-04-25T01:45:31.594 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:30.999+0000 7f0e55cff700 1 -- 172.21.15.31:0/3195768655 wait complete. 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.033+0000 7f66baf61700 1 Processor -- start 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.033+0000 7f66baf61700 1 -- start start 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.033+0000 7f66baf61700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b410aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.033+0000 7f66baf61700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66b407be90 con 0x7f66b4108670 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.033+0000 7f66b8cfd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b410aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.033+0000 7f66b8cfd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b410aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49276/0 (socket says 172.21.15.31:49276) 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.033+0000 7f66b8cfd700 1 -- 172.21.15.31:0/2891198659 learned_addr learned my addr 172.21.15.31:0/2891198659 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.033+0000 7f66b8cfd700 1 -- 172.21.15.31:0/2891198659 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b410afa0 con 0x7f66b4108670 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.034+0000 7f66b8cfd700 1 --2- 172.21.15.31:0/2891198659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b410aa60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66a80098a0 tx=0x7f66a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a58a7a6bba1430 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.034+0000 7f66b37fe700 1 -- 172.21.15.31:0/2891198659 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66a8004160 con 0x7f66b4108670 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.034+0000 7f66b37fe700 1 -- 172.21.15.31:0/2891198659 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a8004300 con 0x7f66b4108670 2024-04-25T01:45:32.931 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.034+0000 7f66b37fe700 1 -- 172.21.15.31:0/2891198659 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66a8011a50 con 0x7f66b4108670 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.034+0000 7f66baf61700 1 -- 172.21.15.31:0/2891198659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 msgr2=0x7f66b410aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.034+0000 7f66baf61700 1 --2- 172.21.15.31:0/2891198659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b410aa60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66a80098a0 tx=0x7f66a8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.034+0000 7f66baf61700 1 -- 172.21.15.31:0/2891198659 shutdown_connections 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.034+0000 7f66baf61700 1 --2- 172.21.15.31:0/2891198659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b410aa60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.034+0000 7f66baf61700 1 -- 172.21.15.31:0/2891198659 >> 172.21.15.31:0/2891198659 conn(0x7f66b4102040 msgr2=0x7f66b41044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.034+0000 7f66baf61700 1 -- 172.21.15.31:0/2891198659 shutdown_connections 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.034+0000 7f66baf61700 1 -- 172.21.15.31:0/2891198659 wait complete. 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.035+0000 7f66baf61700 1 Processor -- start 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.035+0000 7f66baf61700 1 -- start start 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.035+0000 7f66baf61700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b419ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.035+0000 7f66baf61700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66b410bfa0 con 0x7f66b4108670 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.035+0000 7f66b8cfd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b419ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.035+0000 7f66b8cfd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b419ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49284/0 (socket says 172.21.15.31:49284) 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.035+0000 7f66b8cfd700 1 -- 172.21.15.31:0/517804952 learned_addr learned my addr 172.21.15.31:0/517804952 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:32.932 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.035+0000 7f66b8cfd700 1 -- 172.21.15.31:0/517804952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b419bf70 con 0x7f66b4108670 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.036+0000 7f66b8cfd700 1 --2- 172.21.15.31:0/517804952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b419ba30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f66a8011a20 tx=0x7f66a8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.036+0000 7f66b1ffb700 1 -- 172.21.15.31:0/517804952 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66a8003b70 con 0x7f66b4108670 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.036+0000 7f66b1ffb700 1 -- 172.21.15.31:0/517804952 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a801a8e0 con 0x7f66b4108670 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.036+0000 7f66baf61700 1 -- 172.21.15.31:0/517804952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b419c200 con 0x7f66b4108670 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.036+0000 7f66b1ffb700 1 -- 172.21.15.31:0/517804952 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66a8024990 con 0x7f66b4108670 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.036+0000 7f66baf61700 1 -- 172.21.15.31:0/517804952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b419eef0 con 0x7f66b4108670 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.037+0000 7f66b1ffb700 1 -- 172.21.15.31:0/517804952 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f66a8022020 con 0x7f66b4108670 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.037+0000 7f66baf61700 1 -- 172.21.15.31:0/517804952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b4063890 con 0x7f66b4108670 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.037+0000 7f66b1ffb700 1 --2- 172.21.15.31:0/517804952 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f66a4039150 0x7f66a403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.037+0000 7f66b1ffb700 1 -- 172.21.15.31:0/517804952 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f66a804d7d0 con 0x7f66b4108670 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.037+0000 7f66b3fff700 1 --2- 172.21.15.31:0/517804952 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f66a4039150 0x7f66a403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.038+0000 7f66b3fff700 1 --2- 172.21.15.31:0/517804952 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f66a4039150 0x7f66a403b610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f66a0006fd0 tx=0x7f66a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.042+0000 7f66b1ffb700 1 -- 172.21.15.31:0/517804952 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f66a802c350 con 0x7f66b4108670 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.360+0000 7f66baf61700 1 -- 172.21.15.31:0/517804952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f66b419f4a0 con 0x7f66b4108670 2024-04-25T01:45:32.933 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.362+0000 7f66b1ffb700 1 -- 172.21.15.31:0/517804952 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f66a800fdd0 con 0x7f66b4108670 2024-04-25T01:45:32.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.364+0000 7f66baf61700 1 -- 172.21.15.31:0/517804952 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f66a4039150 msgr2=0x7f66a403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:32.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.364+0000 7f66baf61700 1 --2- 172.21.15.31:0/517804952 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f66a4039150 0x7f66a403b610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f66a0006fd0 tx=0x7f66a0006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:32.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.364+0000 7f66baf61700 1 -- 172.21.15.31:0/517804952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 msgr2=0x7f66b419ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:32.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.364+0000 7f66baf61700 1 --2- 172.21.15.31:0/517804952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b419ba30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f66a8011a20 tx=0x7f66a8011b00 comp rx=0 tx=0).stop 2024-04-25T01:45:32.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.365+0000 7f66baf61700 1 -- 172.21.15.31:0/517804952 shutdown_connections 2024-04-25T01:45:32.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.365+0000 7f66baf61700 1 --2- 172.21.15.31:0/517804952 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f66a4039150 0x7f66a403b610 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:32.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.365+0000 7f66baf61700 1 --2- 172.21.15.31:0/517804952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b4108670 0x7f66b419ba30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:32.934 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.365+0000 7f66baf61700 1 -- 172.21.15.31:0/517804952 >> 172.21.15.31:0/517804952 conn(0x7f66b4102040 msgr2=0x7f66b4102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.365+0000 7f66baf61700 1 -- 172.21.15.31:0/517804952 shutdown_connections 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T01:45:32.365+0000 7f66baf61700 1 -- 172.21.15.31:0/517804952 wait complete. 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout: ceph telemetry on 2024-04-25T01:45:32.935 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:32.936 INFO:teuthology.orchestra.run.smithi031.stdout:For more information see: 2024-04-25T01:45:32.936 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:32.936 INFO:teuthology.orchestra.run.smithi031.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T01:45:32.936 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:32.936 INFO:teuthology.orchestra.run.smithi031.stdout:Bootstrap complete. 2024-04-25T01:45:32.977 INFO:tasks.cephadm:Fetching config... 2024-04-25T01:45:32.977 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:45:32.977 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T01:45:32.994 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T01:45:32.994 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:45:32.994 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T01:45:33.051 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T01:45:33.052 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:45:33.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T01:45:33.118 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T01:45:33.119 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:45:33.119 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T01:45:33.175 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T01:45:33.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC75hcTaPfNof6hT475tM8Eur1tKkeGsBmM7b+kF8p0dxuMwUe50pyovyw03S1ySRdHWLtVdI4lAH+/04sjLDftySfLeo5oMiSvUO46knjK6tFnuFizYcEFl+09bInPxttROpvPylzycg0wjWk8ks1+bCrMGVWzsmQZHjGt+0eQ5JmdZc8NL/RuwVNCmCLN10NTIMvT5Z9LL5ylBylxeJuBFh9yQY7Zlam/7oOyjxmEd35/JGH0a/QXX69QVNeVy8YOfPSAutTQ8y/vG404ai4jjdPLD72YDi2UerHx9SLPzCRuBrrUdhq2KFIbIBiTLCaVzVdSETYRJtTZwAfqgS/XsnCj/vyD9NznJCD87gEYY2iHzDw0SEU+/RzHOP4cYlF/+VnRw8GchNbvsT65qpo5GhcSAj+roU33uIQMZdmzTTUFyBfmnHEC4qM1hJnuvr6pc9+ZjiAAF+k9sG/QlOKyvmp2HrfuMZIwESSNcu6b3MKVvZT+8ZYyqb3iATyuCLU= ceph-36afbea4-02a5-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:45:33.253 INFO:teuthology.orchestra.run.smithi031.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC75hcTaPfNof6hT475tM8Eur1tKkeGsBmM7b+kF8p0dxuMwUe50pyovyw03S1ySRdHWLtVdI4lAH+/04sjLDftySfLeo5oMiSvUO46knjK6tFnuFizYcEFl+09bInPxttROpvPylzycg0wjWk8ks1+bCrMGVWzsmQZHjGt+0eQ5JmdZc8NL/RuwVNCmCLN10NTIMvT5Z9LL5ylBylxeJuBFh9yQY7Zlam/7oOyjxmEd35/JGH0a/QXX69QVNeVy8YOfPSAutTQ8y/vG404ai4jjdPLD72YDi2UerHx9SLPzCRuBrrUdhq2KFIbIBiTLCaVzVdSETYRJtTZwAfqgS/XsnCj/vyD9NznJCD87gEYY2iHzDw0SEU+/RzHOP4cYlF/+VnRw8GchNbvsT65qpo5GhcSAj+roU33uIQMZdmzTTUFyBfmnHEC4qM1hJnuvr6pc9+ZjiAAF+k9sG/QlOKyvmp2HrfuMZIwESSNcu6b3MKVvZT+8ZYyqb3iATyuCLU= ceph-36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:33.266 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC75hcTaPfNof6hT475tM8Eur1tKkeGsBmM7b+kF8p0dxuMwUe50pyovyw03S1ySRdHWLtVdI4lAH+/04sjLDftySfLeo5oMiSvUO46knjK6tFnuFizYcEFl+09bInPxttROpvPylzycg0wjWk8ks1+bCrMGVWzsmQZHjGt+0eQ5JmdZc8NL/RuwVNCmCLN10NTIMvT5Z9LL5ylBylxeJuBFh9yQY7Zlam/7oOyjxmEd35/JGH0a/QXX69QVNeVy8YOfPSAutTQ8y/vG404ai4jjdPLD72YDi2UerHx9SLPzCRuBrrUdhq2KFIbIBiTLCaVzVdSETYRJtTZwAfqgS/XsnCj/vyD9NznJCD87gEYY2iHzDw0SEU+/RzHOP4cYlF/+VnRw8GchNbvsT65qpo5GhcSAj+roU33uIQMZdmzTTUFyBfmnHEC4qM1hJnuvr6pc9+ZjiAAF+k9sG/QlOKyvmp2HrfuMZIwESSNcu6b3MKVvZT+8ZYyqb3iATyuCLU= ceph-36afbea4-02a5-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:45:33.307 INFO:teuthology.orchestra.run.smithi040.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC75hcTaPfNof6hT475tM8Eur1tKkeGsBmM7b+kF8p0dxuMwUe50pyovyw03S1ySRdHWLtVdI4lAH+/04sjLDftySfLeo5oMiSvUO46knjK6tFnuFizYcEFl+09bInPxttROpvPylzycg0wjWk8ks1+bCrMGVWzsmQZHjGt+0eQ5JmdZc8NL/RuwVNCmCLN10NTIMvT5Z9LL5ylBylxeJuBFh9yQY7Zlam/7oOyjxmEd35/JGH0a/QXX69QVNeVy8YOfPSAutTQ8y/vG404ai4jjdPLD72YDi2UerHx9SLPzCRuBrrUdhq2KFIbIBiTLCaVzVdSETYRJtTZwAfqgS/XsnCj/vyD9NznJCD87gEYY2iHzDw0SEU+/RzHOP4cYlF/+VnRw8GchNbvsT65qpo5GhcSAj+roU33uIQMZdmzTTUFyBfmnHEC4qM1hJnuvr6pc9+ZjiAAF+k9sG/QlOKyvmp2HrfuMZIwESSNcu6b3MKVvZT+8ZYyqb3iATyuCLU= ceph-36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:33.319 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC75hcTaPfNof6hT475tM8Eur1tKkeGsBmM7b+kF8p0dxuMwUe50pyovyw03S1ySRdHWLtVdI4lAH+/04sjLDftySfLeo5oMiSvUO46knjK6tFnuFizYcEFl+09bInPxttROpvPylzycg0wjWk8ks1+bCrMGVWzsmQZHjGt+0eQ5JmdZc8NL/RuwVNCmCLN10NTIMvT5Z9LL5ylBylxeJuBFh9yQY7Zlam/7oOyjxmEd35/JGH0a/QXX69QVNeVy8YOfPSAutTQ8y/vG404ai4jjdPLD72YDi2UerHx9SLPzCRuBrrUdhq2KFIbIBiTLCaVzVdSETYRJtTZwAfqgS/XsnCj/vyD9NznJCD87gEYY2iHzDw0SEU+/RzHOP4cYlF/+VnRw8GchNbvsT65qpo5GhcSAj+roU33uIQMZdmzTTUFyBfmnHEC4qM1hJnuvr6pc9+ZjiAAF+k9sG/QlOKyvmp2HrfuMZIwESSNcu6b3MKVvZT+8ZYyqb3iATyuCLU= ceph-36afbea4-02a5-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:45:33.358 INFO:teuthology.orchestra.run.smithi134.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC75hcTaPfNof6hT475tM8Eur1tKkeGsBmM7b+kF8p0dxuMwUe50pyovyw03S1ySRdHWLtVdI4lAH+/04sjLDftySfLeo5oMiSvUO46knjK6tFnuFizYcEFl+09bInPxttROpvPylzycg0wjWk8ks1+bCrMGVWzsmQZHjGt+0eQ5JmdZc8NL/RuwVNCmCLN10NTIMvT5Z9LL5ylBylxeJuBFh9yQY7Zlam/7oOyjxmEd35/JGH0a/QXX69QVNeVy8YOfPSAutTQ8y/vG404ai4jjdPLD72YDi2UerHx9SLPzCRuBrrUdhq2KFIbIBiTLCaVzVdSETYRJtTZwAfqgS/XsnCj/vyD9NznJCD87gEYY2iHzDw0SEU+/RzHOP4cYlF/+VnRw8GchNbvsT65qpo5GhcSAj+roU33uIQMZdmzTTUFyBfmnHEC4qM1hJnuvr6pc9+ZjiAAF+k9sG/QlOKyvmp2HrfuMZIwESSNcu6b3MKVvZT+8ZYyqb3iATyuCLU= ceph-36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:33.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T01:45:33.615 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:33.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:33 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/517804952' entity='client.admin' 2024-04-25T01:45:36.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.123+0000 7f2bae9e3700 1 -- 172.21.15.31:0/506071475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba8100bb0 msgr2=0x7f2ba8100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:36.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.123+0000 7f2bae9e3700 1 --2- 172.21.15.31:0/506071475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba8100bb0 0x7f2ba8100fd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009910 tx=0x7f2b98009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:36.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.123+0000 7f2bae9e3700 1 -- 172.21.15.31:0/506071475 shutdown_connections 2024-04-25T01:45:36.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.123+0000 7f2bae9e3700 1 --2- 172.21.15.31:0/506071475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba8100bb0 0x7f2ba8100fd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.123+0000 7f2bae9e3700 1 -- 172.21.15.31:0/506071475 >> 172.21.15.31:0/506071475 conn(0x7f2ba80fc170 msgr2=0x7f2ba80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.123+0000 7f2bae9e3700 1 -- 172.21.15.31:0/506071475 shutdown_connections 2024-04-25T01:45:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.123+0000 7f2bae9e3700 1 -- 172.21.15.31:0/506071475 wait complete. 2024-04-25T01:45:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.123+0000 7f2bae9e3700 1 Processor -- start 2024-04-25T01:45:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.123+0000 7f2bae9e3700 1 -- start start 2024-04-25T01:45:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.124+0000 7f2bae9e3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba8100bb0 0x7f2ba8196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.124+0000 7f2bae9e3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba8106af0 con 0x7f2ba8100bb0 2024-04-25T01:45:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.124+0000 7f2ba7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba8100bb0 0x7f2ba8196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.124+0000 7f2ba7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba8100bb0 0x7f2ba8196250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44326/0 (socket says 172.21.15.31:44326) 2024-04-25T01:45:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.124+0000 7f2ba7fff700 1 -- 172.21.15.31:0/451208521 learned_addr learned my addr 172.21.15.31:0/451208521 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.124+0000 7f2ba7fff700 1 -- 172.21.15.31:0/451208521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba8196790 con 0x7f2ba8100bb0 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.124+0000 7f2ba7fff700 1 --2- 172.21.15.31:0/451208521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba8100bb0 0x7f2ba8196250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b98004fb0 tx=0x7f2b98009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.124+0000 7f2ba57fa700 1 -- 172.21.15.31:0/451208521 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b98027070 con 0x7f2ba8100bb0 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.124+0000 7f2ba57fa700 1 -- 172.21.15.31:0/451208521 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b9800be10 con 0x7f2ba8100bb0 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.124+0000 7f2bae9e3700 1 -- 172.21.15.31:0/451208521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba8196a20 con 0x7f2ba8100bb0 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.125+0000 7f2ba57fa700 1 -- 172.21.15.31:0/451208521 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b98022de0 con 0x7f2ba8100bb0 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.125+0000 7f2bae9e3700 1 -- 172.21.15.31:0/451208521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba806a0a0 con 0x7f2ba8100bb0 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.125+0000 7f2ba57fa700 1 -- 172.21.15.31:0/451208521 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f2b9802a070 con 0x7f2ba8100bb0 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.126+0000 7f2ba57fa700 1 --2- 172.21.15.31:0/451208521 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2b9003b300 0x7f2b9003d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.126+0000 7f2ba57fa700 1 -- 172.21.15.31:0/451208521 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2b98054060 con 0x7f2ba8100bb0 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.126+0000 7f2ba77fe700 1 --2- 172.21.15.31:0/451208521 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2b9003b300 0x7f2b9003d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.126+0000 7f2bae9e3700 1 -- 172.21.15.31:0/451208521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba8107270 con 0x7f2ba8100bb0 2024-04-25T01:45:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.126+0000 7f2ba77fe700 1 --2- 172.21.15.31:0/451208521 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2b9003b300 0x7f2b9003d7c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c006fd0 tx=0x7f2b9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:36.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.131+0000 7f2ba57fa700 1 -- 172.21.15.31:0/451208521 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2b98013070 con 0x7f2ba8100bb0 2024-04-25T01:45:36.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.339+0000 7f2bae9e3700 1 -- 172.21.15.31:0/451208521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f2ba806ab60 con 0x7f2ba8100bb0 2024-04-25T01:45:36.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.341+0000 7f2ba57fa700 1 -- 172.21.15.31:0/451208521 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f2b980213d0 con 0x7f2ba8100bb0 2024-04-25T01:45:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.344+0000 7f2bae9e3700 1 -- 172.21.15.31:0/451208521 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2b9003b300 msgr2=0x7f2b9003d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.344+0000 7f2bae9e3700 1 --2- 172.21.15.31:0/451208521 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2b9003b300 0x7f2b9003d7c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c006fd0 tx=0x7f2b9c006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.344+0000 7f2bae9e3700 1 -- 172.21.15.31:0/451208521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba8100bb0 msgr2=0x7f2ba8196250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.344+0000 7f2bae9e3700 1 --2- 172.21.15.31:0/451208521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba8100bb0 0x7f2ba8196250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b98004fb0 tx=0x7f2b98009f60 comp rx=0 tx=0).stop 2024-04-25T01:45:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.344+0000 7f2bae9e3700 1 -- 172.21.15.31:0/451208521 shutdown_connections 2024-04-25T01:45:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.344+0000 7f2bae9e3700 1 --2- 172.21.15.31:0/451208521 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2b9003b300 0x7f2b9003d7c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.344+0000 7f2bae9e3700 1 --2- 172.21.15.31:0/451208521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ba8100bb0 0x7f2ba8196250 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.344+0000 7f2bae9e3700 1 -- 172.21.15.31:0/451208521 >> 172.21.15.31:0/451208521 conn(0x7f2ba80fc170 msgr2=0x7f2ba80fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.344+0000 7f2bae9e3700 1 -- 172.21.15.31:0/451208521 shutdown_connections 2024-04-25T01:45:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:36.344+0000 7f2bae9e3700 1 -- 172.21.15.31:0/451208521 wait complete. 2024-04-25T01:45:37.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:37 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/451208521' entity='client.admin' 2024-04-25T01:45:37.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:37 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:37.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:37 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:37.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:37 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:45:37.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:37 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:37.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:37 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:37.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:37 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:37.620 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T01:45:37.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T01:45:37.863 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:39.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.379+0000 7f4d594fb700 1 -- 172.21.15.31:0/1177173365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d54102730 msgr2=0x7f4d54104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:39.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.379+0000 7f4d594fb700 1 --2- 172.21.15.31:0/1177173365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d54102730 0x7f4d54104b20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009910 tx=0x7f4d44009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:39.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.379+0000 7f4d594fb700 1 -- 172.21.15.31:0/1177173365 shutdown_connections 2024-04-25T01:45:39.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.379+0000 7f4d594fb700 1 --2- 172.21.15.31:0/1177173365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d54102730 0x7f4d54104b20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:39.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.379+0000 7f4d594fb700 1 -- 172.21.15.31:0/1177173365 >> 172.21.15.31:0/1177173365 conn(0x7f4d540fc180 msgr2=0x7f4d540fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:39.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.379+0000 7f4d594fb700 1 -- 172.21.15.31:0/1177173365 shutdown_connections 2024-04-25T01:45:39.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.379+0000 7f4d594fb700 1 -- 172.21.15.31:0/1177173365 wait complete. 2024-04-25T01:45:39.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.379+0000 7f4d594fb700 1 Processor -- start 2024-04-25T01:45:39.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.380+0000 7f4d594fb700 1 -- start start 2024-04-25T01:45:39.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.380+0000 7f4d594fb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d54102730 0x7f4d54101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:39.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.380+0000 7f4d594fb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4d54107b80 con 0x7f4d54102730 2024-04-25T01:45:39.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.380+0000 7f4d52ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d54102730 0x7f4d54101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:39.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.380+0000 7f4d52ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d54102730 0x7f4d54101fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44332/0 (socket says 172.21.15.31:44332) 2024-04-25T01:45:39.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.380+0000 7f4d52ffd700 1 -- 172.21.15.31:0/3792021313 learned_addr learned my addr 172.21.15.31:0/3792021313 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:39.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.380+0000 7f4d52ffd700 1 -- 172.21.15.31:0/3792021313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d541024f0 con 0x7f4d54102730 2024-04-25T01:45:39.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.380+0000 7f4d52ffd700 1 --2- 172.21.15.31:0/3792021313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d54102730 0x7f4d54101fb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d44005190 tx=0x7f4d44004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:39.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.381+0000 7f4d3bfff700 1 -- 172.21.15.31:0/3792021313 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d44027070 con 0x7f4d54102730 2024-04-25T01:45:39.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.381+0000 7f4d594fb700 1 -- 172.21.15.31:0/3792021313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d541006e0 con 0x7f4d54102730 2024-04-25T01:45:39.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.381+0000 7f4d3bfff700 1 -- 172.21.15.31:0/3792021313 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d44022470 con 0x7f4d54102730 2024-04-25T01:45:39.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.381+0000 7f4d3bfff700 1 -- 172.21.15.31:0/3792021313 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d4400f5f0 con 0x7f4d54102730 2024-04-25T01:45:39.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.381+0000 7f4d594fb700 1 -- 172.21.15.31:0/3792021313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d54100c20 con 0x7f4d54102730 2024-04-25T01:45:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.381+0000 7f4d3bfff700 1 -- 172.21.15.31:0/3792021313 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f4d4402a070 con 0x7f4d54102730 2024-04-25T01:45:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.382+0000 7f4d594fb700 1 -- 172.21.15.31:0/3792021313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d54108300 con 0x7f4d54102730 2024-04-25T01:45:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.382+0000 7f4d3bfff700 1 --2- 172.21.15.31:0/3792021313 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d3c0390f0 0x7f4d3c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.382+0000 7f4d3bfff700 1 -- 172.21.15.31:0/3792021313 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4d4401e030 con 0x7f4d54102730 2024-04-25T01:45:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.382+0000 7f4d527fc700 1 --2- 172.21.15.31:0/3792021313 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d3c0390f0 0x7f4d3c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:39.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.382+0000 7f4d527fc700 1 --2- 172.21.15.31:0/3792021313 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d3c0390f0 0x7f4d3c03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4d48006fd0 tx=0x7f4d48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.387+0000 7f4d3bfff700 1 -- 172.21.15.31:0/3792021313 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4d44013070 con 0x7f4d54102730 2024-04-25T01:45:39.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.598+0000 7f4d594fb700 1 -- 172.21.15.31:0/3792021313 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f4d54190720 con 0x7f4d3c0390f0 2024-04-25T01:45:39.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.601+0000 7f4d3bfff700 1 -- 172.21.15.31:0/3792021313 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4d54190720 con 0x7f4d3c0390f0 2024-04-25T01:45:39.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.603+0000 7f4d594fb700 1 -- 172.21.15.31:0/3792021313 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d3c0390f0 msgr2=0x7f4d3c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:39.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.603+0000 7f4d594fb700 1 --2- 172.21.15.31:0/3792021313 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d3c0390f0 0x7f4d3c03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4d48006fd0 tx=0x7f4d48006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:39.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.603+0000 7f4d594fb700 1 -- 172.21.15.31:0/3792021313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d54102730 msgr2=0x7f4d54101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:39.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.603+0000 7f4d594fb700 1 --2- 172.21.15.31:0/3792021313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d54102730 0x7f4d54101fb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d44005190 tx=0x7f4d44004f80 comp rx=0 tx=0).stop 2024-04-25T01:45:39.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.603+0000 7f4d594fb700 1 -- 172.21.15.31:0/3792021313 shutdown_connections 2024-04-25T01:45:39.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.603+0000 7f4d594fb700 1 --2- 172.21.15.31:0/3792021313 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d3c0390f0 0x7f4d3c03b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:39.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.603+0000 7f4d594fb700 1 --2- 172.21.15.31:0/3792021313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d54102730 0x7f4d54101fb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:39.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.603+0000 7f4d594fb700 1 -- 172.21.15.31:0/3792021313 >> 172.21.15.31:0/3792021313 conn(0x7f4d540fc180 msgr2=0x7f4d540fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:39.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.604+0000 7f4d594fb700 1 -- 172.21.15.31:0/3792021313 shutdown_connections 2024-04-25T01:45:39.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:39.604+0000 7f4d594fb700 1 -- 172.21.15.31:0/3792021313 wait complete. 2024-04-25T01:45:40.320 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi040 2024-04-25T01:45:40.321 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-25T01:45:40.321 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/etc/ceph/ceph.conf 2024-04-25T01:45:40.340 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-25T01:45:40.340 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:40.398 INFO:tasks.cephadm:Adding host smithi040 to orchestrator... 2024-04-25T01:45:40.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch host add smithi040 2024-04-25T01:45:40.649 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:40.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:40 smithi031 ceph-mon[29204]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:40.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:40 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:40.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:40 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:40.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:40 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:40.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:40 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:40.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:40 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:40.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:40 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:40.879 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:40 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:41.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:41 smithi031 ceph-mon[29204]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-25T01:45:41.730 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:41 smithi031 ceph-mon[29204]: Updating smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:41.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:41 smithi031 ceph-mon[29204]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:41.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:41 smithi031 ceph-mon[29204]: Updating smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:45:42.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.142+0000 7fcea7afe700 1 -- 172.21.15.31:0/4054428608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0101ea0 msgr2=0x7fcea0102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:42.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.142+0000 7fcea7afe700 1 --2- 172.21.15.31:0/4054428608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0101ea0 0x7fcea0102280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009910 tx=0x7fce9c009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:42.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.142+0000 7fcea7afe700 1 -- 172.21.15.31:0/4054428608 shutdown_connections 2024-04-25T01:45:42.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.142+0000 7fcea7afe700 1 --2- 172.21.15.31:0/4054428608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0101ea0 0x7fcea0102280 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:42.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.142+0000 7fcea7afe700 1 -- 172.21.15.31:0/4054428608 >> 172.21.15.31:0/4054428608 conn(0x7fcea00fd770 msgr2=0x7fcea00ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:42.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.143+0000 7fcea7afe700 1 -- 172.21.15.31:0/4054428608 shutdown_connections 2024-04-25T01:45:42.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.143+0000 7fcea7afe700 1 -- 172.21.15.31:0/4054428608 wait complete. 2024-04-25T01:45:42.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.143+0000 7fcea7afe700 1 Processor -- start 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.143+0000 7fcea7afe700 1 -- start start 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.143+0000 7fcea7afe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0101ea0 0x7fcea0197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.143+0000 7fcea7afe700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcea010c120 con 0x7fcea0101ea0 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.143+0000 7fcea589a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0101ea0 0x7fcea0197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.143+0000 7fcea589a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0101ea0 0x7fcea0197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44338/0 (socket says 172.21.15.31:44338) 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.144+0000 7fcea589a700 1 -- 172.21.15.31:0/2514112939 learned_addr learned my addr 172.21.15.31:0/2514112939 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.144+0000 7fcea589a700 1 -- 172.21.15.31:0/2514112939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea0197bc0 con 0x7fcea0101ea0 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.144+0000 7fcea589a700 1 --2- 172.21.15.31:0/2514112939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0101ea0 0x7fcea0197680 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0056d0 tx=0x7fce9c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.144+0000 7fce96ffd700 1 -- 172.21.15.31:0/2514112939 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce9c026070 con 0x7fcea0101ea0 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.144+0000 7fce96ffd700 1 -- 172.21.15.31:0/2514112939 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce9c00bba0 con 0x7fcea0101ea0 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.144+0000 7fcea7afe700 1 -- 172.21.15.31:0/2514112939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea0197eb0 con 0x7fcea0101ea0 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.144+0000 7fce96ffd700 1 -- 172.21.15.31:0/2514112939 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce9c0037f0 con 0x7fcea0101ea0 2024-04-25T01:45:42.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.144+0000 7fcea7afe700 1 -- 172.21.15.31:0/2514112939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea019bf50 con 0x7fcea0101ea0 2024-04-25T01:45:42.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.145+0000 7fce96ffd700 1 -- 172.21.15.31:0/2514112939 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fce9c029070 con 0x7fcea0101ea0 2024-04-25T01:45:42.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.145+0000 7fcea7afe700 1 -- 172.21.15.31:0/2514112939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea0103380 con 0x7fcea0101ea0 2024-04-25T01:45:42.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.145+0000 7fce96ffd700 1 --2- 172.21.15.31:0/2514112939 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fce8c039150 0x7fce8c03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:42.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.145+0000 7fce96ffd700 1 -- 172.21.15.31:0/2514112939 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fce9c01e070 con 0x7fcea0101ea0 2024-04-25T01:45:42.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.145+0000 7fcea5099700 1 --2- 172.21.15.31:0/2514112939 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fce8c039150 0x7fce8c03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:42.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.146+0000 7fcea5099700 1 --2- 172.21.15.31:0/2514112939 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fce8c039150 0x7fce8c03b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fce90006fd0 tx=0x7fce90006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:42.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.150+0000 7fce96ffd700 1 -- 172.21.15.31:0/2514112939 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fce9c013070 con 0x7fcea0101ea0 2024-04-25T01:45:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:42.361+0000 7fcea7afe700 1 -- 172.21.15.31:0/2514112939 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi040", "target": ["mon-mgr", ""]}) -- 0x7fcea01989e0 con 0x7fce8c039150 2024-04-25T01:45:42.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:42 smithi031 ceph-mon[29204]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi040", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:43.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:43 smithi031 ceph-mon[29204]: Deploying cephadm binary to smithi040 2024-04-25T01:45:44.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:44 smithi031 ceph-mon[29204]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:46.622+0000 7fce96ffd700 1 -- 172.21.15.31:0/2514112939 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fcea01989e0 con 0x7fce8c039150 2024-04-25T01:45:46.616 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi040' with addr '172.21.15.40' 2024-04-25T01:45:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:46.625+0000 7fcea7afe700 1 -- 172.21.15.31:0/2514112939 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fce8c039150 msgr2=0x7fce8c03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:46.625+0000 7fcea7afe700 1 --2- 172.21.15.31:0/2514112939 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fce8c039150 0x7fce8c03b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fce90006fd0 tx=0x7fce90006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:46.625+0000 7fcea7afe700 1 -- 172.21.15.31:0/2514112939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0101ea0 msgr2=0x7fcea0197680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:46.625+0000 7fcea7afe700 1 --2- 172.21.15.31:0/2514112939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0101ea0 0x7fcea0197680 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0056d0 tx=0x7fce9c0057b0 comp rx=0 tx=0).stop 2024-04-25T01:45:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:46.625+0000 7fcea7afe700 1 -- 172.21.15.31:0/2514112939 shutdown_connections 2024-04-25T01:45:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:46.625+0000 7fcea7afe700 1 --2- 172.21.15.31:0/2514112939 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fce8c039150 0x7fce8c03b610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:46.625+0000 7fcea7afe700 1 --2- 172.21.15.31:0/2514112939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0101ea0 0x7fcea0197680 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:46.625+0000 7fcea7afe700 1 -- 172.21.15.31:0/2514112939 >> 172.21.15.31:0/2514112939 conn(0x7fcea00fd770 msgr2=0x7fcea0103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:46.625+0000 7fcea7afe700 1 -- 172.21.15.31:0/2514112939 shutdown_connections 2024-04-25T01:45:46.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:46.625+0000 7fcea7afe700 1 -- 172.21.15.31:0/2514112939 wait complete. 2024-04-25T01:45:46.928 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:46 smithi031 ceph-mon[29204]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.928 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:46 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:46.928 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:46 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:47.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T01:45:47.445 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:48.460 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:48 smithi031 ceph-mon[29204]: Added host smithi040 2024-04-25T01:45:48.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:48 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:48.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:48 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:48.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.991+0000 7fef47498700 1 -- 172.21.15.31:0/2240588772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef40100de0 msgr2=0x7fef401011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:48.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.991+0000 7fef47498700 1 --2- 172.21.15.31:0/2240588772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef40100de0 0x7fef401011c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fef30009910 tx=0x7fef30009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:48.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.991+0000 7fef47498700 1 -- 172.21.15.31:0/2240588772 shutdown_connections 2024-04-25T01:45:48.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.991+0000 7fef47498700 1 --2- 172.21.15.31:0/2240588772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef40100de0 0x7fef401011c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:48.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.991+0000 7fef47498700 1 -- 172.21.15.31:0/2240588772 >> 172.21.15.31:0/2240588772 conn(0x7fef400fc690 msgr2=0x7fef400feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:48.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.991+0000 7fef47498700 1 -- 172.21.15.31:0/2240588772 shutdown_connections 2024-04-25T01:45:48.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.991+0000 7fef47498700 1 -- 172.21.15.31:0/2240588772 wait complete. 2024-04-25T01:45:48.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.992+0000 7fef47498700 1 Processor -- start 2024-04-25T01:45:48.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.992+0000 7fef47498700 1 -- start start 2024-04-25T01:45:48.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.992+0000 7fef47498700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef40100de0 0x7fef401965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:48.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.992+0000 7fef47498700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fef4010b060 con 0x7fef40100de0 2024-04-25T01:45:48.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.993+0000 7fef45234700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef40100de0 0x7fef401965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.993+0000 7fef45234700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef40100de0 0x7fef401965c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42732/0 (socket says 172.21.15.31:42732) 2024-04-25T01:45:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.993+0000 7fef45234700 1 -- 172.21.15.31:0/3140529907 learned_addr learned my addr 172.21.15.31:0/3140529907 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.993+0000 7fef45234700 1 -- 172.21.15.31:0/3140529907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef40196b00 con 0x7fef40100de0 2024-04-25T01:45:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.993+0000 7fef45234700 1 --2- 172.21.15.31:0/3140529907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef40100de0 0x7fef401965c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fef300056d0 tx=0x7fef300057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.993+0000 7fef367fc700 1 -- 172.21.15.31:0/3140529907 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef30026070 con 0x7fef40100de0 2024-04-25T01:45:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.993+0000 7fef367fc700 1 -- 172.21.15.31:0/3140529907 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef3000bba0 con 0x7fef40100de0 2024-04-25T01:45:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.993+0000 7fef47498700 1 -- 172.21.15.31:0/3140529907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef40196df0 con 0x7fef40100de0 2024-04-25T01:45:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.993+0000 7fef367fc700 1 -- 172.21.15.31:0/3140529907 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef300037f0 con 0x7fef40100de0 2024-04-25T01:45:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.993+0000 7fef47498700 1 -- 172.21.15.31:0/3140529907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef4019ae90 con 0x7fef40100de0 2024-04-25T01:45:48.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.994+0000 7fef367fc700 1 -- 172.21.15.31:0/3140529907 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fef30029070 con 0x7fef40100de0 2024-04-25T01:45:48.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.994+0000 7fef47498700 1 -- 172.21.15.31:0/3140529907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef401022c0 con 0x7fef40100de0 2024-04-25T01:45:48.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.994+0000 7fef367fc700 1 --2- 172.21.15.31:0/3140529907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef2c0390f0 0x7fef2c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:48.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.994+0000 7fef367fc700 1 -- 172.21.15.31:0/3140529907 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fef3001e070 con 0x7fef40100de0 2024-04-25T01:45:48.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.994+0000 7fef44a33700 1 --2- 172.21.15.31:0/3140529907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef2c0390f0 0x7fef2c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:48.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.995+0000 7fef44a33700 1 --2- 172.21.15.31:0/3140529907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef2c0390f0 0x7fef2c03b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fef3c006fd0 tx=0x7fef3c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:48.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:48.999+0000 7fef367fc700 1 -- 172.21.15.31:0/3140529907 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fef30013070 con 0x7fef40100de0 2024-04-25T01:45:49.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.211+0000 7fef47498700 1 -- 172.21.15.31:0/3140529907 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fef40197920 con 0x7fef2c0390f0 2024-04-25T01:45:49.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.212+0000 7fef367fc700 1 -- 172.21.15.31:0/3140529907 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7fef40197920 con 0x7fef2c0390f0 2024-04-25T01:45:49.207 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:49.207 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}] 2024-04-25T01:45:49.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.214+0000 7fef47498700 1 -- 172.21.15.31:0/3140529907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef2c0390f0 msgr2=0x7fef2c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:49.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.214+0000 7fef47498700 1 --2- 172.21.15.31:0/3140529907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef2c0390f0 0x7fef2c03b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fef3c006fd0 tx=0x7fef3c006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:49.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.214+0000 7fef47498700 1 -- 172.21.15.31:0/3140529907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef40100de0 msgr2=0x7fef401965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:49.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.214+0000 7fef47498700 1 --2- 172.21.15.31:0/3140529907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef40100de0 0x7fef401965c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fef300056d0 tx=0x7fef300057b0 comp rx=0 tx=0).stop 2024-04-25T01:45:49.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.214+0000 7fef47498700 1 -- 172.21.15.31:0/3140529907 shutdown_connections 2024-04-25T01:45:49.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.214+0000 7fef47498700 1 --2- 172.21.15.31:0/3140529907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef2c0390f0 0x7fef2c03b5b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:49.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.214+0000 7fef47498700 1 --2- 172.21.15.31:0/3140529907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef40100de0 0x7fef401965c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:49.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.214+0000 7fef47498700 1 -- 172.21.15.31:0/3140529907 >> 172.21.15.31:0/3140529907 conn(0x7fef400fc690 msgr2=0x7fef401027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:49.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.214+0000 7fef47498700 1 -- 172.21.15.31:0/3140529907 shutdown_connections 2024-04-25T01:45:49.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:49.214+0000 7fef47498700 1 -- 172.21.15.31:0/3140529907 wait complete. 2024-04-25T01:45:49.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:49 smithi031 ceph-mon[29204]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:49.850 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi134 2024-04-25T01:45:49.851 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T01:45:49.851 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.conf 2024-04-25T01:45:49.869 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T01:45:49.869 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:49.925 INFO:tasks.cephadm:Adding host smithi134 to orchestrator... 2024-04-25T01:45:49.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch host add smithi134 2024-04-25T01:45:50.186 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:50.595 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:50 smithi031 ceph-mon[29204]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:45:50.596 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:50 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:51.446 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:51 smithi031 ceph-mon[29204]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:51.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.676+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/425652050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcac101ed0 msgr2=0x7fdcac1022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:51.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.676+0000 7fdcb2a7c700 1 --2- 172.21.15.31:0/425652050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcac101ed0 0x7fdcac1022b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdca0009910 tx=0x7fdca0009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:51.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.676+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/425652050 shutdown_connections 2024-04-25T01:45:51.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.676+0000 7fdcb2a7c700 1 --2- 172.21.15.31:0/425652050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcac101ed0 0x7fdcac1022b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:51.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.676+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/425652050 >> 172.21.15.31:0/425652050 conn(0x7fdcac0fd760 msgr2=0x7fdcac0ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:51.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.676+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/425652050 shutdown_connections 2024-04-25T01:45:51.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.676+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/425652050 wait complete. 2024-04-25T01:45:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.677+0000 7fdcb2a7c700 1 Processor -- start 2024-04-25T01:45:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.677+0000 7fdcb2a7c700 1 -- start start 2024-04-25T01:45:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.677+0000 7fdcb2a7c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcac101ed0 0x7fdcac074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.677+0000 7fdcb2a7c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdcac10c150 con 0x7fdcac101ed0 2024-04-25T01:45:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.677+0000 7fdcb0818700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcac101ed0 0x7fdcac074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.677+0000 7fdcb0818700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcac101ed0 0x7fdcac074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42750/0 (socket says 172.21.15.31:42750) 2024-04-25T01:45:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.677+0000 7fdcb0818700 1 -- 172.21.15.31:0/3971973867 learned_addr learned my addr 172.21.15.31:0/3971973867 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.678+0000 7fdcb0818700 1 -- 172.21.15.31:0/3971973867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcac076420 con 0x7fdcac101ed0 2024-04-25T01:45:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.678+0000 7fdcb0818700 1 --2- 172.21.15.31:0/3971973867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcac101ed0 0x7fdcac074530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdca00056d0 tx=0x7fdca00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.678+0000 7fdca9ffb700 1 -- 172.21.15.31:0/3971973867 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdca0026070 con 0x7fdcac101ed0 2024-04-25T01:45:51.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.678+0000 7fdca9ffb700 1 -- 172.21.15.31:0/3971973867 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdca000bc00 con 0x7fdcac101ed0 2024-04-25T01:45:51.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.678+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/3971973867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcac074ad0 con 0x7fdcac101ed0 2024-04-25T01:45:51.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.678+0000 7fdca9ffb700 1 -- 172.21.15.31:0/3971973867 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdca00039b0 con 0x7fdcac101ed0 2024-04-25T01:45:51.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.678+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/3971973867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcac074f90 con 0x7fdcac101ed0 2024-04-25T01:45:51.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.679+0000 7fdca9ffb700 1 -- 172.21.15.31:0/3971973867 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fdca0029070 con 0x7fdcac101ed0 2024-04-25T01:45:51.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.679+0000 7fdca9ffb700 1 --2- 172.21.15.31:0/3971973867 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc9c039150 0x7fdc9c03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:51.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.679+0000 7fdca9ffb700 1 -- 172.21.15.31:0/3971973867 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdca001e070 con 0x7fdcac101ed0 2024-04-25T01:45:51.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.679+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/3971973867 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcac1033b0 con 0x7fdcac101ed0 2024-04-25T01:45:51.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.679+0000 7fdcabfff700 1 --2- 172.21.15.31:0/3971973867 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc9c039150 0x7fdc9c03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:51.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.680+0000 7fdcabfff700 1 --2- 172.21.15.31:0/3971973867 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc9c039150 0x7fdc9c03b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdc98006fd0 tx=0x7fdc98006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:51.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.684+0000 7fdca9ffb700 1 -- 172.21.15.31:0/3971973867 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdca0030070 con 0x7fdcac101ed0 2024-04-25T01:45:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:51.894+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/3971973867 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}) -- 0x7fdcac075a60 con 0x7fdc9c039150 2024-04-25T01:45:53.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:53.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-25T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:53.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:53 smithi031 ceph-mon[29204]: Deploying cephadm binary to smithi134 2024-04-25T01:45:54.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:54 smithi031 ceph-mon[29204]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:54.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:54 smithi031 ceph-mon[29204]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:45:55.480 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:55 smithi031 ceph-mon[29204]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:56.104 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi134' with addr '172.21.15.134' 2024-04-25T01:45:56.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:56.110+0000 7fdca9ffb700 1 -- 172.21.15.31:0/3971973867 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fdcac075a60 con 0x7fdc9c039150 2024-04-25T01:45:56.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:56.112+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/3971973867 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc9c039150 msgr2=0x7fdc9c03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:56.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:56.112+0000 7fdcb2a7c700 1 --2- 172.21.15.31:0/3971973867 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc9c039150 0x7fdc9c03b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdc98006fd0 tx=0x7fdc98006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:56.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:56.113+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/3971973867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcac101ed0 msgr2=0x7fdcac074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:56.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:56.113+0000 7fdcb2a7c700 1 --2- 172.21.15.31:0/3971973867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcac101ed0 0x7fdcac074530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdca00056d0 tx=0x7fdca00057b0 comp rx=0 tx=0).stop 2024-04-25T01:45:56.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:56.113+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/3971973867 shutdown_connections 2024-04-25T01:45:56.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:56.113+0000 7fdcb2a7c700 1 --2- 172.21.15.31:0/3971973867 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc9c039150 0x7fdc9c03b610 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:56.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:56.113+0000 7fdcb2a7c700 1 --2- 172.21.15.31:0/3971973867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcac101ed0 0x7fdcac074530 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:56.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:56.113+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/3971973867 >> 172.21.15.31:0/3971973867 conn(0x7fdcac0fd760 msgr2=0x7fdcac1038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:56.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:56.113+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/3971973867 shutdown_connections 2024-04-25T01:45:56.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:56.113+0000 7fdcb2a7c700 1 -- 172.21.15.31:0/3971973867 wait complete. 2024-04-25T01:45:56.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:56.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:56.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:56.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:56.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:56.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T01:45:56.927 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:57.711 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:57 smithi031 ceph-mon[29204]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:57.711 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:57 smithi031 ceph-mon[29204]: Added host smithi134 2024-04-25T01:45:57.711 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:57.711 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:45:58.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.467+0000 7fdb28e3a700 1 -- 172.21.15.31:0/1632901677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb24100de0 msgr2=0x7fdb241011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:58.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.467+0000 7fdb28e3a700 1 --2- 172.21.15.31:0/1632901677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb24100de0 0x7fdb241011c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009910 tx=0x7fdb18009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:58.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.467+0000 7fdb28e3a700 1 -- 172.21.15.31:0/1632901677 shutdown_connections 2024-04-25T01:45:58.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.467+0000 7fdb28e3a700 1 --2- 172.21.15.31:0/1632901677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb24100de0 0x7fdb241011c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:58.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.467+0000 7fdb28e3a700 1 -- 172.21.15.31:0/1632901677 >> 172.21.15.31:0/1632901677 conn(0x7fdb240fc690 msgr2=0x7fdb240feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:58.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.467+0000 7fdb28e3a700 1 -- 172.21.15.31:0/1632901677 shutdown_connections 2024-04-25T01:45:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.467+0000 7fdb28e3a700 1 -- 172.21.15.31:0/1632901677 wait complete. 2024-04-25T01:45:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.468+0000 7fdb28e3a700 1 Processor -- start 2024-04-25T01:45:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.468+0000 7fdb28e3a700 1 -- start start 2024-04-25T01:45:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.468+0000 7fdb28e3a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb24100de0 0x7fdb241976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.468+0000 7fdb28e3a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2410b060 con 0x7fdb24100de0 2024-04-25T01:45:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.468+0000 7fdb2259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb24100de0 0x7fdb241976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.468+0000 7fdb2259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb24100de0 0x7fdb241976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47840/0 (socket says 172.21.15.31:47840) 2024-04-25T01:45:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.468+0000 7fdb2259c700 1 -- 172.21.15.31:0/919799985 learned_addr learned my addr 172.21.15.31:0/919799985 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:45:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.469+0000 7fdb2259c700 1 -- 172.21.15.31:0/919799985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb24197c20 con 0x7fdb24100de0 2024-04-25T01:45:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.469+0000 7fdb2259c700 1 --2- 172.21.15.31:0/919799985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb24100de0 0x7fdb241976e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdb180056d0 tx=0x7fdb180057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.469+0000 7fdb137fe700 1 -- 172.21.15.31:0/919799985 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb18026070 con 0x7fdb24100de0 2024-04-25T01:45:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.469+0000 7fdb137fe700 1 -- 172.21.15.31:0/919799985 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb1800bba0 con 0x7fdb24100de0 2024-04-25T01:45:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.469+0000 7fdb28e3a700 1 -- 172.21.15.31:0/919799985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb24197f10 con 0x7fdb24100de0 2024-04-25T01:45:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.469+0000 7fdb137fe700 1 -- 172.21.15.31:0/919799985 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb180037f0 con 0x7fdb24100de0 2024-04-25T01:45:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.469+0000 7fdb28e3a700 1 -- 172.21.15.31:0/919799985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb2419bf60 con 0x7fdb24100de0 2024-04-25T01:45:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.470+0000 7fdb137fe700 1 -- 172.21.15.31:0/919799985 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fdb18029070 con 0x7fdb24100de0 2024-04-25T01:45:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.470+0000 7fdb28e3a700 1 -- 172.21.15.31:0/919799985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb241022c0 con 0x7fdb24100de0 2024-04-25T01:45:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.470+0000 7fdb137fe700 1 --2- 172.21.15.31:0/919799985 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdb0c0390f0 0x7fdb0c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.470+0000 7fdb137fe700 1 -- 172.21.15.31:0/919799985 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdb1801e070 con 0x7fdb24100de0 2024-04-25T01:45:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.470+0000 7fdb21d9b700 1 --2- 172.21.15.31:0/919799985 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdb0c0390f0 0x7fdb0c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.470+0000 7fdb21d9b700 1 --2- 172.21.15.31:0/919799985 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdb0c0390f0 0x7fdb0c03b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdb14006fd0 tx=0x7fdb14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:58.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.475+0000 7fdb137fe700 1 -- 172.21.15.31:0/919799985 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdb18013070 con 0x7fdb24100de0 2024-04-25T01:45:58.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.688+0000 7fdb28e3a700 1 -- 172.21.15.31:0/919799985 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdb241989f0 con 0x7fdb0c0390f0 2024-04-25T01:45:58.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.689+0000 7fdb137fe700 1 -- 172.21.15.31:0/919799985 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fdb241989f0 con 0x7fdb0c0390f0 2024-04-25T01:45:58.683 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:45:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.134", "hostname": "smithi134", "labels": [], "status": ""}] 2024-04-25T01:45:58.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.691+0000 7fdb28e3a700 1 -- 172.21.15.31:0/919799985 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdb0c0390f0 msgr2=0x7fdb0c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:58.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.691+0000 7fdb28e3a700 1 --2- 172.21.15.31:0/919799985 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdb0c0390f0 0x7fdb0c03b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdb14006fd0 tx=0x7fdb14006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:58.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.691+0000 7fdb28e3a700 1 -- 172.21.15.31:0/919799985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb24100de0 msgr2=0x7fdb241976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:58.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.691+0000 7fdb28e3a700 1 --2- 172.21.15.31:0/919799985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb24100de0 0x7fdb241976e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdb180056d0 tx=0x7fdb180057b0 comp rx=0 tx=0).stop 2024-04-25T01:45:58.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.691+0000 7fdb28e3a700 1 -- 172.21.15.31:0/919799985 shutdown_connections 2024-04-25T01:45:58.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.691+0000 7fdb28e3a700 1 --2- 172.21.15.31:0/919799985 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdb0c0390f0 0x7fdb0c03b5b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:58.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.691+0000 7fdb28e3a700 1 --2- 172.21.15.31:0/919799985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb24100de0 0x7fdb241976e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:58.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.691+0000 7fdb28e3a700 1 -- 172.21.15.31:0/919799985 >> 172.21.15.31:0/919799985 conn(0x7fdb240fc690 msgr2=0x7fdb2406a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:58.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.691+0000 7fdb28e3a700 1 -- 172.21.15.31:0/919799985 shutdown_connections 2024-04-25T01:45:58.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:45:58.692+0000 7fdb28e3a700 1 -- 172.21.15.31:0/919799985 wait complete. 2024-04-25T01:45:59.288 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T01:45:59.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T01:45:59.534 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:59.854 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:59 smithi031 ceph-mon[29204]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:59.854 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:45:59 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:00.835 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:00 smithi031 ceph-mon[29204]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:46:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.074+0000 7fbcb0a65700 1 -- 172.21.15.31:0/3996856483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101ed0 msgr2=0x7fbcac1022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.074+0000 7fbcb0a65700 1 --2- 172.21.15.31:0/3996856483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101ed0 0x7fbcac1022b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c009910 tx=0x7fbc9c009c20 comp rx=0 tx=0).stop 2024-04-25T01:46:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.074+0000 7fbcb0a65700 1 -- 172.21.15.31:0/3996856483 shutdown_connections 2024-04-25T01:46:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.074+0000 7fbcb0a65700 1 --2- 172.21.15.31:0/3996856483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101ed0 0x7fbcac1022b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.074+0000 7fbcb0a65700 1 -- 172.21.15.31:0/3996856483 >> 172.21.15.31:0/3996856483 conn(0x7fbcac0fd760 msgr2=0x7fbcac0ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.074+0000 7fbcb0a65700 1 -- 172.21.15.31:0/3996856483 shutdown_connections 2024-04-25T01:46:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.074+0000 7fbcb0a65700 1 -- 172.21.15.31:0/3996856483 wait complete. 2024-04-25T01:46:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.075+0000 7fbcb0a65700 1 Processor -- start 2024-04-25T01:46:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.075+0000 7fbcb0a65700 1 -- start start 2024-04-25T01:46:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.075+0000 7fbcb0a65700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101ed0 0x7fbcac074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.075+0000 7fbcb0a65700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac10c150 con 0x7fbcac101ed0 2024-04-25T01:46:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.075+0000 7fbcaa59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101ed0 0x7fbcac074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.075+0000 7fbcaa59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101ed0 0x7fbcac074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47854/0 (socket says 172.21.15.31:47854) 2024-04-25T01:46:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.075+0000 7fbcaa59c700 1 -- 172.21.15.31:0/628920232 learned_addr learned my addr 172.21.15.31:0/628920232 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:46:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.075+0000 7fbcaa59c700 1 -- 172.21.15.31:0/628920232 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcac076420 con 0x7fbcac101ed0 2024-04-25T01:46:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.075+0000 7fbcaa59c700 1 --2- 172.21.15.31:0/628920232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101ed0 0x7fbcac074530 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0056d0 tx=0x7fbc9c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.076+0000 7fbc9b7fe700 1 -- 172.21.15.31:0/628920232 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc9c026070 con 0x7fbcac101ed0 2024-04-25T01:46:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.076+0000 7fbc9b7fe700 1 -- 172.21.15.31:0/628920232 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc9c00bc00 con 0x7fbcac101ed0 2024-04-25T01:46:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.076+0000 7fbcb0a65700 1 -- 172.21.15.31:0/628920232 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcac074ad0 con 0x7fbcac101ed0 2024-04-25T01:46:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.076+0000 7fbc9b7fe700 1 -- 172.21.15.31:0/628920232 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc9c0039b0 con 0x7fbcac101ed0 2024-04-25T01:46:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.076+0000 7fbcb0a65700 1 -- 172.21.15.31:0/628920232 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcac074f90 con 0x7fbcac101ed0 2024-04-25T01:46:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.077+0000 7fbc9b7fe700 1 -- 172.21.15.31:0/628920232 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fbc9c029070 con 0x7fbcac101ed0 2024-04-25T01:46:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.077+0000 7fbcb0a65700 1 -- 172.21.15.31:0/628920232 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcac1033b0 con 0x7fbcac101ed0 2024-04-25T01:46:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.077+0000 7fbc9b7fe700 1 --2- 172.21.15.31:0/628920232 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc940390f0 0x7fbc9403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.077+0000 7fbc9b7fe700 1 -- 172.21.15.31:0/628920232 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbc9c01e070 con 0x7fbcac101ed0 2024-04-25T01:46:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.077+0000 7fbca9d9b700 1 --2- 172.21.15.31:0/628920232 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc940390f0 0x7fbc9403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.077+0000 7fbca9d9b700 1 --2- 172.21.15.31:0/628920232 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc940390f0 0x7fbc9403b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbca0006fd0 tx=0x7fbca0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:01.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.082+0000 7fbc9b7fe700 1 -- 172.21.15.31:0/628920232 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbc9c013070 con 0x7fbcac101ed0 2024-04-25T01:46:01.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.292+0000 7fbcb0a65700 1 -- 172.21.15.31:0/628920232 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fbcac075b00 con 0x7fbcac101ed0 2024-04-25T01:46:01.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.567+0000 7fbc9b7fe700 1 -- 172.21.15.31:0/628920232 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fbc9c003c10 con 0x7fbcac101ed0 2024-04-25T01:46:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.569+0000 7fbcb0a65700 1 -- 172.21.15.31:0/628920232 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc940390f0 msgr2=0x7fbc9403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.569+0000 7fbcb0a65700 1 --2- 172.21.15.31:0/628920232 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc940390f0 0x7fbc9403b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbca0006fd0 tx=0x7fbca0006e40 comp rx=0 tx=0).stop 2024-04-25T01:46:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.569+0000 7fbcb0a65700 1 -- 172.21.15.31:0/628920232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101ed0 msgr2=0x7fbcac074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.569+0000 7fbcb0a65700 1 --2- 172.21.15.31:0/628920232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101ed0 0x7fbcac074530 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0056d0 tx=0x7fbc9c0057b0 comp rx=0 tx=0).stop 2024-04-25T01:46:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.569+0000 7fbcb0a65700 1 -- 172.21.15.31:0/628920232 shutdown_connections 2024-04-25T01:46:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.569+0000 7fbcb0a65700 1 --2- 172.21.15.31:0/628920232 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc940390f0 0x7fbc9403b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.569+0000 7fbcb0a65700 1 --2- 172.21.15.31:0/628920232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101ed0 0x7fbcac074530 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.569+0000 7fbcb0a65700 1 -- 172.21.15.31:0/628920232 >> 172.21.15.31:0/628920232 conn(0x7fbcac0fd760 msgr2=0x7fbcac1038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.569+0000 7fbcb0a65700 1 -- 172.21.15.31:0/628920232 shutdown_connections 2024-04-25T01:46:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:01.570+0000 7fbcb0a65700 1 -- 172.21.15.31:0/628920232 wait complete. 2024-04-25T01:46:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-04-25T01:46:01.914 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:01 smithi031 ceph-mon[29204]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:01.914 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:01 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/628920232' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:46:02.227 INFO:tasks.cephadm:Adding mon.a on smithi031 2024-04-25T01:46:02.227 INFO:tasks.cephadm:Adding mon.b on smithi040 2024-04-25T01:46:02.228 INFO:tasks.cephadm:Adding mon.c on smithi134 2024-04-25T01:46:02.228 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi134:172.21.15.134=c' 2024-04-25T01:46:02.463 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:02.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:02 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/628920232' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:46:02.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:02 smithi031 ceph-mon[29204]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:46:02.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:02.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:02.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:02.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:02.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:46:02.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:02.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:03.980 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:03 smithi031 ceph-mon[29204]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:03.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:03 smithi031 ceph-mon[29204]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-25T01:46:03.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:03 smithi031 ceph-mon[29204]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:03.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:03 smithi031 ceph-mon[29204]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:46:03.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:03 smithi031 ceph-mon[29204]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:46:03.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:03 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:03.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:03 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:03.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:03 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:04.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.057+0000 7f6b0661b700 1 -- 172.21.15.134:0/211346122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b00100dd0 msgr2=0x7f6b001011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:04.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.057+0000 7f6b0661b700 1 --2- 172.21.15.134:0/211346122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b00100dd0 0x7f6b001011b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009910 tx=0x7f6af0009c20 comp rx=0 tx=0).stop 2024-04-25T01:46:04.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.057+0000 7f6b0661b700 1 -- 172.21.15.134:0/211346122 shutdown_connections 2024-04-25T01:46:04.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.057+0000 7f6b0661b700 1 --2- 172.21.15.134:0/211346122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b00100dd0 0x7f6b001011b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:04.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.057+0000 7f6b0661b700 1 -- 172.21.15.134:0/211346122 >> 172.21.15.134:0/211346122 conn(0x7f6b000fc680 msgr2=0x7f6b000feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:04.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.057+0000 7f6b0661b700 1 -- 172.21.15.134:0/211346122 shutdown_connections 2024-04-25T01:46:04.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.057+0000 7f6b0661b700 1 -- 172.21.15.134:0/211346122 wait complete. 2024-04-25T01:46:04.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.058+0000 7f6b0661b700 1 Processor -- start 2024-04-25T01:46:04.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.058+0000 7f6b0661b700 1 -- start start 2024-04-25T01:46:04.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.058+0000 7f6b0661b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b00100dd0 0x7f6b00197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:04.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.058+0000 7f6b0661b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0010b050 con 0x7f6b00100dd0 2024-04-25T01:46:04.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.058+0000 7f6afffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b00100dd0 0x7f6b00197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:04.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.058+0000 7f6afffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b00100dd0 0x7f6b00197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.134:45486/0 (socket says 172.21.15.134:45486) 2024-04-25T01:46:04.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.058+0000 7f6afffff700 1 -- 172.21.15.134:0/49280553 learned_addr learned my addr 172.21.15.134:0/49280553 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-25T01:46:04.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.059+0000 7f6afffff700 1 -- 172.21.15.134:0/49280553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b00197bb0 con 0x7f6b00100dd0 2024-04-25T01:46:04.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.059+0000 7f6afffff700 1 --2- 172.21.15.134:0/49280553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b00100dd0 0x7f6b00197670 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6af00056d0 tx=0x7f6af00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:04.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.059+0000 7f6afd7fa700 1 -- 172.21.15.134:0/49280553 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6af0026070 con 0x7f6b00100dd0 2024-04-25T01:46:04.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.059+0000 7f6afd7fa700 1 -- 172.21.15.134:0/49280553 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6af000bba0 con 0x7f6b00100dd0 2024-04-25T01:46:04.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.059+0000 7f6b0661b700 1 -- 172.21.15.134:0/49280553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b00197ea0 con 0x7f6b00100dd0 2024-04-25T01:46:04.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.059+0000 7f6afd7fa700 1 -- 172.21.15.134:0/49280553 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6af00037f0 con 0x7f6b00100dd0 2024-04-25T01:46:04.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.059+0000 7f6b0661b700 1 -- 172.21.15.134:0/49280553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b0019bf40 con 0x7f6b00100dd0 2024-04-25T01:46:04.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.060+0000 7f6b0661b700 1 -- 172.21.15.134:0/49280553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b001022b0 con 0x7f6b00100dd0 2024-04-25T01:46:04.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.060+0000 7f6afd7fa700 1 -- 172.21.15.134:0/49280553 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f6af0029070 con 0x7f6b00100dd0 2024-04-25T01:46:04.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.060+0000 7f6afd7fa700 1 --2- 172.21.15.134:0/49280553 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6ae8039150 0x7f6ae803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:04.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.060+0000 7f6afd7fa700 1 -- 172.21.15.134:0/49280553 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6af001e070 con 0x7f6b00100dd0 2024-04-25T01:46:04.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.060+0000 7f6aff7fe700 1 --2- 172.21.15.134:0/49280553 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6ae8039150 0x7f6ae803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:04.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.061+0000 7f6aff7fe700 1 --2- 172.21.15.134:0/49280553 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6ae8039150 0x7f6ae803b610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6af4006fd0 tx=0x7f6af4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:04.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.066+0000 7f6afd7fa700 1 -- 172.21.15.134:0/49280553 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6af0013070 con 0x7f6b00100dd0 2024-04-25T01:46:04.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.272+0000 7f6b0661b700 1 -- 172.21.15.134:0/49280553 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi134:172.21.15.134=c", "target": ["mon-mgr", ""]}) -- 0x7f6b001989d0 con 0x7f6ae8039150 2024-04-25T01:46:04.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.279+0000 7f6afd7fa700 1 -- 172.21.15.134:0/49280553 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6b001989d0 con 0x7f6ae8039150 2024-04-25T01:46:04.272 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled mon update... 2024-04-25T01:46:04.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.281+0000 7f6b0661b700 1 -- 172.21.15.134:0/49280553 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6ae8039150 msgr2=0x7f6ae803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:04.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.281+0000 7f6b0661b700 1 --2- 172.21.15.134:0/49280553 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6ae8039150 0x7f6ae803b610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6af4006fd0 tx=0x7f6af4006e40 comp rx=0 tx=0).stop 2024-04-25T01:46:04.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.281+0000 7f6b0661b700 1 -- 172.21.15.134:0/49280553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b00100dd0 msgr2=0x7f6b00197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:04.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.281+0000 7f6b0661b700 1 --2- 172.21.15.134:0/49280553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b00100dd0 0x7f6b00197670 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6af00056d0 tx=0x7f6af00057b0 comp rx=0 tx=0).stop 2024-04-25T01:46:04.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.281+0000 7f6b0661b700 1 -- 172.21.15.134:0/49280553 shutdown_connections 2024-04-25T01:46:04.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.281+0000 7f6b0661b700 1 --2- 172.21.15.134:0/49280553 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6ae8039150 0x7f6ae803b610 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:04.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.281+0000 7f6b0661b700 1 --2- 172.21.15.134:0/49280553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b00100dd0 0x7f6b00197670 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:04.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.281+0000 7f6b0661b700 1 -- 172.21.15.134:0/49280553 >> 172.21.15.134:0/49280553 conn(0x7f6b000fc680 msgr2=0x7f6b0006a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:04.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.281+0000 7f6b0661b700 1 -- 172.21.15.134:0/49280553 shutdown_connections 2024-04-25T01:46:04.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:04.281+0000 7f6b0661b700 1 -- 172.21.15.134:0/49280553 wait complete. 2024-04-25T01:46:04.863 DEBUG:teuthology.orchestra.run.smithi040:mon.b> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T01:46:04.879 DEBUG:teuthology.orchestra.run.smithi134:mon.c> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T01:46:04.892 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:46:04.893 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:46:05.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:05 smithi031 ceph-mon[29204]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:05.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:05 smithi031 ceph-mon[29204]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi134:172.21.15.134=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:05.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:05 smithi031 ceph-mon[29204]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi134:172.21.15.134=c;count:3 2024-04-25T01:46:05.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:05.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:05.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:05.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:05.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:05.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:05.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:05.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:05 smithi031 ceph-mon[29204]: Deploying daemon mon.c on smithi134 2024-04-25T01:46:07.017 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:46:07.532 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:07 smithi031 ceph-mon[29204]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:08.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.769+0000 7f8a58205700 1 -- 172.21.15.134:0/2119617958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a50100e90 msgr2=0x7f8a50101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.769+0000 7f8a58205700 1 --2- 172.21.15.134:0/2119617958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a50100e90 0x7f8a50101270 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800b0a0 tx=0x7f8a4800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.769+0000 7f8a58205700 1 -- 172.21.15.134:0/2119617958 shutdown_connections 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.769+0000 7f8a58205700 1 --2- 172.21.15.134:0/2119617958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a50100e90 0x7f8a50101270 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.769+0000 7f8a58205700 1 -- 172.21.15.134:0/2119617958 >> 172.21.15.134:0/2119617958 conn(0x7f8a500fc760 msgr2=0x7f8a500feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.770+0000 7f8a58205700 1 -- 172.21.15.134:0/2119617958 shutdown_connections 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.770+0000 7f8a58205700 1 -- 172.21.15.134:0/2119617958 wait complete. 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.770+0000 7f8a58205700 1 Processor -- start 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.770+0000 7f8a58205700 1 -- start start 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.770+0000 7f8a58205700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a50100e90 0x7f8a50197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.770+0000 7f8a58205700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5010b110 con 0x7f8a50100e90 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.771+0000 7f8a55fa1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a50100e90 0x7f8a50197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.771+0000 7f8a55fa1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a50100e90 0x7f8a50197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.134:39302/0 (socket says 172.21.15.134:39302) 2024-04-25T01:46:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.771+0000 7f8a55fa1700 1 -- 172.21.15.134:0/546709857 learned_addr learned my addr 172.21.15.134:0/546709857 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-25T01:46:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.771+0000 7f8a55fa1700 1 -- 172.21.15.134:0/546709857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a50197c90 con 0x7f8a50100e90 2024-04-25T01:46:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.771+0000 7f8a55fa1700 1 --2- 172.21.15.134:0/546709857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a50100e90 0x7f8a50197750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800d010 tx=0x7f8a48009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.771+0000 7f8a46ffd700 1 -- 172.21.15.134:0/546709857 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a48017040 con 0x7f8a50100e90 2024-04-25T01:46:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.771+0000 7f8a46ffd700 1 -- 172.21.15.134:0/546709857 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a48022070 con 0x7f8a50100e90 2024-04-25T01:46:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.771+0000 7f8a58205700 1 -- 172.21.15.134:0/546709857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a50197f20 con 0x7f8a50100e90 2024-04-25T01:46:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.771+0000 7f8a46ffd700 1 -- 172.21.15.134:0/546709857 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a4801d5e0 con 0x7f8a50100e90 2024-04-25T01:46:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.771+0000 7f8a58205700 1 -- 172.21.15.134:0/546709857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5019bfd0 con 0x7f8a50100e90 2024-04-25T01:46:08.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.772+0000 7f8a46ffd700 1 -- 172.21.15.134:0/546709857 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f8a4801d8e0 con 0x7f8a50100e90 2024-04-25T01:46:08.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.772+0000 7f8a58205700 1 -- 172.21.15.134:0/546709857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a50102370 con 0x7f8a50100e90 2024-04-25T01:46:08.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.772+0000 7f8a46ffd700 1 --2- 172.21.15.134:0/546709857 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8a3c039140 0x7f8a3c03b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:08.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.772+0000 7f8a46ffd700 1 -- 172.21.15.134:0/546709857 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8a4800f050 con 0x7f8a50100e90 2024-04-25T01:46:08.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.772+0000 7f8a557a0700 1 --2- 172.21.15.134:0/546709857 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8a3c039140 0x7f8a3c03b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:08.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.773+0000 7f8a557a0700 1 --2- 172.21.15.134:0/546709857 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8a3c039140 0x7f8a3c03b600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8a40006fd0 tx=0x7f8a40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:08.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:08.777+0000 7f8a46ffd700 1 -- 172.21.15.134:0/546709857 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8a4801ade0 con 0x7f8a50100e90 2024-04-25T01:46:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.072+0000 7f8a58205700 1 -- 172.21.15.134:0/546709857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f8a50198a60 con 0x7f8a50100e90 2024-04-25T01:46:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.073+0000 7f8a46ffd700 1 -- 172.21.15.134:0/546709857 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f8a48004180 con 0x7f8a50100e90 2024-04-25T01:46:09.066 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:46:09.066 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","modified":"2024-04-25T01:44:14.223274Z","created":"2024-04-25T01:44:14.223274Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T01:46:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.075+0000 7f8a58205700 1 -- 172.21.15.134:0/546709857 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8a3c039140 msgr2=0x7f8a3c03b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.075+0000 7f8a58205700 1 --2- 172.21.15.134:0/546709857 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8a3c039140 0x7f8a3c03b600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8a40006fd0 tx=0x7f8a40006e40 comp rx=0 tx=0).stop 2024-04-25T01:46:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.075+0000 7f8a58205700 1 -- 172.21.15.134:0/546709857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a50100e90 msgr2=0x7f8a50197750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.075+0000 7f8a58205700 1 --2- 172.21.15.134:0/546709857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a50100e90 0x7f8a50197750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800d010 tx=0x7f8a48009790 comp rx=0 tx=0).stop 2024-04-25T01:46:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.075+0000 7f8a58205700 1 -- 172.21.15.134:0/546709857 shutdown_connections 2024-04-25T01:46:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.075+0000 7f8a58205700 1 --2- 172.21.15.134:0/546709857 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8a3c039140 0x7f8a3c03b600 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.075+0000 7f8a58205700 1 --2- 172.21.15.134:0/546709857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a50100e90 0x7f8a50197750 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.075+0000 7f8a58205700 1 -- 172.21.15.134:0/546709857 >> 172.21.15.134:0/546709857 conn(0x7f8a500fc760 msgr2=0x7f8a5006a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.075+0000 7f8a58205700 1 -- 172.21.15.134:0/546709857 shutdown_connections 2024-04-25T01:46:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:09.075+0000 7f8a58205700 1 -- 172.21.15.134:0/546709857 wait complete. 2024-04-25T01:46:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-25T01:46:10.889 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:46:10.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:46:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:46:14.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:46:14.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: mon.a calling monitor election 2024-04-25T01:46:14.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:14.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:14.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:14.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:14.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: mon.c calling monitor election 2024-04-25T01:46:14.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:14.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:14.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:14.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:14.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: monmap epoch 2 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: last_changed 2024-04-25T01:46:09.163035+0000 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: created 2024-04-25T01:44:14.223274+0000 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: min_mon_release 19 (squid) 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: election_strategy: 1 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.c 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: fsmap 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: mgrmap e13: x(active, since 50s) 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: overall HEALTH_OK 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:14.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:14.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:14.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:14.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:14.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:14 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:15.432 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:15 smithi134 ceph-mon[33098]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:15.432 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:15 smithi134 ceph-mon[33098]: Deploying daemon mon.b on smithi040 2024-04-25T01:46:15.432 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:15.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:15 smithi031 ceph-mon[29204]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:15.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:15 smithi031 ceph-mon[29204]: Deploying daemon mon.b on smithi040 2024-04-25T01:46:15.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:15.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.529+0000 7f7ece20b700 1 -- 172.21.15.134:0/1301682702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7eb40035c0 msgr2=0x7f7eb4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:15.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.529+0000 7f7ece20b700 1 --2- 172.21.15.134:0/1301682702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7eb40035c0 0x7f7eb4005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8078360 tx=0x7f7eb800feb0 comp rx=0 tx=0).stop 2024-04-25T01:46:15.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.529+0000 7f7ece20b700 1 -- 172.21.15.134:0/1301682702 shutdown_connections 2024-04-25T01:46:15.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.529+0000 7f7ece20b700 1 --2- 172.21.15.134:0/1301682702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7eb40035c0 0x7f7eb4005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:15.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.529+0000 7f7ece20b700 1 -- 172.21.15.134:0/1301682702 >> 172.21.15.134:0/1301682702 conn(0x7f7ec80770b0 msgr2=0x7f7ec80774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:15.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.529+0000 7f7ece20b700 1 -- 172.21.15.134:0/1301682702 shutdown_connections 2024-04-25T01:46:15.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.529+0000 7f7ece20b700 1 -- 172.21.15.134:0/1301682702 wait complete. 2024-04-25T01:46:15.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.531+0000 7f7ece20b700 1 Processor -- start 2024-04-25T01:46:15.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.531+0000 7f7ece20b700 1 -- start start 2024-04-25T01:46:15.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.531+0000 7f7ece20b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7eb40035c0 0x7f7ec819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:15.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.531+0000 7f7ece20b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ec819d920 0x7f7ec81975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:15.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.531+0000 7f7ece20b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec8109c70 con 0x7f7eb40035c0 2024-04-25T01:46:15.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.531+0000 7f7ece20b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec8109af0 con 0x7f7ec819d920 2024-04-25T01:46:15.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ebeffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ec819d920 0x7f7ec81975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:15.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ec77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7eb40035c0 0x7f7ec819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:15.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ebeffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ec819d920 0x7f7ec81975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40326/0 (socket says 172.21.15.134:40326) 2024-04-25T01:46:15.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ebeffd700 1 -- 172.21.15.134:0/324644064 learned_addr learned my addr 172.21.15.134:0/324644064 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-25T01:46:15.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ebeffd700 1 -- 172.21.15.134:0/324644064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ec819d920 msgr2=0x7f7ec81975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T01:46:15.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ebeffd700 1 -- 172.21.15.134:0/324644064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ec819d920 msgr2=0x7f7ec81975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:46:15.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ebeffd700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ec819d920 0x7f7ec81975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:46:15.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ebeffd700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ec819d920 0x7f7ec81975f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:46:15.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ec77fe700 1 -- 172.21.15.134:0/324644064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ec819d920 msgr2=0x7f7ec81975f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:46:15.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ec77fe700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ec819d920 0x7f7ec81975f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:15.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ec77fe700 1 -- 172.21.15.134:0/324644064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec819de70 con 0x7f7eb40035c0 2024-04-25T01:46:15.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.532+0000 7f7ec77fe700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7eb40035c0 0x7f7ec819d3e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8078360 tx=0x7f7eb80046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:15.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.533+0000 7f7ec57fa700 1 -- 172.21.15.134:0/324644064 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7eb8013b00 con 0x7f7eb40035c0 2024-04-25T01:46:15.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.533+0000 7f7ec57fa700 1 -- 172.21.15.134:0/324644064 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7eb80044e0 con 0x7f7eb40035c0 2024-04-25T01:46:15.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.533+0000 7f7ece20b700 1 -- 172.21.15.134:0/324644064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec8197b30 con 0x7f7eb40035c0 2024-04-25T01:46:15.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.533+0000 7f7ec57fa700 1 -- 172.21.15.134:0/324644064 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7eb8003d80 con 0x7f7eb40035c0 2024-04-25T01:46:15.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.533+0000 7f7ece20b700 1 -- 172.21.15.134:0/324644064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec81980c0 con 0x7f7eb40035c0 2024-04-25T01:46:15.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.534+0000 7f7ece20b700 1 -- 172.21.15.134:0/324644064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ec8198430 con 0x7f7eb40035c0 2024-04-25T01:46:15.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.534+0000 7f7ec57fa700 1 -- 172.21.15.134:0/324644064 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f7eb8023070 con 0x7f7eb40035c0 2024-04-25T01:46:15.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.534+0000 7f7ec57fa700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7ea8038ee0 0x7f7ea803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:15.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.534+0000 7f7ec57fa700 1 -- 172.21.15.134:0/324644064 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7eb8020050 con 0x7f7eb40035c0 2024-04-25T01:46:15.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.534+0000 7f7ebeffd700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7ea8038ee0 0x7f7ea803b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:15.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.535+0000 7f7ebeffd700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7ea8038ee0 0x7f7ea803b3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0009180 tx=0x7f7eb0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:15.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.540+0000 7f7ec57fa700 1 -- 172.21.15.134:0/324644064 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7eb8028020 con 0x7f7eb40035c0 2024-04-25T01:46:15.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.827+0000 7f7ece20b700 1 -- 172.21.15.134:0/324644064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7ec8063890 con 0x7f7eb40035c0 2024-04-25T01:46:15.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.828+0000 7f7ec57fa700 1 -- 172.21.15.134:0/324644064 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f7eb800f8b0 con 0x7f7eb40035c0 2024-04-25T01:46:15.822 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:46:15.823 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":2,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","modified":"2024-04-25T01:46:09.163035Z","created":"2024-04-25T01:44:14.223274Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T01:46:15.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.831+0000 7f7ece20b700 1 -- 172.21.15.134:0/324644064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7ea8038ee0 msgr2=0x7f7ea803b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:15.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.831+0000 7f7ece20b700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7ea8038ee0 0x7f7ea803b3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0009180 tx=0x7f7eb0008040 comp rx=0 tx=0).stop 2024-04-25T01:46:15.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.831+0000 7f7ece20b700 1 -- 172.21.15.134:0/324644064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7eb40035c0 msgr2=0x7f7ec819d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:15.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.831+0000 7f7ece20b700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7eb40035c0 0x7f7ec819d3e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8078360 tx=0x7f7eb80046f0 comp rx=0 tx=0).stop 2024-04-25T01:46:15.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.831+0000 7f7ece20b700 1 -- 172.21.15.134:0/324644064 shutdown_connections 2024-04-25T01:46:15.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.831+0000 7f7ece20b700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7ea8038ee0 0x7f7ea803b3a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:15.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.831+0000 7f7ece20b700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7eb40035c0 0x7f7ec819d3e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:15.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.831+0000 7f7ece20b700 1 --2- 172.21.15.134:0/324644064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ec819d920 0x7f7ec81975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:15.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.831+0000 7f7ece20b700 1 -- 172.21.15.134:0/324644064 >> 172.21.15.134:0/324644064 conn(0x7f7ec80770b0 msgr2=0x7f7ec80fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:15.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.831+0000 7f7ece20b700 1 -- 172.21.15.134:0/324644064 shutdown_connections 2024-04-25T01:46:15.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:15.831+0000 7f7ece20b700 1 -- 172.21.15.134:0/324644064 wait complete. 2024-04-25T01:46:15.827 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 2 2024-04-25T01:46:16.383 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:16 smithi134 ceph-mon[33098]: from='client.? 172.21.15.134:0/324644064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:46:16.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:16 smithi031 ceph-mon[29204]: from='client.? 172.21.15.134:0/324644064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:46:17.413 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:46:17.413 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:46:17.656 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:46:17.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:17 smithi134 ceph-mon[33098]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:17.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:17 smithi031 ceph-mon[29204]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:17.990 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(???) e0 preinit fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:46:17.990 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).mds e1 new map 2024-04-25T01:46:17.990 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-25T01:46:17.990 INFO:journalctl@ceph.mon.b.smithi040.stdout: e1 2024-04-25T01:46:17.990 INFO:journalctl@ceph.mon.b.smithi040.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:46:17.990 INFO:journalctl@ceph.mon.b.smithi040.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:46:17.991 INFO:journalctl@ceph.mon.b.smithi040.stdout: legacy client fscid: -1 2024-04-25T01:46:17.991 INFO:journalctl@ceph.mon.b.smithi040.stdout: 2024-04-25T01:46:17.991 INFO:journalctl@ceph.mon.b.smithi040.stdout: No filesystems configured 2024-04-25T01:46:17.991 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T01:46:17.991 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T01:46:18.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T01:46:18.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T01:46:18.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T01:46:18.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Added host smithi040 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Deploying cephadm binary to smithi134 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Added host smithi134 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/628920232' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:46:18.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/628920232' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi134:172.21.15.134=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:18.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi134:172.21.15.134=c;count:3 2024-04-25T01:46:18.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:18.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:18.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Deploying daemon mon.c on smithi134 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.a calling monitor election 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.c calling monitor election 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: monmap epoch 2 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: last_changed 2024-04-25T01:46:09.163035+0000 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: created 2024-04-25T01:44:14.223274+0000 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: min_mon_release 19 (squid) 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: election_strategy: 1 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.c 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: fsmap 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mgrmap e13: x(active, since 50s) 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: overall HEALTH_OK 2024-04-25T01:46:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: Deploying daemon mon.b on smithi040 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: from='client.? 172.21.15.134:0/324644064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:17 smithi040 ceph-mon[32418]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:18 smithi040 bash[32398]: 6695eaeb0133542ceea77d2d5bea3d67affcc8a5eb58c4268e162fcb25e95dec 2024-04-25T01:46:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:18 smithi040 systemd[1]: Started Ceph mon.b for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: mon.a calling monitor election 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: mon.c calling monitor election 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:23.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: monmap epoch 3 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: last_changed 2024-04-25T01:46:18.002062+0000 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: created 2024-04-25T01:44:14.223274+0000 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: min_mon_release 19 (squid) 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: election_strategy: 1 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.c 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: fsmap 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: mgrmap e13: x(active, since 59s) 2024-04-25T01:46:23.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: overall HEALTH_OK 2024-04-25T01:46:23.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:23.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:23.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:23.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:23.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:23.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:46:23.481 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: mon.a calling monitor election 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: mon.c calling monitor election 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: monmap epoch 3 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:46:23.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: last_changed 2024-04-25T01:46:18.002062+0000 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: created 2024-04-25T01:44:14.223274+0000 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: min_mon_release 19 (squid) 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: election_strategy: 1 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.c 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: fsmap 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: mgrmap e13: x(active, since 59s) 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: overall HEALTH_OK 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:23.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:23.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.526+0000 7f14243b4700 1 -- 172.21.15.134:0/4105953498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14080057d0 msgr2=0x7f1408007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:23.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.526+0000 7f14243b4700 1 --2- 172.21.15.134:0/4105953498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14080057d0 0x7f1408007c60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f141c100b60 tx=0x7f141800fe60 comp rx=0 tx=0).stop 2024-04-25T01:46:23.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.527+0000 7f14243b4700 1 -- 172.21.15.134:0/4105953498 shutdown_connections 2024-04-25T01:46:23.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.527+0000 7f14243b4700 1 --2- 172.21.15.134:0/4105953498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14080057d0 0x7f1408007c60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:23.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.527+0000 7f14243b4700 1 -- 172.21.15.134:0/4105953498 >> 172.21.15.134:0/4105953498 conn(0x7f141c0fc760 msgr2=0x7f141c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:23.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.527+0000 7f14243b4700 1 -- 172.21.15.134:0/4105953498 shutdown_connections 2024-04-25T01:46:23.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.527+0000 7f14243b4700 1 -- 172.21.15.134:0/4105953498 wait complete. 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.527+0000 7f14243b4700 1 Processor -- start 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.527+0000 7f14243b4700 1 -- start start 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f14243b4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14080057d0 0x7f141c195530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f14243b4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f141c199490 0x7f141c195a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f14243b4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141c1960c0 0x7f141c1aa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f14243b4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f141c107b10 con 0x7f14080057d0 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f14243b4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f141c06b280 con 0x7f141c1960c0 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f14243b4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f141c107c90 con 0x7f141c199490 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f142194f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f141c199490 0x7f141c195a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f142194f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f141c199490 0x7f141c195a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40342/0 (socket says 172.21.15.134:40342) 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f142194f700 1 -- 172.21.15.134:0/3761730852 learned_addr learned my addr 172.21.15.134:0/3761730852 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-25T01:46:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f142194f700 1 -- 172.21.15.134:0/3761730852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f141c199490 msgr2=0x7f141c195a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f1422951700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141c1960c0 0x7f141c1aa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f142194f700 1 -- 172.21.15.134:0/3761730852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f141c199490 msgr2=0x7f141c195a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f142194f700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f141c199490 0x7f141c195a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f1422150700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14080057d0 0x7f141c195530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f142194f700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f141c199490 0x7f141c195a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f1422951700 1 -- 172.21.15.134:0/3761730852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141c1960c0 msgr2=0x7f141c1aa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f1422951700 1 -- 172.21.15.134:0/3761730852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141c1960c0 msgr2=0x7f141c1aa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f1422951700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141c1960c0 0x7f141c1aa0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f1422951700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141c1960c0 0x7f141c1aa0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f1422150700 1 -- 172.21.15.134:0/3761730852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f141c199490 msgr2=0x7f141c195a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f1422150700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f141c199490 0x7f141c195a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f1422150700 1 -- 172.21.15.134:0/3761730852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141c1960c0 msgr2=0x7f141c1aa0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f1422150700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141c1960c0 0x7f141c1aa0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:23.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.528+0000 7f1422150700 1 -- 172.21.15.134:0/3761730852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141c1aa9b0 con 0x7f14080057d0 2024-04-25T01:46:23.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.529+0000 7f1422150700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14080057d0 0x7f141c195530 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f141c100b60 tx=0x7f14180054f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:23.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.529+0000 7f14137fe700 1 -- 172.21.15.134:0/3761730852 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141802b020 con 0x7f14080057d0 2024-04-25T01:46:23.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.529+0000 7f14137fe700 1 -- 172.21.15.134:0/3761730852 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f141801d440 con 0x7f14080057d0 2024-04-25T01:46:23.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.529+0000 7f14243b4700 1 -- 172.21.15.134:0/3761730852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141c1aac40 con 0x7f14080057d0 2024-04-25T01:46:23.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.529+0000 7f14137fe700 1 -- 172.21.15.134:0/3761730852 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418022040 con 0x7f14080057d0 2024-04-25T01:46:23.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.529+0000 7f14243b4700 1 -- 172.21.15.134:0/3761730852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f141c1ab180 con 0x7f14080057d0 2024-04-25T01:46:23.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.530+0000 7f14137fe700 1 -- 172.21.15.134:0/3761730852 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f141802e020 con 0x7f14080057d0 2024-04-25T01:46:23.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.530+0000 7f14243b4700 1 -- 172.21.15.134:0/3761730852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f141c102370 con 0x7f14080057d0 2024-04-25T01:46:23.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.530+0000 7f14137fe700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1400039580 0x7f140003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:23.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.530+0000 7f14137fe700 1 -- 172.21.15.134:0/3761730852 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f141804eac0 con 0x7f14080057d0 2024-04-25T01:46:23.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.530+0000 7f142194f700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1400039580 0x7f140003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:23.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.531+0000 7f142194f700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1400039580 0x7f140003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f141c196cf0 tx=0x7f140c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:23.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.535+0000 7f14137fe700 1 -- 172.21.15.134:0/3761730852 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1418035020 con 0x7f14080057d0 2024-04-25T01:46:23.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.827+0000 7f14243b4700 1 -- 172.21.15.134:0/3761730852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f141c02db30 con 0x7f14080057d0 2024-04-25T01:46:23.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.828+0000 7f14137fe700 1 -- 172.21.15.134:0/3761730852 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f141804ee20 con 0x7f14080057d0 2024-04-25T01:46:23.823 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-25T01:46:23.823 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":3,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","modified":"2024-04-25T01:46:18.002062Z","created":"2024-04-25T01:44:14.223274Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T01:46:23.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 -- 172.21.15.134:0/3761730852 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1400039580 msgr2=0x7f140003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:23.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1400039580 0x7f140003ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f141c196cf0 tx=0x7f140c008040 comp rx=0 tx=0).stop 2024-04-25T01:46:23.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 -- 172.21.15.134:0/3761730852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14080057d0 msgr2=0x7f141c195530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:23.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14080057d0 0x7f141c195530 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f141c100b60 tx=0x7f14180054f0 comp rx=0 tx=0).stop 2024-04-25T01:46:23.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 -- 172.21.15.134:0/3761730852 shutdown_connections 2024-04-25T01:46:23.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1400039580 0x7f140003ba40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:23.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141c1960c0 0x7f141c1aa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:23.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14080057d0 0x7f141c195530 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:23.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 --2- 172.21.15.134:0/3761730852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f141c199490 0x7f141c195a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:23.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 -- 172.21.15.134:0/3761730852 >> 172.21.15.134:0/3761730852 conn(0x7f141c0fc760 msgr2=0x7f141c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:23.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 -- 172.21.15.134:0/3761730852 shutdown_connections 2024-04-25T01:46:23.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:23.831+0000 7f14243b4700 1 -- 172.21.15.134:0/3761730852 wait complete. 2024-04-25T01:46:23.826 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 3 2024-04-25T01:46:24.154 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:24.155 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:24.155 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:24 smithi134 ceph-mon[33098]: from='client.? 172.21.15.134:0/3761730852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:46:24.155 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:24.441 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T01:46:24.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T01:46:24.468 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:24.469 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:24.469 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:24 smithi031 ceph-mon[29204]: from='client.? 172.21.15.134:0/3761730852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:46:24.469 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:24.698 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:46:25.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:25.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:46:25.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: mon.a calling monitor election 2024-04-25T01:46:25.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: mon.c calling monitor election 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: monmap epoch 3 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: last_changed 2024-04-25T01:46:18.002062+0000 2024-04-25T01:46:25.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: created 2024-04-25T01:44:14.223274+0000 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: min_mon_release 19 (squid) 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: election_strategy: 1 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.c 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: fsmap 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: mgrmap e13: x(active, since 59s) 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: overall HEALTH_OK 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:25.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='client.? 172.21.15.134:0/3761730852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:46:25.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:25.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: mon.b calling monitor election 2024-04-25T01:46:25.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: mon.b calling monitor election 2024-04-25T01:46:25.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: mon.a calling monitor election 2024-04-25T01:46:25.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: mon.c calling monitor election 2024-04-25T01:46:25.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:46:25.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: monmap epoch 3 2024-04-25T01:46:25.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:46:25.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: last_changed 2024-04-25T01:46:18.002062+0000 2024-04-25T01:46:25.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: created 2024-04-25T01:44:14.223274+0000 2024-04-25T01:46:25.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: min_mon_release 19 (squid) 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: election_strategy: 1 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.c 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: fsmap 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: mgrmap e13: x(active, since 61s) 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: overall HEALTH_OK 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:25.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:25.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: mon.b calling monitor election 2024-04-25T01:46:25.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: mon.b calling monitor election 2024-04-25T01:46:25.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: mon.a calling monitor election 2024-04-25T01:46:25.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: mon.c calling monitor election 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: monmap epoch 3 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: last_changed 2024-04-25T01:46:18.002062+0000 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: created 2024-04-25T01:44:14.223274+0000 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: min_mon_release 19 (squid) 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: election_strategy: 1 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.c 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: fsmap 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: mgrmap e13: x(active, since 61s) 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: overall HEALTH_OK 2024-04-25T01:46:25.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:25.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:25.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:25.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:25.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: mon.b calling monitor election 2024-04-25T01:46:25.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: mon.b calling monitor election 2024-04-25T01:46:25.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: mon.a calling monitor election 2024-04-25T01:46:25.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: mon.c calling monitor election 2024-04-25T01:46:25.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: monmap epoch 3 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: last_changed 2024-04-25T01:46:18.002062+0000 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: created 2024-04-25T01:44:14.223274+0000 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: min_mon_release 19 (squid) 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: election_strategy: 1 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.c 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: 2: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: fsmap 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: mgrmap e13: x(active, since 61s) 2024-04-25T01:46:25.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: overall HEALTH_OK 2024-04-25T01:46:25.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:25.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:25.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:25.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:26.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.377+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/2124856601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce0107cd0 msgr2=0x7f1ce01080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:26.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.377+0000 7f1ce5ee3700 1 --2- 172.21.15.31:0/2124856601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce0107cd0 0x7f1ce01080b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009910 tx=0x7f1cd0009c20 comp rx=0 tx=0).stop 2024-04-25T01:46:26.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.377+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/2124856601 shutdown_connections 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.377+0000 7f1ce5ee3700 1 --2- 172.21.15.31:0/2124856601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce0107cd0 0x7f1ce01080b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.377+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/2124856601 >> 172.21.15.31:0/2124856601 conn(0x7f1ce0076690 msgr2=0x7f1ce0076aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.377+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/2124856601 shutdown_connections 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.377+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/2124856601 wait complete. 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1ce5ee3700 1 Processor -- start 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1ce5ee3700 1 -- start start 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1ce5ee3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ce0107cd0 0x7f1ce0100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1ce5ee3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce0101190 0x7f1ce0101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1ce5ee3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce01a3ab0 0x7f1ce01a5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1ce5ee3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce010c110 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1ce5ee3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce010bf90 con 0x7f1ce0107cd0 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1ce5ee3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce010c290 con 0x7f1ce0101190 2024-04-25T01:46:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1cdffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce01a3ab0 0x7f1ce01a5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1cdffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce01a3ab0 0x7f1ce01a5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56622/0 (socket says 172.21.15.31:56622) 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1cdffff700 1 -- 172.21.15.31:0/655389720 learned_addr learned my addr 172.21.15.31:0/655389720 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1cdf7fe700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ce0107cd0 0x7f1ce0100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.378+0000 7f1cdeffd700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce0101190 0x7f1ce0101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.379+0000 7f1cdffff700 1 -- 172.21.15.31:0/655389720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce0101190 msgr2=0x7f1ce0101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.379+0000 7f1cdffff700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce0101190 0x7f1ce0101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.379+0000 7f1cdffff700 1 -- 172.21.15.31:0/655389720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ce0107cd0 msgr2=0x7f1ce0100c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.379+0000 7f1cdffff700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ce0107cd0 0x7f1ce0100c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.379+0000 7f1cdffff700 1 -- 172.21.15.31:0/655389720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce01a65d0 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.379+0000 7f1cdffff700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce01a3ab0 0x7f1ce01a5ea0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800ba50 tx=0x7f1cd800bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.379+0000 7f1cdcff9700 1 -- 172.21.15.31:0/655389720 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8019070 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.379+0000 7f1cdcff9700 1 -- 172.21.15.31:0/655389720 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cd800cde0 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.379+0000 7f1cdcff9700 1 -- 172.21.15.31:0/655389720 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8007350 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.379+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/655389720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce01a68c0 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.379+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/655389720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce01a6d50 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.380+0000 7f1cdcff9700 1 -- 172.21.15.31:0/655389720 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1cd801c070 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.380+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/655389720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ce0063890 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.380+0000 7f1cdcff9700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1ccc039580 0x7f1ccc03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:26.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.380+0000 7f1cdcff9700 1 -- 172.21.15.31:0/655389720 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1cd804b720 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.380+0000 7f1cdf7fe700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1ccc039580 0x7f1ccc03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:26.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.381+0000 7f1cdf7fe700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1ccc039580 0x7f1ccc03ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0006010 tx=0x7f1cd001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:26.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.385+0000 7f1cdcff9700 1 -- 172.21.15.31:0/655389720 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1cd80135e0 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.598+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/655389720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f1ce01a7220 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.598+0000 7f1cdcff9700 1 -- 172.21.15.31:0/655389720 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f1cd80074f0 con 0x7f1ce01a3ab0 2024-04-25T01:46:26.593 INFO:teuthology.orchestra.run.smithi031.stdout:# minimal ceph.conf for 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:46:26.593 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-04-25T01:46:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:46:26.594 INFO:teuthology.orchestra.run.smithi031.stdout: mon_host = [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-04-25T01:46:26.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/655389720 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1ccc039580 msgr2=0x7f1ccc03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:26.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1ccc039580 0x7f1ccc03ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0006010 tx=0x7f1cd001a040 comp rx=0 tx=0).stop 2024-04-25T01:46:26.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/655389720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce01a3ab0 msgr2=0x7f1ce01a5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:26.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce01a3ab0 0x7f1ce01a5ea0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800ba50 tx=0x7f1cd800bd60 comp rx=0 tx=0).stop 2024-04-25T01:46:26.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/655389720 shutdown_connections 2024-04-25T01:46:26.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1ccc039580 0x7f1ccc03ba40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:26.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ce01a3ab0 0x7f1ce01a5ea0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:26.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ce0107cd0 0x7f1ce0100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:26.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 --2- 172.21.15.31:0/655389720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce0101190 0x7f1ce0101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:26.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/655389720 >> 172.21.15.31:0/655389720 conn(0x7f1ce0076690 msgr2=0x7f1ce00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:26.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/655389720 shutdown_connections 2024-04-25T01:46:26.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:26.600+0000 7f1ce5ee3700 1 -- 172.21.15.31:0/655389720 wait complete. 2024-04-25T01:46:26.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: Updating smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:26.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:46:26.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:26.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:26.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-25T01:46:26.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-25T01:46:26.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-25T01:46:26.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: Updating smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:26.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:26.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:26.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:26.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:46:26.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:26.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:26.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-25T01:46:26.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-25T01:46:26.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-25T01:46:26.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: Updating smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:26.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:46:26.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:26.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:46:27.608 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:27 smithi031 ceph-mon[29204]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:27.608 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:27 smithi031 ceph-mon[29204]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:46:27.608 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:27 smithi031 ceph-mon[29204]: Reconfiguring daemon mon.a on smithi031 2024-04-25T01:46:27.608 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:27 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/655389720' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:27.609 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T01:46:27.609 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:46:27.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:46:27.678 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:46:27.679 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:46:27.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:27 smithi134 ceph-mon[33098]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:27.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:27 smithi134 ceph-mon[33098]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:46:27.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:27 smithi134 ceph-mon[33098]: Reconfiguring daemon mon.a on smithi031 2024-04-25T01:46:27.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:27 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/655389720' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:27.744 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-25T01:46:27.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:46:27.772 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:27 smithi040 ceph-mon[32418]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:27.772 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:27 smithi040 ceph-mon[32418]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:46:27.772 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:27 smithi040 ceph-mon[32418]: Reconfiguring daemon mon.a on smithi031 2024-04-25T01:46:27.772 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:27 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/655389720' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:27.778 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-25T01:46:27.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:46:27.846 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T01:46:27.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:46:27.879 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T01:46:27.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:46:27.946 INFO:tasks.cephadm:Adding mgr.x on smithi031 2024-04-25T01:46:27.946 INFO:tasks.cephadm:Adding mgr.y on smithi040 2024-04-25T01:46:27.946 INFO:tasks.cephadm:Adding mgr.z on smithi134 2024-04-25T01:46:27.946 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi031=x;smithi040=y;smithi134=z' 2024-04-25T01:46:28.220 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:46:29.231 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:29 smithi031 ceph-mon[29204]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:29.231 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:29.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:29.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:29 smithi031 ceph-mon[29204]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:46:29.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:29.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:46:29.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:29.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:29 smithi031 ceph-mon[29204]: Reconfiguring daemon mon.b on smithi040 2024-04-25T01:46:29.320 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:29 smithi134 ceph-mon[33098]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:29.320 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:29.321 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:29.321 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:29 smithi134 ceph-mon[33098]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:46:29.321 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:29.321 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:46:29.321 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:29.321 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:29 smithi134 ceph-mon[33098]: Reconfiguring daemon mon.b on smithi040 2024-04-25T01:46:29.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:29 smithi040 ceph-mon[32418]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:29.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:29.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:29.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:29 smithi040 ceph-mon[32418]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:46:29.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:29.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:46:29.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:29.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:29 smithi040 ceph-mon[32418]: Reconfiguring daemon mon.b on smithi040 2024-04-25T01:46:29.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.786+0000 7fa477c61700 1 -- 172.21.15.134:0/1853996987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa470100ca0 msgr2=0x7fa4701010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:29.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.786+0000 7fa477c61700 1 --2- 172.21.15.134:0/1853996987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa470100ca0 0x7fa4701010c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009910 tx=0x7fa46c009c20 comp rx=0 tx=0).stop 2024-04-25T01:46:29.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.786+0000 7fa477c61700 1 -- 172.21.15.134:0/1853996987 shutdown_connections 2024-04-25T01:46:29.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.786+0000 7fa477c61700 1 --2- 172.21.15.134:0/1853996987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa470100ca0 0x7fa4701010c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:29.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.786+0000 7fa477c61700 1 -- 172.21.15.134:0/1853996987 >> 172.21.15.134:0/1853996987 conn(0x7fa4700fc240 msgr2=0x7fa4700fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.786+0000 7fa477c61700 1 -- 172.21.15.134:0/1853996987 shutdown_connections 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.786+0000 7fa477c61700 1 -- 172.21.15.134:0/1853996987 wait complete. 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa477c61700 1 Processor -- start 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa477c61700 1 -- start start 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa477c61700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa470100ca0 0x7fa470196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa477c61700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa470196940 0x7fa47019b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa477c61700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa470196dc0 0x7fa47019df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa477c61700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa470106be0 con 0x7fa470196dc0 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa477c61700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa470106a60 con 0x7fa470100ca0 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa477c61700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa470106d60 con 0x7fa470196940 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa4751fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa470196940 0x7fa47019b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa4751fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa470196940 0x7fa47019b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:44402/0 (socket says 172.21.15.134:44402) 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa4751fc700 1 -- 172.21.15.134:0/380370685 learned_addr learned my addr 172.21.15.134:0/380370685 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-25T01:46:29.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa4761fe700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa470196dc0 0x7fa47019df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa4759fd700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa470100ca0 0x7fa470196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa4751fc700 1 -- 172.21.15.134:0/380370685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa470100ca0 msgr2=0x7fa470196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa4751fc700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa470100ca0 0x7fa470196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa4751fc700 1 -- 172.21.15.134:0/380370685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa470196dc0 msgr2=0x7fa47019df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa4751fc700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa470196dc0 0x7fa47019df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.787+0000 7fa4751fc700 1 -- 172.21.15.134:0/380370685 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47019e810 con 0x7fa470196940 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.788+0000 7fa4751fc700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa470196940 0x7fa47019b9b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa46000b780 tx=0x7fa46000ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.788+0000 7fa466ffd700 1 -- 172.21.15.134:0/380370685 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa460004230 con 0x7fa470196940 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.788+0000 7fa466ffd700 1 -- 172.21.15.134:0/380370685 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4600043d0 con 0x7fa470196940 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.788+0000 7fa477c61700 1 -- 172.21.15.134:0/380370685 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa47006a4a0 con 0x7fa470196940 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.788+0000 7fa466ffd700 1 -- 172.21.15.134:0/380370685 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa460011b60 con 0x7fa470196940 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.788+0000 7fa477c61700 1 -- 172.21.15.134:0/380370685 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa47006aab0 con 0x7fa470196940 2024-04-25T01:46:29.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.789+0000 7fa466ffd700 1 -- 172.21.15.134:0/380370685 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fa460011d00 con 0x7fa470196940 2024-04-25T01:46:29.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.789+0000 7fa466ffd700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa45c039580 0x7fa45c03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:29.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.789+0000 7fa466ffd700 1 -- 172.21.15.134:0/380370685 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa46004bcc0 con 0x7fa470196940 2024-04-25T01:46:29.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.789+0000 7fa477c61700 1 -- 172.21.15.134:0/380370685 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa470063890 con 0x7fa470196940 2024-04-25T01:46:29.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.789+0000 7fa4759fd700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa45c039580 0x7fa45c03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:29.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.790+0000 7fa4759fd700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa45c039580 0x7fa45c03ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00b5c0 tx=0x7fa46c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:29.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:29.794+0000 7fa466ffd700 1 -- 172.21.15.134:0/380370685 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa460020500 con 0x7fa470196940 2024-04-25T01:46:30.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.016+0000 7fa477c61700 1 -- 172.21.15.134:0/380370685 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi040=y;smithi134=z", "target": ["mon-mgr", ""]}) -- 0x7fa47006ad70 con 0x7fa45c039580 2024-04-25T01:46:30.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.025+0000 7fa466ffd700 1 -- 172.21.15.134:0/380370685 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa47006ad70 con 0x7fa45c039580 2024-04-25T01:46:30.020 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled mgr update... 2024-04-25T01:46:30.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 -- 172.21.15.134:0/380370685 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa45c039580 msgr2=0x7fa45c03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:30.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa45c039580 0x7fa45c03ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00b5c0 tx=0x7fa46c01a040 comp rx=0 tx=0).stop 2024-04-25T01:46:30.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 -- 172.21.15.134:0/380370685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa470196940 msgr2=0x7fa47019b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:30.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa470196940 0x7fa47019b9b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa46000b780 tx=0x7fa46000ba90 comp rx=0 tx=0).stop 2024-04-25T01:46:30.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 -- 172.21.15.134:0/380370685 shutdown_connections 2024-04-25T01:46:30.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa45c039580 0x7fa45c03ba40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:30.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa470196dc0 0x7fa47019df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:30.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa470100ca0 0x7fa470196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:30.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 --2- 172.21.15.134:0/380370685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa470196940 0x7fa47019b9b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:30.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 -- 172.21.15.134:0/380370685 >> 172.21.15.134:0/380370685 conn(0x7fa4700fc240 msgr2=0x7fa4700fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:30.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 -- 172.21.15.134:0/380370685 shutdown_connections 2024-04-25T01:46:30.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:46:30.027+0000 7fa477c61700 1 -- 172.21.15.134:0/380370685 wait complete. 2024-04-25T01:46:30.526 DEBUG:teuthology.orchestra.run.smithi040:mgr.y> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T01:46:30.529 DEBUG:teuthology.orchestra.run.smithi134:mgr.z> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T01:46:30.531 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T01:46:30.531 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:46:30.531 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:46:30.551 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:46:30.551 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-04-25T01:46:30.610 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:46:30.610 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:30.611 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T01:46:30.611 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:30.611 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:30.611 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-25 01:45:36.361782912 +0000 2024-04-25T01:46:30.611 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-25 01:40:51.013417895 +0000 2024-04-25T01:46:30.611 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-25 01:40:51.013417895 +0000 2024-04-25T01:46:30.611 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-25 01:40:51.013417895 +0000 2024-04-25T01:46:30.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:46:30.680 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-25T01:46:30.681 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-25T01:46:30.681 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.0002906 s, 1.8 MB/s 2024-04-25T01:46:30.682 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:46:30.741 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-04-25T01:46:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:46:30.800 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:30.800 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-25T01:46:30.800 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:30.800 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:30.801 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-25 01:45:36.392782420 +0000 2024-04-25T01:46:30.801 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-25 01:40:51.351412473 +0000 2024-04-25T01:46:30.801 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-25 01:40:51.351412473 +0000 2024-04-25T01:46:30.801 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-25 01:40:51.351412473 +0000 2024-04-25T01:46:30.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:46:30.847 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:30 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:30.847 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:30 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:30.847 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:30 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:30.847 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:30 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:46:30.848 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:30 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:30.848 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:30 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:30.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:30 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:30.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:30 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:30.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:30 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:30.865 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:30 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:46:30.865 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:30 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:30.865 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:30 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:30.867 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-25T01:46:30.867 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-25T01:46:30.867 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000268002 s, 1.9 MB/s 2024-04-25T01:46:30.869 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:46:30.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:30 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:30.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:30 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:30.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:30 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:46:30.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:30 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:46:30.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:30 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:30.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:30 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:30.928 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-04-25T01:46:30.986 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:46:30.986 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:30.986 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-25T01:46:30.986 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:30.986 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:30.986 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-25 01:45:36.419781992 +0000 2024-04-25T01:46:30.986 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-25 01:40:51.682407163 +0000 2024-04-25T01:46:30.986 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-25 01:40:51.682407163 +0000 2024-04-25T01:46:30.986 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-25 01:40:51.682407163 +0000 2024-04-25T01:46:30.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:46:31.053 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-25T01:46:31.053 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-25T01:46:31.053 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.00019094 s, 2.7 MB/s 2024-04-25T01:46:31.054 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:46:31.115 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-04-25T01:46:31.178 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:46:31.179 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:31.179 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T01:46:31.179 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:31.179 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:31.179 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-25 01:45:36.445781580 +0000 2024-04-25T01:46:31.179 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-25 01:40:52.031401564 +0000 2024-04-25T01:46:31.179 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-25 01:40:52.031401564 +0000 2024-04-25T01:46:31.179 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-25 01:40:52.031401564 +0000 2024-04-25T01:46:31.180 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:46:31.247 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-25T01:46:31.247 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-25T01:46:31.247 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000187654 s, 2.7 MB/s 2024-04-25T01:46:31.249 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:46:31.317 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-25T01:46:31.317 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:46:31.340 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:46:31.340 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-04-25T01:46:31.405 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:46:31.405 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:31.406 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-25T01:46:31.406 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:31.406 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:31.406 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-25 01:46:17.453681968 +0000 2024-04-25T01:46:31.406 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-25 01:40:53.136658070 +0000 2024-04-25T01:46:31.406 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-25 01:40:53.136658070 +0000 2024-04-25T01:46:31.406 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-25 01:40:53.136658070 +0000 2024-04-25T01:46:31.407 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:46:31.476 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-25T01:46:31.477 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-25T01:46:31.477 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000372926 s, 1.4 MB/s 2024-04-25T01:46:31.478 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:46:31.539 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-04-25T01:46:31.606 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:46:31.606 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:31.606 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-25T01:46:31.606 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:31.607 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:31.607 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-25 01:46:17.453681968 +0000 2024-04-25T01:46:31.607 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-25 01:40:53.480653632 +0000 2024-04-25T01:46:31.607 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-25 01:40:53.480653632 +0000 2024-04-25T01:46:31.607 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-25 01:40:53.480653632 +0000 2024-04-25T01:46:31.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:46:31.677 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-25T01:46:31.677 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-25T01:46:31.677 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.00020456 s, 2.5 MB/s 2024-04-25T01:46:31.679 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:46:31.741 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-04-25T01:46:31.802 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:46:31.802 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:31.802 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T01:46:31.802 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:31.802 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:31.802 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-25 01:46:17.453681968 +0000 2024-04-25T01:46:31.802 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-25 01:40:53.829649129 +0000 2024-04-25T01:46:31.802 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-25 01:40:53.829649129 +0000 2024-04-25T01:46:31.802 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-25 01:40:53.829649129 +0000 2024-04-25T01:46:31.803 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:46:31.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:31 smithi040 ceph-mon[32418]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:46:31.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:31 smithi040 ceph-mon[32418]: Reconfiguring daemon mon.c on smithi134 2024-04-25T01:46:31.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:31 smithi040 ceph-mon[32418]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:31.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:31 smithi040 ceph-mon[32418]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi040=y;smithi134=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:31.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:31 smithi040 ceph-mon[32418]: Saving service mgr spec with placement smithi031=x;smithi040=y;smithi134=z;count:3 2024-04-25T01:46:31.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:31 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:31.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:31 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:31.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:31 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-25T01:46:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-25T01:46:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000220859 s, 2.3 MB/s 2024-04-25T01:46:31.873 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:46:31.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:31 smithi134 ceph-mon[33098]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:46:31.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:31 smithi134 ceph-mon[33098]: Reconfiguring daemon mon.c on smithi134 2024-04-25T01:46:31.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:31 smithi134 ceph-mon[33098]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:31.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:31 smithi134 ceph-mon[33098]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi040=y;smithi134=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:31.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:31 smithi134 ceph-mon[33098]: Saving service mgr spec with placement smithi031=x;smithi040=y;smithi134=z;count:3 2024-04-25T01:46:31.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:31 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:31.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:31 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:31.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:31 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:31.934 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-04-25T01:46:31.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:31 smithi031 ceph-mon[29204]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:46:31.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:31 smithi031 ceph-mon[29204]: Reconfiguring daemon mon.c on smithi134 2024-04-25T01:46:31.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:31 smithi031 ceph-mon[29204]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:31.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:31 smithi031 ceph-mon[29204]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi040=y;smithi134=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:31.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:31 smithi031 ceph-mon[29204]: Saving service mgr spec with placement smithi031=x;smithi040=y;smithi134=z;count:3 2024-04-25T01:46:31.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:31 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:31.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:31 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:31.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:31 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:31.993 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:46:31.993 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:31.993 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-25T01:46:31.993 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:31.993 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:31.993 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-25 01:46:17.454681956 +0000 2024-04-25T01:46:31.993 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-25 01:40:54.178644627 +0000 2024-04-25T01:46:31.993 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-25 01:40:54.178644627 +0000 2024-04-25T01:46:31.993 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-25 01:40:54.178644627 +0000 2024-04-25T01:46:31.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:46:32.060 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-25T01:46:32.060 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-25T01:46:32.060 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000208379 s, 2.5 MB/s 2024-04-25T01:46:32.062 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:46:32.120 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T01:46:32.120 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:46:32.138 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:46:32.138 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-04-25T01:46:32.194 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:46:32.195 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:32.195 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T01:46:32.195 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:32.195 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:32.195 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-25 01:46:08.070052616 +0000 2024-04-25T01:46:32.195 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-25 01:40:47.702064824 +0000 2024-04-25T01:46:32.195 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-25 01:40:47.702064824 +0000 2024-04-25T01:46:32.195 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-04-25 01:40:47.702064824 +0000 2024-04-25T01:46:32.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:46:32.259 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-25T01:46:32.259 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-25T01:46:32.259 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000216825 s, 2.4 MB/s 2024-04-25T01:46:32.260 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:46:32.316 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-04-25T01:46:32.371 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:46:32.372 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:32.372 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-25T01:46:32.372 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:32.372 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:32.372 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-25 01:46:08.070052616 +0000 2024-04-25T01:46:32.372 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-25 01:40:48.049056784 +0000 2024-04-25T01:46:32.373 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-25 01:40:48.049056784 +0000 2024-04-25T01:46:32.373 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-04-25 01:40:48.049056784 +0000 2024-04-25T01:46:32.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:46:32.434 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-25T01:46:32.434 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-25T01:46:32.434 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000206819 s, 2.5 MB/s 2024-04-25T01:46:32.435 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:46:32.491 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-04-25T01:46:32.548 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:46:32.548 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:32.548 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-25T01:46:32.548 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:32.548 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:32.548 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-25 01:46:08.070052616 +0000 2024-04-25T01:46:32.548 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-25 01:40:48.382049068 +0000 2024-04-25T01:46:32.548 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-25 01:40:48.382049068 +0000 2024-04-25T01:46:32.549 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-04-25 01:40:48.382049068 +0000 2024-04-25T01:46:32.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:46:32.614 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-25T01:46:32.614 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-25T01:46:32.614 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000176715 s, 2.9 MB/s 2024-04-25T01:46:32.615 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:46:32.671 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-04-25T01:46:32.726 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:46:32.726 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:32.726 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-25T01:46:32.726 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:32.726 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:32.726 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-25 01:46:08.071052594 +0000 2024-04-25T01:46:32.726 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-25 01:40:48.721041213 +0000 2024-04-25T01:46:32.726 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-25 01:40:48.721041213 +0000 2024-04-25T01:46:32.726 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-04-25 01:40:48.721041213 +0000 2024-04-25T01:46:32.727 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:46:32.790 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-25T01:46:32.790 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-25T01:46:32.790 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000207752 s, 2.5 MB/s 2024-04-25T01:46:32.791 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:46:32.847 INFO:tasks.cephadm:Deploying osd.0 on smithi031 with /dev/vg_nvme/lv_4... 2024-04-25T01:46:32.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:46:32.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:32.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:32.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:32.873 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:32.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:32.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:32.874 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:32.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:32.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:32.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:32.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:32.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:32.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:32.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:32.907 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:32.907 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:32.907 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:32.908 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:32.908 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:32.908 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:32.908 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:33.100 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:46:33.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:33 smithi040 ceph-mon[32418]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:33.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:33 smithi040 ceph-mon[32418]: Deploying daemon mgr.z on smithi134 2024-04-25T01:46:33.932 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:33 smithi134 ceph-mon[33098]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:33.932 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:33 smithi134 ceph-mon[33098]: Deploying daemon mgr.z on smithi134 2024-04-25T01:46:33.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:33 smithi031 ceph-mon[29204]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:33.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:33 smithi031 ceph-mon[29204]: Deploying daemon mgr.z on smithi134 2024-04-25T01:46:34.367 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:34 smithi134 systemd[1]: Starting Ceph mgr.z for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:46:34.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:34 smithi134 ceph-mon[33098]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:34.644 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:34 smithi134 podman[34940]: 2024-04-25 01:46:34.373116962 +0000 UTC m=+0.111754788 container create 9bee8c413a8c21ec1e3e6ec280a99124f5c3b1069d9b11af7b2091398ea42d08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T01:46:34.644 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:34 smithi134 podman[34940]: 2024-04-25 01:46:34.279152733 +0000 UTC m=+0.017790573 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:46:34.644 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:34 smithi134 podman[34940]: 2024-04-25 01:46:34.57543578 +0000 UTC m=+0.314073606 container init 9bee8c413a8c21ec1e3e6ec280a99124f5c3b1069d9b11af7b2091398ea42d08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-25T01:46:34.644 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:34 smithi134 podman[34940]: 2024-04-25 01:46:34.577986206 +0000 UTC m=+0.316624039 container start 9bee8c413a8c21ec1e3e6ec280a99124f5c3b1069d9b11af7b2091398ea42d08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:46:34.644 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:34 smithi134 ceph-mgr[34959]: -- 172.21.15.134:0/3912460318 <== mon.1 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559e71c365a0 con 0x559e71c0b000 2024-04-25T01:46:34.644 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:34 smithi134 bash[34940]: 9bee8c413a8c21ec1e3e6ec280a99124f5c3b1069d9b11af7b2091398ea42d08 2024-04-25T01:46:34.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:34 smithi040 ceph-mon[32418]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:34.911 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:34 smithi134 systemd[1]: Started Ceph mgr.z for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:46:34.911 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:34.731+0000 7f91f0009200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:46:34.911 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:34.916+0000 7f91f0009200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:46:34.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:34 smithi031 ceph-mon[29204]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:34.947 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:46:34.947 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:46:34.947 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-25T01:46:34.947 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-25T01:46:34.947 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149909 s, 699 MB/s 2024-04-25T01:46:34.947 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-25T01:46:35.182 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:35 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:35.016+0000 7f91f0009200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:46:35.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_4 2024-04-25T01:46:35.682 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:35 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:35.321+0000 7f91f0009200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:46:35.682 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:35 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:35.404+0000 7f91f0009200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:46:35.853 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:46:35.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:35 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:35.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:35 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:35.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:35 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:35.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:35 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:35.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:35 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:35.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:35 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:35.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:35 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:35.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:35 smithi031 ceph-mon[29204]: Deploying daemon mgr.y on smithi040 2024-04-25T01:46:35.999 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:35 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:35.999 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:35 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:35.999 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:35 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:35.999 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:35 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:35.999 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:35 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:35.999 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:35 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:35.999 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:35 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:35.999 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:35 smithi134 ceph-mon[33098]: Deploying daemon mgr.y on smithi040 2024-04-25T01:46:36.153 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:35 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:36.153 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:35 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:36.153 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:35 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:36.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:35 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:36.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:35 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:36.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:35 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:36.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:35 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:36.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:35 smithi040 ceph-mon[32418]: Deploying daemon mgr.y on smithi040 2024-04-25T01:46:36.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:36 smithi040 ceph-mon[32418]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:36.932 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:36 smithi134 ceph-mon[33098]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:36.932 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:36 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:36.646+0000 7f91f0009200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:46:36.933 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:36 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:36.819+0000 7f91f0009200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:46:37.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:36 smithi031 ceph-mon[29204]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:37.200 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:37 smithi040 ceph-mgr[33800]: -- 172.21.15.40:0/1060198371 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558da468c5a0 con 0x558da4660c00 2024-04-25T01:46:37.200 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:37 smithi040 bash[33779]: b66e1f6b463d956a929cca7bc5ff887448b026a370000fbca5bd8a5dba184246 2024-04-25T01:46:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.272+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2889116398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f18103e30 msgr2=0x7f2f1810a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.272+0000 7f2f1c96d700 1 --2- 172.21.15.31:0/2889116398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f18103e30 0x7f2f1810a6e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000b0d0 tx=0x7f2f1000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:46:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.272+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2889116398 shutdown_connections 2024-04-25T01:46:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.272+0000 7f2f1c96d700 1 --2- 172.21.15.31:0/2889116398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f18103e30 0x7f2f1810a6e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.272+0000 7f2f1c96d700 1 --2- 172.21.15.31:0/2889116398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f18102290 0x7f2f181026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.272+0000 7f2f1c96d700 1 --2- 172.21.15.31:0/2889116398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f18103490 0x7f2f181038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.272+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2889116398 >> 172.21.15.31:0/2889116398 conn(0x7f2f180fd830 msgr2=0x7f2f180ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.272+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2889116398 shutdown_connections 2024-04-25T01:46:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.272+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2889116398 wait complete. 2024-04-25T01:46:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.272+0000 7f2f1c96d700 1 Processor -- start 2024-04-25T01:46:37.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1c96d700 1 -- start start 2024-04-25T01:46:37.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1c96d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f18102290 0x7f2f18198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:37.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1c96d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f18103490 0x7f2f18198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:37.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1c96d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f18103e30 0x7f2f1819f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:37.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1c96d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1810dd00 con 0x7f2f18102290 2024-04-25T01:46:37.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1c96d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1810db80 con 0x7f2f18103e30 2024-04-25T01:46:37.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1c96d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1810de80 con 0x7f2f18103490 2024-04-25T01:46:37.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f18102290 0x7f2f18198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:37.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f18102290 0x7f2f18198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59028/0 (socket says 172.21.15.31:59028) 2024-04-25T01:46:37.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1659c700 1 -- 172.21.15.31:0/2606380781 learned_addr learned my addr 172.21.15.31:0/2606380781 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:46:37.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f15d9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f18103490 0x7f2f18198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f16d9d700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f18103e30 0x7f2f1819f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1659c700 1 -- 172.21.15.31:0/2606380781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f18103490 msgr2=0x7f2f18198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1659c700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f18103490 0x7f2f18198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1659c700 1 -- 172.21.15.31:0/2606380781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f18103e30 msgr2=0x7f2f1819f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1659c700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f18103e30 0x7f2f1819f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1659c700 1 -- 172.21.15.31:0/2606380781 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1819ffa0 con 0x7f2f18102290 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.273+0000 7f2f1659c700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f18102290 0x7f2f18198340 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800d9b0 tx=0x7f2f0800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.274+0000 7f2f077fe700 1 -- 172.21.15.31:0/2606380781 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08016070 con 0x7f2f18102290 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.274+0000 7f2f077fe700 1 -- 172.21.15.31:0/2606380781 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f0800be00 con 0x7f2f18102290 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.274+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2606380781 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f181a0290 con 0x7f2f18102290 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.274+0000 7f2f077fe700 1 -- 172.21.15.31:0/2606380781 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08010e30 con 0x7f2f18102290 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.274+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2606380781 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f181a0820 con 0x7f2f18102290 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.275+0000 7f2f077fe700 1 -- 172.21.15.31:0/2606380781 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f2f08019070 con 0x7f2f18102290 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.275+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2606380781 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f18068f20 con 0x7f2f18102290 2024-04-25T01:46:37.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.275+0000 7f2f077fe700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2f00039330 0x7f2f0003b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:37.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.275+0000 7f2f077fe700 1 -- 172.21.15.31:0/2606380781 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2f0804be40 con 0x7f2f18102290 2024-04-25T01:46:37.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.275+0000 7f2f15d9b700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2f00039330 0x7f2f0003b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:37.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.275+0000 7f2f15d9b700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2f00039330 0x7f2f0003b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009be0 tx=0x7f2f0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:37.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.280+0000 7f2f077fe700 1 -- 172.21.15.31:0/2606380781 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2f08020030 con 0x7f2f18102290 2024-04-25T01:46:37.278 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:36 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:36.997+0000 7f91f0009200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:46:37.478 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:37 smithi040 systemd[1]: Started Ceph mgr.y for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:46:37.478 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:37 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:37.302+0000 7f8966348200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:46:37.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:37.502+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2606380781 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2f18062670 con 0x7f2f00039330 2024-04-25T01:46:37.579 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:37 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:37.283+0000 7f91f0009200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:46:37.579 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:37 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:37.376+0000 7f91f0009200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:46:37.579 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:37 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:37.584+0000 7f91f0009200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:46:37.903 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:37 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:37.500+0000 7f8966348200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:46:37.903 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:37 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:37.611+0000 7f8966348200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:46:37.932 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:37 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:37.785+0000 7f91f0009200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:46:38.280 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:37 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:37.947+0000 7f8966348200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:46:38.280 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:38 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:38.051+0000 7f8966348200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:46:38.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:38 smithi031 ceph-mon[29204]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:38.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:46:38.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:46:38.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:38.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:38.646 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.646 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.646 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.646 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.647 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:38 smithi040 ceph-mon[32418]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:38.647 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:46:38.647 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:46:38.647 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:38.647 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:38.682 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:38.418+0000 7f91f0009200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:46:38.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:38.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-mon[33098]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:38.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:46:38.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:46:38.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:38.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:39.135 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:38.846+0000 7f91f0009200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:46:39.136 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:38 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:38.942+0000 7f91f0009200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:46:39.136 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:39.140+0000 7f91f0009200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:46:39.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:39.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:39.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:39.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:39.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:39.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:39.653 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:39.382+0000 7f8966348200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:46:39.653 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:39.557+0000 7f8966348200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:46:39.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:39.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:39.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:39.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:39.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:39.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:39.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:39.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:39.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:39.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:39.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:39.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:39.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:39.932 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:39.666+0000 7f91f0009200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:46:39.933 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:39 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:39.759+0000 7f91f0009200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:46:40.024 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:39 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:39.732+0000 7f8966348200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:46:40.024 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:40 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:40.029+0000 7f8966348200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:46:40.328 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:40 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:40.124+0000 7f8966348200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:46:40.328 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:40 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:40.333+0000 7f8966348200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:46:40.339 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:40 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:39.999+0000 7f91f0009200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:46:40.339 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:40 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:40.131+0000 7f91f0009200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:46:40.339 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:40 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:40.226+0000 7f91f0009200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:46:40.339 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:40 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:40.342+0000 7f91f0009200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:46:40.653 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:40 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:40.531+0000 7f8966348200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:46:40.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:40 smithi040 ceph-mon[32418]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:46:40.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:40 smithi040 ceph-mon[32418]: Reconfiguring daemon mgr.x on smithi031 2024-04-25T01:46:40.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:40 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1921439715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b99beda-ae8b-4e4b-8569-1c3b5f44654b"}]: dispatch 2024-04-25T01:46:40.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:40 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1921439715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b99beda-ae8b-4e4b-8569-1c3b5f44654b"}]': finished 2024-04-25T01:46:40.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:40 smithi040 ceph-mon[32418]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:46:40.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:40 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:40.682 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:40 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:40.434+0000 7f91f0009200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:46:40.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:40 smithi134 ceph-mon[33098]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:46:40.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:40 smithi134 ceph-mon[33098]: Reconfiguring daemon mgr.x on smithi031 2024-04-25T01:46:40.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:40 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1921439715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b99beda-ae8b-4e4b-8569-1c3b5f44654b"}]: dispatch 2024-04-25T01:46:40.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:40 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1921439715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b99beda-ae8b-4e4b-8569-1c3b5f44654b"}]': finished 2024-04-25T01:46:40.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:40 smithi134 ceph-mon[33098]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:46:40.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:40 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:40.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:40 smithi031 ceph-mon[29204]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:46:40.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:40 smithi031 ceph-mon[29204]: Reconfiguring daemon mgr.x on smithi031 2024-04-25T01:46:40.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:40 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1921439715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b99beda-ae8b-4e4b-8569-1c3b5f44654b"}]: dispatch 2024-04-25T01:46:40.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:40 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1921439715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b99beda-ae8b-4e4b-8569-1c3b5f44654b"}]': finished 2024-04-25T01:46:40.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:40 smithi031 ceph-mon[29204]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:46:40.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:40 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:41.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:41 smithi040 ceph-mon[32418]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:41.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:41 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/4207012447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:46:41.403 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:41 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:41.155+0000 7f8966348200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:46:41.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:41 smithi134 ceph-mon[33098]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:41.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:41 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/4207012447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:46:41.683 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:41 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:41.245+0000 7f91f0009200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:46:41.683 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:46:41 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:46:41.461+0000 7f91f0009200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:46:41.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:41 smithi031 ceph-mon[29204]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:41.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:41 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/4207012447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:46:41.888 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:41 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:41.589+0000 7f8966348200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:46:41.888 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:41 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:41.688+0000 7f8966348200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:46:41.888 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:41 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:41.893+0000 7f8966348200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:46:42.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:42.356+0000 7f2f077fe700 1 -- 172.21.15.31:0/2606380781 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7f2f0801acd0 con 0x7f2f18102290 2024-04-25T01:46:42.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:42 smithi040 ceph-mon[32418]: Standby manager daemon z started 2024-04-25T01:46:42.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:42 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:46:42.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:42 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:42.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:42 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:46:42.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:42 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:42.654 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:42 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:42.436+0000 7f8966348200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:46:42.654 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:42 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:42.530+0000 7f8966348200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:46:42.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:42 smithi134 ceph-mon[33098]: Standby manager daemon z started 2024-04-25T01:46:42.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:42 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:46:42.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:42 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:42.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:42 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:46:42.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:42 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:42.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:42 smithi031 ceph-mon[29204]: Standby manager daemon z started 2024-04-25T01:46:42.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:42 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:46:42.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:42 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:42.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:42 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:46:42.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:42 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.134:0/290449043' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:43.126 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:42 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:42.779+0000 7f8966348200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:46:43.127 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:42 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:42.916+0000 7f8966348200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:46:43.127 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:43 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:43.013+0000 7f8966348200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:46:43.127 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:43 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:43.131+0000 7f8966348200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:46:43.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:43 smithi040 ceph-mon[32418]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:43.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:43 smithi040 ceph-mon[32418]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T01:46:43.403 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:43 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:46:43.404 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:43 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:43.225+0000 7f8966348200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:46:43.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:43 smithi134 ceph-mon[33098]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:43.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:43 smithi134 ceph-mon[33098]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T01:46:43.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:43 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:46:43.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:43 smithi031 ceph-mon[29204]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:43.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:43 smithi031 ceph-mon[29204]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T01:46:43.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:43 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:46:44.350 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:44 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:44.065+0000 7f8966348200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:46:44.350 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:46:44 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:46:44.284+0000 7f8966348200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:46:44.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:44.365+0000 7f2f077fe700 1 -- 172.21.15.31:0/2606380781 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f2f0800f3c0 con 0x7f2f18102290 2024-04-25T01:46:44.532 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:44 smithi031 ceph-mon[29204]: Standby manager daemon y started 2024-04-25T01:46:44.532 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:44 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:46:44.532 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:44 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:44.532 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:44 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:46:44.532 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:44 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:44.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:44 smithi040 ceph-mon[32418]: Standby manager daemon y started 2024-04-25T01:46:44.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:44 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:46:44.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:44 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:44.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:44 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:46:44.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:44 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:44.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:44 smithi134 ceph-mon[33098]: Standby manager daemon y started 2024-04-25T01:46:44.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:44 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:46:44.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:44 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:44.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:44 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:46:44.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:44 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.40:0/2211797874' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:45.622 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:45 smithi031 ceph-mon[29204]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:45.622 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:45 smithi031 ceph-mon[29204]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T01:46:45.622 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:45 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:46:45.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:45 smithi040 ceph-mon[32418]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:45.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:45 smithi040 ceph-mon[32418]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T01:46:45.653 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:45 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:46:45.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:45 smithi134 ceph-mon[33098]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:45.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:45 smithi134 ceph-mon[33098]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T01:46:45.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:45 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:46:46.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:46 smithi031 ceph-mon[29204]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:46.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:46 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:46.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:46 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:46.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:46 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:46.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:46 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:46.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:46 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:46.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:46 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:47.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:46 smithi134 ceph-mon[33098]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:47.010 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:46 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:47.010 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:46 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:47.010 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:46 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:47.010 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:46 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:47.010 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:46 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:47.010 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:46 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:47.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:46 smithi040 ceph-mon[32418]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:47.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:46 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:47.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:46 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:47.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:46 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:47.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:46 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:47.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:46 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:47.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:46 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:48.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:48 smithi031 ceph-mon[29204]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:48.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:48 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:46:48.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:48 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:48.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:48 smithi031 ceph-mon[29204]: Deploying daemon osd.0 on smithi031 2024-04-25T01:46:49.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:48 smithi040 ceph-mon[32418]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:49.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:48 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:46:49.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:48 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:49.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:48 smithi040 ceph-mon[32418]: Deploying daemon osd.0 on smithi031 2024-04-25T01:46:49.183 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:48 smithi134 ceph-mon[33098]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:49.183 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:48 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:46:49.183 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:48 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:49.183 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:48 smithi134 ceph-mon[33098]: Deploying daemon osd.0 on smithi031 2024-04-25T01:46:50.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:50 smithi031 ceph-mon[29204]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:51.153 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:50 smithi040 ceph-mon[32418]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:51.183 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:50 smithi134 ceph-mon[33098]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:52.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:52 smithi031 ceph-mon[29204]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:53.153 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:52 smithi040 ceph-mon[32418]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:53.183 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:52 smithi134 ceph-mon[33098]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:54.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:54 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:54.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:54 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:54.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:54 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:54.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:54 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:54.903 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:54 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:54.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:54 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:54.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:54 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:54.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:54 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:54.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:54 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:55.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:55 smithi031 ceph-mon[29204]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:55.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:55.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:55.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:55.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:55.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:55.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:55 smithi040 ceph-mon[32418]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:55.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:55.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:55.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:55.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:55.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:55.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:55 smithi134 ceph-mon[33098]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:55.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:55.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:55.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:55.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:55.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:57.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.264+0000 7f2f077fe700 1 -- 172.21.15.31:0/2606380781 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2f18062670 con 0x7f2f00039330 2024-04-25T01:46:57.259 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 0 on host 'smithi031' 2024-04-25T01:46:57.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.266+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2606380781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2f00039330 msgr2=0x7f2f0003b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:57.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.266+0000 7f2f1c96d700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2f00039330 0x7f2f0003b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009be0 tx=0x7f2f0c009260 comp rx=0 tx=0).stop 2024-04-25T01:46:57.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.266+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2606380781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f18102290 msgr2=0x7f2f18198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:57.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.266+0000 7f2f1c96d700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f18102290 0x7f2f18198340 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800d9b0 tx=0x7f2f0800dcc0 comp rx=0 tx=0).stop 2024-04-25T01:46:57.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.266+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2606380781 shutdown_connections 2024-04-25T01:46:57.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.266+0000 7f2f1c96d700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2f00039330 0x7f2f0003b7f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:57.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.266+0000 7f2f1c96d700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f18103e30 0x7f2f1819f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:57.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.266+0000 7f2f1c96d700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f18102290 0x7f2f18198340 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:57.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.266+0000 7f2f1c96d700 1 --2- 172.21.15.31:0/2606380781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f18103490 0x7f2f18198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:57.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.266+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2606380781 >> 172.21.15.31:0/2606380781 conn(0x7f2f180fd830 msgr2=0x7f2f18106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:57.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.266+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2606380781 shutdown_connections 2024-04-25T01:46:57.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:46:57.267+0000 7f2f1c96d700 1 -- 172.21.15.31:0/2606380781 wait complete. 2024-04-25T01:46:57.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:57 smithi031 ceph-mon[29204]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:57.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:57.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:57.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:57.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:57 smithi031 ceph-mon[29204]: from='osd.0 [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:46:57.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:57 smithi031 ceph-mon[29204]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:46:57.898 DEBUG:teuthology.orchestra.run.smithi031:osd.0> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T01:46:57.901 INFO:tasks.cephadm:Deploying osd.1 on smithi040 with /dev/vg_nvme/lv_4... 2024-04-25T01:46:57.901 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:46:57.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:57 smithi040 ceph-mon[32418]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:57.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:57.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:57.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:57.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:57 smithi040 ceph-mon[32418]: from='osd.0 [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:46:57.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:57 smithi040 ceph-mon[32418]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:46:57.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:57 smithi134 ceph-mon[33098]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:57.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:57.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:46:57.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:57.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:57 smithi134 ceph-mon[33098]: from='osd.0 [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:46:57.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:57 smithi134 ceph-mon[33098]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:46:58.156 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:46:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:59 smithi031 ceph-mon[29204]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:59.366 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:59 smithi031 ceph-mon[29204]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:46:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:59 smithi031 ceph-mon[29204]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:46:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:59 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:59 smithi031 ceph-mon[29204]: from='osd.0 [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:46:59.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:46:59 smithi031 ceph-mon[29204]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:46:59.367 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 01:46:59 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0[43534]: 2024-04-25T01:46:59.255+0000 7f48268b6700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T01:46:59.367 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 01:46:59 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0[43534]: 2024-04-25T01:46:59.258+0000 7f48226a9700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:46:59.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:59 smithi040 ceph-mon[32418]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:59.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:59 smithi040 ceph-mon[32418]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:46:59.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:59 smithi040 ceph-mon[32418]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:46:59.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:59 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:59.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:59 smithi040 ceph-mon[32418]: from='osd.0 [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:46:59.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:46:59 smithi040 ceph-mon[32418]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:46:59.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:59 smithi134 ceph-mon[33098]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:59.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:59 smithi134 ceph-mon[33098]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:46:59.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:59 smithi134 ceph-mon[33098]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:46:59.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:59 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:59.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:59 smithi134 ceph-mon[33098]: from='osd.0 [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:46:59.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:46:59 smithi134 ceph-mon[33098]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:47:00.010 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:47:00.010 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:47:00.010 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-25T01:47:00.010 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-25T01:47:00.010 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013396 s, 783 MB/s 2024-04-25T01:47:00.011 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-25T01:47:00.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:00 smithi031 ceph-mon[29204]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:47:00.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:00 smithi031 ceph-mon[29204]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:47:00.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:00 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:00.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:00 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:00.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:00 smithi040 ceph-mon[32418]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:47:00.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:00 smithi040 ceph-mon[32418]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:47:00.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:00 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:00.506 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:00 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:00.675 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_4 2024-04-25T01:47:00.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:00 smithi134 ceph-mon[33098]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:47:00.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:00 smithi134 ceph-mon[33098]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:47:00.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:00 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:00.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:00 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:00.922 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:47:01.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:47:01.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:47:01.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:47:01.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:01.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: osd.0 [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401] boot 2024-04-25T01:47:01.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:47:01.316 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:01.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:01.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:01.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:01.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:01.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:01.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:01.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:01.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:47:01.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:47:01.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:47:01.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:01.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: osd.0 [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401] boot 2024-04-25T01:47:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:47:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: osd.0 [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401] boot 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:01.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:01.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:01.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:01.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:01.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:01.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:02.372 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:02 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi031 2024-04-25T01:47:02.373 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:02 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi031 to 17093M 2024-04-25T01:47:02.373 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:02 smithi040 ceph-mon[32418]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:47:02.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.380+0000 7fdff00cc700 1 -- 172.21.15.40:0/3130756390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfe8101180 msgr2=0x7fdfe81015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:02.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.380+0000 7fdff00cc700 1 --2- 172.21.15.40:0/3130756390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfe8101180 0x7fdfe81015a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40099c0 tx=0x7fdfe4009cd0 comp rx=0 tx=0).stop 2024-04-25T01:47:02.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.380+0000 7fdff00cc700 1 -- 172.21.15.40:0/3130756390 shutdown_connections 2024-04-25T01:47:02.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.380+0000 7fdff00cc700 1 --2- 172.21.15.40:0/3130756390 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8102d20 0x7fdfe81095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:02.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.380+0000 7fdff00cc700 1 --2- 172.21.15.40:0/3130756390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfe8101180 0x7fdfe81015a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:02.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.380+0000 7fdff00cc700 1 --2- 172.21.15.40:0/3130756390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8102380 0x7fdfe81027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:02.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.380+0000 7fdff00cc700 1 -- 172.21.15.40:0/3130756390 >> 172.21.15.40:0/3130756390 conn(0x7fdfe80fc720 msgr2=0x7fdfe80feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:47:02.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.380+0000 7fdff00cc700 1 -- 172.21.15.40:0/3130756390 shutdown_connections 2024-04-25T01:47:02.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.380+0000 7fdff00cc700 1 -- 172.21.15.40:0/3130756390 wait complete. 2024-04-25T01:47:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdff00cc700 1 Processor -- start 2024-04-25T01:47:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdff00cc700 1 -- start start 2024-04-25T01:47:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdff00cc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8101180 0x7fdfe806b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdff00cc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8102380 0x7fdfe819d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdff00cc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfe8102d20 0x7fdfe819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdff00cc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe810cc10 con 0x7fdfe8102380 2024-04-25T01:47:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdff00cc700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe810ca90 con 0x7fdfe8102d20 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdff00cc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe810cd90 con 0x7fdfe8101180 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdfee669700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfe8102d20 0x7fdfe819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdfee669700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfe8102d20 0x7fdfe819f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34312/0 (socket says 172.21.15.40:34312) 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdfee669700 1 -- 172.21.15.40:0/481156346 learned_addr learned my addr 172.21.15.40:0/481156346 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdfed667700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8102380 0x7fdfe819d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdfede68700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8101180 0x7fdfe806b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdfee669700 1 -- 172.21.15.40:0/481156346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8101180 msgr2=0x7fdfe806b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdfee669700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8101180 0x7fdfe806b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdfee669700 1 -- 172.21.15.40:0/481156346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8102380 msgr2=0x7fdfe819d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdfee669700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8102380 0x7fdfe819d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.381+0000 7fdfee669700 1 -- 172.21.15.40:0/481156346 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe819ff90 con 0x7fdfe8102d20 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.382+0000 7fdfee669700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfe8102d20 0x7fdfe819f6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000ada0 tx=0x7fdfe000c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.382+0000 7fdfdeffd700 1 -- 172.21.15.40:0/481156346 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0017020 con 0x7fdfe8102d20 2024-04-25T01:47:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.382+0000 7fdff00cc700 1 -- 172.21.15.40:0/481156346 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe81a0280 con 0x7fdfe8102d20 2024-04-25T01:47:02.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.382+0000 7fdfdeffd700 1 -- 172.21.15.40:0/481156346 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfe000ce30 con 0x7fdfe8102d20 2024-04-25T01:47:02.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.382+0000 7fdfdeffd700 1 -- 172.21.15.40:0/481156346 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0005350 con 0x7fdfe8102d20 2024-04-25T01:47:02.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.382+0000 7fdff00cc700 1 -- 172.21.15.40:0/481156346 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe81a0810 con 0x7fdfe8102d20 2024-04-25T01:47:02.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.383+0000 7fdff00cc700 1 -- 172.21.15.40:0/481156346 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfe8068f20 con 0x7fdfe8102d20 2024-04-25T01:47:02.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.384+0000 7fdfdeffd700 1 -- 172.21.15.40:0/481156346 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fdfe000c980 con 0x7fdfe8102d20 2024-04-25T01:47:02.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.384+0000 7fdfdeffd700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdfd40a2b90 0x7fdfd40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:02.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.385+0000 7fdfdeffd700 1 -- 172.21.15.40:0/481156346 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fdfe00cccc0 con 0x7fdfe8102d20 2024-04-25T01:47:02.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.385+0000 7fdfede68700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdfd40a2b90 0x7fdfd40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:02.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.385+0000 7fdfede68700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdfd40a2b90 0x7fdfd40a5050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4000c00 tx=0x7fdfe4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:47:02.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.389+0000 7fdfdeffd700 1 -- 172.21.15.40:0/481156346 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdfe0021020 con 0x7fdfe8102d20 2024-04-25T01:47:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:02.606+0000 7fdff00cc700 1 -- 172.21.15.40:0/481156346 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fdfe8062730 con 0x7fdfd40a2b90 2024-04-25T01:47:02.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:02 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi031 2024-04-25T01:47:02.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:02 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi031 to 17093M 2024-04-25T01:47:02.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:02 smithi134 ceph-mon[33098]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:47:02.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:02 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi031 2024-04-25T01:47:02.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:02 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi031 to 17093M 2024-04-25T01:47:02.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:02 smithi031 ceph-mon[29204]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:47:03.550 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:03 smithi040 ceph-mon[32418]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:03.550 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:03 smithi040 ceph-mon[32418]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:03.550 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:03 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:03.550 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:03 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:03.551 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:03 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:03.584 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:03 smithi134 ceph-mon[33098]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:03.584 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:03 smithi134 ceph-mon[33098]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:03.585 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:03 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:03.585 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:03 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:03.585 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:03 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:03.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:03 smithi031 ceph-mon[29204]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:03.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:03 smithi031 ceph-mon[29204]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:03.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:03 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:03.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:03 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:03.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:03 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:05.613 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:05 smithi040 ceph-mon[32418]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:05.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:05 smithi134 ceph-mon[33098]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:05.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:05 smithi031 ceph-mon[29204]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:06.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:06 smithi040 ceph-mon[32418]: from='client.? 172.21.15.40:0/1954974865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285db63c-f115-4517-894f-01b97208ba8d"}]: dispatch 2024-04-25T01:47:06.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:06 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285db63c-f115-4517-894f-01b97208ba8d"}]: dispatch 2024-04-25T01:47:06.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:06 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "285db63c-f115-4517-894f-01b97208ba8d"}]': finished 2024-04-25T01:47:06.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:06 smithi040 ceph-mon[32418]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:47:06.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:06 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:06.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:06 smithi040 ceph-mon[32418]: from='client.? 172.21.15.40:0/1838512586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:06.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:06 smithi134 ceph-mon[33098]: from='client.? 172.21.15.40:0/1954974865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285db63c-f115-4517-894f-01b97208ba8d"}]: dispatch 2024-04-25T01:47:06.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:06 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285db63c-f115-4517-894f-01b97208ba8d"}]: dispatch 2024-04-25T01:47:06.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:06 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "285db63c-f115-4517-894f-01b97208ba8d"}]': finished 2024-04-25T01:47:06.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:06 smithi134 ceph-mon[33098]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:47:06.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:06 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:06.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:06 smithi134 ceph-mon[33098]: from='client.? 172.21.15.40:0/1838512586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:06.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:06 smithi031 ceph-mon[29204]: from='client.? 172.21.15.40:0/1954974865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285db63c-f115-4517-894f-01b97208ba8d"}]: dispatch 2024-04-25T01:47:06.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:06 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "285db63c-f115-4517-894f-01b97208ba8d"}]: dispatch 2024-04-25T01:47:06.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:06 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "285db63c-f115-4517-894f-01b97208ba8d"}]': finished 2024-04-25T01:47:06.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:06 smithi031 ceph-mon[29204]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:47:06.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:06 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:06.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:06 smithi031 ceph-mon[29204]: from='client.? 172.21.15.40:0/1838512586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:07.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:07 smithi031 ceph-mon[29204]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:07.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:07 smithi040 ceph-mon[32418]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:07.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:07 smithi134 ceph-mon[33098]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:09.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:09 smithi031 ceph-mon[29204]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:09.587 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:09 smithi040 ceph-mon[32418]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:09.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:09 smithi134 ceph-mon[33098]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:11.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:11 smithi040 ceph-mon[32418]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:11.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:11 smithi134 ceph-mon[33098]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:11.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:11 smithi031 ceph-mon[29204]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:13.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:13 smithi040 ceph-mon[32418]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:13.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:47:13.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:13.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:13 smithi040 ceph-mon[32418]: Deploying daemon osd.1 on smithi040 2024-04-25T01:47:13.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:13 smithi134 ceph-mon[33098]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:13.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:47:13.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:13.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:13 smithi134 ceph-mon[33098]: Deploying daemon osd.1 on smithi040 2024-04-25T01:47:13.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:13 smithi031 ceph-mon[29204]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:13.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:47:13.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:13.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:13 smithi031 ceph-mon[29204]: Deploying daemon osd.1 on smithi040 2024-04-25T01:47:15.552 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:15 smithi040 ceph-mon[32418]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:15.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:15 smithi134 ceph-mon[33098]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:15.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:15 smithi031 ceph-mon[29204]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:17.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:17 smithi040 ceph-mon[32418]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:17.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:17 smithi134 ceph-mon[33098]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:17.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:17 smithi031 ceph-mon[29204]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:19.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:19 smithi040 ceph-mon[32418]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:19.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:19.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:19.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:19.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:19 smithi134 ceph-mon[33098]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:19.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:19.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:19.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:19.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:19 smithi031 ceph-mon[29204]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:19.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:19.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:19.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:21.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:21 smithi040 ceph-mon[32418]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:21.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:21 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:21.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:21 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:21.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:21 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:21.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:21 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:21.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:21 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:21.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:21 smithi134 ceph-mon[33098]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:21.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:21 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:21.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:21 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:21.433 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:21 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:21.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:21 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:21.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:21 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:21.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:21 smithi031 ceph-mon[29204]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:21.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:21 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:21.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:21 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:21.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:21 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:21.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:21 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:21.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:21 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:21.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.824+0000 7fdfdeffd700 1 -- 172.21.15.40:0/481156346 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdfe8062730 con 0x7fdfd40a2b90 2024-04-25T01:47:21.820 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 1 on host 'smithi040' 2024-04-25T01:47:21.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 -- 172.21.15.40:0/481156346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdfd40a2b90 msgr2=0x7fdfd40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdfd40a2b90 0x7fdfd40a5050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4000c00 tx=0x7fdfe4005fd0 comp rx=0 tx=0).stop 2024-04-25T01:47:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 -- 172.21.15.40:0/481156346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfe8102d20 msgr2=0x7fdfe819f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfe8102d20 0x7fdfe819f6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000ada0 tx=0x7fdfe000c330 comp rx=0 tx=0).stop 2024-04-25T01:47:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 -- 172.21.15.40:0/481156346 shutdown_connections 2024-04-25T01:47:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdfd40a2b90 0x7fdfd40a5050 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfe8102d20 0x7fdfe819f6c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8101180 0x7fdfe806b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 --2- 172.21.15.40:0/481156346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8102380 0x7fdfe819d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 -- 172.21.15.40:0/481156346 >> 172.21.15.40:0/481156346 conn(0x7fdfe80fc720 msgr2=0x7fdfe81056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:47:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 -- 172.21.15.40:0/481156346 shutdown_connections 2024-04-25T01:47:21.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:47:21.826+0000 7fdff00cc700 1 -- 172.21.15.40:0/481156346 wait complete. 2024-04-25T01:47:22.418 DEBUG:teuthology.orchestra.run.smithi040:osd.1> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T01:47:22.420 INFO:tasks.cephadm:Deploying osd.2 on smithi134 with /dev/vg_nvme/lv_4... 2024-04-25T01:47:22.421 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:47:22.665 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:47:23.064 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:22 smithi134 ceph-mon[33098]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:23.064 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:22 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:23.064 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:22 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:23.064 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:22 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:23.065 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:22 smithi134 ceph-mon[33098]: from='osd.1 [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:23.065 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:22 smithi134 ceph-mon[33098]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:23.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:22 smithi040 ceph-mon[32418]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:23.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:22 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:23.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:22 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:23.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:22 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:23.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:22 smithi040 ceph-mon[32418]: from='osd.1 [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:23.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:22 smithi040 ceph-mon[32418]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:23.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:22 smithi031 ceph-mon[29204]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:23.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:22 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:23.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:22 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:23.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:22 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:23.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:22 smithi031 ceph-mon[29204]: from='osd.1 [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:23.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:22 smithi031 ceph-mon[29204]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:23.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:23 smithi040 ceph-mon[32418]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:47:23.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:23 smithi040 ceph-mon[32418]: from='osd.1 [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:47:23.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:23 smithi040 ceph-mon[32418]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:47:23.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:23.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:23 smithi040 ceph-mon[32418]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:47:23.925 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:23 smithi134 ceph-mon[33098]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:47:23.925 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:23 smithi134 ceph-mon[33098]: from='osd.1 [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:47:23.925 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:23 smithi134 ceph-mon[33098]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:47:23.926 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:23.926 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:23 smithi134 ceph-mon[33098]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:47:24.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:23 smithi031 ceph-mon[29204]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:47:24.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:23 smithi031 ceph-mon[29204]: from='osd.1 [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:47:24.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:23 smithi031 ceph-mon[29204]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:47:24.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:24.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:23 smithi031 ceph-mon[29204]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:47:24.518 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:47:24.518 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:47:24.518 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-04-25T01:47:24.519 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-04-25T01:47:24.519 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122326 s, 857 MB/s 2024-04-25T01:47:24.519 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-04-25T01:47:24.654 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 01:47:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1[37010]: 2024-04-25T01:47:24.241+0000 7ffaebc2d700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T01:47:24.654 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 01:47:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1[37010]: 2024-04-25T01:47:24.245+0000 7ffae3a18700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:47:25.114 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:24 smithi040 ceph-mon[32418]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:25.114 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:24 smithi040 ceph-mon[32418]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:47:25.114 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:24 smithi040 ceph-mon[32418]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:47:25.114 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:25.114 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:25.114 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:24 smithi040 ceph-mon[32418]: from='osd.1 ' entity='osd.1' 2024-04-25T01:47:25.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:24 smithi134 ceph-mon[33098]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:25.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:24 smithi134 ceph-mon[33098]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:47:25.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:24 smithi134 ceph-mon[33098]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:47:25.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:25.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:25.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:24 smithi134 ceph-mon[33098]: from='osd.1 ' entity='osd.1' 2024-04-25T01:47:25.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi134:vg_nvme/lv_4 2024-04-25T01:47:25.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:24 smithi031 ceph-mon[29204]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:25.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:24 smithi031 ceph-mon[29204]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:47:25.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:24 smithi031 ceph-mon[29204]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:47:25.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:25.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:25.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:24 smithi031 ceph-mon[29204]: from='osd.1 ' entity='osd.1' 2024-04-25T01:47:25.391 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:47:26.011 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: osd.1 [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525] boot 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi040 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi040 to 17093M 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:26.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:26.013 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:47:26.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:47:26.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: osd.1 [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525] boot 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi040 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi040 to 17093M 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:26.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:47:26.232 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: osd.1 [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525] boot 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi040 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi040 to 17093M 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:26.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:26.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.722+0000 7f5c3c24f700 1 -- 172.21.15.134:0/355961664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c34102390 msgr2=0x7f5c341027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:26.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.722+0000 7f5c3c24f700 1 --2- 172.21.15.134:0/355961664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c34102390 0x7f5c341027f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009990 tx=0x7f5c30009ca0 comp rx=0 tx=0).stop 2024-04-25T01:47:26.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.722+0000 7f5c3c24f700 1 -- 172.21.15.134:0/355961664 shutdown_connections 2024-04-25T01:47:26.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.722+0000 7f5c3c24f700 1 --2- 172.21.15.134:0/355961664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c34102d30 0x7f5c341095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:26.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.722+0000 7f5c3c24f700 1 --2- 172.21.15.134:0/355961664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c34101190 0x7f5c341015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:26.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.722+0000 7f5c3c24f700 1 --2- 172.21.15.134:0/355961664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c34102390 0x7f5c341027f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:26.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.722+0000 7f5c3c24f700 1 -- 172.21.15.134:0/355961664 >> 172.21.15.134:0/355961664 conn(0x7f5c340fc750 msgr2=0x7f5c340feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:47:26.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.723+0000 7f5c3c24f700 1 -- 172.21.15.134:0/355961664 shutdown_connections 2024-04-25T01:47:26.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.723+0000 7f5c3c24f700 1 -- 172.21.15.134:0/355961664 wait complete. 2024-04-25T01:47:26.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.723+0000 7f5c3c24f700 1 Processor -- start 2024-04-25T01:47:26.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.723+0000 7f5c3c24f700 1 -- start start 2024-04-25T01:47:26.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.723+0000 7f5c3c24f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c34101190 0x7f5c341984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.723+0000 7f5c3c24f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c34102390 0x7f5c34198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.723+0000 7f5c3c24f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c34102d30 0x7f5c3419f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.723+0000 7f5c3c24f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3410cc90 con 0x7f5c34102390 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.723+0000 7f5c3c24f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3410cb10 con 0x7f5c34102d30 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.723+0000 7f5c3c24f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3410ce10 con 0x7f5c34101190 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.723+0000 7f5c39feb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c34101190 0x7f5c341984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c39feb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c34101190 0x7f5c341984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43492/0 (socket says 172.21.15.134:43492) 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c39feb700 1 -- 172.21.15.134:0/3066487944 learned_addr learned my addr 172.21.15.134:0/3066487944 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c397ea700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c34102390 0x7f5c34198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c3a7ec700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c34102d30 0x7f5c3419f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c39feb700 1 -- 172.21.15.134:0/3066487944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c34102d30 msgr2=0x7f5c3419f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c39feb700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c34102d30 0x7f5c3419f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c39feb700 1 -- 172.21.15.134:0/3066487944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c34102390 msgr2=0x7f5c34198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c39feb700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c34102390 0x7f5c34198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c39feb700 1 -- 172.21.15.134:0/3066487944 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c341a0080 con 0x7f5c34101190 2024-04-25T01:47:26.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c39feb700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c34101190 0x7f5c341984c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400c9a0 tx=0x7f5c2400ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:47:26.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c2affd700 1 -- 172.21.15.134:0/3066487944 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24004220 con 0x7f5c34101190 2024-04-25T01:47:26.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c3c24f700 1 -- 172.21.15.134:0/3066487944 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c341a0370 con 0x7f5c34101190 2024-04-25T01:47:26.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c2affd700 1 -- 172.21.15.134:0/3066487944 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c24007d10 con 0x7f5c34101190 2024-04-25T01:47:26.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c2affd700 1 -- 172.21.15.134:0/3066487944 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24003d00 con 0x7f5c34101190 2024-04-25T01:47:26.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.724+0000 7f5c3c24f700 1 -- 172.21.15.134:0/3066487944 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c341a0980 con 0x7f5c34101190 2024-04-25T01:47:26.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.725+0000 7f5c3c24f700 1 -- 172.21.15.134:0/3066487944 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c34068f20 con 0x7f5c34101190 2024-04-25T01:47:26.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.726+0000 7f5c2affd700 1 -- 172.21.15.134:0/3066487944 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f5c24007860 con 0x7f5c34101190 2024-04-25T01:47:26.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.727+0000 7f5c2affd700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5c200a2b90 0x7f5c200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:26.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.727+0000 7f5c2affd700 1 -- 172.21.15.134:0/3066487944 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7f5c240cd770 con 0x7f5c34101190 2024-04-25T01:47:26.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.728+0000 7f5c397ea700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5c200a2b90 0x7f5c200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:26.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.728+0000 7f5c397ea700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5c200a2b90 0x7f5c200a5050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000b5c0 tx=0x7f5c30005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:47:26.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.731+0000 7f5c2affd700 1 -- 172.21.15.134:0/3066487944 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5c24014030 con 0x7f5c34101190 2024-04-25T01:47:26.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:26.934+0000 7f5c3c24f700 1 -- 172.21.15.134:0/3066487944 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f5c341a0c40 con 0x7f5c200a2b90 2024-04-25T01:47:27.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:27 smithi031 ceph-mon[29204]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:27.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:27 smithi031 ceph-mon[29204]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:47:27.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:27 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:27.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:27 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:27.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:27 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:27.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:27 smithi134 ceph-mon[33098]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:27.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:27 smithi134 ceph-mon[33098]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:47:27.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:27 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:27.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:27 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:27.836 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:27 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:27.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:27 smithi040 ceph-mon[32418]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:27.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:27 smithi040 ceph-mon[32418]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:47:27.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:27 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:27.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:27 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:27.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:27 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:28.697 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:28 smithi134 ceph-mon[33098]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:28.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:28 smithi040 ceph-mon[32418]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:28.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:28 smithi031 ceph-mon[29204]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:29.850 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:29 smithi134 ceph-mon[33098]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:29.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:29 smithi040 ceph-mon[32418]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:29.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:29 smithi031 ceph-mon[29204]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:30.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:30 smithi040 ceph-mon[32418]: from='client.? 172.21.15.134:0/3898560207' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af88e288-e53a-46a6-93a0-576c53c16b71"}]: dispatch 2024-04-25T01:47:30.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:30 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af88e288-e53a-46a6-93a0-576c53c16b71"}]: dispatch 2024-04-25T01:47:30.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:30 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af88e288-e53a-46a6-93a0-576c53c16b71"}]': finished 2024-04-25T01:47:30.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:30 smithi040 ceph-mon[32418]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:47:30.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:30 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:30.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:30 smithi040 ceph-mon[32418]: from='client.? 172.21.15.134:0/880656155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:30.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:30 smithi134 ceph-mon[33098]: from='client.? 172.21.15.134:0/3898560207' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af88e288-e53a-46a6-93a0-576c53c16b71"}]: dispatch 2024-04-25T01:47:30.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:30 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af88e288-e53a-46a6-93a0-576c53c16b71"}]: dispatch 2024-04-25T01:47:30.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:30 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af88e288-e53a-46a6-93a0-576c53c16b71"}]': finished 2024-04-25T01:47:30.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:30 smithi134 ceph-mon[33098]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:47:30.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:30 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:30.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:30 smithi134 ceph-mon[33098]: from='client.? 172.21.15.134:0/880656155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:30.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:30 smithi031 ceph-mon[29204]: from='client.? 172.21.15.134:0/3898560207' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af88e288-e53a-46a6-93a0-576c53c16b71"}]: dispatch 2024-04-25T01:47:30.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:30 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af88e288-e53a-46a6-93a0-576c53c16b71"}]: dispatch 2024-04-25T01:47:30.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:30 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af88e288-e53a-46a6-93a0-576c53c16b71"}]': finished 2024-04-25T01:47:30.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:30 smithi031 ceph-mon[29204]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:47:30.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:30 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:30.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:30 smithi031 ceph-mon[29204]: from='client.? 172.21.15.134:0/880656155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:31.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:31 smithi040 ceph-mon[32418]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:31.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:31 smithi134 ceph-mon[33098]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:31.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:31 smithi031 ceph-mon[29204]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:33.678 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:33 smithi134 ceph-mon[33098]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:33.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:33 smithi040 ceph-mon[32418]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:33.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:33 smithi031 ceph-mon[29204]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:35.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:35 smithi134 ceph-mon[33098]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:35.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:35 smithi040 ceph-mon[32418]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:35.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:35 smithi031 ceph-mon[29204]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:37.785 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:37 smithi031 ceph-mon[29204]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:37.785 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:37 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:47:37.785 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:37 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:37.785 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:37 smithi031 ceph-mon[29204]: Deploying daemon osd.2 on smithi134 2024-04-25T01:47:37.821 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:37 smithi134 ceph-mon[33098]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:37.821 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:37 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:47:37.821 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:37 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:37.821 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:37 smithi134 ceph-mon[33098]: Deploying daemon osd.2 on smithi134 2024-04-25T01:47:37.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:37 smithi040 ceph-mon[32418]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:37.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:37 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:47:37.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:37 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:37.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:37 smithi040 ceph-mon[32418]: Deploying daemon osd.2 on smithi134 2024-04-25T01:47:39.832 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:39 smithi134 ceph-mon[33098]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:39.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:39 smithi040 ceph-mon[32418]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:39.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:39 smithi031 ceph-mon[29204]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:41.732 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:41 smithi134 ceph-mon[33098]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:41.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:41 smithi040 ceph-mon[32418]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:41.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:41 smithi031 ceph-mon[29204]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:43.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:42 smithi134 ceph-mon[33098]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:43.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:43.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:43.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:43.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:42 smithi040 ceph-mon[32418]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:43.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:43.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:43.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:43.482 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:42 smithi031 ceph-mon[29204]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:43.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:43.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:43.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:44.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.952+0000 7f5c2affd700 1 -- 172.21.15.134:0/3066487944 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5c341a0c40 con 0x7f5c200a2b90 2024-04-25T01:47:44.949 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 2 on host 'smithi134' 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 -- 172.21.15.134:0/3066487944 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5c200a2b90 msgr2=0x7f5c200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5c200a2b90 0x7f5c200a5050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000b5c0 tx=0x7f5c30005fd0 comp rx=0 tx=0).stop 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 -- 172.21.15.134:0/3066487944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c34101190 msgr2=0x7f5c341984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c34101190 0x7f5c341984c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400c9a0 tx=0x7f5c2400ccb0 comp rx=0 tx=0).stop 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 -- 172.21.15.134:0/3066487944 shutdown_connections 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5c200a2b90 0x7f5c200a5050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c34102d30 0x7f5c3419f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c34101190 0x7f5c341984c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 --2- 172.21.15.134:0/3066487944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c34102390 0x7f5c34198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 -- 172.21.15.134:0/3066487944 >> 172.21.15.134:0/3066487944 conn(0x7f5c340fc750 msgr2=0x7f5c3406a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 -- 172.21.15.134:0/3066487944 shutdown_connections 2024-04-25T01:47:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:47:44.955+0000 7f5c3c24f700 1 -- 172.21.15.134:0/3066487944 wait complete. 2024-04-25T01:47:45.291 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:44 smithi134 ceph-mon[33098]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:45.292 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:44 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:45.292 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:44 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:45.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:44 smithi040 ceph-mon[32418]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:45.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:44 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:45.404 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:44 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:45.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:44 smithi031 ceph-mon[29204]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:45.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:44 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:45.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:44 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:45.926 DEBUG:teuthology.orchestra.run.smithi134:osd.2> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T01:47:45.929 INFO:tasks.cephadm:Deploying osd.3 on smithi031 with /dev/vg_nvme/lv_3... 2024-04-25T01:47:45.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:47:46.185 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:47:46.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:46 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:46.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:46 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:46.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:46 smithi031 ceph-mon[29204]: from='osd.2 [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:46.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:46 smithi031 ceph-mon[29204]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:46.774 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:46 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:46.774 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:46 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:46.774 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:46 smithi134 ceph-mon[33098]: from='osd.2 [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:46.774 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:46 smithi134 ceph-mon[33098]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:46.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:46 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:46.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:46 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:46.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:46 smithi040 ceph-mon[32418]: from='osd.2 [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:46.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:46 smithi040 ceph-mon[32418]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:47.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:47 smithi031 ceph-mon[29204]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:47.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:47 smithi031 ceph-mon[29204]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:47:47.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:47 smithi031 ceph-mon[29204]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:47:47.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:47.563 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:47 smithi031 ceph-mon[29204]: from='osd.2 [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:47:47.564 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:47 smithi031 ceph-mon[29204]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:47:47.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:47 smithi134 ceph-mon[33098]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:47.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:47 smithi134 ceph-mon[33098]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:47:47.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:47 smithi134 ceph-mon[33098]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:47:47.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:47.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:47 smithi134 ceph-mon[33098]: from='osd.2 [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:47:47.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:47 smithi134 ceph-mon[33098]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:47:47.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:47 smithi040 ceph-mon[32418]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:47.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:47 smithi040 ceph-mon[32418]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:47:47.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:47 smithi040 ceph-mon[32418]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:47:47.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:47.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:47 smithi040 ceph-mon[32418]: from='osd.2 [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:47:47.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:47 smithi040 ceph-mon[32418]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:47:48.162 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:47:48.162 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:47:48.162 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-25T01:47:48.163 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-25T01:47:48.163 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011324 s, 926 MB/s 2024-04-25T01:47:48.163 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-25T01:47:48.434 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 01:47:47 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2[38239]: 2024-04-25T01:47:47.979+0000 7f6540196700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T01:47:48.434 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 01:47:47 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2[38239]: 2024-04-25T01:47:47.983+0000 7f6537780700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:47:48.794 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:48 smithi134 ceph-mon[33098]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:47:48.794 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:48 smithi134 ceph-mon[33098]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:47:48.794 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:48 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:48.794 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:48 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:48.794 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:48 smithi134 ceph-mon[33098]: from='osd.2 ' entity='osd.2' 2024-04-25T01:47:48.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:48 smithi031 ceph-mon[29204]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:47:48.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:48 smithi031 ceph-mon[29204]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:47:48.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:48 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:48.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:48 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:48.809 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:48 smithi031 ceph-mon[29204]: from='osd.2 ' entity='osd.2' 2024-04-25T01:47:48.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_3 2024-04-25T01:47:48.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:48 smithi040 ceph-mon[32418]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:47:48.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:48 smithi040 ceph-mon[32418]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:47:48.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:48 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:48.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:48 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:48.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:48 smithi040 ceph-mon[32418]: from='osd.2 ' entity='osd.2' 2024-04-25T01:47:49.097 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: osd.2 [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802] boot 2024-04-25T01:47:49.865 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:47:49.865 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:49.865 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:49.865 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:49.865 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:49.865 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:47:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:47:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: osd.2 [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802] boot 2024-04-25T01:47:49.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:47:49.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:49.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:49.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:49.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:49.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:47:49.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:47:49.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:49.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:49.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: osd.2 [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802] boot 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:49.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.621+0000 7f1f552d1700 1 -- 172.21.15.31:0/214547419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f50108000 msgr2=0x7f1f5010a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.621+0000 7f1f552d1700 1 --2- 172.21.15.31:0/214547419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f50108000 0x7f1f5010a3f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800b0a0 tx=0x7f1f4800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:47:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.621+0000 7f1f552d1700 1 -- 172.21.15.31:0/214547419 shutdown_connections 2024-04-25T01:47:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.621+0000 7f1f552d1700 1 --2- 172.21.15.31:0/214547419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f50108000 0x7f1f5010a3f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.621+0000 7f1f552d1700 1 --2- 172.21.15.31:0/214547419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f50102da0 0x7f1f50105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.621+0000 7f1f552d1700 1 --2- 172.21.15.31:0/214547419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f501056d0 0x7f1f50107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.621+0000 7f1f552d1700 1 -- 172.21.15.31:0/214547419 >> 172.21.15.31:0/214547419 conn(0x7f1f500fc730 msgr2=0x7f1f500feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:47:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.621+0000 7f1f552d1700 1 -- 172.21.15.31:0/214547419 shutdown_connections 2024-04-25T01:47:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.621+0000 7f1f552d1700 1 -- 172.21.15.31:0/214547419 wait complete. 2024-04-25T01:47:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.621+0000 7f1f552d1700 1 Processor -- start 2024-04-25T01:47:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f552d1700 1 -- start start 2024-04-25T01:47:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f552d1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f50102da0 0x7f1f501983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f552d1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f501056d0 0x7f1f50198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f552d1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f50108000 0x7f1f5019f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f552d1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5010dd50 con 0x7f1f50108000 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f552d1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5010dbd0 con 0x7f1f501056d0 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f552d1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5010ded0 con 0x7f1f50102da0 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f50108000 0x7f1f5019f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f50108000 0x7f1f5019f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42986/0 (socket says 172.21.15.31:42986) 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4f7fe700 1 -- 172.21.15.31:0/2418014754 learned_addr learned my addr 172.21.15.31:0/2418014754 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4effd700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f50102da0 0x7f1f501983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4f7fe700 1 -- 172.21.15.31:0/2418014754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f50102da0 msgr2=0x7f1f501983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4f7fe700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f50102da0 0x7f1f501983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4f7fe700 1 -- 172.21.15.31:0/2418014754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f501056d0 msgr2=0x7f1f50198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:50.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4e7fc700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f501056d0 0x7f1f50198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4f7fe700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f501056d0 0x7f1f50198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4f7fe700 1 -- 172.21.15.31:0/2418014754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f501a0020 con 0x7f1f50108000 2024-04-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4effd700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f50102da0 0x7f1f501983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.622+0000 7f1f4e7fc700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f501056d0 0x7f1f50198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.623+0000 7f1f4f7fe700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f50108000 0x7f1f5019f980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800ced0 tx=0x7f1f4800cfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.623+0000 7f1f37fff700 1 -- 172.21.15.31:0/2418014754 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f48021070 con 0x7f1f50108000 2024-04-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.623+0000 7f1f37fff700 1 -- 172.21.15.31:0/2418014754 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1f48007de0 con 0x7f1f50108000 2024-04-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.623+0000 7f1f552d1700 1 -- 172.21.15.31:0/2418014754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f501a02b0 con 0x7f1f50108000 2024-04-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.623+0000 7f1f37fff700 1 -- 172.21.15.31:0/2418014754 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4801d450 con 0x7f1f50108000 2024-04-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.623+0000 7f1f552d1700 1 -- 172.21.15.31:0/2418014754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f501a0710 con 0x7f1f50108000 2024-04-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.624+0000 7f1f552d1700 1 -- 172.21.15.31:0/2418014754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f50068f20 con 0x7f1f50108000 2024-04-25T01:47:50.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.625+0000 7f1f37fff700 1 -- 172.21.15.31:0/2418014754 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f1f48003a00 con 0x7f1f50108000 2024-04-25T01:47:50.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.626+0000 7f1f37fff700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1f380a2b90 0x7f1f380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:50.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.626+0000 7f1f4effd700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1f380a2b90 0x7f1f380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:50.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.626+0000 7f1f37fff700 1 -- 172.21.15.31:0/2418014754 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f1f480cee40 con 0x7f1f50108000 2024-04-25T01:47:50.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.626+0000 7f1f4effd700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1f380a2b90 0x7f1f380a5050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1f40005fd0 tx=0x7f1f40005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:47:50.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.630+0000 7f1f37fff700 1 -- 172.21.15.31:0/2418014754 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1f4809d250 con 0x7f1f50108000 2024-04-25T01:47:50.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:50 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi134 2024-04-25T01:47:50.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:50 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi134 to 17088M 2024-04-25T01:47:50.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:50 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:50.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:50.847+0000 7f1f552d1700 1 -- 172.21.15.31:0/2418014754 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1f50062670 con 0x7f1f380a2b90 2024-04-25T01:47:50.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:50 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi134 2024-04-25T01:47:50.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:50 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi134 to 17088M 2024-04-25T01:47:50.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:50 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:50.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:50 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi134 2024-04-25T01:47:50.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:50 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi134 to 17088M 2024-04-25T01:47:50.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:50 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:51.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:51 smithi031 ceph-mon[29204]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:51.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:51 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:51.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:51 smithi031 ceph-mon[29204]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:47:51.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:51 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:51.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:51 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:51.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:51 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:51.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:51 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:51.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:51 smithi040 ceph-mon[32418]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:51.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:51 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:51.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:51 smithi040 ceph-mon[32418]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:47:51.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:51 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:51.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:51 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:51.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:51 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:51.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:51 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:51.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:51 smithi134 ceph-mon[33098]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:51.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:51 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:51.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:51 smithi134 ceph-mon[33098]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:47:51.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:51 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:51.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:51 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:51.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:51 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:51.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:51 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:52.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:52 smithi031 ceph-mon[29204]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:52.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:52 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:52.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:52 smithi031 ceph-mon[29204]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:47:52.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:52 smithi040 ceph-mon[32418]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:52.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:52 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:52.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:52 smithi040 ceph-mon[32418]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:47:52.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:52 smithi134 ceph-mon[33098]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:52.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:52 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:52.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:52 smithi134 ceph-mon[33098]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:47:53.233 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 01:47:53 smithi031 sudo[47983]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:53.233 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 01:47:53 smithi031 sudo[47983]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:53.233 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 01:47:53 smithi031 sudo[47983]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:53.233 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 01:47:53 smithi031 sudo[47983]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:53.829 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 01:47:53 smithi031 sudo[48020]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:53.829 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 01:47:53 smithi031 sudo[48020]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:53.829 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 01:47:53 smithi031 sudo[48020]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:53.829 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 01:47:53 smithi031 sudo[48020]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:53.829 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:53 smithi031 ceph-mon[29204]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:53.829 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:53 smithi031 ceph-mon[29204]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:47:53.830 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:53 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2914489890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc257cfc-3310-4d0f-bb39-ec5aed0f18f0"}]: dispatch 2024-04-25T01:47:53.830 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:53 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2914489890' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc257cfc-3310-4d0f-bb39-ec5aed0f18f0"}]': finished 2024-04-25T01:47:53.830 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:53 smithi031 ceph-mon[29204]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:47:53.830 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:47:53.830 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:53.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:53 smithi040 ceph-mon[32418]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:53.904 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:53 smithi040 ceph-mon[32418]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:47:53.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:53 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2914489890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc257cfc-3310-4d0f-bb39-ec5aed0f18f0"}]: dispatch 2024-04-25T01:47:53.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:53 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2914489890' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc257cfc-3310-4d0f-bb39-ec5aed0f18f0"}]': finished 2024-04-25T01:47:53.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:53 smithi040 ceph-mon[32418]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:47:53.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:53 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:47:53.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:53 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:53.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:53 smithi134 ceph-mon[33098]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:53.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:53 smithi134 ceph-mon[33098]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:47:53.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:53 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2914489890' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc257cfc-3310-4d0f-bb39-ec5aed0f18f0"}]: dispatch 2024-04-25T01:47:53.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:53 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2914489890' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc257cfc-3310-4d0f-bb39-ec5aed0f18f0"}]': finished 2024-04-25T01:47:53.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:53 smithi134 ceph-mon[33098]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:47:53.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:53 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:47:53.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:53 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:54.404 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 01:47:53 smithi040 sudo[40725]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:54.405 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 01:47:53 smithi040 sudo[40725]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:54.405 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 01:47:53 smithi040 sudo[40725]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:54.405 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 01:47:53 smithi040 sudo[40725]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:54.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:47:54.572+0000 7f1f37fff700 1 -- 172.21.15.31:0/2418014754 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1f480991c0 con 0x7f1f50108000 2024-04-25T01:47:54.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:54 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1532604566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:54.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:54 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:54.905 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 01:47:54 smithi040 sudo[40728]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:54.905 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 01:47:54 smithi040 sudo[40728]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:54.905 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 01:47:54 smithi040 sudo[40728]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:54.905 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 01:47:54 smithi040 sudo[40728]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:54.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:54 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1532604566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:54.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:54 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:54.934 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 01:47:54 smithi134 sudo[41855]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:54.934 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 01:47:54 smithi134 sudo[41855]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:54.934 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 01:47:54 smithi134 sudo[41855]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:54.934 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 01:47:54 smithi134 sudo[41855]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:54.982 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:54 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1532604566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:54.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:54 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:55.434 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 01:47:54 smithi134 sudo[41858]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:55.434 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 01:47:54 smithi134 sudo[41858]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:55.434 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 01:47:54 smithi134 sudo[41858]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:55.434 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 01:47:54 smithi134 sudo[41858]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:55.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 sudo[48470]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:55.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 sudo[48470]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:55.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 sudo[48470]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:55.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 sudo[48470]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:56.151 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:55 smithi134 ceph-mon[33098]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:56.151 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:55 smithi134 ceph-mon[33098]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:47:56.151 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:56.151 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:55 smithi134 ceph-mon[33098]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:56.151 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:56 smithi134 sudo[41861]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:56.151 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:56 smithi134 sudo[41861]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:56.151 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:56 smithi134 sudo[41861]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:56.151 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:56 smithi134 sudo[41861]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:56.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:55 smithi040 ceph-mon[32418]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:56.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:55 smithi040 ceph-mon[32418]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:47:56.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:56.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:55 smithi040 ceph-mon[32418]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:56.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 sudo[48839]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:56.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 sudo[48839]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:56.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 sudo[48839]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:56.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 sudo[48839]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:56.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 ceph-mon[29204]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:56.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 ceph-mon[29204]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:47:56.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:47:56.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:55 smithi031 ceph-mon[29204]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:56.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:56 smithi134 sudo[41864]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:56.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:56 smithi134 sudo[41864]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:56.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:56 smithi134 sudo[41864]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:56.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:56 smithi134 sudo[41864]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:57.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:56 smithi040 sudo[40731]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:57.154 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:56 smithi040 sudo[40731]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:57.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:56 smithi040 sudo[40731]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:57.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:56 smithi040 sudo[40731]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:57.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:57.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:57.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:57.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:57.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:57.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:57.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:57.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:57.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:57.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:57.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:57.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:57.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 sudo[40734]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 sudo[40734]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 sudo[40734]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 sudo[40734]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:57.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:57.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:57.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:57.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:57.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:57.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:57.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:57.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:57.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:57.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:57.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:57.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:57.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:57.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:57.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:57.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:57.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:58.604 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:58 smithi031 ceph-mon[29204]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:58.604 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:58 smithi031 ceph-mon[29204]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:58.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:58 smithi040 ceph-mon[32418]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:58.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:58 smithi040 ceph-mon[32418]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:58.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:58 smithi134 ceph-mon[33098]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:58.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:58 smithi134 ceph-mon[33098]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:59.654 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:47:59 smithi040 ceph-mon[32418]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:59.656 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:47:59 smithi031 ceph-mon[29204]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:59.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:47:59 smithi134 ceph-mon[33098]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:48:01.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:01 smithi040 ceph-mon[32418]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:48:01.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:48:01.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:01.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:01 smithi031 ceph-mon[29204]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:48:01.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:48:01.663 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:01 smithi134 ceph-mon[33098]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:48:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:48:01.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:02.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:02 smithi040 ceph-mon[32418]: Deploying daemon osd.3 on smithi031 2024-04-25T01:48:02.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:02 smithi134 ceph-mon[33098]: Deploying daemon osd.3 on smithi031 2024-04-25T01:48:02.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:02 smithi031 ceph-mon[29204]: Deploying daemon osd.3 on smithi031 2024-04-25T01:48:03.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:03 smithi031 ceph-mon[29204]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-04-25T01:48:03.585 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:03 smithi134 ceph-mon[33098]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-04-25T01:48:03.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:03 smithi040 ceph-mon[32418]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-04-25T01:48:05.655 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:05 smithi031 ceph-mon[29204]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-04-25T01:48:05.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:05 smithi040 ceph-mon[32418]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-04-25T01:48:05.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:05 smithi134 ceph-mon[33098]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-04-25T01:48:07.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:07 smithi040 ceph-mon[32418]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-04-25T01:48:07.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:07 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:07.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:07 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:07.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:07 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:07.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:07 smithi134 ceph-mon[33098]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-04-25T01:48:07.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:07 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:07.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:07 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:07.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:07 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:07.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:07 smithi031 ceph-mon[29204]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-04-25T01:48:07.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:07 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:07.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:07 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:07.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:07 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:09.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:09 smithi031 ceph-mon[29204]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:48:09.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:09 smithi040 ceph-mon[32418]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:48:09.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:09 smithi134 ceph-mon[33098]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:48:10.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.520+0000 7f1f37fff700 1 -- 172.21.15.31:0/2418014754 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1f50062670 con 0x7f1f380a2b90 2024-04-25T01:48:10.517 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 3 on host 'smithi031' 2024-04-25T01:48:10.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 -- 172.21.15.31:0/2418014754 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1f380a2b90 msgr2=0x7f1f380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1f380a2b90 0x7f1f380a5050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1f40005fd0 tx=0x7f1f40005dc0 comp rx=0 tx=0).stop 2024-04-25T01:48:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 -- 172.21.15.31:0/2418014754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f50108000 msgr2=0x7f1f5019f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f50108000 0x7f1f5019f980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800ced0 tx=0x7f1f4800cfb0 comp rx=0 tx=0).stop 2024-04-25T01:48:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 -- 172.21.15.31:0/2418014754 shutdown_connections 2024-04-25T01:48:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1f380a2b90 0x7f1f380a5050 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f50108000 0x7f1f5019f980 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f50102da0 0x7f1f501983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 --2- 172.21.15.31:0/2418014754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f501056d0 0x7f1f50198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 -- 172.21.15.31:0/2418014754 >> 172.21.15.31:0/2418014754 conn(0x7f1f500fc730 msgr2=0x7f1f501036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:48:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 -- 172.21.15.31:0/2418014754 shutdown_connections 2024-04-25T01:48:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:48:10.522+0000 7f1f552d1700 1 -- 172.21.15.31:0/2418014754 wait complete. 2024-04-25T01:48:10.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:10.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:10.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:10 smithi031 ceph-mon[29204]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:48:10.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:10 smithi031 ceph-mon[29204]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:48:10.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:10.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:10.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:10.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:10 smithi040 ceph-mon[32418]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:48:10.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:10 smithi040 ceph-mon[32418]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:48:10.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:10.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:10.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:10.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:10 smithi134 ceph-mon[33098]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:48:10.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:10 smithi134 ceph-mon[33098]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:48:10.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:10.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:11.205 DEBUG:teuthology.orchestra.run.smithi031:osd.3> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T01:48:11.207 INFO:tasks.cephadm:Deploying osd.4 on smithi040 with /dev/vg_nvme/lv_3... 2024-04-25T01:48:11.207 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:48:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:48:11.849 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:11 smithi040 ceph-mon[32418]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:48:11.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:11 smithi040 ceph-mon[32418]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:48:11.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:11 smithi040 ceph-mon[32418]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:48:11.850 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:11 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:11.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:11 smithi134 ceph-mon[33098]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:48:11.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:11 smithi134 ceph-mon[33098]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:48:11.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:11 smithi134 ceph-mon[33098]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:48:11.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:11 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:11.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:11 smithi031 ceph-mon[29204]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:48:11.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:11 smithi031 ceph-mon[29204]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:48:11.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:11 smithi031 ceph-mon[29204]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:48:11.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:11 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:11.984 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 01:48:11 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3[50065]: 2024-04-25T01:48:11.674+0000 7fee0f6fd700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T01:48:11.984 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 01:48:11 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3[50065]: 2024-04-25T01:48:11.681+0000 7fee0b4f0700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:48:12.707 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:12 smithi040 ceph-mon[32418]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:48:12.707 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:12 smithi040 ceph-mon[32418]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:48:12.707 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:12 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:12.707 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:12 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:12.708 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:12 smithi040 ceph-mon[32418]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:48:12.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:12 smithi134 ceph-mon[33098]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:48:12.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:12 smithi134 ceph-mon[33098]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:48:12.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:12 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:12.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:12 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:12.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:12 smithi134 ceph-mon[33098]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:48:12.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:12 smithi031 ceph-mon[29204]: from='osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:48:12.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:12 smithi031 ceph-mon[29204]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:48:12.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:12 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:12.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:12 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:12.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:12 smithi031 ceph-mon[29204]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:48:13.293 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:48:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:48:13.294 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-25T01:48:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-25T01:48:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137726 s, 761 MB/s 2024-04-25T01:48:13.294 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 2024-04-25T01:48:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-25T01:48:13.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:13 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:48:13.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:13 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:48:13.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:13.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:13 smithi031 ceph-mon[29204]: osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424] boot 2024-04-25T01:48:13.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:13 smithi031 ceph-mon[29204]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T01:48:13.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:13.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:13 smithi031 ceph-mon[29204]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:48:13.885 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:13 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:48:13.885 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:13 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:48:13.885 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:13.885 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:13 smithi040 ceph-mon[32418]: osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424] boot 2024-04-25T01:48:13.886 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:13 smithi040 ceph-mon[32418]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T01:48:13.886 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:13.886 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:13 smithi040 ceph-mon[32418]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:48:13.912 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_3 2024-04-25T01:48:13.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:13 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:48:13.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:13 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:48:13.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:13.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:13 smithi134 ceph-mon[33098]: osd.3 [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424] boot 2024-04-25T01:48:13.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:13 smithi134 ceph-mon[33098]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T01:48:13.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:13.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:13 smithi134 ceph-mon[33098]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:48:14.165 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:48:15.455 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:15.455 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi031 2024-04-25T01:48:15.455 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.456 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.456 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:15.456 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:15.456 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi031 to 8546M 2024-04-25T01:48:15.456 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.456 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:15.456 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:15.456 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.456 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:15 smithi040 ceph-mon[32418]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:48:15.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:15.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi031 2024-04-25T01:48:15.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:15.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:15.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi031 to 8546M 2024-04-25T01:48:15.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.483 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:15.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:15.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:15 smithi031 ceph-mon[29204]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:48:15.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.681+0000 7fd76e821700 1 -- 172.21.15.40:0/2145588298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd768105670 msgr2=0x7fd768107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:15.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.681+0000 7fd76e821700 1 --2- 172.21.15.40:0/2145588298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd768105670 0x7fd768107a60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009990 tx=0x7fd75c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:48:15.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.681+0000 7fd76e821700 1 -- 172.21.15.40:0/2145588298 shutdown_connections 2024-04-25T01:48:15.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.681+0000 7fd76e821700 1 --2- 172.21.15.40:0/2145588298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd768107fa0 0x7fd76810a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:15.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.681+0000 7fd76e821700 1 --2- 172.21.15.40:0/2145588298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd768102d40 0x7fd768105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:15.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.681+0000 7fd76e821700 1 --2- 172.21.15.40:0/2145588298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd768105670 0x7fd768107a60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:15.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.681+0000 7fd76e821700 1 -- 172.21.15.40:0/2145588298 >> 172.21.15.40:0/2145588298 conn(0x7fd7680fc750 msgr2=0x7fd7680feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:48:15.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.681+0000 7fd76e821700 1 -- 172.21.15.40:0/2145588298 shutdown_connections 2024-04-25T01:48:15.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.681+0000 7fd76e821700 1 -- 172.21.15.40:0/2145588298 wait complete. 2024-04-25T01:48:15.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd76e821700 1 Processor -- start 2024-04-25T01:48:15.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd76e821700 1 -- start start 2024-04-25T01:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd76e821700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd768102d40 0x7fd7681026a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd76e821700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd768105670 0x7fd768100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd76e821700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd768107fa0 0x7fd768101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd76e821700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd76810dd40 con 0x7fd768105670 2024-04-25T01:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd76e821700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd76810dbc0 con 0x7fd768102d40 2024-04-25T01:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd76e821700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd76810dec0 con 0x7fd768107fa0 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd767fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd768102d40 0x7fd7681026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd767fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd768102d40 0x7fd7681026a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33828/0 (socket says 172.21.15.40:33828) 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd767fff700 1 -- 172.21.15.40:0/3644220321 learned_addr learned my addr 172.21.15.40:0/3644220321 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.682+0000 7fd76cdbe700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd768107fa0 0x7fd768101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd767fff700 1 -- 172.21.15.40:0/3644220321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd768107fa0 msgr2=0x7fd768101230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd7677fe700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd768105670 0x7fd768100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd767fff700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd768107fa0 0x7fd768101230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd767fff700 1 -- 172.21.15.40:0/3644220321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd768105670 msgr2=0x7fd768100cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd767fff700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd768105670 0x7fd768100cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd767fff700 1 -- 172.21.15.40:0/3644220321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd768101ae0 con 0x7fd768102d40 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd767fff700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd768102d40 0x7fd7681026a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c9b0 tx=0x7fd75800cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd7657fa700 1 -- 172.21.15.40:0/3644220321 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758007b00 con 0x7fd768102d40 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd76e821700 1 -- 172.21.15.40:0/3644220321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7681a2490 con 0x7fd768102d40 2024-04-25T01:48:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd7657fa700 1 -- 172.21.15.40:0/3644220321 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd758007ca0 con 0x7fd768102d40 2024-04-25T01:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd7657fa700 1 -- 172.21.15.40:0/3644220321 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758018730 con 0x7fd768102d40 2024-04-25T01:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.683+0000 7fd76e821700 1 -- 172.21.15.40:0/3644220321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7681a29d0 con 0x7fd768102d40 2024-04-25T01:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.684+0000 7fd76e821700 1 -- 172.21.15.40:0/3644220321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd768068f20 con 0x7fd768102d40 2024-04-25T01:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.685+0000 7fd7657fa700 1 -- 172.21.15.40:0/3644220321 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd75800f450 con 0x7fd768102d40 2024-04-25T01:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.686+0000 7fd7657fa700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd7500a2c20 0x7fd7500a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.686+0000 7fd7657fa700 1 -- 172.21.15.40:0/3644220321 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7fd7580cdf80 con 0x7fd768102d40 2024-04-25T01:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.686+0000 7fd7677fe700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd7500a2c20 0x7fd7500a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.687+0000 7fd7677fe700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd7500a2c20 0x7fd7500a50e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00b670 tx=0x7fd75c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:48:15.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:15.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi031 2024-04-25T01:48:15.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:15.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:15.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi031 to 8546M 2024-04-25T01:48:15.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:15.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:15.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:15.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:15 smithi134 ceph-mon[33098]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:48:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.690+0000 7fd7657fa700 1 -- 172.21.15.40:0/3644220321 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd758014030 con 0x7fd768102d40 2024-04-25T01:48:15.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:15.906+0000 7fd76e821700 1 -- 172.21.15.40:0/3644220321 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd768062670 con 0x7fd7500a2c20 2024-04-25T01:48:16.831 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:16 smithi040 ceph-mon[32418]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:48:16.831 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:16 smithi040 ceph-mon[32418]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:16.831 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:16 smithi040 ceph-mon[32418]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:16.831 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:16.831 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:16.831 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:16.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:16 smithi134 ceph-mon[33098]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:48:16.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:16 smithi134 ceph-mon[33098]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:16.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:16 smithi134 ceph-mon[33098]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:16.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:16.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:16.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:16.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:16 smithi031 ceph-mon[29204]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:48:16.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:16 smithi031 ceph-mon[29204]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:16.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:16 smithi031 ceph-mon[29204]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:16.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:16.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:16.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:17.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:17 smithi040 ceph-mon[32418]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:48:17.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:17 smithi134 ceph-mon[33098]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:48:17.961 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:17 smithi031 ceph-mon[29204]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:48:18.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:18 smithi134 ceph-mon[33098]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-25T01:48:18.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:18 smithi031 ceph-mon[29204]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-25T01:48:19.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:18 smithi040 ceph-mon[32418]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-25T01:48:19.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:19 smithi134 ceph-mon[33098]: from='client.? 172.21.15.40:0/3911736977' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dd91e15-4315-40e0-a93a-aa3e7d5ebed3"}]: dispatch 2024-04-25T01:48:19.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:19 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dd91e15-4315-40e0-a93a-aa3e7d5ebed3"}]: dispatch 2024-04-25T01:48:19.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:19 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0dd91e15-4315-40e0-a93a-aa3e7d5ebed3"}]': finished 2024-04-25T01:48:19.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:19 smithi134 ceph-mon[33098]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T01:48:19.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:19.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:19 smithi134 ceph-mon[33098]: from='client.? 172.21.15.40:0/840218543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:19.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:19 smithi031 ceph-mon[29204]: from='client.? 172.21.15.40:0/3911736977' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dd91e15-4315-40e0-a93a-aa3e7d5ebed3"}]: dispatch 2024-04-25T01:48:19.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:19 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dd91e15-4315-40e0-a93a-aa3e7d5ebed3"}]: dispatch 2024-04-25T01:48:19.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:19 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0dd91e15-4315-40e0-a93a-aa3e7d5ebed3"}]': finished 2024-04-25T01:48:19.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:19 smithi031 ceph-mon[29204]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T01:48:19.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:19.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:19 smithi031 ceph-mon[29204]: from='client.? 172.21.15.40:0/840218543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:20.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:19 smithi040 ceph-mon[32418]: from='client.? 172.21.15.40:0/3911736977' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dd91e15-4315-40e0-a93a-aa3e7d5ebed3"}]: dispatch 2024-04-25T01:48:20.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:19 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dd91e15-4315-40e0-a93a-aa3e7d5ebed3"}]: dispatch 2024-04-25T01:48:20.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:19 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0dd91e15-4315-40e0-a93a-aa3e7d5ebed3"}]': finished 2024-04-25T01:48:20.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:19 smithi040 ceph-mon[32418]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T01:48:20.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:20.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:19 smithi040 ceph-mon[32418]: from='client.? 172.21.15.40:0/840218543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:20.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:20 smithi031 ceph-mon[29204]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T01:48:21.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:20 smithi040 ceph-mon[32418]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T01:48:21.184 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:20 smithi134 ceph-mon[33098]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T01:48:22.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:22 smithi040 ceph-mon[32418]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:48:22.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:22 smithi031 ceph-mon[29204]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:48:23.184 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:22 smithi134 ceph-mon[33098]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:48:24.942 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:24 smithi040 ceph-mon[32418]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:48:24.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:24 smithi031 ceph-mon[29204]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:48:25.184 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:24 smithi134 ceph-mon[33098]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:48:26.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:26 smithi031 ceph-mon[29204]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:26.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:48:26.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:26.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:26 smithi031 ceph-mon[29204]: Deploying daemon osd.4 on smithi040 2024-04-25T01:48:26.991 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:26 smithi040 ceph-mon[32418]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:26.991 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:48:26.991 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:26.992 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:26 smithi040 ceph-mon[32418]: Deploying daemon osd.4 on smithi040 2024-04-25T01:48:27.184 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:26 smithi134 ceph-mon[33098]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:27.185 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:48:27.185 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:27.185 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:26 smithi134 ceph-mon[33098]: Deploying daemon osd.4 on smithi040 2024-04-25T01:48:28.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:28 smithi040 ceph-mon[32418]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:28.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:28 smithi031 ceph-mon[29204]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:29.184 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:28 smithi134 ceph-mon[33098]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:30.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:30 smithi031 ceph-mon[29204]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:31.030 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:30 smithi040 ceph-mon[32418]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:31.184 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:30 smithi134 ceph-mon[33098]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:33.155 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:32 smithi040 ceph-mon[32418]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:33.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:33.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:33.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:33.184 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:32 smithi134 ceph-mon[33098]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:33.184 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:33.185 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:33.185 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:33.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:32 smithi031 ceph-mon[29204]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:33.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:33.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:33.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:34.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:34 smithi040 ceph-mon[32418]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:35.184 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:34 smithi134 ceph-mon[33098]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:35.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:34 smithi031 ceph-mon[29204]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:35.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.381+0000 7fd7657fa700 1 -- 172.21.15.40:0/3644220321 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd768062670 con 0x7fd7500a2c20 2024-04-25T01:48:35.379 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 4 on host 'smithi040' 2024-04-25T01:48:35.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 -- 172.21.15.40:0/3644220321 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd7500a2c20 msgr2=0x7fd7500a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:35.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd7500a2c20 0x7fd7500a50e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00b670 tx=0x7fd75c005fb0 comp rx=0 tx=0).stop 2024-04-25T01:48:35.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 -- 172.21.15.40:0/3644220321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd768102d40 msgr2=0x7fd7681026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:35.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd768102d40 0x7fd7681026a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c9b0 tx=0x7fd75800cd70 comp rx=0 tx=0).stop 2024-04-25T01:48:35.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 -- 172.21.15.40:0/3644220321 shutdown_connections 2024-04-25T01:48:35.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd7500a2c20 0x7fd7500a50e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:35.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd768107fa0 0x7fd768101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:35.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd768102d40 0x7fd7681026a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:35.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 --2- 172.21.15.40:0/3644220321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd768105670 0x7fd768100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:35.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 -- 172.21.15.40:0/3644220321 >> 172.21.15.40:0/3644220321 conn(0x7fd7680fc750 msgr2=0x7fd768103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:48:35.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 -- 172.21.15.40:0/3644220321 shutdown_connections 2024-04-25T01:48:35.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:48:35.383+0000 7fd76e821700 1 -- 172.21.15.40:0/3644220321 wait complete. 2024-04-25T01:48:36.032 DEBUG:teuthology.orchestra.run.smithi040:osd.4> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T01:48:36.034 INFO:tasks.cephadm:Deploying osd.5 on smithi134 with /dev/vg_nvme/lv_3... 2024-04-25T01:48:36.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:48:36.277 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:48:36.608 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:36 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.608 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:36 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.608 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:36 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.608 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:36 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.608 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:36 smithi040 ceph-mon[32418]: from='osd.4 [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:36.608 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:36 smithi040 ceph-mon[32418]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:36.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:36 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:36 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:36 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:36 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:36 smithi134 ceph-mon[33098]: from='osd.4 [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:36.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:36 smithi134 ceph-mon[33098]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:36.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:36 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:36 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:36 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:36 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:36.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:36 smithi031 ceph-mon[29204]: from='osd.4 [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:36.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:36 smithi031 ceph-mon[29204]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:37.607 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:37 smithi040 ceph-mon[32418]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:37.607 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:37 smithi040 ceph-mon[32418]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:48:37.607 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:37 smithi040 ceph-mon[32418]: from='osd.4 [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:48:37.607 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:37 smithi040 ceph-mon[32418]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:48:37.607 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:37 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:37.607 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:37 smithi040 ceph-mon[32418]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:48:37.607 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 01:48:37 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4[43546]: 2024-04-25T01:48:37.431+0000 7f8d4cb00700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T01:48:37.608 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 01:48:37 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4[43546]: 2024-04-25T01:48:37.439+0000 7f8d488f3700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:48:37.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:37 smithi134 ceph-mon[33098]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:37.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:37 smithi134 ceph-mon[33098]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:48:37.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:37 smithi134 ceph-mon[33098]: from='osd.4 [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:48:37.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:37 smithi134 ceph-mon[33098]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:48:37.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:37 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:37.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:37 smithi134 ceph-mon[33098]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:48:37.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:37 smithi031 ceph-mon[29204]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:37.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:37 smithi031 ceph-mon[29204]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:48:37.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:37 smithi031 ceph-mon[29204]: from='osd.4 [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:48:37.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:37 smithi031 ceph-mon[29204]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:48:37.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:37 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:37.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:37 smithi031 ceph-mon[29204]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:48:38.489 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:48:38.489 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:48:38.489 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-04-25T01:48:38.489 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-04-25T01:48:38.489 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123842 s, 847 MB/s 2024-04-25T01:48:38.489 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-04-25T01:48:38.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:38 smithi134 ceph-mon[33098]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:48:38.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:38 smithi134 ceph-mon[33098]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:48:38.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:38.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:38.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:38.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:38 smithi031 ceph-mon[29204]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:48:38.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:38 smithi031 ceph-mon[29204]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:48:38.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:38.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:38.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:38.785 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:38 smithi040 ceph-mon[32418]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:48:38.785 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:38 smithi040 ceph-mon[32418]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:48:38.785 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:38.785 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:38.785 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:39.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi134:vg_nvme/lv_3 2024-04-25T01:48:39.426 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:48:39.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:48:39.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:48:39.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:39.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: osd.4 [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093] boot 2024-04-25T01:48:39.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T01:48:39.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:39.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:39.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:39.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:39.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:39.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:39.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:39.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:39.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:48:39.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:48:39.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:39.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: osd.4 [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093] boot 2024-04-25T01:48:39.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T01:48:39.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:39.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:39.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:39.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:39.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:39.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:39.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:39.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:39.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:48:39.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:48:39.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:39.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: osd.4 [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093] boot 2024-04-25T01:48:39.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T01:48:39.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:39.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:39.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:39.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:39.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:39.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:39.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:39.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:39.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:39.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:40.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:40 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi040 2024-04-25T01:48:40.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:40 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi040 to 8546M 2024-04-25T01:48:40.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:40 smithi134 ceph-mon[33098]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:48:40.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:40 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi040 2024-04-25T01:48:40.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:40 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi040 to 8546M 2024-04-25T01:48:40.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:40 smithi031 ceph-mon[29204]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:48:40.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:40 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi040 2024-04-25T01:48:40.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:40 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi040 to 8546M 2024-04-25T01:48:40.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:40 smithi040 ceph-mon[32418]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:48:40.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.912+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2299050101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e841022b0 msgr2=0x7f2e841026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:40.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.912+0000 7f2e88f0d700 1 --2- 172.21.15.134:0/2299050101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e841022b0 0x7f2e841026d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009990 tx=0x7f2e6c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:48:40.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.912+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2299050101 shutdown_connections 2024-04-25T01:48:40.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.912+0000 7f2e88f0d700 1 --2- 172.21.15.134:0/2299050101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e84103e50 0x7f2e8410a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:40.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.912+0000 7f2e88f0d700 1 --2- 172.21.15.134:0/2299050101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e841022b0 0x7f2e841026d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:40.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.912+0000 7f2e88f0d700 1 --2- 172.21.15.134:0/2299050101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e841034b0 0x7f2e84103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:40.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.912+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2299050101 >> 172.21.15.134:0/2299050101 conn(0x7f2e840fd850 msgr2=0x7f2e840ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:48:40.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.912+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2299050101 shutdown_connections 2024-04-25T01:48:40.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.912+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2299050101 wait complete. 2024-04-25T01:48:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e88f0d700 1 Processor -- start 2024-04-25T01:48:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e88f0d700 1 -- start start 2024-04-25T01:48:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e88f0d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e841022b0 0x7f2e841984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e88f0d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e841034b0 0x7f2e841989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e88f0d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e84103e50 0x7f2e8419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e88f0d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8410de10 con 0x7f2e84103e50 2024-04-25T01:48:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e88f0d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8410dc90 con 0x7f2e841034b0 2024-04-25T01:48:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e88f0d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8410df90 con 0x7f2e841022b0 2024-04-25T01:48:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e8259c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e841022b0 0x7f2e841984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e8259c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e841022b0 0x7f2e841984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:33512/0 (socket says 172.21.15.134:33512) 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e8259c700 1 -- 172.21.15.134:0/2140977072 learned_addr learned my addr 172.21.15.134:0/2140977072 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e82d9d700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e84103e50 0x7f2e8419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e81d9b700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e841034b0 0x7f2e841989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e8259c700 1 -- 172.21.15.134:0/2140977072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e841034b0 msgr2=0x7f2e841989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e8259c700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e841034b0 0x7f2e841989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e8259c700 1 -- 172.21.15.134:0/2140977072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e84103e50 msgr2=0x7f2e8419fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e8259c700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e84103e50 0x7f2e8419fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e8259c700 1 -- 172.21.15.134:0/2140977072 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e841a0110 con 0x7f2e841022b0 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.913+0000 7f2e82d9d700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e84103e50 0x7f2e8419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.914+0000 7f2e8259c700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e841022b0 0x7f2e841984b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0037f0 tx=0x7f2e6c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.914+0000 7f2e7b7fe700 1 -- 172.21.15.134:0/2140977072 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c003c90 con 0x7f2e841022b0 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.914+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2140977072 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e841a03a0 con 0x7f2e841022b0 2024-04-25T01:48:40.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.914+0000 7f2e7b7fe700 1 -- 172.21.15.134:0/2140977072 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e6c004830 con 0x7f2e841022b0 2024-04-25T01:48:40.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.914+0000 7f2e7b7fe700 1 -- 172.21.15.134:0/2140977072 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c0054e0 con 0x7f2e841022b0 2024-04-25T01:48:40.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.914+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2140977072 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e841a0820 con 0x7f2e841022b0 2024-04-25T01:48:40.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.915+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2140977072 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e84068f20 con 0x7f2e841022b0 2024-04-25T01:48:40.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.916+0000 7f2e7b7fe700 1 -- 172.21.15.134:0/2140977072 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2e6c0049e0 con 0x7f2e841022b0 2024-04-25T01:48:40.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.916+0000 7f2e7b7fe700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e700a2c20 0x7f2e700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:40.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.917+0000 7f2e7b7fe700 1 -- 172.21.15.134:0/2140977072 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f2e6c0cf4c0 con 0x7f2e841022b0 2024-04-25T01:48:40.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.917+0000 7f2e81d9b700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e700a2c20 0x7f2e700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:40.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.917+0000 7f2e81d9b700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e700a2c20 0x7f2e700a50e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2e74005950 tx=0x7f2e740058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:48:40.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:40.920+0000 7f2e7b7fe700 1 -- 172.21.15.134:0/2140977072 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2e6c013070 con 0x7f2e841022b0 2024-04-25T01:48:41.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:48:41.122+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2140977072 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2e84062670 con 0x7f2e700a2c20 2024-04-25T01:48:41.727 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:41 smithi134 ceph-mon[33098]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:41.727 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:41 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:41.728 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:41 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:41.728 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:41 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:41.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:41 smithi031 ceph-mon[29204]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:41.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:41 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:41.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:41 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:41.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:41 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:41.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:41 smithi040 ceph-mon[32418]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:41.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:41 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:41.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:41 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:41.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:41 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:42.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:42 smithi134 ceph-mon[33098]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:42.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:42 smithi031 ceph-mon[29204]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:42.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:42 smithi040 ceph-mon[32418]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:43.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:43 smithi031 ceph-mon[29204]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:43.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:43 smithi040 ceph-mon[32418]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:43.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:43 smithi134 ceph-mon[33098]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:44 smithi031 ceph-mon[29204]: from='client.? 172.21.15.134:0/2931439585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e992053c-b5dd-41cc-a72f-c371eadc508d"}]: dispatch 2024-04-25T01:48:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:44 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e992053c-b5dd-41cc-a72f-c371eadc508d"}]: dispatch 2024-04-25T01:48:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:44 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e992053c-b5dd-41cc-a72f-c371eadc508d"}]': finished 2024-04-25T01:48:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:44 smithi031 ceph-mon[29204]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T01:48:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:44 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:44.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:44 smithi040 ceph-mon[32418]: from='client.? 172.21.15.134:0/2931439585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e992053c-b5dd-41cc-a72f-c371eadc508d"}]: dispatch 2024-04-25T01:48:44.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:44 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e992053c-b5dd-41cc-a72f-c371eadc508d"}]: dispatch 2024-04-25T01:48:44.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:44 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e992053c-b5dd-41cc-a72f-c371eadc508d"}]': finished 2024-04-25T01:48:44.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:44 smithi040 ceph-mon[32418]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T01:48:44.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:44 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:44.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:44 smithi134 ceph-mon[33098]: from='client.? 172.21.15.134:0/2931439585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e992053c-b5dd-41cc-a72f-c371eadc508d"}]: dispatch 2024-04-25T01:48:44.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:44 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e992053c-b5dd-41cc-a72f-c371eadc508d"}]: dispatch 2024-04-25T01:48:44.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:44 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e992053c-b5dd-41cc-a72f-c371eadc508d"}]': finished 2024-04-25T01:48:44.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:44 smithi134 ceph-mon[33098]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T01:48:44.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:44 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:45.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:45 smithi031 ceph-mon[29204]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:45.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:45 smithi031 ceph-mon[29204]: from='client.? 172.21.15.134:0/3083760603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:45.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:45 smithi040 ceph-mon[32418]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:45.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:45 smithi040 ceph-mon[32418]: from='client.? 172.21.15.134:0/3083760603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:45.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:45 smithi134 ceph-mon[33098]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:45.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:45 smithi134 ceph-mon[33098]: from='client.? 172.21.15.134:0/3083760603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:47.655 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:47 smithi040 ceph-mon[32418]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:47.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:47 smithi031 ceph-mon[29204]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:47.821 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:47 smithi134 ceph-mon[33098]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:49.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:49 smithi134 ceph-mon[33098]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:49.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:49 smithi031 ceph-mon[29204]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:49.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:49 smithi040 ceph-mon[32418]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:51.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:51 smithi031 ceph-mon[29204]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:51.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:51 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:48:51.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:51 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:51.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:51 smithi134 ceph-mon[33098]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:51.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:51 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:48:51.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:51 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:51.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:51 smithi040 ceph-mon[32418]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:51.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:51 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:48:51.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:51 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:52.593 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:52 smithi134 ceph-mon[33098]: Deploying daemon osd.5 on smithi134 2024-04-25T01:48:52.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:52 smithi031 ceph-mon[29204]: Deploying daemon osd.5 on smithi134 2024-04-25T01:48:52.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:52 smithi040 ceph-mon[32418]: Deploying daemon osd.5 on smithi134 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:53 smithi031 ceph-mon[29204]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:53.858 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:53 smithi134 ceph-mon[33098]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:53.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:53 smithi040 ceph-mon[32418]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:55.559 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:55 smithi134 ceph-mon[33098]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:55.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:55 smithi031 ceph-mon[29204]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:55.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:55 smithi040 ceph-mon[32418]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:57.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:57 smithi040 ceph-mon[32418]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:57.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:57.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:57.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:57.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:57 smithi134 ceph-mon[33098]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:57.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:57.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:57.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:57.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:57 smithi031 ceph-mon[29204]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:57.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:57.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:57.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:48:59.905 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:48:59 smithi040 ceph-mon[32418]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:59.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:48:59 smithi134 ceph-mon[33098]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:59.983 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:48:59 smithi031 ceph-mon[29204]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:49:00.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.041+0000 7f2e7b7fe700 1 -- 172.21.15.134:0/2140977072 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2e84062670 con 0x7f2e700a2c20 2024-04-25T01:49:00.038 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 5 on host 'smithi134' 2024-04-25T01:49:00.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.043+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2140977072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e700a2c20 msgr2=0x7f2e700a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:00.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.043+0000 7f2e88f0d700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e700a2c20 0x7f2e700a50e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2e74005950 tx=0x7f2e740058e0 comp rx=0 tx=0).stop 2024-04-25T01:49:00.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.044+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2140977072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e841022b0 msgr2=0x7f2e841984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:00.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.044+0000 7f2e88f0d700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e841022b0 0x7f2e841984b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0037f0 tx=0x7f2e6c0047b0 comp rx=0 tx=0).stop 2024-04-25T01:49:00.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.044+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2140977072 shutdown_connections 2024-04-25T01:49:00.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.044+0000 7f2e88f0d700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e700a2c20 0x7f2e700a50e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:00.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.044+0000 7f2e88f0d700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e84103e50 0x7f2e8419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:00.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.044+0000 7f2e88f0d700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e841022b0 0x7f2e841984b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:00.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.044+0000 7f2e88f0d700 1 --2- 172.21.15.134:0/2140977072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e841034b0 0x7f2e841989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:00.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.044+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2140977072 >> 172.21.15.134:0/2140977072 conn(0x7f2e840fd850 msgr2=0x7f2e84106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:49:00.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.044+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2140977072 shutdown_connections 2024-04-25T01:49:00.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:00.044+0000 7f2e88f0d700 1 -- 172.21.15.134:0/2140977072 wait complete. 2024-04-25T01:49:00.639 DEBUG:teuthology.orchestra.run.smithi134:osd.5> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T01:49:00.641 INFO:tasks.cephadm:Deploying osd.6 on smithi031 with /dev/vg_nvme/lv_2... 2024-04-25T01:49:00.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:49:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:49:01.319 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:01 smithi134 ceph-mon[33098]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:49:01.319 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:01.319 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:01.319 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:01 smithi134 ceph-mon[33098]: from='osd.5 [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:49:01.319 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:01 smithi134 ceph-mon[33098]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:49:01.319 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:01.319 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:01 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:01.352 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:01 smithi031 ceph-mon[29204]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:49:01.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:01.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:01.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:01 smithi031 ceph-mon[29204]: from='osd.5 [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:49:01.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:01 smithi031 ceph-mon[29204]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:49:01.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:01.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:01 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:01.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:01 smithi040 ceph-mon[32418]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:49:01.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:01.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:01.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:01 smithi040 ceph-mon[32418]: from='osd.5 [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:49:01.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:01 smithi040 ceph-mon[32418]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:49:01.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:01.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:01 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:02.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:02 smithi031 ceph-mon[29204]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:49:02.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:02 smithi031 ceph-mon[29204]: from='osd.5 [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:49:02.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:02 smithi031 ceph-mon[29204]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:49:02.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:02.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:02 smithi031 ceph-mon[29204]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:49:02.310 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:02 smithi134 ceph-mon[33098]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:49:02.310 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:02 smithi134 ceph-mon[33098]: from='osd.5 [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:49:02.310 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:02 smithi134 ceph-mon[33098]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:49:02.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:02 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:02.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:02 smithi134 ceph-mon[33098]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:49:02.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:02 smithi040 ceph-mon[32418]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:49:02.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:02 smithi040 ceph-mon[32418]: from='osd.5 [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:49:02.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:02 smithi040 ceph-mon[32418]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:49:02.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:02 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:02.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:02 smithi040 ceph-mon[32418]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:49:02.656 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 01:49:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T01:49:02.482+0000 7fe310a74700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T01:49:02.656 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 01:49:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T01:49:02.488+0000 7fe30805e700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:49:02.855 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:49:02.855 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:49:02.855 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-25T01:49:02.856 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-25T01:49:02.856 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122879 s, 853 MB/s 2024-04-25T01:49:02.856 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-25T01:49:03.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:03 smithi040 ceph-mon[32418]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:49:03.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:03 smithi040 ceph-mon[32418]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:49:03.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:03 smithi040 ceph-mon[32418]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:49:03.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:03 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:03.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:03 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:03.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:03 smithi040 ceph-mon[32418]: from='osd.5 ' entity='osd.5' 2024-04-25T01:49:03.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:03 smithi031 ceph-mon[29204]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:49:03.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:03 smithi031 ceph-mon[29204]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:49:03.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:03 smithi031 ceph-mon[29204]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:49:03.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:03 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:03.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:03 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:03.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:03 smithi031 ceph-mon[29204]: from='osd.5 ' entity='osd.5' 2024-04-25T01:49:03.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:03 smithi134 ceph-mon[33098]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:49:03.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:03 smithi134 ceph-mon[33098]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:49:03.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:03 smithi134 ceph-mon[33098]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:49:03.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:03 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:03.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:03 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:03.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:03 smithi134 ceph-mon[33098]: from='osd.5 ' entity='osd.5' 2024-04-25T01:49:03.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_2 2024-04-25T01:49:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:49:04.177 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:04 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:49:04.177 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:04 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:49:04.177 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:04 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:04.178 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:04 smithi134 ceph-mon[33098]: osd.5 [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767] boot 2024-04-25T01:49:04.178 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:04 smithi134 ceph-mon[33098]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T01:49:04.178 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:04 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:04.178 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:04 smithi134 ceph-mon[33098]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T01:49:04.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:04 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:49:04.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:04 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:49:04.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:04 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:04.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:04 smithi040 ceph-mon[32418]: osd.5 [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767] boot 2024-04-25T01:49:04.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:04 smithi040 ceph-mon[32418]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T01:49:04.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:04 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:04.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:04 smithi040 ceph-mon[32418]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T01:49:04.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:04 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:49:04.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:04 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:49:04.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:04 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:04.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:04 smithi031 ceph-mon[29204]: osd.5 [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767] boot 2024-04-25T01:49:04.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:04 smithi031 ceph-mon[29204]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T01:49:04.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:04 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:49:04.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:04 smithi031 ceph-mon[29204]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T01:49:05.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.424+0000 7f69a2686700 1 -- 172.21.15.31:0/1725408917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f699c101190 msgr2=0x7f699c1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:05.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.424+0000 7f69a2686700 1 --2- 172.21.15.31:0/1725408917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f699c101190 0x7f699c1015b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f698c0099c0 tx=0x7f698c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:49:05.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.424+0000 7f69a2686700 1 -- 172.21.15.31:0/1725408917 shutdown_connections 2024-04-25T01:49:05.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.424+0000 7f69a2686700 1 --2- 172.21.15.31:0/1725408917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699c102d30 0x7f699c1095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:05.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.424+0000 7f69a2686700 1 --2- 172.21.15.31:0/1725408917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f699c101190 0x7f699c1015b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:05.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.424+0000 7f69a2686700 1 --2- 172.21.15.31:0/1725408917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c102390 0x7f699c1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:05.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.424+0000 7f69a2686700 1 -- 172.21.15.31:0/1725408917 >> 172.21.15.31:0/1725408917 conn(0x7f699c0fc730 msgr2=0x7f699c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:49:05.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.424+0000 7f69a2686700 1 -- 172.21.15.31:0/1725408917 shutdown_connections 2024-04-25T01:49:05.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.424+0000 7f69a2686700 1 -- 172.21.15.31:0/1725408917 wait complete. 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi134 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi134 to 8544M 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:05.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:05.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:05.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:05 smithi031 ceph-mon[29204]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:49:05.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f69a2686700 1 Processor -- start 2024-04-25T01:49:05.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f69a2686700 1 -- start start 2024-04-25T01:49:05.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f69a2686700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699c101190 0x7f699c198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:05.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f69a2686700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c102390 0x7f699c19b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f69a2686700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f699c102d30 0x7f699c19b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f69a2686700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f699c10cc50 con 0x7f699c102d30 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f69a2686700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f699c10cad0 con 0x7f699c101190 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f69a2686700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f699c10cdd0 con 0x7f699c102390 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f69a0c23700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f699c102d30 0x7f699c19b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f69a0c23700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f699c102d30 0x7f699c19b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45576/0 (socket says 172.21.15.31:45576) 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f69a0c23700 1 -- 172.21.15.31:0/2441168579 learned_addr learned my addr 172.21.15.31:0/2441168579 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f6993fff700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c102390 0x7f699c19b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.425+0000 7f699bfff700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699c101190 0x7f699c198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f69a0c23700 1 -- 172.21.15.31:0/2441168579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c102390 msgr2=0x7f699c19b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f69a0c23700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c102390 0x7f699c19b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f69a0c23700 1 -- 172.21.15.31:0/2441168579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699c101190 msgr2=0x7f699c198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f69a0c23700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699c101190 0x7f699c198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:05.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f69a0c23700 1 -- 172.21.15.31:0/2441168579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699c19be00 con 0x7f699c102d30 2024-04-25T01:49:05.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f6993fff700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c102390 0x7f699c19b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:49:05.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f69a0c23700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f699c102d30 0x7f699c19b570 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6994007ee0 tx=0x7f699400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:49:05.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f6999ffb700 1 -- 172.21.15.31:0/2441168579 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6994004bb0 con 0x7f699c102d30 2024-04-25T01:49:05.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f6999ffb700 1 -- 172.21.15.31:0/2441168579 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6994004d50 con 0x7f699c102d30 2024-04-25T01:49:05.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f6999ffb700 1 -- 172.21.15.31:0/2441168579 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69940058e0 con 0x7f699c102d30 2024-04-25T01:49:05.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f69a2686700 1 -- 172.21.15.31:0/2441168579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f699c19c0f0 con 0x7f699c102d30 2024-04-25T01:49:05.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.426+0000 7f69a2686700 1 -- 172.21.15.31:0/2441168579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f699c19c5e0 con 0x7f699c102d30 2024-04-25T01:49:05.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.427+0000 7f69a2686700 1 -- 172.21.15.31:0/2441168579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f699c068f20 con 0x7f699c102d30 2024-04-25T01:49:05.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.429+0000 7f6999ffb700 1 -- 172.21.15.31:0/2441168579 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f699401a020 con 0x7f699c102d30 2024-04-25T01:49:05.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.430+0000 7f6999ffb700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69880a2ca0 0x7f69880a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:05.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.430+0000 7f699bfff700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69880a2ca0 0x7f69880a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:05.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.430+0000 7f6999ffb700 1 -- 172.21.15.31:0/2441168579 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f69940cdf60 con 0x7f699c102d30 2024-04-25T01:49:05.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.430+0000 7f699bfff700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69880a2ca0 0x7f69880a5160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f698c000c00 tx=0x7f698c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:49:05.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.433+0000 7f6999ffb700 1 -- 172.21.15.31:0/2441168579 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6994010070 con 0x7f699c102d30 2024-04-25T01:49:05.596 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:49:05.596 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi134 2024-04-25T01:49:05.596 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.596 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.596 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:05.596 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:05.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi134 to 8544M 2024-04-25T01:49:05.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:05.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:05.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:05.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:05.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:05.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:05 smithi134 ceph-mon[33098]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:49:05.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:05.649+0000 7f69a2686700 1 -- 172.21.15.31:0/2441168579 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f699c19c840 con 0x7f69880a2ca0 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi134 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi134 to 8544M 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:05.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:05.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:05.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:05.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:05 smithi040 ceph-mon[32418]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:49:06.501 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:06 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:06.501 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:06 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:06.501 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:06 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:06.502 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:06 smithi031 ceph-mon[29204]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:49:06.600 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:06 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:06.600 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:06 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:06.600 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:06 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:06.600 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:06 smithi040 ceph-mon[32418]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:49:06.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:06 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:06.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:06 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:06.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:06 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:06.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:06 smithi134 ceph-mon[33098]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:49:07.516 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:07 smithi031 ceph-mon[29204]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:07.516 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:07 smithi031 ceph-mon[29204]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:07.516 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:07 smithi031 ceph-mon[29204]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:49:07.567 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:07 smithi040 ceph-mon[32418]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:07.567 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:07 smithi040 ceph-mon[32418]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:07.567 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:07 smithi040 ceph-mon[32418]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:49:07.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:07 smithi134 ceph-mon[33098]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:07.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:07 smithi134 ceph-mon[33098]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:07.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:07 smithi134 ceph-mon[33098]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:49:09.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:09 smithi040 ceph-mon[32418]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-25T01:49:09.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:09 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/217378660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c03259a-375f-482d-be6e-410eed785c7d"}]: dispatch 2024-04-25T01:49:09.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:09 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/217378660' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c03259a-375f-482d-be6e-410eed785c7d"}]': finished 2024-04-25T01:49:09.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:09 smithi040 ceph-mon[32418]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T01:49:09.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:09 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:09.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:09 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/187099811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:09.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:09 smithi134 ceph-mon[33098]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-25T01:49:09.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:09 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/217378660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c03259a-375f-482d-be6e-410eed785c7d"}]: dispatch 2024-04-25T01:49:09.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:09 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/217378660' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c03259a-375f-482d-be6e-410eed785c7d"}]': finished 2024-04-25T01:49:09.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:09 smithi134 ceph-mon[33098]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T01:49:09.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:09 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:09.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:09 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/187099811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:09.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:09 smithi031 ceph-mon[29204]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-25T01:49:09.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:09 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/217378660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c03259a-375f-482d-be6e-410eed785c7d"}]: dispatch 2024-04-25T01:49:09.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:09 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/217378660' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c03259a-375f-482d-be6e-410eed785c7d"}]': finished 2024-04-25T01:49:09.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:09 smithi031 ceph-mon[29204]: osdmap e43: 7 total, 6 up, 7 in 2024-04-25T01:49:09.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:09 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:09.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:09 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/187099811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:11.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:11 smithi040 ceph-mon[32418]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T01:49:11.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:11 smithi134 ceph-mon[33098]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T01:49:11.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:11 smithi031 ceph-mon[29204]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T01:49:13.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:13 smithi040 ceph-mon[32418]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:49:13.671 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:13 smithi031 ceph-mon[29204]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:49:13.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:13 smithi134 ceph-mon[33098]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:49:15.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:15 smithi040 ceph-mon[32418]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:49:15.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:15 smithi134 ceph-mon[33098]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:49:15.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:15 smithi031 ceph-mon[29204]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:49:16.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:49:16.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:16.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:49:16.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:16.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:49:16.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:17.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:17 smithi040 ceph-mon[32418]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:17.406 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:17 smithi040 ceph-mon[32418]: Deploying daemon osd.6 on smithi031 2024-04-25T01:49:17.470 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:17 smithi031 ceph-mon[29204]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:17.471 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:17 smithi031 ceph-mon[29204]: Deploying daemon osd.6 on smithi031 2024-04-25T01:49:17.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:17 smithi134 ceph-mon[33098]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:17.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:17 smithi134 ceph-mon[33098]: Deploying daemon osd.6 on smithi031 2024-04-25T01:49:19.647 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:19 smithi031 ceph-mon[29204]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:19.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:19 smithi040 ceph-mon[32418]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:19.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:19 smithi134 ceph-mon[33098]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:21.484 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:21 smithi031 ceph-mon[29204]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:21.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:21 smithi040 ceph-mon[32418]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:21.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:21 smithi134 ceph-mon[33098]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:22.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:22 smithi031 ceph-mon[29204]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:22.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:22 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:22.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:22 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:22.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:22 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:23.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:22 smithi040 ceph-mon[32418]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:23.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:22 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:23.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:22 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:23.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:22 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:23.185 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:22 smithi134 ceph-mon[33098]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:23.185 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:22 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:23.185 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:22 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:23.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:22 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:24.943 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:24 smithi031 ceph-mon[29204]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:25.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:24 smithi040 ceph-mon[32418]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:25.185 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:24 smithi134 ceph-mon[33098]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:25.450 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 6 on host 'smithi031' 2024-04-25T01:49:25.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.452+0000 7f6999ffb700 1 -- 172.21.15.31:0/2441168579 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f699c19c840 con 0x7f69880a2ca0 2024-04-25T01:49:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 -- 172.21.15.31:0/2441168579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69880a2ca0 msgr2=0x7f69880a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:25.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69880a2ca0 0x7f69880a5160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f698c000c00 tx=0x7f698c005fb0 comp rx=0 tx=0).stop 2024-04-25T01:49:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 -- 172.21.15.31:0/2441168579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f699c102d30 msgr2=0x7f699c19b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f699c102d30 0x7f699c19b570 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6994007ee0 tx=0x7f699400d3b0 comp rx=0 tx=0).stop 2024-04-25T01:49:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 -- 172.21.15.31:0/2441168579 shutdown_connections 2024-04-25T01:49:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69880a2ca0 0x7f69880a5160 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f699c102d30 0x7f699c19b570 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699c101190 0x7f699c198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 --2- 172.21.15.31:0/2441168579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f699c102390 0x7f699c19b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 -- 172.21.15.31:0/2441168579 >> 172.21.15.31:0/2441168579 conn(0x7f699c0fc730 msgr2=0x7f699c105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:49:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 -- 172.21.15.31:0/2441168579 shutdown_connections 2024-04-25T01:49:25.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:49:25.455+0000 7f69a2686700 1 -- 172.21.15.31:0/2441168579 wait complete. 2024-04-25T01:49:26.154 DEBUG:teuthology.orchestra.run.smithi031:osd.6> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T01:49:26.156 INFO:tasks.cephadm:Deploying osd.7 on smithi040 with /dev/vg_nvme/lv_2... 2024-04-25T01:49:26.157 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:49:26.414 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:49:26.537 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.537 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.537 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.537 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.537 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:26 smithi040 ceph-mon[32418]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:49:26.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.685 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:26 smithi134 ceph-mon[33098]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:49:26.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:26.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:26 smithi031 ceph-mon[29204]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:49:27.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:27 smithi031 ceph-mon[29204]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:27.690 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:27 smithi031 ceph-mon[29204]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:49:27.690 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:27 smithi031 ceph-mon[29204]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T01:49:27.690 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:27 smithi031 ceph-mon[29204]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:49:27.690 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:27 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:27.828 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:27 smithi040 ceph-mon[32418]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:27.828 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:27 smithi040 ceph-mon[32418]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:49:27.828 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:27 smithi040 ceph-mon[32418]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T01:49:27.828 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:27 smithi040 ceph-mon[32418]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:49:27.828 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:27 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:27.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:27 smithi134 ceph-mon[33098]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:27.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:27 smithi134 ceph-mon[33098]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:49:27.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:27 smithi134 ceph-mon[33098]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T01:49:27.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:27 smithi134 ceph-mon[33098]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:49:27.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:27 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:28.485 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 01:49:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T01:49:27.999+0000 7f256829f700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T01:49:28.485 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 01:49:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T01:49:28.008+0000 7f256008a700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:49:28.686 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:28 smithi040 ceph-mon[32418]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:49:28.687 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:28 smithi040 ceph-mon[32418]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:49:28.687 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:28 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:28.687 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:28 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:28.687 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:28 smithi040 ceph-mon[32418]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' 2024-04-25T01:49:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:49:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:49:28.757 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-25T01:49:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-25T01:49:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012422 s, 844 MB/s 2024-04-25T01:49:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-25T01:49:28.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:28 smithi031 ceph-mon[29204]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:49:28.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:28 smithi031 ceph-mon[29204]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:49:28.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:28 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:28.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:28 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:28.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:28 smithi031 ceph-mon[29204]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' 2024-04-25T01:49:28.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:28 smithi134 ceph-mon[33098]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:49:28.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:28 smithi134 ceph-mon[33098]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:49:28.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:28 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:28.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:28 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:28.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:28 smithi134 ceph-mon[33098]: from='osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542]' entity='osd.6' 2024-04-25T01:49:29.463 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_2 2024-04-25T01:49:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:49:29.834 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:49:29.834 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:49:29.834 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:29.834 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:29.835 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542] boot 2024-04-25T01:49:29.835 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T01:49:29.835 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:29.835 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.835 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.835 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.836 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.836 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.836 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.836 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:29.836 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:29.836 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.836 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:29.837 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:29.837 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:29.837 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:49:29.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:49:29.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:29.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:29.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542] boot 2024-04-25T01:49:29.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T01:49:29.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:29.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:29.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:29.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:29.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:29.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:29.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: osd.6 [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542] boot 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: osdmap e46: 7 total, 7 up, 7 in 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:29.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:29.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:29.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:29.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:29.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:30.851 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:30 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi031 2024-04-25T01:49:30.852 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:30 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi031 to 5697M 2024-04-25T01:49:30.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:30 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi031 2024-04-25T01:49:30.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:30 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi031 to 5697M 2024-04-25T01:49:30.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:30 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi031 2024-04-25T01:49:30.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:30 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi031 to 5697M 2024-04-25T01:49:31.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.412+0000 7f2e626e4700 1 -- 172.21.15.40:0/3773510678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e5c1034b0 msgr2=0x7f2e5c103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:31.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.412+0000 7f2e626e4700 1 --2- 172.21.15.40:0/3773510678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e5c1034b0 0x7f2e5c103910 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0099c0 tx=0x7f2e4c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:49:31.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.412+0000 7f2e626e4700 1 -- 172.21.15.40:0/3773510678 shutdown_connections 2024-04-25T01:49:31.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.412+0000 7f2e626e4700 1 --2- 172.21.15.40:0/3773510678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c103e50 0x7f2e5c10a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:31.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.412+0000 7f2e626e4700 1 --2- 172.21.15.40:0/3773510678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5c1022b0 0x7f2e5c1026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:31.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.412+0000 7f2e626e4700 1 --2- 172.21.15.40:0/3773510678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e5c1034b0 0x7f2e5c103910 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.412+0000 7f2e626e4700 1 -- 172.21.15.40:0/3773510678 >> 172.21.15.40:0/3773510678 conn(0x7f2e5c0fd850 msgr2=0x7f2e5c0ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.412+0000 7f2e626e4700 1 -- 172.21.15.40:0/3773510678 shutdown_connections 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.412+0000 7f2e626e4700 1 -- 172.21.15.40:0/3773510678 wait complete. 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.412+0000 7f2e626e4700 1 Processor -- start 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.412+0000 7f2e626e4700 1 -- start start 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e626e4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c1022b0 0x7f2e5c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e626e4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e5c1034b0 0x7f2e5c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e626e4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5c103e50 0x7f2e5c1a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e626e4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5c10dd30 con 0x7f2e5c1022b0 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e626e4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5c10dbb0 con 0x7f2e5c1034b0 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e626e4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5c10deb0 con 0x7f2e5c103e50 2024-04-25T01:49:31.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e5b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e5c1034b0 0x7f2e5c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e5b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e5c1034b0 0x7f2e5c19aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45700/0 (socket says 172.21.15.40:45700) 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e5b7fe700 1 -- 172.21.15.40:0/2290888781 learned_addr learned my addr 172.21.15.40:0/2290888781 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e5bfff700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c1022b0 0x7f2e5c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e60c81700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5c103e50 0x7f2e5c1a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e5b7fe700 1 -- 172.21.15.40:0/2290888781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5c103e50 msgr2=0x7f2e5c1a1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e5b7fe700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5c103e50 0x7f2e5c1a1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e5b7fe700 1 -- 172.21.15.40:0/2290888781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c1022b0 msgr2=0x7f2e5c19a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e5b7fe700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c1022b0 0x7f2e5c19a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e5b7fe700 1 -- 172.21.15.40:0/2290888781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5c1a21f0 con 0x7f2e5c1034b0 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e60c81700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5c103e50 0x7f2e5c1a1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e5b7fe700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e5c1034b0 0x7f2e5c19aad0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0098e0 tx=0x7f2e4c003e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.413+0000 7f2e597fa700 1 -- 172.21.15.40:0/2290888781 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c009e60 con 0x7f2e5c1034b0 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.414+0000 7f2e626e4700 1 -- 172.21.15.40:0/2290888781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5c1a2480 con 0x7f2e5c1034b0 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.414+0000 7f2e597fa700 1 -- 172.21.15.40:0/2290888781 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e4c00f460 con 0x7f2e5c1034b0 2024-04-25T01:49:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.414+0000 7f2e597fa700 1 -- 172.21.15.40:0/2290888781 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c017650 con 0x7f2e5c1034b0 2024-04-25T01:49:31.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.414+0000 7f2e626e4700 1 -- 172.21.15.40:0/2290888781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e5c1a2930 con 0x7f2e5c1034b0 2024-04-25T01:49:31.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.414+0000 7f2e626e4700 1 -- 172.21.15.40:0/2290888781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e5c068f20 con 0x7f2e5c1034b0 2024-04-25T01:49:31.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.416+0000 7f2e597fa700 1 -- 172.21.15.40:0/2290888781 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2e4c00f610 con 0x7f2e5c1034b0 2024-04-25T01:49:31.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.416+0000 7f2e597fa700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e480a2c20 0x7f2e480a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:31.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.416+0000 7f2e597fa700 1 -- 172.21.15.40:0/2290888781 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f2e4c0cf220 con 0x7f2e5c1034b0 2024-04-25T01:49:31.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.417+0000 7f2e5bfff700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e480a2c20 0x7f2e480a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:31.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.417+0000 7f2e5bfff700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e480a2c20 0x7f2e480a50e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c103310 tx=0x7f2e4400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:49:31.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.420+0000 7f2e597fa700 1 -- 172.21.15.40:0/2290888781 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2e4c013070 con 0x7f2e5c1034b0 2024-04-25T01:49:31.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:31.637+0000 7f2e626e4700 1 -- 172.21.15.40:0/2290888781 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2e5c1a2c50 con 0x7f2e480a2c20 2024-04-25T01:49:31.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:31 smithi040 ceph-mon[32418]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:31.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:31 smithi040 ceph-mon[32418]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T01:49:31.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:31 smithi134 ceph-mon[33098]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:31.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:31 smithi134 ceph-mon[33098]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T01:49:31.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:31 smithi031 ceph-mon[29204]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:31.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:31 smithi031 ceph-mon[29204]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T01:49:32.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:32 smithi040 ceph-mon[32418]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:32.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:32.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:32.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:32.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:32 smithi134 ceph-mon[33098]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:32.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:32.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:32.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:32.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:32 smithi031 ceph-mon[29204]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:32.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:32.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:32.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:33.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:33 smithi040 ceph-mon[32418]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:33.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:33 smithi134 ceph-mon[33098]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:33.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:33 smithi031 ceph-mon[29204]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:35.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:35 smithi040 ceph-mon[32418]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:35.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:35 smithi040 ceph-mon[32418]: from='client.? 172.21.15.40:0/2231012069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d408419f-02da-4563-ae2a-e6b436cc4f6b"}]: dispatch 2024-04-25T01:49:35.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:35 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d408419f-02da-4563-ae2a-e6b436cc4f6b"}]: dispatch 2024-04-25T01:49:35.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:35 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d408419f-02da-4563-ae2a-e6b436cc4f6b"}]': finished 2024-04-25T01:49:35.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:35 smithi040 ceph-mon[32418]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T01:49:35.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:35 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:35.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:35 smithi040 ceph-mon[32418]: from='client.? 172.21.15.40:0/429075695' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:35.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:35 smithi134 ceph-mon[33098]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:35.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:35 smithi134 ceph-mon[33098]: from='client.? 172.21.15.40:0/2231012069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d408419f-02da-4563-ae2a-e6b436cc4f6b"}]: dispatch 2024-04-25T01:49:35.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:35 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d408419f-02da-4563-ae2a-e6b436cc4f6b"}]: dispatch 2024-04-25T01:49:35.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:35 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d408419f-02da-4563-ae2a-e6b436cc4f6b"}]': finished 2024-04-25T01:49:35.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:35 smithi134 ceph-mon[33098]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T01:49:35.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:35 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:35.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:35 smithi134 ceph-mon[33098]: from='client.? 172.21.15.40:0/429075695' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:35.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:35 smithi031 ceph-mon[29204]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:35.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:35 smithi031 ceph-mon[29204]: from='client.? 172.21.15.40:0/2231012069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d408419f-02da-4563-ae2a-e6b436cc4f6b"}]: dispatch 2024-04-25T01:49:35.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:35 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d408419f-02da-4563-ae2a-e6b436cc4f6b"}]: dispatch 2024-04-25T01:49:35.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:35 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d408419f-02da-4563-ae2a-e6b436cc4f6b"}]': finished 2024-04-25T01:49:35.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:35 smithi031 ceph-mon[29204]: osdmap e48: 8 total, 7 up, 8 in 2024-04-25T01:49:35.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:35 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:35.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:35 smithi031 ceph-mon[29204]: from='client.? 172.21.15.40:0/429075695' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:37.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:37 smithi040 ceph-mon[32418]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:37.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:37 smithi134 ceph-mon[33098]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:37.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:37 smithi031 ceph-mon[29204]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:39.875 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:39 smithi040 ceph-mon[32418]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:39.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:39 smithi134 ceph-mon[33098]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:39.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:39 smithi031 ceph-mon[29204]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:41.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:41 smithi040 ceph-mon[32418]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:41.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:41 smithi134 ceph-mon[33098]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:41.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:41 smithi031 ceph-mon[29204]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:42.887 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:49:42.888 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:42.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:49:42.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:42.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:49:42.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:43.869 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:43 smithi040 ceph-mon[32418]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:43.870 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:43 smithi040 ceph-mon[32418]: Deploying daemon osd.7 on smithi040 2024-04-25T01:49:43.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:43 smithi134 ceph-mon[33098]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:43.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:43 smithi134 ceph-mon[33098]: Deploying daemon osd.7 on smithi040 2024-04-25T01:49:43.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:43 smithi031 ceph-mon[29204]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:43.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:43 smithi031 ceph-mon[29204]: Deploying daemon osd.7 on smithi040 2024-04-25T01:49:45.656 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:45 smithi040 ceph-mon[32418]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:45.935 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:45 smithi134 ceph-mon[33098]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:45.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:45 smithi031 ceph-mon[29204]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:47.881 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:47 smithi040 ceph-mon[32418]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:47.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:47 smithi134 ceph-mon[33098]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:47.984 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:47 smithi031 ceph-mon[29204]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:49.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:48 smithi040 ceph-mon[32418]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:49.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:48 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:49.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:48 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:49.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:48 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:49.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:48 smithi134 ceph-mon[33098]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:49.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:48 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:49.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:48 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:49.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:48 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:49.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:48 smithi031 ceph-mon[29204]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:49.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:48 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:49.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:48 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:49.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:48 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:51.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:50 smithi031 ceph-mon[29204]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:51.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:50 smithi040 ceph-mon[32418]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:51.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:50 smithi134 ceph-mon[33098]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:51.506 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 7 on host 'smithi040' 2024-04-25T01:49:51.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.508+0000 7f2e597fa700 1 -- 172.21.15.40:0/2290888781 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2e5c1a2c50 con 0x7f2e480a2c20 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 -- 172.21.15.40:0/2290888781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e480a2c20 msgr2=0x7f2e480a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e480a2c20 0x7f2e480a50e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c103310 tx=0x7f2e4400a3b0 comp rx=0 tx=0).stop 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 -- 172.21.15.40:0/2290888781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e5c1034b0 msgr2=0x7f2e5c19aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e5c1034b0 0x7f2e5c19aad0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0098e0 tx=0x7f2e4c003e60 comp rx=0 tx=0).stop 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 -- 172.21.15.40:0/2290888781 shutdown_connections 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2e480a2c20 0x7f2e480a50e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5c103e50 0x7f2e5c1a1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e5c1022b0 0x7f2e5c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 --2- 172.21.15.40:0/2290888781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e5c1034b0 0x7f2e5c19aad0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 -- 172.21.15.40:0/2290888781 >> 172.21.15.40:0/2290888781 conn(0x7f2e5c0fd850 msgr2=0x7f2e5c074460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 -- 172.21.15.40:0/2290888781 shutdown_connections 2024-04-25T01:49:51.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:49:51.511+0000 7f2e626e4700 1 -- 172.21.15.40:0/2290888781 wait complete. 2024-04-25T01:49:52.190 DEBUG:teuthology.orchestra.run.smithi040:osd.7> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T01:49:52.193 INFO:tasks.cephadm:Deploying osd.8 on smithi134 with /dev/vg_nvme/lv_2... 2024-04-25T01:49:52.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:49:52.437 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:49:52.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:52 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:52.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:52 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:52.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:52 smithi134 ceph-mon[33098]: from='osd.7 [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:52.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:52 smithi134 ceph-mon[33098]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:52.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:52 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:52.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:52 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:52.901 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:52 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:52.901 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:52 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:52.901 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:52 smithi040 ceph-mon[32418]: from='osd.7 [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:52.901 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:52 smithi040 ceph-mon[32418]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:52.901 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:52 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:52.901 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:52 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:52.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:52 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:52.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:52 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:52.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:52 smithi031 ceph-mon[29204]: from='osd.7 [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:52.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:52 smithi031 ceph-mon[29204]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:52.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:52 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:52.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:52 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:53.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:53 smithi134 ceph-mon[33098]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:53.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:53 smithi134 ceph-mon[33098]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:49:53.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:53 smithi134 ceph-mon[33098]: from='osd.7 [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:49:53.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:53 smithi134 ceph-mon[33098]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T01:49:53.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:53 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:53.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:53 smithi134 ceph-mon[33098]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:49:53.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:53 smithi040 ceph-mon[32418]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:53.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:53 smithi040 ceph-mon[32418]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:49:53.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:53 smithi040 ceph-mon[32418]: from='osd.7 [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:49:53.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:53 smithi040 ceph-mon[32418]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T01:49:53.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:53 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:53.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:53 smithi040 ceph-mon[32418]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:49:53.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:53 smithi031 ceph-mon[29204]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:53.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:53 smithi031 ceph-mon[29204]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:49:53.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:53 smithi031 ceph-mon[29204]: from='osd.7 [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:49:53.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:53 smithi031 ceph-mon[29204]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T01:49:53.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:53.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:53 smithi031 ceph-mon[29204]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:49:54.228 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 01:49:53 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T01:49:53.966+0000 7f2361737700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T01:49:54.229 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 01:49:53 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T01:49:53.979+0000 7f2359522700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:49:54.519 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:49:54.519 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:49:54.519 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-04-25T01:49:54.519 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-04-25T01:49:54.519 INFO:teuthology.orchestra.run.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127811 s, 820 MB/s 2024-04-25T01:49:54.519 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-04-25T01:49:54.856 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:54 smithi134 ceph-mon[33098]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:49:54.856 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:54 smithi134 ceph-mon[33098]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:49:54.857 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:54 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:54.857 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:54 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:54.857 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:54 smithi134 ceph-mon[33098]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T01:49:54.857 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:54 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:54.857 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:54 smithi134 ceph-mon[33098]: from='osd.7 ' entity='osd.7' 2024-04-25T01:49:54.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:54 smithi040 ceph-mon[32418]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:49:54.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:54 smithi040 ceph-mon[32418]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:49:54.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:54 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:54.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:54 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:54.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:54 smithi040 ceph-mon[32418]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T01:49:54.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:54 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:54.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:54 smithi040 ceph-mon[32418]: from='osd.7 ' entity='osd.7' 2024-04-25T01:49:54.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:54 smithi031 ceph-mon[29204]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:49:54.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:54 smithi031 ceph-mon[29204]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:49:54.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:54 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:54.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:54 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:54.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:54 smithi031 ceph-mon[29204]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T01:49:54.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:54 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:54.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:54 smithi031 ceph-mon[29204]: from='osd.7 ' entity='osd.7' 2024-04-25T01:49:55.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi134:vg_nvme/lv_2 2024-04-25T01:49:55.430 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:49:55.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:55 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:49:55.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:55 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:49:55.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:55 smithi134 ceph-mon[33098]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:55.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:55.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:55 smithi134 ceph-mon[33098]: osd.7 [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732] boot 2024-04-25T01:49:55.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:55 smithi134 ceph-mon[33098]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T01:49:55.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:55.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:55 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:49:55.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:55 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:49:55.906 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:55 smithi040 ceph-mon[32418]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:55.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:55.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:55 smithi040 ceph-mon[32418]: osd.7 [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732] boot 2024-04-25T01:49:55.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:55 smithi040 ceph-mon[32418]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T01:49:55.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:55.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:55 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:49:55.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:55 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:49:55.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:55 smithi031 ceph-mon[29204]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:55.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:55.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:55 smithi031 ceph-mon[29204]: osd.7 [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732] boot 2024-04-25T01:49:55.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:55 smithi031 ceph-mon[29204]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T01:49:55.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:56.785 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:49:56.785 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:56.785 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi040 2024-04-25T01:49:56.785 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi040 to 5697M 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:56.786 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:56.787 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:56.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:49:56.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:56.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi040 2024-04-25T01:49:56.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:56.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:56.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:56.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:56.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:56.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi040 to 5697M 2024-04-25T01:49:56.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:56.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:56.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:56.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:56.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:56.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:56.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:56.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:56.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.997+0000 7f7bfb10e700 1 -- 172.21.15.134:0/891461822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bf4102290 msgr2=0x7f7bf41026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:56.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.997+0000 7f7bfb10e700 1 --2- 172.21.15.134:0/891461822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bf4102290 0x7f7bf41026b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009990 tx=0x7f7be8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.997+0000 7f7bfb10e700 1 -- 172.21.15.134:0/891461822 shutdown_connections 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.997+0000 7f7bfb10e700 1 --2- 172.21.15.134:0/891461822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bf4103e30 0x7f7bf410a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.997+0000 7f7bfb10e700 1 --2- 172.21.15.134:0/891461822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bf4102290 0x7f7bf41026b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.997+0000 7f7bfb10e700 1 --2- 172.21.15.134:0/891461822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bf4103490 0x7f7bf41038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.997+0000 7f7bfb10e700 1 -- 172.21.15.134:0/891461822 >> 172.21.15.134:0/891461822 conn(0x7f7bf40fd830 msgr2=0x7f7bf40ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.997+0000 7f7bfb10e700 1 -- 172.21.15.134:0/891461822 shutdown_connections 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.997+0000 7f7bfb10e700 1 -- 172.21.15.134:0/891461822 wait complete. 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bfb10e700 1 Processor -- start 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bfb10e700 1 -- start start 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bfb10e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bf4102290 0x7f7bf41983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bfb10e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bf4103490 0x7f7bf4198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:56.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bfb10e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bf4103e30 0x7f7bf419f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bfb10e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf410dd00 con 0x7f7bf4103e30 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bfb10e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf410db80 con 0x7f7bf4102290 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bfb10e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf410de80 con 0x7f7bf4103490 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bf3fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bf4103490 0x7f7bf4198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bf3fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bf4103490 0x7f7bf4198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38974/0 (socket says 172.21.15.134:38974) 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bf3fff700 1 -- 172.21.15.134:0/889084374 learned_addr learned my addr 172.21.15.134:0/889084374 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bf8eaa700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bf4102290 0x7f7bf41983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.998+0000 7f7bf96ab700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bf4103e30 0x7f7bf419f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.999+0000 7f7bf3fff700 1 -- 172.21.15.134:0/889084374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bf4102290 msgr2=0x7f7bf41983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.999+0000 7f7bf3fff700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bf4102290 0x7f7bf41983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.999+0000 7f7bf3fff700 1 -- 172.21.15.134:0/889084374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bf4103e30 msgr2=0x7f7bf419f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.999+0000 7f7bf3fff700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bf4103e30 0x7f7bf419f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.999+0000 7f7bf3fff700 1 -- 172.21.15.134:0/889084374 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf419ffe0 con 0x7f7bf4103490 2024-04-25T01:49:56.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.999+0000 7f7bf3fff700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bf4103490 0x7f7bf4198910 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7be000d980 tx=0x7f7be000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:49:57.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.999+0000 7f7bf1ffb700 1 -- 172.21.15.134:0/889084374 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be0016070 con 0x7f7bf4103490 2024-04-25T01:49:57.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.999+0000 7f7bf1ffb700 1 -- 172.21.15.134:0/889084374 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7be0009d80 con 0x7f7bf4103490 2024-04-25T01:49:57.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.999+0000 7f7bfb10e700 1 -- 172.21.15.134:0/889084374 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf41a02d0 con 0x7f7bf4103490 2024-04-25T01:49:57.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.999+0000 7f7bf1ffb700 1 -- 172.21.15.134:0/889084374 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be000f3c0 con 0x7f7bf4103490 2024-04-25T01:49:57.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:56.999+0000 7f7bfb10e700 1 -- 172.21.15.134:0/889084374 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf41a0810 con 0x7f7bf4103490 2024-04-25T01:49:57.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:57.000+0000 7f7bfb10e700 1 -- 172.21.15.134:0/889084374 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf4068f20 con 0x7f7bf4103490 2024-04-25T01:49:57.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:57.001+0000 7f7bf1ffb700 1 -- 172.21.15.134:0/889084374 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7be0019070 con 0x7f7bf4103490 2024-04-25T01:49:57.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:57.002+0000 7f7bf1ffb700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7be40a2c20 0x7f7be40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:57.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:57.002+0000 7f7bf1ffb700 1 -- 172.21.15.134:0/889084374 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7f7be00cdfb0 con 0x7f7bf4103490 2024-04-25T01:49:57.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:57.002+0000 7f7bf8eaa700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7be40a2c20 0x7f7be40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:57.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:57.003+0000 7f7bf8eaa700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7be40a2c20 0x7f7be40a50e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7be80037f0 tx=0x7f7be80038c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:49:57.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:57.006+0000 7f7bf1ffb700 1 -- 172.21.15.134:0/889084374 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7be009ba50 con 0x7f7bf4103490 2024-04-25T01:49:57.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:49:57.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:57.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi040 2024-04-25T01:49:57.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:57.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:57.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:57.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:57.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:57.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi040 to 5697M 2024-04-25T01:49:57.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:57.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:57.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:57.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:57.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:57.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:57.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:57.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:49:57.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:49:57.212+0000 7f7bfb10e700 1 -- 172.21.15.134:0/889084374 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7bf4062670 con 0x7f7be40a2c20 2024-04-25T01:49:57.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:57 smithi031 ceph-mon[29204]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:49:57.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:57 smithi031 ceph-mon[29204]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:57.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:57.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:57.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:57 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:58.014 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:57 smithi134 ceph-mon[33098]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:49:58.014 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:57 smithi134 ceph-mon[33098]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:58.014 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:58.014 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:58.014 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:57 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:58.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:57 smithi040 ceph-mon[32418]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:49:58.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:57 smithi040 ceph-mon[32418]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:58.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:58.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:58.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:57 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:58.784 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:58 smithi134 ceph-mon[33098]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:58.784 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:58 smithi134 ceph-mon[33098]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:49:58.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:58 smithi031 ceph-mon[29204]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:58.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:58 smithi031 ceph-mon[29204]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:49:59.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:58 smithi040 ceph-mon[32418]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:59.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:58 smithi040 ceph-mon[32418]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:49:59.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:49:59 smithi031 ceph-mon[29204]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T01:50:00.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:49:59 smithi040 ceph-mon[32418]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T01:50:00.162 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:49:59 smithi134 ceph-mon[33098]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T01:50:00.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:00 smithi031 ceph-mon[29204]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:00.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:00 smithi031 ceph-mon[29204]: from='client.? 172.21.15.134:0/4145668725' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71eaf5df-507d-4ea3-b5e6-fbc65d839f10"}]: dispatch 2024-04-25T01:50:00.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:00 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71eaf5df-507d-4ea3-b5e6-fbc65d839f10"}]: dispatch 2024-04-25T01:50:00.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:00 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71eaf5df-507d-4ea3-b5e6-fbc65d839f10"}]': finished 2024-04-25T01:50:00.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:00 smithi031 ceph-mon[29204]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T01:50:00.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:00 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:00.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:00 smithi031 ceph-mon[29204]: overall HEALTH_OK 2024-04-25T01:50:00.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:00 smithi031 ceph-mon[29204]: from='client.? 172.21.15.134:0/2977430603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:01.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:00 smithi040 ceph-mon[32418]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:01.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:00 smithi040 ceph-mon[32418]: from='client.? 172.21.15.134:0/4145668725' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71eaf5df-507d-4ea3-b5e6-fbc65d839f10"}]: dispatch 2024-04-25T01:50:01.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:00 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71eaf5df-507d-4ea3-b5e6-fbc65d839f10"}]: dispatch 2024-04-25T01:50:01.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:00 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71eaf5df-507d-4ea3-b5e6-fbc65d839f10"}]': finished 2024-04-25T01:50:01.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:00 smithi040 ceph-mon[32418]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T01:50:01.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:00 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:01.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:00 smithi040 ceph-mon[32418]: overall HEALTH_OK 2024-04-25T01:50:01.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:00 smithi040 ceph-mon[32418]: from='client.? 172.21.15.134:0/2977430603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:01.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:00 smithi134 ceph-mon[33098]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:01.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:00 smithi134 ceph-mon[33098]: from='client.? 172.21.15.134:0/4145668725' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71eaf5df-507d-4ea3-b5e6-fbc65d839f10"}]: dispatch 2024-04-25T01:50:01.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:00 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71eaf5df-507d-4ea3-b5e6-fbc65d839f10"}]: dispatch 2024-04-25T01:50:01.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:00 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71eaf5df-507d-4ea3-b5e6-fbc65d839f10"}]': finished 2024-04-25T01:50:01.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:00 smithi134 ceph-mon[33098]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T01:50:01.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:00 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:01.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:00 smithi134 ceph-mon[33098]: overall HEALTH_OK 2024-04-25T01:50:01.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:00 smithi134 ceph-mon[33098]: from='client.? 172.21.15.134:0/2977430603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:02.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:02 smithi031 ceph-mon[29204]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:03.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:02 smithi040 ceph-mon[32418]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:03.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:02 smithi134 ceph-mon[33098]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:04.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:04 smithi031 ceph-mon[29204]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:04.996 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:04 smithi134 ceph-mon[33098]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:05.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:04 smithi040 ceph-mon[32418]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:06.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:06 smithi134 ceph-mon[33098]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:06.978 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:06 smithi040 ceph-mon[32418]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:06.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:06 smithi031 ceph-mon[29204]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:07.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:07 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:50:07.950 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:07 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:07.950 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:07 smithi134 ceph-mon[33098]: Deploying daemon osd.8 on smithi134 2024-04-25T01:50:07.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:07 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:50:07.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:07 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:07.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:07 smithi031 ceph-mon[29204]: Deploying daemon osd.8 on smithi134 2024-04-25T01:50:08.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:07 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:50:08.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:07 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:08.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:07 smithi040 ceph-mon[32418]: Deploying daemon osd.8 on smithi134 2024-04-25T01:50:08.826 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:08 smithi134 ceph-mon[33098]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:08.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:08 smithi031 ceph-mon[29204]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:09.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:08 smithi040 ceph-mon[32418]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:11.050 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:10 smithi134 ceph-mon[33098]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:11.156 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:10 smithi040 ceph-mon[32418]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:11.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:10 smithi031 ceph-mon[29204]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:12.754 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:12 smithi134 ceph-mon[33098]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:13.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:12 smithi040 ceph-mon[32418]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:13.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:12 smithi031 ceph-mon[29204]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:14.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:14.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:14.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:14.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:14.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:14.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:14.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:14.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:14.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:15.094 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:14 smithi134 ceph-mon[33098]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:15.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:14 smithi040 ceph-mon[32418]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:15.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:14 smithi031 ceph-mon[29204]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:16.330 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 8 on host 'smithi134' 2024-04-25T01:50:16.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.332+0000 7f7bf1ffb700 1 -- 172.21.15.134:0/889084374 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7bf4062670 con 0x7f7be40a2c20 2024-04-25T01:50:16.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.335+0000 7f7bfb10e700 1 -- 172.21.15.134:0/889084374 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7be40a2c20 msgr2=0x7f7be40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:16.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.335+0000 7f7bfb10e700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7be40a2c20 0x7f7be40a50e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7be80037f0 tx=0x7f7be80038c0 comp rx=0 tx=0).stop 2024-04-25T01:50:16.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.336+0000 7f7bfb10e700 1 -- 172.21.15.134:0/889084374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bf4103490 msgr2=0x7f7bf4198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:16.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.336+0000 7f7bfb10e700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bf4103490 0x7f7bf4198910 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7be000d980 tx=0x7f7be000dc90 comp rx=0 tx=0).stop 2024-04-25T01:50:16.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.336+0000 7f7bfb10e700 1 -- 172.21.15.134:0/889084374 shutdown_connections 2024-04-25T01:50:16.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.336+0000 7f7bfb10e700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7be40a2c20 0x7f7be40a50e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:16.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.336+0000 7f7bfb10e700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bf4103e30 0x7f7bf419f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:16.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.336+0000 7f7bfb10e700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bf4102290 0x7f7bf41983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:16.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.336+0000 7f7bfb10e700 1 --2- 172.21.15.134:0/889084374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bf4103490 0x7f7bf4198910 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:16.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.336+0000 7f7bfb10e700 1 -- 172.21.15.134:0/889084374 >> 172.21.15.134:0/889084374 conn(0x7f7bf40fd830 msgr2=0x7f7bf41068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:50:16.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.336+0000 7f7bfb10e700 1 -- 172.21.15.134:0/889084374 shutdown_connections 2024-04-25T01:50:16.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:50:16.336+0000 7f7bfb10e700 1 -- 172.21.15.134:0/889084374 wait complete. 2024-04-25T01:50:16.948 DEBUG:teuthology.orchestra.run.smithi134:osd.8> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T01:50:16.951 INFO:tasks.cephadm:Deploying osd.9 on smithi031 with /dev/vg_nvme/lv_1... 2024-04-25T01:50:16.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:50:17.212 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:50:17.551 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:17 smithi031 ceph-mon[29204]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:17.551 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:17 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:17.551 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:17 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:17.551 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:17 smithi031 ceph-mon[29204]: from='osd.8 [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:17.551 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:17 smithi031 ceph-mon[29204]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:17.551 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:17 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:17.551 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:17 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:17.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:17 smithi040 ceph-mon[32418]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:17.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:17.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:17.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:17 smithi040 ceph-mon[32418]: from='osd.8 [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:17.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:17 smithi040 ceph-mon[32418]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:17.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:17.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:17 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:17.663 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:17 smithi134 ceph-mon[33098]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:17.663 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:17 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:17.663 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:17 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:17.663 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:17 smithi134 ceph-mon[33098]: from='osd.8 [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:17.663 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:17 smithi134 ceph-mon[33098]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:17.663 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:17 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:17.663 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:17 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:18.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:18 smithi031 ceph-mon[29204]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:50:18.561 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:18 smithi031 ceph-mon[29204]: from='osd.8 [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:50:18.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:18 smithi031 ceph-mon[29204]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:50:18.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:18 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:18.562 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:18 smithi031 ceph-mon[29204]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:50:18.596 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:18 smithi134 ceph-mon[33098]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:50:18.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:18 smithi134 ceph-mon[33098]: from='osd.8 [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:50:18.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:18 smithi134 ceph-mon[33098]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:50:18.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:18 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:18.597 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:18 smithi134 ceph-mon[33098]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:50:18.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:18 smithi040 ceph-mon[32418]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:50:18.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:18 smithi040 ceph-mon[32418]: from='osd.8 [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:50:18.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:18 smithi040 ceph-mon[32418]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:50:18.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:18 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:18.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:18 smithi040 ceph-mon[32418]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:50:18.924 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 01:50:18 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T01:50:18.745+0000 7f7771e5c700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T01:50:18.924 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 01:50:18 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T01:50:18.754+0000 7f7769c47700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:50:19.326 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:50:19.327 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:50:19.327 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-25T01:50:19.327 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-25T01:50:19.327 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01239 s, 846 MB/s 2024-04-25T01:50:19.327 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-25T01:50:19.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:19 smithi031 ceph-mon[29204]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:19.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:19 smithi031 ceph-mon[29204]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:50:19.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:19 smithi031 ceph-mon[29204]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T01:50:19.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:19.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:19.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:19 smithi031 ceph-mon[29204]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T01:50:19.635 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:19.635 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:19 smithi031 ceph-mon[29204]: from='osd.8 ' entity='osd.8' 2024-04-25T01:50:19.635 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:19.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:19 smithi040 ceph-mon[32418]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:19.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:19 smithi040 ceph-mon[32418]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:50:19.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:19 smithi040 ceph-mon[32418]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T01:50:19.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:19.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:19.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:19 smithi040 ceph-mon[32418]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T01:50:19.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:19.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:19 smithi040 ceph-mon[32418]: from='osd.8 ' entity='osd.8' 2024-04-25T01:50:19.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:19.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:19 smithi134 ceph-mon[33098]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:19.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:19 smithi134 ceph-mon[33098]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:50:19.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:19 smithi134 ceph-mon[33098]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T01:50:19.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:19.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:19.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:19 smithi134 ceph-mon[33098]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T01:50:19.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:19.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:19 smithi134 ceph-mon[33098]: from='osd.8 ' entity='osd.8' 2024-04-25T01:50:19.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:20.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_1 2024-04-25T01:50:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:50:20.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:50:20.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:50:20.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: osd.8 [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048] boot 2024-04-25T01:50:20.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:50:20.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:20.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:20.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi134 2024-04-25T01:50:20.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:20.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:20.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:20.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:20.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:20.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi134 to 5696M 2024-04-25T01:50:20.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:20.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:20.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:20.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:20.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:20.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:20.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:20.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:20 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:21.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:50:21.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:50:21.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: osd.8 [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048] boot 2024-04-25T01:50:21.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:50:21.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:21.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi134 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi134 to 5696M 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:21.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:20 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:21.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:50:21.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:50:21.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: osd.8 [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048] boot 2024-04-25T01:50:21.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:50:21.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:21.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:21.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi134 2024-04-25T01:50:21.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:21.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:21.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:21.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:21.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:21.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi134 to 5696M 2024-04-25T01:50:21.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:21.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:21.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:21.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:21.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:21.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:21.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:21.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:20 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:21.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:21 smithi031 ceph-mon[29204]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:50:21.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.945+0000 7fd892a74700 1 -- 172.21.15.31:0/764703073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd88c100580 msgr2=0x7fd88c1009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:21.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.945+0000 7fd892a74700 1 --2- 172.21.15.31:0/764703073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd88c100580 0x7fd88c1009a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd880009990 tx=0x7fd880009ca0 comp rx=0 tx=0).stop 2024-04-25T01:50:21.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.945+0000 7fd892a74700 1 -- 172.21.15.31:0/764703073 shutdown_connections 2024-04-25T01:50:21.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.945+0000 7fd892a74700 1 --2- 172.21.15.31:0/764703073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd88c103fa0 0x7fd88c10a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:21.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.945+0000 7fd892a74700 1 --2- 172.21.15.31:0/764703073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd88c100580 0x7fd88c1009a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:21.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.945+0000 7fd892a74700 1 --2- 172.21.15.31:0/764703073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c103600 0x7fd88c103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:21.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.945+0000 7fd892a74700 1 -- 172.21.15.31:0/764703073 >> 172.21.15.31:0/764703073 conn(0x7fd88c0fbee0 msgr2=0x7fd88c0fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:50:21.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.945+0000 7fd892a74700 1 -- 172.21.15.31:0/764703073 shutdown_connections 2024-04-25T01:50:21.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.945+0000 7fd892a74700 1 -- 172.21.15.31:0/764703073 wait complete. 2024-04-25T01:50:21.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.946+0000 7fd892a74700 1 Processor -- start 2024-04-25T01:50:21.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.946+0000 7fd892a74700 1 -- start start 2024-04-25T01:50:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.946+0000 7fd892a74700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c100580 0x7fd88c198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.946+0000 7fd892a74700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd88c103600 0x7fd88c1988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.946+0000 7fd892a74700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd88c103fa0 0x7fd88c19f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.946+0000 7fd892a74700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c10dd20 con 0x7fd88c103fa0 2024-04-25T01:50:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.946+0000 7fd892a74700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c10dba0 con 0x7fd88c103600 2024-04-25T01:50:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.946+0000 7fd892a74700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c10dea0 con 0x7fd88c100580 2024-04-25T01:50:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd891011700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd88c103fa0 0x7fd88c19f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd891011700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd88c103fa0 0x7fd88c19f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38374/0 (socket says 172.21.15.31:38374) 2024-04-25T01:50:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd891011700 1 -- 172.21.15.31:0/219240244 learned_addr learned my addr 172.21.15.31:0/219240244 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:50:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd88bfff700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd88c103600 0x7fd88c1988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd890810700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c100580 0x7fd88c198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd891011700 1 -- 172.21.15.31:0/219240244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c100580 msgr2=0x7fd88c198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd891011700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c100580 0x7fd88c198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd891011700 1 -- 172.21.15.31:0/219240244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd88c103600 msgr2=0x7fd88c1988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd891011700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd88c103600 0x7fd88c1988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd891011700 1 -- 172.21.15.31:0/219240244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88c19fff0 con 0x7fd88c103fa0 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd891011700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd88c103fa0 0x7fd88c19f950 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd884007f10 tx=0x7fd88400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd889ffb700 1 -- 172.21.15.31:0/219240244 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd884004bb0 con 0x7fd88c103fa0 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd889ffb700 1 -- 172.21.15.31:0/219240244 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd884004d50 con 0x7fd88c103fa0 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd892a74700 1 -- 172.21.15.31:0/219240244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88c1a02e0 con 0x7fd88c103fa0 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.947+0000 7fd889ffb700 1 -- 172.21.15.31:0/219240244 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8840058e0 con 0x7fd88c103fa0 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.948+0000 7fd892a74700 1 -- 172.21.15.31:0/219240244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88c1a07d0 con 0x7fd88c103fa0 2024-04-25T01:50:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.948+0000 7fd892a74700 1 -- 172.21.15.31:0/219240244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd88c068f20 con 0x7fd88c103fa0 2024-04-25T01:50:21.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.949+0000 7fd889ffb700 1 -- 172.21.15.31:0/219240244 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd88401a020 con 0x7fd88c103fa0 2024-04-25T01:50:21.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.950+0000 7fd889ffb700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd87c0a2c20 0x7fd87c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:21.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.950+0000 7fd890810700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd87c0a2c20 0x7fd87c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:21.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.950+0000 7fd889ffb700 1 -- 172.21.15.31:0/219240244 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5858+0+0 (secure 0 0 0) 0x7fd8840cddd0 con 0x7fd88c103fa0 2024-04-25T01:50:21.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.951+0000 7fd890810700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd87c0a2c20 0x7fd87c0a50e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd880003780 tx=0x7fd880005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:50:21.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:21.955+0000 7fd889ffb700 1 -- 172.21.15.31:0/219240244 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd884010070 con 0x7fd88c103fa0 2024-04-25T01:50:22.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:21 smithi040 ceph-mon[32418]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:50:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:22.173+0000 7fd892a74700 1 -- 172.21.15.31:0/219240244 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd88c062670 con 0x7fd87c0a2c20 2024-04-25T01:50:22.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:21 smithi134 ceph-mon[33098]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:50:22.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:22 smithi031 ceph-mon[29204]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:50:22.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:22 smithi031 ceph-mon[29204]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:22.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:22 smithi031 ceph-mon[29204]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:22.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:22 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:22.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:22 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:22.981 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:22 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:23.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:22 smithi040 ceph-mon[32418]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:50:23.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:22 smithi040 ceph-mon[32418]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:23.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:22 smithi040 ceph-mon[32418]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:23.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:22 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:23.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:22 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:23.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:22 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:23.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:22 smithi134 ceph-mon[33098]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:50:23.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:22 smithi134 ceph-mon[33098]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:23.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:22 smithi134 ceph-mon[33098]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:23.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:22 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:23.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:22 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:23.187 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:22 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:23.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:23 smithi031 ceph-mon[29204]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T01:50:24.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:23 smithi040 ceph-mon[32418]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T01:50:24.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:23 smithi134 ceph-mon[33098]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T01:50:24.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:24 smithi031 ceph-mon[29204]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:24.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:24 smithi031 ceph-mon[29204]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T01:50:25.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:24 smithi040 ceph-mon[32418]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:25.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:24 smithi040 ceph-mon[32418]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T01:50:25.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:24 smithi134 ceph-mon[33098]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:25.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:24 smithi134 ceph-mon[33098]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T01:50:25.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:25 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/543049246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eda04a7b-4e18-470d-9678-e21ffeb1f7ed"}]: dispatch 2024-04-25T01:50:25.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:25 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/543049246' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eda04a7b-4e18-470d-9678-e21ffeb1f7ed"}]': finished 2024-04-25T01:50:25.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:25 smithi031 ceph-mon[29204]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:50:25.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:25 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:25.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:25 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2764477607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:26.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:25 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/543049246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eda04a7b-4e18-470d-9678-e21ffeb1f7ed"}]: dispatch 2024-04-25T01:50:26.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:25 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/543049246' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eda04a7b-4e18-470d-9678-e21ffeb1f7ed"}]': finished 2024-04-25T01:50:26.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:25 smithi040 ceph-mon[32418]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:50:26.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:25 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:26.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:25 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2764477607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:26.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:25 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/543049246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eda04a7b-4e18-470d-9678-e21ffeb1f7ed"}]: dispatch 2024-04-25T01:50:26.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:25 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/543049246' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eda04a7b-4e18-470d-9678-e21ffeb1f7ed"}]': finished 2024-04-25T01:50:26.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:25 smithi134 ceph-mon[33098]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:50:26.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:25 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:26.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:25 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2764477607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:27.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:26 smithi134 ceph-mon[33098]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:50:27.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:26 smithi031 ceph-mon[29204]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:50:27.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:26 smithi040 ceph-mon[32418]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:50:29.186 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:28 smithi134 ceph-mon[33098]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:50:29.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:28 smithi031 ceph-mon[29204]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:50:29.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:28 smithi040 ceph-mon[32418]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:50:30.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:30 smithi031 ceph-mon[29204]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-25T01:50:31.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:30 smithi040 ceph-mon[32418]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-25T01:50:31.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:30 smithi134 ceph-mon[33098]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-25T01:50:33.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:32 smithi031 ceph-mon[29204]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:50:33.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:50:33.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:32 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:33.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:32 smithi031 ceph-mon[29204]: Deploying daemon osd.9 on smithi031 2024-04-25T01:50:33.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:32 smithi040 ceph-mon[32418]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:50:33.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:50:33.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:32 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:33.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:32 smithi040 ceph-mon[32418]: Deploying daemon osd.9 on smithi031 2024-04-25T01:50:33.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:32 smithi134 ceph-mon[33098]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:50:33.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:50:33.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:32 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:33.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:32 smithi134 ceph-mon[33098]: Deploying daemon osd.9 on smithi031 2024-04-25T01:50:35.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:34 smithi031 ceph-mon[29204]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:50:35.238 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:34 smithi134 ceph-mon[33098]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:50:35.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:34 smithi040 ceph-mon[32418]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:50:37.209 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:36 smithi040 ceph-mon[32418]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-25T01:50:37.235 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:36 smithi031 ceph-mon[29204]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-25T01:50:37.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:36 smithi134 ceph-mon[33098]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-25T01:50:39.638 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:39 smithi031 ceph-mon[29204]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:39.638 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:39.638 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:39.638 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:39.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:39 smithi040 ceph-mon[32418]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:39.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:39.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:39.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:39.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:39 smithi134 ceph-mon[33098]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:39.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:39.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:39.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:41.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:41 smithi031 ceph-mon[29204]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:41.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:41 smithi040 ceph-mon[32418]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:41.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:41 smithi134 ceph-mon[33098]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:42.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.588+0000 7fd889ffb700 1 -- 172.21.15.31:0/219240244 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd88c062670 con 0x7fd87c0a2c20 2024-04-25T01:50:42.587 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 9 on host 'smithi031' 2024-04-25T01:50:42.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.590+0000 7fd892a74700 1 -- 172.21.15.31:0/219240244 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd87c0a2c20 msgr2=0x7fd87c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:42.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.590+0000 7fd892a74700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd87c0a2c20 0x7fd87c0a50e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd880003780 tx=0x7fd880005fd0 comp rx=0 tx=0).stop 2024-04-25T01:50:42.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.590+0000 7fd892a74700 1 -- 172.21.15.31:0/219240244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd88c103fa0 msgr2=0x7fd88c19f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:42.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.590+0000 7fd892a74700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd88c103fa0 0x7fd88c19f950 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd884007f10 tx=0x7fd88400d3b0 comp rx=0 tx=0).stop 2024-04-25T01:50:42.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.590+0000 7fd892a74700 1 -- 172.21.15.31:0/219240244 shutdown_connections 2024-04-25T01:50:42.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.590+0000 7fd892a74700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd87c0a2c20 0x7fd87c0a50e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:42.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.590+0000 7fd892a74700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd88c103fa0 0x7fd88c19f950 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:42.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.590+0000 7fd892a74700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c100580 0x7fd88c198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:42.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.590+0000 7fd892a74700 1 --2- 172.21.15.31:0/219240244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd88c103600 0x7fd88c1988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:42.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.590+0000 7fd892a74700 1 -- 172.21.15.31:0/219240244 >> 172.21.15.31:0/219240244 conn(0x7fd88c0fbee0 msgr2=0x7fd88c106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:50:42.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.591+0000 7fd892a74700 1 -- 172.21.15.31:0/219240244 shutdown_connections 2024-04-25T01:50:42.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:50:42.591+0000 7fd892a74700 1 -- 172.21.15.31:0/219240244 wait complete. 2024-04-25T01:50:42.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:42.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:42.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:42 smithi040 ceph-mon[32418]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:50:42.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:42 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:42.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:42.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:42.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:42 smithi134 ceph-mon[33098]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:50:42.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:42 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:42.963 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.964 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.964 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:42.964 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:42.964 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.964 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:42 smithi031 ceph-mon[29204]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:50:42.964 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.964 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:42.964 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:42 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:43.283 DEBUG:teuthology.orchestra.run.smithi031:osd.9> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T01:50:43.286 INFO:tasks.cephadm:Deploying osd.10 on smithi040 with /dev/vg_nvme/lv_1... 2024-04-25T01:50:43.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:50:43.541 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:50:43.801 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:43 smithi031 ceph-mon[29204]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:43.802 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:43 smithi031 ceph-mon[29204]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:50:43.802 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:43 smithi031 ceph-mon[29204]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T01:50:43.802 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:43 smithi031 ceph-mon[29204]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:50:43.802 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:43 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:43.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:43 smithi040 ceph-mon[32418]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:43.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:43 smithi040 ceph-mon[32418]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:50:43.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:43 smithi040 ceph-mon[32418]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T01:50:43.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:43 smithi040 ceph-mon[32418]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:50:43.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:43 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:43.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:43 smithi134 ceph-mon[33098]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:43.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:43 smithi134 ceph-mon[33098]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:50:43.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:43 smithi134 ceph-mon[33098]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T01:50:43.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:43 smithi134 ceph-mon[33098]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T01:50:43.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:43 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:44.485 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 01:50:44 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T01:50:44.059+0000 7fc41c962700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T01:50:44.486 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 01:50:44 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T01:50:44.074+0000 7fc41474d700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:50:44.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:44 smithi040 ceph-mon[32418]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:50:44.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:44 smithi040 ceph-mon[32418]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T01:50:44.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:44 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:44.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:44 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:44.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:44 smithi040 ceph-mon[32418]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' 2024-04-25T01:50:44.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:44 smithi134 ceph-mon[33098]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:50:44.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:44 smithi134 ceph-mon[33098]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T01:50:44.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:44 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:44.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:44 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:44.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:44 smithi134 ceph-mon[33098]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' 2024-04-25T01:50:44.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:44 smithi031 ceph-mon[29204]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T01:50:44.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:44 smithi031 ceph-mon[29204]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T01:50:44.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:44 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:44.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:44 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:44.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:44 smithi031 ceph-mon[29204]: from='osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088]' entity='osd.9' 2024-04-25T01:50:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:50:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:50:45.787 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-25T01:50:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-25T01:50:45.787 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119589 s, 877 MB/s 2024-04-25T01:50:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-25T01:50:45.889 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:45 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:50:45.889 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:45 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:50:45.889 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:45 smithi031 ceph-mon[29204]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:45.889 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:45 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:45.889 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:45 smithi031 ceph-mon[29204]: osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088] boot 2024-04-25T01:50:45.889 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:45 smithi031 ceph-mon[29204]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T01:50:45.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:45 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:45.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:45 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:50:45.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:45 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:50:45.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:45 smithi040 ceph-mon[32418]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:45.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:45 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:45.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:45 smithi040 ceph-mon[32418]: osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088] boot 2024-04-25T01:50:45.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:45 smithi040 ceph-mon[32418]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T01:50:45.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:45 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:45.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:45 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:50:45.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:45 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:50:45.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:45 smithi134 ceph-mon[33098]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:45.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:45 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:45.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:45 smithi134 ceph-mon[33098]: osd.9 [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088] boot 2024-04-25T01:50:45.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:45 smithi134 ceph-mon[33098]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T01:50:45.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:45 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:46.506 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_1 2024-04-25T01:50:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:50:47.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:47.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: osdmap e70: 10 total, 10 up, 10 in 2024-04-25T01:50:47.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:47.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:47.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:47.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:47.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:47.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:47.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:47.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: osdmap e70: 10 total, 10 up, 10 in 2024-04-25T01:50:47.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:47.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:47.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:47.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:47.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:47.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:47.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:47.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: osdmap e70: 10 total, 10 up, 10 in 2024-04-25T01:50:47.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:47.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:47.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:47.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:47.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:47.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:47.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:50:48.290 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:48 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi031 2024-04-25T01:50:48.290 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:48 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi031 to 4273M 2024-04-25T01:50:48.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:48 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi031 2024-04-25T01:50:48.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:48 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi031 to 4273M 2024-04-25T01:50:48.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:48 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi031 2024-04-25T01:50:48.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:48 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi031 to 4273M 2024-04-25T01:50:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.526+0000 7f3be4103700 1 -- 172.21.15.40:0/428869508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bdc103e50 msgr2=0x7f3bdc10a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.526+0000 7f3be4103700 1 --2- 172.21.15.40:0/428869508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bdc103e50 0x7f3bdc10a700 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400b0d0 tx=0x7f3bd400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:50:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.526+0000 7f3be4103700 1 -- 172.21.15.40:0/428869508 shutdown_connections 2024-04-25T01:50:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 --2- 172.21.15.40:0/428869508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bdc103e50 0x7f3bdc10a700 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 --2- 172.21.15.40:0/428869508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bdc1022b0 0x7f3bdc1026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 --2- 172.21.15.40:0/428869508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bdc1034b0 0x7f3bdc103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 -- 172.21.15.40:0/428869508 >> 172.21.15.40:0/428869508 conn(0x7f3bdc0fd850 msgr2=0x7f3bdc0ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:50:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 -- 172.21.15.40:0/428869508 shutdown_connections 2024-04-25T01:50:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 -- 172.21.15.40:0/428869508 wait complete. 2024-04-25T01:50:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 Processor -- start 2024-04-25T01:50:48.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 -- start start 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bdc1022b0 0x7f3bdc075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bdc1034b0 0x7f3bdc074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bdc103e50 0x7f3bdc074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3bdc10dd40 con 0x7f3bdc103e50 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3bdc10dbc0 con 0x7f3bdc1034b0 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be4103700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3bdc10dec0 con 0x7f3bdc1022b0 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.527+0000 7f3be169e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bdc1034b0 0x7f3bdc074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be169e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bdc1034b0 0x7f3bdc074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41970/0 (socket says 172.21.15.40:41970) 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be169e700 1 -- 172.21.15.40:0/511817183 learned_addr learned my addr 172.21.15.40:0/511817183 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be1e9f700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bdc1022b0 0x7f3bdc075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be26a0700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bdc103e50 0x7f3bdc074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be169e700 1 -- 172.21.15.40:0/511817183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bdc1022b0 msgr2=0x7f3bdc075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be169e700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bdc1022b0 0x7f3bdc075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:48.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be169e700 1 -- 172.21.15.40:0/511817183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bdc103e50 msgr2=0x7f3bdc074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:48.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be169e700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bdc103e50 0x7f3bdc074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:48.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be169e700 1 -- 172.21.15.40:0/511817183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bdc075410 con 0x7f3bdc1034b0 2024-04-25T01:50:48.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be169e700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bdc1034b0 0x7f3bdc074620 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800d980 tx=0x7f3bd800dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:50:48.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3bd2ffd700 1 -- 172.21.15.40:0/511817183 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8017070 con 0x7f3bdc1034b0 2024-04-25T01:50:48.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3bd2ffd700 1 -- 172.21.15.40:0/511817183 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3bd8004830 con 0x7f3bdc1034b0 2024-04-25T01:50:48.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be4103700 1 -- 172.21.15.40:0/511817183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bdc1a2480 con 0x7f3bdc1034b0 2024-04-25T01:50:48.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3bd2ffd700 1 -- 172.21.15.40:0/511817183 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8005440 con 0x7f3bdc1034b0 2024-04-25T01:50:48.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.528+0000 7f3be4103700 1 -- 172.21.15.40:0/511817183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bdc1a2990 con 0x7f3bdc1034b0 2024-04-25T01:50:48.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.529+0000 7f3be4103700 1 -- 172.21.15.40:0/511817183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bdc068f20 con 0x7f3bdc1034b0 2024-04-25T01:50:48.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.531+0000 7f3bd2ffd700 1 -- 172.21.15.40:0/511817183 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3bd80055e0 con 0x7f3bdc1034b0 2024-04-25T01:50:48.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.531+0000 7f3bd2ffd700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3bc80a2ca0 0x7f3bc80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:48.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.531+0000 7f3bd2ffd700 1 -- 172.21.15.40:0/511817183 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6261+0+0 (secure 0 0 0) 0x7f3bd80ceb00 con 0x7f3bdc1034b0 2024-04-25T01:50:48.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.531+0000 7f3be1e9f700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3bc80a2ca0 0x7f3bc80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:48.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.532+0000 7f3be1e9f700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3bc80a2ca0 0x7f3bc80a5160 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc005fd0 tx=0x7f3bcc005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:50:48.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.535+0000 7f3bd2ffd700 1 -- 172.21.15.40:0/511817183 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3bd8020020 con 0x7f3bdc1034b0 2024-04-25T01:50:48.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:50:48.751+0000 7f3be4103700 1 -- 172.21.15.40:0/511817183 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3bdc062670 con 0x7f3bc80a2ca0 2024-04-25T01:50:49.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:49 smithi040 ceph-mon[32418]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:49.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:49.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:49.390 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:49.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:49 smithi134 ceph-mon[33098]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:49.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:49.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:49.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:49.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:49 smithi031 ceph-mon[29204]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:49.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:49.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:49.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:50.236 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:50 smithi031 ceph-mon[29204]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:50.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:50 smithi040 ceph-mon[32418]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:50.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:50 smithi134 ceph-mon[33098]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:51.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:51 smithi040 ceph-mon[32418]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:51.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:51 smithi134 ceph-mon[33098]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:51.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:51 smithi031 ceph-mon[29204]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:52.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:52 smithi040 ceph-mon[32418]: from='client.? 172.21.15.40:0/2475184876' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb"}]: dispatch 2024-04-25T01:50:52.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:52 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb"}]: dispatch 2024-04-25T01:50:52.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:52 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb"}]': finished 2024-04-25T01:50:52.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:52 smithi040 ceph-mon[32418]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:50:52.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:52 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:50:52.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:52 smithi134 ceph-mon[33098]: from='client.? 172.21.15.40:0/2475184876' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb"}]: dispatch 2024-04-25T01:50:52.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:52 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb"}]: dispatch 2024-04-25T01:50:52.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:52 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb"}]': finished 2024-04-25T01:50:52.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:52 smithi134 ceph-mon[33098]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:50:52.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:52 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:50:52.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:52 smithi031 ceph-mon[29204]: from='client.? 172.21.15.40:0/2475184876' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb"}]: dispatch 2024-04-25T01:50:52.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:52 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb"}]: dispatch 2024-04-25T01:50:52.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:52 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb"}]': finished 2024-04-25T01:50:52.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:52 smithi031 ceph-mon[29204]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:50:52.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:52 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:50:53.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:53 smithi040 ceph-mon[32418]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:53.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:53 smithi040 ceph-mon[32418]: from='client.? 172.21.15.40:0/896877309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:53.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:53 smithi134 ceph-mon[33098]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:53.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:53 smithi134 ceph-mon[33098]: from='client.? 172.21.15.40:0/896877309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:53.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:53 smithi031 ceph-mon[29204]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:53.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:53 smithi031 ceph-mon[29204]: from='client.? 172.21.15.40:0/896877309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:55.361 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:55 smithi040 ceph-mon[32418]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:55.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:55 smithi134 ceph-mon[33098]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:55.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:55 smithi031 ceph-mon[29204]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:57.157 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:57 smithi040 ceph-mon[32418]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:57.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:57 smithi134 ceph-mon[33098]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:57.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:57 smithi031 ceph-mon[29204]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:59.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:50:59 smithi040 ceph-mon[32418]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:59.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:50:59 smithi134 ceph-mon[33098]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:59.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:50:59 smithi031 ceph-mon[29204]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:00.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:00 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:51:00.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:00 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:00.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:00 smithi040 ceph-mon[32418]: Deploying daemon osd.10 on smithi040 2024-04-25T01:51:00.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:00 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:51:00.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:00 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:00.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:00 smithi134 ceph-mon[33098]: Deploying daemon osd.10 on smithi040 2024-04-25T01:51:00.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:00 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:51:00.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:00 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:00.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:00 smithi031 ceph-mon[29204]: Deploying daemon osd.10 on smithi040 2024-04-25T01:51:01.181 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:01 smithi040 ceph-mon[32418]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:01.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:01 smithi134 ceph-mon[33098]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:01.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:01 smithi031 ceph-mon[29204]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:03.380 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:03 smithi040 ceph-mon[32418]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:03.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:03 smithi134 ceph-mon[33098]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:03.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:03 smithi031 ceph-mon[29204]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:05.407 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:05 smithi040 ceph-mon[32418]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:05.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:05 smithi134 ceph-mon[33098]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:05.485 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:05 smithi031 ceph-mon[29204]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:06.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:06 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:06.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:06 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:06.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:06 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:06.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:06 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:06.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:06 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:06.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:06 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:06.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:06 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:06.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:06 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:06.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:06 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:07.623 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:07 smithi040 ceph-mon[32418]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:07.637 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:07 smithi031 ceph-mon[29204]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:07.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:07 smithi134 ceph-mon[33098]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:09.269 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 10 on host 'smithi040' 2024-04-25T01:51:09.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.270+0000 7f3bd2ffd700 1 -- 172.21.15.40:0/511817183 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3bdc062670 con 0x7f3bc80a2ca0 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 -- 172.21.15.40:0/511817183 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3bc80a2ca0 msgr2=0x7f3bc80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3bc80a2ca0 0x7f3bc80a5160 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc005fd0 tx=0x7f3bcc005e50 comp rx=0 tx=0).stop 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 -- 172.21.15.40:0/511817183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bdc1034b0 msgr2=0x7f3bdc074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bdc1034b0 0x7f3bdc074620 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800d980 tx=0x7f3bd800dd40 comp rx=0 tx=0).stop 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 -- 172.21.15.40:0/511817183 shutdown_connections 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3bc80a2ca0 0x7f3bc80a5160 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3bdc103e50 0x7f3bdc074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bdc1022b0 0x7f3bdc075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 --2- 172.21.15.40:0/511817183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bdc1034b0 0x7f3bdc074620 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 -- 172.21.15.40:0/511817183 >> 172.21.15.40:0/511817183 conn(0x7f3bdc0fd850 msgr2=0x7f3bdc106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 -- 172.21.15.40:0/511817183 shutdown_connections 2024-04-25T01:51:09.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-25T01:51:09.272+0000 7f3be4103700 1 -- 172.21.15.40:0/511817183 wait complete. 2024-04-25T01:51:09.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:09 smithi040 ceph-mon[32418]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:09.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:09 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:09 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:09 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:09.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:09 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:09.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:09 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:09 smithi040 ceph-mon[32418]: from='osd.10 [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:09.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:09 smithi040 ceph-mon[32418]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:09.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:09 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:09 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:09 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:09.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:09 smithi134 ceph-mon[33098]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:09.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:09 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:09 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:09 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:09.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:09 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:09.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:09 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:09 smithi134 ceph-mon[33098]: from='osd.10 [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:09.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:09 smithi134 ceph-mon[33098]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:09.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:09 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:09 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:09 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:09.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:09 smithi031 ceph-mon[29204]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:09.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:09 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:09 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:09 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:09.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:09 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:09.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:09 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:09 smithi031 ceph-mon[29204]: from='osd.10 [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:09.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:09 smithi031 ceph-mon[29204]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:09.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:09 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:09 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:09.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:09 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:10.072 DEBUG:teuthology.orchestra.run.smithi040:osd.10> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T01:51:10.074 INFO:tasks.cephadm:Deploying osd.11 on smithi134 with /dev/vg_nvme/lv_1... 2024-04-25T01:51:10.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:51:10.319 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:51:10.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:10 smithi040 ceph-mon[32418]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:51:10.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:10 smithi040 ceph-mon[32418]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T01:51:10.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:10.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:10 smithi040 ceph-mon[32418]: from='osd.10 [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:51:10.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:10 smithi040 ceph-mon[32418]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:51:10.699 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:10 smithi134 ceph-mon[33098]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:51:10.699 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:10 smithi134 ceph-mon[33098]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T01:51:10.699 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:10.699 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:10 smithi134 ceph-mon[33098]: from='osd.10 [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:51:10.699 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:10 smithi134 ceph-mon[33098]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:51:10.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:10 smithi031 ceph-mon[29204]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:51:10.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:10 smithi031 ceph-mon[29204]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T01:51:10.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:10.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:10 smithi031 ceph-mon[29204]: from='osd.10 [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:51:10.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:10 smithi031 ceph-mon[29204]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-25T01:51:11.259 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 01:51:11 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T01:51:10.997+0000 7fda95264700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T01:51:11.259 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 01:51:11 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T01:51:11.009+0000 7fda8c84e700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:51:11.737 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:11 smithi134 ceph-mon[33098]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:11.737 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:11 smithi134 ceph-mon[33098]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:51:11.737 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:11 smithi134 ceph-mon[33098]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T01:51:11.738 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:11 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:11.738 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:11 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:11.738 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:11 smithi134 ceph-mon[33098]: from='osd.10 ' entity='osd.10' 2024-04-25T01:51:11.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:11 smithi040 ceph-mon[32418]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:11.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:11 smithi040 ceph-mon[32418]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:51:11.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:11 smithi040 ceph-mon[32418]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T01:51:11.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:11 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:11.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:11 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:11.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:11 smithi040 ceph-mon[32418]: from='osd.10 ' entity='osd.10' 2024-04-25T01:51:11.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:11 smithi031 ceph-mon[29204]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:11.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:11 smithi031 ceph-mon[29204]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-25T01:51:11.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:11 smithi031 ceph-mon[29204]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T01:51:11.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:11 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:11.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:11 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:11.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:11 smithi031 ceph-mon[29204]: from='osd.10 ' entity='osd.10' 2024-04-25T01:51:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:51:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:51:12.455 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-04-25T01:51:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-04-25T01:51:12.456 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114181 s, 918 MB/s 2024-04-25T01:51:12.456 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-04-25T01:51:12.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:12 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:51:12.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:12 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:51:12.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:12 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:12.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:12 smithi134 ceph-mon[33098]: osd.10 [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197] boot 2024-04-25T01:51:12.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:12 smithi134 ceph-mon[33098]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:51:12.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:12 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:12.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:12 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:51:12.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:12 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:51:12.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:12 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:12.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:12 smithi040 ceph-mon[32418]: osd.10 [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197] boot 2024-04-25T01:51:12.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:12 smithi040 ceph-mon[32418]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:51:12.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:12 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:12.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:12 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:51:12.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:12 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:51:12.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:12 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:12.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:12 smithi031 ceph-mon[29204]: osd.10 [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197] boot 2024-04-25T01:51:12.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:12 smithi031 ceph-mon[29204]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:51:12.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:12 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:13.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi134:vg_nvme/lv_1 2024-04-25T01:51:13.394 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:51:13.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:51:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:13.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:13.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:51:13.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:13.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:13.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.750 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:13.750 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:13.750 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:13.750 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:13.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:13.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:51:13.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:13.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:13.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:13.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:13.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:13.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:13.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:14.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:14 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi040 2024-04-25T01:51:14.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:14 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-25T01:51:14.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:14 smithi134 ceph-mon[33098]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:51:14.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:14 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi040 2024-04-25T01:51:14.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:14 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-25T01:51:14.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:14 smithi040 ceph-mon[32418]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:51:14.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:14 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi040 2024-04-25T01:51:14.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:14 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-25T01:51:14.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:14 smithi031 ceph-mon[29204]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:51:15.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.004+0000 7fa9d8452700 1 -- 172.21.15.134:0/2847254468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d0102d20 msgr2=0x7fa9d01095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:15.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.004+0000 7fa9d8452700 1 --2- 172.21.15.134:0/2847254468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d0102d20 0x7fa9d01095d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800b0a0 tx=0x7fa9c800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:51:15.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.004+0000 7fa9d8452700 1 -- 172.21.15.134:0/2847254468 shutdown_connections 2024-04-25T01:51:15.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.004+0000 7fa9d8452700 1 --2- 172.21.15.134:0/2847254468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d0102d20 0x7fa9d01095d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:15.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.004+0000 7fa9d8452700 1 --2- 172.21.15.134:0/2847254468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9d0101180 0x7fa9d01015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:15.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.004+0000 7fa9d8452700 1 --2- 172.21.15.134:0/2847254468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9d0102380 0x7fa9d01027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:15.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.004+0000 7fa9d8452700 1 -- 172.21.15.134:0/2847254468 >> 172.21.15.134:0/2847254468 conn(0x7fa9d00fc720 msgr2=0x7fa9d00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:15.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.004+0000 7fa9d8452700 1 -- 172.21.15.134:0/2847254468 shutdown_connections 2024-04-25T01:51:15.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.004+0000 7fa9d8452700 1 -- 172.21.15.134:0/2847254468 wait complete. 2024-04-25T01:51:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.004+0000 7fa9d8452700 1 Processor -- start 2024-04-25T01:51:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.004+0000 7fa9d8452700 1 -- start start 2024-04-25T01:51:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d8452700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d0101180 0x7fa9d006b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d8452700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9d0102380 0x7fa9d019d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d8452700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9d0102d20 0x7fa9d019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d8452700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d010cd10 con 0x7fa9d0102d20 2024-04-25T01:51:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d8452700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d010cb90 con 0x7fa9d0102380 2024-04-25T01:51:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d8452700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d010ce90 con 0x7fa9d0101180 2024-04-25T01:51:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d61ee700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d0101180 0x7fa9d006b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d61ee700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d0101180 0x7fa9d006b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34482/0 (socket says 172.21.15.134:34482) 2024-04-25T01:51:15.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d61ee700 1 -- 172.21.15.134:0/709331236 learned_addr learned my addr 172.21.15.134:0/709331236 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d59ed700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9d0102380 0x7fa9d019d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d69ef700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9d0102d20 0x7fa9d019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d61ee700 1 -- 172.21.15.134:0/709331236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9d0102380 msgr2=0x7fa9d019d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d61ee700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9d0102380 0x7fa9d019d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d61ee700 1 -- 172.21.15.134:0/709331236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9d0102d20 msgr2=0x7fa9d019f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d61ee700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9d0102d20 0x7fa9d019f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d61ee700 1 -- 172.21.15.134:0/709331236 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d01a0130 con 0x7fa9d0101180 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d69ef700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9d0102d20 0x7fa9d019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d61ee700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d0101180 0x7fa9d006b540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00b500 tx=0x7fa9cc00b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9c77fe700 1 -- 172.21.15.134:0/709331236 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc015020 con 0x7fa9d0101180 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9c77fe700 1 -- 172.21.15.134:0/709331236 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9cc011ce0 con 0x7fa9d0101180 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9c77fe700 1 -- 172.21.15.134:0/709331236 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc010d60 con 0x7fa9d0101180 2024-04-25T01:51:15.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d8452700 1 -- 172.21.15.134:0/709331236 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d01a0420 con 0x7fa9d0101180 2024-04-25T01:51:15.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.005+0000 7fa9d8452700 1 -- 172.21.15.134:0/709331236 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d01a0a30 con 0x7fa9d0101180 2024-04-25T01:51:15.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.006+0000 7fa9d8452700 1 -- 172.21.15.134:0/709331236 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9d0068f20 con 0x7fa9d0101180 2024-04-25T01:51:15.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.008+0000 7fa9c77fe700 1 -- 172.21.15.134:0/709331236 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa9cc018020 con 0x7fa9d0101180 2024-04-25T01:51:15.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.009+0000 7fa9c77fe700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa9bc0a2ca0 0x7fa9bc0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:15.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.009+0000 7fa9c77fe700 1 -- 172.21.15.134:0/709331236 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6694+0+0 (secure 0 0 0) 0x7fa9cc0ce720 con 0x7fa9d0101180 2024-04-25T01:51:15.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.009+0000 7fa9d59ed700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa9bc0a2ca0 0x7fa9bc0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:15.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.009+0000 7fa9d59ed700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa9bc0a2ca0 0x7fa9bc0a5160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0006fd0 tx=0x7fa9c0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:15.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.012+0000 7fa9c77fe700 1 -- 172.21.15.134:0/709331236 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa9cc01f020 con 0x7fa9d0101180 2024-04-25T01:51:15.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:15.216+0000 7fa9d8452700 1 -- 172.21.15.134:0/709331236 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fa9d0062670 con 0x7fa9bc0a2ca0 2024-04-25T01:51:15.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:15 smithi040 ceph-mon[32418]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:15.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:15 smithi040 ceph-mon[32418]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T01:51:15.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:15 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:51:15.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:15 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:51:15.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:15 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:15.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:15 smithi134 ceph-mon[33098]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:15.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:15 smithi134 ceph-mon[33098]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T01:51:15.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:51:15.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:51:15.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:15 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:15.985 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:15 smithi031 ceph-mon[29204]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:15.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:15 smithi031 ceph-mon[29204]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T01:51:15.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:51:15.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:51:15.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:15 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:16.907 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:16 smithi134 ceph-mon[33098]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:16.907 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:16 smithi134 ceph-mon[33098]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T01:51:16.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:16 smithi040 ceph-mon[32418]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:16.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:16 smithi040 ceph-mon[32418]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T01:51:16.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:16 smithi031 ceph-mon[29204]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:16.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:16 smithi031 ceph-mon[29204]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T01:51:17.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:17 smithi040 ceph-mon[32418]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:51:17.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:17 smithi134 ceph-mon[33098]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:51:17.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:17 smithi031 ceph-mon[29204]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:51:18.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:18 smithi134 ceph-mon[33098]: from='client.? 172.21.15.134:0/4209653365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5113a81e-7097-4d97-8998-b59224bb6b28"}]: dispatch 2024-04-25T01:51:18.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:18 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5113a81e-7097-4d97-8998-b59224bb6b28"}]: dispatch 2024-04-25T01:51:18.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:18 smithi134 ceph-mon[33098]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5113a81e-7097-4d97-8998-b59224bb6b28"}]': finished 2024-04-25T01:51:18.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:18 smithi134 ceph-mon[33098]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:51:18.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:18 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:18.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:18 smithi031 ceph-mon[29204]: from='client.? 172.21.15.134:0/4209653365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5113a81e-7097-4d97-8998-b59224bb6b28"}]: dispatch 2024-04-25T01:51:18.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:18 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5113a81e-7097-4d97-8998-b59224bb6b28"}]: dispatch 2024-04-25T01:51:18.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:18 smithi031 ceph-mon[29204]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5113a81e-7097-4d97-8998-b59224bb6b28"}]': finished 2024-04-25T01:51:18.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:18 smithi031 ceph-mon[29204]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:51:18.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:18 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:18.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:18 smithi040 ceph-mon[32418]: from='client.? 172.21.15.134:0/4209653365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5113a81e-7097-4d97-8998-b59224bb6b28"}]: dispatch 2024-04-25T01:51:18.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:18 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5113a81e-7097-4d97-8998-b59224bb6b28"}]: dispatch 2024-04-25T01:51:18.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:18 smithi040 ceph-mon[32418]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5113a81e-7097-4d97-8998-b59224bb6b28"}]': finished 2024-04-25T01:51:18.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:18 smithi040 ceph-mon[32418]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:51:18.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:18 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:19.657 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:19 smithi040 ceph-mon[32418]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-25T01:51:19.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:19 smithi040 ceph-mon[32418]: from='client.? 172.21.15.134:0/1228568014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:51:19.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:19 smithi134 ceph-mon[33098]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-25T01:51:19.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:19 smithi134 ceph-mon[33098]: from='client.? 172.21.15.134:0/1228568014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:51:19.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:19 smithi031 ceph-mon[29204]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-25T01:51:19.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:19 smithi031 ceph-mon[29204]: from='client.? 172.21.15.134:0/1228568014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:51:21.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:21 smithi040 ceph-mon[32418]: pgmap v247: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:51:21.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:21 smithi134 ceph-mon[33098]: pgmap v247: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:51:21.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:21 smithi031 ceph-mon[29204]: pgmap v247: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:51:23.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:23 smithi040 ceph-mon[32418]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-25T01:51:23.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:23 smithi134 ceph-mon[33098]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-25T01:51:23.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:23 smithi031 ceph-mon[29204]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-25T01:51:25.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:25 smithi040 ceph-mon[32418]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:25.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:25 smithi134 ceph-mon[33098]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:25.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:25 smithi031 ceph-mon[29204]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:26.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:51:26.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:26.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:26 smithi040 ceph-mon[32418]: Deploying daemon osd.11 on smithi134 2024-04-25T01:51:26.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:51:26.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:26.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:26 smithi134 ceph-mon[33098]: Deploying daemon osd.11 on smithi134 2024-04-25T01:51:26.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:51:26.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:26.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:26 smithi031 ceph-mon[29204]: Deploying daemon osd.11 on smithi134 2024-04-25T01:51:27.898 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:27 smithi134 ceph-mon[33098]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:27.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:27 smithi040 ceph-mon[32418]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:27.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:27 smithi031 ceph-mon[29204]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:29.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:29 smithi040 ceph-mon[32418]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:29.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:29 smithi134 ceph-mon[33098]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:29.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:29 smithi031 ceph-mon[29204]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:31.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:31 smithi040 ceph-mon[32418]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:31.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:31 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:31.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:31 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:31.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:31 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:31.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:31 smithi134 ceph-mon[33098]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:31.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:31 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:31.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:31 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:31.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:31 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:31.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:31 smithi031 ceph-mon[29204]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:31.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:31 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:31.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:31 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:31.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:31 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:33.907 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:33 smithi040 ceph-mon[32418]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:33.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:33 smithi134 ceph-mon[33098]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:33.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:33 smithi031 ceph-mon[29204]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:34.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.992+0000 7fa9c77fe700 1 -- 172.21.15.134:0/709331236 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fa9d0062670 con 0x7fa9bc0a2ca0 2024-04-25T01:51:34.991 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 11 on host 'smithi134' 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 -- 172.21.15.134:0/709331236 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa9bc0a2ca0 msgr2=0x7fa9bc0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa9bc0a2ca0 0x7fa9bc0a5160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0006fd0 tx=0x7fa9c0008040 comp rx=0 tx=0).stop 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 -- 172.21.15.134:0/709331236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d0101180 msgr2=0x7fa9d006b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d0101180 0x7fa9d006b540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00b500 tx=0x7fa9cc00b8c0 comp rx=0 tx=0).stop 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 -- 172.21.15.134:0/709331236 shutdown_connections 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa9bc0a2ca0 0x7fa9bc0a5160 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9d0102d20 0x7fa9d019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d0101180 0x7fa9d006b540 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 --2- 172.21.15.134:0/709331236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9d0102380 0x7fa9d019d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 -- 172.21.15.134:0/709331236 >> 172.21.15.134:0/709331236 conn(0x7fa9d00fc720 msgr2=0x7fa9d01056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 -- 172.21.15.134:0/709331236 shutdown_connections 2024-04-25T01:51:34.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:34.994+0000 7fa9d8452700 1 -- 172.21.15.134:0/709331236 wait complete. 2024-04-25T01:51:35.673 DEBUG:teuthology.orchestra.run.smithi134:osd.11> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T01:51:35.678 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T01:51:35.678 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T01:51:35.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:35 smithi040 ceph-mon[32418]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:35.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:35 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:35.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:35 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:35.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:35 smithi040 ceph-mon[32418]: from='osd.11 [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:35.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:35 smithi040 ceph-mon[32418]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:35.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:35 smithi134 ceph-mon[33098]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:35.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:35 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:35.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:35 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:35.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:35 smithi134 ceph-mon[33098]: from='osd.11 [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:35.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:35 smithi134 ceph-mon[33098]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:35.939 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:51:35.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:35 smithi031 ceph-mon[29204]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:35.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:35 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:35.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:35 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:35.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:35 smithi031 ceph-mon[29204]: from='osd.11 [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:35.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:35 smithi031 ceph-mon[29204]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:36.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:36 smithi134 ceph-mon[33098]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:36.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:36 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:36.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:36 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:36.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:36 smithi134 ceph-mon[33098]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:51:36.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:36 smithi134 ceph-mon[33098]: from='osd.11 [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:51:36.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:36 smithi134 ceph-mon[33098]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T01:51:36.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:36 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:36.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:36 smithi134 ceph-mon[33098]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:51:36.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:36 smithi031 ceph-mon[29204]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:36.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:36 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:36.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:36 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:36.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:36 smithi031 ceph-mon[29204]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:51:36.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:36 smithi031 ceph-mon[29204]: from='osd.11 [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:51:36.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:36 smithi031 ceph-mon[29204]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T01:51:36.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:36 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:36.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:36 smithi031 ceph-mon[29204]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:51:37.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:36 smithi040 ceph-mon[32418]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:37.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:36 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:37.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:36 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:37.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:36 smithi040 ceph-mon[32418]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:51:37.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:36 smithi040 ceph-mon[32418]: from='osd.11 [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:51:37.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:36 smithi040 ceph-mon[32418]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T01:51:37.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:36 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:37.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:36 smithi040 ceph-mon[32418]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-25T01:51:37.345 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 01:51:36 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T01:51:36.977+0000 7f932eb4e700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T01:51:37.346 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 01:51:36 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T01:51:36.991+0000 7f932a941700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:51:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.818+0000 7f959a4b7700 1 -- 172.21.15.31:0/2174143286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95941034b0 msgr2=0x7f9594103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.818+0000 7f959a4b7700 1 --2- 172.21.15.31:0/2174143286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95941034b0 0x7f9594103910 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f957c0099c0 tx=0x7f957c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:51:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.818+0000 7f959a4b7700 1 -- 172.21.15.31:0/2174143286 shutdown_connections 2024-04-25T01:51:37.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.818+0000 7f959a4b7700 1 --2- 172.21.15.31:0/2174143286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9594103e50 0x7f959410a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.818+0000 7f959a4b7700 1 --2- 172.21.15.31:0/2174143286 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95941022b0 0x7f95941026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.818+0000 7f959a4b7700 1 --2- 172.21.15.31:0/2174143286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95941034b0 0x7f9594103910 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.818+0000 7f959a4b7700 1 -- 172.21.15.31:0/2174143286 >> 172.21.15.31:0/2174143286 conn(0x7f95940fd850 msgr2=0x7f95940ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.818+0000 7f959a4b7700 1 -- 172.21.15.31:0/2174143286 shutdown_connections 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.818+0000 7f959a4b7700 1 -- 172.21.15.31:0/2174143286 wait complete. 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f959a4b7700 1 Processor -- start 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f959a4b7700 1 -- start start 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f959a4b7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95941022b0 0x7f95941984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f959a4b7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95941034b0 0x7f95941989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f959a4b7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9594103e50 0x7f959419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f959a4b7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f959410de10 con 0x7f95941022b0 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f959a4b7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f959410dc90 con 0x7f95941034b0 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f959a4b7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f959410df90 con 0x7f9594103e50 2024-04-25T01:51:37.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f9593fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95941022b0 0x7f95941984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f9593fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95941022b0 0x7f95941984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41712/0 (socket says 172.21.15.31:41712) 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f9593fff700 1 -- 172.21.15.31:0/3398535338 learned_addr learned my addr 172.21.15.31:0/3398535338 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f9598a54700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9594103e50 0x7f959419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.819+0000 7f95937fe700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95941034b0 0x7f95941989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.820+0000 7f9593fff700 1 -- 172.21.15.31:0/3398535338 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9594103e50 msgr2=0x7f959419fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.820+0000 7f9593fff700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9594103e50 0x7f959419fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.820+0000 7f9593fff700 1 -- 172.21.15.31:0/3398535338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95941034b0 msgr2=0x7f95941989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.820+0000 7f9593fff700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95941034b0 0x7f95941989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.820+0000 7f9593fff700 1 -- 172.21.15.31:0/3398535338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95941a0110 con 0x7f95941022b0 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.820+0000 7f9593fff700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95941022b0 0x7f95941984b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f958400d9b0 tx=0x7f958400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.820+0000 7f95917fa700 1 -- 172.21.15.31:0/3398535338 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584017070 con 0x7f95941022b0 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.820+0000 7f95917fa700 1 -- 172.21.15.31:0/3398535338 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9584009de0 con 0x7f95941022b0 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.820+0000 7f959a4b7700 1 -- 172.21.15.31:0/3398535338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95941a0400 con 0x7f95941022b0 2024-04-25T01:51:37.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.820+0000 7f95917fa700 1 -- 172.21.15.31:0/3398535338 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958400f3c0 con 0x7f95941022b0 2024-04-25T01:51:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.820+0000 7f959a4b7700 1 -- 172.21.15.31:0/3398535338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95941a09c0 con 0x7f95941022b0 2024-04-25T01:51:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.821+0000 7f959a4b7700 1 -- 172.21.15.31:0/3398535338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9594068f20 con 0x7f95941022b0 2024-04-25T01:51:37.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.822+0000 7f95917fa700 1 -- 172.21.15.31:0/3398535338 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9584009930 con 0x7f95941022b0 2024-04-25T01:51:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.823+0000 7f95917fa700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f95800a2ca0 0x7f95800a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.823+0000 7f95937fe700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f95800a2ca0 0x7f95800a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.823+0000 7f95917fa700 1 -- 172.21.15.31:0/3398535338 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6847+0+0 (secure 0 0 0) 0x7f95840ce560 con 0x7f95941022b0 2024-04-25T01:51:37.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.823+0000 7f95937fe700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f95800a2ca0 0x7f95800a5160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f957c0038f0 tx=0x7f957c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:37.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:37.827+0000 7f95917fa700 1 -- 172.21.15.31:0/3398535338 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f958409ba60 con 0x7f95941022b0 2024-04-25T01:51:38.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.035+0000 7f959a4b7700 1 -- 172.21.15.31:0/3398535338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f9594062e40 con 0x7f95941022b0 2024-04-25T01:51:38.033 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:37 smithi031 ceph-mon[29204]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:51:38.034 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:37 smithi031 ceph-mon[29204]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T01:51:38.034 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:37 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:38.034 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:37 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:38.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.035+0000 7f95917fa700 1 -- 172.21.15.31:0/3398535338 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f958409b360 con 0x7f95941022b0 2024-04-25T01:51:38.034 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:51:38.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 -- 172.21.15.31:0/3398535338 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f95800a2ca0 msgr2=0x7f95800a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:38.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f95800a2ca0 0x7f95800a5160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f957c0038f0 tx=0x7f957c005fb0 comp rx=0 tx=0).stop 2024-04-25T01:51:38.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 -- 172.21.15.31:0/3398535338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95941022b0 msgr2=0x7f95941984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:38.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95941022b0 0x7f95941984b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f958400d9b0 tx=0x7f958400dcc0 comp rx=0 tx=0).stop 2024-04-25T01:51:38.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 -- 172.21.15.31:0/3398535338 shutdown_connections 2024-04-25T01:51:38.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f95800a2ca0 0x7f95800a5160 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:38.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9594103e50 0x7f959419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:38.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95941022b0 0x7f95941984b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:38.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 --2- 172.21.15.31:0/3398535338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95941034b0 0x7f95941989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:38.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 -- 172.21.15.31:0/3398535338 >> 172.21.15.31:0/3398535338 conn(0x7f95940fd850 msgr2=0x7f9594106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:38.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 -- 172.21.15.31:0/3398535338 shutdown_connections 2024-04-25T01:51:38.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:38.037+0000 7f959a4b7700 1 -- 172.21.15.31:0/3398535338 wait complete. 2024-04-25T01:51:38.187 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:37 smithi134 ceph-mon[33098]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:51:38.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:37 smithi134 ceph-mon[33098]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T01:51:38.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:37 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:38.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:37 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:38.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:37 smithi040 ceph-mon[32418]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-25T01:51:38.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:37 smithi040 ceph-mon[32418]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T01:51:38.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:37 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:38.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:37 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:38.836 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1714009897,"num_in_osds":12,"osd_in_since":1714009878,"num_remapped_pgs":0} 2024-04-25T01:51:38.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:51:39.096 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:51:39.187 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:38 smithi134 ceph-mon[33098]: purged_snaps scrub starts 2024-04-25T01:51:39.187 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:38 smithi134 ceph-mon[33098]: purged_snaps scrub ok 2024-04-25T01:51:39.187 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:38 smithi134 ceph-mon[33098]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:39.187 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:38 smithi134 ceph-mon[33098]: osd.11 [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774] boot 2024-04-25T01:51:39.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:38 smithi134 ceph-mon[33098]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:51:39.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:38 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:39.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:38 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3398535338' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:51:39.219 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:38 smithi031 ceph-mon[29204]: purged_snaps scrub starts 2024-04-25T01:51:39.219 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:38 smithi031 ceph-mon[29204]: purged_snaps scrub ok 2024-04-25T01:51:39.219 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:38 smithi031 ceph-mon[29204]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:39.219 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:38 smithi031 ceph-mon[29204]: osd.11 [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774] boot 2024-04-25T01:51:39.219 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:38 smithi031 ceph-mon[29204]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:51:39.219 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:38 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:39.219 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:38 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3398535338' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:51:39.350 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:38 smithi040 ceph-mon[32418]: purged_snaps scrub starts 2024-04-25T01:51:39.351 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:38 smithi040 ceph-mon[32418]: purged_snaps scrub ok 2024-04-25T01:51:39.351 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:38 smithi040 ceph-mon[32418]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:39.351 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:38 smithi040 ceph-mon[32418]: osd.11 [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774] boot 2024-04-25T01:51:39.351 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:38 smithi040 ceph-mon[32418]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:51:39.351 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:38 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:39.351 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:38 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3398535338' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:51:40.317 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:51:40.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: Detected new or changed devices on smithi134 2024-04-25T01:51:40.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi134 to 4272M 2024-04-25T01:51:40.318 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.319 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:40.319 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:40.319 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.319 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:39 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:40.377 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:51:40.377 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: Detected new or changed devices on smithi134 2024-04-25T01:51:40.377 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi134 to 4272M 2024-04-25T01:51:40.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:40.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:40.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:39 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:40.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: Detected new or changed devices on smithi134 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi134 to 4272M 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:40.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:39 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:40.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.736+0000 7fbaf4184700 1 -- 172.21.15.31:0/2187030833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaec1022b0 msgr2=0x7fbaec102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:40.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.736+0000 7fbaf4184700 1 --2- 172.21.15.31:0/2187030833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaec1022b0 0x7fbaec102710 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbadc009990 tx=0x7fbadc009ca0 comp rx=0 tx=0).stop 2024-04-25T01:51:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.736+0000 7fbaf4184700 1 -- 172.21.15.31:0/2187030833 shutdown_connections 2024-04-25T01:51:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.736+0000 7fbaf4184700 1 --2- 172.21.15.31:0/2187030833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaec102c50 0x7fbaec10f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.736+0000 7fbaf4184700 1 --2- 172.21.15.31:0/2187030833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaec1082f0 0x7fbaec1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.736+0000 7fbaf4184700 1 --2- 172.21.15.31:0/2187030833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaec1022b0 0x7fbaec102710 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.736+0000 7fbaf4184700 1 -- 172.21.15.31:0/2187030833 >> 172.21.15.31:0/2187030833 conn(0x7fbaec0793e0 msgr2=0x7fbaec0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.736+0000 7fbaf4184700 1 -- 172.21.15.31:0/2187030833 shutdown_connections 2024-04-25T01:51:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.736+0000 7fbaf4184700 1 -- 172.21.15.31:0/2187030833 wait complete. 2024-04-25T01:51:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf4184700 1 Processor -- start 2024-04-25T01:51:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf4184700 1 -- start start 2024-04-25T01:51:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf4184700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaec1022b0 0x7fbaec19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf4184700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaec102c50 0x7fbaec19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf4184700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaec1082f0 0x7fbaec198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf4184700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec112420 con 0x7fbaec1022b0 2024-04-25T01:51:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf4184700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec1122a0 con 0x7fbaec102c50 2024-04-25T01:51:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf4184700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec1125a0 con 0x7fbaec1082f0 2024-04-25T01:51:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf1f20700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaec1022b0 0x7fbaec19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:40.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf1f20700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaec1022b0 0x7fbaec19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41728/0 (socket says 172.21.15.31:41728) 2024-04-25T01:51:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf1f20700 1 -- 172.21.15.31:0/3620238625 learned_addr learned my addr 172.21.15.31:0/3620238625 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:51:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.737+0000 7fbaf2721700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaec1082f0 0x7fbaec198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbaf171f700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaec102c50 0x7fbaec19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbaf1f20700 1 -- 172.21.15.31:0/3620238625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaec1082f0 msgr2=0x7fbaec198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbaf1f20700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaec1082f0 0x7fbaec198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbaf1f20700 1 -- 172.21.15.31:0/3620238625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaec102c50 msgr2=0x7fbaec19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbaf1f20700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaec102c50 0x7fbaec19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbaf1f20700 1 -- 172.21.15.31:0/3620238625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaec199020 con 0x7fbaec1022b0 2024-04-25T01:51:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbaf1f20700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaec1022b0 0x7fbaec19e670 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbae800d980 tx=0x7fbae800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:40.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbae2ffd700 1 -- 172.21.15.31:0/3620238625 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae8016070 con 0x7fbaec1022b0 2024-04-25T01:51:40.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbaf4184700 1 -- 172.21.15.31:0/3620238625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaec199310 con 0x7fbaec1022b0 2024-04-25T01:51:40.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbae2ffd700 1 -- 172.21.15.31:0/3620238625 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbae8004830 con 0x7fbaec1022b0 2024-04-25T01:51:40.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbae2ffd700 1 -- 172.21.15.31:0/3620238625 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae8005530 con 0x7fbaec1022b0 2024-04-25T01:51:40.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.738+0000 7fbaf4184700 1 -- 172.21.15.31:0/3620238625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaec1a55f0 con 0x7fbaec1022b0 2024-04-25T01:51:40.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.739+0000 7fbaf4184700 1 -- 172.21.15.31:0/3620238625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaec1039a0 con 0x7fbaec1022b0 2024-04-25T01:51:40.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.740+0000 7fbae2ffd700 1 -- 172.21.15.31:0/3620238625 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbae80049e0 con 0x7fbaec1022b0 2024-04-25T01:51:40.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.741+0000 7fbae2ffd700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbad80a2c20 0x7fbad80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:40.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.741+0000 7fbaf171f700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbad80a2c20 0x7fbad80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:40.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.741+0000 7fbae2ffd700 1 -- 172.21.15.31:0/3620238625 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fbae80ced80 con 0x7fbaec1022b0 2024-04-25T01:51:40.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.741+0000 7fbaf171f700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbad80a2c20 0x7fbad80a50e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00b5c0 tx=0x7fbadc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:40.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.745+0000 7fbae2ffd700 1 -- 172.21.15.31:0/3620238625 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbae801f020 con 0x7fbaec1022b0 2024-04-25T01:51:40.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.951+0000 7fbaf4184700 1 -- 172.21.15.31:0/3620238625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbaec199c70 con 0x7fbaec1022b0 2024-04-25T01:51:40.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.953+0000 7fbae2ffd700 1 -- 172.21.15.31:0/3620238625 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18384 (secure 0 0 0) 0x7fbae809c180 con 0x7fbaec1022b0 2024-04-25T01:51:40.952 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:51:40.952 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:51:38.978864+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:51:40.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 -- 172.21.15.31:0/3620238625 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbad80a2c20 msgr2=0x7fbad80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbad80a2c20 0x7fbad80a50e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00b5c0 tx=0x7fbadc005960 comp rx=0 tx=0).stop 2024-04-25T01:51:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 -- 172.21.15.31:0/3620238625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaec1022b0 msgr2=0x7fbaec19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaec1022b0 0x7fbaec19e670 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbae800d980 tx=0x7fbae800dc90 comp rx=0 tx=0).stop 2024-04-25T01:51:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 -- 172.21.15.31:0/3620238625 shutdown_connections 2024-04-25T01:51:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbad80a2c20 0x7fbad80a50e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbaec1082f0 0x7fbaec198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaec1022b0 0x7fbaec19e670 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 --2- 172.21.15.31:0/3620238625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaec102c50 0x7fbaec19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 -- 172.21.15.31:0/3620238625 >> 172.21.15.31:0/3620238625 conn(0x7fbaec0793e0 msgr2=0x7fbaec1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 -- 172.21.15.31:0/3620238625 shutdown_connections 2024-04-25T01:51:40.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:40.955+0000 7fbaf4184700 1 -- 172.21.15.31:0/3620238625 wait complete. 2024-04-25T01:51:41.236 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:40 smithi031 ceph-mon[29204]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:41.236 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:40 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:41.236 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:40 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:41.236 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:40 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:41.236 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:40 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3620238625' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:41.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:40 smithi040 ceph-mon[32418]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:41.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:40 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:41.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:40 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:41.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:40 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:41.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:40 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3620238625' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:41.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:40 smithi134 ceph-mon[33098]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:41.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:40 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:41.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:40 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:41.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:40 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:41.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:40 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3620238625' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:41.594 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:51:41.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T01:51:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:51:43.231 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:42 smithi031 ceph-mon[29204]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:43.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:42 smithi040 ceph-mon[32418]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:43.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:42 smithi134 ceph-mon[33098]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.449+0000 7f7d2c445700 1 -- 172.21.15.31:0/256056176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d241071a0 msgr2=0x7f7d24107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.449+0000 7f7d2c445700 1 --2- 172.21.15.31:0/256056176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d241071a0 0x7f7d24107580 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c00b0a0 tx=0x7f7d1c00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:51:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.449+0000 7f7d2c445700 1 -- 172.21.15.31:0/256056176 shutdown_connections 2024-04-25T01:51:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.449+0000 7f7d2c445700 1 --2- 172.21.15.31:0/256056176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d24101af0 0x7f7d2410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.449+0000 7f7d2c445700 1 --2- 172.21.15.31:0/256056176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d241071a0 0x7f7d24107580 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.449+0000 7f7d2c445700 1 --2- 172.21.15.31:0/256056176 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d24101130 0x7f7d241015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.449+0000 7f7d2c445700 1 -- 172.21.15.31:0/256056176 >> 172.21.15.31:0/256056176 conn(0x7f7d240780e0 msgr2=0x7f7d240784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.449+0000 7f7d2c445700 1 -- 172.21.15.31:0/256056176 shutdown_connections 2024-04-25T01:51:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.449+0000 7f7d2c445700 1 -- 172.21.15.31:0/256056176 wait complete. 2024-04-25T01:51:43.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2c445700 1 Processor -- start 2024-04-25T01:51:43.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2c445700 1 -- start start 2024-04-25T01:51:43.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2c445700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d24101130 0x7f7d2419e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:43.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2c445700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d24101af0 0x7f7d2419e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2c445700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d241071a0 0x7f7d24198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2c445700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d241123f0 con 0x7f7d241071a0 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2c445700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7d24112270 con 0x7f7d24101af0 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2c445700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7d24112570 con 0x7f7d24101130 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2a1e1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d24101130 0x7f7d2419e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2a9e2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d241071a0 0x7f7d24198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2a1e1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d24101130 0x7f7d2419e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:38204/0 (socket says 172.21.15.31:38204) 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d2a1e1700 1 -- 172.21.15.31:0/2217122900 learned_addr learned my addr 172.21.15.31:0/2217122900 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.450+0000 7f7d299e0700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d24101af0 0x7f7d2419e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d2a9e2700 1 -- 172.21.15.31:0/2217122900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d24101130 msgr2=0x7f7d2419e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d2a9e2700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d24101130 0x7f7d2419e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d2a9e2700 1 -- 172.21.15.31:0/2217122900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d24101af0 msgr2=0x7f7d2419e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d2a9e2700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d24101af0 0x7f7d2419e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d2a9e2700 1 -- 172.21.15.31:0/2217122900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d24198dc0 con 0x7f7d241071a0 2024-04-25T01:51:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d2a1e1700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d24101130 0x7f7d2419e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:51:43.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d2a9e2700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d241071a0 0x7f7d24198660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7d1400c970 tx=0x7f7d1400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:43.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d237fe700 1 -- 172.21.15.31:0/2217122900 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1400e880 con 0x7f7d241071a0 2024-04-25T01:51:43.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d2c445700 1 -- 172.21.15.31:0/2217122900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d241990b0 con 0x7f7d241071a0 2024-04-25T01:51:43.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d237fe700 1 -- 172.21.15.31:0/2217122900 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d1400f450 con 0x7f7d241071a0 2024-04-25T01:51:43.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d237fe700 1 -- 172.21.15.31:0/2217122900 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d14018660 con 0x7f7d241071a0 2024-04-25T01:51:43.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.451+0000 7f7d2c445700 1 -- 172.21.15.31:0/2217122900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d241a5320 con 0x7f7d241071a0 2024-04-25T01:51:43.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.452+0000 7f7d2c445700 1 -- 172.21.15.31:0/2217122900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d24068f20 con 0x7f7d241071a0 2024-04-25T01:51:43.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.454+0000 7f7d237fe700 1 -- 172.21.15.31:0/2217122900 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7d1400f600 con 0x7f7d241071a0 2024-04-25T01:51:43.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.455+0000 7f7d237fe700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d0c0a2c20 0x7f7d0c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:43.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.455+0000 7f7d2a1e1700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d0c0a2c20 0x7f7d0c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:43.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.455+0000 7f7d237fe700 1 -- 172.21.15.31:0/2217122900 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f7d140cea40 con 0x7f7d241071a0 2024-04-25T01:51:43.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.455+0000 7f7d2a1e1700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d0c0a2c20 0x7f7d0c0a50e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009e60 tx=0x7f7d1c00b7d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:43.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.458+0000 7f7d237fe700 1 -- 172.21.15.31:0/2217122900 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7d14014030 con 0x7f7d241071a0 2024-04-25T01:51:43.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.664+0000 7f7d2c445700 1 -- 172.21.15.31:0/2217122900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7d24199b40 con 0x7f7d241071a0 2024-04-25T01:51:43.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.664+0000 7f7d237fe700 1 -- 172.21.15.31:0/2217122900 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f7d1409be30 con 0x7f7d241071a0 2024-04-25T01:51:43.664 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T01:51:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 -- 172.21.15.31:0/2217122900 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d0c0a2c20 msgr2=0x7f7d0c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d0c0a2c20 0x7f7d0c0a50e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009e60 tx=0x7f7d1c00b7d0 comp rx=0 tx=0).stop 2024-04-25T01:51:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 -- 172.21.15.31:0/2217122900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d241071a0 msgr2=0x7f7d24198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d241071a0 0x7f7d24198660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7d1400c970 tx=0x7f7d1400cc80 comp rx=0 tx=0).stop 2024-04-25T01:51:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 -- 172.21.15.31:0/2217122900 shutdown_connections 2024-04-25T01:51:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7d0c0a2c20 0x7f7d0c0a50e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d241071a0 0x7f7d24198660 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d24101130 0x7f7d2419e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 --2- 172.21.15.31:0/2217122900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d24101af0 0x7f7d2419e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 -- 172.21.15.31:0/2217122900 >> 172.21.15.31:0/2217122900 conn(0x7f7d240780e0 msgr2=0x7f7d240ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:43.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 -- 172.21.15.31:0/2217122900 shutdown_connections 2024-04-25T01:51:43.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:43.666+0000 7f7d2c445700 1 -- 172.21.15.31:0/2217122900 wait complete. 2024-04-25T01:51:44.236 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:43 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2217122900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:51:44.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:43 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2217122900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:51:44.425 INFO:tasks.cephadm:Adding mds.a on smithi031 2024-04-25T01:51:44.425 INFO:tasks.cephadm:Adding mds.d on smithi031 2024-04-25T01:51:44.425 INFO:tasks.cephadm:Adding mds.g on smithi031 2024-04-25T01:51:44.425 INFO:tasks.cephadm:Adding mds.j on smithi031 2024-04-25T01:51:44.425 INFO:tasks.cephadm:Adding mds.b on smithi040 2024-04-25T01:51:44.425 INFO:tasks.cephadm:Adding mds.e on smithi040 2024-04-25T01:51:44.425 INFO:tasks.cephadm:Adding mds.h on smithi040 2024-04-25T01:51:44.425 INFO:tasks.cephadm:Adding mds.k on smithi040 2024-04-25T01:51:44.425 INFO:tasks.cephadm:Adding mds.c on smithi134 2024-04-25T01:51:44.425 INFO:tasks.cephadm:Adding mds.f on smithi134 2024-04-25T01:51:44.426 INFO:tasks.cephadm:Adding mds.i on smithi134 2024-04-25T01:51:44.426 INFO:tasks.cephadm:Adding mds.l on smithi134 2024-04-25T01:51:44.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi134=c;smithi134=f;smithi134=i;smithi134=l' 2024-04-25T01:51:44.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:44 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2217122900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:51:44.675 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:51:45.134 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:45 smithi134 ceph-mon[33098]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:45.408 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:45 smithi040 ceph-mon[32418]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:45.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:45 smithi031 ceph-mon[29204]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:46.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.287+0000 7fbcb3e51700 1 -- 172.21.15.134:0/2519984182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcac1014b0 msgr2=0x7fbcac101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:46.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.287+0000 7fbcb3e51700 1 --2- 172.21.15.134:0/2519984182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcac1014b0 0x7fbcac101910 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0099c0 tx=0x7fbc9c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.287+0000 7fbcb3e51700 1 -- 172.21.15.134:0/2519984182 shutdown_connections 2024-04-25T01:51:46.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.287+0000 7fbcb3e51700 1 --2- 172.21.15.134:0/2519984182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcac101e50 0x7fbcac109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.287+0000 7fbcb3e51700 1 --2- 172.21.15.134:0/2519984182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac102770 0x7fbcac102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.287+0000 7fbcb3e51700 1 --2- 172.21.15.134:0/2519984182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcac1014b0 0x7fbcac101910 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.287+0000 7fbcb3e51700 1 -- 172.21.15.134:0/2519984182 >> 172.21.15.134:0/2519984182 conn(0x7fbcac0fbf00 msgr2=0x7fbcac0fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.287+0000 7fbcb3e51700 1 -- 172.21.15.134:0/2519984182 shutdown_connections 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.287+0000 7fbcb3e51700 1 -- 172.21.15.134:0/2519984182 wait complete. 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.287+0000 7fbcb3e51700 1 Processor -- start 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.287+0000 7fbcb3e51700 1 -- start start 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb3e51700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcac1014b0 0x7fbcac100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb3e51700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101e50 0x7fbcac19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb3e51700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcac102770 0x7fbcac1a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb3e51700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac10cc90 con 0x7fbcac101e50 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb3e51700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac10cb10 con 0x7fbcac1014b0 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb3e51700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac10ce10 con 0x7fbcac102770 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb23ee700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcac102770 0x7fbcac1a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb23ee700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcac102770 0x7fbcac1a1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:44582/0 (socket says 172.21.15.134:44582) 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb23ee700 1 -- 172.21.15.134:0/976758700 learned_addr learned my addr 172.21.15.134:0/976758700 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-25T01:51:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb13ec700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101e50 0x7fbcac19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb1bed700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcac1014b0 0x7fbcac100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb23ee700 1 -- 172.21.15.134:0/976758700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcac1014b0 msgr2=0x7fbcac100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb23ee700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcac1014b0 0x7fbcac100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb23ee700 1 -- 172.21.15.134:0/976758700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101e50 msgr2=0x7fbcac19f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb23ee700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101e50 0x7fbcac19f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb23ee700 1 -- 172.21.15.134:0/976758700 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcac1a2290 con 0x7fbcac102770 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb23ee700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcac102770 0x7fbcac1a1ab0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbca400ef30 tx=0x7fbca400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbca2ffd700 1 -- 172.21.15.134:0/976758700 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca40092e0 con 0x7fbcac102770 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbca2ffd700 1 -- 172.21.15.134:0/976758700 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbca4004bb0 con 0x7fbcac102770 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb3e51700 1 -- 172.21.15.134:0/976758700 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcac1a2520 con 0x7fbcac102770 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbca2ffd700 1 -- 172.21.15.134:0/976758700 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca4004060 con 0x7fbcac102770 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.288+0000 7fbcb3e51700 1 -- 172.21.15.134:0/976758700 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcac1a29e0 con 0x7fbcac102770 2024-04-25T01:51:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.289+0000 7fbcb3e51700 1 -- 172.21.15.134:0/976758700 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcac068f20 con 0x7fbcac102770 2024-04-25T01:51:46.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.291+0000 7fbca2ffd700 1 -- 172.21.15.134:0/976758700 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbca401b030 con 0x7fbcac102770 2024-04-25T01:51:46.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.291+0000 7fbca2ffd700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc980a2c20 0x7fbc980a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:46.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.291+0000 7fbca2ffd700 1 -- 172.21.15.134:0/976758700 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fbca40cf130 con 0x7fbcac102770 2024-04-25T01:51:46.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.291+0000 7fbcb1bed700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc980a2c20 0x7fbc980a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:46.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.292+0000 7fbcb1bed700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc980a2c20 0x7fbc980a50e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbcac06a700 tx=0x7fbca800bef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:46.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.295+0000 7fbca2ffd700 1 -- 172.21.15.134:0/976758700 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbca4022020 con 0x7fbcac102770 2024-04-25T01:51:46.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.503+0000 7fbcb3e51700 1 -- 172.21.15.134:0/976758700 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi134=c;smithi134=f;smithi134=i;smithi134=l", "target": ["mon-mgr", ""]}) -- 0x7fbcac062930 con 0x7fbc980a2c20 2024-04-25T01:51:46.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.511+0000 7fbca2ffd700 1 -- 172.21.15.134:0/976758700 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fbcac062930 con 0x7fbc980a2c20 2024-04-25T01:51:46.510 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled mds.all update... 2024-04-25T01:51:46.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 -- 172.21.15.134:0/976758700 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc980a2c20 msgr2=0x7fbc980a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc980a2c20 0x7fbc980a50e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbcac06a700 tx=0x7fbca800bef0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 -- 172.21.15.134:0/976758700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcac102770 msgr2=0x7fbcac1a1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcac102770 0x7fbcac1a1ab0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbca400ef30 tx=0x7fbca400c3a0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 -- 172.21.15.134:0/976758700 shutdown_connections 2024-04-25T01:51:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fbc980a2c20 0x7fbc980a50e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcac102770 0x7fbcac1a1ab0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcac1014b0 0x7fbcac100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 --2- 172.21.15.134:0/976758700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcac101e50 0x7fbcac19f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 -- 172.21.15.134:0/976758700 >> 172.21.15.134:0/976758700 conn(0x7fbcac0fbf00 msgr2=0x7fbcac105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 -- 172.21.15.134:0/976758700 shutdown_connections 2024-04-25T01:51:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-25T01:51:46.514+0000 7fbcb3e51700 1 -- 172.21.15.134:0/976758700 wait complete. 2024-04-25T01:51:47.188 DEBUG:teuthology.orchestra.run.smithi031:mds.a> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T01:51:47.191 DEBUG:teuthology.orchestra.run.smithi031:mds.d> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T01:51:47.234 DEBUG:teuthology.orchestra.run.smithi031:mds.g> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T01:51:47.277 DEBUG:teuthology.orchestra.run.smithi031:mds.j> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T01:51:47.321 DEBUG:teuthology.orchestra.run.smithi040:mds.b> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T01:51:47.324 DEBUG:teuthology.orchestra.run.smithi040:mds.e> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T01:51:47.367 DEBUG:teuthology.orchestra.run.smithi040:mds.h> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T01:51:47.410 DEBUG:teuthology.orchestra.run.smithi040:mds.k> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T01:51:47.453 DEBUG:teuthology.orchestra.run.smithi134:mds.c> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T01:51:47.455 DEBUG:teuthology.orchestra.run.smithi134:mds.f> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T01:51:47.498 DEBUG:teuthology.orchestra.run.smithi134:mds.i> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T01:51:47.541 DEBUG:teuthology.orchestra.run.smithi134:mds.l> sudo journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T01:51:47.584 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T01:51:47.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:51:47.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:47.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi134=c;smithi134=f;smithi134=i;smithi134=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:47.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi134=c;smithi134=f;smithi134=i;smithi134=l;count:12 2024-04-25T01:51:47.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:47.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:47.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:47.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:47.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:47.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:47.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:47.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:47.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:47 smithi040 ceph-mon[32418]: Deploying daemon mds.l on smithi134 2024-04-25T01:51:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.705+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2756969055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb58102070 msgr2=0x7fcb58108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.705+0000 7fcb5e39e640 1 --2- 172.21.15.31:0/2756969055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb58102070 0x7fcb58108900 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcb50009ca0 tx=0x7fcb5002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:51:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.705+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2756969055 shutdown_connections 2024-04-25T01:51:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.705+0000 7fcb5e39e640 1 --2- 172.21.15.31:0/2756969055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb58102070 0x7fcb58108900 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.705+0000 7fcb5e39e640 1 --2- 172.21.15.31:0/2756969055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb581016b0 0x7fcb58101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.705+0000 7fcb5e39e640 1 --2- 172.21.15.31:0/2756969055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb581004c0 0x7fcb581008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.705+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2756969055 >> 172.21.15.31:0/2756969055 conn(0x7fcb580fba10 msgr2=0x7fcb580fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.705+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2756969055 shutdown_connections 2024-04-25T01:51:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.705+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2756969055 wait complete. 2024-04-25T01:51:47.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5e39e640 1 Processor -- start 2024-04-25T01:51:47.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5e39e640 1 -- start start 2024-04-25T01:51:47.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5e39e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb581004c0 0x7fcb581977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:47.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5e39e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb581016b0 0x7fcb58197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5e39e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb58102070 0x7fcb5819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5e39e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5810c050 con 0x7fcb581016b0 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5e39e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5810bed0 con 0x7fcb581004c0 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5e39e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5810c1d0 con 0x7fcb58102070 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5cb9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb581016b0 0x7fcb58197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5cb9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb581016b0 0x7fcb58197d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53548/0 (socket says 172.21.15.31:53548) 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5cb9b640 1 -- 172.21.15.31:0/2185595943 learned_addr learned my addr 172.21.15.31:0/2185595943 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5db9d640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb58102070 0x7fcb5819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5d39c640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb581004c0 0x7fcb581977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5cb9b640 1 -- 172.21.15.31:0/2185595943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb58102070 msgr2=0x7fcb5819ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5cb9b640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb58102070 0x7fcb5819ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5cb9b640 1 -- 172.21.15.31:0/2185595943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb581004c0 msgr2=0x7fcb581977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5cb9b640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb581004c0 0x7fcb581977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5cb9b640 1 -- 172.21.15.31:0/2185595943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5819f3f0 con 0x7fcb581016b0 2024-04-25T01:51:47.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5db9d640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb58102070 0x7fcb5819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:51:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.706+0000 7fcb5cb9b640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb581016b0 0x7fcb58197d10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800d950 tx=0x7fcb4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.707+0000 7fcb4e7fc640 1 -- 172.21.15.31:0/2185595943 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48017070 con 0x7fcb581016b0 2024-04-25T01:51:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.707+0000 7fcb4e7fc640 1 -- 172.21.15.31:0/2185595943 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb48009c30 con 0x7fcb581016b0 2024-04-25T01:51:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.707+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2185595943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb5819f6e0 con 0x7fcb581016b0 2024-04-25T01:51:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.707+0000 7fcb4e7fc640 1 -- 172.21.15.31:0/2185595943 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4800f620 con 0x7fcb581016b0 2024-04-25T01:51:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.707+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2185595943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb5819fc30 con 0x7fcb581016b0 2024-04-25T01:51:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.708+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2185595943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb5819fe90 con 0x7fcb581016b0 2024-04-25T01:51:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.709+0000 7fcb4e7fc640 1 -- 172.21.15.31:0/2185595943 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcb480109a0 con 0x7fcb581016b0 2024-04-25T01:51:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.710+0000 7fcb4e7fc640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fcb440a2b90 0x7fcb440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.710+0000 7fcb5d39c640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fcb440a2b90 0x7fcb440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.710+0000 7fcb4e7fc640 1 -- 172.21.15.31:0/2185595943 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fcb480ce900 con 0x7fcb581016b0 2024-04-25T01:51:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.710+0000 7fcb5d39c640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fcb440a2b90 0x7fcb440a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcb58101510 tx=0x7fcb54005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:47.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.714+0000 7fcb4e7fc640 1 -- 172.21.15.31:0/2185595943 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcb48009de0 con 0x7fcb581016b0 2024-04-25T01:51:47.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:47.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi134=c;smithi134=f;smithi134=i;smithi134=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:47.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi134=c;smithi134=f;smithi134=i;smithi134=l;count:12 2024-04-25T01:51:47.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:47.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:47.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:47.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:47.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:47.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:47.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:47.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:47.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:47 smithi134 ceph-mon[33098]: Deploying daemon mds.l on smithi134 2024-04-25T01:51:47.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.902+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2185595943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb581a0060 con 0x7fcb581016b0 2024-04-25T01:51:47.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.903+0000 7fcb4e7fc640 1 -- 172.21.15.31:0/2185595943 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18384 (secure 0 0 0) 0x7fcb4809bc60 con 0x7fcb581016b0 2024-04-25T01:51:47.903 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:51:47.903 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:51:38.978864+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:51:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.905+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2185595943 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fcb440a2b90 msgr2=0x7fcb440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.905+0000 7fcb5e39e640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fcb440a2b90 0x7fcb440a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcb58101510 tx=0x7fcb54005f90 comp rx=0 tx=0).stop 2024-04-25T01:51:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.905+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2185595943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb581016b0 msgr2=0x7fcb58197d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.905+0000 7fcb5e39e640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb581016b0 0x7fcb58197d10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800d950 tx=0x7fcb4800de10 comp rx=0 tx=0).stop 2024-04-25T01:51:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.905+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2185595943 shutdown_connections 2024-04-25T01:51:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.905+0000 7fcb5e39e640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fcb440a2b90 0x7fcb440a5050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.905+0000 7fcb5e39e640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb58102070 0x7fcb5819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.905+0000 7fcb5e39e640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb581016b0 0x7fcb58197d10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.905+0000 7fcb5e39e640 1 --2- 172.21.15.31:0/2185595943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb581004c0 0x7fcb581977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.905+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2185595943 >> 172.21.15.31:0/2185595943 conn(0x7fcb580fba10 msgr2=0x7fcb580fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:47.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.906+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2185595943 shutdown_connections 2024-04-25T01:51:47.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:47.906+0000 7fcb5e39e640 1 -- 172.21.15.31:0/2185595943 wait complete. 2024-04-25T01:51:47.915 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:47.915 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi134=c;smithi134=f;smithi134=i;smithi134=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:47.915 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi134=c;smithi134=f;smithi134=i;smithi134=l;count:12 2024-04-25T01:51:47.915 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:47.915 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:47.915 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:47.915 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:47.915 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:47.915 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:47.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:47.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:47.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:47 smithi031 ceph-mon[29204]: Deploying daemon mds.l on smithi134 2024-04-25T01:51:47.917 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:51:47.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:51:48.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.023+0000 7f6c56e58640 1 -- 172.21.15.31:0/1785264303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c50102af0 msgr2=0x7f6c50102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:48.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.023+0000 7f6c56e58640 1 --2- 172.21.15.31:0/1785264303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c50102af0 0x7f6c50102f70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c009930 tx=0x7f6c3c02a420 comp rx=0 tx=0).stop 2024-04-25T01:51:48.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.023+0000 7f6c56e58640 1 -- 172.21.15.31:0/1785264303 shutdown_connections 2024-04-25T01:51:48.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.023+0000 7f6c56e58640 1 --2- 172.21.15.31:0/1785264303 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c501034b0 0x7f6c50109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.023+0000 7f6c56e58640 1 --2- 172.21.15.31:0/1785264303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c50102af0 0x7f6c50102f70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.023+0000 7f6c56e58640 1 --2- 172.21.15.31:0/1785264303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c500ffa80 0x7f6c500ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.023+0000 7f6c56e58640 1 -- 172.21.15.31:0/1785264303 >> 172.21.15.31:0/1785264303 conn(0x7f6c500fb370 msgr2=0x7f6c500fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.023+0000 7f6c56e58640 1 -- 172.21.15.31:0/1785264303 shutdown_connections 2024-04-25T01:51:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.023+0000 7f6c56e58640 1 -- 172.21.15.31:0/1785264303 wait complete. 2024-04-25T01:51:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.023+0000 7f6c56e58640 1 Processor -- start 2024-04-25T01:51:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.023+0000 7f6c56e58640 1 -- start start 2024-04-25T01:51:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c56e58640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c500ffa80 0x7f6c501979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:48.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c56e58640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c50102af0 0x7f6c50197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c56e58640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c501034b0 0x7f6c5019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c56e58640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5010d2e0 con 0x7f6c500ffa80 2024-04-25T01:51:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c56e58640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5010d160 con 0x7f6c501034b0 2024-04-25T01:51:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c56e58640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5010d460 con 0x7f6c50102af0 2024-04-25T01:51:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c54bf4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c500ffa80 0x7f6c501979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c54bf4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c500ffa80 0x7f6c501979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53556/0 (socket says 172.21.15.31:53556) 2024-04-25T01:51:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c54bf4640 1 -- 172.21.15.31:0/1154329575 learned_addr learned my addr 172.21.15.31:0/1154329575 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:51:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c4ffff640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c50102af0 0x7f6c50197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:48.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c553f5640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c501034b0 0x7f6c5019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c54bf4640 1 -- 172.21.15.31:0/1154329575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c50102af0 msgr2=0x7f6c50197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c54bf4640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c50102af0 0x7f6c50197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c54bf4640 1 -- 172.21.15.31:0/1154329575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c501034b0 msgr2=0x7f6c5019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c54bf4640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c501034b0 0x7f6c5019ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c54bf4640 1 -- 172.21.15.31:0/1154329575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c5019f600 con 0x7f6c500ffa80 2024-04-25T01:51:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c54bf4640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c500ffa80 0x7f6c501979e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400c970 tx=0x7f6c4400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c4dffb640 1 -- 172.21.15.31:0/1154329575 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c44004220 con 0x7f6c500ffa80 2024-04-25T01:51:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c4dffb640 1 -- 172.21.15.31:0/1154329575 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c440043c0 con 0x7f6c500ffa80 2024-04-25T01:51:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c56e58640 1 -- 172.21.15.31:0/1154329575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c5019f8f0 con 0x7f6c500ffa80 2024-04-25T01:51:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.024+0000 7f6c4dffb640 1 -- 172.21.15.31:0/1154329575 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4400f450 con 0x7f6c500ffa80 2024-04-25T01:51:48.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.025+0000 7f6c56e58640 1 -- 172.21.15.31:0/1154329575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c5019fe60 con 0x7f6c500ffa80 2024-04-25T01:51:48.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.026+0000 7f6c56e58640 1 -- 172.21.15.31:0/1154329575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c50108ca0 con 0x7f6c500ffa80 2024-04-25T01:51:48.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.026+0000 7f6c4dffb640 1 -- 172.21.15.31:0/1154329575 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6c440079b0 con 0x7f6c500ffa80 2024-04-25T01:51:48.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.027+0000 7f6c4dffb640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c400a2c10 0x7f6c400a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:48.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.027+0000 7f6c4ffff640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c400a2c10 0x7f6c400a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:48.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.027+0000 7f6c4dffb640 1 -- 172.21.15.31:0/1154329575 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f6c440ce910 con 0x7f6c500ffa80 2024-04-25T01:51:48.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.028+0000 7f6c4ffff640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c400a2c10 0x7f6c400a50d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c004770 tx=0x7f6c3c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:48.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.032+0000 7f6c4dffb640 1 -- 172.21.15.31:0/1154329575 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6c44014030 con 0x7f6c500ffa80 2024-04-25T01:51:48.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.215+0000 7f6c56e58640 1 -- 172.21.15.31:0/1154329575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6c5002d340 con 0x7f6c500ffa80 2024-04-25T01:51:48.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.216+0000 7f6c4dffb640 1 -- 172.21.15.31:0/1154329575 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f6c4409bd10 con 0x7f6c500ffa80 2024-04-25T01:51:48.215 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T01:51:48.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 -- 172.21.15.31:0/1154329575 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c400a2c10 msgr2=0x7f6c400a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:48.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c400a2c10 0x7f6c400a50d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c004770 tx=0x7f6c3c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 -- 172.21.15.31:0/1154329575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c500ffa80 msgr2=0x7f6c501979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c500ffa80 0x7f6c501979e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400c970 tx=0x7f6c4400ce30 comp rx=0 tx=0).stop 2024-04-25T01:51:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 -- 172.21.15.31:0/1154329575 shutdown_connections 2024-04-25T01:51:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c400a2c10 0x7f6c400a50d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c501034b0 0x7f6c5019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c50102af0 0x7f6c50197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 --2- 172.21.15.31:0/1154329575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c500ffa80 0x7f6c501979e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 -- 172.21.15.31:0/1154329575 >> 172.21.15.31:0/1154329575 conn(0x7f6c500fb370 msgr2=0x7f6c50105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 -- 172.21.15.31:0/1154329575 shutdown_connections 2024-04-25T01:51:48.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.218+0000 7f6c56e58640 1 -- 172.21.15.31:0/1154329575 wait complete. 2024-04-25T01:51:48.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:51:48.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.332+0000 7f9130eca640 1 -- 172.21.15.31:0/848116765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f912c1017b0 msgr2=0x7f912c101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:48.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.332+0000 7f9130eca640 1 --2- 172.21.15.31:0/848116765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f912c1017b0 0x7f912c101bd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f911c009900 tx=0x7f911c02a410 comp rx=0 tx=0).stop 2024-04-25T01:51:48.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.332+0000 7f9130eca640 1 -- 172.21.15.31:0/848116765 shutdown_connections 2024-04-25T01:51:48.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.332+0000 7f9130eca640 1 --2- 172.21.15.31:0/848116765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f912c103360 0x7f912c109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.332+0000 7f9130eca640 1 --2- 172.21.15.31:0/848116765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f912c1029a0 0x7f912c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.332+0000 7f9130eca640 1 --2- 172.21.15.31:0/848116765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f912c1017b0 0x7f912c101bd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.332+0000 7f9130eca640 1 -- 172.21.15.31:0/848116765 >> 172.21.15.31:0/848116765 conn(0x7f912c0fcd40 msgr2=0x7f912c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.332+0000 7f9130eca640 1 -- 172.21.15.31:0/848116765 shutdown_connections 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.332+0000 7f9130eca640 1 -- 172.21.15.31:0/848116765 wait complete. 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9130eca640 1 Processor -- start 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9130eca640 1 -- start start 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9130eca640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f912c1017b0 0x7f912c0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9130eca640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f912c1029a0 0x7f912c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9130eca640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f912c103360 0x7f912c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9130eca640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f912c10d320 con 0x7f912c1029a0 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9130eca640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f912c10d1a0 con 0x7f912c103360 2024-04-25T01:51:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9130eca640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f912c10d4a0 con 0x7f912c1017b0 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9129d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f912c1029a0 0x7f912c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9129d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f912c1029a0 0x7f912c073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53570/0 (socket says 172.21.15.31:53570) 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9129d9b640 1 -- 172.21.15.31:0/1211875616 learned_addr learned my addr 172.21.15.31:0/1211875616 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f912a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f912c1017b0 0x7f912c0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f912ad9d640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f912c103360 0x7f912c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9129d9b640 1 -- 172.21.15.31:0/1211875616 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f912c1017b0 msgr2=0x7f912c0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9129d9b640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f912c1017b0 0x7f912c0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9129d9b640 1 -- 172.21.15.31:0/1211875616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f912c103360 msgr2=0x7f912c074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9129d9b640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f912c103360 0x7f912c074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9129d9b640 1 -- 172.21.15.31:0/1211875616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f912c074900 con 0x7f912c1029a0 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f912a59c640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f912c1017b0 0x7f912c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f9129d9b640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f912c1029a0 0x7f912c073b30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f912000e9f0 tx=0x7f912000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.333+0000 7f911b7fe640 1 -- 172.21.15.31:0/1211875616 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912000cd80 con 0x7f912c1029a0 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.334+0000 7f911b7fe640 1 -- 172.21.15.31:0/1211875616 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91200054d0 con 0x7f912c1029a0 2024-04-25T01:51:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.334+0000 7f9130eca640 1 -- 172.21.15.31:0/1211875616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f912c1a1b10 con 0x7f912c1029a0 2024-04-25T01:51:48.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.334+0000 7f911b7fe640 1 -- 172.21.15.31:0/1211875616 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120010490 con 0x7f912c1029a0 2024-04-25T01:51:48.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.334+0000 7f9130eca640 1 -- 172.21.15.31:0/1211875616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f912c1a2010 con 0x7f912c1029a0 2024-04-25T01:51:48.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.334+0000 7f9130eca640 1 -- 172.21.15.31:0/1211875616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f912c108b70 con 0x7f912c1029a0 2024-04-25T01:51:48.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.336+0000 7f911b7fe640 1 -- 172.21.15.31:0/1211875616 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9120005000 con 0x7f912c1029a0 2024-04-25T01:51:48.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.336+0000 7f911b7fe640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f91140a2b90 0x7f91140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:48.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.337+0000 7f912a59c640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f91140a2b90 0x7f91140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:48.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.337+0000 7f911b7fe640 1 -- 172.21.15.31:0/1211875616 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f91200cf2c0 con 0x7f912c1029a0 2024-04-25T01:51:48.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.337+0000 7f912a59c640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f91140a2b90 0x7f91140a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f911c005ab0 tx=0x7f911c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:48.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.340+0000 7f911b7fe640 1 -- 172.21.15.31:0/1211875616 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9120053740 con 0x7f912c1029a0 2024-04-25T01:51:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.565+0000 7f9130eca640 1 -- 172.21.15.31:0/1211875616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f912c02d170 con 0x7f912c1029a0 2024-04-25T01:51:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.565+0000 7f911b7fe640 1 -- 172.21.15.31:0/1211875616 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f91200052f0 con 0x7f912c1029a0 2024-04-25T01:51:48.565 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:51:48.565 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T01:51:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 1 2024-04-25T01:51:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 -- 172.21.15.31:0/1211875616 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f91140a2b90 msgr2=0x7f91140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f91140a2b90 0x7f91140a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f911c005ab0 tx=0x7f911c005a20 comp rx=0 tx=0).stop 2024-04-25T01:51:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 -- 172.21.15.31:0/1211875616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f912c1029a0 msgr2=0x7f912c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f912c1029a0 0x7f912c073b30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f912000e9f0 tx=0x7f912000edf0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 -- 172.21.15.31:0/1211875616 shutdown_connections 2024-04-25T01:51:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f91140a2b90 0x7f91140a5050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f912c103360 0x7f912c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f912c1029a0 0x7f912c073b30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 --2- 172.21.15.31:0/1211875616 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f912c1017b0 0x7f912c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 -- 172.21.15.31:0/1211875616 >> 172.21.15.31:0/1211875616 conn(0x7f912c0fcd40 msgr2=0x7f912c105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 -- 172.21.15.31:0/1211875616 shutdown_connections 2024-04-25T01:51:48.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:48.567+0000 7f9130eca640 1 -- 172.21.15.31:0/1211875616 wait complete. 2024-04-25T01:51:48.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:48 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2185595943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:48.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:48 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1154329575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:51:48.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:48 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2185595943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:48.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:48 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1154329575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:51:48.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:48 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2185595943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:48.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:48 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1154329575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:51:49.097 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 01:51:48 smithi134 systemd[1]: Starting Ceph mds.l for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:51:49.367 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 01:51:49 smithi134 podman[63042]: 2024-04-25 01:51:49.000959918 +0000 UTC m=+0.017522813 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:51:49.367 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 01:51:49 smithi134 podman[63042]: 2024-04-25 01:51:49.132984925 +0000 UTC m=+0.149547877 container create 7dcfa65053c5e5b3bb73a4e3b91d84edae8163e4cc6ccf1f27be2ef1d78bb86f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T01:51:49.367 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 01:51:49 smithi134 podman[63042]: 2024-04-25 01:51:49.369169601 +0000 UTC m=+0.385732497 container init 7dcfa65053c5e5b3bb73a4e3b91d84edae8163e4cc6ccf1f27be2ef1d78bb86f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T01:51:49.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:49 smithi134 ceph-mon[33098]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:49.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:49 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1211875616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:49.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:49 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.134:6832/400138993,v1:172.21.15.134:6833/400138993] up:boot 2024-04-25T01:51:49.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:49 smithi134 ceph-mon[33098]: fsmap 1 up:standby 2024-04-25T01:51:49.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:51:49.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:49 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:49.688 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 01:51:49 smithi134 podman[63042]: 2024-04-25 01:51:49.372042884 +0000 UTC m=+0.388605794 container start 7dcfa65053c5e5b3bb73a4e3b91d84edae8163e4cc6ccf1f27be2ef1d78bb86f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:51:49.688 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 01:51:49 smithi134 ceph-mds[63061]: -- 172.21.15.134:0/3070714069 <== mon.1 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eb40871a40 con 0x55eb40844800 2024-04-25T01:51:49.688 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 01:51:49 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-l[63057]: starting mds.l at 2024-04-25T01:51:49.688 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 01:51:49 smithi134 bash[63042]: 7dcfa65053c5e5b3bb73a4e3b91d84edae8163e4cc6ccf1f27be2ef1d78bb86f 2024-04-25T01:51:49.688 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 01:51:49 smithi134 systemd[1]: Started Ceph mds.l for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:51:49.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:49 smithi031 ceph-mon[29204]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:49.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:49 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1211875616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:49.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:49 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.134:6832/400138993,v1:172.21.15.134:6833/400138993] up:boot 2024-04-25T01:51:49.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:49 smithi031 ceph-mon[29204]: fsmap 1 up:standby 2024-04-25T01:51:49.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:51:49.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:49 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:49.802 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:49 smithi040 ceph-mon[32418]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:49.802 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:49 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1211875616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:49.802 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:49 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.134:6832/400138993,v1:172.21.15.134:6833/400138993] up:boot 2024-04-25T01:51:49.802 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:49 smithi040 ceph-mon[32418]: fsmap 1 up:standby 2024-04-25T01:51:49.802 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:51:49.802 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:49 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:50.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:51:50.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.694+0000 7fc7af161640 1 -- 172.21.15.31:0/1229249085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7a8102980 msgr2=0x7fc7a8102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.694+0000 7fc7af161640 1 --2- 172.21.15.31:0/1229249085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7a8102980 0x7fc7a8102e00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009900 tx=0x7fc79c02a410 comp rx=0 tx=0).stop 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.694+0000 7fc7af161640 1 -- 172.21.15.31:0/1229249085 shutdown_connections 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.694+0000 7fc7af161640 1 --2- 172.21.15.31:0/1229249085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7a8103340 0x7fc7a8109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.694+0000 7fc7af161640 1 --2- 172.21.15.31:0/1229249085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7a8102980 0x7fc7a8102e00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.694+0000 7fc7af161640 1 --2- 172.21.15.31:0/1229249085 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7a8101790 0x7fc7a8101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.694+0000 7fc7af161640 1 -- 172.21.15.31:0/1229249085 >> 172.21.15.31:0/1229249085 conn(0x7fc7a80fcd20 msgr2=0x7fc7a80ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.694+0000 7fc7af161640 1 -- 172.21.15.31:0/1229249085 shutdown_connections 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.694+0000 7fc7af161640 1 -- 172.21.15.31:0/1229249085 wait complete. 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7af161640 1 Processor -- start 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7af161640 1 -- start start 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7af161640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7a8101790 0x7fc7a80754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7af161640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7a8102980 0x7fc7a8073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7af161640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7a8103340 0x7fc7a8074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7af161640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a810d2a0 con 0x7fc7a8102980 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7af161640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a810d120 con 0x7fc7a8103340 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7af161640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a810d420 con 0x7fc7a8101790 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7a7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7a8102980 0x7fc7a8073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7a7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7a8102980 0x7fc7a8073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53594/0 (socket says 172.21.15.31:53594) 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7a7fff640 1 -- 172.21.15.31:0/801701961 learned_addr learned my addr 172.21.15.31:0/801701961 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7ad6fe640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7a8103340 0x7fc7a8074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7acefd640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7a8101790 0x7fc7a80754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7a7fff640 1 -- 172.21.15.31:0/801701961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7a8101790 msgr2=0x7fc7a80754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7a7fff640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7a8101790 0x7fc7a80754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7a7fff640 1 -- 172.21.15.31:0/801701961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7a8103340 msgr2=0x7fc7a8074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7a7fff640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7a8103340 0x7fc7a8074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.695+0000 7fc7a7fff640 1 -- 172.21.15.31:0/801701961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a8074900 con 0x7fc7a8102980 2024-04-25T01:51:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.696+0000 7fc7a7fff640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7a8102980 0x7fc7a8073b30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc79c004770 tx=0x7fc79c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.696+0000 7fc7a5ffb640 1 -- 172.21.15.31:0/801701961 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c03f030 con 0x7fc7a8102980 2024-04-25T01:51:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.696+0000 7fc7a5ffb640 1 -- 172.21.15.31:0/801701961 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc79c003cb0 con 0x7fc7a8102980 2024-04-25T01:51:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.696+0000 7fc7af161640 1 -- 172.21.15.31:0/801701961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7a81a1a30 con 0x7fc7a8102980 2024-04-25T01:51:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.696+0000 7fc7a5ffb640 1 -- 172.21.15.31:0/801701961 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c035070 con 0x7fc7a8102980 2024-04-25T01:51:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.696+0000 7fc7af161640 1 -- 172.21.15.31:0/801701961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7a81a1f80 con 0x7fc7a8102980 2024-04-25T01:51:50.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.697+0000 7fc7af161640 1 -- 172.21.15.31:0/801701961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7a8108b00 con 0x7fc7a8102980 2024-04-25T01:51:50.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.698+0000 7fc7a5ffb640 1 -- 172.21.15.31:0/801701961 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc79c042030 con 0x7fc7a8102980 2024-04-25T01:51:50.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.699+0000 7fc7a5ffb640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc7900a2b90 0x7fc7900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:50.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.699+0000 7fc7acefd640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc7900a2b90 0x7fc7900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:50.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.699+0000 7fc7a5ffb640 1 -- 172.21.15.31:0/801701961 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc79c0ef5d0 con 0x7fc7a8102980 2024-04-25T01:51:50.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.699+0000 7fc7acefd640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc7900a2b90 0x7fc7900a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc798005fd0 tx=0x7fc798005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:50.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.703+0000 7fc7a5ffb640 1 -- 172.21.15.31:0/801701961 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc79c0bc930 con 0x7fc7a8102980 2024-04-25T01:51:50.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:50 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:50.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:50 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:50.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:50 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:50.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:50 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:50.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:50 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:50.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:50 smithi040 ceph-mon[32418]: Deploying daemon mds.k on smithi040 2024-04-25T01:51:50.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.925+0000 7fc7af161640 1 -- 172.21.15.31:0/801701961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7a8062340 con 0x7fc7a8102980 2024-04-25T01:51:50.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.926+0000 7fc7a5ffb640 1 -- 172.21.15.31:0/801701961 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7fc79c0bc230 con 0x7fc7a8102980 2024-04-25T01:51:50.926 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:51:50.926 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:51:50.926 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 2 2024-04-25T01:51:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 -- 172.21.15.31:0/801701961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc7900a2b90 msgr2=0x7fc7900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc7900a2b90 0x7fc7900a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc798005fd0 tx=0x7fc798005850 comp rx=0 tx=0).stop 2024-04-25T01:51:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 -- 172.21.15.31:0/801701961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7a8102980 msgr2=0x7fc7a8073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7a8102980 0x7fc7a8073b30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc79c004770 tx=0x7fc79c0042f0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 -- 172.21.15.31:0/801701961 shutdown_connections 2024-04-25T01:51:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc7900a2b90 0x7fc7900a5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7a8103340 0x7fc7a8074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7a8102980 0x7fc7a8073b30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 --2- 172.21.15.31:0/801701961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7a8101790 0x7fc7a80754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 -- 172.21.15.31:0/801701961 >> 172.21.15.31:0/801701961 conn(0x7fc7a80fcd20 msgr2=0x7fc7a8105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:50.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 -- 172.21.15.31:0/801701961 shutdown_connections 2024-04-25T01:51:50.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:50.928+0000 7fc7af161640 1 -- 172.21.15.31:0/801701961 wait complete. 2024-04-25T01:51:50.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:50 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:50.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:50 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:50.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:50 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:50.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:50 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:50.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:50 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:50.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:50 smithi134 ceph-mon[33098]: Deploying daemon mds.k on smithi040 2024-04-25T01:51:50.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:50 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:50.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:50 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:50.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:50 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:50.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:50 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:50.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:50 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:50.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:50 smithi031 ceph-mon[29204]: Deploying daemon mds.k on smithi040 2024-04-25T01:51:51.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:51 smithi040 ceph-mon[32418]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:51.820 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:51 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/801701961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:51.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:51 smithi134 ceph-mon[33098]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:51.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:51 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/801701961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:51.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:51 smithi031 ceph-mon[29204]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:51.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:51 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/801701961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:52.158 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 01:51:52 smithi040 podman[61624]: 2024-04-25 01:51:52.047892913 +0000 UTC m=+0.120923795 container create 7a908d565424fa5a70c93e51bcdca2ef19dcd78b624ba1eb214436f35e1bed18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T01:51:52.658 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 01:51:52 smithi040 podman[61624]: 2024-04-25 01:51:52.295770322 +0000 UTC m=+0.368801158 container init 7a908d565424fa5a70c93e51bcdca2ef19dcd78b624ba1eb214436f35e1bed18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T01:51:52.658 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 01:51:52 smithi040 podman[61624]: 2024-04-25 01:51:52.298486089 +0000 UTC m=+0.371516925 container start 7a908d565424fa5a70c93e51bcdca2ef19dcd78b624ba1eb214436f35e1bed18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0) 2024-04-25T01:51:52.658 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 01:51:52 smithi040 ceph-mds[61643]: -- 172.21.15.40:0/632869386 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e7a45cba40 con 0x55e7a459ec00 2024-04-25T01:51:52.658 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 01:51:52 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-k[61639]: starting mds.k at 2024-04-25T01:51:52.658 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 01:51:52 smithi040 bash[61624]: 7a908d565424fa5a70c93e51bcdca2ef19dcd78b624ba1eb214436f35e1bed18 2024-04-25T01:51:52.658 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 01:51:52 smithi040 systemd[1]: Started Ceph mds.k for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:51:52.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:51:53.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.053+0000 7f6ea7163640 1 -- 172.21.15.31:0/1785798657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea01029a0 msgr2=0x7f6ea0102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:53.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.053+0000 7f6ea7163640 1 --2- 172.21.15.31:0/1785798657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea01029a0 0x7f6ea0102e20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c009900 tx=0x7f6e8c02a410 comp rx=0 tx=0).stop 2024-04-25T01:51:53.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.053+0000 7f6ea7163640 1 -- 172.21.15.31:0/1785798657 shutdown_connections 2024-04-25T01:51:53.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.053+0000 7f6ea7163640 1 --2- 172.21.15.31:0/1785798657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea0103360 0x7f6ea0109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.053+0000 7f6ea7163640 1 --2- 172.21.15.31:0/1785798657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea01029a0 0x7f6ea0102e20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.053+0000 7f6ea7163640 1 --2- 172.21.15.31:0/1785798657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea01017b0 0x7f6ea0101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.053+0000 7f6ea7163640 1 -- 172.21.15.31:0/1785798657 >> 172.21.15.31:0/1785798657 conn(0x7f6ea00fcd40 msgr2=0x7f6ea00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.053+0000 7f6ea7163640 1 -- 172.21.15.31:0/1785798657 shutdown_connections 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.053+0000 7f6ea7163640 1 -- 172.21.15.31:0/1785798657 wait complete. 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6ea7163640 1 Processor -- start 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6ea7163640 1 -- start start 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6ea7163640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea01017b0 0x7f6ea01979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6ea7163640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea01029a0 0x7f6ea0197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6ea7163640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea0103360 0x7f6ea019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6ea7163640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea010d2e0 con 0x7f6ea01017b0 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6ea7163640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea010d160 con 0x7f6ea01029a0 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6ea7163640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea010d460 con 0x7f6ea0103360 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6ea4eff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea01017b0 0x7f6ea01979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6ea4eff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea01017b0 0x7f6ea01979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48568/0 (socket says 172.21.15.31:48568) 2024-04-25T01:51:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6ea4eff640 1 -- 172.21.15.31:0/186255525 learned_addr learned my addr 172.21.15.31:0/186255525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.054+0000 7f6e9ffff640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea01029a0 0x7f6ea0197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6ea5700640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea0103360 0x7f6ea019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6ea4eff640 1 -- 172.21.15.31:0/186255525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea0103360 msgr2=0x7f6ea019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6ea4eff640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea0103360 0x7f6ea019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6ea4eff640 1 -- 172.21.15.31:0/186255525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea01029a0 msgr2=0x7f6ea0197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6ea4eff640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea01029a0 0x7f6ea0197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6ea4eff640 1 -- 172.21.15.31:0/186255525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea019f600 con 0x7f6ea01017b0 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6ea4eff640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea01017b0 0x7f6ea01979e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6e9400e9c0 tx=0x7f6e9400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6e9dffb640 1 -- 172.21.15.31:0/186255525 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9400cd80 con 0x7f6ea01017b0 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6e9dffb640 1 -- 172.21.15.31:0/186255525 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e940035f0 con 0x7f6ea01017b0 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6e9dffb640 1 -- 172.21.15.31:0/186255525 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94010690 con 0x7f6ea01017b0 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6ea7163640 1 -- 172.21.15.31:0/186255525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea019f8f0 con 0x7f6ea01017b0 2024-04-25T01:51:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.055+0000 7f6ea7163640 1 -- 172.21.15.31:0/186255525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea019fe60 con 0x7f6ea01017b0 2024-04-25T01:51:53.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.056+0000 7f6ea7163640 1 -- 172.21.15.31:0/186255525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ea0108ba0 con 0x7f6ea01017b0 2024-04-25T01:51:53.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.057+0000 7f6e9dffb640 1 -- 172.21.15.31:0/186255525 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6e940037a0 con 0x7f6ea01017b0 2024-04-25T01:51:53.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.058+0000 7f6e9dffb640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6e900a2c90 0x7f6e900a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:53.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.058+0000 7f6e9ffff640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6e900a2c90 0x7f6e900a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:53.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.058+0000 7f6e9dffb640 1 -- 172.21.15.31:0/186255525 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f6e940cf020 con 0x7f6ea01017b0 2024-04-25T01:51:53.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.059+0000 7f6e9ffff640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6e900a2c90 0x7f6e900a5150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c006010 tx=0x7f6e8c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.066+0000 7f6e9dffb640 1 -- 172.21.15.31:0/186255525 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6e9409c380 con 0x7f6ea01017b0 2024-04-25T01:51:53.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.291+0000 7f6ea7163640 1 -- 172.21.15.31:0/186255525 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ea002d170 con 0x7f6ea01017b0 2024-04-25T01:51:53.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.291+0000 7f6e9dffb640 1 -- 172.21.15.31:0/186255525 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f6e9409bc80 con 0x7f6ea01017b0 2024-04-25T01:51:53.291 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:51:53.291 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:51:53.291 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 3 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 -- 172.21.15.31:0/186255525 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6e900a2c90 msgr2=0x7f6e900a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6e900a2c90 0x7f6e900a5150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c006010 tx=0x7f6e8c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 -- 172.21.15.31:0/186255525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea01017b0 msgr2=0x7f6ea01979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea01017b0 0x7f6ea01979e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6e9400e9c0 tx=0x7f6e9400ee80 comp rx=0 tx=0).stop 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 -- 172.21.15.31:0/186255525 shutdown_connections 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6e900a2c90 0x7f6e900a5150 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ea0103360 0x7f6ea019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea01029a0 0x7f6ea0197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 --2- 172.21.15.31:0/186255525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ea01017b0 0x7f6ea01979e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 -- 172.21.15.31:0/186255525 >> 172.21.15.31:0/186255525 conn(0x7f6ea00fcd40 msgr2=0x7f6ea0105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 -- 172.21.15.31:0/186255525 shutdown_connections 2024-04-25T01:51:53.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:53.293+0000 7f6ea7163640 1 -- 172.21.15.31:0/186255525 wait complete. 2024-04-25T01:51:53.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:53.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:53.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:53.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:53.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:53.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:53.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:53.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: Deploying daemon mds.j on smithi031 2024-04-25T01:51:53.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.40:6832/3875547004,v1:172.21.15.40:6833/3875547004] up:boot 2024-04-25T01:51:53.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: fsmap 2 up:standby 2024-04-25T01:51:53.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:51:53.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:53 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/186255525' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:53.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:53.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:53.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:53.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:53.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:53.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:53.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:53.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: Deploying daemon mds.j on smithi031 2024-04-25T01:51:53.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.40:6832/3875547004,v1:172.21.15.40:6833/3875547004] up:boot 2024-04-25T01:51:53.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: fsmap 2 up:standby 2024-04-25T01:51:53.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:51:53.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:53 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/186255525' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:53.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:53.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:53.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:53.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:53.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:53.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:53.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:53.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: Deploying daemon mds.j on smithi031 2024-04-25T01:51:53.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.40:6832/3875547004,v1:172.21.15.40:6833/3875547004] up:boot 2024-04-25T01:51:53.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: fsmap 2 up:standby 2024-04-25T01:51:53.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:51:53.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:53 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/186255525' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:54.954 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 01:51:54 smithi031 systemd[1]: Starting Ceph mds.j for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:51:55.204 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 01:51:54 smithi031 podman[69067]: 2024-04-25 01:51:54.857597028 +0000 UTC m=+0.018250905 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:51:55.205 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 01:51:54 smithi031 podman[69067]: 2024-04-25 01:51:54.95886904 +0000 UTC m=+0.119522951 container create 14c1d80366404aefe0ebd26f3b210bc51733c571408b934a9bb597b1c31e2420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9) 2024-04-25T01:51:55.205 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 01:51:55 smithi031 podman[69067]: 2024-04-25 01:51:55.203192433 +0000 UTC m=+0.363846298 container init 14c1d80366404aefe0ebd26f3b210bc51733c571408b934a9bb597b1c31e2420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0) 2024-04-25T01:51:55.205 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 01:51:55 smithi031 podman[69067]: 2024-04-25 01:51:55.205940292 +0000 UTC m=+0.366594158 container start 14c1d80366404aefe0ebd26f3b210bc51733c571408b934a9bb597b1c31e2420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-j, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T01:51:55.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.413+0000 7f1ba989a640 1 -- 172.21.15.31:0/842152671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba41006a0 msgr2=0x7f1ba4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.413+0000 7f1ba989a640 1 --2- 172.21.15.31:0/842152671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba41006a0 0x7f1ba4100ac0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009900 tx=0x7f1b9402a410 comp rx=0 tx=0).stop 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.413+0000 7f1ba989a640 1 -- 172.21.15.31:0/842152671 shutdown_connections 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.413+0000 7f1ba989a640 1 --2- 172.21.15.31:0/842152671 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba4102250 0x7f1ba4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.413+0000 7f1ba989a640 1 --2- 172.21.15.31:0/842152671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba4101890 0x7f1ba4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.413+0000 7f1ba989a640 1 --2- 172.21.15.31:0/842152671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba41006a0 0x7f1ba4100ac0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.413+0000 7f1ba989a640 1 -- 172.21.15.31:0/842152671 >> 172.21.15.31:0/842152671 conn(0x7f1ba40fbc30 msgr2=0x7f1ba40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1ba989a640 1 -- 172.21.15.31:0/842152671 shutdown_connections 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1ba989a640 1 -- 172.21.15.31:0/842152671 wait complete. 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1ba989a640 1 Processor -- start 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1ba989a640 1 -- start start 2024-04-25T01:51:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1ba989a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba41006a0 0x7f1ba41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1ba989a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba4101890 0x7f1ba4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1ba989a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba4102250 0x7f1ba419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1ba989a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba410d2f0 con 0x7f1ba4101890 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1ba989a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba410d170 con 0x7f1ba41006a0 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1ba989a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba410d470 con 0x7f1ba4102250 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1b9bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba4101890 0x7f1ba4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1ba2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba41006a0 0x7f1ba41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1b9bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba4101890 0x7f1ba4197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48592/0 (socket says 172.21.15.31:48592) 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.414+0000 7f1b9bfff640 1 -- 172.21.15.31:0/1663721903 learned_addr learned my addr 172.21.15.31:0/1663721903 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.415+0000 7f1ba37fe640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba4102250 0x7f1ba419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.415+0000 7f1b9bfff640 1 -- 172.21.15.31:0/1663721903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba4102250 msgr2=0x7f1ba419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.415+0000 7f1b9bfff640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba4102250 0x7f1ba419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.415+0000 7f1b9bfff640 1 -- 172.21.15.31:0/1663721903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba41006a0 msgr2=0x7f1ba41979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.415+0000 7f1b9bfff640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba41006a0 0x7f1ba41979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.415+0000 7f1b9bfff640 1 -- 172.21.15.31:0/1663721903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba419f600 con 0x7f1ba4101890 2024-04-25T01:51:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.415+0000 7f1ba37fe640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba4102250 0x7f1ba419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:51:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.415+0000 7f1b9bfff640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba4101890 0x7f1ba4197f20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00e9f0 tx=0x7f1b8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.416+0000 7f1ba8898640 1 -- 172.21.15.31:0/1663721903 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c00cd80 con 0x7f1ba4101890 2024-04-25T01:51:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.416+0000 7f1ba8898640 1 -- 172.21.15.31:0/1663721903 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b8c0108e0 con 0x7f1ba4101890 2024-04-25T01:51:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.416+0000 7f1ba989a640 1 -- 172.21.15.31:0/1663721903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba419f8f0 con 0x7f1ba4101890 2024-04-25T01:51:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.416+0000 7f1ba8898640 1 -- 172.21.15.31:0/1663721903 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c018920 con 0x7f1ba4101890 2024-04-25T01:51:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.416+0000 7f1ba989a640 1 -- 172.21.15.31:0/1663721903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba419fe90 con 0x7f1ba4101890 2024-04-25T01:51:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.417+0000 7f1ba989a640 1 -- 172.21.15.31:0/1663721903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ba4107ab0 con 0x7f1ba4101890 2024-04-25T01:51:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.419+0000 7f1ba8898640 1 -- 172.21.15.31:0/1663721903 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1b8c010a90 con 0x7f1ba4101890 2024-04-25T01:51:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.420+0000 7f1ba8898640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1b900a2c10 0x7f1b900a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.420+0000 7f1ba2ffd640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1b900a2c10 0x7f1b900a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.420+0000 7f1ba8898640 1 -- 172.21.15.31:0/1663721903 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f1b8c0cf3b0 con 0x7f1ba4101890 2024-04-25T01:51:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.420+0000 7f1ba2ffd640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1b900a2c10 0x7f1b900a50d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1b94004770 tx=0x7f1b94000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:55.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.424+0000 7f1ba8898640 1 -- 172.21.15.31:0/1663721903 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1b8c09c710 con 0x7f1ba4101890 2024-04-25T01:51:55.486 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 01:51:55 smithi031 ceph-mds[69087]: -- 172.21.15.31:0/3103767164 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ffc90a7a40 con 0x55ffc907ac00 2024-04-25T01:51:55.486 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 01:51:55 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-j[69083]: starting mds.j at 2024-04-25T01:51:55.486 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 01:51:55 smithi031 bash[69067]: 14c1d80366404aefe0ebd26f3b210bc51733c571408b934a9bb597b1c31e2420 2024-04-25T01:51:55.486 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 01:51:55 smithi031 systemd[1]: Started Ceph mds.j for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:51:55.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:55 smithi031 ceph-mon[29204]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:55.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:55.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:55.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:55.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:55.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:55.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:55 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:55.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.652+0000 7f1ba989a640 1 -- 172.21.15.31:0/1663721903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ba402d170 con 0x7f1ba4101890 2024-04-25T01:51:55.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.653+0000 7f1ba8898640 1 -- 172.21.15.31:0/1663721903 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f1b8c09c010 con 0x7f1ba4101890 2024-04-25T01:51:55.653 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:51:55.653 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:51:55.653 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 4 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 -- 172.21.15.31:0/1663721903 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1b900a2c10 msgr2=0x7f1b900a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1b900a2c10 0x7f1b900a50d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1b94004770 tx=0x7f1b94000bc0 comp rx=0 tx=0).stop 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 -- 172.21.15.31:0/1663721903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba4101890 msgr2=0x7f1ba4197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba4101890 0x7f1ba4197f20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00e9f0 tx=0x7f1b8c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 -- 172.21.15.31:0/1663721903 shutdown_connections 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1b900a2c10 0x7f1b900a50d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba4102250 0x7f1ba419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba4101890 0x7f1ba4197f20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 --2- 172.21.15.31:0/1663721903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba41006a0 0x7f1ba41979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 -- 172.21.15.31:0/1663721903 >> 172.21.15.31:0/1663721903 conn(0x7f1ba40fbc30 msgr2=0x7f1ba4104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 -- 172.21.15.31:0/1663721903 shutdown_connections 2024-04-25T01:51:55.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:55.655+0000 7f1ba989a640 1 -- 172.21.15.31:0/1663721903 wait complete. 2024-04-25T01:51:55.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:55 smithi040 ceph-mon[32418]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:55.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:55.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:55.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:55.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:55.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:55.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:55 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:55.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:55 smithi134 ceph-mon[33098]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:55.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:55.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:55.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:55.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:55.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:55.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:55 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:56.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:56 smithi031 ceph-mon[29204]: Deploying daemon mds.d on smithi031 2024-04-25T01:51:56.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:56 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.31:6834/3566811410,v1:172.21.15.31:6835/3566811410] up:boot 2024-04-25T01:51:56.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:56 smithi031 ceph-mon[29204]: fsmap 3 up:standby 2024-04-25T01:51:56.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:56 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:51:56.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:56 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1663721903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:56.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:56 smithi040 ceph-mon[32418]: Deploying daemon mds.d on smithi031 2024-04-25T01:51:56.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:56 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.31:6834/3566811410,v1:172.21.15.31:6835/3566811410] up:boot 2024-04-25T01:51:56.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:56 smithi040 ceph-mon[32418]: fsmap 3 up:standby 2024-04-25T01:51:56.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:56 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:51:56.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:56 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1663721903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:56.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:56 smithi134 ceph-mon[33098]: Deploying daemon mds.d on smithi031 2024-04-25T01:51:56.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:56 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.31:6834/3566811410,v1:172.21.15.31:6835/3566811410] up:boot 2024-04-25T01:51:56.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:56 smithi134 ceph-mon[33098]: fsmap 3 up:standby 2024-04-25T01:51:56.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:56 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:51:56.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:56 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1663721903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:57.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:51:57.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:57 smithi134 ceph-mon[33098]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:57.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:57 smithi031 ceph-mon[29204]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.794+0000 7f01ad509640 1 -- 172.21.15.31:0/2289688527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01a8100680 msgr2=0x7f01a8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.794+0000 7f01ad509640 1 --2- 172.21.15.31:0/2289688527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01a8100680 0x7f01a8100aa0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0198009900 tx=0x7f019802a410 comp rx=0 tx=0).stop 2024-04-25T01:51:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.794+0000 7f01ad509640 1 -- 172.21.15.31:0/2289688527 shutdown_connections 2024-04-25T01:51:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.794+0000 7f01ad509640 1 --2- 172.21.15.31:0/2289688527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a8102230 0x7f01a8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.794+0000 7f01ad509640 1 --2- 172.21.15.31:0/2289688527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01a8101870 0x7f01a8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.794+0000 7f01ad509640 1 --2- 172.21.15.31:0/2289688527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01a8100680 0x7f01a8100aa0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.794+0000 7f01ad509640 1 -- 172.21.15.31:0/2289688527 >> 172.21.15.31:0/2289688527 conn(0x7f01a80fbbf0 msgr2=0x7f01a80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:57.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.794+0000 7f01ad509640 1 -- 172.21.15.31:0/2289688527 shutdown_connections 2024-04-25T01:51:57.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.794+0000 7f01ad509640 1 -- 172.21.15.31:0/2289688527 wait complete. 2024-04-25T01:51:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.795+0000 7f01ad509640 1 Processor -- start 2024-04-25T01:51:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.795+0000 7f01ad509640 1 -- start start 2024-04-25T01:51:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.795+0000 7f01ad509640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a8100680 0x7f01a8069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.795+0000 7f01ad509640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01a8101870 0x7f01a819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:57.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.795+0000 7f01ad509640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01a8102230 0x7f01a819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01ad509640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f01a810c1b0 con 0x7f01a8101870 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01ad509640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01a810c030 con 0x7f01a8100680 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01ad509640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f01a810c330 con 0x7f01a8102230 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01a8101870 0x7f01a819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01a8101870 0x7f01a819c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48622/0 (socket says 172.21.15.31:48622) 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a67fc640 1 -- 172.21.15.31:0/437419715 learned_addr learned my addr 172.21.15.31:0/437419715 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a6ffd640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a8100680 0x7f01a8069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a77fe640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01a8102230 0x7f01a819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a67fc640 1 -- 172.21.15.31:0/437419715 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01a8102230 msgr2=0x7f01a819eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a67fc640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01a8102230 0x7f01a819eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a67fc640 1 -- 172.21.15.31:0/437419715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a8100680 msgr2=0x7f01a8069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a67fc640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a8100680 0x7f01a8069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:57.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a67fc640 1 -- 172.21.15.31:0/437419715 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a819f660 con 0x7f01a8101870 2024-04-25T01:51:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a77fe640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01a8102230 0x7f01a819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:51:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01a67fc640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01a8101870 0x7f01a819c760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f019c00e9f0 tx=0x7f019c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f018ffff640 1 -- 172.21.15.31:0/437419715 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c00cd80 con 0x7f01a8101870 2024-04-25T01:51:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f018ffff640 1 -- 172.21.15.31:0/437419715 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f019c0054d0 con 0x7f01a8101870 2024-04-25T01:51:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01ad509640 1 -- 172.21.15.31:0/437419715 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a819f950 con 0x7f01a8101870 2024-04-25T01:51:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f018ffff640 1 -- 172.21.15.31:0/437419715 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c010490 con 0x7f01a8101870 2024-04-25T01:51:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.796+0000 7f01ad509640 1 -- 172.21.15.31:0/437419715 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a819fe50 con 0x7f01a8101870 2024-04-25T01:51:57.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.798+0000 7f01ad509640 1 -- 172.21.15.31:0/437419715 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a8107a60 con 0x7f01a8101870 2024-04-25T01:51:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.799+0000 7f018ffff640 1 -- 172.21.15.31:0/437419715 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f019c005020 con 0x7f01a8101870 2024-04-25T01:51:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.800+0000 7f018ffff640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f01900a2b90 0x7f01900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.800+0000 7f01a6ffd640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f01900a2b90 0x7f01900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.800+0000 7f018ffff640 1 -- 172.21.15.31:0/437419715 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f019c0cf080 con 0x7f01a8101870 2024-04-25T01:51:57.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.800+0000 7f01a6ffd640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f01900a2b90 0x7f01900a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0198000c00 tx=0x7f01980314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:57.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:57.805+0000 7f018ffff640 1 -- 172.21.15.31:0/437419715 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f019c09c480 con 0x7f01a8101870 2024-04-25T01:51:57.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:57 smithi040 ceph-mon[32418]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:58.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.031+0000 7f01ad509640 1 -- 172.21.15.31:0/437419715 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01a802d170 con 0x7f01a8101870 2024-04-25T01:51:58.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.032+0000 7f018ffff640 1 -- 172.21.15.31:0/437419715 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f019c09bd80 con 0x7f01a8101870 2024-04-25T01:51:58.035 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:51:58.035 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 4 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 -- 172.21.15.31:0/437419715 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f01900a2b90 msgr2=0x7f01900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f01900a2b90 0x7f01900a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0198000c00 tx=0x7f01980314a0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 -- 172.21.15.31:0/437419715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01a8101870 msgr2=0x7f01a819c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01a8101870 0x7f01a819c760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f019c00e9f0 tx=0x7f019c00edf0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 -- 172.21.15.31:0/437419715 shutdown_connections 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f01900a2b90 0x7f01900a5050 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01a8102230 0x7f01a819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01a8101870 0x7f01a819c760 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 --2- 172.21.15.31:0/437419715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a8100680 0x7f01a8069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 -- 172.21.15.31:0/437419715 >> 172.21.15.31:0/437419715 conn(0x7f01a80fbbf0 msgr2=0x7f01a8104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 -- 172.21.15.31:0/437419715 shutdown_connections 2024-04-25T01:51:58.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:51:58.035+0000 7f01ad509640 1 -- 172.21.15.31:0/437419715 wait complete. 2024-04-25T01:51:58.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:58 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/437419715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:58.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:58 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/437419715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:58.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:58 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/437419715' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:51:59.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:59 smithi031 ceph-mon[29204]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:59.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:59 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.31:6836/2948364489,v1:172.21.15.31:6837/2948364489] up:boot 2024-04-25T01:51:59.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:59 smithi031 ceph-mon[29204]: fsmap 4 up:standby 2024-04-25T01:51:59.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:59 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:51:59.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:59 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:59.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:59 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:59.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:59 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:59.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:59 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:59.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:59 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:59.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:59 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:59.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:51:59 smithi031 ceph-mon[29204]: Deploying daemon mds.e on smithi040 2024-04-25T01:51:59.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:59 smithi040 ceph-mon[32418]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:59.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:59 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.31:6836/2948364489,v1:172.21.15.31:6837/2948364489] up:boot 2024-04-25T01:51:59.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:59 smithi040 ceph-mon[32418]: fsmap 4 up:standby 2024-04-25T01:51:59.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:59 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:51:59.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:59 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:59.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:59 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:59.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:59 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:59.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:59 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:59.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:59 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:59.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:59 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:59.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:51:59 smithi040 ceph-mon[32418]: Deploying daemon mds.e on smithi040 2024-04-25T01:51:59.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:59 smithi134 ceph-mon[33098]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:59.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:59 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.31:6836/2948364489,v1:172.21.15.31:6837/2948364489] up:boot 2024-04-25T01:51:59.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:59 smithi134 ceph-mon[33098]: fsmap 4 up:standby 2024-04-25T01:51:59.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:59 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:51:59.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:59 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:59.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:59 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:59.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:59 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:51:59.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:59 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:59.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:59 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:59.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:59 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:59.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:51:59 smithi134 ceph-mon[33098]: Deploying daemon mds.e on smithi040 2024-04-25T01:52:00.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.164+0000 7f528e05d640 1 -- 172.21.15.31:0/2172698705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5288102980 msgr2=0x7f5288102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.164+0000 7f528e05d640 1 --2- 172.21.15.31:0/2172698705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5288102980 0x7f5288102e00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f527c009900 tx=0x7f527c02a410 comp rx=0 tx=0).stop 2024-04-25T01:52:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.164+0000 7f528e05d640 1 -- 172.21.15.31:0/2172698705 shutdown_connections 2024-04-25T01:52:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.164+0000 7f528e05d640 1 --2- 172.21.15.31:0/2172698705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5288103340 0x7f5288109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.164+0000 7f528e05d640 1 --2- 172.21.15.31:0/2172698705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5288102980 0x7f5288102e00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.164+0000 7f528e05d640 1 --2- 172.21.15.31:0/2172698705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5288101790 0x7f5288101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:00.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.164+0000 7f528e05d640 1 -- 172.21.15.31:0/2172698705 >> 172.21.15.31:0/2172698705 conn(0x7f52880fcd20 msgr2=0x7f52880ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.164+0000 7f528e05d640 1 -- 172.21.15.31:0/2172698705 shutdown_connections 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.164+0000 7f528e05d640 1 -- 172.21.15.31:0/2172698705 wait complete. 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.164+0000 7f528e05d640 1 Processor -- start 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.164+0000 7f528e05d640 1 -- start start 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f528e05d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5288101790 0x7f52880754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f528e05d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5288102980 0x7f5288073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f528e05d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5288103340 0x7f5288074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f528e05d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f528810d2c0 con 0x7f5288102980 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f528e05d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f528810d140 con 0x7f5288103340 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f528e05d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f528810d440 con 0x7f5288101790 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5286ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5288102980 0x7f5288073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5286ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5288102980 0x7f5288073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48662/0 (socket says 172.21.15.31:48662) 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5286ffd640 1 -- 172.21.15.31:0/996808757 learned_addr learned my addr 172.21.15.31:0/996808757 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:00.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f52877fe640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5288101790 0x7f52880754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5287fff640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5288103340 0x7f5288074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5286ffd640 1 -- 172.21.15.31:0/996808757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5288101790 msgr2=0x7f52880754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5286ffd640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5288101790 0x7f52880754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5286ffd640 1 -- 172.21.15.31:0/996808757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5288103340 msgr2=0x7f5288074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5286ffd640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5288103340 0x7f5288074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5286ffd640 1 -- 172.21.15.31:0/996808757 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5288074900 con 0x7f5288102980 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5286ffd640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5288102980 0x7f5288073b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f527c005940 tx=0x7f527c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5284ff9640 1 -- 172.21.15.31:0/996808757 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c03f030 con 0x7f5288102980 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f528e05d640 1 -- 172.21.15.31:0/996808757 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52881a1a80 con 0x7f5288102980 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.165+0000 7f5284ff9640 1 -- 172.21.15.31:0/996808757 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f527c0318f0 con 0x7f5288102980 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.166+0000 7f5284ff9640 1 -- 172.21.15.31:0/996808757 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c035070 con 0x7f5288102980 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.166+0000 7f528e05d640 1 -- 172.21.15.31:0/996808757 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52881a1fd0 con 0x7f5288102980 2024-04-25T01:52:00.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.167+0000 7f528e05d640 1 -- 172.21.15.31:0/996808757 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5288108b00 con 0x7f5288102980 2024-04-25T01:52:00.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.168+0000 7f5284ff9640 1 -- 172.21.15.31:0/996808757 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f527c042030 con 0x7f5288102980 2024-04-25T01:52:00.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.168+0000 7f5284ff9640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f52700a2b90 0x7f52700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:00.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.168+0000 7f52877fe640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f52700a2b90 0x7f52700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:00.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.168+0000 7f5284ff9640 1 -- 172.21.15.31:0/996808757 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f527c0ef040 con 0x7f5288102980 2024-04-25T01:52:00.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.169+0000 7f52877fe640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f52700a2b90 0x7f52700a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f52881027e0 tx=0x7f5278005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:00.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.173+0000 7f5284ff9640 1 -- 172.21.15.31:0/996808757 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f527c0bbdf0 con 0x7f5288102980 2024-04-25T01:52:00.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.397+0000 7f528e05d640 1 -- 172.21.15.31:0/996808757 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f528802d350 con 0x7f5288102980 2024-04-25T01:52:00.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.398+0000 7f5284ff9640 1 -- 172.21.15.31:0/996808757 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f527c0c00f0 con 0x7f5288102980 2024-04-25T01:52:00.397 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:00.398 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:52:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 5 2024-04-25T01:52:00.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.399+0000 7f528e05d640 1 -- 172.21.15.31:0/996808757 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f52700a2b90 msgr2=0x7f52700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:00.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.399+0000 7f528e05d640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f52700a2b90 0x7f52700a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f52881027e0 tx=0x7f5278005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.399+0000 7f528e05d640 1 -- 172.21.15.31:0/996808757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5288102980 msgr2=0x7f5288073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.399+0000 7f528e05d640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5288102980 0x7f5288073b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f527c005940 tx=0x7f527c0042f0 comp rx=0 tx=0).stop 2024-04-25T01:52:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.399+0000 7f528e05d640 1 -- 172.21.15.31:0/996808757 shutdown_connections 2024-04-25T01:52:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.400+0000 7f528e05d640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f52700a2b90 0x7f52700a5050 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.400+0000 7f528e05d640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5288103340 0x7f5288074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.400+0000 7f528e05d640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5288102980 0x7f5288073b30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.400+0000 7f528e05d640 1 --2- 172.21.15.31:0/996808757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5288101790 0x7f52880754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.400+0000 7f528e05d640 1 -- 172.21.15.31:0/996808757 >> 172.21.15.31:0/996808757 conn(0x7f52880fcd20 msgr2=0x7f5288105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.400+0000 7f528e05d640 1 -- 172.21.15.31:0/996808757 shutdown_connections 2024-04-25T01:52:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:00.400+0000 7f528e05d640 1 -- 172.21.15.31:0/996808757 wait complete. 2024-04-25T01:52:00.723 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:00 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/996808757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:00.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:00 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/996808757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:00.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:00 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/996808757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:01.143 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:52:00 smithi040 systemd[1]: Starting Ceph mds.e for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:01.402 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:52:01 smithi040 podman[61942]: 2024-04-25 01:52:01.047614728 +0000 UTC m=+0.023845463 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:01.402 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:52:01 smithi040 podman[61942]: 2024-04-25 01:52:01.166337626 +0000 UTC m=+0.142568336 container create 71f06a6cc78c7224382cfddcd23992c06b762ce66dbde46f45ab75d4faf1279e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9) 2024-04-25T01:52:01.402 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:52:01 smithi040 podman[61942]: 2024-04-25 01:52:01.377333525 +0000 UTC m=+0.353564246 container init 71f06a6cc78c7224382cfddcd23992c06b762ce66dbde46f45ab75d4faf1279e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T01:52:01.402 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:52:01 smithi040 podman[61942]: 2024-04-25 01:52:01.379998963 +0000 UTC m=+0.356229674 container start 71f06a6cc78c7224382cfddcd23992c06b762ce66dbde46f45ab75d4faf1279e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:52:01.403 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:52:01 smithi040 ceph-mds[61961]: -- 172.21.15.40:0/3187375849 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558e905bda40 con 0x558e90590c00 2024-04-25T01:52:01.666 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:01 smithi040 ceph-mon[32418]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:01.667 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:52:01 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e[61957]: starting mds.e at 2024-04-25T01:52:01.667 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:52:01 smithi040 bash[61942]: 71f06a6cc78c7224382cfddcd23992c06b762ce66dbde46f45ab75d4faf1279e 2024-04-25T01:52:01.667 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:52:01 smithi040 systemd[1]: Started Ceph mds.e for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:01.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:01 smithi031 ceph-mon[29204]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:01.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:01 smithi134 ceph-mon[33098]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:02.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:02.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 -- 172.21.15.31:0/3174226031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a8c1012e0 msgr2=0x7f1a8c108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:02.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 --2- 172.21.15.31:0/3174226031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a8c1012e0 0x7f1a8c108b80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c009930 tx=0x7f1a7c02a420 comp rx=0 tx=0).stop 2024-04-25T01:52:02.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 -- 172.21.15.31:0/3174226031 shutdown_connections 2024-04-25T01:52:02.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 --2- 172.21.15.31:0/3174226031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a8c1012e0 0x7f1a8c108b80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:02.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 --2- 172.21.15.31:0/3174226031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8c100920 0x7f1a8c100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:02.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 --2- 172.21.15.31:0/3174226031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a8c101c80 0x7f1a8c1020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:02.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 -- 172.21.15.31:0/3174226031 >> 172.21.15.31:0/3174226031 conn(0x7f1a8c0fb3b0 msgr2=0x7f1a8c0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:02.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 -- 172.21.15.31:0/3174226031 shutdown_connections 2024-04-25T01:52:02.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 -- 172.21.15.31:0/3174226031 wait complete. 2024-04-25T01:52:02.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 Processor -- start 2024-04-25T01:52:02.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 -- start start 2024-04-25T01:52:02.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8c100920 0x7f1a8c100380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a8c1012e0 0x7f1a8c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a8c101c80 0x7f1a8c19eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10c210 con 0x7f1a8c1012e0 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10c090 con 0x7f1a8c100920 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.527+0000 7f1a92f09640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10c390 con 0x7f1a8c101c80 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a8bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a8c1012e0 0x7f1a8c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a8bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a8c1012e0 0x7f1a8c19c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48682/0 (socket says 172.21.15.31:48682) 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a8bfff640 1 -- 172.21.15.31:0/2877595009 learned_addr learned my addr 172.21.15.31:0/2877595009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a914a6640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a8c101c80 0x7f1a8c19eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a90ca5640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8c100920 0x7f1a8c100380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a8bfff640 1 -- 172.21.15.31:0/2877595009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a8c101c80 msgr2=0x7f1a8c19eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a8bfff640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a8c101c80 0x7f1a8c19eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a8bfff640 1 -- 172.21.15.31:0/2877595009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8c100920 msgr2=0x7f1a8c100380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a8bfff640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8c100920 0x7f1a8c100380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:02.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a8bfff640 1 -- 172.21.15.31:0/2877595009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8c19f5b0 con 0x7f1a8c1012e0 2024-04-25T01:52:02.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a914a6640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a8c101c80 0x7f1a8c19eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:52:02.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a8bfff640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a8c1012e0 0x7f1a8c19c680 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000d950 tx=0x7f1a8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:02.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a89ffb640 1 -- 172.21.15.31:0/2877595009 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80017030 con 0x7f1a8c1012e0 2024-04-25T01:52:02.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a89ffb640 1 -- 172.21.15.31:0/2877595009 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a80004020 con 0x7f1a8c1012e0 2024-04-25T01:52:02.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a92f09640 1 -- 172.21.15.31:0/2877595009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a8c19f840 con 0x7f1a8c1012e0 2024-04-25T01:52:02.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.528+0000 7f1a89ffb640 1 -- 172.21.15.31:0/2877595009 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a800047a0 con 0x7f1a8c1012e0 2024-04-25T01:52:02.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.529+0000 7f1a92f09640 1 -- 172.21.15.31:0/2877595009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a8c19fde0 con 0x7f1a8c1012e0 2024-04-25T01:52:02.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.530+0000 7f1a92f09640 1 -- 172.21.15.31:0/2877595009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a8c107a00 con 0x7f1a8c1012e0 2024-04-25T01:52:02.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.531+0000 7f1a89ffb640 1 -- 172.21.15.31:0/2877595009 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1a800041d0 con 0x7f1a8c1012e0 2024-04-25T01:52:02.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.531+0000 7f1a89ffb640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1a740a2c10 0x7f1a740a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:02.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.531+0000 7f1a89ffb640 1 -- 172.21.15.31:0/2877595009 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f1a800ce6c0 con 0x7f1a8c1012e0 2024-04-25T01:52:02.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.531+0000 7f1a90ca5640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1a740a2c10 0x7f1a740a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:02.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.532+0000 7f1a90ca5640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1a740a2c10 0x7f1a740a50d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1a840092f0 tx=0x7f1a84009200 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:02.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.536+0000 7f1a89ffb640 1 -- 172.21.15.31:0/2877595009 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1a8009bac0 con 0x7f1a8c1012e0 2024-04-25T01:52:02.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:02 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.40:6834/538218297,v1:172.21.15.40:6835/538218297] up:boot 2024-04-25T01:52:02.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:02 smithi134 ceph-mon[33098]: fsmap 5 up:standby 2024-04-25T01:52:02.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:02 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:52:02.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:02 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:02.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:02 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:02.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:02 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:02.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:02 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:02.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:02 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:02.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:02 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:02.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:02 smithi134 ceph-mon[33098]: Deploying daemon mds.f on smithi134 2024-04-25T01:52:02.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:02 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.40:6834/538218297,v1:172.21.15.40:6835/538218297] up:boot 2024-04-25T01:52:02.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:02 smithi031 ceph-mon[29204]: fsmap 5 up:standby 2024-04-25T01:52:02.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:52:02.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:02.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:02.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:02.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:02.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:02.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:02 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:02.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:02 smithi031 ceph-mon[29204]: Deploying daemon mds.f on smithi134 2024-04-25T01:52:02.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.760+0000 7f1a92f09640 1 -- 172.21.15.31:0/2877595009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a8c02d360 con 0x7f1a8c1012e0 2024-04-25T01:52:02.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.761+0000 7f1a89ffb640 1 -- 172.21.15.31:0/2877595009 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7f1a8009b3c0 con 0x7f1a8c1012e0 2024-04-25T01:52:02.761 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:02.762 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T01:52:02.762 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 6 2024-04-25T01:52:02.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 -- 172.21.15.31:0/2877595009 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1a740a2c10 msgr2=0x7f1a740a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:02.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1a740a2c10 0x7f1a740a50d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1a840092f0 tx=0x7f1a84009200 comp rx=0 tx=0).stop 2024-04-25T01:52:02.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 -- 172.21.15.31:0/2877595009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a8c1012e0 msgr2=0x7f1a8c19c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:02.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a8c1012e0 0x7f1a8c19c680 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000d950 tx=0x7f1a8000de10 comp rx=0 tx=0).stop 2024-04-25T01:52:02.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 -- 172.21.15.31:0/2877595009 shutdown_connections 2024-04-25T01:52:02.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1a740a2c10 0x7f1a740a50d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:02.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a8c101c80 0x7f1a8c19eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:02.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a8c1012e0 0x7f1a8c19c680 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:02.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 --2- 172.21.15.31:0/2877595009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8c100920 0x7f1a8c100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:02.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 -- 172.21.15.31:0/2877595009 >> 172.21.15.31:0/2877595009 conn(0x7f1a8c0fb3b0 msgr2=0x7f1a8c104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:02.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 -- 172.21.15.31:0/2877595009 shutdown_connections 2024-04-25T01:52:02.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:02.763+0000 7f1a92f09640 1 -- 172.21.15.31:0/2877595009 wait complete. 2024-04-25T01:52:02.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:02 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.40:6834/538218297,v1:172.21.15.40:6835/538218297] up:boot 2024-04-25T01:52:02.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:02 smithi040 ceph-mon[32418]: fsmap 5 up:standby 2024-04-25T01:52:02.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:02 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:52:02.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:02 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:02.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:02 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:02.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:02 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:02.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:02 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:02.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:02 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:02.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:02 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:02.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:02 smithi040 ceph-mon[32418]: Deploying daemon mds.f on smithi134 2024-04-25T01:52:03.842 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:03 smithi134 ceph-mon[33098]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:03.842 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:03 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2877595009' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:03.843 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 01:52:03 smithi134 systemd[1]: Starting Ceph mds.f for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:03.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:03 smithi040 ceph-mon[32418]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:03.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:03 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2877595009' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:03.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:03 smithi031 ceph-mon[29204]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:03.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:03 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2877595009' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:04.134 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 01:52:03 smithi134 podman[63353]: 2024-04-25 01:52:03.746423148 +0000 UTC m=+0.018449616 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:04.134 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 01:52:03 smithi134 podman[63353]: 2024-04-25 01:52:03.862149316 +0000 UTC m=+0.134175776 container create f69574034f9fde8b9b2c5cf717e66fa3fbbb2ea3589bc16fb76f61dc835e875a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T01:52:04.134 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 01:52:04 smithi134 podman[63353]: 2024-04-25 01:52:04.135253869 +0000 UTC m=+0.407280330 container init f69574034f9fde8b9b2c5cf717e66fa3fbbb2ea3589bc16fb76f61dc835e875a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:52:04.437 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 01:52:04 smithi134 podman[63353]: 2024-04-25 01:52:04.137832135 +0000 UTC m=+0.409858597 container start f69574034f9fde8b9b2c5cf717e66fa3fbbb2ea3589bc16fb76f61dc835e875a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T01:52:04.437 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 01:52:04 smithi134 ceph-mds[63372]: -- 172.21.15.134:0/4125539461 <== mon.1 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56135a63ba40 con 0x56135a60ec00 2024-04-25T01:52:04.438 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 01:52:04 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-f[63368]: starting mds.f at 2024-04-25T01:52:04.438 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 01:52:04 smithi134 bash[63353]: f69574034f9fde8b9b2c5cf717e66fa3fbbb2ea3589bc16fb76f61dc835e875a 2024-04-25T01:52:04.438 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 01:52:04 smithi134 systemd[1]: Started Ceph mds.f for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:04.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:04.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.890+0000 7fc08b73b640 1 -- 172.21.15.31:0/2404536300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0841029a0 msgr2=0x7fc084102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:04.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.890+0000 7fc08b73b640 1 --2- 172.21.15.31:0/2404536300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0841029a0 0x7fc084102e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc074009900 tx=0x7fc07402a410 comp rx=0 tx=0).stop 2024-04-25T01:52:04.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.890+0000 7fc08b73b640 1 -- 172.21.15.31:0/2404536300 shutdown_connections 2024-04-25T01:52:04.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.890+0000 7fc08b73b640 1 --2- 172.21.15.31:0/2404536300 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc084103360 0x7fc084109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.890+0000 7fc08b73b640 1 --2- 172.21.15.31:0/2404536300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0841029a0 0x7fc084102e20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.890+0000 7fc08b73b640 1 --2- 172.21.15.31:0/2404536300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0841017b0 0x7fc084101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.890+0000 7fc08b73b640 1 -- 172.21.15.31:0/2404536300 >> 172.21.15.31:0/2404536300 conn(0x7fc0840fcd40 msgr2=0x7fc0840ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.891+0000 7fc08b73b640 1 -- 172.21.15.31:0/2404536300 shutdown_connections 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.891+0000 7fc08b73b640 1 -- 172.21.15.31:0/2404536300 wait complete. 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.891+0000 7fc08b73b640 1 Processor -- start 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.891+0000 7fc08b73b640 1 -- start start 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.891+0000 7fc08b73b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0841017b0 0x7fc084199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.891+0000 7fc08b73b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0841029a0 0x7fc08419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.891+0000 7fc08b73b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc084103360 0x7fc0841a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.891+0000 7fc08b73b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc08410d300 con 0x7fc0841029a0 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.891+0000 7fc08b73b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc08410d180 con 0x7fc084103360 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.891+0000 7fc08b73b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc08410d480 con 0x7fc0841017b0 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc088cd6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0841029a0 0x7fc08419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc088cd6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0841029a0 0x7fc08419a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54158/0 (socket says 172.21.15.31:54158) 2024-04-25T01:52:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc088cd6640 1 -- 172.21.15.31:0/241975375 learned_addr learned my addr 172.21.15.31:0/241975375 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc0894d7640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0841017b0 0x7fc084199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc089cd8640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc084103360 0x7fc0841a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc088cd6640 1 -- 172.21.15.31:0/241975375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0841017b0 msgr2=0x7fc084199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc088cd6640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0841017b0 0x7fc084199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc088cd6640 1 -- 172.21.15.31:0/241975375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc084103360 msgr2=0x7fc0841a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc088cd6640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc084103360 0x7fc0841a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc088cd6640 1 -- 172.21.15.31:0/241975375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0841a1800 con 0x7fc0841029a0 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc088cd6640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0841029a0 0x7fc08419a120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc074005940 tx=0x7fc0740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc07a7fc640 1 -- 172.21.15.31:0/241975375 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc07403f030 con 0x7fc0841029a0 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc07a7fc640 1 -- 172.21.15.31:0/241975375 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0740318f0 con 0x7fc0841029a0 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc08b73b640 1 -- 172.21.15.31:0/241975375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0841a1a90 con 0x7fc0841029a0 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc07a7fc640 1 -- 172.21.15.31:0/241975375 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074035070 con 0x7fc0841029a0 2024-04-25T01:52:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.892+0000 7fc08b73b640 1 -- 172.21.15.31:0/241975375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0841a1fd0 con 0x7fc0841029a0 2024-04-25T01:52:04.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.893+0000 7fc08b73b640 1 -- 172.21.15.31:0/241975375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0841a21a0 con 0x7fc0841029a0 2024-04-25T01:52:04.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.894+0000 7fc07a7fc640 1 -- 172.21.15.31:0/241975375 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc074042030 con 0x7fc0841029a0 2024-04-25T01:52:04.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.895+0000 7fc07a7fc640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc0700a2c10 0x7fc0700a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:04.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.895+0000 7fc0894d7640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc0700a2c10 0x7fc0700a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:04.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.895+0000 7fc07a7fc640 1 -- 172.21.15.31:0/241975375 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc0740ef100 con 0x7fc0841029a0 2024-04-25T01:52:04.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.896+0000 7fc0894d7640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc0700a2c10 0x7fc0700a50d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc084102800 tx=0x7fc080005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:04.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:04.900+0000 7fc07a7fc640 1 -- 172.21.15.31:0/241975375 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc0740bc460 con 0x7fc0841029a0 2024-04-25T01:52:05.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.122+0000 7fc08b73b640 1 -- 172.21.15.31:0/241975375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0841a2530 con 0x7fc0841029a0 2024-04-25T01:52:05.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.122+0000 7fc07a7fc640 1 -- 172.21.15.31:0/241975375 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7fc0740bbd60 con 0x7fc0841029a0 2024-04-25T01:52:05.122 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:05.122 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T01:52:05.122 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 7 2024-04-25T01:52:05.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 -- 172.21.15.31:0/241975375 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc0700a2c10 msgr2=0x7fc0700a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:05.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc0700a2c10 0x7fc0700a50d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc084102800 tx=0x7fc080005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:05.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 -- 172.21.15.31:0/241975375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0841029a0 msgr2=0x7fc08419a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:05.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0841029a0 0x7fc08419a120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc074005940 tx=0x7fc0740042f0 comp rx=0 tx=0).stop 2024-04-25T01:52:05.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 -- 172.21.15.31:0/241975375 shutdown_connections 2024-04-25T01:52:05.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc0700a2c10 0x7fc0700a50d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:05.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc084103360 0x7fc0841a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:05.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0841029a0 0x7fc08419a120 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:05.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 --2- 172.21.15.31:0/241975375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0841017b0 0x7fc084199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:05.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 -- 172.21.15.31:0/241975375 >> 172.21.15.31:0/241975375 conn(0x7fc0840fcd40 msgr2=0x7fc0841081c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:05.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 -- 172.21.15.31:0/241975375 shutdown_connections 2024-04-25T01:52:05.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:05.124+0000 7fc08b73b640 1 -- 172.21.15.31:0/241975375 wait complete. 2024-04-25T01:52:05.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:05.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:05.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:05.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:05.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:05.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:05.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:05.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: Deploying daemon mds.c on smithi134 2024-04-25T01:52:05.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] up:boot 2024-04-25T01:52:05.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: fsmap 6 up:standby 2024-04-25T01:52:05.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:52:05.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:05 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/241975375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:05.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:05.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:05.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:05.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:05.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:05.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:05.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:05.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: Deploying daemon mds.c on smithi134 2024-04-25T01:52:05.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] up:boot 2024-04-25T01:52:05.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: fsmap 6 up:standby 2024-04-25T01:52:05.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:52:05.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:05 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/241975375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:05.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:05.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:05.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:05.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:05.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:05.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:05.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:05.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: Deploying daemon mds.c on smithi134 2024-04-25T01:52:05.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] up:boot 2024-04-25T01:52:05.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: fsmap 6 up:standby 2024-04-25T01:52:05.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:52:05.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:05 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/241975375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:06.688 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 01:52:06 smithi134 systemd[1]: Starting Ceph mds.c for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:06.977 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 01:52:06 smithi134 podman[63661]: 2024-04-25 01:52:06.604488211 +0000 UTC m=+0.018032685 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:06.989 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 01:52:06 smithi134 podman[63661]: 2024-04-25 01:52:06.734289479 +0000 UTC m=+0.147833941 container create 1498fd4b1901db8a5d0ea6e3b350e3459981b789b4dbd065db2384637fcdb18c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T01:52:06.989 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 01:52:06 smithi134 podman[63661]: 2024-04-25 01:52:06.978723967 +0000 UTC m=+0.392268428 container init 1498fd4b1901db8a5d0ea6e3b350e3459981b789b4dbd065db2384637fcdb18c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-25T01:52:07.136 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:07.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 -- 172.21.15.31:0/800125048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1030100680 msgr2=0x7f1030100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 --2- 172.21.15.31:0/800125048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1030100680 0x7f1030100aa0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f102c009900 tx=0x7f102c02a410 comp rx=0 tx=0).stop 2024-04-25T01:52:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 -- 172.21.15.31:0/800125048 shutdown_connections 2024-04-25T01:52:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 --2- 172.21.15.31:0/800125048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1030102230 0x7f1030108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 --2- 172.21.15.31:0/800125048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1030101870 0x7f1030101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 --2- 172.21.15.31:0/800125048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1030100680 0x7f1030100aa0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 -- 172.21.15.31:0/800125048 >> 172.21.15.31:0/800125048 conn(0x7f10300fbbf0 msgr2=0x7f10300fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 -- 172.21.15.31:0/800125048 shutdown_connections 2024-04-25T01:52:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 -- 172.21.15.31:0/800125048 wait complete. 2024-04-25T01:52:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 Processor -- start 2024-04-25T01:52:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 -- start start 2024-04-25T01:52:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1030100680 0x7f1030197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1030101870 0x7f1030197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1030102230 0x7f103019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f103010c180 con 0x7f1030101870 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f103010c000 con 0x7f1030100680 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.252+0000 7f10376ba640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f103010c300 con 0x7f1030102230 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f1034c55640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1030101870 0x7f1030197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f1034c55640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1030101870 0x7f1030197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54172/0 (socket says 172.21.15.31:54172) 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f1034c55640 1 -- 172.21.15.31:0/2387748238 learned_addr learned my addr 172.21.15.31:0/2387748238 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f1035456640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1030100680 0x7f1030197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f1035c57640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1030102230 0x7f103019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f1034c55640 1 -- 172.21.15.31:0/2387748238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1030102230 msgr2=0x7f103019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f1034c55640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1030102230 0x7f103019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f1034c55640 1 -- 172.21.15.31:0/2387748238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1030100680 msgr2=0x7f1030197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f1034c55640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1030100680 0x7f1030197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f1034c55640 1 -- 172.21.15.31:0/2387748238 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103019f690 con 0x7f1030101870 2024-04-25T01:52:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f1034c55640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1030101870 0x7f1030197fb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f102000d950 tx=0x7f102000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f10267fc640 1 -- 172.21.15.31:0/2387748238 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1020017030 con 0x7f1030101870 2024-04-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f10267fc640 1 -- 172.21.15.31:0/2387748238 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1020004020 con 0x7f1030101870 2024-04-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f10376ba640 1 -- 172.21.15.31:0/2387748238 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103019f980 con 0x7f1030101870 2024-04-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f10267fc640 1 -- 172.21.15.31:0/2387748238 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10200047a0 con 0x7f1030101870 2024-04-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.253+0000 7f10376ba640 1 -- 172.21.15.31:0/2387748238 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f103019ff20 con 0x7f1030101870 2024-04-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.254+0000 7f10376ba640 1 -- 172.21.15.31:0/2387748238 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10301a01b0 con 0x7f1030101870 2024-04-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.255+0000 7f10267fc640 1 -- 172.21.15.31:0/2387748238 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f10200041d0 con 0x7f1030101870 2024-04-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.256+0000 7f10267fc640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f101c0a2c10 0x7f101c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.256+0000 7f1035456640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f101c0a2c10 0x7f101c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.256+0000 7f10267fc640 1 -- 172.21.15.31:0/2387748238 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f10200ce580 con 0x7f1030101870 2024-04-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.257+0000 7f1035456640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f101c0a2c10 0x7f101c0a50d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f102c005ab0 tx=0x7f102c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:07.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.260+0000 7f10267fc640 1 -- 172.21.15.31:0/2387748238 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f102009b980 con 0x7f1030101870 2024-04-25T01:52:07.315 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 01:52:06 smithi134 podman[63661]: 2024-04-25 01:52:06.981355505 +0000 UTC m=+0.394899967 container start 1498fd4b1901db8a5d0ea6e3b350e3459981b789b4dbd065db2384637fcdb18c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-c, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T01:52:07.315 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 01:52:07 smithi134 ceph-mds[63680]: -- 172.21.15.134:0/1016595848 <== mon.1 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d1a8cda40 con 0x562d1a8a1000 2024-04-25T01:52:07.315 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 01:52:07 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-c[63676]: starting mds.c at 2024-04-25T01:52:07.315 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 01:52:07 smithi134 bash[63661]: 1498fd4b1901db8a5d0ea6e3b350e3459981b789b4dbd065db2384637fcdb18c 2024-04-25T01:52:07.315 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 01:52:07 smithi134 systemd[1]: Started Ceph mds.c for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:07.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:07 smithi031 ceph-mon[29204]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:07.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:07 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:07.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:07 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:07.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:07 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:07.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:07 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:07.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:07 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:07.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:07 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:07.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.489+0000 7f10376ba640 1 -- 172.21.15.31:0/2387748238 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10301a0380 con 0x7f1030101870 2024-04-25T01:52:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.489+0000 7f10267fc640 1 -- 172.21.15.31:0/2387748238 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7f102009b280 con 0x7f1030101870 2024-04-25T01:52:07.489 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:07.489 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T01:52:07.489 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-04-25T01:52:07.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.491+0000 7f10376ba640 1 -- 172.21.15.31:0/2387748238 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f101c0a2c10 msgr2=0x7f101c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:07.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.491+0000 7f10376ba640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f101c0a2c10 0x7f101c0a50d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f102c005ab0 tx=0x7f102c031400 comp rx=0 tx=0).stop 2024-04-25T01:52:07.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.491+0000 7f10376ba640 1 -- 172.21.15.31:0/2387748238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1030101870 msgr2=0x7f1030197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:07.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.491+0000 7f10376ba640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1030101870 0x7f1030197fb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f102000d950 tx=0x7f102000de10 comp rx=0 tx=0).stop 2024-04-25T01:52:07.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.491+0000 7f10376ba640 1 -- 172.21.15.31:0/2387748238 shutdown_connections 2024-04-25T01:52:07.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.491+0000 7f10376ba640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f101c0a2c10 0x7f101c0a50d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:07.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.491+0000 7f10376ba640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1030102230 0x7f103019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:07.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.491+0000 7f10376ba640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1030101870 0x7f1030197fb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:07.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.491+0000 7f10376ba640 1 --2- 172.21.15.31:0/2387748238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1030100680 0x7f1030197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:07.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.491+0000 7f10376ba640 1 -- 172.21.15.31:0/2387748238 >> 172.21.15.31:0/2387748238 conn(0x7f10300fbbf0 msgr2=0x7f1030068960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:07.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.492+0000 7f10376ba640 1 -- 172.21.15.31:0/2387748238 shutdown_connections 2024-04-25T01:52:07.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:07.492+0000 7f10376ba640 1 -- 172.21.15.31:0/2387748238 wait complete. 2024-04-25T01:52:07.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:07 smithi040 ceph-mon[32418]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:07.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:07 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:07.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:07 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:07.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:07 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:07.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:07 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:07.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:07 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:07.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:07 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:07.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:07 smithi134 ceph-mon[33098]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:07.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:07 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:07.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:07 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:07.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:07 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:07.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:07 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:07.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:07 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:07.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:07 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:08.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:08 smithi040 ceph-mon[32418]: Deploying daemon mds.h on smithi040 2024-04-25T01:52:08.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:08 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] up:boot 2024-04-25T01:52:08.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:08 smithi040 ceph-mon[32418]: fsmap 7 up:standby 2024-04-25T01:52:08.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:08 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:52:08.388 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:08 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2387748238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:08.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:08 smithi134 ceph-mon[33098]: Deploying daemon mds.h on smithi040 2024-04-25T01:52:08.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:08 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] up:boot 2024-04-25T01:52:08.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:08 smithi134 ceph-mon[33098]: fsmap 7 up:standby 2024-04-25T01:52:08.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:08 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:52:08.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:08 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2387748238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:08.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:08 smithi031 ceph-mon[29204]: Deploying daemon mds.h on smithi040 2024-04-25T01:52:08.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:08 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] up:boot 2024-04-25T01:52:08.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:08 smithi031 ceph-mon[29204]: fsmap 7 up:standby 2024-04-25T01:52:08.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:08 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:52:08.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:08 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2387748238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:09.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:09.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.621+0000 7f223a31b640 1 -- 172.21.15.31:0/4085529299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2234102240 msgr2=0x7f2234108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.621+0000 7f223a31b640 1 --2- 172.21.15.31:0/4085529299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2234102240 0x7f2234108ad0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f222c009ca0 tx=0x7f222c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.621+0000 7f223a31b640 1 -- 172.21.15.31:0/4085529299 shutdown_connections 2024-04-25T01:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.621+0000 7f223a31b640 1 --2- 172.21.15.31:0/4085529299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2234102240 0x7f2234108ad0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.621+0000 7f223a31b640 1 --2- 172.21.15.31:0/4085529299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2234101880 0x7f2234101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.621+0000 7f223a31b640 1 --2- 172.21.15.31:0/4085529299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2234100690 0x7f2234100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.621+0000 7f223a31b640 1 -- 172.21.15.31:0/4085529299 >> 172.21.15.31:0/4085529299 conn(0x7f22340fbc20 msgr2=0x7f22340fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.621+0000 7f223a31b640 1 -- 172.21.15.31:0/4085529299 shutdown_connections 2024-04-25T01:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.622+0000 7f223a31b640 1 -- 172.21.15.31:0/4085529299 wait complete. 2024-04-25T01:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.622+0000 7f223a31b640 1 Processor -- start 2024-04-25T01:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.622+0000 7f223a31b640 1 -- start start 2024-04-25T01:52:09.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.622+0000 7f223a31b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2234100690 0x7f223406a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:09.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.622+0000 7f223a31b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2234101880 0x7f223419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f223a31b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2234102240 0x7f22341a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f223a31b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f223410c1c0 con 0x7f2234101880 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f223a31b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f223410c040 con 0x7f2234100690 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f223a31b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f223410c340 con 0x7f2234102240 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22337fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2234101880 0x7f223419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22337fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2234101880 0x7f223419ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54190/0 (socket says 172.21.15.31:54190) 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22337fe640 1 -- 172.21.15.31:0/1538274916 learned_addr learned my addr 172.21.15.31:0/1538274916 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22388b8640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2234102240 0x7f22341a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f2233fff640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2234100690 0x7f223406a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22337fe640 1 -- 172.21.15.31:0/1538274916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2234102240 msgr2=0x7f22341a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22337fe640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2234102240 0x7f22341a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22337fe640 1 -- 172.21.15.31:0/1538274916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2234100690 msgr2=0x7f223406a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22337fe640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2234100690 0x7f223406a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22337fe640 1 -- 172.21.15.31:0/1538274916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22341a1810 con 0x7f2234101880 2024-04-25T01:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f2233fff640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2234100690 0x7f223406a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:52:09.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22337fe640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2234101880 0x7f223419ead0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f221c00e9f0 tx=0x7f221c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:09.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22317fa640 1 -- 172.21.15.31:0/1538274916 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221c00cd80 con 0x7f2234101880 2024-04-25T01:52:09.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22317fa640 1 -- 172.21.15.31:0/1538274916 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f221c0108e0 con 0x7f2234101880 2024-04-25T01:52:09.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f223a31b640 1 -- 172.21.15.31:0/1538274916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22341a1aa0 con 0x7f2234101880 2024-04-25T01:52:09.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f22317fa640 1 -- 172.21.15.31:0/1538274916 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221c018920 con 0x7f2234101880 2024-04-25T01:52:09.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.623+0000 7f223a31b640 1 -- 172.21.15.31:0/1538274916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22341a2040 con 0x7f2234101880 2024-04-25T01:52:09.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.625+0000 7f223a31b640 1 -- 172.21.15.31:0/1538274916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2234107a80 con 0x7f2234101880 2024-04-25T01:52:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.626+0000 7f22317fa640 1 -- 172.21.15.31:0/1538274916 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f221c010a90 con 0x7f2234101880 2024-04-25T01:52:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.627+0000 7f22317fa640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f22200a2c40 0x7f22200a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.627+0000 7f2233fff640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f22200a2c40 0x7f22200a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.627+0000 7f22317fa640 1 -- 172.21.15.31:0/1538274916 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f221c0ceef0 con 0x7f2234101880 2024-04-25T01:52:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.627+0000 7f2233fff640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f22200a2c40 0x7f22200a5100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f22240058c0 tx=0x7f2224005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:09.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.631+0000 7f22317fa640 1 -- 172.21.15.31:0/1538274916 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f221c09c2f0 con 0x7f2234101880 2024-04-25T01:52:09.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:09 smithi040 ceph-mon[32418]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:09.659 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 01:52:09 smithi040 systemd[1]: Starting Ceph mds.h for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:09.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:09 smithi134 ceph-mon[33098]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:09.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:09 smithi031 ceph-mon[29204]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:09.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.860+0000 7f223a31b640 1 -- 172.21.15.31:0/1538274916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f223402d390 con 0x7f2234101880 2024-04-25T01:52:09.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.861+0000 7f22317fa640 1 -- 172.21.15.31:0/1538274916 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7f221c09bbf0 con 0x7f2234101880 2024-04-25T01:52:09.861 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:09.861 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T01:52:09.861 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-04-25T01:52:09.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.863+0000 7f223a31b640 1 -- 172.21.15.31:0/1538274916 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f22200a2c40 msgr2=0x7f22200a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:09.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.863+0000 7f223a31b640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f22200a2c40 0x7f22200a5100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f22240058c0 tx=0x7f2224005d50 comp rx=0 tx=0).stop 2024-04-25T01:52:09.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.863+0000 7f223a31b640 1 -- 172.21.15.31:0/1538274916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2234101880 msgr2=0x7f223419ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:09.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.863+0000 7f223a31b640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2234101880 0x7f223419ead0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f221c00e9f0 tx=0x7f221c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.863+0000 7f223a31b640 1 -- 172.21.15.31:0/1538274916 shutdown_connections 2024-04-25T01:52:09.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.863+0000 7f223a31b640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f22200a2c40 0x7f22200a5100 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.863+0000 7f223a31b640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2234102240 0x7f22341a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.863+0000 7f223a31b640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2234101880 0x7f223419ead0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.863+0000 7f223a31b640 1 --2- 172.21.15.31:0/1538274916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2234100690 0x7f223406a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.863+0000 7f223a31b640 1 -- 172.21.15.31:0/1538274916 >> 172.21.15.31:0/1538274916 conn(0x7f22340fbc20 msgr2=0x7f2234104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:09.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.863+0000 7f223a31b640 1 -- 172.21.15.31:0/1538274916 shutdown_connections 2024-04-25T01:52:09.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:09.864+0000 7f223a31b640 1 -- 172.21.15.31:0/1538274916 wait complete. 2024-04-25T01:52:09.974 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 01:52:09 smithi040 podman[62264]: 2024-04-25 01:52:09.585746266 +0000 UTC m=+0.018554443 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:09.975 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 01:52:09 smithi040 podman[62264]: 2024-04-25 01:52:09.7147597 +0000 UTC m=+0.147567865 container create 46c3665ae4d9b93d04aa269e8c5910fe722bcbac69775d7c34b1f7ac7b404ec8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9) 2024-04-25T01:52:09.975 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 01:52:09 smithi040 podman[62264]: 2024-04-25 01:52:09.975953282 +0000 UTC m=+0.408761448 container init 46c3665ae4d9b93d04aa269e8c5910fe722bcbac69775d7c34b1f7ac7b404ec8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T01:52:10.326 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 01:52:09 smithi040 podman[62264]: 2024-04-25 01:52:09.978796621 +0000 UTC m=+0.411604790 container start 46c3665ae4d9b93d04aa269e8c5910fe722bcbac69775d7c34b1f7ac7b404ec8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T01:52:10.326 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 01:52:10 smithi040 ceph-mds[62283]: -- 172.21.15.40:0/1728532256 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d73d5cda40 con 0x55d73d5a1000 2024-04-25T01:52:10.326 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 01:52:10 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-h[62279]: starting mds.h at 2024-04-25T01:52:10.326 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 01:52:10 smithi040 bash[62264]: 46c3665ae4d9b93d04aa269e8c5910fe722bcbac69775d7c34b1f7ac7b404ec8 2024-04-25T01:52:10.326 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 01:52:10 smithi040 systemd[1]: Started Ceph mds.h for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:10.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:10 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1538274916' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:10.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:10.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:10.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:10.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:10.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:10.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:10 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:10.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:10 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1538274916' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:10.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:10.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:10.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:10.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:10.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:10.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:10 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:10.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:10 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1538274916' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:10.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:10.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:10.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:10.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:10.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:10.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:10 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:11.437 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:11 smithi134 ceph-mon[33098]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:11.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:11 smithi134 ceph-mon[33098]: Deploying daemon mds.i on smithi134 2024-04-25T01:52:11.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:11 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] up:boot 2024-04-25T01:52:11.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:11 smithi134 ceph-mon[33098]: fsmap 8 up:standby 2024-04-25T01:52:11.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:11 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:52:11.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:11 smithi040 ceph-mon[32418]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:11.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:11 smithi040 ceph-mon[32418]: Deploying daemon mds.i on smithi134 2024-04-25T01:52:11.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:11 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] up:boot 2024-04-25T01:52:11.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:11 smithi040 ceph-mon[32418]: fsmap 8 up:standby 2024-04-25T01:52:11.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:11 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:52:11.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:11 smithi031 ceph-mon[29204]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:11.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:11 smithi031 ceph-mon[29204]: Deploying daemon mds.i on smithi134 2024-04-25T01:52:11.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:11 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] up:boot 2024-04-25T01:52:11.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:11 smithi031 ceph-mon[29204]: fsmap 8 up:standby 2024-04-25T01:52:11.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:11 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:52:11.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:11.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.991+0000 7fe265b34640 1 -- 172.21.15.31:0/2153425553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe260104b50 msgr2=0x7fe260106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:11.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.991+0000 7fe265b34640 1 --2- 172.21.15.31:0/2153425553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe260104b50 0x7fe260106f40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe254009900 tx=0x7fe25402a410 comp rx=0 tx=0).stop 2024-04-25T01:52:11.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.991+0000 7fe265b34640 1 -- 172.21.15.31:0/2153425553 shutdown_connections 2024-04-25T01:52:11.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.991+0000 7fe265b34640 1 --2- 172.21.15.31:0/2153425553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe260107480 0x7fe2601098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.991+0000 7fe265b34640 1 --2- 172.21.15.31:0/2153425553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe260104b50 0x7fe260106f40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.991+0000 7fe265b34640 1 --2- 172.21.15.31:0/2153425553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe260102220 0x7fe260104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.991+0000 7fe265b34640 1 -- 172.21.15.31:0/2153425553 >> 172.21.15.31:0/2153425553 conn(0x7fe2600fbc30 msgr2=0x7fe2600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.991+0000 7fe265b34640 1 -- 172.21.15.31:0/2153425553 shutdown_connections 2024-04-25T01:52:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.991+0000 7fe265b34640 1 -- 172.21.15.31:0/2153425553 wait complete. 2024-04-25T01:52:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe265b34640 1 Processor -- start 2024-04-25T01:52:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe265b34640 1 -- start start 2024-04-25T01:52:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe265b34640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe260102220 0x7fe260101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe265b34640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe260104b50 0x7fe2601001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe265b34640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe260107480 0x7fe260100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe265b34640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe26010d330 con 0x7fe260107480 2024-04-25T01:52:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe265b34640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe26010d1b0 con 0x7fe260102220 2024-04-25T01:52:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe265b34640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe26010d4b0 con 0x7fe260104b50 2024-04-25T01:52:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe25ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe260107480 0x7fe260100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe25ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe260107480 0x7fe260100710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54216/0 (socket says 172.21.15.31:54216) 2024-04-25T01:52:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe25ffff640 1 -- 172.21.15.31:0/2762462752 learned_addr learned my addr 172.21.15.31:0/2762462752 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe25f7fe640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe260102220 0x7fe260101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.992+0000 7fe25effd640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe260104b50 0x7fe2601001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.993+0000 7fe25ffff640 1 -- 172.21.15.31:0/2762462752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe260104b50 msgr2=0x7fe2601001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.993+0000 7fe25ffff640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe260104b50 0x7fe2601001d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.993+0000 7fe25ffff640 1 -- 172.21.15.31:0/2762462752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe260102220 msgr2=0x7fe260101b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.993+0000 7fe25ffff640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe260102220 0x7fe260101b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.993+0000 7fe25ffff640 1 -- 172.21.15.31:0/2762462752 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe260100fa0 con 0x7fe260107480 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.993+0000 7fe25ffff640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe260107480 0x7fe260100710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe25800eaa0 tx=0x7fe25800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.993+0000 7fe25cff9640 1 -- 172.21.15.31:0/2762462752 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25800cdb0 con 0x7fe260107480 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.993+0000 7fe25cff9640 1 -- 172.21.15.31:0/2762462752 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe258005020 con 0x7fe260107480 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.993+0000 7fe265b34640 1 -- 172.21.15.31:0/2762462752 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2601a1b80 con 0x7fe260107480 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.993+0000 7fe25cff9640 1 -- 172.21.15.31:0/2762462752 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2580116d0 con 0x7fe260107480 2024-04-25T01:52:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.993+0000 7fe265b34640 1 -- 172.21.15.31:0/2762462752 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2601a20a0 con 0x7fe260107480 2024-04-25T01:52:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.994+0000 7fe265b34640 1 -- 172.21.15.31:0/2762462752 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2600689b0 con 0x7fe260107480 2024-04-25T01:52:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.995+0000 7fe25cff9640 1 -- 172.21.15.31:0/2762462752 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe2580051d0 con 0x7fe260107480 2024-04-25T01:52:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.996+0000 7fe25cff9640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fe2480a2c10 0x7fe2480a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.996+0000 7fe25f7fe640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fe2480a2c10 0x7fe2480a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.996+0000 7fe25cff9640 1 -- 172.21.15.31:0/2762462752 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fe2580ced60 con 0x7fe260107480 2024-04-25T01:52:11.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:11.996+0000 7fe25f7fe640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fe2480a2c10 0x7fe2480a50d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe250009bd0 tx=0x7fe250005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:12.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.000+0000 7fe25cff9640 1 -- 172.21.15.31:0/2762462752 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe258016030 con 0x7fe260107480 2024-04-25T01:52:12.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.224+0000 7fe265b34640 1 -- 172.21.15.31:0/2762462752 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe260062340 con 0x7fe260107480 2024-04-25T01:52:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.225+0000 7fe25cff9640 1 -- 172.21.15.31:0/2762462752 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7fe25809ba70 con 0x7fe260107480 2024-04-25T01:52:12.225 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:12.225 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T01:52:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 9 2024-04-25T01:52:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 -- 172.21.15.31:0/2762462752 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fe2480a2c10 msgr2=0x7fe2480a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fe2480a2c10 0x7fe2480a50d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe250009bd0 tx=0x7fe250005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 -- 172.21.15.31:0/2762462752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe260107480 msgr2=0x7fe260100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe260107480 0x7fe260100710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe25800eaa0 tx=0x7fe25800ef60 comp rx=0 tx=0).stop 2024-04-25T01:52:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 -- 172.21.15.31:0/2762462752 shutdown_connections 2024-04-25T01:52:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fe2480a2c10 0x7fe2480a50d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe260107480 0x7fe260100710 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe260104b50 0x7fe2601001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 --2- 172.21.15.31:0/2762462752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe260102220 0x7fe260101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 -- 172.21.15.31:0/2762462752 >> 172.21.15.31:0/2762462752 conn(0x7fe2600fbc30 msgr2=0x7fe260102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 -- 172.21.15.31:0/2762462752 shutdown_connections 2024-04-25T01:52:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:12.227+0000 7fe265b34640 1 -- 172.21.15.31:0/2762462752 wait complete. 2024-04-25T01:52:12.339 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 01:52:12 smithi134 systemd[1]: Starting Ceph mds.i for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:12.649 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:12 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2762462752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:12.649 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 01:52:12 smithi134 podman[63983]: 2024-04-25 01:52:12.261045674 +0000 UTC m=+0.017909139 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:12.649 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 01:52:12 smithi134 podman[63983]: 2024-04-25 01:52:12.389304427 +0000 UTC m=+0.146167877 container create 4aaa4ddc19e624a140786bbfdd92b5d5d05996ee26c609d312f4719cb10abb93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T01:52:12.649 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 01:52:12 smithi134 podman[63983]: 2024-04-25 01:52:12.650520392 +0000 UTC m=+0.407383843 container init 4aaa4ddc19e624a140786bbfdd92b5d5d05996ee26c609d312f4719cb10abb93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-i, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T01:52:12.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:12 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2762462752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:12.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:12 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2762462752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:12.938 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 01:52:12 smithi134 podman[63983]: 2024-04-25 01:52:12.653311784 +0000 UTC m=+0.410175234 container start 4aaa4ddc19e624a140786bbfdd92b5d5d05996ee26c609d312f4719cb10abb93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9) 2024-04-25T01:52:12.938 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 01:52:12 smithi134 ceph-mds[64002]: -- 172.21.15.134:0/2434444055 <== mon.1 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558d0a637a40 con 0x558d0a60b000 2024-04-25T01:52:12.938 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 01:52:12 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-i[63998]: starting mds.i at 2024-04-25T01:52:12.938 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 01:52:12 smithi134 bash[63983]: 4aaa4ddc19e624a140786bbfdd92b5d5d05996ee26c609d312f4719cb10abb93 2024-04-25T01:52:12.938 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 01:52:12 smithi134 systemd[1]: Started Ceph mds.i for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:13 smithi040 ceph-mon[32418]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:13.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:13.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:13.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:13.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:13.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:13 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:13.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:13 smithi031 ceph-mon[29204]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:13.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:13.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:13.660 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:13.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:13.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:13.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:13 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:13.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:13 smithi134 ceph-mon[33098]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:13.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:13.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:13.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:13.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:13.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:13.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:13 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:14.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:14.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.355+0000 7f53170cd640 1 -- 172.21.15.31:0/607625153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5310103350 msgr2=0x7f5310109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:14.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.355+0000 7f53170cd640 1 --2- 172.21.15.31:0/607625153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5310103350 0x7f5310109be0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5308009ca0 tx=0x7f530802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.355+0000 7f53170cd640 1 -- 172.21.15.31:0/607625153 shutdown_connections 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.355+0000 7f53170cd640 1 --2- 172.21.15.31:0/607625153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5310103350 0x7f5310109be0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.355+0000 7f53170cd640 1 --2- 172.21.15.31:0/607625153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5310102990 0x7f5310102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.355+0000 7f53170cd640 1 --2- 172.21.15.31:0/607625153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53101017a0 0x7f5310101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.355+0000 7f53170cd640 1 -- 172.21.15.31:0/607625153 >> 172.21.15.31:0/607625153 conn(0x7f53100fccf0 msgr2=0x7f53100ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f53170cd640 1 -- 172.21.15.31:0/607625153 shutdown_connections 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f53170cd640 1 -- 172.21.15.31:0/607625153 wait complete. 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f53170cd640 1 Processor -- start 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f53170cd640 1 -- start start 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f53170cd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53101017a0 0x7f53101978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f53170cd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5310102990 0x7f5310197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f53170cd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5310103350 0x7f531019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f53170cd640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f531010d2b0 con 0x7f5310103350 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f53170cd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f531010d130 con 0x7f53101017a0 2024-04-25T01:52:14.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f53170cd640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f531010d430 con 0x7f5310102990 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f531566a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5310103350 0x7f531019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.356+0000 7f531566a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5310103350 0x7f531019ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59984/0 (socket says 172.21.15.31:59984) 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f531566a640 1 -- 172.21.15.31:0/172705312 learned_addr learned my addr 172.21.15.31:0/172705312 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f530ffff640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5310102990 0x7f5310197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f5314e69640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53101017a0 0x7f53101978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f531566a640 1 -- 172.21.15.31:0/172705312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5310102990 msgr2=0x7f5310197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f531566a640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5310102990 0x7f5310197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f531566a640 1 -- 172.21.15.31:0/172705312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53101017a0 msgr2=0x7f53101978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f531566a640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53101017a0 0x7f53101978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f531566a640 1 -- 172.21.15.31:0/172705312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531019f4f0 con 0x7f5310103350 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f531566a640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5310103350 0x7f531019ee80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5308009dc0 tx=0x7f5308003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f530dffb640 1 -- 172.21.15.31:0/172705312 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5308032ec0 con 0x7f5310103350 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f530dffb640 1 -- 172.21.15.31:0/172705312 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53080044f0 con 0x7f5310103350 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f530dffb640 1 -- 172.21.15.31:0/172705312 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5308030d40 con 0x7f5310103350 2024-04-25T01:52:14.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f53170cd640 1 -- 172.21.15.31:0/172705312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f531019f780 con 0x7f5310103350 2024-04-25T01:52:14.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.357+0000 7f53170cd640 1 -- 172.21.15.31:0/172705312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f531019fcd0 con 0x7f5310103350 2024-04-25T01:52:14.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.358+0000 7f53170cd640 1 -- 172.21.15.31:0/172705312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5310108ac0 con 0x7f5310103350 2024-04-25T01:52:14.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.360+0000 7f530dffb640 1 -- 172.21.15.31:0/172705312 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5308004040 con 0x7f5310103350 2024-04-25T01:52:14.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.360+0000 7f530dffb640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f53000a2db0 0x7f53000a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:14.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.360+0000 7f5314e69640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f53000a2db0 0x7f53000a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:14.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.360+0000 7f530dffb640 1 -- 172.21.15.31:0/172705312 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f53080f02f0 con 0x7f5310103350 2024-04-25T01:52:14.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.361+0000 7f5314e69640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f53000a2db0 0x7f53000a5270 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5304005fd0 tx=0x7f5304005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:14.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.364+0000 7f530dffb640 1 -- 172.21.15.31:0/172705312 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f53080bd6f0 con 0x7f5310103350 2024-04-25T01:52:14.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:14 smithi031 ceph-mon[29204]: Deploying daemon mds.g on smithi031 2024-04-25T01:52:14.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:14 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] up:boot 2024-04-25T01:52:14.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:14 smithi031 ceph-mon[29204]: fsmap 9 up:standby 2024-04-25T01:52:14.528 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:14 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:52:14.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.589+0000 7f53170cd640 1 -- 172.21.15.31:0/172705312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f531002d290 con 0x7f5310103350 2024-04-25T01:52:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.590+0000 7f530dffb640 1 -- 172.21.15.31:0/172705312 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7998 (secure 0 0 0) 0x7f53080bcff0 con 0x7f5310103350 2024-04-25T01:52:14.590 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:14.590 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:52:14.590 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 10 2024-04-25T01:52:14.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 -- 172.21.15.31:0/172705312 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f53000a2db0 msgr2=0x7f53000a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:14.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f53000a2db0 0x7f53000a5270 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5304005fd0 tx=0x7f5304005850 comp rx=0 tx=0).stop 2024-04-25T01:52:14.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 -- 172.21.15.31:0/172705312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5310103350 msgr2=0x7f531019ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:14.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5310103350 0x7f531019ee80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5308009dc0 tx=0x7f5308003ba0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 -- 172.21.15.31:0/172705312 shutdown_connections 2024-04-25T01:52:14.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f53000a2db0 0x7f53000a5270 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5310103350 0x7f531019ee80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5310102990 0x7f5310197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 --2- 172.21.15.31:0/172705312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53101017a0 0x7f53101978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 -- 172.21.15.31:0/172705312 >> 172.21.15.31:0/172705312 conn(0x7f53100fccf0 msgr2=0x7f5310105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:14.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 -- 172.21.15.31:0/172705312 shutdown_connections 2024-04-25T01:52:14.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:14.592+0000 7f53170cd640 1 -- 172.21.15.31:0/172705312 wait complete. 2024-04-25T01:52:14.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:14 smithi040 ceph-mon[32418]: Deploying daemon mds.g on smithi031 2024-04-25T01:52:14.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:14 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] up:boot 2024-04-25T01:52:14.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:14 smithi040 ceph-mon[32418]: fsmap 9 up:standby 2024-04-25T01:52:14.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:14 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:52:14.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:14 smithi134 ceph-mon[33098]: Deploying daemon mds.g on smithi031 2024-04-25T01:52:14.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:14 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] up:boot 2024-04-25T01:52:14.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:14 smithi134 ceph-mon[33098]: fsmap 9 up:standby 2024-04-25T01:52:14.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:14 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:52:15.237 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 01:52:15 smithi031 systemd[1]: Starting Ceph mds.g for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:15.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:15 smithi031 ceph-mon[29204]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:15.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:15 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/172705312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:15.631 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 01:52:15 smithi031 podman[70064]: 2024-04-25 01:52:15.240334909 +0000 UTC m=+0.018409238 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:15.631 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 01:52:15 smithi031 podman[70064]: 2024-04-25 01:52:15.370911142 +0000 UTC m=+0.148985449 container create 5c78e932ce5f7b5a093f68a8aacd2c75d5b59836ed19e1e7f433f32a5fbe2bbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:52:15.631 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 01:52:15 smithi031 podman[70064]: 2024-04-25 01:52:15.632030753 +0000 UTC m=+0.410105061 container init 5c78e932ce5f7b5a093f68a8aacd2c75d5b59836ed19e1e7f433f32a5fbe2bbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T01:52:15.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:15 smithi040 ceph-mon[32418]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:15.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:15 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/172705312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:15.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:15 smithi134 ceph-mon[33098]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:15.688 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:15 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/172705312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:15.986 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 01:52:15 smithi031 podman[70064]: 2024-04-25 01:52:15.635054526 +0000 UTC m=+0.413128841 container start 5c78e932ce5f7b5a093f68a8aacd2c75d5b59836ed19e1e7f433f32a5fbe2bbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-25T01:52:15.987 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 01:52:15 smithi031 ceph-mds[70083]: -- 172.21.15.31:0/446848498 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5639a47d9a40 con 0x5639a47acc00 2024-04-25T01:52:15.987 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 01:52:15 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-g[70079]: starting mds.g at 2024-04-25T01:52:15.987 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 01:52:15 smithi031 bash[70064]: 5c78e932ce5f7b5a093f68a8aacd2c75d5b59836ed19e1e7f433f32a5fbe2bbe 2024-04-25T01:52:15.987 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 01:52:15 smithi031 systemd[1]: Started Ceph mds.g for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:16.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.711+0000 7f51588ee640 1 -- 172.21.15.31:0/862153138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5154101790 msgr2=0x7f5154101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.711+0000 7f51588ee640 1 --2- 172.21.15.31:0/862153138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5154101790 0x7f5154101bb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5144009ca0 tx=0x7f514402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.711+0000 7f51588ee640 1 -- 172.21.15.31:0/862153138 shutdown_connections 2024-04-25T01:52:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.711+0000 7f51588ee640 1 --2- 172.21.15.31:0/862153138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5154103340 0x7f5154109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.711+0000 7f51588ee640 1 --2- 172.21.15.31:0/862153138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5154102980 0x7f5154102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.711+0000 7f51588ee640 1 --2- 172.21.15.31:0/862153138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5154101790 0x7f5154101bb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.711+0000 7f51588ee640 1 -- 172.21.15.31:0/862153138 >> 172.21.15.31:0/862153138 conn(0x7f51540fcd20 msgr2=0x7f51540ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.711+0000 7f51588ee640 1 -- 172.21.15.31:0/862153138 shutdown_connections 2024-04-25T01:52:16.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.711+0000 7f51588ee640 1 -- 172.21.15.31:0/862153138 wait complete. 2024-04-25T01:52:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f51588ee640 1 Processor -- start 2024-04-25T01:52:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f51588ee640 1 -- start start 2024-04-25T01:52:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f51588ee640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5154101790 0x7f51540754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f51588ee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5154102980 0x7f5154073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f51588ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5154103340 0x7f5154074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f51588ee640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f515410d2b0 con 0x7f5154102980 2024-04-25T01:52:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f51588ee640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f515410d130 con 0x7f5154103340 2024-04-25T01:52:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f51588ee640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f515410d430 con 0x7f5154101790 2024-04-25T01:52:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f5151d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5154102980 0x7f5154073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f5151d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5154102980 0x7f5154073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60026/0 (socket says 172.21.15.31:60026) 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f5151d9b640 1 -- 172.21.15.31:0/233513727 learned_addr learned my addr 172.21.15.31:0/233513727 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f515259c640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5154101790 0x7f51540754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f5151d9b640 1 -- 172.21.15.31:0/233513727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5154101790 msgr2=0x7f51540754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f5151d9b640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5154101790 0x7f51540754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f5152d9d640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5154103340 0x7f5154074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f5151d9b640 1 -- 172.21.15.31:0/233513727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5154103340 msgr2=0x7f5154074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f5151d9b640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5154103340 0x7f5154074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f5151d9b640 1 -- 172.21.15.31:0/233513727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5154074900 con 0x7f5154102980 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.712+0000 7f515259c640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5154101790 0x7f51540754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.713+0000 7f5151d9b640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5154102980 0x7f5154073b30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f514800d770 tx=0x7f514800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.713+0000 7f51437fe640 1 -- 172.21.15.31:0/233513727 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5148017070 con 0x7f5154102980 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.713+0000 7f51588ee640 1 -- 172.21.15.31:0/233513727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51541a1b60 con 0x7f5154102980 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.713+0000 7f51437fe640 1 -- 172.21.15.31:0/233513727 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51480054d0 con 0x7f5154102980 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.713+0000 7f51437fe640 1 -- 172.21.15.31:0/233513727 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5148004cf0 con 0x7f5154102980 2024-04-25T01:52:16.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.713+0000 7f51588ee640 1 -- 172.21.15.31:0/233513727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51541a2060 con 0x7f5154102980 2024-04-25T01:52:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.714+0000 7f51588ee640 1 -- 172.21.15.31:0/233513727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5154108bc0 con 0x7f5154102980 2024-04-25T01:52:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.715+0000 7f51437fe640 1 -- 172.21.15.31:0/233513727 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5148004e90 con 0x7f5154102980 2024-04-25T01:52:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.716+0000 7f51437fe640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f513c0a2b90 0x7f513c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.716+0000 7f515259c640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f513c0a2b90 0x7f513c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.716+0000 7f51437fe640 1 -- 172.21.15.31:0/233513727 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f51480ce6c0 con 0x7f5154102980 2024-04-25T01:52:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.716+0000 7f515259c640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f513c0a2b90 0x7f513c0a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5144006010 tx=0x7f5144009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.721+0000 7f51437fe640 1 -- 172.21.15.31:0/233513727 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f514809bac0 con 0x7f5154102980 2024-04-25T01:52:16.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.943+0000 7f51588ee640 1 -- 172.21.15.31:0/233513727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f515402d370 con 0x7f5154102980 2024-04-25T01:52:16.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.944+0000 7f51437fe640 1 -- 172.21.15.31:0/233513727 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8824 (secure 0 0 0) 0x7f514809b3c0 con 0x7f5154102980 2024-04-25T01:52:16.944 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:16.944 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:52:16.944 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 11 2024-04-25T01:52:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 -- 172.21.15.31:0/233513727 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f513c0a2b90 msgr2=0x7f513c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f513c0a2b90 0x7f513c0a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5144006010 tx=0x7f5144009360 comp rx=0 tx=0).stop 2024-04-25T01:52:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 -- 172.21.15.31:0/233513727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5154102980 msgr2=0x7f5154073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5154102980 0x7f5154073b30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f514800d770 tx=0x7f514800dc30 comp rx=0 tx=0).stop 2024-04-25T01:52:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 -- 172.21.15.31:0/233513727 shutdown_connections 2024-04-25T01:52:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f513c0a2b90 0x7f513c0a5050 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5154103340 0x7f5154074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5154102980 0x7f5154073b30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 --2- 172.21.15.31:0/233513727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5154101790 0x7f51540754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 -- 172.21.15.31:0/233513727 >> 172.21.15.31:0/233513727 conn(0x7f51540fcd20 msgr2=0x7f5154105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 -- 172.21.15.31:0/233513727 shutdown_connections 2024-04-25T01:52:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:16.946+0000 7f51588ee640 1 -- 172.21.15.31:0/233513727 wait complete. 2024-04-25T01:52:16.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:16 smithi031 ceph-mon[29204]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:16.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:16.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:16.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:16.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:16.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:16.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:16.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:16 smithi031 ceph-mon[29204]: Deploying daemon mds.a on smithi031 2024-04-25T01:52:16.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:16 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.31:6838/3704800465,v1:172.21.15.31:6839/3704800465] up:boot 2024-04-25T01:52:16.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:16 smithi031 ceph-mon[29204]: fsmap 10 up:standby 2024-04-25T01:52:16.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:16 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:52:17.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:16 smithi040 ceph-mon[32418]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:17.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:17.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:17.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:17.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:17.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:17.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:17.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:16 smithi040 ceph-mon[32418]: Deploying daemon mds.a on smithi031 2024-04-25T01:52:17.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:16 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.31:6838/3704800465,v1:172.21.15.31:6839/3704800465] up:boot 2024-04-25T01:52:17.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:16 smithi040 ceph-mon[32418]: fsmap 10 up:standby 2024-04-25T01:52:17.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:16 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:52:17.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:16 smithi134 ceph-mon[33098]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:17.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:17.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:17.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:17.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:17.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:17.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:17.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:16 smithi134 ceph-mon[33098]: Deploying daemon mds.a on smithi031 2024-04-25T01:52:17.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:16 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.31:6838/3704800465,v1:172.21.15.31:6839/3704800465] up:boot 2024-04-25T01:52:17.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:16 smithi134 ceph-mon[33098]: fsmap 10 up:standby 2024-04-25T01:52:17.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:16 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:52:17.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:17 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/233513727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:17.987 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 01:52:17 smithi031 systemd[1]: Starting Ceph mds.a for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:18.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:17 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/233513727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:18.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:17 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/233513727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:18.336 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 01:52:18 smithi031 podman[70413]: 2024-04-25 01:52:17.965997839 +0000 UTC m=+0.018726704 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:18.336 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 01:52:18 smithi031 podman[70413]: 2024-04-25 01:52:18.111366017 +0000 UTC m=+0.164094882 container create 375c20a58b875ed22b1c42712145e178424bb8818b1fdc231631114a246b172e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T01:52:18.336 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 01:52:18 smithi031 podman[70413]: 2024-04-25 01:52:18.337372404 +0000 UTC m=+0.390101270 container init 375c20a58b875ed22b1c42712145e178424bb8818b1fdc231631114a246b172e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-a, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T01:52:18.737 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 01:52:18 smithi031 podman[70413]: 2024-04-25 01:52:18.341059444 +0000 UTC m=+0.393788312 container start 375c20a58b875ed22b1c42712145e178424bb8818b1fdc231631114a246b172e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T01:52:18.737 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 01:52:18 smithi031 ceph-mds[70432]: -- 172.21.15.31:0/1955649504 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ad79e39a40 con 0x55ad79e0cc00 2024-04-25T01:52:18.737 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 01:52:18 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-a[70428]: starting mds.a at 2024-04-25T01:52:18.737 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 01:52:18 smithi031 bash[70413]: 375c20a58b875ed22b1c42712145e178424bb8818b1fdc231631114a246b172e 2024-04-25T01:52:18.737 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 01:52:18 smithi031 systemd[1]: Started Ceph mds.a for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:18.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.071+0000 7f09e4b89640 1 -- 172.21.15.31:0/2006708560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0103360 msgr2=0x7f09e0109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.071+0000 7f09e4b89640 1 --2- 172.21.15.31:0/2006708560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0103360 0x7f09e0109bf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009ca0 tx=0x7f09d802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.071+0000 7f09e4b89640 1 -- 172.21.15.31:0/2006708560 shutdown_connections 2024-04-25T01:52:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.071+0000 7f09e4b89640 1 --2- 172.21.15.31:0/2006708560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0103360 0x7f09e0109bf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.071+0000 7f09e4b89640 1 --2- 172.21.15.31:0/2006708560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e01029a0 0x7f09e0102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.071+0000 7f09e4b89640 1 --2- 172.21.15.31:0/2006708560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09e01017b0 0x7f09e0101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.071+0000 7f09e4b89640 1 -- 172.21.15.31:0/2006708560 >> 172.21.15.31:0/2006708560 conn(0x7f09e00fcd40 msgr2=0x7f09e00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:19.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.071+0000 7f09e4b89640 1 -- 172.21.15.31:0/2006708560 shutdown_connections 2024-04-25T01:52:19.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.071+0000 7f09e4b89640 1 -- 172.21.15.31:0/2006708560 wait complete. 2024-04-25T01:52:19.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09e4b89640 1 Processor -- start 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09e4b89640 1 -- start start 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09e4b89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e01017b0 0x7f09e00754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09e4b89640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09e01029a0 0x7f09e0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09e4b89640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0103360 0x7f09e0074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09e4b89640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f09e010d330 con 0x7f09e0103360 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09e4b89640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f09e010d1b0 con 0x7f09e01029a0 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09e4b89640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f09e010d4b0 con 0x7f09e01017b0 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09ded9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0103360 0x7f09e0074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09ded9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0103360 0x7f09e0074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60068/0 (socket says 172.21.15.31:60068) 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09ded9d640 1 -- 172.21.15.31:0/970354792 learned_addr learned my addr 172.21.15.31:0/970354792 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.072+0000 7f09ddd9b640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09e01029a0 0x7f09e0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09de59c640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e01017b0 0x7f09e00754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09ded9d640 1 -- 172.21.15.31:0/970354792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e01017b0 msgr2=0x7f09e00754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09ded9d640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e01017b0 0x7f09e00754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09ded9d640 1 -- 172.21.15.31:0/970354792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09e01029a0 msgr2=0x7f09e0073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09ded9d640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09e01029a0 0x7f09e0073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09ded9d640 1 -- 172.21.15.31:0/970354792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e0074900 con 0x7f09e0103360 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09de59c640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e01017b0 0x7f09e00754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09ddd9b640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09e01029a0 0x7f09e0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09ded9d640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0103360 0x7f09e0074070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f09d8031040 tx=0x7f09d8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09cf7fe640 1 -- 172.21.15.31:0/970354792 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8032ec0 con 0x7f09e0103360 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09cf7fe640 1 -- 172.21.15.31:0/970354792 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09d80044f0 con 0x7f09e0103360 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09cf7fe640 1 -- 172.21.15.31:0/970354792 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8030cb0 con 0x7f09e0103360 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09e4b89640 1 -- 172.21.15.31:0/970354792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e01a1aa0 con 0x7f09e0103360 2024-04-25T01:52:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.073+0000 7f09e4b89640 1 -- 172.21.15.31:0/970354792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e01a2040 con 0x7f09e0103360 2024-04-25T01:52:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.074+0000 7f09e4b89640 1 -- 172.21.15.31:0/970354792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09e0108b50 con 0x7f09e0103360 2024-04-25T01:52:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.075+0000 7f09cf7fe640 1 -- 172.21.15.31:0/970354792 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f09d8004040 con 0x7f09e0103360 2024-04-25T01:52:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.076+0000 7f09cf7fe640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09c80a2db0 0x7f09c80a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.076+0000 7f09de59c640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09c80a2db0 0x7f09c80a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.076+0000 7f09cf7fe640 1 -- 172.21.15.31:0/970354792 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f09d80f00a0 con 0x7f09e0103360 2024-04-25T01:52:19.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.076+0000 7f09de59c640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09c80a2db0 0x7f09c80a5270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f09e0102800 tx=0x7f09d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:19.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.080+0000 7f09cf7fe640 1 -- 172.21.15.31:0/970354792 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f09d80bd4a0 con 0x7f09e0103360 2024-04-25T01:52:19.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.303+0000 7f09e4b89640 1 -- 172.21.15.31:0/970354792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09e0062340 con 0x7f09e0103360 2024-04-25T01:52:19.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.304+0000 7f09cf7fe640 1 -- 172.21.15.31:0/970354792 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7f09d80bcda0 con 0x7f09e0103360 2024-04-25T01:52:19.304 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:19.304 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:52:19.304 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 12 2024-04-25T01:52:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 -- 172.21.15.31:0/970354792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09c80a2db0 msgr2=0x7f09c80a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09c80a2db0 0x7f09c80a5270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f09e0102800 tx=0x7f09d0005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 -- 172.21.15.31:0/970354792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0103360 msgr2=0x7f09e0074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0103360 0x7f09e0074070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f09d8031040 tx=0x7f09d8003ba0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 -- 172.21.15.31:0/970354792 shutdown_connections 2024-04-25T01:52:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09c80a2db0 0x7f09c80a5270 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09e0103360 0x7f09e0074070 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09e01029a0 0x7f09e0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 --2- 172.21.15.31:0/970354792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09e01017b0 0x7f09e00754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 -- 172.21.15.31:0/970354792 >> 172.21.15.31:0/970354792 conn(0x7f09e00fcd40 msgr2=0x7f09e0105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:19.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 -- 172.21.15.31:0/970354792 shutdown_connections 2024-04-25T01:52:19.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:19.306+0000 7f09e4b89640 1 -- 172.21.15.31:0/970354792 wait complete. 2024-04-25T01:52:19.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:19.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:19.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:19.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:19.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:19.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:19.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:19.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: Deploying daemon mds.b on smithi040 2024-04-25T01:52:19.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.31:6840/4277792607,v1:172.21.15.31:6841/4277792607] up:boot 2024-04-25T01:52:19.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: fsmap 11 up:standby 2024-04-25T01:52:19.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:52:19.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:19 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/970354792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:19.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:19.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:19.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:19.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:19.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:19.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:19.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:19.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: Deploying daemon mds.b on smithi040 2024-04-25T01:52:19.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.31:6840/4277792607,v1:172.21.15.31:6841/4277792607] up:boot 2024-04-25T01:52:19.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: fsmap 11 up:standby 2024-04-25T01:52:19.874 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:52:19.875 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:19 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/970354792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:19.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:19.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:19.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:19.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:19.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:19.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:19.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:19.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: Deploying daemon mds.b on smithi040 2024-04-25T01:52:19.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.31:6840/4277792607,v1:172.21.15.31:6841/4277792607] up:boot 2024-04-25T01:52:19.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: fsmap 11 up:standby 2024-04-25T01:52:19.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:52:19.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:19 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/970354792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:20.908 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:20 smithi040 systemd[1]: Starting Ceph mds.b for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:21.282 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:21 smithi040 podman[62573]: 2024-04-25 01:52:20.917698013 +0000 UTC m=+0.019347408 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:21.282 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:21 smithi040 podman[62573]: 2024-04-25 01:52:21.02088026 +0000 UTC m=+0.122529677 container create ac5fc400b968e0a3b1a9c024bf3a8fe321921ab87a59de26f8344ffd6a6fc677 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T01:52:21.282 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:21 smithi040 podman[62573]: 2024-04-25 01:52:21.257106336 +0000 UTC m=+0.358755716 container init ac5fc400b968e0a3b1a9c024bf3a8fe321921ab87a59de26f8344ffd6a6fc677 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-25T01:52:21.283 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:21 smithi040 podman[62573]: 2024-04-25 01:52:21.259744379 +0000 UTC m=+0.361393755 container start ac5fc400b968e0a3b1a9c024bf3a8fe321921ab87a59de26f8344ffd6a6fc677 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T01:52:21.283 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:21 smithi040 ceph-mds[62592]: -- 172.21.15.40:0/1611911498 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5599e8d33a40 con 0x5599e8d06800 2024-04-25T01:52:21.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.435+0000 7f09a9212640 1 -- 172.21.15.31:0/3607523901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a41017b0 msgr2=0x7f09a4101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.435+0000 7f09a9212640 1 --2- 172.21.15.31:0/3607523901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a41017b0 0x7f09a4101bd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0994009900 tx=0x7f099402a410 comp rx=0 tx=0).stop 2024-04-25T01:52:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.435+0000 7f09a9212640 1 -- 172.21.15.31:0/3607523901 shutdown_connections 2024-04-25T01:52:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.435+0000 7f09a9212640 1 --2- 172.21.15.31:0/3607523901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a4103360 0x7f09a4109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.435+0000 7f09a9212640 1 --2- 172.21.15.31:0/3607523901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09a41029a0 0x7f09a4102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.435+0000 7f09a9212640 1 --2- 172.21.15.31:0/3607523901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a41017b0 0x7f09a4101bd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.435+0000 7f09a9212640 1 -- 172.21.15.31:0/3607523901 >> 172.21.15.31:0/3607523901 conn(0x7f09a40fcd40 msgr2=0x7f09a40ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.435+0000 7f09a9212640 1 -- 172.21.15.31:0/3607523901 shutdown_connections 2024-04-25T01:52:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.435+0000 7f09a9212640 1 -- 172.21.15.31:0/3607523901 wait complete. 2024-04-25T01:52:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.436+0000 7f09a9212640 1 Processor -- start 2024-04-25T01:52:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.436+0000 7f09a9212640 1 -- start start 2024-04-25T01:52:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.436+0000 7f09a9212640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41017b0 0x7f09a41979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.436+0000 7f09a9212640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09a41029a0 0x7f09a4197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.436+0000 7f09a9212640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a4103360 0x7f09a419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.436+0000 7f09a9212640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f09a410d320 con 0x7f09a4103360 2024-04-25T01:52:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.436+0000 7f09a9212640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f09a410d1a0 con 0x7f09a41017b0 2024-04-25T01:52:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.436+0000 7f09a9212640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f09a410d4a0 con 0x7f09a41029a0 2024-04-25T01:52:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a4103360 0x7f09a419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a4103360 0x7f09a419efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60076/0 (socket says 172.21.15.31:60076) 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a359e640 1 -- 172.21.15.31:0/541489937 learned_addr learned my addr 172.21.15.31:0/541489937 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a2d9d640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41017b0 0x7f09a41979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f099a59c640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09a41029a0 0x7f09a4197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a359e640 1 -- 172.21.15.31:0/541489937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09a41029a0 msgr2=0x7f09a4197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a359e640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09a41029a0 0x7f09a4197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a359e640 1 -- 172.21.15.31:0/541489937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41017b0 msgr2=0x7f09a41979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a359e640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41017b0 0x7f09a41979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a359e640 1 -- 172.21.15.31:0/541489937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a419f610 con 0x7f09a4103360 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f099a59c640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09a41029a0 0x7f09a4197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a359e640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a4103360 0x7f09a419efa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f099c00eb10 tx=0x7f099c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a0d99640 1 -- 172.21.15.31:0/541489937 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c004b20 con 0x7f09a4103360 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a0d99640 1 -- 172.21.15.31:0/541489937 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f099c004cc0 con 0x7f09a4103360 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a9212640 1 -- 172.21.15.31:0/541489937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a419f900 con 0x7f09a4103360 2024-04-25T01:52:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a0d99640 1 -- 172.21.15.31:0/541489937 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c010690 con 0x7f09a4103360 2024-04-25T01:52:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.437+0000 7f09a9212640 1 -- 172.21.15.31:0/541489937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a419fe00 con 0x7f09a4103360 2024-04-25T01:52:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.438+0000 7f09a9212640 1 -- 172.21.15.31:0/541489937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a4108b70 con 0x7f09a4103360 2024-04-25T01:52:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.440+0000 7f09a0d99640 1 -- 172.21.15.31:0/541489937 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f099c005000 con 0x7f09a4103360 2024-04-25T01:52:21.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.440+0000 7f09a0d99640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09900a2b90 0x7f09900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.440+0000 7f09a2d9d640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09900a2b90 0x7f09900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.440+0000 7f09a0d99640 1 -- 172.21.15.31:0/541489937 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f099c0cf3b0 con 0x7f09a4103360 2024-04-25T01:52:21.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.441+0000 7f09a2d9d640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09900a2b90 0x7f09900a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0994005ab0 tx=0x7f0994005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:21.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.444+0000 7f09a0d99640 1 -- 172.21.15.31:0/541489937 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f099c053740 con 0x7f09a4103360 2024-04-25T01:52:21.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:21 smithi040 ceph-mon[32418]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:21.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:21 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:21 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.658 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:21 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:21 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:21 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.659 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b[62588]: starting mds.b at 2024-04-25T01:52:21.659 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:21 smithi040 bash[62573]: ac5fc400b968e0a3b1a9c024bf3a8fe321921ab87a59de26f8344ffd6a6fc677 2024-04-25T01:52:21.659 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:21 smithi040 systemd[1]: Started Ceph mds.b for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:21.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.671+0000 7f09a9212640 1 -- 172.21.15.31:0/541489937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09a402d170 con 0x7f09a4103360 2024-04-25T01:52:21.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.672+0000 7f09a0d99640 1 -- 172.21.15.31:0/541489937 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10476 (secure 0 0 0) 0x7f099c09c7b0 con 0x7f09a4103360 2024-04-25T01:52:21.672 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:21.672 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24503,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T01:52:21.673 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2024-04-25T01:52:21.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 -- 172.21.15.31:0/541489937 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09900a2b90 msgr2=0x7f09900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09900a2b90 0x7f09900a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0994005ab0 tx=0x7f0994005a20 comp rx=0 tx=0).stop 2024-04-25T01:52:21.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 -- 172.21.15.31:0/541489937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a4103360 msgr2=0x7f09a419efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a4103360 0x7f09a419efa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f099c00eb10 tx=0x7f099c00c2d0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 -- 172.21.15.31:0/541489937 shutdown_connections 2024-04-25T01:52:21.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f09900a2b90 0x7f09900a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09a4103360 0x7f09a419efa0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09a41029a0 0x7f09a4197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 --2- 172.21.15.31:0/541489937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a41017b0 0x7f09a41979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 -- 172.21.15.31:0/541489937 >> 172.21.15.31:0/541489937 conn(0x7f09a40fcd40 msgr2=0x7f09a4105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:21.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 -- 172.21.15.31:0/541489937 shutdown_connections 2024-04-25T01:52:21.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.675+0000 7f09a9212640 1 -- 172.21.15.31:0/541489937 wait complete. 2024-04-25T01:52:21.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:21 smithi031 ceph-mon[29204]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:21.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:21 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:21 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:21 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:21 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:21 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:52:21.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.793+0000 7f00f3038640 1 -- 172.21.15.31:0/2655162225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec102230 msgr2=0x7f00ec108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.793+0000 7f00f3038640 1 --2- 172.21.15.31:0/2655162225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec102230 0x7f00ec108ac0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009ca0 tx=0x7f00e402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.793+0000 7f00f3038640 1 -- 172.21.15.31:0/2655162225 shutdown_connections 2024-04-25T01:52:21.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.793+0000 7f00f3038640 1 --2- 172.21.15.31:0/2655162225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec102230 0x7f00ec108ac0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.793+0000 7f00f3038640 1 --2- 172.21.15.31:0/2655162225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00ec101870 0x7f00ec101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.793+0000 7f00f3038640 1 --2- 172.21.15.31:0/2655162225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ec100680 0x7f00ec100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.793+0000 7f00f3038640 1 -- 172.21.15.31:0/2655162225 >> 172.21.15.31:0/2655162225 conn(0x7f00ec0fbbf0 msgr2=0x7f00ec0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.793+0000 7f00f3038640 1 -- 172.21.15.31:0/2655162225 shutdown_connections 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.793+0000 7f00f3038640 1 -- 172.21.15.31:0/2655162225 wait complete. 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00f3038640 1 Processor -- start 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00f3038640 1 -- start start 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00f3038640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00ec100680 0x7f00ec1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00f3038640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec101870 0x7f00ec197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00f3038640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ec102230 0x7f00ec19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00f3038640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec10c1b0 con 0x7f00ec101870 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00f3038640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec10c030 con 0x7f00ec102230 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00f3038640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f00ec10c330 con 0x7f00ec100680 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00ebfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec101870 0x7f00ec197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00ebfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec101870 0x7f00ec197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60096/0 (socket says 172.21.15.31:60096) 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00ebfff640 1 -- 172.21.15.31:0/4273483147 learned_addr learned my addr 172.21.15.31:0/4273483147 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00f0dd4640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00ec100680 0x7f00ec1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00f15d5640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ec102230 0x7f00ec19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00ebfff640 1 -- 172.21.15.31:0/4273483147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00ec100680 msgr2=0x7f00ec1979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00ebfff640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00ec100680 0x7f00ec1979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00ebfff640 1 -- 172.21.15.31:0/4273483147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ec102230 msgr2=0x7f00ec19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00ebfff640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ec102230 0x7f00ec19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.794+0000 7f00ebfff640 1 -- 172.21.15.31:0/4273483147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ec19f5d0 con 0x7f00ec101870 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.795+0000 7f00ebfff640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec101870 0x7f00ec197ef0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f00d800e9f0 tx=0x7f00d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.795+0000 7f00e9ffb640 1 -- 172.21.15.31:0/4273483147 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d800cd80 con 0x7f00ec101870 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.795+0000 7f00f3038640 1 -- 172.21.15.31:0/4273483147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00ec19f8c0 con 0x7f00ec101870 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.795+0000 7f00e9ffb640 1 -- 172.21.15.31:0/4273483147 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00d80054d0 con 0x7f00ec101870 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.795+0000 7f00e9ffb640 1 -- 172.21.15.31:0/4273483147 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8010490 con 0x7f00ec101870 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.795+0000 7f00f3038640 1 -- 172.21.15.31:0/4273483147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00ec19fe10 con 0x7f00ec101870 2024-04-25T01:52:21.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.796+0000 7f00f3038640 1 -- 172.21.15.31:0/4273483147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00ec1a00a0 con 0x7f00ec101870 2024-04-25T01:52:21.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.797+0000 7f00e9ffb640 1 -- 172.21.15.31:0/4273483147 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f00d8005020 con 0x7f00ec101870 2024-04-25T01:52:21.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.797+0000 7f00e9ffb640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f00dc0a2b90 0x7f00dc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.798+0000 7f00e9ffb640 1 -- 172.21.15.31:0/4273483147 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f00d80cef00 con 0x7f00ec101870 2024-04-25T01:52:21.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.798+0000 7f00f0dd4640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f00dc0a2b90 0x7f00dc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.798+0000 7f00f0dd4640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f00dc0a2b90 0x7f00dc0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f00ec1016d0 tx=0x7f00e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:21.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.802+0000 7f00e9ffb640 1 -- 172.21.15.31:0/4273483147 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f00d809c300 con 0x7f00ec101870 2024-04-25T01:52:21.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:21 smithi134 ceph-mon[33098]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:21.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:21 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:21 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:21 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:21 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:21 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:21.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.990+0000 7f00f3038640 1 -- 172.21.15.31:0/4273483147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00ec1a0270 con 0x7f00ec101870 2024-04-25T01:52:21.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.991+0000 7f00e9ffb640 1 -- 172.21.15.31:0/4273483147 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18384 (secure 0 0 0) 0x7f00d809bc00 con 0x7f00ec101870 2024-04-25T01:52:21.991 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:21.992 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:51:38.978864+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:52:21.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.993+0000 7f00f3038640 1 -- 172.21.15.31:0/4273483147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f00dc0a2b90 msgr2=0x7f00dc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.993+0000 7f00f3038640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f00dc0a2b90 0x7f00dc0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f00ec1016d0 tx=0x7f00e0005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.993+0000 7f00f3038640 1 -- 172.21.15.31:0/4273483147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec101870 msgr2=0x7f00ec197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.993+0000 7f00f3038640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec101870 0x7f00ec197ef0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f00d800e9f0 tx=0x7f00d800edf0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.994+0000 7f00f3038640 1 -- 172.21.15.31:0/4273483147 shutdown_connections 2024-04-25T01:52:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.994+0000 7f00f3038640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f00dc0a2b90 0x7f00dc0a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.994+0000 7f00f3038640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00ec102230 0x7f00ec19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.994+0000 7f00f3038640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec101870 0x7f00ec197ef0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.994+0000 7f00f3038640 1 --2- 172.21.15.31:0/4273483147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00ec100680 0x7f00ec1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.994+0000 7f00f3038640 1 -- 172.21.15.31:0/4273483147 >> 172.21.15.31:0/4273483147 conn(0x7f00ec0fbbf0 msgr2=0x7f00ec0689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.994+0000 7f00f3038640 1 -- 172.21.15.31:0/4273483147 shutdown_connections 2024-04-25T01:52:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:21.994+0000 7f00f3038640 1 -- 172.21.15.31:0/4273483147 wait complete. 2024-04-25T01:52:22.005 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:52:22.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:52:22.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.112+0000 7f6c559e7640 1 -- 172.21.15.31:0/4172274228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c501006a0 msgr2=0x7f6c50100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.112+0000 7f6c559e7640 1 --2- 172.21.15.31:0/4172274228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c501006a0 0x7f6c50100ac0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6c40009900 tx=0x7f6c4002a410 comp rx=0 tx=0).stop 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.112+0000 7f6c559e7640 1 -- 172.21.15.31:0/4172274228 shutdown_connections 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 --2- 172.21.15.31:0/4172274228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c50102250 0x7f6c50108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 --2- 172.21.15.31:0/4172274228 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c50101890 0x7f6c50101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 --2- 172.21.15.31:0/4172274228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c501006a0 0x7f6c50100ac0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 -- 172.21.15.31:0/4172274228 >> 172.21.15.31:0/4172274228 conn(0x7f6c500fbc30 msgr2=0x7f6c500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 -- 172.21.15.31:0/4172274228 shutdown_connections 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 -- 172.21.15.31:0/4172274228 wait complete. 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 Processor -- start 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 -- start start 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c501006a0 0x7f6c5006a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c50101890 0x7f6c5019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c50102250 0x7f6c5019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5010c1d0 con 0x7f6c50102250 2024-04-25T01:52:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5010c050 con 0x7f6c501006a0 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.113+0000 7f6c559e7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5010c350 con 0x7f6c50101890 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c4f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c50102250 0x7f6c5019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c4effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c501006a0 0x7f6c5006a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c4f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c50102250 0x7f6c5019edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60122/0 (socket says 172.21.15.31:60122) 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c4f7fe640 1 -- 172.21.15.31:0/4113318427 learned_addr learned my addr 172.21.15.31:0/4113318427 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c47fff640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c50101890 0x7f6c5019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c4f7fe640 1 -- 172.21.15.31:0/4113318427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c50101890 msgr2=0x7f6c5019c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c4f7fe640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c50101890 0x7f6c5019c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c4f7fe640 1 -- 172.21.15.31:0/4113318427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c501006a0 msgr2=0x7f6c5006a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c4f7fe640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c501006a0 0x7f6c5006a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c4f7fe640 1 -- 172.21.15.31:0/4113318427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c5019f6e0 con 0x7f6c50102250 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c4f7fe640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c50102250 0x7f6c5019edd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800eb10 tx=0x7f6c4800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c549e5640 1 -- 172.21.15.31:0/4113318427 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c48004b20 con 0x7f6c50102250 2024-04-25T01:52:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c549e5640 1 -- 172.21.15.31:0/4113318427 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c48004cc0 con 0x7f6c50102250 2024-04-25T01:52:22.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c559e7640 1 -- 172.21.15.31:0/4113318427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c5019f970 con 0x7f6c50102250 2024-04-25T01:52:22.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c549e5640 1 -- 172.21.15.31:0/4113318427 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c48010690 con 0x7f6c50102250 2024-04-25T01:52:22.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.114+0000 7f6c559e7640 1 -- 172.21.15.31:0/4113318427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c5019fe70 con 0x7f6c50102250 2024-04-25T01:52:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.115+0000 7f6c559e7640 1 -- 172.21.15.31:0/4113318427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c50107ba0 con 0x7f6c50102250 2024-04-25T01:52:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.117+0000 7f6c549e5640 1 -- 172.21.15.31:0/4113318427 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6c48005000 con 0x7f6c50102250 2024-04-25T01:52:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.117+0000 7f6c549e5640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c3c0a2b90 0x7f6c3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.117+0000 7f6c549e5640 1 -- 172.21.15.31:0/4113318427 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f6c480cf320 con 0x7f6c50102250 2024-04-25T01:52:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.117+0000 7f6c4effd640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c3c0a2b90 0x7f6c3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.118+0000 7f6c4effd640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c3c0a2b90 0x7f6c3c0a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6c40000c00 tx=0x7f6c400314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:22.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.123+0000 7f6c549e5640 1 -- 172.21.15.31:0/4113318427 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6c48053740 con 0x7f6c50102250 2024-04-25T01:52:22.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.335+0000 7f6c559e7640 1 -- 172.21.15.31:0/4113318427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6c5002d3c0 con 0x7f6c50102250 2024-04-25T01:52:22.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.336+0000 7f6c549e5640 1 -- 172.21.15.31:0/4113318427 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f6c4809c720 con 0x7f6c50102250 2024-04-25T01:52:22.335 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T01:52:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 -- 172.21.15.31:0/4113318427 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c3c0a2b90 msgr2=0x7f6c3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c3c0a2b90 0x7f6c3c0a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6c40000c00 tx=0x7f6c400314a0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 -- 172.21.15.31:0/4113318427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c50102250 msgr2=0x7f6c5019edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c50102250 0x7f6c5019edd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800eb10 tx=0x7f6c4800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 -- 172.21.15.31:0/4113318427 shutdown_connections 2024-04-25T01:52:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6c3c0a2b90 0x7f6c3c0a5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c50102250 0x7f6c5019edd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c50101890 0x7f6c5019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 --2- 172.21.15.31:0/4113318427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c501006a0 0x7f6c5006a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 -- 172.21.15.31:0/4113318427 >> 172.21.15.31:0/4113318427 conn(0x7f6c500fbc30 msgr2=0x7f6c50104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 -- 172.21.15.31:0/4113318427 shutdown_connections 2024-04-25T01:52:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.338+0000 7f6c559e7640 1 -- 172.21.15.31:0/4113318427 wait complete. 2024-04-25T01:52:22.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T01:52:22.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.451+0000 7fb997d10640 1 -- 172.21.15.31:0/983068716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9901006a0 msgr2=0x7fb990100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.451+0000 7fb997d10640 1 --2- 172.21.15.31:0/983068716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9901006a0 0x7fb990100ac0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009900 tx=0x7fb98c02a410 comp rx=0 tx=0).stop 2024-04-25T01:52:22.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.451+0000 7fb997d10640 1 -- 172.21.15.31:0/983068716 shutdown_connections 2024-04-25T01:52:22.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.451+0000 7fb997d10640 1 --2- 172.21.15.31:0/983068716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb990102250 0x7fb990108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.451+0000 7fb997d10640 1 --2- 172.21.15.31:0/983068716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb990101890 0x7fb990101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.451+0000 7fb997d10640 1 --2- 172.21.15.31:0/983068716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9901006a0 0x7fb990100ac0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.451+0000 7fb997d10640 1 -- 172.21.15.31:0/983068716 >> 172.21.15.31:0/983068716 conn(0x7fb9900fbc30 msgr2=0x7fb9900fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:22.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.451+0000 7fb997d10640 1 -- 172.21.15.31:0/983068716 shutdown_connections 2024-04-25T01:52:22.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.451+0000 7fb997d10640 1 -- 172.21.15.31:0/983068716 wait complete. 2024-04-25T01:52:22.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb997d10640 1 Processor -- start 2024-04-25T01:52:22.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb997d10640 1 -- start start 2024-04-25T01:52:22.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb997d10640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9901006a0 0x7fb99006a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb997d10640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb990101890 0x7fb99019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb997d10640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb990102250 0x7fb99019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb997d10640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb99010c1d0 con 0x7fb990102250 2024-04-25T01:52:22.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb997d10640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb99010c050 con 0x7fb990101890 2024-04-25T01:52:22.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb997d10640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb99010c350 con 0x7fb9901006a0 2024-04-25T01:52:22.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb9962ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb990102250 0x7fb99019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb9962ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb990102250 0x7fb99019ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60154/0 (socket says 172.21.15.31:60154) 2024-04-25T01:52:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb9962ad640 1 -- 172.21.15.31:0/2871152133 learned_addr learned my addr 172.21.15.31:0/2871152133 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.452+0000 7fb995aac640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9901006a0 0x7fb99006a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb9952ab640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb990101890 0x7fb99019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb9962ad640 1 -- 172.21.15.31:0/2871152133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9901006a0 msgr2=0x7fb99006a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb9962ad640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9901006a0 0x7fb99006a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb9962ad640 1 -- 172.21.15.31:0/2871152133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb990101890 msgr2=0x7fb99019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb9962ad640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb990101890 0x7fb99019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb9962ad640 1 -- 172.21.15.31:0/2871152133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99019f690 con 0x7fb990102250 2024-04-25T01:52:22.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb9962ad640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb990102250 0x7fb99019ed80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb98800efd0 tx=0x7fb98800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:22.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb986ffd640 1 -- 172.21.15.31:0/2871152133 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb988003b70 con 0x7fb990102250 2024-04-25T01:52:22.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb997d10640 1 -- 172.21.15.31:0/2871152133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb99019f920 con 0x7fb990102250 2024-04-25T01:52:22.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb986ffd640 1 -- 172.21.15.31:0/2871152133 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb988003d10 con 0x7fb990102250 2024-04-25T01:52:22.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb986ffd640 1 -- 172.21.15.31:0/2871152133 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb988010980 con 0x7fb990102250 2024-04-25T01:52:22.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.453+0000 7fb997d10640 1 -- 172.21.15.31:0/2871152133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb99019fe20 con 0x7fb990102250 2024-04-25T01:52:22.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.454+0000 7fb997d10640 1 -- 172.21.15.31:0/2871152133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb990107b30 con 0x7fb990102250 2024-04-25T01:52:22.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.455+0000 7fb986ffd640 1 -- 172.21.15.31:0/2871152133 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb988004020 con 0x7fb990102250 2024-04-25T01:52:22.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.456+0000 7fb986ffd640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb97c0a2b90 0x7fb97c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.456+0000 7fb995aac640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb97c0a2b90 0x7fb97c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.456+0000 7fb986ffd640 1 -- 172.21.15.31:0/2871152133 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fb9880cf090 con 0x7fb990102250 2024-04-25T01:52:22.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.456+0000 7fb995aac640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb97c0a2b90 0x7fb97c0a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb98c004770 tx=0x7fb98c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:22.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.460+0000 7fb986ffd640 1 -- 172.21.15.31:0/2871152133 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb98809c490 con 0x7fb990102250 2024-04-25T01:52:22.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:22 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] up:boot 2024-04-25T01:52:22.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:22 smithi031 ceph-mon[29204]: fsmap 12 up:standby 2024-04-25T01:52:22.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:22 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:52:22.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:22 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.652 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:22 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/541489937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:22 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/4273483147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:22 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/4113318427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:52:22.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.657+0000 7fb997d10640 1 -- 172.21.15.31:0/2871152133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb99002d3c0 con 0x7fb990102250 2024-04-25T01:52:22.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.657+0000 7fb986ffd640 1 -- 172.21.15.31:0/2871152133 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) ==== 84+0+65 (secure 0 0 0) 0x7fb98809bd90 con 0x7fb990102250 2024-04-25T01:52:22.657 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:22.657 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-04-25T01:52:22.657 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T01:52:22.657 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-04-25T01:52:22.657 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-04-25T01:52:22.658 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T01:52:22.658 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-04-25T01:52:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.659+0000 7fb997d10640 1 -- 172.21.15.31:0/2871152133 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb97c0a2b90 msgr2=0x7fb97c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.659+0000 7fb997d10640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb97c0a2b90 0x7fb97c0a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb98c004770 tx=0x7fb98c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.659+0000 7fb997d10640 1 -- 172.21.15.31:0/2871152133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb990102250 msgr2=0x7fb99019ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.660+0000 7fb997d10640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb990102250 0x7fb99019ed80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb98800efd0 tx=0x7fb98800c6a0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.660+0000 7fb997d10640 1 -- 172.21.15.31:0/2871152133 shutdown_connections 2024-04-25T01:52:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.660+0000 7fb997d10640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb97c0a2b90 0x7fb97c0a5050 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.660+0000 7fb997d10640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb990102250 0x7fb99019ed80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.660+0000 7fb997d10640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb990101890 0x7fb99019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.660+0000 7fb997d10640 1 --2- 172.21.15.31:0/2871152133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9901006a0 0x7fb99006a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.660+0000 7fb997d10640 1 -- 172.21.15.31:0/2871152133 >> 172.21.15.31:0/2871152133 conn(0x7fb9900fbc30 msgr2=0x7fb990104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:22.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.660+0000 7fb997d10640 1 -- 172.21.15.31:0/2871152133 shutdown_connections 2024-04-25T01:52:22.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.660+0000 7fb997d10640 1 -- 172.21.15.31:0/2871152133 wait complete. 2024-04-25T01:52:22.671 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T01:52:22.671 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T01:52:22.726 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:22 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] up:boot 2024-04-25T01:52:22.727 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:22 smithi134 ceph-mon[33098]: fsmap 12 up:standby 2024-04-25T01:52:22.727 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:22 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:52:22.727 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:22 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.727 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:22 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/541489937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.727 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:22 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/4273483147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.727 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:22 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/4113318427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:52:22.776 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:22 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] up:boot 2024-04-25T01:52:22.777 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:22 smithi040 ceph-mon[32418]: fsmap 12 up:standby 2024-04-25T01:52:22.777 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:22 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:52:22.777 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:22 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.777 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:22 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/541489937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.777 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:22 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/4273483147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.777 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:22 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/4113318427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:52:22.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.778+0000 7fa19e6a3640 1 -- 172.21.15.31:0/4069710720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa198101790 msgr2=0x7fa198101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.778+0000 7fa19e6a3640 1 --2- 172.21.15.31:0/4069710720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa198101790 0x7fa198101bb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa188009900 tx=0x7fa18802a410 comp rx=0 tx=0).stop 2024-04-25T01:52:22.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.778+0000 7fa19e6a3640 1 -- 172.21.15.31:0/4069710720 shutdown_connections 2024-04-25T01:52:22.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.778+0000 7fa19e6a3640 1 --2- 172.21.15.31:0/4069710720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198103340 0x7fa198109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.778+0000 7fa19e6a3640 1 --2- 172.21.15.31:0/4069710720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa198102980 0x7fa198102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.778+0000 7fa19e6a3640 1 --2- 172.21.15.31:0/4069710720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa198101790 0x7fa198101bb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.778+0000 7fa19e6a3640 1 -- 172.21.15.31:0/4069710720 >> 172.21.15.31:0/4069710720 conn(0x7fa1980fcd20 msgr2=0x7fa1980ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:22.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.778+0000 7fa19e6a3640 1 -- 172.21.15.31:0/4069710720 shutdown_connections 2024-04-25T01:52:22.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.778+0000 7fa19e6a3640 1 -- 172.21.15.31:0/4069710720 wait complete. 2024-04-25T01:52:22.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.778+0000 7fa19e6a3640 1 Processor -- start 2024-04-25T01:52:22.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.778+0000 7fa19e6a3640 1 -- start start 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa19e6a3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa198101790 0x7fa198197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa19e6a3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102980 0x7fa198197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa19e6a3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa198103340 0x7fa19819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa19e6a3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa19810d2b0 con 0x7fa198101790 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa19e6a3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa19810d130 con 0x7fa198102980 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa19e6a3640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa19810d430 con 0x7fa198103340 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa197fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa198101790 0x7fa198197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa197fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa198101790 0x7fa198197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44580/0 (socket says 172.21.15.31:44580) 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa197fff640 1 -- 172.21.15.31:0/1056778472 learned_addr learned my addr 172.21.15.31:0/1056778472 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa19cc40640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa198103340 0x7fa19819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa1977fe640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102980 0x7fa198197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa197fff640 1 -- 172.21.15.31:0/1056778472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa198103340 msgr2=0x7fa19819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa197fff640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa198103340 0x7fa19819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa197fff640 1 -- 172.21.15.31:0/1056778472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102980 msgr2=0x7fa198197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:22.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa197fff640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102980 0x7fa198197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:22.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa197fff640 1 -- 172.21.15.31:0/1056778472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19819f660 con 0x7fa198101790 2024-04-25T01:52:22.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa19cc40640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa198103340 0x7fa19819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:22.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.779+0000 7fa197fff640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa198101790 0x7fa198197a40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa1880314e0 tx=0x7fa188031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:22.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.780+0000 7fa1957fa640 1 -- 172.21.15.31:0/1056778472 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18803f030 con 0x7fa198101790 2024-04-25T01:52:22.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.780+0000 7fa19e6a3640 1 -- 172.21.15.31:0/1056778472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa19819f8f0 con 0x7fa198101790 2024-04-25T01:52:22.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.780+0000 7fa1957fa640 1 -- 172.21.15.31:0/1056778472 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1880044b0 con 0x7fa198101790 2024-04-25T01:52:22.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.780+0000 7fa1957fa640 1 -- 172.21.15.31:0/1056778472 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188035070 con 0x7fa198101790 2024-04-25T01:52:22.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.780+0000 7fa19e6a3640 1 -- 172.21.15.31:0/1056778472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa19819fe30 con 0x7fa198101790 2024-04-25T01:52:22.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.781+0000 7fa19e6a3640 1 -- 172.21.15.31:0/1056778472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa198108bc0 con 0x7fa198101790 2024-04-25T01:52:22.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.782+0000 7fa1957fa640 1 -- 172.21.15.31:0/1056778472 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa188042030 con 0x7fa198101790 2024-04-25T01:52:22.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.783+0000 7fa1957fa640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa1800a2b90 0x7fa1800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:22.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.783+0000 7fa1977fe640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa1800a2b90 0x7fa1800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:22.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.783+0000 7fa1957fa640 1 -- 172.21.15.31:0/1056778472 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fa1880ef150 con 0x7fa198101790 2024-04-25T01:52:22.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.783+0000 7fa1977fe640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa1800a2b90 0x7fa1800a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa198198f50 tx=0x7fa18c00a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:22.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.787+0000 7fa1957fa640 1 -- 172.21.15.31:0/1056778472 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa1880bbe60 con 0x7fa198101790 2024-04-25T01:52:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:22.975+0000 7fa19e6a3640 1 -- 172.21.15.31:0/1056778472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa198062340 con 0x7fa198101790 2024-04-25T01:52:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-04-25T01:52:23.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.492+0000 7fa1957fa640 1 -- 172.21.15.31:0/1056778472 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) ==== 138+0+0 (secure 0 0 0) 0x7fa1880c0160 con 0x7fa198101790 2024-04-25T01:52:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.495+0000 7fa19e6a3640 1 -- 172.21.15.31:0/1056778472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa1800a2b90 msgr2=0x7fa1800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.495+0000 7fa19e6a3640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa1800a2b90 0x7fa1800a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa198198f50 tx=0x7fa18c00a250 comp rx=0 tx=0).stop 2024-04-25T01:52:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.495+0000 7fa19e6a3640 1 -- 172.21.15.31:0/1056778472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa198101790 msgr2=0x7fa198197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.495+0000 7fa19e6a3640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa198101790 0x7fa198197a40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa1880314e0 tx=0x7fa188031a30 comp rx=0 tx=0).stop 2024-04-25T01:52:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.495+0000 7fa19e6a3640 1 -- 172.21.15.31:0/1056778472 shutdown_connections 2024-04-25T01:52:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.495+0000 7fa19e6a3640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa1800a2b90 0x7fa1800a5050 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.495+0000 7fa19e6a3640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa198103340 0x7fa19819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.495+0000 7fa19e6a3640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102980 0x7fa198197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.495+0000 7fa19e6a3640 1 --2- 172.21.15.31:0/1056778472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa198101790 0x7fa198197a40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.495+0000 7fa19e6a3640 1 -- 172.21.15.31:0/1056778472 >> 172.21.15.31:0/1056778472 conn(0x7fa1980fcd20 msgr2=0x7fa198105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.496+0000 7fa19e6a3640 1 -- 172.21.15.31:0/1056778472 shutdown_connections 2024-04-25T01:52:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.496+0000 7fa19e6a3640 1 -- 172.21.15.31:0/1056778472 wait complete. 2024-04-25T01:52:23.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T01:52:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.618+0000 7f39d7b77640 1 -- 172.21.15.31:0/2180914666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0101890 msgr2=0x7f39d0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.618+0000 7f39d7b77640 1 --2- 172.21.15.31:0/2180914666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0101890 0x7f39d0101d10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009900 tx=0x7f39c002a410 comp rx=0 tx=0).stop 2024-04-25T01:52:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.618+0000 7f39d7b77640 1 -- 172.21.15.31:0/2180914666 shutdown_connections 2024-04-25T01:52:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.618+0000 7f39d7b77640 1 --2- 172.21.15.31:0/2180914666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39d0102250 0x7f39d0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.618+0000 7f39d7b77640 1 --2- 172.21.15.31:0/2180914666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0101890 0x7f39d0101d10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.618+0000 7f39d7b77640 1 --2- 172.21.15.31:0/2180914666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39d01006a0 0x7f39d0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.618+0000 7f39d7b77640 1 -- 172.21.15.31:0/2180914666 >> 172.21.15.31:0/2180914666 conn(0x7f39d00fbc30 msgr2=0x7f39d00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.618+0000 7f39d7b77640 1 -- 172.21.15.31:0/2180914666 shutdown_connections 2024-04-25T01:52:23.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.618+0000 7f39d7b77640 1 -- 172.21.15.31:0/2180914666 wait complete. 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d7b77640 1 Processor -- start 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d7b77640 1 -- start start 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d7b77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39d01006a0 0x7f39d01968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d7b77640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39d0101890 0x7f39d0196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d7b77640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102250 0x7f39d019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d7b77640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39d010c1d0 con 0x7f39d0102250 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d7b77640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f39d010c050 con 0x7f39d01006a0 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d7b77640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f39d010c350 con 0x7f39d0101890 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d6114640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102250 0x7f39d019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d6114640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102250 0x7f39d019de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44592/0 (socket says 172.21.15.31:44592) 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d6114640 1 -- 172.21.15.31:0/653541064 learned_addr learned my addr 172.21.15.31:0/653541064 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d5112640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39d0101890 0x7f39d0196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.619+0000 7f39d5913640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39d01006a0 0x7f39d01968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:23.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.620+0000 7f39d6114640 1 -- 172.21.15.31:0/653541064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39d0101890 msgr2=0x7f39d0196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.620+0000 7f39d6114640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39d0101890 0x7f39d0196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.620+0000 7f39d6114640 1 -- 172.21.15.31:0/653541064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39d01006a0 msgr2=0x7f39d01968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.620+0000 7f39d6114640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39d01006a0 0x7f39d01968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.620+0000 7f39d6114640 1 -- 172.21.15.31:0/653541064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d019e4f0 con 0x7f39d0102250 2024-04-25T01:52:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.620+0000 7f39d6114640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102250 0x7f39d019de80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f39c800eca0 tx=0x7f39c800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.620+0000 7f39c6ffd640 1 -- 172.21.15.31:0/653541064 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c8004b20 con 0x7f39d0102250 2024-04-25T01:52:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.620+0000 7f39c6ffd640 1 -- 172.21.15.31:0/653541064 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39c8004cc0 con 0x7f39d0102250 2024-04-25T01:52:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.620+0000 7f39d7b77640 1 -- 172.21.15.31:0/653541064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d0069da0 con 0x7f39d0102250 2024-04-25T01:52:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.620+0000 7f39c6ffd640 1 -- 172.21.15.31:0/653541064 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c8017670 con 0x7f39d0102250 2024-04-25T01:52:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.620+0000 7f39d7b77640 1 -- 172.21.15.31:0/653541064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d006a2c0 con 0x7f39d0102250 2024-04-25T01:52:23.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.621+0000 7f39d7b77640 1 -- 172.21.15.31:0/653541064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d00689b0 con 0x7f39d0102250 2024-04-25T01:52:23.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.622+0000 7f39c6ffd640 1 -- 172.21.15.31:0/653541064 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f39c8005020 con 0x7f39d0102250 2024-04-25T01:52:23.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.623+0000 7f39c6ffd640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39bc0a2b90 0x7f39bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.623+0000 7f39d5913640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39bc0a2b90 0x7f39bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.623+0000 7f39c6ffd640 1 -- 172.21.15.31:0/653541064 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7f39c80d06f0 con 0x7f39d0102250 2024-04-25T01:52:23.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.624+0000 7f39d5913640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39bc0a2b90 0x7f39bc0a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0095f0 tx=0x7f39cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:23.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.628+0000 7f39c6ffd640 1 -- 172.21.15.31:0/653541064 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f39c809d330 con 0x7f39d0102250 2024-04-25T01:52:23.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:23.818+0000 7f39d7b77640 1 -- 172.21.15.31:0/653541064 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f39d019e780 con 0x7f39d0102250 2024-04-25T01:52:23.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:23 smithi040 ceph-mon[32418]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:23.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:23 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2871152133' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:52:23.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:23 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1056778472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:52:23.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:23 smithi134 ceph-mon[33098]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:23.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:23 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2871152133' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:52:23.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:23 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1056778472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:52:23.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:23 smithi031 ceph-mon[29204]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:23.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:23 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2871152133' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:52:23.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:23 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1056778472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:52:23.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:23.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:24.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.494+0000 7f39c6ffd640 1 -- 172.21.15.31:0/653541064 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v85) ==== 170+0+0 (secure 0 0 0) 0x7f39c80a1630 con 0x7f39d0102250 2024-04-25T01:52:24.495 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-04-25T01:52:24.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.505+0000 7f39d7b77640 1 -- 172.21.15.31:0/653541064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39bc0a2b90 msgr2=0x7f39bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:24.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.505+0000 7f39d7b77640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39bc0a2b90 0x7f39bc0a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0095f0 tx=0x7f39cc005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:24.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.505+0000 7f39d7b77640 1 -- 172.21.15.31:0/653541064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102250 msgr2=0x7f39d019de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:24.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.505+0000 7f39d7b77640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102250 0x7f39d019de80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f39c800eca0 tx=0x7f39c800b690 comp rx=0 tx=0).stop 2024-04-25T01:52:24.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.507+0000 7f39d7b77640 1 -- 172.21.15.31:0/653541064 shutdown_connections 2024-04-25T01:52:24.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.507+0000 7f39d7b77640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39bc0a2b90 0x7f39bc0a5050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.507+0000 7f39d7b77640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102250 0x7f39d019de80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.507+0000 7f39d7b77640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39d0101890 0x7f39d0196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.507+0000 7f39d7b77640 1 --2- 172.21.15.31:0/653541064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39d01006a0 0x7f39d01968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.507+0000 7f39d7b77640 1 -- 172.21.15.31:0/653541064 >> 172.21.15.31:0/653541064 conn(0x7f39d00fbc30 msgr2=0x7f39d0104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:24.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.507+0000 7f39d7b77640 1 -- 172.21.15.31:0/653541064 shutdown_connections 2024-04-25T01:52:24.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.507+0000 7f39d7b77640 1 -- 172.21.15.31:0/653541064 wait complete. 2024-04-25T01:52:24.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T01:52:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.635+0000 7f659719e640 1 -- 172.21.15.31:0/1766201482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6590102260 msgr2=0x7f6590104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:24.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.635+0000 7f659719e640 1 --2- 172.21.15.31:0/1766201482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6590102260 0x7f6590104650 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6584009900 tx=0x7f658402a410 comp rx=0 tx=0).stop 2024-04-25T01:52:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.635+0000 7f659719e640 1 -- 172.21.15.31:0/1766201482 shutdown_connections 2024-04-25T01:52:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.635+0000 7f659719e640 1 --2- 172.21.15.31:0/1766201482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65901074c0 0x7f65901098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.635+0000 7f659719e640 1 --2- 172.21.15.31:0/1766201482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6590104b90 0x7f6590106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.635+0000 7f659719e640 1 --2- 172.21.15.31:0/1766201482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6590102260 0x7f6590104650 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.635+0000 7f659719e640 1 -- 172.21.15.31:0/1766201482 >> 172.21.15.31:0/1766201482 conn(0x7f65900fbbf0 msgr2=0x7f65900fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.635+0000 7f659719e640 1 -- 172.21.15.31:0/1766201482 shutdown_connections 2024-04-25T01:52:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.635+0000 7f659719e640 1 -- 172.21.15.31:0/1766201482 wait complete. 2024-04-25T01:52:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659719e640 1 Processor -- start 2024-04-25T01:52:24.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659719e640 1 -- start start 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659719e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6590102260 0x7f6590101bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659719e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6590104b90 0x7f6590100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659719e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65901074c0 0x7f6590100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659719e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f659010d2a0 con 0x7f65901074c0 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659719e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f659010d120 con 0x7f6590104b90 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659719e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f659010d420 con 0x7f6590102260 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659573b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65901074c0 0x7f6590100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659573b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65901074c0 0x7f6590100750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44600/0 (socket says 172.21.15.31:44600) 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659573b640 1 -- 172.21.15.31:0/621655006 learned_addr learned my addr 172.21.15.31:0/621655006 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f658ffff640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6590104b90 0x7f6590100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f6594f3a640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6590102260 0x7f6590101bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:24.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659573b640 1 -- 172.21.15.31:0/621655006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6590102260 msgr2=0x7f6590101bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659573b640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6590102260 0x7f6590101bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659573b640 1 -- 172.21.15.31:0/621655006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6590104b90 msgr2=0x7f6590100210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659573b640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6590104b90 0x7f6590100210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.636+0000 7f659573b640 1 -- 172.21.15.31:0/621655006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6590100fe0 con 0x7f65901074c0 2024-04-25T01:52:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.637+0000 7f659573b640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65901074c0 0x7f6590100750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6588009fd0 tx=0x7f658800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.637+0000 7f658dffb640 1 -- 172.21.15.31:0/621655006 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6588017070 con 0x7f65901074c0 2024-04-25T01:52:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.637+0000 7f659719e640 1 -- 172.21.15.31:0/621655006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65901a1b80 con 0x7f65901074c0 2024-04-25T01:52:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.637+0000 7f658dffb640 1 -- 172.21.15.31:0/621655006 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6588003e20 con 0x7f65901074c0 2024-04-25T01:52:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.637+0000 7f658dffb640 1 -- 172.21.15.31:0/621655006 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65880106c0 con 0x7f65901074c0 2024-04-25T01:52:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.637+0000 7f659719e640 1 -- 172.21.15.31:0/621655006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65901a2080 con 0x7f65901074c0 2024-04-25T01:52:24.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.638+0000 7f659719e640 1 -- 172.21.15.31:0/621655006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65900fd6e0 con 0x7f65901074c0 2024-04-25T01:52:24.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.639+0000 7f658dffb640 1 -- 172.21.15.31:0/621655006 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6588003970 con 0x7f65901074c0 2024-04-25T01:52:24.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.640+0000 7f658dffb640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65800a2b90 0x7f65800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:24.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.640+0000 7f6594f3a640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65800a2b90 0x7f65800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:24.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.640+0000 7f658dffb640 1 -- 172.21.15.31:0/621655006 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7879+0+0 (secure 0 0 0) 0x7f65880ceb80 con 0x7f65901074c0 2024-04-25T01:52:24.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.641+0000 7f6594f3a640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65800a2b90 0x7f65800a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6584006010 tx=0x7f6584000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:24.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.644+0000 7f658dffb640 1 -- 172.21.15.31:0/621655006 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f65880047e0 con 0x7f65901074c0 2024-04-25T01:52:24.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:24.896+0000 7f659719e640 1 -- 172.21.15.31:0/621655006 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f65900623b0 con 0x7f65901074c0 2024-04-25T01:52:24.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:24 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1056778472' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:52:24.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:24 smithi040 ceph-mon[32418]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:52:24.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:24.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:24 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/653541064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:52:24.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:24.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:24.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:24.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:24.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:24 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1056778472' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:52:24.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:24 smithi134 ceph-mon[33098]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:52:24.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:24.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:24 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/653541064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:52:24.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:24.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:24.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:24.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:24.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:24 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1056778472' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:52:24.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:24 smithi031 ceph-mon[29204]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:52:24.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:24.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:24 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/653541064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:52:24.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:24.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:24.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:24.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:25.509 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:25 smithi134 ceph-mon[33098]: pgmap v284: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:25.509 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:25 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/653541064' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:52:25.509 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:25 smithi134 ceph-mon[33098]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:52:25.509 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:25 smithi134 ceph-mon[33098]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:52:25.509 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:25 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/621655006' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:52:25.509 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:25 smithi134 ceph-mon[33098]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:52:25.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.723+0000 7f658dffb640 1 -- 172.21.15.31:0/621655006 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f658809b5a0 con 0x7f65901074c0 2024-04-25T01:52:25.723 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T01:52:25.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 -- 172.21.15.31:0/621655006 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65800a2b90 msgr2=0x7f65800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:25.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65800a2b90 0x7f65800a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6584006010 tx=0x7f6584000bc0 comp rx=0 tx=0).stop 2024-04-25T01:52:25.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 -- 172.21.15.31:0/621655006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65901074c0 msgr2=0x7f6590100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:25.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65901074c0 0x7f6590100750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6588009fd0 tx=0x7f658800b260 comp rx=0 tx=0).stop 2024-04-25T01:52:25.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 -- 172.21.15.31:0/621655006 shutdown_connections 2024-04-25T01:52:25.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65800a2b90 0x7f65800a5050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:25.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65901074c0 0x7f6590100750 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:25.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6590104b90 0x7f6590100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:25.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 --2- 172.21.15.31:0/621655006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6590102260 0x7f6590101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:25.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 -- 172.21.15.31:0/621655006 >> 172.21.15.31:0/621655006 conn(0x7f65900fbbf0 msgr2=0x7f6590102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:25.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 -- 172.21.15.31:0/621655006 shutdown_connections 2024-04-25T01:52:25.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.725+0000 7f659719e640 1 -- 172.21.15.31:0/621655006 wait complete. 2024-04-25T01:52:25.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:25 smithi031 ceph-mon[29204]: pgmap v284: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:25.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:25 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/653541064' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:52:25.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:25 smithi031 ceph-mon[29204]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:52:25.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:25 smithi031 ceph-mon[29204]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:52:25.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:25 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/621655006' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:52:25.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:25 smithi031 ceph-mon[29204]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:52:25.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:52:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.850+0000 7fa74afb7640 1 -- 172.21.15.31:0/3697980616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74410c1b0 msgr2=0x7fa74410c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.850+0000 7fa74afb7640 1 --2- 172.21.15.31:0/3697980616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74410c1b0 0x7fa74410c5d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009900 tx=0x7fa72c02a410 comp rx=0 tx=0).stop 2024-04-25T01:52:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.850+0000 7fa74afb7640 1 -- 172.21.15.31:0/3697980616 shutdown_connections 2024-04-25T01:52:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.850+0000 7fa74afb7640 1 --2- 172.21.15.31:0/3697980616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74410dd60 0x7fa7441145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.850+0000 7fa74afb7640 1 --2- 172.21.15.31:0/3697980616 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa74410d3a0 0x7fa74410d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.850+0000 7fa74afb7640 1 --2- 172.21.15.31:0/3697980616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74410c1b0 0x7fa74410c5d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.850+0000 7fa74afb7640 1 -- 172.21.15.31:0/3697980616 >> 172.21.15.31:0/3697980616 conn(0x7fa74406c210 msgr2=0x7fa74406c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.850+0000 7fa74afb7640 1 -- 172.21.15.31:0/3697980616 shutdown_connections 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.850+0000 7fa74afb7640 1 -- 172.21.15.31:0/3697980616 wait complete. 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa74afb7640 1 Processor -- start 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa74afb7640 1 -- start start 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa74afb7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74410c1b0 0x7fa7441a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa74afb7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74410d3a0 0x7fa7441a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa74afb7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa74410dd60 0x7fa7441a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa74afb7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa744117cf0 con 0x7fa74410d3a0 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa74afb7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa744117b70 con 0x7fa74410c1b0 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa74afb7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa744117e70 con 0x7fa74410dd60 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa743fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74410d3a0 0x7fa7441a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa748d53640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74410c1b0 0x7fa7441a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa743fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74410d3a0 0x7fa7441a2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44618/0 (socket says 172.21.15.31:44618) 2024-04-25T01:52:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa743fff640 1 -- 172.21.15.31:0/1576214980 learned_addr learned my addr 172.21.15.31:0/1576214980 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa748d53640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74410c1b0 0x7fa7441a2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37996/0 (socket says 172.21.15.31:37996) 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.851+0000 7fa749554640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa74410dd60 0x7fa7441a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa743fff640 1 -- 172.21.15.31:0/1576214980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa74410dd60 msgr2=0x7fa7441a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa743fff640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa74410dd60 0x7fa7441a99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa743fff640 1 -- 172.21.15.31:0/1576214980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74410c1b0 msgr2=0x7fa7441a2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa743fff640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74410c1b0 0x7fa7441a2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa743fff640 1 -- 172.21.15.31:0/1576214980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7441aa030 con 0x7fa74410d3a0 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa748d53640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74410c1b0 0x7fa7441a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa743fff640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74410d3a0 0x7fa7441a2950 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa73400e9f0 tx=0x7fa73400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa741ffb640 1 -- 172.21.15.31:0/1576214980 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73400cd80 con 0x7fa74410d3a0 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa741ffb640 1 -- 172.21.15.31:0/1576214980 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7340054d0 con 0x7fa74410d3a0 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa741ffb640 1 -- 172.21.15.31:0/1576214980 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa734010490 con 0x7fa74410d3a0 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa74afb7640 1 -- 172.21.15.31:0/1576214980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7441aa320 con 0x7fa74410d3a0 2024-04-25T01:52:25.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.852+0000 7fa74afb7640 1 -- 172.21.15.31:0/1576214980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7441aa820 con 0x7fa74410d3a0 2024-04-25T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.853+0000 7fa74afb7640 1 -- 172.21.15.31:0/1576214980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7441135f0 con 0x7fa74410d3a0 2024-04-25T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.855+0000 7fa741ffb640 1 -- 172.21.15.31:0/1576214980 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa734005020 con 0x7fa74410d3a0 2024-04-25T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.855+0000 7fa741ffb640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa7300a2b90 0x7fa7300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.856+0000 7fa748d53640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa7300a2b90 0x7fa7300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.856+0000 7fa741ffb640 1 -- 172.21.15.31:0/1576214980 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fa7340cf2f0 con 0x7fa74410d3a0 2024-04-25T01:52:25.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.856+0000 7fa748d53640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa7300a2b90 0x7fa7300a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa72c004770 tx=0x7fa72c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:25.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:25.861+0000 7fa741ffb640 1 -- 172.21.15.31:0/1576214980 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa73409c300 con 0x7fa74410d3a0 2024-04-25T01:52:25.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:25 smithi040 ceph-mon[32418]: pgmap v284: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:25.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:25 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/653541064' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:52:25.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:25 smithi040 ceph-mon[32418]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:52:25.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:25 smithi040 ceph-mon[32418]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:52:25.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:25 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/621655006' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:52:25.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:25 smithi040 ceph-mon[32418]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:52:26.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.060+0000 7fa74afb7640 1 -- 172.21.15.31:0/1576214980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa744062340 con 0x7fa74410d3a0 2024-04-25T01:52:26.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.063+0000 7fa741ffb640 1 -- 172.21.15.31:0/1576214980 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22160 (secure 0 0 0) 0x7fa73409bc00 con 0x7fa74410d3a0 2024-04-25T01:52:26.063 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:26.063 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":86,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:52:24.897958+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:52:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 -- 172.21.15.31:0/1576214980 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa7300a2b90 msgr2=0x7fa7300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa7300a2b90 0x7fa7300a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa72c004770 tx=0x7fa72c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 -- 172.21.15.31:0/1576214980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74410d3a0 msgr2=0x7fa7441a2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74410d3a0 0x7fa7441a2950 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa73400e9f0 tx=0x7fa73400edf0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 -- 172.21.15.31:0/1576214980 shutdown_connections 2024-04-25T01:52:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa7300a2b90 0x7fa7300a5050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa74410dd60 0x7fa7441a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74410d3a0 0x7fa7441a2950 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 --2- 172.21.15.31:0/1576214980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74410c1b0 0x7fa7441a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 -- 172.21.15.31:0/1576214980 >> 172.21.15.31:0/1576214980 conn(0x7fa74406c210 msgr2=0x7fa7441107c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:26.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 -- 172.21.15.31:0/1576214980 shutdown_connections 2024-04-25T01:52:26.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.066+0000 7fa74afb7640 1 -- 172.21.15.31:0/1576214980 wait complete. 2024-04-25T01:52:26.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:52:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.197+0000 7f8ded370640 1 -- 172.21.15.31:0/596449870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de80ffe20 msgr2=0x7f8de8100240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.197+0000 7f8ded370640 1 --2- 172.21.15.31:0/596449870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de80ffe20 0x7f8de8100240 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8009900 tx=0x7f8dd802a410 comp rx=0 tx=0).stop 2024-04-25T01:52:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.197+0000 7f8ded370640 1 -- 172.21.15.31:0/596449870 shutdown_connections 2024-04-25T01:52:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.197+0000 7f8ded370640 1 --2- 172.21.15.31:0/596449870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8de8101930 0x7f8de8108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.197+0000 7f8ded370640 1 --2- 172.21.15.31:0/596449870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de8100f70 0x7f8de81013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.197+0000 7f8ded370640 1 --2- 172.21.15.31:0/596449870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de80ffe20 0x7f8de8100240 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.197+0000 7f8ded370640 1 -- 172.21.15.31:0/596449870 >> 172.21.15.31:0/596449870 conn(0x7f8de80fb3b0 msgr2=0x7f8de80fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.197+0000 7f8ded370640 1 -- 172.21.15.31:0/596449870 shutdown_connections 2024-04-25T01:52:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.197+0000 7f8ded370640 1 -- 172.21.15.31:0/596449870 wait complete. 2024-04-25T01:52:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.197+0000 7f8ded370640 1 Processor -- start 2024-04-25T01:52:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8ded370640 1 -- start start 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8ded370640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de80ffe20 0x7f8de8197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8ded370640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8de8100f70 0x7f8de8197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8ded370640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8101930 0x7f8de819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8ded370640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8de810d310 con 0x7f8de8101930 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8ded370640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8de810d190 con 0x7f8de80ffe20 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8ded370640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8de810d490 con 0x7f8de8100f70 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8de77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8101930 0x7f8de819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8de77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8101930 0x7f8de819eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44632/0 (socket says 172.21.15.31:44632) 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8de77fe640 1 -- 172.21.15.31:0/846297072 learned_addr learned my addr 172.21.15.31:0/846297072 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8de67fc640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8de8100f70 0x7f8de8197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8de6ffd640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de80ffe20 0x7f8de8197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8de77fe640 1 -- 172.21.15.31:0/846297072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8de8100f70 msgr2=0x7f8de8197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8de77fe640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8de8100f70 0x7f8de8197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8de77fe640 1 -- 172.21.15.31:0/846297072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de80ffe20 msgr2=0x7f8de8197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8de77fe640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de80ffe20 0x7f8de8197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8de77fe640 1 -- 172.21.15.31:0/846297072 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de819f550 con 0x7f8de8101930 2024-04-25T01:52:26.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.198+0000 7f8de77fe640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8101930 0x7f8de819eee0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8de000eaa0 tx=0x7f8de000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:26.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.199+0000 7f8dcffff640 1 -- 172.21.15.31:0/846297072 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de000cdb0 con 0x7f8de8101930 2024-04-25T01:52:26.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.199+0000 7f8ded370640 1 -- 172.21.15.31:0/846297072 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de819f840 con 0x7f8de8101930 2024-04-25T01:52:26.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.199+0000 7f8dcffff640 1 -- 172.21.15.31:0/846297072 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8de0005020 con 0x7f8de8101930 2024-04-25T01:52:26.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.199+0000 7f8dcffff640 1 -- 172.21.15.31:0/846297072 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de00116d0 con 0x7f8de8101930 2024-04-25T01:52:26.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.199+0000 7f8ded370640 1 -- 172.21.15.31:0/846297072 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de819fde0 con 0x7f8de8101930 2024-04-25T01:52:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.200+0000 7f8ded370640 1 -- 172.21.15.31:0/846297072 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de8107130 con 0x7f8de8101930 2024-04-25T01:52:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.201+0000 7f8dcffff640 1 -- 172.21.15.31:0/846297072 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8de00051d0 con 0x7f8de8101930 2024-04-25T01:52:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.201+0000 7f8dcffff640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8dd00a2b90 0x7f8dd00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.201+0000 7f8de6ffd640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8dd00a2b90 0x7f8dd00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.201+0000 7f8dcffff640 1 -- 172.21.15.31:0/846297072 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f8de00cf080 con 0x7f8de8101930 2024-04-25T01:52:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.202+0000 7f8de6ffd640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8dd00a2b90 0x7f8dd00a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8000c00 tx=0x7f8dd80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:26.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.206+0000 7f8dcffff640 1 -- 172.21.15.31:0/846297072 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8de0016030 con 0x7f8de8101930 2024-04-25T01:52:26.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.396+0000 7f8ded370640 1 -- 172.21.15.31:0/846297072 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8de8062340 con 0x7f8de8101930 2024-04-25T01:52:26.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.397+0000 7f8dcffff640 1 -- 172.21.15.31:0/846297072 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22160 (secure 0 0 0) 0x7f8de009c130 con 0x7f8de8101930 2024-04-25T01:52:26.398 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:26.398 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":86,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:52:24.897958+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:52:26.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.399+0000 7f8ded370640 1 -- 172.21.15.31:0/846297072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8dd00a2b90 msgr2=0x7f8dd00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.399+0000 7f8ded370640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8dd00a2b90 0x7f8dd00a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8000c00 tx=0x7f8dd80314a0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.399+0000 7f8ded370640 1 -- 172.21.15.31:0/846297072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8101930 msgr2=0x7f8de819eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.399+0000 7f8ded370640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8101930 0x7f8de819eee0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8de000eaa0 tx=0x7f8de000ef60 comp rx=0 tx=0).stop 2024-04-25T01:52:26.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.399+0000 7f8ded370640 1 -- 172.21.15.31:0/846297072 shutdown_connections 2024-04-25T01:52:26.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.399+0000 7f8ded370640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8dd00a2b90 0x7f8dd00a5050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.399+0000 7f8ded370640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8de8101930 0x7f8de819eee0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.399+0000 7f8ded370640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8de8100f70 0x7f8de8197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.399+0000 7f8ded370640 1 --2- 172.21.15.31:0/846297072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8de80ffe20 0x7f8de8197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.399+0000 7f8ded370640 1 -- 172.21.15.31:0/846297072 >> 172.21.15.31:0/846297072 conn(0x7f8de80fb3b0 msgr2=0x7f8de8104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:26.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.400+0000 7f8ded370640 1 -- 172.21.15.31:0/846297072 shutdown_connections 2024-04-25T01:52:26.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.400+0000 7f8ded370640 1 -- 172.21.15.31:0/846297072 wait complete. 2024-04-25T01:52:26.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T01:52:26.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.522+0000 7f0a77db0640 1 -- 172.21.15.31:0/3583254929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a70107460 msgr2=0x7f0a701098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.522+0000 7f0a77db0640 1 --2- 172.21.15.31:0/3583254929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a70107460 0x7f0a701098b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009be0 tx=0x7f0a6802e840 comp rx=0 tx=0).stop 2024-04-25T01:52:26.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.522+0000 7f0a77db0640 1 -- 172.21.15.31:0/3583254929 shutdown_connections 2024-04-25T01:52:26.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.522+0000 7f0a77db0640 1 --2- 172.21.15.31:0/3583254929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a70107460 0x7f0a701098b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.522+0000 7f0a77db0640 1 --2- 172.21.15.31:0/3583254929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a70104b30 0x7f0a70106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.522+0000 7f0a77db0640 1 --2- 172.21.15.31:0/3583254929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a70102200 0x7f0a701045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.522+0000 7f0a77db0640 1 -- 172.21.15.31:0/3583254929 >> 172.21.15.31:0/3583254929 conn(0x7f0a700fbbf0 msgr2=0x7f0a700fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.522+0000 7f0a77db0640 1 -- 172.21.15.31:0/3583254929 shutdown_connections 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.522+0000 7f0a77db0640 1 -- 172.21.15.31:0/3583254929 wait complete. 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.523+0000 7f0a77db0640 1 Processor -- start 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.523+0000 7f0a77db0640 1 -- start start 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.523+0000 7f0a77db0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a70102200 0x7f0a70195790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.523+0000 7f0a77db0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a70104b30 0x7f0a70195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.523+0000 7f0a77db0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a70107460 0x7f0a7019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.523+0000 7f0a77db0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7010b0f0 con 0x7f0a70102200 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.523+0000 7f0a77db0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7010af70 con 0x7f0a70107460 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.523+0000 7f0a77db0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7010b270 con 0x7f0a70104b30 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.523+0000 7f0a75b4c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a70102200 0x7f0a70195790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.523+0000 7f0a75b4c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a70102200 0x7f0a70195790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44658/0 (socket says 172.21.15.31:44658) 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.523+0000 7f0a75b4c640 1 -- 172.21.15.31:0/3865268880 learned_addr learned my addr 172.21.15.31:0/3865268880 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a7634d640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a70107460 0x7f0a7019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a75b4c640 1 -- 172.21.15.31:0/3865268880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a70104b30 msgr2=0x7f0a70195cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a7534b640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a70104b30 0x7f0a70195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a75b4c640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a70104b30 0x7f0a70195cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a75b4c640 1 -- 172.21.15.31:0/3865268880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a70107460 msgr2=0x7f0a7019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a75b4c640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a70107460 0x7f0a7019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a75b4c640 1 -- 172.21.15.31:0/3865268880 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a7019d3b0 con 0x7f0a70102200 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a7634d640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a70107460 0x7f0a7019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a75b4c640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a70102200 0x7f0a70195790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00d950 tx=0x7f0a6c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a66ffd640 1 -- 172.21.15.31:0/3865268880 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c017070 con 0x7f0a70102200 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a77db0640 1 -- 172.21.15.31:0/3865268880 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a7019d6a0 con 0x7f0a70102200 2024-04-25T01:52:26.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a66ffd640 1 -- 172.21.15.31:0/3865268880 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a6c009c30 con 0x7f0a70102200 2024-04-25T01:52:26.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a66ffd640 1 -- 172.21.15.31:0/3865268880 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c00f620 con 0x7f0a70102200 2024-04-25T01:52:26.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.524+0000 7f0a77db0640 1 -- 172.21.15.31:0/3865268880 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a7019dbf0 con 0x7f0a70102200 2024-04-25T01:52:26.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.525+0000 7f0a77db0640 1 -- 172.21.15.31:0/3865268880 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a700fd6e0 con 0x7f0a70102200 2024-04-25T01:52:26.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.526+0000 7f0a66ffd640 1 -- 172.21.15.31:0/3865268880 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0a6c0109a0 con 0x7f0a70102200 2024-04-25T01:52:26.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.527+0000 7f0a66ffd640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0a5c0a2b90 0x7f0a5c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.527+0000 7f0a7534b640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0a5c0a2b90 0x7f0a5c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.527+0000 7f0a66ffd640 1 -- 172.21.15.31:0/3865268880 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f0a6c0cebf0 con 0x7f0a70102200 2024-04-25T01:52:26.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.527+0000 7f0a7534b640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0a5c0a2b90 0x7f0a5c0a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0a70196ca0 tx=0x7f0a60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:26.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.531+0000 7f0a66ffd640 1 -- 172.21.15.31:0/3865268880 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0a6c009de0 con 0x7f0a70102200 2024-04-25T01:52:26.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:52:26.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/621655006' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:52:26.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:52:26.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:52:26.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1576214980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:26.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:26.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:26.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:26 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/846297072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:26.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:26.757+0000 7f0a77db0640 1 -- 172.21.15.31:0/3865268880 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0a7002d170 con 0x7f0a70102200 2024-04-25T01:52:26.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/621655006' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1576214980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:26.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:26 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/846297072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:26.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:52:26.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/621655006' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:52:26.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:52:26.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:52:26.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1576214980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:26.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:26.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:26.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:52:26.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:26 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/846297072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:27.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.736+0000 7f0a66ffd640 1 -- 172.21.15.31:0/3865268880 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f0a6c09bca0 con 0x7f0a70102200 2024-04-25T01:52:27.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.738+0000 7f0a77db0640 1 -- 172.21.15.31:0/3865268880 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0a5c0a2b90 msgr2=0x7f0a5c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:27.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.738+0000 7f0a77db0640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0a5c0a2b90 0x7f0a5c0a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0a70196ca0 tx=0x7f0a60009260 comp rx=0 tx=0).stop 2024-04-25T01:52:27.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.738+0000 7f0a77db0640 1 -- 172.21.15.31:0/3865268880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a70102200 msgr2=0x7f0a70195790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:27.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.738+0000 7f0a77db0640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a70102200 0x7f0a70195790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00d950 tx=0x7f0a6c00de10 comp rx=0 tx=0).stop 2024-04-25T01:52:27.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.738+0000 7f0a77db0640 1 -- 172.21.15.31:0/3865268880 shutdown_connections 2024-04-25T01:52:27.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.738+0000 7f0a77db0640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0a5c0a2b90 0x7f0a5c0a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:27.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.739+0000 7f0a77db0640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a70107460 0x7f0a7019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:27.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.739+0000 7f0a77db0640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a70104b30 0x7f0a70195cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:27.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.739+0000 7f0a77db0640 1 --2- 172.21.15.31:0/3865268880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a70102200 0x7f0a70195790 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:27.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.739+0000 7f0a77db0640 1 -- 172.21.15.31:0/3865268880 >> 172.21.15.31:0/3865268880 conn(0x7f0a700fbbf0 msgr2=0x7f0a70102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:27.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.739+0000 7f0a77db0640 1 -- 172.21.15.31:0/3865268880 shutdown_connections 2024-04-25T01:52:27.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.739+0000 7f0a77db0640 1 -- 172.21.15.31:0/3865268880 wait complete. 2024-04-25T01:52:27.750 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-25T01:52:27.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-25T01:52:27.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:27 smithi031 ceph-mon[29204]: pgmap v287: 97 pgs: 24 creating+peering, 29 active+clean, 44 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:27.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:27 smithi031 ceph-mon[29204]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:52:27.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:27 smithi031 ceph-mon[29204]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:52:27.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:27 smithi031 ceph-mon[29204]: Cluster is now healthy 2024-04-25T01:52:27.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:27 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] up:active 2024-04-25T01:52:27.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:27 smithi031 ceph-mon[29204]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:27.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:27 smithi031 ceph-mon[29204]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:27.778 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:27 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3865268880' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:52:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.862+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/2170764938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc80741a0 msgr2=0x7f3dc8074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.862+0000 7f3dcf5ce640 1 --2- 172.21.15.31:0/2170764938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc80741a0 0x7f3dc8074620 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009930 tx=0x7f3db802a420 comp rx=0 tx=0).stop 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.862+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/2170764938 shutdown_connections 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.862+0000 7f3dcf5ce640 1 --2- 172.21.15.31:0/2170764938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3dc810dd60 0x7f3dc8110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.862+0000 7f3dcf5ce640 1 --2- 172.21.15.31:0/2170764938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc80741a0 0x7f3dc8074620 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.862+0000 7f3dcf5ce640 1 --2- 172.21.15.31:0/2170764938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc8075780 0x7f3dc8073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.862+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/2170764938 >> 172.21.15.31:0/2170764938 conn(0x7f3dc80fcd20 msgr2=0x7f3dc80ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.862+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/2170764938 shutdown_connections 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.862+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/2170764938 wait complete. 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dcf5ce640 1 Processor -- start 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dcf5ce640 1 -- start start 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dcf5ce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc80741a0 0x7f3dc819e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:27.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dcf5ce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc8075780 0x7f3dc819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dcf5ce640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3dc810dd60 0x7f3dc81a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dcf5ce640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc8113960 con 0x7f3dc80741a0 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dcf5ce640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc81137e0 con 0x7f3dc8075780 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dcf5ce640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc8113ae0 con 0x7f3dc810dd60 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dcd36a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc80741a0 0x7f3dc819e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dcd36a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc80741a0 0x7f3dc819e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44680/0 (socket says 172.21.15.31:44680) 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dcd36a640 1 -- 172.21.15.31:0/3932296190 learned_addr learned my addr 172.21.15.31:0/3932296190 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.863+0000 7f3dccb69640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc8075780 0x7f3dc819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.864+0000 7f3dcd36a640 1 -- 172.21.15.31:0/3932296190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3dc810dd60 msgr2=0x7f3dc81a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.864+0000 7f3dcd36a640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3dc810dd60 0x7f3dc81a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.864+0000 7f3dcd36a640 1 -- 172.21.15.31:0/3932296190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc8075780 msgr2=0x7f3dc819e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.864+0000 7f3dcd36a640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc8075780 0x7f3dc819e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.864+0000 7f3dcd36a640 1 -- 172.21.15.31:0/3932296190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc81a5c50 con 0x7f3dc80741a0 2024-04-25T01:52:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.864+0000 7f3dcd36a640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc80741a0 0x7f3dc819e030 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400c920 tx=0x7f3dc400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:27.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.864+0000 7f3dbe7fc640 1 -- 172.21.15.31:0/3932296190 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4007c70 con 0x7f3dc80741a0 2024-04-25T01:52:27.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.864+0000 7f3dbe7fc640 1 -- 172.21.15.31:0/3932296190 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3dc4007e10 con 0x7f3dc80741a0 2024-04-25T01:52:27.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.864+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/3932296190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc81a5f40 con 0x7f3dc80741a0 2024-04-25T01:52:27.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.864+0000 7f3dbe7fc640 1 -- 172.21.15.31:0/3932296190 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4018820 con 0x7f3dc80741a0 2024-04-25T01:52:27.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.864+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/3932296190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc81a6420 con 0x7f3dc80741a0 2024-04-25T01:52:27.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.865+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/3932296190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc8103f30 con 0x7f3dc80741a0 2024-04-25T01:52:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.866+0000 7f3dbe7fc640 1 -- 172.21.15.31:0/3932296190 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3dc40189c0 con 0x7f3dc80741a0 2024-04-25T01:52:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.867+0000 7f3dbe7fc640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3db40a2b90 0x7f3db40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.867+0000 7f3dccb69640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3db40a2b90 0x7f3db40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.867+0000 7f3dbe7fc640 1 -- 172.21.15.31:0/3932296190 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f3dc40cf0c0 con 0x7f3dc80741a0 2024-04-25T01:52:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.867+0000 7f3dccb69640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3db40a2b90 0x7f3db40a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3db8005ab0 tx=0x7f3db8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:27.871+0000 7f3dbe7fc640 1 -- 172.21.15.31:0/3932296190 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3dc4014030 con 0x7f3dc80741a0 2024-04-25T01:52:27.908 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:27 smithi040 ceph-mon[32418]: pgmap v287: 97 pgs: 24 creating+peering, 29 active+clean, 44 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:27.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:27 smithi040 ceph-mon[32418]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:52:27.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:27 smithi040 ceph-mon[32418]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:52:27.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:27 smithi040 ceph-mon[32418]: Cluster is now healthy 2024-04-25T01:52:27.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:27 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] up:active 2024-04-25T01:52:27.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:27 smithi040 ceph-mon[32418]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:27.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:27 smithi040 ceph-mon[32418]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:27.909 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:27 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3865268880' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:52:27.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:27 smithi134 ceph-mon[33098]: pgmap v287: 97 pgs: 24 creating+peering, 29 active+clean, 44 unknown; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:27.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:27 smithi134 ceph-mon[33098]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:52:27.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:27 smithi134 ceph-mon[33098]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:52:27.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:27 smithi134 ceph-mon[33098]: Cluster is now healthy 2024-04-25T01:52:27.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:27 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] up:active 2024-04-25T01:52:27.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:27 smithi134 ceph-mon[33098]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:27.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:27 smithi134 ceph-mon[33098]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:27.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:27 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3865268880' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:52:28.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.096+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/3932296190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f3dc8062480 con 0x7f3dc80741a0 2024-04-25T01:52:28.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.710+0000 7f3dbe7fc640 1 -- 172.21.15.31:0/3932296190 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f3dc409c0d0 con 0x7f3dc80741a0 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/3932296190 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3db40a2b90 msgr2=0x7f3db40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3db40a2b90 0x7f3db40a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3db8005ab0 tx=0x7f3db8005a20 comp rx=0 tx=0).stop 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/3932296190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc80741a0 msgr2=0x7f3dc819e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc80741a0 0x7f3dc819e030 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400c920 tx=0x7f3dc400cde0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/3932296190 shutdown_connections 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3db40a2b90 0x7f3db40a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3dc810dd60 0x7f3dc81a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3dc8075780 0x7f3dc819e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 --2- 172.21.15.31:0/3932296190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dc80741a0 0x7f3dc819e030 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/3932296190 >> 172.21.15.31:0/3932296190 conn(0x7f3dc80fcd20 msgr2=0x7f3dc8101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/3932296190 shutdown_connections 2024-04-25T01:52:28.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.714+0000 7f3dcf5ce640 1 -- 172.21.15.31:0/3932296190 wait complete. 2024-04-25T01:52:28.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-25T01:52:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.882+0000 7fa82e590640 1 -- 172.21.15.31:0/355317726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa828104c10 msgr2=0x7fa828107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.882+0000 7fa82e590640 1 --2- 172.21.15.31:0/355317726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa828104c10 0x7fa828107000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009900 tx=0x7fa81c02a410 comp rx=0 tx=0).stop 2024-04-25T01:52:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.882+0000 7fa82e590640 1 -- 172.21.15.31:0/355317726 shutdown_connections 2024-04-25T01:52:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.882+0000 7fa82e590640 1 --2- 172.21.15.31:0/355317726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa828107540 0x7fa828109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.882+0000 7fa82e590640 1 --2- 172.21.15.31:0/355317726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa828104c10 0x7fa828107000 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.882+0000 7fa82e590640 1 --2- 172.21.15.31:0/355317726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8281022e0 0x7fa8281046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.882+0000 7fa82e590640 1 -- 172.21.15.31:0/355317726 >> 172.21.15.31:0/355317726 conn(0x7fa8280fbc30 msgr2=0x7fa8280fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.882+0000 7fa82e590640 1 -- 172.21.15.31:0/355317726 shutdown_connections 2024-04-25T01:52:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.882+0000 7fa82e590640 1 -- 172.21.15.31:0/355317726 wait complete. 2024-04-25T01:52:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.882+0000 7fa82e590640 1 Processor -- start 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.882+0000 7fa82e590640 1 -- start start 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82e590640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8281022e0 0x7fa828197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82e590640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa828104c10 0x7fa828198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82e590640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa828107540 0x7fa82819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82e590640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa82810d320 con 0x7fa828107540 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82e590640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa82810d1a0 con 0x7fa828104c10 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82e590640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa82810d4a0 con 0x7fa8281022e0 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82cb2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa828107540 0x7fa82819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82cb2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa828107540 0x7fa82819f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44698/0 (socket says 172.21.15.31:44698) 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82cb2d640 1 -- 172.21.15.31:0/552680748 learned_addr learned my addr 172.21.15.31:0/552680748 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa827fff640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8281022e0 0x7fa828197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa8277fe640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa828104c10 0x7fa828198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82cb2d640 1 -- 172.21.15.31:0/552680748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8281022e0 msgr2=0x7fa828197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82cb2d640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8281022e0 0x7fa828197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82cb2d640 1 -- 172.21.15.31:0/552680748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa828104c10 msgr2=0x7fa828198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82cb2d640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa828104c10 0x7fa828198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82cb2d640 1 -- 172.21.15.31:0/552680748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82819f690 con 0x7fa828107540 2024-04-25T01:52:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82cb2d640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa828107540 0x7fa82819f020 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa820009fd0 tx=0x7fa82000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa8257fa640 1 -- 172.21.15.31:0/552680748 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820017070 con 0x7fa828107540 2024-04-25T01:52:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa82e590640 1 -- 172.21.15.31:0/552680748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82819f980 con 0x7fa828107540 2024-04-25T01:52:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa8257fa640 1 -- 172.21.15.31:0/552680748 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa820003e20 con 0x7fa828107540 2024-04-25T01:52:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.883+0000 7fa8257fa640 1 -- 172.21.15.31:0/552680748 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820010460 con 0x7fa828107540 2024-04-25T01:52:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.884+0000 7fa82e590640 1 -- 172.21.15.31:0/552680748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa82819fe50 con 0x7fa828107540 2024-04-25T01:52:28.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.885+0000 7fa82e590640 1 -- 172.21.15.31:0/552680748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8280fd720 con 0x7fa828107540 2024-04-25T01:52:28.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.886+0000 7fa8257fa640 1 -- 172.21.15.31:0/552680748 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa820003970 con 0x7fa828107540 2024-04-25T01:52:28.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.886+0000 7fa8257fa640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa8100a2b90 0x7fa8100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:28.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.886+0000 7fa8257fa640 1 -- 172.21.15.31:0/552680748 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fa8200cec70 con 0x7fa828107540 2024-04-25T01:52:28.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.886+0000 7fa827fff640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa8100a2b90 0x7fa8100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:28.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.887+0000 7fa827fff640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa8100a2b90 0x7fa8100a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa818005fd0 tx=0x7fa81800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:28.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:28.891+0000 7fa8257fa640 1 -- 172.21.15.31:0/552680748 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa82009b6d0 con 0x7fa828107540 2024-04-25T01:52:28.986 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:28 smithi031 ceph-mon[29204]: pgmap v289: 97 pgs: 24 creating+peering, 51 active+clean, 22 unknown; 578 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 725 B/s wr, 1 op/s 2024-04-25T01:52:28.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:28 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3865268880' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:52:28.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:28 smithi031 ceph-mon[29204]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:28.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:28 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3932296190' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T01:52:28.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:28 smithi031 ceph-mon[29204]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:52:28.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:28 smithi031 ceph-mon[29204]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:52:28.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:28 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3932296190' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T01:52:28.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:28 smithi031 ceph-mon[29204]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:52:28.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:28 smithi031 ceph-mon[29204]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:52:29.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.113+0000 7fa82e590640 1 -- 172.21.15.31:0/552680748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fa828062340 con 0x7fa828107540 2024-04-25T01:52:29.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:28 smithi040 ceph-mon[32418]: pgmap v289: 97 pgs: 24 creating+peering, 51 active+clean, 22 unknown; 578 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 725 B/s wr, 1 op/s 2024-04-25T01:52:29.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:28 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3865268880' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:52:29.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:28 smithi040 ceph-mon[32418]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:29.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:28 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3932296190' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T01:52:29.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:28 smithi040 ceph-mon[32418]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:52:29.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:28 smithi040 ceph-mon[32418]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:52:29.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:28 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3932296190' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T01:52:29.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:28 smithi040 ceph-mon[32418]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:52:29.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:28 smithi040 ceph-mon[32418]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:52:29.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:28 smithi134 ceph-mon[33098]: pgmap v289: 97 pgs: 24 creating+peering, 51 active+clean, 22 unknown; 578 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 725 B/s wr, 1 op/s 2024-04-25T01:52:29.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:28 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3865268880' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:52:29.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:28 smithi134 ceph-mon[33098]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:29.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:28 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3932296190' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T01:52:29.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:28 smithi134 ceph-mon[33098]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:52:29.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:28 smithi134 ceph-mon[33098]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:52:29.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:28 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3932296190' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T01:52:29.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:28 smithi134 ceph-mon[33098]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:52:29.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:28 smithi134 ceph-mon[33098]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:52:29.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.718+0000 7fa8257fa640 1 -- 172.21.15.31:0/552680748 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7fa82009f9d0 con 0x7fa828107540 2024-04-25T01:52:29.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.721+0000 7fa82e590640 1 -- 172.21.15.31:0/552680748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa8100a2b90 msgr2=0x7fa8100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:29.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.721+0000 7fa82e590640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa8100a2b90 0x7fa8100a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa818005fd0 tx=0x7fa81800b390 comp rx=0 tx=0).stop 2024-04-25T01:52:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.721+0000 7fa82e590640 1 -- 172.21.15.31:0/552680748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa828107540 msgr2=0x7fa82819f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.721+0000 7fa82e590640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa828107540 0x7fa82819f020 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa820009fd0 tx=0x7fa82000b260 comp rx=0 tx=0).stop 2024-04-25T01:52:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.722+0000 7fa82e590640 1 -- 172.21.15.31:0/552680748 shutdown_connections 2024-04-25T01:52:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.722+0000 7fa82e590640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa8100a2b90 0x7fa8100a5050 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.722+0000 7fa82e590640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa828107540 0x7fa82819f020 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.722+0000 7fa82e590640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa828104c10 0x7fa828198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.722+0000 7fa82e590640 1 --2- 172.21.15.31:0/552680748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8281022e0 0x7fa828197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.722+0000 7fa82e590640 1 -- 172.21.15.31:0/552680748 >> 172.21.15.31:0/552680748 conn(0x7fa8280fbc30 msgr2=0x7fa828102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.722+0000 7fa82e590640 1 -- 172.21.15.31:0/552680748 shutdown_connections 2024-04-25T01:52:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.722+0000 7fa82e590640 1 -- 172.21.15.31:0/552680748 wait complete. 2024-04-25T01:52:29.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T01:52:29.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.847+0000 7f5a0219e640 1 -- 172.21.15.31:0/3633453726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59fc1006a0 msgr2=0x7f59fc100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:29.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.847+0000 7f5a0219e640 1 --2- 172.21.15.31:0/3633453726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59fc1006a0 0x7f59fc100ac0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009900 tx=0x7f59f002a410 comp rx=0 tx=0).stop 2024-04-25T01:52:29.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.847+0000 7f5a0219e640 1 -- 172.21.15.31:0/3633453726 shutdown_connections 2024-04-25T01:52:29.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.847+0000 7f5a0219e640 1 --2- 172.21.15.31:0/3633453726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc102250 0x7f59fc108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.847+0000 7f5a0219e640 1 --2- 172.21.15.31:0/3633453726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59fc101890 0x7f59fc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.847+0000 7f5a0219e640 1 --2- 172.21.15.31:0/3633453726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59fc1006a0 0x7f59fc100ac0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.847+0000 7f5a0219e640 1 -- 172.21.15.31:0/3633453726 >> 172.21.15.31:0/3633453726 conn(0x7f59fc0fbc30 msgr2=0x7f59fc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.847+0000 7f5a0219e640 1 -- 172.21.15.31:0/3633453726 shutdown_connections 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.847+0000 7f5a0219e640 1 -- 172.21.15.31:0/3633453726 wait complete. 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f5a0219e640 1 Processor -- start 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f5a0219e640 1 -- start start 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f5a0219e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59fc1006a0 0x7f59fc196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f5a0219e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59fc101890 0x7f59fc196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f5a0219e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc102250 0x7f59fc19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f5a0219e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc10c220 con 0x7f59fc101890 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f5a0219e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc10c0a0 con 0x7f59fc1006a0 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f5a0219e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc10c3a0 con 0x7f59fc102250 2024-04-25T01:52:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59faffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59fc101890 0x7f59fc196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59fb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59fc1006a0 0x7f59fc196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59faffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59fc101890 0x7f59fc196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44716/0 (socket says 172.21.15.31:44716) 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59faffd640 1 -- 172.21.15.31:0/3777253567 learned_addr learned my addr 172.21.15.31:0/3777253567 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59fbfff640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc102250 0x7f59fc19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59faffd640 1 -- 172.21.15.31:0/3777253567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc102250 msgr2=0x7f59fc19df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59faffd640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc102250 0x7f59fc19df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59faffd640 1 -- 172.21.15.31:0/3777253567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59fc1006a0 msgr2=0x7f59fc196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59faffd640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59fc1006a0 0x7f59fc196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59faffd640 1 -- 172.21.15.31:0/3777253567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc19e580 con 0x7f59fc101890 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59fb7fe640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59fc1006a0 0x7f59fc196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.848+0000 7f59fbfff640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc102250 0x7f59fc19df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.849+0000 7f59faffd640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59fc101890 0x7f59fc196ea0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f59ec00e9f0 tx=0x7f59ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.849+0000 7f59f8ff9640 1 -- 172.21.15.31:0/3777253567 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec00cd80 con 0x7f59fc101890 2024-04-25T01:52:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.849+0000 7f59f8ff9640 1 -- 172.21.15.31:0/3777253567 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59ec0108e0 con 0x7f59fc101890 2024-04-25T01:52:29.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.849+0000 7f5a0219e640 1 -- 172.21.15.31:0/3777253567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59fc19e870 con 0x7f59fc101890 2024-04-25T01:52:29.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.849+0000 7f59f8ff9640 1 -- 172.21.15.31:0/3777253567 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec018920 con 0x7f59fc101890 2024-04-25T01:52:29.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.849+0000 7f5a0219e640 1 -- 172.21.15.31:0/3777253567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59fc19ee90 con 0x7f59fc101890 2024-04-25T01:52:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.850+0000 7f5a0219e640 1 -- 172.21.15.31:0/3777253567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59fc107ae0 con 0x7f59fc101890 2024-04-25T01:52:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.851+0000 7f59f8ff9640 1 -- 172.21.15.31:0/3777253567 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f59ec010a90 con 0x7f59fc101890 2024-04-25T01:52:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.852+0000 7f59f8ff9640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f59e40a2c10 0x7f59e40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.852+0000 7f59fb7fe640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f59e40a2c10 0x7f59e40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.852+0000 7f59f8ff9640 1 -- 172.21.15.31:0/3777253567 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f59ec0cf270 con 0x7f59fc101890 2024-04-25T01:52:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.852+0000 7f59fb7fe640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f59e40a2c10 0x7f59e40a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f59f0004770 tx=0x7f59f0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:29.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:29.856+0000 7f59f8ff9640 1 -- 172.21.15.31:0/3777253567 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f59ec09c280 con 0x7f59fc101890 2024-04-25T01:52:30.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.080+0000 7f5a0219e640 1 -- 172.21.15.31:0/3777253567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f59fc062340 con 0x7f59fc101890 2024-04-25T01:52:30.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:29 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/552680748' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:52:30.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:29 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/552680748' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:52:30.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:29 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] up:active 2024-04-25T01:52:30.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:29 smithi040 ceph-mon[32418]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:52:30.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:29 smithi040 ceph-mon[32418]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:52:30.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:29 smithi040 ceph-mon[32418]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:52:30.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:29 smithi040 ceph-mon[32418]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:52:30.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:29 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/552680748' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:52:30.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:29 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/552680748' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:52:30.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:29 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] up:active 2024-04-25T01:52:30.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:29 smithi134 ceph-mon[33098]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:52:30.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:29 smithi134 ceph-mon[33098]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:52:30.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:29 smithi134 ceph-mon[33098]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:52:30.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:29 smithi134 ceph-mon[33098]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:52:30.236 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:29 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/552680748' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:52:30.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:29 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/552680748' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:52:30.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:29 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] up:active 2024-04-25T01:52:30.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:29 smithi031 ceph-mon[29204]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:52:30.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:29 smithi031 ceph-mon[29204]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:52:30.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:29 smithi031 ceph-mon[29204]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:52:30.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:29 smithi031 ceph-mon[29204]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:52:30.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.725+0000 7f59f8ff9640 1 -- 172.21.15.31:0/3777253567 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7f59ec09bb80 con 0x7f59fc101890 2024-04-25T01:52:30.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 -- 172.21.15.31:0/3777253567 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f59e40a2c10 msgr2=0x7f59e40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:30.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f59e40a2c10 0x7f59e40a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f59f0004770 tx=0x7f59f0000bc0 comp rx=0 tx=0).stop 2024-04-25T01:52:30.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 -- 172.21.15.31:0/3777253567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59fc101890 msgr2=0x7f59fc196ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:30.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59fc101890 0x7f59fc196ea0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f59ec00e9f0 tx=0x7f59ec00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:52:30.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 -- 172.21.15.31:0/3777253567 shutdown_connections 2024-04-25T01:52:30.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f59e40a2c10 0x7f59e40a50d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:30.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59fc102250 0x7f59fc19df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:30.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59fc101890 0x7f59fc196ea0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:30.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 --2- 172.21.15.31:0/3777253567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59fc1006a0 0x7f59fc196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:30.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 -- 172.21.15.31:0/3777253567 >> 172.21.15.31:0/3777253567 conn(0x7f59fc0fbc30 msgr2=0x7f59fc104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:30.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 -- 172.21.15.31:0/3777253567 shutdown_connections 2024-04-25T01:52:30.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.729+0000 7f5a0219e640 1 -- 172.21.15.31:0/3777253567 wait complete. 2024-04-25T01:52:30.746 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-25T01:52:30.747 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-25T01:52:30.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.857+0000 7efdb35c1640 1 -- 172.21.15.31:0/3441168224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac102230 msgr2=0x7efdac108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.857+0000 7efdb35c1640 1 --2- 172.21.15.31:0/3441168224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac102230 0x7efdac108ac0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efda4009900 tx=0x7efda402a410 comp rx=0 tx=0).stop 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 -- 172.21.15.31:0/3441168224 shutdown_connections 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 --2- 172.21.15.31:0/3441168224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac102230 0x7efdac108ac0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 --2- 172.21.15.31:0/3441168224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdac101870 0x7efdac101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 --2- 172.21.15.31:0/3441168224 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdac100680 0x7efdac100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 -- 172.21.15.31:0/3441168224 >> 172.21.15.31:0/3441168224 conn(0x7efdac0fbbf0 msgr2=0x7efdac0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 -- 172.21.15.31:0/3441168224 shutdown_connections 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 -- 172.21.15.31:0/3441168224 wait complete. 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 Processor -- start 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 -- start start 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac100680 0x7efdac104800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdac101870 0x7efdac102e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:30.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdac102230 0x7efdac103380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.858+0000 7efdb35c1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efdac10c190 con 0x7efdac100680 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb35c1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efdac10c010 con 0x7efdac102230 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb35c1640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efdac10c310 con 0x7efdac101870 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb135d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac100680 0x7efdac104800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb135d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac100680 0x7efdac104800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44740/0 (socket says 172.21.15.31:44740) 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb135d640 1 -- 172.21.15.31:0/2863869206 learned_addr learned my addr 172.21.15.31:0/2863869206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb1b5e640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdac102230 0x7efdac103380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb0b5c640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdac101870 0x7efdac102e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb135d640 1 -- 172.21.15.31:0/2863869206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdac101870 msgr2=0x7efdac102e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb135d640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdac101870 0x7efdac102e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb135d640 1 -- 172.21.15.31:0/2863869206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdac102230 msgr2=0x7efdac103380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb135d640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdac102230 0x7efdac103380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb135d640 1 -- 172.21.15.31:0/2863869206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdac103c10 con 0x7efdac100680 2024-04-25T01:52:30.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb135d640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac100680 0x7efdac104800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00ea80 tx=0x7efd9c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:30.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdaa7fc640 1 -- 172.21.15.31:0/2863869206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c00ce40 con 0x7efdac100680 2024-04-25T01:52:30.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdaa7fc640 1 -- 172.21.15.31:0/2863869206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd9c011920 con 0x7efdac100680 2024-04-25T01:52:30.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb35c1640 1 -- 172.21.15.31:0/2863869206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdac1a1ab0 con 0x7efdac100680 2024-04-25T01:52:30.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdaa7fc640 1 -- 172.21.15.31:0/2863869206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c01aaa0 con 0x7efdac100680 2024-04-25T01:52:30.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.859+0000 7efdb35c1640 1 -- 172.21.15.31:0/2863869206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdac1a2050 con 0x7efdac100680 2024-04-25T01:52:30.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.860+0000 7efdb35c1640 1 -- 172.21.15.31:0/2863869206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdac1a2280 con 0x7efdac100680 2024-04-25T01:52:30.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.861+0000 7efdaa7fc640 1 -- 172.21.15.31:0/2863869206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7efd9c011ad0 con 0x7efdac100680 2024-04-25T01:52:30.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.862+0000 7efdaa7fc640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd940a2c10 0x7efd940a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:30.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.862+0000 7efdb0b5c640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd940a2c10 0x7efd940a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:30.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.862+0000 7efdaa7fc640 1 -- 172.21.15.31:0/2863869206 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7efd9c0cf500 con 0x7efdac100680 2024-04-25T01:52:30.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.862+0000 7efdb0b5c640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd940a2c10 0x7efd940a50d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efdac104570 tx=0x7efda0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:30.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:30.866+0000 7efdaa7fc640 1 -- 172.21.15.31:0/2863869206 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efd9c016030 con 0x7efdac100680 2024-04-25T01:52:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.080+0000 7efdb35c1640 1 -- 172.21.15.31:0/2863869206 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7efdac068fe0 con 0x7efd940a2c10 2024-04-25T01:52:31.085 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:30 smithi031 ceph-mon[29204]: pgmap v290: 97 pgs: 4 creating+peering, 93 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 5 op/s 2024-04-25T01:52:31.085 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:30 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3777253567' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:52:31.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:30 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3777253567' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:52:31.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:30 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] up:active 2024-04-25T01:52:31.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:30 smithi031 ceph-mon[29204]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T01:52:31.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:30 smithi031 ceph-mon[29204]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:52:31.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:30 smithi031 ceph-mon[29204]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-25T01:52:31.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:30 smithi031 ceph-mon[29204]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-25T01:52:31.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:30 smithi040 ceph-mon[32418]: pgmap v290: 97 pgs: 4 creating+peering, 93 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 5 op/s 2024-04-25T01:52:31.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:30 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3777253567' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:52:31.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:30 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3777253567' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:52:31.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:30 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] up:active 2024-04-25T01:52:31.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:30 smithi040 ceph-mon[32418]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T01:52:31.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:30 smithi040 ceph-mon[32418]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:52:31.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:30 smithi040 ceph-mon[32418]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-25T01:52:31.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:30 smithi040 ceph-mon[32418]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-25T01:52:31.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:30 smithi134 ceph-mon[33098]: pgmap v290: 97 pgs: 4 creating+peering, 93 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 5 op/s 2024-04-25T01:52:31.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:30 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3777253567' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:52:31.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:30 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3777253567' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:52:31.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:30 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] up:active 2024-04-25T01:52:31.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:30 smithi134 ceph-mon[33098]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T01:52:31.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:30 smithi134 ceph-mon[33098]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:52:31.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:30 smithi134 ceph-mon[33098]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-25T01:52:31.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:30 smithi134 ceph-mon[33098]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:52:31.084+0000 7f8307e5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:52:31.084+0000 7f8307e5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:52:31.084+0000 7f8307e5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:52:31.084+0000 7f8307e5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:52:31.084+0000 7f8307e5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:52:31.754 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:31 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b[62588]: 2024-04-25T01:52:31.485+0000 7fa3082fa700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:52:31.754 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:31 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b[62588]: 2024-04-25T01:52:31.485+0000 7fa3082fa700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T01:52:31.754 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:31 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b[62588]: 2024-04-25T01:52:31.485+0000 7fa3082fa700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T01:52:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.760+0000 7efdaa7fc640 1 -- 172.21.15.31:0/2863869206 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efdac068fe0 con 0x7efd940a2c10 2024-04-25T01:52:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 -- 172.21.15.31:0/2863869206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd940a2c10 msgr2=0x7efd940a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd940a2c10 0x7efd940a50d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efdac104570 tx=0x7efda0005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 -- 172.21.15.31:0/2863869206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac100680 msgr2=0x7efdac104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac100680 0x7efdac104800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00ea80 tx=0x7efd9c00ef40 comp rx=0 tx=0).stop 2024-04-25T01:52:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 -- 172.21.15.31:0/2863869206 shutdown_connections 2024-04-25T01:52:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd940a2c10 0x7efd940a50d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdac102230 0x7efdac103380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdac101870 0x7efdac102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 --2- 172.21.15.31:0/2863869206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdac100680 0x7efdac104800 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 -- 172.21.15.31:0/2863869206 >> 172.21.15.31:0/2863869206 conn(0x7efdac0fbbf0 msgr2=0x7efdac0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:31.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 -- 172.21.15.31:0/2863869206 shutdown_connections 2024-04-25T01:52:31.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.764+0000 7efdb35c1640 1 -- 172.21.15.31:0/2863869206 wait complete. 2024-04-25T01:52:31.778 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T01:52:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.891+0000 7f0223638640 1 -- 172.21.15.31:0/4028785484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f021c0694d0 msgr2=0x7f021c0ffe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.891+0000 7f0223638640 1 --2- 172.21.15.31:0/4028785484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f021c0694d0 0x7f021c0ffe10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0218009930 tx=0x7f021802a420 comp rx=0 tx=0).stop 2024-04-25T01:52:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.891+0000 7f0223638640 1 -- 172.21.15.31:0/4028785484 shutdown_connections 2024-04-25T01:52:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.891+0000 7f0223638640 1 --2- 172.21.15.31:0/4028785484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021c100f60 0x7f021c101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.891+0000 7f0223638640 1 --2- 172.21.15.31:0/4028785484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021c0698f0 0x7f021c100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.891+0000 7f0223638640 1 --2- 172.21.15.31:0/4028785484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f021c0694d0 0x7f021c0ffe10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.891+0000 7f0223638640 1 -- 172.21.15.31:0/4028785484 >> 172.21.15.31:0/4028785484 conn(0x7f021c0fb380 msgr2=0x7f021c0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.891+0000 7f0223638640 1 -- 172.21.15.31:0/4028785484 shutdown_connections 2024-04-25T01:52:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.891+0000 7f0223638640 1 -- 172.21.15.31:0/4028785484 wait complete. 2024-04-25T01:52:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f0223638640 1 Processor -- start 2024-04-25T01:52:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f0223638640 1 -- start start 2024-04-25T01:52:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f0223638640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f021c0694d0 0x7f021c1969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f0223638640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021c0698f0 0x7f021c196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f0223638640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021c100f60 0x7f021c19df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f0223638640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f021c10c290 con 0x7f021c0694d0 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f0223638640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f021c10c110 con 0x7f021c0698f0 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f0223638640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f021c10c410 con 0x7f021c100f60 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f02213d4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f021c0694d0 0x7f021c1969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f02213d4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f021c0694d0 0x7f021c1969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44788/0 (socket says 172.21.15.31:44788) 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f02213d4640 1 -- 172.21.15.31:0/865984879 learned_addr learned my addr 172.21.15.31:0/865984879 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.892+0000 7f0220bd3640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021c0698f0 0x7f021c196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f0221bd5640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021c100f60 0x7f021c19df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f02213d4640 1 -- 172.21.15.31:0/865984879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021c100f60 msgr2=0x7f021c19df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f02213d4640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021c100f60 0x7f021c19df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f02213d4640 1 -- 172.21.15.31:0/865984879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021c0698f0 msgr2=0x7f021c196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f02213d4640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021c0698f0 0x7f021c196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f02213d4640 1 -- 172.21.15.31:0/865984879 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021c19e5f0 con 0x7f021c0694d0 2024-04-25T01:52:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f0221bd5640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021c100f60 0x7f021c19df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:52:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f0220bd3640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021c0698f0 0x7f021c196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f02213d4640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f021c0694d0 0x7f021c1969d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f02180314e0 tx=0x7f0218031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f02127fc640 1 -- 172.21.15.31:0/865984879 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021803f030 con 0x7f021c0694d0 2024-04-25T01:52:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f02127fc640 1 -- 172.21.15.31:0/865984879 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02180044b0 con 0x7f021c0694d0 2024-04-25T01:52:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f0223638640 1 -- 172.21.15.31:0/865984879 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f021c19e880 con 0x7f021c0694d0 2024-04-25T01:52:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f02127fc640 1 -- 172.21.15.31:0/865984879 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218035070 con 0x7f021c0694d0 2024-04-25T01:52:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.893+0000 7f0223638640 1 -- 172.21.15.31:0/865984879 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f021c19ee40 con 0x7f021c0694d0 2024-04-25T01:52:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.894+0000 7f0223638640 1 -- 172.21.15.31:0/865984879 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f021c107ce0 con 0x7f021c0694d0 2024-04-25T01:52:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.895+0000 7f02127fc640 1 -- 172.21.15.31:0/865984879 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0218042030 con 0x7f021c0694d0 2024-04-25T01:52:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.896+0000 7f02127fc640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f02080a2c10 0x7f02080a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.896+0000 7f0220bd3640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f02080a2c10 0x7f02080a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.896+0000 7f02127fc640 1 -- 172.21.15.31:0/865984879 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f02180ef8a0 con 0x7f021c0694d0 2024-04-25T01:52:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.896+0000 7f0220bd3640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f02080a2c10 0x7f02080a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f021c197ee0 tx=0x7f020c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:31.899+0000 7f02127fc640 1 -- 172.21.15.31:0/865984879 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f02180bc8b0 con 0x7f021c0694d0 2024-04-25T01:52:32.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.113+0000 7f0223638640 1 -- 172.21.15.31:0/865984879 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f021c061c10 con 0x7f02080a2c10 2024-04-25T01:52:32.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.138+0000 7f02127fc640 1 -- 172.21.15.31:0/865984879 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f021c061c10 con 0x7f02080a2c10 2024-04-25T01:52:32.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 -- 172.21.15.31:0/865984879 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f02080a2c10 msgr2=0x7f02080a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f02080a2c10 0x7f02080a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f021c197ee0 tx=0x7f020c005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:32.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 -- 172.21.15.31:0/865984879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f021c0694d0 msgr2=0x7f021c1969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f021c0694d0 0x7f021c1969d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f02180314e0 tx=0x7f0218031a30 comp rx=0 tx=0).stop 2024-04-25T01:52:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 -- 172.21.15.31:0/865984879 shutdown_connections 2024-04-25T01:52:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f02080a2c10 0x7f02080a50d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f021c100f60 0x7f021c19df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021c0698f0 0x7f021c196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 --2- 172.21.15.31:0/865984879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f021c0694d0 0x7f021c1969d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 -- 172.21.15.31:0/865984879 >> 172.21.15.31:0/865984879 conn(0x7f021c0fb380 msgr2=0x7f021c0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 -- 172.21.15.31:0/865984879 shutdown_connections 2024-04-25T01:52:32.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.140+0000 7f0223638640 1 -- 172.21.15.31:0/865984879 wait complete. 2024-04-25T01:52:32.150 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-mon[29204]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:32.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] up:active 2024-04-25T01:52:32.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-mon[29204]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T01:52:32.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-mon[29204]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:52:32.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-mon[29204]: Cluster is now healthy 2024-04-25T01:52:32.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-mon[29204]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-04-25T01:52:32.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:31 smithi031 ceph-mon[29204]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-25T01:52:32.152 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T01:52:32.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:31 smithi040 ceph-mon[32418]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:32.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:31 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] up:active 2024-04-25T01:52:32.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:31 smithi040 ceph-mon[32418]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T01:52:32.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:31 smithi040 ceph-mon[32418]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:52:32.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:31 smithi040 ceph-mon[32418]: Cluster is now healthy 2024-04-25T01:52:32.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:31 smithi040 ceph-mon[32418]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-04-25T01:52:32.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:31 smithi040 ceph-mon[32418]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-25T01:52:32.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:31 smithi134 ceph-mon[33098]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:32.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:31 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] up:active 2024-04-25T01:52:32.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:31 smithi134 ceph-mon[33098]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T01:52:32.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:31 smithi134 ceph-mon[33098]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:52:32.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:31 smithi134 ceph-mon[33098]: Cluster is now healthy 2024-04-25T01:52:32.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:31 smithi134 ceph-mon[33098]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-04-25T01:52:32.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:31 smithi134 ceph-mon[33098]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-25T01:52:32.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.258+0000 7f58d131b640 1 -- 172.21.15.31:0/1729007434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58cc100680 msgr2=0x7f58cc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.258+0000 7f58d131b640 1 --2- 172.21.15.31:0/1729007434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58cc100680 0x7f58cc100aa0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009900 tx=0x7f58bc02a410 comp rx=0 tx=0).stop 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.258+0000 7f58d131b640 1 -- 172.21.15.31:0/1729007434 shutdown_connections 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.258+0000 7f58d131b640 1 --2- 172.21.15.31:0/1729007434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58cc102230 0x7f58cc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.258+0000 7f58d131b640 1 --2- 172.21.15.31:0/1729007434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58cc101870 0x7f58cc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.258+0000 7f58d131b640 1 --2- 172.21.15.31:0/1729007434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58cc100680 0x7f58cc100aa0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.258+0000 7f58d131b640 1 -- 172.21.15.31:0/1729007434 >> 172.21.15.31:0/1729007434 conn(0x7f58cc0fbbf0 msgr2=0x7f58cc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.258+0000 7f58d131b640 1 -- 172.21.15.31:0/1729007434 shutdown_connections 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.258+0000 7f58d131b640 1 -- 172.21.15.31:0/1729007434 wait complete. 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58d131b640 1 Processor -- start 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58d131b640 1 -- start start 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58d131b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58cc100680 0x7f58cc196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58d131b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58cc101870 0x7f58cc196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58d131b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58cc102230 0x7f58cc19df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58d131b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc10c200 con 0x7f58cc102230 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58d131b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc10c080 con 0x7f58cc100680 2024-04-25T01:52:32.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58d131b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc10c380 con 0x7f58cc101870 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58cc102230 0x7f58cc19df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58caffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58cc100680 0x7f58cc196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58caffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58cc100680 0x7f58cc196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38148/0 (socket says 172.21.15.31:38148) 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58cc102230 0x7f58cc19df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44812/0 (socket says 172.21.15.31:44812) 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58caffd640 1 -- 172.21.15.31:0/3627367097 learned_addr learned my addr 172.21.15.31:0/3627367097 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58c3fff640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58cc101870 0x7f58cc196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58cb7fe640 1 -- 172.21.15.31:0/3627367097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58cc101870 msgr2=0x7f58cc196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58cb7fe640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58cc101870 0x7f58cc196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58cb7fe640 1 -- 172.21.15.31:0/3627367097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58cc100680 msgr2=0x7f58cc196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58cb7fe640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58cc100680 0x7f58cc196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58cb7fe640 1 -- 172.21.15.31:0/3627367097 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58cc19e5b0 con 0x7f58cc102230 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58c3fff640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58cc101870 0x7f58cc196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.259+0000 7f58caffd640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58cc100680 0x7f58cc196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:32.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.260+0000 7f58cb7fe640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58cc102230 0x7f58cc19df40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009fd0 tx=0x7f58c400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:32.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.260+0000 7f58c8ff9640 1 -- 172.21.15.31:0/3627367097 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c400ce20 con 0x7f58cc102230 2024-04-25T01:52:32.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.260+0000 7f58c8ff9640 1 -- 172.21.15.31:0/3627367097 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58c4005020 con 0x7f58cc102230 2024-04-25T01:52:32.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.260+0000 7f58d131b640 1 -- 172.21.15.31:0/3627367097 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58cc068930 con 0x7f58cc102230 2024-04-25T01:52:32.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.260+0000 7f58c8ff9640 1 -- 172.21.15.31:0/3627367097 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4010690 con 0x7f58cc102230 2024-04-25T01:52:32.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.260+0000 7f58d131b640 1 -- 172.21.15.31:0/3627367097 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58cc068eb0 con 0x7f58cc102230 2024-04-25T01:52:32.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.261+0000 7f58d131b640 1 -- 172.21.15.31:0/3627367097 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58cc107580 con 0x7f58cc102230 2024-04-25T01:52:32.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.262+0000 7f58c8ff9640 1 -- 172.21.15.31:0/3627367097 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f58c40051d0 con 0x7f58cc102230 2024-04-25T01:52:32.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.263+0000 7f58c8ff9640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f58b80a2b90 0x7f58b80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:32.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.263+0000 7f58caffd640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f58b80a2b90 0x7f58b80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:32.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.263+0000 7f58c8ff9640 1 -- 172.21.15.31:0/3627367097 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f58c40cf670 con 0x7f58cc102230 2024-04-25T01:52:32.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.263+0000 7f58caffd640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f58b80a2b90 0x7f58b80a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f58bc032fd0 tx=0x7f58bc005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:32.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.267+0000 7f58c8ff9640 1 -- 172.21.15.31:0/3627367097 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f58c409c720 con 0x7f58cc102230 2024-04-25T01:52:32.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.482+0000 7f58d131b640 1 -- 172.21.15.31:0/3627367097 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f58cc069170 con 0x7f58b80a2b90 2024-04-25T01:52:32.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.487+0000 7f58c8ff9640 1 -- 172.21.15.31:0/3627367097 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f58cc069170 con 0x7f58b80a2b90 2024-04-25T01:52:32.487 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.489+0000 7f58d131b640 1 -- 172.21.15.31:0/3627367097 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f58b80a2b90 msgr2=0x7f58b80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.489+0000 7f58d131b640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f58b80a2b90 0x7f58b80a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f58bc032fd0 tx=0x7f58bc005dd0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.489+0000 7f58d131b640 1 -- 172.21.15.31:0/3627367097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58cc102230 msgr2=0x7f58cc19df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.489+0000 7f58d131b640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58cc102230 0x7f58cc19df40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009fd0 tx=0x7f58c400ef10 comp rx=0 tx=0).stop 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.489+0000 7f58d131b640 1 -- 172.21.15.31:0/3627367097 shutdown_connections 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.489+0000 7f58d131b640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f58b80a2b90 0x7f58b80a5050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.489+0000 7f58d131b640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58cc102230 0x7f58cc19df40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.489+0000 7f58d131b640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58cc101870 0x7f58cc196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.489+0000 7f58d131b640 1 --2- 172.21.15.31:0/3627367097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58cc100680 0x7f58cc196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.489+0000 7f58d131b640 1 -- 172.21.15.31:0/3627367097 >> 172.21.15.31:0/3627367097 conn(0x7f58cc0fbbf0 msgr2=0x7f58cc104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.490+0000 7f58d131b640 1 -- 172.21.15.31:0/3627367097 shutdown_connections 2024-04-25T01:52:32.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.490+0000 7f58d131b640 1 -- 172.21.15.31:0/3627367097 wait complete. 2024-04-25T01:52:32.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T01:52:32.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.609+0000 7f92a999e640 1 -- 172.21.15.31:0/3277779144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92a4101650 msgr2=0x7f92a4101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.609+0000 7f92a999e640 1 --2- 172.21.15.31:0/3277779144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92a4101650 0x7f92a4101a70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f929c009ca0 tx=0x7f929c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.609+0000 7f92a999e640 1 -- 172.21.15.31:0/3277779144 shutdown_connections 2024-04-25T01:52:32.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.609+0000 7f92a999e640 1 --2- 172.21.15.31:0/3277779144 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92a4103200 0x7f92a4109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.609+0000 7f92a999e640 1 --2- 172.21.15.31:0/3277779144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92a4102840 0x7f92a4102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.609+0000 7f92a999e640 1 --2- 172.21.15.31:0/3277779144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92a4101650 0x7f92a4101a70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.609+0000 7f92a999e640 1 -- 172.21.15.31:0/3277779144 >> 172.21.15.31:0/3277779144 conn(0x7f92a40fcbe0 msgr2=0x7f92a40ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:32.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.609+0000 7f92a999e640 1 -- 172.21.15.31:0/3277779144 shutdown_connections 2024-04-25T01:52:32.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.609+0000 7f92a999e640 1 -- 172.21.15.31:0/3277779144 wait complete. 2024-04-25T01:52:32.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a999e640 1 Processor -- start 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a999e640 1 -- start start 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a999e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92a4101650 0x7f92a41978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a999e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92a4102840 0x7f92a4197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a999e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92a4103200 0x7f92a419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a999e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f92a410d180 con 0x7f92a4103200 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a999e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f92a410d000 con 0x7f92a4101650 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a999e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f92a410d300 con 0x7f92a4102840 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a919d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92a4103200 0x7f92a419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a899c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92a4101650 0x7f92a41978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a919d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92a4103200 0x7f92a419ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44824/0 (socket says 172.21.15.31:44824) 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a919d640 1 -- 172.21.15.31:0/3016347472 learned_addr learned my addr 172.21.15.31:0/3016347472 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a3fff640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92a4102840 0x7f92a4197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a919d640 1 -- 172.21.15.31:0/3016347472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92a4102840 msgr2=0x7f92a4197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a919d640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92a4102840 0x7f92a4197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a919d640 1 -- 172.21.15.31:0/3016347472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92a4101650 msgr2=0x7f92a41978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a919d640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92a4101650 0x7f92a41978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a919d640 1 -- 172.21.15.31:0/3016347472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a419f4e0 con 0x7f92a4103200 2024-04-25T01:52:32.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a3fff640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92a4102840 0x7f92a4197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:52:32.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a919d640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92a4103200 0x7f92a419ee70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f929400d750 tx=0x7f929400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:32.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a1ffb640 1 -- 172.21.15.31:0/3016347472 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9294017070 con 0x7f92a4103200 2024-04-25T01:52:32.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a1ffb640 1 -- 172.21.15.31:0/3016347472 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92940054d0 con 0x7f92a4103200 2024-04-25T01:52:32.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.610+0000 7f92a999e640 1 -- 172.21.15.31:0/3016347472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a419f7d0 con 0x7f92a4103200 2024-04-25T01:52:32.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.611+0000 7f92a1ffb640 1 -- 172.21.15.31:0/3016347472 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9294004c60 con 0x7f92a4103200 2024-04-25T01:52:32.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.611+0000 7f92a999e640 1 -- 172.21.15.31:0/3016347472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a419fd20 con 0x7f92a4103200 2024-04-25T01:52:32.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.612+0000 7f92a999e640 1 -- 172.21.15.31:0/3016347472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a4108a90 con 0x7f92a4103200 2024-04-25T01:52:32.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.613+0000 7f92a1ffb640 1 -- 172.21.15.31:0/3016347472 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9294004e00 con 0x7f92a4103200 2024-04-25T01:52:32.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.613+0000 7f92a1ffb640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f92900a2b90 0x7f92900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:32.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.613+0000 7f92a899c640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f92900a2b90 0x7f92900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:32.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.613+0000 7f92a1ffb640 1 -- 172.21.15.31:0/3016347472 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f92940ce890 con 0x7f92a4103200 2024-04-25T01:52:32.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.614+0000 7f92a899c640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f92900a2b90 0x7f92900a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f929c000f80 tx=0x7f929c031000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:32.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.618+0000 7f92a1ffb640 1 -- 172.21.15.31:0/3016347472 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f929409b940 con 0x7f92a4103200 2024-04-25T01:52:32.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.772+0000 7f92a1ffb640 1 -- 172.21.15.31:0/3016347472 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f92940978b0 con 0x7f92a4103200 2024-04-25T01:52:32.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.843+0000 7f92a999e640 1 -- 172.21.15.31:0/3016347472 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f92a4061b80 con 0x7f92900a2b90 2024-04-25T01:52:32.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.867+0000 7f92a1ffb640 1 -- 172.21.15.31:0/3016347472 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f92a4061b80 con 0x7f92900a2b90 2024-04-25T01:52:32.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 -- 172.21.15.31:0/3016347472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f92900a2b90 msgr2=0x7f92900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f92900a2b90 0x7f92900a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f929c000f80 tx=0x7f929c031000 comp rx=0 tx=0).stop 2024-04-25T01:52:32.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 -- 172.21.15.31:0/3016347472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92a4103200 msgr2=0x7f92a419ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92a4103200 0x7f92a419ee70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f929400d750 tx=0x7f929400dc10 comp rx=0 tx=0).stop 2024-04-25T01:52:32.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 -- 172.21.15.31:0/3016347472 shutdown_connections 2024-04-25T01:52:32.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f92900a2b90 0x7f92900a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92a4103200 0x7f92a419ee70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92a4102840 0x7f92a4197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 --2- 172.21.15.31:0/3016347472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92a4101650 0x7f92a41978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 -- 172.21.15.31:0/3016347472 >> 172.21.15.31:0/3016347472 conn(0x7f92a40fcbe0 msgr2=0x7f92a4105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:32.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 -- 172.21.15.31:0/3016347472 shutdown_connections 2024-04-25T01:52:32.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.869+0000 7f92a999e640 1 -- 172.21.15.31:0/3016347472 wait complete. 2024-04-25T01:52:32.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T01:52:32.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.995+0000 7f8844c1c640 1 -- 172.21.15.31:0/2168614215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8840101890 msgr2=0x7f8840101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.995+0000 7f8844c1c640 1 --2- 172.21.15.31:0/2168614215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8840101890 0x7f8840101d10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8828009900 tx=0x7f882802a410 comp rx=0 tx=0).stop 2024-04-25T01:52:32.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.995+0000 7f8844c1c640 1 -- 172.21.15.31:0/2168614215 shutdown_connections 2024-04-25T01:52:32.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.995+0000 7f8844c1c640 1 --2- 172.21.15.31:0/2168614215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8840102250 0x7f8840108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.995+0000 7f8844c1c640 1 --2- 172.21.15.31:0/2168614215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8840101890 0x7f8840101d10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.995+0000 7f8844c1c640 1 --2- 172.21.15.31:0/2168614215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88401006a0 0x7f8840100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.995+0000 7f8844c1c640 1 -- 172.21.15.31:0/2168614215 >> 172.21.15.31:0/2168614215 conn(0x7f88400fbc30 msgr2=0x7f88400fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.995+0000 7f8844c1c640 1 -- 172.21.15.31:0/2168614215 shutdown_connections 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.995+0000 7f8844c1c640 1 -- 172.21.15.31:0/2168614215 wait complete. 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f8844c1c640 1 Processor -- start 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f8844c1c640 1 -- start start 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f8844c1c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88401006a0 0x7f8840197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f8844c1c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8840101890 0x7f8840197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f8844c1c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8840102250 0x7f884019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f8844c1c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f884010d2f0 con 0x7f8840102250 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f8844c1c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f884010d170 con 0x7f88401006a0 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f8844c1c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f884010d470 con 0x7f8840101890 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f883ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8840102250 0x7f884019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f883ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8840102250 0x7f884019efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38002/0 (socket says 172.21.15.31:38002) 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f883e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88401006a0 0x7f8840197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:32.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f883ed9d640 1 -- 172.21.15.31:0/708982104 learned_addr learned my addr 172.21.15.31:0/708982104 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.996+0000 7f8837fff640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8840101890 0x7f8840197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f883ed9d640 1 -- 172.21.15.31:0/708982104 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8840101890 msgr2=0x7f8840197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f883ed9d640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8840101890 0x7f8840197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f883ed9d640 1 -- 172.21.15.31:0/708982104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88401006a0 msgr2=0x7f8840197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f883ed9d640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88401006a0 0x7f8840197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f883ed9d640 1 -- 172.21.15.31:0/708982104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884019f650 con 0x7f8840102250 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f883e59c640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88401006a0 0x7f8840197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f883ed9d640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8840102250 0x7f884019efe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f883800eb10 tx=0x7f883800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f88377fe640 1 -- 172.21.15.31:0/708982104 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838004b20 con 0x7f8840102250 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f8844c1c640 1 -- 172.21.15.31:0/708982104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f884019f940 con 0x7f8840102250 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f88377fe640 1 -- 172.21.15.31:0/708982104 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8838004cc0 con 0x7f8840102250 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f88377fe640 1 -- 172.21.15.31:0/708982104 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838010690 con 0x7f8840102250 2024-04-25T01:52:32.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.997+0000 7f8844c1c640 1 -- 172.21.15.31:0/708982104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f884019fe10 con 0x7f8840102250 2024-04-25T01:52:33.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.998+0000 7f8844c1c640 1 -- 172.21.15.31:0/708982104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8840107a80 con 0x7f8840102250 2024-04-25T01:52:33.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:32.999+0000 7f88377fe640 1 -- 172.21.15.31:0/708982104 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8838005020 con 0x7f8840102250 2024-04-25T01:52:33.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.000+0000 7f88377fe640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f882c0a2c20 0x7f882c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:33.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.000+0000 7f883e59c640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f882c0a2c20 0x7f882c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:33.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.000+0000 7f88377fe640 1 -- 172.21.15.31:0/708982104 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f88380cf6b0 con 0x7f8840102250 2024-04-25T01:52:33.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.000+0000 7f883e59c640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f882c0a2c20 0x7f882c0a50e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8830005fd0 tx=0x7f883000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:33.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.004+0000 7f88377fe640 1 -- 172.21.15.31:0/708982104 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f883809c560 con 0x7f8840102250 2024-04-25T01:52:33.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:32 smithi040 ceph-mon[32418]: pgmap v291: 97 pgs: 97 active+clean; 580 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 12 op/s 2024-04-25T01:52:33.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:32 smithi040 ceph-mon[32418]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-25T01:52:33.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:32 smithi040 ceph-mon[32418]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:33.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:32 smithi040 ceph-mon[32418]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:33.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:32 smithi134 ceph-mon[33098]: pgmap v291: 97 pgs: 97 active+clean; 580 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 12 op/s 2024-04-25T01:52:33.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:32 smithi134 ceph-mon[33098]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-25T01:52:33.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:32 smithi134 ceph-mon[33098]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:33.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:32 smithi134 ceph-mon[33098]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:33.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.217+0000 7f8844c1c640 1 -- 172.21.15.31:0/708982104 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f8840061b80 con 0x7f882c0a2c20 2024-04-25T01:52:33.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.221+0000 7f88377fe640 1 -- 172.21.15.31:0/708982104 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8840061b80 con 0x7f882c0a2c20 2024-04-25T01:52:33.222 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/qa/sv_1/3b71f681-d37b-4bb4-9bd7-1d9dc0aa7714 2024-04-25T01:52:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.223+0000 7f8844c1c640 1 -- 172.21.15.31:0/708982104 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f882c0a2c20 msgr2=0x7f882c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.223+0000 7f8844c1c640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f882c0a2c20 0x7f882c0a50e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8830005fd0 tx=0x7f883000b390 comp rx=0 tx=0).stop 2024-04-25T01:52:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.223+0000 7f8844c1c640 1 -- 172.21.15.31:0/708982104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8840102250 msgr2=0x7f884019efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.223+0000 7f8844c1c640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8840102250 0x7f884019efe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f883800eb10 tx=0x7f883800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.224+0000 7f8844c1c640 1 -- 172.21.15.31:0/708982104 shutdown_connections 2024-04-25T01:52:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.224+0000 7f8844c1c640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f882c0a2c20 0x7f882c0a50e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.224+0000 7f8844c1c640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8840102250 0x7f884019efe0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.224+0000 7f8844c1c640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8840101890 0x7f8840197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.224+0000 7f8844c1c640 1 --2- 172.21.15.31:0/708982104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88401006a0 0x7f8840197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.224+0000 7f8844c1c640 1 -- 172.21.15.31:0/708982104 >> 172.21.15.31:0/708982104 conn(0x7f88400fbc30 msgr2=0x7f8840104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:33.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.224+0000 7f8844c1c640 1 -- 172.21.15.31:0/708982104 shutdown_connections 2024-04-25T01:52:33.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.224+0000 7f8844c1c640 1 -- 172.21.15.31:0/708982104 wait complete. 2024-04-25T01:52:33.233 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:32 smithi031 ceph-mon[29204]: pgmap v291: 97 pgs: 97 active+clean; 580 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 12 op/s 2024-04-25T01:52:33.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:32 smithi031 ceph-mon[29204]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-25T01:52:33.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:32 smithi031 ceph-mon[29204]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:33.234 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:32 smithi031 ceph-mon[29204]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:33.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:33.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.337+0000 7f55dd7ed640 1 -- 172.21.15.31:0/3132086561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d8102230 msgr2=0x7f55d8108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.337+0000 7f55dd7ed640 1 --2- 172.21.15.31:0/3132086561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d8102230 0x7f55d8108ac0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f55d0009900 tx=0x7f55d002a410 comp rx=0 tx=0).stop 2024-04-25T01:52:33.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.337+0000 7f55dd7ed640 1 -- 172.21.15.31:0/3132086561 shutdown_connections 2024-04-25T01:52:33.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.337+0000 7f55dd7ed640 1 --2- 172.21.15.31:0/3132086561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d8102230 0x7f55d8108ac0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.337+0000 7f55dd7ed640 1 --2- 172.21.15.31:0/3132086561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d8101870 0x7f55d8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.337+0000 7f55dd7ed640 1 --2- 172.21.15.31:0/3132086561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55d8100680 0x7f55d8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.337+0000 7f55dd7ed640 1 -- 172.21.15.31:0/3132086561 >> 172.21.15.31:0/3132086561 conn(0x7f55d80fbbf0 msgr2=0x7f55d80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:33.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.337+0000 7f55dd7ed640 1 -- 172.21.15.31:0/3132086561 shutdown_connections 2024-04-25T01:52:33.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.337+0000 7f55dd7ed640 1 -- 172.21.15.31:0/3132086561 wait complete. 2024-04-25T01:52:33.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55dd7ed640 1 Processor -- start 2024-04-25T01:52:33.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55dd7ed640 1 -- start start 2024-04-25T01:52:33.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55dd7ed640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d8100680 0x7f55d8197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:33.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55dd7ed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d8101870 0x7f55d8197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:33.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55dd7ed640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55d8102230 0x7f55d819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:33.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55dd7ed640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f55d810c1b0 con 0x7f55d8100680 2024-04-25T01:52:33.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55dd7ed640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f55d810c030 con 0x7f55d8101870 2024-04-25T01:52:33.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55dd7ed640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f55d810c330 con 0x7f55d8102230 2024-04-25T01:52:33.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55d6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d8100680 0x7f55d8197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:33.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55d6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d8100680 0x7f55d8197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38024/0 (socket says 172.21.15.31:38024) 2024-04-25T01:52:33.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55d6ffd640 1 -- 172.21.15.31:0/2099065277 learned_addr learned my addr 172.21.15.31:0/2099065277 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:33.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55d67fc640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d8101870 0x7f55d8197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:33.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.338+0000 7f55d77fe640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55d8102230 0x7f55d819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.339+0000 7f55d6ffd640 1 -- 172.21.15.31:0/2099065277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55d8102230 msgr2=0x7f55d819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.339+0000 7f55d6ffd640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55d8102230 0x7f55d819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.339+0000 7f55d6ffd640 1 -- 172.21.15.31:0/2099065277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d8101870 msgr2=0x7f55d8197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.339+0000 7f55d6ffd640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d8101870 0x7f55d8197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.339+0000 7f55d6ffd640 1 -- 172.21.15.31:0/2099065277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d819f5b0 con 0x7f55d8100680 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.339+0000 7f55d6ffd640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d8100680 0x7f55d8197990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f55c800eaa0 tx=0x7f55c800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.339+0000 7f55cffff640 1 -- 172.21.15.31:0/2099065277 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8016070 con 0x7f55d8100680 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.339+0000 7f55cffff640 1 -- 172.21.15.31:0/2099065277 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55c8003e20 con 0x7f55d8100680 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.339+0000 7f55dd7ed640 1 -- 172.21.15.31:0/2099065277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d819f8a0 con 0x7f55d8100680 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.339+0000 7f55cffff640 1 -- 172.21.15.31:0/2099065277 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c80116d0 con 0x7f55d8100680 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.339+0000 7f55dd7ed640 1 -- 172.21.15.31:0/2099065277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d819fda0 con 0x7f55d8100680 2024-04-25T01:52:33.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.340+0000 7f55dd7ed640 1 -- 172.21.15.31:0/2099065277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d81a0000 con 0x7f55d8100680 2024-04-25T01:52:33.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.341+0000 7f55cffff640 1 -- 172.21.15.31:0/2099065277 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f55c8019070 con 0x7f55d8100680 2024-04-25T01:52:33.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.342+0000 7f55cffff640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f55c40a2c20 0x7f55c40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:33.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.342+0000 7f55d67fc640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f55c40a2c20 0x7f55c40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:33.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.342+0000 7f55cffff640 1 -- 172.21.15.31:0/2099065277 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f55c80cf080 con 0x7f55d8100680 2024-04-25T01:52:33.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.342+0000 7f55d67fc640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f55c40a2c20 0x7f55c40a50e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f55c00095f0 tx=0x7f55c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:33.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.346+0000 7f55cffff640 1 -- 172.21.15.31:0/2099065277 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f55c809b9e0 con 0x7f55d8100680 2024-04-25T01:52:33.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.573+0000 7f55dd7ed640 1 -- 172.21.15.31:0/2099065277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55d81a01d0 con 0x7f55d8100680 2024-04-25T01:52:33.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.574+0000 7f55cffff640 1 -- 172.21.15.31:0/2099065277 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11923 (secure 0 0 0) 0x7f55c809fce0 con 0x7f55d8100680 2024-04-25T01:52:33.574 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:33.574 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:52:32.772296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24487,24473,24503,24463,24451]},"id":1}]} 2024-04-25T01:52:33.575 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T01:52:33.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 -- 172.21.15.31:0/2099065277 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f55c40a2c20 msgr2=0x7f55c40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f55c40a2c20 0x7f55c40a50e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f55c00095f0 tx=0x7f55c0005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:33.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 -- 172.21.15.31:0/2099065277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d8100680 msgr2=0x7f55d8197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d8100680 0x7f55d8197990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f55c800eaa0 tx=0x7f55c800ef60 comp rx=0 tx=0).stop 2024-04-25T01:52:33.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 -- 172.21.15.31:0/2099065277 shutdown_connections 2024-04-25T01:52:33.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f55c40a2c20 0x7f55c40a50e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55d8102230 0x7f55d819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d8101870 0x7f55d8197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 --2- 172.21.15.31:0/2099065277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d8100680 0x7f55d8197990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 -- 172.21.15.31:0/2099065277 >> 172.21.15.31:0/2099065277 conn(0x7f55d80fbbf0 msgr2=0x7f55d80fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:33.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 -- 172.21.15.31:0/2099065277 shutdown_connections 2024-04-25T01:52:33.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.576+0000 7f55dd7ed640 1 -- 172.21.15.31:0/2099065277 wait complete. 2024-04-25T01:52:33.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:52:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.695+0000 7f7730de9640 1 -- 172.21.15.31:0/3985707817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f772c100690 msgr2=0x7f772c100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.695+0000 7f7730de9640 1 --2- 172.21.15.31:0/3985707817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f772c100690 0x7f772c100ab0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7724009900 tx=0x7f772402a410 comp rx=0 tx=0).stop 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.695+0000 7f7730de9640 1 -- 172.21.15.31:0/3985707817 shutdown_connections 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.695+0000 7f7730de9640 1 --2- 172.21.15.31:0/3985707817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f772c102240 0x7f772c108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.695+0000 7f7730de9640 1 --2- 172.21.15.31:0/3985707817 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f772c101880 0x7f772c101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.695+0000 7f7730de9640 1 --2- 172.21.15.31:0/3985707817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f772c100690 0x7f772c100ab0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.695+0000 7f7730de9640 1 -- 172.21.15.31:0/3985707817 >> 172.21.15.31:0/3985707817 conn(0x7f772c0fbc20 msgr2=0x7f772c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.696+0000 7f7730de9640 1 -- 172.21.15.31:0/3985707817 shutdown_connections 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.696+0000 7f7730de9640 1 -- 172.21.15.31:0/3985707817 wait complete. 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.696+0000 7f7730de9640 1 Processor -- start 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.696+0000 7f7730de9640 1 -- start start 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.696+0000 7f7730de9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f772c100690 0x7f772c06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.696+0000 7f7730de9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f772c101880 0x7f772c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.696+0000 7f7730de9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f772c102240 0x7f772c19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.696+0000 7f7730de9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f772c10c1c0 con 0x7f772c101880 2024-04-25T01:52:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.696+0000 7f7730de9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f772c10c040 con 0x7f772c102240 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.696+0000 7f7730de9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f772c10c340 con 0x7f772c100690 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f7729d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f772c101880 0x7f772c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f772a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f772c100690 0x7f772c06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f7729d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f772c101880 0x7f772c19c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38038/0 (socket says 172.21.15.31:38038) 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f7729d9b640 1 -- 172.21.15.31:0/2040775782 learned_addr learned my addr 172.21.15.31:0/2040775782 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f772ad9d640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f772c102240 0x7f772c19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f7729d9b640 1 -- 172.21.15.31:0/2040775782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f772c100690 msgr2=0x7f772c06a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f7729d9b640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f772c100690 0x7f772c06a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f7729d9b640 1 -- 172.21.15.31:0/2040775782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f772c102240 msgr2=0x7f772c19eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f7729d9b640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f772c102240 0x7f772c19eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f7729d9b640 1 -- 172.21.15.31:0/2040775782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f772c19f6b0 con 0x7f772c101880 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f7729d9b640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f772c101880 0x7f772c19c780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f772000d770 tx=0x7f772000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f771b7fe640 1 -- 172.21.15.31:0/2040775782 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7720017070 con 0x7f772c101880 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f771b7fe640 1 -- 172.21.15.31:0/2040775782 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77200054d0 con 0x7f772c101880 2024-04-25T01:52:33.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f7730de9640 1 -- 172.21.15.31:0/2040775782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f772c19f940 con 0x7f772c101880 2024-04-25T01:52:33.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f771b7fe640 1 -- 172.21.15.31:0/2040775782 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7720004cf0 con 0x7f772c101880 2024-04-25T01:52:33.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.697+0000 7f7730de9640 1 -- 172.21.15.31:0/2040775782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f772c19fe90 con 0x7f772c101880 2024-04-25T01:52:33.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.698+0000 7f7730de9640 1 -- 172.21.15.31:0/2040775782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f772c107a20 con 0x7f772c101880 2024-04-25T01:52:33.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.699+0000 7f771b7fe640 1 -- 172.21.15.31:0/2040775782 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7720004e90 con 0x7f772c101880 2024-04-25T01:52:33.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.700+0000 7f771b7fe640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f77140a2ca0 0x7f77140a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:33.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.700+0000 7f771b7fe640 1 -- 172.21.15.31:0/2040775782 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f77200ce960 con 0x7f772c101880 2024-04-25T01:52:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.700+0000 7f772a59c640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f77140a2ca0 0x7f77140a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:33.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.700+0000 7f772a59c640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f77140a2ca0 0x7f77140a5160 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7724004770 tx=0x7f7724000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:33.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.704+0000 7f771b7fe640 1 -- 172.21.15.31:0/2040775782 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f772009b2c0 con 0x7f772c101880 2024-04-25T01:52:33.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.894+0000 7f7730de9640 1 -- 172.21.15.31:0/2040775782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f772c02d320 con 0x7f772c101880 2024-04-25T01:52:33.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.895+0000 7f771b7fe640 1 -- 172.21.15.31:0/2040775782 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22147 (secure 0 0 0) 0x7f772009f5c0 con 0x7f772c101880 2024-04-25T01:52:33.896 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:33.896 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:52:26.507181+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:52:33.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 -- 172.21.15.31:0/2040775782 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f77140a2ca0 msgr2=0x7f77140a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f77140a2ca0 0x7f77140a5160 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7724004770 tx=0x7f7724000bc0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 -- 172.21.15.31:0/2040775782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f772c101880 msgr2=0x7f772c19c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:33.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f772c101880 0x7f772c19c780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f772000d770 tx=0x7f772000dc30 comp rx=0 tx=0).stop 2024-04-25T01:52:33.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 -- 172.21.15.31:0/2040775782 shutdown_connections 2024-04-25T01:52:33.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f77140a2ca0 0x7f77140a5160 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f772c102240 0x7f772c19eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f772c101880 0x7f772c19c780 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 --2- 172.21.15.31:0/2040775782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f772c100690 0x7f772c06a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:33.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 -- 172.21.15.31:0/2040775782 >> 172.21.15.31:0/2040775782 conn(0x7f772c0fbc20 msgr2=0x7f772c104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:33.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 -- 172.21.15.31:0/2040775782 shutdown_connections 2024-04-25T01:52:33.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:33.898+0000 7f7730de9640 1 -- 172.21.15.31:0/2040775782 wait complete. 2024-04-25T01:52:33.908 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:33 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] up:active 2024-04-25T01:52:33.909 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:33 smithi031 ceph-mon[29204]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-25T01:52:33.909 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:33 smithi031 ceph-mon[29204]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:52:33.909 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:33 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:52:33.909 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:33 smithi031 ceph-mon[29204]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:33.909 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:33 smithi031 ceph-mon[29204]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:33.910 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:33 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2099065277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:33.912 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T01:52:33.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:52:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.016+0000 7f6d1eda5640 1 -- 172.21.15.31:0/3939952924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d18100690 msgr2=0x7f6d18100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.016+0000 7f6d1eda5640 1 --2- 172.21.15.31:0/3939952924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d18100690 0x7f6d18100ab0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009ca0 tx=0x7f6d1002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.016+0000 7f6d1eda5640 1 -- 172.21.15.31:0/3939952924 shutdown_connections 2024-04-25T01:52:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.016+0000 7f6d1eda5640 1 --2- 172.21.15.31:0/3939952924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102240 0x7f6d18108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.016+0000 7f6d1eda5640 1 --2- 172.21.15.31:0/3939952924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d18101880 0x7f6d18101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.016+0000 7f6d1eda5640 1 --2- 172.21.15.31:0/3939952924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d18100690 0x7f6d18100ab0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.016+0000 7f6d1eda5640 1 -- 172.21.15.31:0/3939952924 >> 172.21.15.31:0/3939952924 conn(0x7f6d180fbc20 msgr2=0x7f6d180fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.016+0000 7f6d1eda5640 1 -- 172.21.15.31:0/3939952924 shutdown_connections 2024-04-25T01:52:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.016+0000 7f6d1eda5640 1 -- 172.21.15.31:0/3939952924 wait complete. 2024-04-25T01:52:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d1eda5640 1 Processor -- start 2024-04-25T01:52:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d1eda5640 1 -- start start 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d1eda5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d18100690 0x7f6d18197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d1eda5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d18101880 0x7f6d18197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d1eda5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102240 0x7f6d1819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d1eda5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1810c200 con 0x7f6d18101880 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d1eda5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1810c080 con 0x7f6d18102240 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d1eda5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1810c380 con 0x7f6d18100690 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d17fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d18101880 0x7f6d18197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d17fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d18101880 0x7f6d18197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38056/0 (socket says 172.21.15.31:38056) 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d17fff640 1 -- 172.21.15.31:0/2132534380 learned_addr learned my addr 172.21.15.31:0/2132534380 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d1cb41640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d18100690 0x7f6d18197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d1d342640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102240 0x7f6d1819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d17fff640 1 -- 172.21.15.31:0/2132534380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d18100690 msgr2=0x7f6d18197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d17fff640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d18100690 0x7f6d18197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d17fff640 1 -- 172.21.15.31:0/2132534380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102240 msgr2=0x7f6d1819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d17fff640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102240 0x7f6d1819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d17fff640 1 -- 172.21.15.31:0/2132534380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d1819f660 con 0x7f6d18101880 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.017+0000 7f6d17fff640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d18101880 0x7f6d18197fd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000d770 tx=0x7f6d0000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.018+0000 7f6d15ffb640 1 -- 172.21.15.31:0/2132534380 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d00017070 con 0x7f6d18101880 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.018+0000 7f6d15ffb640 1 -- 172.21.15.31:0/2132534380 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d000054d0 con 0x7f6d18101880 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.018+0000 7f6d1eda5640 1 -- 172.21.15.31:0/2132534380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d1819f950 con 0x7f6d18101880 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.018+0000 7f6d15ffb640 1 -- 172.21.15.31:0/2132534380 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d00004cf0 con 0x7f6d18101880 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.018+0000 7f6d1eda5640 1 -- 172.21.15.31:0/2132534380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d1819fea0 con 0x7f6d18101880 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.019+0000 7f6d1eda5640 1 -- 172.21.15.31:0/2132534380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d181a00d0 con 0x7f6d18101880 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.020+0000 7f6d15ffb640 1 -- 172.21.15.31:0/2132534380 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6d00004e90 con 0x7f6d18101880 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.020+0000 7f6d15ffb640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6d040a2ca0 0x7f6d040a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.020+0000 7f6d1cb41640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6d040a2ca0 0x7f6d040a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.020+0000 7f6d15ffb640 1 -- 172.21.15.31:0/2132534380 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f6d000ce960 con 0x7f6d18101880 2024-04-25T01:52:34.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.021+0000 7f6d1cb41640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6d040a2ca0 0x7f6d040a5160 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6d10000f80 tx=0x7f6d10031000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:34.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.025+0000 7f6d15ffb640 1 -- 172.21.15.31:0/2132534380 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d0009b2c0 con 0x7f6d18101880 2024-04-25T01:52:34.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:33 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] up:active 2024-04-25T01:52:34.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:33 smithi040 ceph-mon[32418]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-25T01:52:34.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:33 smithi040 ceph-mon[32418]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:52:34.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:33 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:52:34.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:33 smithi040 ceph-mon[32418]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:34.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:33 smithi040 ceph-mon[32418]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:34.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:33 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2099065277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:34.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:33 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] up:active 2024-04-25T01:52:34.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:33 smithi134 ceph-mon[33098]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-25T01:52:34.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:33 smithi134 ceph-mon[33098]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:52:34.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:33 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:52:34.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:33 smithi134 ceph-mon[33098]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:34.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:33 smithi134 ceph-mon[33098]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:34.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:33 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2099065277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:34.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.206+0000 7f6d1eda5640 1 -- 172.21.15.31:0/2132534380 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6d1806a5a0 con 0x7f6d040a2ca0 2024-04-25T01:52:34.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.214+0000 7f6d15ffb640 1 -- 172.21.15.31:0/2132534380 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258938 (secure 0 0 0) 0x7f6d1806a5a0 con 0x7f6d040a2ca0 2024-04-25T01:52:34.214 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:34.220 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.217+0000 7f6d1eda5640 1 -- 172.21.15.31:0/2132534380 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6d040a2ca0 msgr2=0x7f6d040a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.217+0000 7f6d1eda5640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6d040a2ca0 0x7f6d040a5160 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6d10000f80 tx=0x7f6d10031000 comp rx=0 tx=0).stop 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.217+0000 7f6d1eda5640 1 -- 172.21.15.31:0/2132534380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d18101880 msgr2=0x7f6d18197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.217+0000 7f6d1eda5640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d18101880 0x7f6d18197fd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000d770 tx=0x7f6d0000dc30 comp rx=0 tx=0).stop 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.218+0000 7f6d1eda5640 1 -- 172.21.15.31:0/2132534380 shutdown_connections 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.218+0000 7f6d1eda5640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6d040a2ca0 0x7f6d040a5160 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.218+0000 7f6d1eda5640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d18102240 0x7f6d1819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.218+0000 7f6d1eda5640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d18101880 0x7f6d18197fd0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.218+0000 7f6d1eda5640 1 --2- 172.21.15.31:0/2132534380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d18100690 0x7f6d18197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.218+0000 7f6d1eda5640 1 -- 172.21.15.31:0/2132534380 >> 172.21.15.31:0/2132534380 conn(0x7f6d180fbc20 msgr2=0x7f6d18069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.218+0000 7f6d1eda5640 1 -- 172.21.15.31:0/2132534380 shutdown_connections 2024-04-25T01:52:34.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.218+0000 7f6d1eda5640 1 -- 172.21.15.31:0/2132534380 wait complete. 2024-04-25T01:52:34.229 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-04-25T01:52:33.732097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":613786,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":311,"num_write_kb":2995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368868,"kb_used_data":11376,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572956,"statfs":{"total":1151940427776,"available":1151562706944,"internally_reserved":0,"allocated":11649024,"data_stored":7200106,"data_compressed":106956,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23418,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.224344"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988531+0000","last_change":"2024-04-25T01:52:25.523199+0000","last_active":"2024-04-25T01:52:27.988531+0000","last_peered":"2024-04-25T01:52:27.988531+0000","last_clean":"2024-04-25T01:52:27.988531+0000","last_became_active":"2024-04-25T01:52:25.522052+0000","last_became_peered":"2024-04-25T01:52:25.522052+0000","last_unstale":"2024-04-25T01:52:27.988531+0000","last_undegraded":"2024-04-25T01:52:27.988531+0000","last_fullsized":"2024-04-25T01:52:27.988531+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:07:27.095370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487912+0000","last_change":"2024-04-25T01:52:25.528431+0000","last_active":"2024-04-25T01:52:27.487912+0000","last_peered":"2024-04-25T01:52:27.487912+0000","last_clean":"2024-04-25T01:52:27.487912+0000","last_became_active":"2024-04-25T01:52:25.528122+0000","last_became_peered":"2024-04-25T01:52:25.528122+0000","last_unstale":"2024-04-25T01:52:27.487912+0000","last_undegraded":"2024-04-25T01:52:27.487912+0000","last_fullsized":"2024-04-25T01:52:27.487912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:26:56.794582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487847+0000","last_change":"2024-04-25T01:52:25.532963+0000","last_active":"2024-04-25T01:52:27.487847+0000","last_peered":"2024-04-25T01:52:27.487847+0000","last_clean":"2024-04-25T01:52:27.487847+0000","last_became_active":"2024-04-25T01:52:25.532770+0000","last_became_peered":"2024-04-25T01:52:25.532770+0000","last_unstale":"2024-04-25T01:52:27.487847+0000","last_undegraded":"2024-04-25T01:52:27.487847+0000","last_fullsized":"2024-04-25T01:52:27.487847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:13.398868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793349+0000","last_change":"2024-04-25T01:52:25.533032+0000","last_active":"2024-04-25T01:52:27.793349+0000","last_peered":"2024-04-25T01:52:27.793349+0000","last_clean":"2024-04-25T01:52:27.793349+0000","last_became_active":"2024-04-25T01:52:25.530437+0000","last_became_peered":"2024-04-25T01:52:25.530437+0000","last_unstale":"2024-04-25T01:52:27.793349+0000","last_undegraded":"2024-04-25T01:52:27.793349+0000","last_fullsized":"2024-04-25T01:52:27.793349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:48.526222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.521921+0000","last_change":"2024-04-25T01:52:25.523624+0000","last_active":"2024-04-25T01:52:26.521921+0000","last_peered":"2024-04-25T01:52:26.521921+0000","last_clean":"2024-04-25T01:52:26.521921+0000","last_became_active":"2024-04-25T01:52:25.523289+0000","last_became_peered":"2024-04-25T01:52:25.523289+0000","last_unstale":"2024-04-25T01:52:26.521921+0000","last_undegraded":"2024-04-25T01:52:26.521921+0000","last_fullsized":"2024-04-25T01:52:26.521921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:09.688472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.007733+0000","last_change":"2024-04-25T01:52:25.527123+0000","last_active":"2024-04-25T01:52:27.007733+0000","last_peered":"2024-04-25T01:52:27.007733+0000","last_clean":"2024-04-25T01:52:27.007733+0000","last_became_active":"2024-04-25T01:52:25.526808+0000","last_became_peered":"2024-04-25T01:52:25.526808+0000","last_unstale":"2024-04-25T01:52:27.007733+0000","last_undegraded":"2024-04-25T01:52:27.007733+0000","last_fullsized":"2024-04-25T01:52:27.007733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:31.654857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377688+0000","last_change":"2024-04-25T01:52:25.517011+0000","last_active":"2024-04-25T01:52:27.377688+0000","last_peered":"2024-04-25T01:52:27.377688+0000","last_clean":"2024-04-25T01:52:27.377688+0000","last_became_active":"2024-04-25T01:52:25.516680+0000","last_became_peered":"2024-04-25T01:52:25.516680+0000","last_unstale":"2024-04-25T01:52:27.377688+0000","last_undegraded":"2024-04-25T01:52:27.377688+0000","last_fullsized":"2024-04-25T01:52:27.377688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:03.388576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732057+0000","last_change":"2024-04-25T01:52:25.531079+0000","last_active":"2024-04-25T01:52:27.732057+0000","last_peered":"2024-04-25T01:52:27.732057+0000","last_clean":"2024-04-25T01:52:27.732057+0000","last_became_active":"2024-04-25T01:52:25.529200+0000","last_became_peered":"2024-04-25T01:52:25.529200+0000","last_unstale":"2024-04-25T01:52:27.732057+0000","last_undegraded":"2024-04-25T01:52:27.732057+0000","last_fullsized":"2024-04-25T01:52:27.732057+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:30.484010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.520111+0000","last_change":"2024-04-25T01:52:25.533421+0000","last_active":"2024-04-25T01:52:26.520111+0000","last_peered":"2024-04-25T01:52:26.520111+0000","last_clean":"2024-04-25T01:52:26.520111+0000","last_became_active":"2024-04-25T01:52:25.531781+0000","last_became_peered":"2024-04-25T01:52:25.531781+0000","last_unstale":"2024-04-25T01:52:26.520111+0000","last_undegraded":"2024-04-25T01:52:26.520111+0000","last_fullsized":"2024-04-25T01:52:26.520111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:32.505793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.008019+0000","last_change":"2024-04-25T01:52:25.530223+0000","last_active":"2024-04-25T01:52:27.008019+0000","last_peered":"2024-04-25T01:52:27.008019+0000","last_clean":"2024-04-25T01:52:27.008019+0000","last_became_active":"2024-04-25T01:52:25.529909+0000","last_became_peered":"2024-04-25T01:52:25.529909+0000","last_unstale":"2024-04-25T01:52:27.008019+0000","last_undegraded":"2024-04-25T01:52:27.008019+0000","last_fullsized":"2024-04-25T01:52:27.008019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:07.429026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.378147+0000","last_change":"2024-04-25T01:52:25.524502+0000","last_active":"2024-04-25T01:52:27.378147+0000","last_peered":"2024-04-25T01:52:27.378147+0000","last_clean":"2024-04-25T01:52:27.378147+0000","last_became_active":"2024-04-25T01:52:25.524018+0000","last_became_peered":"2024-04-25T01:52:25.524018+0000","last_unstale":"2024-04-25T01:52:27.378147+0000","last_undegraded":"2024-04-25T01:52:27.378147+0000","last_fullsized":"2024-04-25T01:52:27.378147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:23.760389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793370+0000","last_change":"2024-04-25T01:52:25.533593+0000","last_active":"2024-04-25T01:52:27.793370+0000","last_peered":"2024-04-25T01:52:27.793370+0000","last_clean":"2024-04-25T01:52:27.793370+0000","last_became_active":"2024-04-25T01:52:25.532745+0000","last_became_peered":"2024-04-25T01:52:25.532745+0000","last_unstale":"2024-04-25T01:52:27.793370+0000","last_undegraded":"2024-04-25T01:52:27.793370+0000","last_fullsized":"2024-04-25T01:52:27.793370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:45.083267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517350+0000","last_change":"2024-04-25T01:52:25.528563+0000","last_active":"2024-04-25T01:52:26.517350+0000","last_peered":"2024-04-25T01:52:26.517350+0000","last_clean":"2024-04-25T01:52:26.517350+0000","last_became_active":"2024-04-25T01:52:25.528236+0000","last_became_peered":"2024-04-25T01:52:25.528236+0000","last_unstale":"2024-04-25T01:52:26.517350+0000","last_undegraded":"2024-04-25T01:52:26.517350+0000","last_fullsized":"2024-04-25T01:52:26.517350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:28:27.222799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487482+0000","last_change":"2024-04-25T01:52:25.524951+0000","last_active":"2024-04-25T01:52:27.487482+0000","last_peered":"2024-04-25T01:52:27.487482+0000","last_clean":"2024-04-25T01:52:27.487482+0000","last_became_active":"2024-04-25T01:52:25.524617+0000","last_became_peered":"2024-04-25T01:52:25.524617+0000","last_unstale":"2024-04-25T01:52:27.487482+0000","last_undegraded":"2024-04-25T01:52:27.487482+0000","last_fullsized":"2024-04-25T01:52:27.487482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:03.561883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487726+0000","last_change":"2024-04-25T01:52:25.532471+0000","last_active":"2024-04-25T01:52:27.487726+0000","last_peered":"2024-04-25T01:52:27.487726+0000","last_clean":"2024-04-25T01:52:27.487726+0000","last_became_active":"2024-04-25T01:52:25.532131+0000","last_became_peered":"2024-04-25T01:52:25.532131+0000","last_unstale":"2024-04-25T01:52:27.487726+0000","last_undegraded":"2024-04-25T01:52:27.487726+0000","last_fullsized":"2024-04-25T01:52:27.487726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:03.478816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.519636+0000","last_change":"2024-04-25T01:52:25.529160+0000","last_active":"2024-04-25T01:52:26.519636+0000","last_peered":"2024-04-25T01:52:26.519636+0000","last_clean":"2024-04-25T01:52:26.519636+0000","last_became_active":"2024-04-25T01:52:25.528854+0000","last_became_peered":"2024-04-25T01:52:25.528854+0000","last_unstale":"2024-04-25T01:52:26.519636+0000","last_undegraded":"2024-04-25T01:52:26.519636+0000","last_fullsized":"2024-04-25T01:52:26.519636+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:54.713715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988660+0000","last_change":"2024-04-25T01:52:25.523372+0000","last_active":"2024-04-25T01:52:27.988660+0000","last_peered":"2024-04-25T01:52:27.988660+0000","last_clean":"2024-04-25T01:52:27.988660+0000","last_became_active":"2024-04-25T01:52:25.522538+0000","last_became_peered":"2024-04-25T01:52:25.522538+0000","last_unstale":"2024-04-25T01:52:27.988660+0000","last_undegraded":"2024-04-25T01:52:27.988660+0000","last_fullsized":"2024-04-25T01:52:27.988660+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:47.213815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.730426+0000","last_change":"2024-04-25T01:52:25.522920+0000","last_active":"2024-04-25T01:52:27.730426+0000","last_peered":"2024-04-25T01:52:27.730426+0000","last_clean":"2024-04-25T01:52:27.730426+0000","last_became_active":"2024-04-25T01:52:25.522594+0000","last_became_peered":"2024-04-25T01:52:25.522594+0000","last_unstale":"2024-04-25T01:52:27.730426+0000","last_undegraded":"2024-04-25T01:52:27.730426+0000","last_fullsized":"2024-04-25T01:52:27.730426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:45.043667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793620+0000","last_change":"2024-04-25T01:52:25.532738+0000","last_active":"2024-04-25T01:52:27.793620+0000","last_peered":"2024-04-25T01:52:27.793620+0000","last_clean":"2024-04-25T01:52:27.793620+0000","last_became_active":"2024-04-25T01:52:25.532237+0000","last_became_peered":"2024-04-25T01:52:25.532237+0000","last_unstale":"2024-04-25T01:52:27.793620+0000","last_undegraded":"2024-04-25T01:52:27.793620+0000","last_fullsized":"2024-04-25T01:52:27.793620+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:54.456704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:29.736267+0000","last_change":"2024-04-25T01:52:24.515686+0000","last_active":"2024-04-25T01:52:29.736267+0000","last_peered":"2024-04-25T01:52:29.736267+0000","last_clean":"2024-04-25T01:52:29.736267+0000","last_became_active":"2024-04-25T01:52:24.515246+0000","last_became_peered":"2024-04-25T01:52:24.515246+0000","last_unstale":"2024-04-25T01:52:29.736267+0000","last_undegraded":"2024-04-25T01:52:29.736267+0000","last_fullsized":"2024-04-25T01:52:29.736267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:14.325747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732403+0000","last_change":"2024-04-25T01:52:25.528796+0000","last_active":"2024-04-25T01:52:27.732403+0000","last_peered":"2024-04-25T01:52:27.732403+0000","last_clean":"2024-04-25T01:52:27.732403+0000","last_became_active":"2024-04-25T01:52:25.528453+0000","last_became_peered":"2024-04-25T01:52:25.528453+0000","last_unstale":"2024-04-25T01:52:27.732403+0000","last_undegraded":"2024-04-25T01:52:27.732403+0000","last_fullsized":"2024-04-25T01:52:27.732403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:29.432519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732524+0000","last_change":"2024-04-25T01:52:24.520229+0000","last_active":"2024-04-25T01:52:27.732524+0000","last_peered":"2024-04-25T01:52:27.732524+0000","last_clean":"2024-04-25T01:52:27.732524+0000","last_became_active":"2024-04-25T01:52:24.519932+0000","last_became_peered":"2024-04-25T01:52:24.519932+0000","last_unstale":"2024-04-25T01:52:27.732524+0000","last_undegraded":"2024-04-25T01:52:27.732524+0000","last_fullsized":"2024-04-25T01:52:27.732524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.786949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.488038+0000","last_change":"2024-04-25T01:52:25.532822+0000","last_active":"2024-04-25T01:52:27.488038+0000","last_peered":"2024-04-25T01:52:27.488038+0000","last_clean":"2024-04-25T01:52:27.488038+0000","last_became_active":"2024-04-25T01:52:25.532462+0000","last_became_peered":"2024-04-25T01:52:25.532462+0000","last_unstale":"2024-04-25T01:52:27.488038+0000","last_undegraded":"2024-04-25T01:52:27.488038+0000","last_fullsized":"2024-04-25T01:52:27.488038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:55.569883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:29.734097+0000","last_change":"2024-04-25T01:52:24.512923+0000","last_active":"2024-04-25T01:52:29.734097+0000","last_peered":"2024-04-25T01:52:29.734097+0000","last_clean":"2024-04-25T01:52:29.734097+0000","last_became_active":"2024-04-25T01:52:24.512604+0000","last_became_peered":"2024-04-25T01:52:24.512604+0000","last_unstale":"2024-04-25T01:52:29.734097+0000","last_undegraded":"2024-04-25T01:52:29.734097+0000","last_fullsized":"2024-04-25T01:52:29.734097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:41.640234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.519837+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:52:26.519837+0000","last_peered":"2024-04-25T01:52:26.519837+0000","last_clean":"2024-04-25T01:52:26.519837+0000","last_became_active":"2024-04-25T01:52:25.526447+0000","last_became_peered":"2024-04-25T01:52:25.526447+0000","last_unstale":"2024-04-25T01:52:26.519837+0000","last_undegraded":"2024-04-25T01:52:26.519837+0000","last_fullsized":"2024-04-25T01:52:26.519837+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:10:18.313830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.522057+0000","last_change":"2024-04-25T01:52:24.511565+0000","last_active":"2024-04-25T01:52:26.522057+0000","last_peered":"2024-04-25T01:52:26.522057+0000","last_clean":"2024-04-25T01:52:26.522057+0000","last_became_active":"2024-04-25T01:52:24.511258+0000","last_became_peered":"2024-04-25T01:52:24.511258+0000","last_unstale":"2024-04-25T01:52:26.522057+0000","last_undegraded":"2024-04-25T01:52:26.522057+0000","last_fullsized":"2024-04-25T01:52:26.522057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:38.733338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793361+0000","last_change":"2024-04-25T01:52:25.527138+0000","last_active":"2024-04-25T01:52:27.793361+0000","last_peered":"2024-04-25T01:52:27.793361+0000","last_clean":"2024-04-25T01:52:27.793361+0000","last_became_active":"2024-04-25T01:52:25.526842+0000","last_became_peered":"2024-04-25T01:52:25.526842+0000","last_unstale":"2024-04-25T01:52:27.793361+0000","last_undegraded":"2024-04-25T01:52:27.793361+0000","last_fullsized":"2024-04-25T01:52:27.793361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:27.305773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.726803+0000","last_change":"2024-04-25T01:52:24.514854+0000","last_active":"2024-04-25T01:52:28.726803+0000","last_peered":"2024-04-25T01:52:28.726803+0000","last_clean":"2024-04-25T01:52:28.726803+0000","last_became_active":"2024-04-25T01:52:24.514548+0000","last_became_peered":"2024-04-25T01:52:24.514548+0000","last_unstale":"2024-04-25T01:52:28.726803+0000","last_undegraded":"2024-04-25T01:52:28.726803+0000","last_fullsized":"2024-04-25T01:52:28.726803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:38.539202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.522069+0000","last_change":"2024-04-25T01:52:25.523888+0000","last_active":"2024-04-25T01:52:26.522069+0000","last_peered":"2024-04-25T01:52:26.522069+0000","last_clean":"2024-04-25T01:52:26.522069+0000","last_became_active":"2024-04-25T01:52:25.523581+0000","last_became_peered":"2024-04-25T01:52:25.523581+0000","last_unstale":"2024-04-25T01:52:26.522069+0000","last_undegraded":"2024-04-25T01:52:26.522069+0000","last_fullsized":"2024-04-25T01:52:26.522069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:59.521757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:30.744219+0000","last_change":"2024-04-25T01:52:24.512774+0000","last_active":"2024-04-25T01:52:30.744219+0000","last_peered":"2024-04-25T01:52:30.744219+0000","last_clean":"2024-04-25T01:52:30.744219+0000","last_became_active":"2024-04-25T01:52:24.512393+0000","last_became_peered":"2024-04-25T01:52:24.512393+0000","last_unstale":"2024-04-25T01:52:30.744219+0000","last_undegraded":"2024-04-25T01:52:30.744219+0000","last_fullsized":"2024-04-25T01:52:30.744219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:41.639961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793080+0000","last_change":"2024-04-25T01:52:25.523381+0000","last_active":"2024-04-25T01:52:27.793080+0000","last_peered":"2024-04-25T01:52:27.793080+0000","last_clean":"2024-04-25T01:52:27.793080+0000","last_became_active":"2024-04-25T01:52:25.522978+0000","last_became_peered":"2024-04-25T01:52:25.522978+0000","last_unstale":"2024-04-25T01:52:27.793080+0000","last_undegraded":"2024-04-25T01:52:27.793080+0000","last_fullsized":"2024-04-25T01:52:27.793080+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:45.702788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517184+0000","last_change":"2024-04-25T01:52:24.521280+0000","last_active":"2024-04-25T01:52:26.517184+0000","last_peered":"2024-04-25T01:52:26.517184+0000","last_clean":"2024-04-25T01:52:26.517184+0000","last_became_active":"2024-04-25T01:52:24.520959+0000","last_became_peered":"2024-04-25T01:52:24.520959+0000","last_unstale":"2024-04-25T01:52:26.517184+0000","last_undegraded":"2024-04-25T01:52:26.517184+0000","last_fullsized":"2024-04-25T01:52:26.517184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:55.037678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.731936+0000","last_change":"2024-04-25T01:52:25.524331+0000","last_active":"2024-04-25T01:52:27.731936+0000","last_peered":"2024-04-25T01:52:27.731936+0000","last_clean":"2024-04-25T01:52:27.731936+0000","last_became_active":"2024-04-25T01:52:25.523431+0000","last_became_peered":"2024-04-25T01:52:25.523431+0000","last_unstale":"2024-04-25T01:52:27.731936+0000","last_undegraded":"2024-04-25T01:52:27.731936+0000","last_fullsized":"2024-04-25T01:52:27.731936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:55.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.792499+0000","last_change":"2024-04-25T01:52:24.515697+0000","last_active":"2024-04-25T01:52:27.792499+0000","last_peered":"2024-04-25T01:52:27.792499+0000","last_clean":"2024-04-25T01:52:27.792499+0000","last_became_active":"2024-04-25T01:52:24.515285+0000","last_became_peered":"2024-04-25T01:52:24.515285+0000","last_unstale":"2024-04-25T01:52:27.792499+0000","last_undegraded":"2024-04-25T01:52:27.792499+0000","last_fullsized":"2024-04-25T01:52:27.792499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:24.249203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.730154+0000","last_change":"2024-04-25T01:52:25.522415+0000","last_active":"2024-04-25T01:52:27.730154+0000","last_peered":"2024-04-25T01:52:27.730154+0000","last_clean":"2024-04-25T01:52:27.730154+0000","last_became_active":"2024-04-25T01:52:25.521270+0000","last_became_peered":"2024-04-25T01:52:25.521270+0000","last_unstale":"2024-04-25T01:52:27.730154+0000","last_undegraded":"2024-04-25T01:52:27.730154+0000","last_fullsized":"2024-04-25T01:52:27.730154+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:53.514169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.723787+0000","last_change":"2024-04-25T01:52:24.509525+0000","last_active":"2024-04-25T01:52:28.723787+0000","last_peered":"2024-04-25T01:52:28.723787+0000","last_clean":"2024-04-25T01:52:28.723787+0000","last_became_active":"2024-04-25T01:52:24.509184+0000","last_became_peered":"2024-04-25T01:52:24.509184+0000","last_unstale":"2024-04-25T01:52:28.723787+0000","last_undegraded":"2024-04-25T01:52:28.723787+0000","last_fullsized":"2024-04-25T01:52:28.723787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:16:43.301401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.520145+0000","last_change":"2024-04-25T01:52:25.525293+0000","last_active":"2024-04-25T01:52:26.520145+0000","last_peered":"2024-04-25T01:52:26.520145+0000","last_clean":"2024-04-25T01:52:26.520145+0000","last_became_active":"2024-04-25T01:52:25.524982+0000","last_became_peered":"2024-04-25T01:52:25.524982+0000","last_unstale":"2024-04-25T01:52:26.520145+0000","last_undegraded":"2024-04-25T01:52:26.520145+0000","last_fullsized":"2024-04-25T01:52:26.520145+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:22.222898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.724420+0000","last_change":"2024-04-25T01:52:24.519834+0000","last_active":"2024-04-25T01:52:28.724420+0000","last_peered":"2024-04-25T01:52:28.724420+0000","last_clean":"2024-04-25T01:52:28.724420+0000","last_became_active":"2024-04-25T01:52:24.519521+0000","last_became_peered":"2024-04-25T01:52:24.519521+0000","last_unstale":"2024-04-25T01:52:28.724420+0000","last_undegraded":"2024-04-25T01:52:28.724420+0000","last_fullsized":"2024-04-25T01:52:28.724420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:02.414527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.488009+0000","last_change":"2024-04-25T01:52:25.523809+0000","last_active":"2024-04-25T01:52:27.488009+0000","last_peered":"2024-04-25T01:52:27.488009+0000","last_clean":"2024-04-25T01:52:27.488009+0000","last_became_active":"2024-04-25T01:52:25.523406+0000","last_became_peered":"2024-04-25T01:52:25.523406+0000","last_unstale":"2024-04-25T01:52:27.488009+0000","last_undegraded":"2024-04-25T01:52:27.488009+0000","last_fullsized":"2024-04-25T01:52:27.488009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:22.169633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793424+0000","last_change":"2024-04-25T01:52:24.510226+0000","last_active":"2024-04-25T01:52:27.793424+0000","last_peered":"2024-04-25T01:52:27.793424+0000","last_clean":"2024-04-25T01:52:27.793424+0000","last_became_active":"2024-04-25T01:52:24.509667+0000","last_became_peered":"2024-04-25T01:52:24.509667+0000","last_unstale":"2024-04-25T01:52:27.793424+0000","last_undegraded":"2024-04-25T01:52:27.793424+0000","last_fullsized":"2024-04-25T01:52:27.793424+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:49.180675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.519894+0000","last_change":"2024-04-25T01:52:25.531567+0000","last_active":"2024-04-25T01:52:26.519894+0000","last_peered":"2024-04-25T01:52:26.519894+0000","last_clean":"2024-04-25T01:52:26.519894+0000","last_became_active":"2024-04-25T01:52:25.531200+0000","last_became_peered":"2024-04-25T01:52:25.531200+0000","last_unstale":"2024-04-25T01:52:26.519894+0000","last_undegraded":"2024-04-25T01:52:26.519894+0000","last_fullsized":"2024-04-25T01:52:26.519894+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:51:14.336413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'17","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:32.130105+0000","last_change":"2024-04-25T01:52:24.512679+0000","last_active":"2024-04-25T01:52:32.130105+0000","last_peered":"2024-04-25T01:52:32.130105+0000","last_clean":"2024-04-25T01:52:32.130105+0000","last_became_active":"2024-04-25T01:52:24.512283+0000","last_became_peered":"2024-04-25T01:52:24.512283+0000","last_unstale":"2024-04-25T01:52:32.130105+0000","last_undegraded":"2024-04-25T01:52:32.130105+0000","last_fullsized":"2024-04-25T01:52:32.130105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:28.045138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20895,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.522210+0000","last_change":"2024-04-25T01:52:25.525931+0000","last_active":"2024-04-25T01:52:26.522210+0000","last_peered":"2024-04-25T01:52:26.522210+0000","last_clean":"2024-04-25T01:52:26.522210+0000","last_became_active":"2024-04-25T01:52:25.525460+0000","last_became_peered":"2024-04-25T01:52:25.525460+0000","last_unstale":"2024-04-25T01:52:26.522210+0000","last_undegraded":"2024-04-25T01:52:26.522210+0000","last_fullsized":"2024-04-25T01:52:26.522210+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:46.569587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.724366+0000","last_change":"2024-04-25T01:52:24.505470+0000","last_active":"2024-04-25T01:52:28.724366+0000","last_peered":"2024-04-25T01:52:28.724366+0000","last_clean":"2024-04-25T01:52:28.724366+0000","last_became_active":"2024-04-25T01:52:24.504677+0000","last_became_peered":"2024-04-25T01:52:24.504677+0000","last_unstale":"2024-04-25T01:52:28.724366+0000","last_undegraded":"2024-04-25T01:52:28.724366+0000","last_fullsized":"2024-04-25T01:52:28.724366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:01.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517150+0000","last_change":"2024-04-25T01:52:25.526364+0000","last_active":"2024-04-25T01:52:26.517150+0000","last_peered":"2024-04-25T01:52:26.517150+0000","last_clean":"2024-04-25T01:52:26.517150+0000","last_became_active":"2024-04-25T01:52:25.525376+0000","last_became_peered":"2024-04-25T01:52:25.525376+0000","last_unstale":"2024-04-25T01:52:26.517150+0000","last_undegraded":"2024-04-25T01:52:26.517150+0000","last_fullsized":"2024-04-25T01:52:26.517150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:26.770994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.726244+0000","last_change":"2024-04-25T01:52:24.520287+0000","last_active":"2024-04-25T01:52:28.726244+0000","last_peered":"2024-04-25T01:52:28.726244+0000","last_clean":"2024-04-25T01:52:28.726244+0000","last_became_active":"2024-04-25T01:52:24.519967+0000","last_became_peered":"2024-04-25T01:52:24.519967+0000","last_unstale":"2024-04-25T01:52:28.726244+0000","last_undegraded":"2024-04-25T01:52:28.726244+0000","last_fullsized":"2024-04-25T01:52:28.726244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:45.078410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377949+0000","last_change":"2024-04-25T01:52:25.525766+0000","last_active":"2024-04-25T01:52:27.377949+0000","last_peered":"2024-04-25T01:52:27.377949+0000","last_clean":"2024-04-25T01:52:27.377949+0000","last_became_active":"2024-04-25T01:52:25.525350+0000","last_became_peered":"2024-04-25T01:52:25.525350+0000","last_unstale":"2024-04-25T01:52:27.377949+0000","last_undegraded":"2024-04-25T01:52:27.377949+0000","last_fullsized":"2024-04-25T01:52:27.377949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:05.387502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.792800+0000","last_change":"2024-04-25T01:52:24.513674+0000","last_active":"2024-04-25T01:52:27.792800+0000","last_peered":"2024-04-25T01:52:27.792800+0000","last_clean":"2024-04-25T01:52:27.792800+0000","last_became_active":"2024-04-25T01:52:24.513270+0000","last_became_peered":"2024-04-25T01:52:24.513270+0000","last_unstale":"2024-04-25T01:52:27.792800+0000","last_undegraded":"2024-04-25T01:52:27.792800+0000","last_fullsized":"2024-04-25T01:52:27.792800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:23.385928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377568+0000","last_change":"2024-04-25T01:52:25.517166+0000","last_active":"2024-04-25T01:52:27.377568+0000","last_peered":"2024-04-25T01:52:27.377568+0000","last_clean":"2024-04-25T01:52:27.377568+0000","last_became_active":"2024-04-25T01:52:25.516853+0000","last_became_peered":"2024-04-25T01:52:25.516853+0000","last_unstale":"2024-04-25T01:52:27.377568+0000","last_undegraded":"2024-04-25T01:52:27.377568+0000","last_fullsized":"2024-04-25T01:52:27.377568+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:18.502558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:29.736794+0000","last_change":"2024-04-25T01:52:24.514695+0000","last_active":"2024-04-25T01:52:29.736794+0000","last_peered":"2024-04-25T01:52:29.736794+0000","last_clean":"2024-04-25T01:52:29.736794+0000","last_became_active":"2024-04-25T01:52:24.514370+0000","last_became_peered":"2024-04-25T01:52:24.514370+0000","last_unstale":"2024-04-25T01:52:29.736794+0000","last_undegraded":"2024-04-25T01:52:29.736794+0000","last_fullsized":"2024-04-25T01:52:29.736794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:28:28.537307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.792696+0000","last_change":"2024-04-25T01:52:25.527062+0000","last_active":"2024-04-25T01:52:27.792696+0000","last_peered":"2024-04-25T01:52:27.792696+0000","last_clean":"2024-04-25T01:52:27.792696+0000","last_became_active":"2024-04-25T01:52:25.526757+0000","last_became_peered":"2024-04-25T01:52:25.526757+0000","last_unstale":"2024-04-25T01:52:27.792696+0000","last_undegraded":"2024-04-25T01:52:27.792696+0000","last_fullsized":"2024-04-25T01:52:27.792696+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:36.360030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.522155+0000","last_change":"2024-04-25T01:52:24.512235+0000","last_active":"2024-04-25T01:52:26.522155+0000","last_peered":"2024-04-25T01:52:26.522155+0000","last_clean":"2024-04-25T01:52:26.522155+0000","last_became_active":"2024-04-25T01:52:24.511910+0000","last_became_peered":"2024-04-25T01:52:24.511910+0000","last_unstale":"2024-04-25T01:52:26.522155+0000","last_undegraded":"2024-04-25T01:52:26.522155+0000","last_fullsized":"2024-04-25T01:52:26.522155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:38.435734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988402+0000","last_change":"2024-04-25T01:52:25.527854+0000","last_active":"2024-04-25T01:52:27.988402+0000","last_peered":"2024-04-25T01:52:27.988402+0000","last_clean":"2024-04-25T01:52:27.988402+0000","last_became_active":"2024-04-25T01:52:25.527481+0000","last_became_peered":"2024-04-25T01:52:25.527481+0000","last_unstale":"2024-04-25T01:52:27.988402+0000","last_undegraded":"2024-04-25T01:52:27.988402+0000","last_fullsized":"2024-04-25T01:52:27.988402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:09:11.281982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.730195+0000","last_change":"2024-04-25T01:52:24.513463+0000","last_active":"2024-04-25T01:52:27.730195+0000","last_peered":"2024-04-25T01:52:27.730195+0000","last_clean":"2024-04-25T01:52:27.730195+0000","last_became_active":"2024-04-25T01:52:24.513176+0000","last_became_peered":"2024-04-25T01:52:24.513176+0000","last_unstale":"2024-04-25T01:52:27.730195+0000","last_undegraded":"2024-04-25T01:52:27.730195+0000","last_fullsized":"2024-04-25T01:52:27.730195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:13:33.507559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.588800+0000","last_change":"2024-04-25T01:52:25.528316+0000","last_active":"2024-04-25T01:52:27.588800+0000","last_peered":"2024-04-25T01:52:27.588800+0000","last_clean":"2024-04-25T01:52:27.588800+0000","last_became_active":"2024-04-25T01:52:25.527991+0000","last_became_peered":"2024-04-25T01:52:25.527991+0000","last_unstale":"2024-04-25T01:52:27.588800+0000","last_undegraded":"2024-04-25T01:52:27.588800+0000","last_fullsized":"2024-04-25T01:52:27.588800+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:55.871663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377711+0000","last_change":"2024-04-25T01:52:24.519357+0000","last_active":"2024-04-25T01:52:27.377711+0000","last_peered":"2024-04-25T01:52:27.377711+0000","last_clean":"2024-04-25T01:52:27.377711+0000","last_became_active":"2024-04-25T01:52:24.519025+0000","last_became_peered":"2024-04-25T01:52:24.519025+0000","last_unstale":"2024-04-25T01:52:27.377711+0000","last_undegraded":"2024-04-25T01:52:27.377711+0000","last_fullsized":"2024-04-25T01:52:27.377711+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:03.476554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.007801+0000","last_change":"2024-04-25T01:52:25.527922+0000","last_active":"2024-04-25T01:52:27.007801+0000","last_peered":"2024-04-25T01:52:27.007801+0000","last_clean":"2024-04-25T01:52:27.007801+0000","last_became_active":"2024-04-25T01:52:25.527596+0000","last_became_peered":"2024-04-25T01:52:25.527596+0000","last_unstale":"2024-04-25T01:52:27.007801+0000","last_undegraded":"2024-04-25T01:52:27.007801+0000","last_fullsized":"2024-04-25T01:52:27.007801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:56.002208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.792833+0000","last_change":"2024-04-25T01:52:25.533800+0000","last_active":"2024-04-25T01:52:27.792833+0000","last_peered":"2024-04-25T01:52:27.792833+0000","last_clean":"2024-04-25T01:52:27.792833+0000","last_became_active":"2024-04-25T01:52:25.533484+0000","last_became_peered":"2024-04-25T01:52:25.533484+0000","last_unstale":"2024-04-25T01:52:27.792833+0000","last_undegraded":"2024-04-25T01:52:27.792833+0000","last_fullsized":"2024-04-25T01:52:27.792833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:13.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517111+0000","last_change":"2024-04-25T01:52:24.506068+0000","last_active":"2024-04-25T01:52:26.517111+0000","last_peered":"2024-04-25T01:52:26.517111+0000","last_clean":"2024-04-25T01:52:26.517111+0000","last_became_active":"2024-04-25T01:52:24.505667+0000","last_became_peered":"2024-04-25T01:52:24.505667+0000","last_unstale":"2024-04-25T01:52:26.517111+0000","last_undegraded":"2024-04-25T01:52:26.517111+0000","last_fullsized":"2024-04-25T01:52:26.517111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:30.494376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732361+0000","last_change":"2024-04-25T01:52:25.530681+0000","last_active":"2024-04-25T01:52:27.732361+0000","last_peered":"2024-04-25T01:52:27.732361+0000","last_clean":"2024-04-25T01:52:27.732361+0000","last_became_active":"2024-04-25T01:52:25.529035+0000","last_became_peered":"2024-04-25T01:52:25.529035+0000","last_unstale":"2024-04-25T01:52:27.732361+0000","last_undegraded":"2024-04-25T01:52:27.732361+0000","last_fullsized":"2024-04-25T01:52:27.732361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:28.787180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.724985+0000","last_change":"2024-04-25T01:52:24.508948+0000","last_active":"2024-04-25T01:52:28.724985+0000","last_peered":"2024-04-25T01:52:28.724985+0000","last_clean":"2024-04-25T01:52:28.724985+0000","last_became_active":"2024-04-25T01:52:24.508563+0000","last_became_peered":"2024-04-25T01:52:24.508563+0000","last_unstale":"2024-04-25T01:52:28.724985+0000","last_undegraded":"2024-04-25T01:52:28.724985+0000","last_fullsized":"2024-04-25T01:52:28.724985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:37.591783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377547+0000","last_change":"2024-04-25T01:51:15.621251+0000","last_active":"2024-04-25T01:52:27.377547+0000","last_peered":"2024-04-25T01:52:27.377547+0000","last_clean":"2024-04-25T01:52:27.377547+0000","last_became_active":"2024-04-25T01:51:15.620771+0000","last_became_peered":"2024-04-25T01:51:15.620771+0000","last_unstale":"2024-04-25T01:52:27.377547+0000","last_undegraded":"2024-04-25T01:52:27.377547+0000","last_fullsized":"2024-04-25T01:52:27.377547+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_clean_scrub_stamp":"2024-04-25T01:47:50.544829+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:44.048065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517069+0000","last_change":"2024-04-25T01:52:25.530216+0000","last_active":"2024-04-25T01:52:26.517069+0000","last_peered":"2024-04-25T01:52:26.517069+0000","last_clean":"2024-04-25T01:52:26.517069+0000","last_became_active":"2024-04-25T01:52:25.529813+0000","last_became_peered":"2024-04-25T01:52:25.529813+0000","last_unstale":"2024-04-25T01:52:26.517069+0000","last_undegraded":"2024-04-25T01:52:26.517069+0000","last_fullsized":"2024-04-25T01:52:26.517069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:57.656375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:29.735263+0000","last_change":"2024-04-25T01:52:24.517069+0000","last_active":"2024-04-25T01:52:29.735263+0000","last_peered":"2024-04-25T01:52:29.735263+0000","last_clean":"2024-04-25T01:52:29.735263+0000","last_became_active":"2024-04-25T01:52:24.516732+0000","last_became_peered":"2024-04-25T01:52:24.516732+0000","last_unstale":"2024-04-25T01:52:29.735263+0000","last_undegraded":"2024-04-25T01:52:29.735263+0000","last_fullsized":"2024-04-25T01:52:29.735263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:58.546314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.522221+0000","last_change":"2024-04-25T01:52:24.514786+0000","last_active":"2024-04-25T01:52:26.522221+0000","last_peered":"2024-04-25T01:52:26.522221+0000","last_clean":"2024-04-25T01:52:26.522221+0000","last_became_active":"2024-04-25T01:52:24.514486+0000","last_became_peered":"2024-04-25T01:52:24.514486+0000","last_unstale":"2024-04-25T01:52:26.522221+0000","last_undegraded":"2024-04-25T01:52:26.522221+0000","last_fullsized":"2024-04-25T01:52:26.522221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:42.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517462+0000","last_change":"2024-04-25T01:52:25.526518+0000","last_active":"2024-04-25T01:52:26.517462+0000","last_peered":"2024-04-25T01:52:26.517462+0000","last_clean":"2024-04-25T01:52:26.517462+0000","last_became_active":"2024-04-25T01:52:25.525917+0000","last_became_peered":"2024-04-25T01:52:25.525917+0000","last_unstale":"2024-04-25T01:52:26.517462+0000","last_undegraded":"2024-04-25T01:52:26.517462+0000","last_fullsized":"2024-04-25T01:52:26.517462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:20.776385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:29.734537+0000","last_change":"2024-04-25T01:52:24.511561+0000","last_active":"2024-04-25T01:52:29.734537+0000","last_peered":"2024-04-25T01:52:29.734537+0000","last_clean":"2024-04-25T01:52:29.734537+0000","last_became_active":"2024-04-25T01:52:24.511165+0000","last_became_peered":"2024-04-25T01:52:24.511165+0000","last_unstale":"2024-04-25T01:52:29.734537+0000","last_undegraded":"2024-04-25T01:52:29.734537+0000","last_fullsized":"2024-04-25T01:52:29.734537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:27:41.006137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732445+0000","last_change":"2024-04-25T01:52:25.527991+0000","last_active":"2024-04-25T01:52:27.732445+0000","last_peered":"2024-04-25T01:52:27.732445+0000","last_clean":"2024-04-25T01:52:27.732445+0000","last_became_active":"2024-04-25T01:52:25.527677+0000","last_became_peered":"2024-04-25T01:52:25.527677+0000","last_unstale":"2024-04-25T01:52:27.732445+0000","last_undegraded":"2024-04-25T01:52:27.732445+0000","last_fullsized":"2024-04-25T01:52:27.732445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:07:24.698774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.725108+0000","last_change":"2024-04-25T01:52:24.523536+0000","last_active":"2024-04-25T01:52:28.725108+0000","last_peered":"2024-04-25T01:52:28.725108+0000","last_clean":"2024-04-25T01:52:28.725108+0000","last_became_active":"2024-04-25T01:52:24.523233+0000","last_became_peered":"2024-04-25T01:52:24.523233+0000","last_unstale":"2024-04-25T01:52:28.725108+0000","last_undegraded":"2024-04-25T01:52:28.725108+0000","last_fullsized":"2024-04-25T01:52:28.725108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:49:44.444968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487949+0000","last_change":"2024-04-25T01:52:25.527090+0000","last_active":"2024-04-25T01:52:27.487949+0000","last_peered":"2024-04-25T01:52:27.487949+0000","last_clean":"2024-04-25T01:52:27.487949+0000","last_became_active":"2024-04-25T01:52:25.526771+0000","last_became_peered":"2024-04-25T01:52:25.526771+0000","last_unstale":"2024-04-25T01:52:27.487949+0000","last_undegraded":"2024-04-25T01:52:27.487949+0000","last_fullsized":"2024-04-25T01:52:27.487949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:58.758748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.725563+0000","last_change":"2024-04-25T01:52:24.516469+0000","last_active":"2024-04-25T01:52:28.725563+0000","last_peered":"2024-04-25T01:52:28.725563+0000","last_clean":"2024-04-25T01:52:28.725563+0000","last_became_active":"2024-04-25T01:52:24.515948+0000","last_became_peered":"2024-04-25T01:52:24.515948+0000","last_unstale":"2024-04-25T01:52:28.725563+0000","last_undegraded":"2024-04-25T01:52:28.725563+0000","last_fullsized":"2024-04-25T01:52:28.725563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:25.122153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.521898+0000","last_change":"2024-04-25T01:52:25.524967+0000","last_active":"2024-04-25T01:52:26.521898+0000","last_peered":"2024-04-25T01:52:26.521898+0000","last_clean":"2024-04-25T01:52:26.521898+0000","last_became_active":"2024-04-25T01:52:25.524670+0000","last_became_peered":"2024-04-25T01:52:25.524670+0000","last_unstale":"2024-04-25T01:52:26.521898+0000","last_undegraded":"2024-04-25T01:52:26.521898+0000","last_fullsized":"2024-04-25T01:52:26.521898+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:08.829052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:31.750420+0000","last_change":"2024-04-25T01:52:24.512187+0000","last_active":"2024-04-25T01:52:31.750420+0000","last_peered":"2024-04-25T01:52:31.750420+0000","last_clean":"2024-04-25T01:52:31.750420+0000","last_became_active":"2024-04-25T01:52:24.510303+0000","last_became_peered":"2024-04-25T01:52:24.510303+0000","last_unstale":"2024-04-25T01:52:31.750420+0000","last_undegraded":"2024-04-25T01:52:31.750420+0000","last_fullsized":"2024-04-25T01:52:31.750420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:42:31.596355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988098+0000","last_change":"2024-04-25T01:52:25.530386+0000","last_active":"2024-04-25T01:52:27.988098+0000","last_peered":"2024-04-25T01:52:27.988098+0000","last_clean":"2024-04-25T01:52:27.988098+0000","last_became_active":"2024-04-25T01:52:25.530034+0000","last_became_peered":"2024-04-25T01:52:25.530034+0000","last_unstale":"2024-04-25T01:52:27.988098+0000","last_undegraded":"2024-04-25T01:52:27.988098+0000","last_fullsized":"2024-04-25T01:52:27.988098+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:11.470123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793046+0000","last_change":"2024-04-25T01:52:24.511881+0000","last_active":"2024-04-25T01:52:27.793046+0000","last_peered":"2024-04-25T01:52:27.793046+0000","last_clean":"2024-04-25T01:52:27.793046+0000","last_became_active":"2024-04-25T01:52:24.511226+0000","last_became_peered":"2024-04-25T01:52:24.511226+0000","last_unstale":"2024-04-25T01:52:27.793046+0000","last_undegraded":"2024-04-25T01:52:27.793046+0000","last_fullsized":"2024-04-25T01:52:27.793046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:49:10.038555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732522+0000","last_change":"2024-04-25T01:52:25.525695+0000","last_active":"2024-04-25T01:52:27.732522+0000","last_peered":"2024-04-25T01:52:27.732522+0000","last_clean":"2024-04-25T01:52:27.732522+0000","last_became_active":"2024-04-25T01:52:25.524830+0000","last_became_peered":"2024-04-25T01:52:25.524830+0000","last_unstale":"2024-04-25T01:52:27.732522+0000","last_undegraded":"2024-04-25T01:52:27.732522+0000","last_fullsized":"2024-04-25T01:52:27.732522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:04:58.449350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.725260+0000","last_change":"2024-04-25T01:52:24.519232+0000","last_active":"2024-04-25T01:52:28.725260+0000","last_peered":"2024-04-25T01:52:28.725260+0000","last_clean":"2024-04-25T01:52:28.725260+0000","last_became_active":"2024-04-25T01:52:24.518902+0000","last_became_peered":"2024-04-25T01:52:24.518902+0000","last_unstale":"2024-04-25T01:52:28.725260+0000","last_undegraded":"2024-04-25T01:52:28.725260+0000","last_fullsized":"2024-04-25T01:52:28.725260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.173730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377940+0000","last_change":"2024-04-25T01:52:25.522955+0000","last_active":"2024-04-25T01:52:27.377940+0000","last_peered":"2024-04-25T01:52:27.377940+0000","last_clean":"2024-04-25T01:52:27.377940+0000","last_became_active":"2024-04-25T01:52:25.522626+0000","last_became_peered":"2024-04-25T01:52:25.522626+0000","last_unstale":"2024-04-25T01:52:27.377940+0000","last_undegraded":"2024-04-25T01:52:27.377940+0000","last_fullsized":"2024-04-25T01:52:27.377940+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:47.218461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793443+0000","last_change":"2024-04-25T01:52:24.516383+0000","last_active":"2024-04-25T01:52:27.793443+0000","last_peered":"2024-04-25T01:52:27.793443+0000","last_clean":"2024-04-25T01:52:27.793443+0000","last_became_active":"2024-04-25T01:52:24.515096+0000","last_became_peered":"2024-04-25T01:52:24.515096+0000","last_unstale":"2024-04-25T01:52:27.793443+0000","last_undegraded":"2024-04-25T01:52:27.793443+0000","last_fullsized":"2024-04-25T01:52:27.793443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:52.658161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.378280+0000","last_change":"2024-04-25T01:52:25.524874+0000","last_active":"2024-04-25T01:52:27.378280+0000","last_peered":"2024-04-25T01:52:27.378280+0000","last_clean":"2024-04-25T01:52:27.378280+0000","last_became_active":"2024-04-25T01:52:25.523532+0000","last_became_peered":"2024-04-25T01:52:25.523532+0000","last_unstale":"2024-04-25T01:52:27.378280+0000","last_undegraded":"2024-04-25T01:52:27.378280+0000","last_fullsized":"2024-04-25T01:52:27.378280+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:10.746446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793126+0000","last_change":"2024-04-25T01:52:24.526212+0000","last_active":"2024-04-25T01:52:27.793126+0000","last_peered":"2024-04-25T01:52:27.793126+0000","last_clean":"2024-04-25T01:52:27.793126+0000","last_became_active":"2024-04-25T01:52:24.525882+0000","last_became_peered":"2024-04-25T01:52:24.525882+0000","last_unstale":"2024-04-25T01:52:27.793126+0000","last_undegraded":"2024-04-25T01:52:27.793126+0000","last_fullsized":"2024-04-25T01:52:27.793126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:04.792560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377867+0000","last_change":"2024-04-25T01:52:25.517387+0000","last_active":"2024-04-25T01:52:27.377867+0000","last_peered":"2024-04-25T01:52:27.377867+0000","last_clean":"2024-04-25T01:52:27.377867+0000","last_became_active":"2024-04-25T01:52:25.517047+0000","last_became_peered":"2024-04-25T01:52:25.517047+0000","last_unstale":"2024-04-25T01:52:27.377867+0000","last_undegraded":"2024-04-25T01:52:27.377867+0000","last_fullsized":"2024-04-25T01:52:27.377867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:39.462474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517458+0000","last_change":"2024-04-25T01:52:24.506510+0000","last_active":"2024-04-25T01:52:26.517458+0000","last_peered":"2024-04-25T01:52:26.517458+0000","last_clean":"2024-04-25T01:52:26.517458+0000","last_became_active":"2024-04-25T01:52:24.506184+0000","last_became_peered":"2024-04-25T01:52:24.506184+0000","last_unstale":"2024-04-25T01:52:26.517458+0000","last_undegraded":"2024-04-25T01:52:26.517458+0000","last_fullsized":"2024-04-25T01:52:26.517458+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:56.754386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.378115+0000","last_change":"2024-04-25T01:52:25.525197+0000","last_active":"2024-04-25T01:52:27.378115+0000","last_peered":"2024-04-25T01:52:27.378115+0000","last_clean":"2024-04-25T01:52:27.378115+0000","last_became_active":"2024-04-25T01:52:25.524692+0000","last_became_peered":"2024-04-25T01:52:25.524692+0000","last_unstale":"2024-04-25T01:52:27.378115+0000","last_undegraded":"2024-04-25T01:52:27.378115+0000","last_fullsized":"2024-04-25T01:52:27.378115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.264227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988215+0000","last_change":"2024-04-25T01:52:25.531060+0000","last_active":"2024-04-25T01:52:27.988215+0000","last_peered":"2024-04-25T01:52:27.988215+0000","last_clean":"2024-04-25T01:52:27.988215+0000","last_became_active":"2024-04-25T01:52:25.530774+0000","last_became_peered":"2024-04-25T01:52:25.530774+0000","last_unstale":"2024-04-25T01:52:27.988215+0000","last_undegraded":"2024-04-25T01:52:27.988215+0000","last_fullsized":"2024-04-25T01:52:27.988215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:32.168699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793070+0000","last_change":"2024-04-25T01:52:25.535381+0000","last_active":"2024-04-25T01:52:27.793070+0000","last_peered":"2024-04-25T01:52:27.793070+0000","last_clean":"2024-04-25T01:52:27.793070+0000","last_became_active":"2024-04-25T01:52:25.533623+0000","last_became_peered":"2024-04-25T01:52:25.533623+0000","last_unstale":"2024-04-25T01:52:27.793070+0000","last_undegraded":"2024-04-25T01:52:27.793070+0000","last_fullsized":"2024-04-25T01:52:27.793070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:14.032508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793201+0000","last_change":"2024-04-25T01:52:25.528803+0000","last_active":"2024-04-25T01:52:27.793201+0000","last_peered":"2024-04-25T01:52:27.793201+0000","last_clean":"2024-04-25T01:52:27.793201+0000","last_became_active":"2024-04-25T01:52:25.528505+0000","last_became_peered":"2024-04-25T01:52:25.528505+0000","last_unstale":"2024-04-25T01:52:27.793201+0000","last_undegraded":"2024-04-25T01:52:27.793201+0000","last_fullsized":"2024-04-25T01:52:27.793201+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:35.811181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487769+0000","last_change":"2024-04-25T01:52:25.523620+0000","last_active":"2024-04-25T01:52:27.487769+0000","last_peered":"2024-04-25T01:52:27.487769+0000","last_clean":"2024-04-25T01:52:27.487769+0000","last_became_active":"2024-04-25T01:52:25.523007+0000","last_became_peered":"2024-04-25T01:52:25.523007+0000","last_unstale":"2024-04-25T01:52:27.487769+0000","last_undegraded":"2024-04-25T01:52:27.487769+0000","last_fullsized":"2024-04-25T01:52:27.487769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:23.587838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517315+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:52:26.517315+0000","last_peered":"2024-04-25T01:52:26.517315+0000","last_clean":"2024-04-25T01:52:26.517315+0000","last_became_active":"2024-04-25T01:52:25.526398+0000","last_became_peered":"2024-04-25T01:52:25.526398+0000","last_unstale":"2024-04-25T01:52:26.517315+0000","last_undegraded":"2024-04-25T01:52:26.517315+0000","last_fullsized":"2024-04-25T01:52:26.517315+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:01:37.286676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988266+0000","last_change":"2024-04-25T01:52:25.524999+0000","last_active":"2024-04-25T01:52:27.988266+0000","last_peered":"2024-04-25T01:52:27.988266+0000","last_clean":"2024-04-25T01:52:27.988266+0000","last_became_active":"2024-04-25T01:52:25.524652+0000","last_became_peered":"2024-04-25T01:52:25.524652+0000","last_unstale":"2024-04-25T01:52:27.988266+0000","last_undegraded":"2024-04-25T01:52:27.988266+0000","last_fullsized":"2024-04-25T01:52:27.988266+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:47:37.463701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793710+0000","last_change":"2024-04-25T01:52:25.532996+0000","last_active":"2024-04-25T01:52:27.793710+0000","last_peered":"2024-04-25T01:52:27.793710+0000","last_clean":"2024-04-25T01:52:27.793710+0000","last_became_active":"2024-04-25T01:52:25.530230+0000","last_became_peered":"2024-04-25T01:52:25.530230+0000","last_unstale":"2024-04-25T01:52:27.793710+0000","last_undegraded":"2024-04-25T01:52:27.793710+0000","last_fullsized":"2024-04-25T01:52:27.793710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:08.996434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732256+0000","last_change":"2024-04-25T01:52:25.523838+0000","last_active":"2024-04-25T01:52:27.732256+0000","last_peered":"2024-04-25T01:52:27.732256+0000","last_clean":"2024-04-25T01:52:27.732256+0000","last_became_active":"2024-04-25T01:52:25.523297+0000","last_became_peered":"2024-04-25T01:52:25.523297+0000","last_unstale":"2024-04-25T01:52:27.732256+0000","last_undegraded":"2024-04-25T01:52:27.732256+0000","last_fullsized":"2024-04-25T01:52:27.732256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:52.938763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.792550+0000","last_change":"2024-04-25T01:52:25.519028+0000","last_active":"2024-04-25T01:52:27.792550+0000","last_peered":"2024-04-25T01:52:27.792550+0000","last_clean":"2024-04-25T01:52:27.792550+0000","last_became_active":"2024-04-25T01:52:25.518716+0000","last_became_peered":"2024-04-25T01:52:25.518716+0000","last_unstale":"2024-04-25T01:52:27.792550+0000","last_undegraded":"2024-04-25T01:52:27.792550+0000","last_fullsized":"2024-04-25T01:52:27.792550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:37:31.573403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732042+0000","last_change":"2024-04-25T01:52:25.533894+0000","last_active":"2024-04-25T01:52:27.732042+0000","last_peered":"2024-04-25T01:52:27.732042+0000","last_clean":"2024-04-25T01:52:27.732042+0000","last_became_active":"2024-04-25T01:52:25.533510+0000","last_became_peered":"2024-04-25T01:52:25.533510+0000","last_unstale":"2024-04-25T01:52:27.732042+0000","last_undegraded":"2024-04-25T01:52:27.732042+0000","last_fullsized":"2024-04-25T01:52:27.732042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:07.128296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.519798+0000","last_change":"2024-04-25T01:52:25.524693+0000","last_active":"2024-04-25T01:52:26.519798+0000","last_peered":"2024-04-25T01:52:26.519798+0000","last_clean":"2024-04-25T01:52:26.519798+0000","last_became_active":"2024-04-25T01:52:25.524366+0000","last_became_peered":"2024-04-25T01:52:25.524366+0000","last_unstale":"2024-04-25T01:52:26.519798+0000","last_undegraded":"2024-04-25T01:52:26.519798+0000","last_fullsized":"2024-04-25T01:52:26.519798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:23.133722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517377+0000","last_change":"2024-04-25T01:52:25.525936+0000","last_active":"2024-04-25T01:52:26.517377+0000","last_peered":"2024-04-25T01:52:26.517377+0000","last_clean":"2024-04-25T01:52:26.517377+0000","last_became_active":"2024-04-25T01:52:25.525586+0000","last_became_peered":"2024-04-25T01:52:25.525586+0000","last_unstale":"2024-04-25T01:52:26.517377+0000","last_undegraded":"2024-04-25T01:52:26.517377+0000","last_fullsized":"2024-04-25T01:52:26.517377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:34.756898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988207+0000","last_change":"2024-04-25T01:52:25.528704+0000","last_active":"2024-04-25T01:52:27.988207+0000","last_peered":"2024-04-25T01:52:27.988207+0000","last_clean":"2024-04-25T01:52:27.988207+0000","last_became_active":"2024-04-25T01:52:25.528379+0000","last_became_peered":"2024-04-25T01:52:25.528379+0000","last_unstale":"2024-04-25T01:52:27.988207+0000","last_undegraded":"2024-04-25T01:52:27.988207+0000","last_fullsized":"2024-04-25T01:52:27.988207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:43.453698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280293,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":23301,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":36409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318285,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":451252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579923,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":1063176,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743446,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":450487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005085,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":450295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":999424,"data_stored":1041241,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":449669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":451281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":451147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":451315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":450333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":449691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":999424,"data_stored":1040219,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":23022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:52:34.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.370+0000 7f84604b3640 1 -- 172.21.15.31:0/1909821054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8458101870 msgr2=0x7f8458101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.370+0000 7f84604b3640 1 --2- 172.21.15.31:0/1909821054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8458101870 0x7f8458101cf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8454009900 tx=0x7f845402a410 comp rx=0 tx=0).stop 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 -- 172.21.15.31:0/1909821054 shutdown_connections 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 --2- 172.21.15.31:0/1909821054 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8458102230 0x7f8458108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 --2- 172.21.15.31:0/1909821054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8458101870 0x7f8458101cf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 --2- 172.21.15.31:0/1909821054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8458100680 0x7f8458100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 -- 172.21.15.31:0/1909821054 >> 172.21.15.31:0/1909821054 conn(0x7f84580fbbf0 msgr2=0x7f84580fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 -- 172.21.15.31:0/1909821054 shutdown_connections 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 -- 172.21.15.31:0/1909821054 wait complete. 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 Processor -- start 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 -- start start 2024-04-25T01:52:34.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8458100680 0x7f84581968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8458101870 0x7f8458196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8458102230 0x7f845819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f845810c1b0 con 0x7f8458100680 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f845810c030 con 0x7f8458102230 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f84604b3640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f845810c330 con 0x7f8458101870 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.371+0000 7f845e24f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8458100680 0x7f84581968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f845e24f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8458100680 0x7f84581968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38066/0 (socket says 172.21.15.31:38066) 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f845e24f640 1 -- 172.21.15.31:0/2776429096 learned_addr learned my addr 172.21.15.31:0/2776429096 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f845ea50640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8458102230 0x7f845819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f845da4e640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8458101870 0x7f8458196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f845e24f640 1 -- 172.21.15.31:0/2776429096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8458101870 msgr2=0x7f8458196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f845e24f640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8458101870 0x7f8458196e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f845e24f640 1 -- 172.21.15.31:0/2776429096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8458102230 msgr2=0x7f845819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f845e24f640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8458102230 0x7f845819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f845e24f640 1 -- 172.21.15.31:0/2776429096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845819e510 con 0x7f8458100680 2024-04-25T01:52:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f845e24f640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8458100680 0x7f84581968f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8448009860 tx=0x7f8448009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f844f7fe640 1 -- 172.21.15.31:0/2776429096 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8448004020 con 0x7f8458100680 2024-04-25T01:52:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f844f7fe640 1 -- 172.21.15.31:0/2776429096 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8448007c50 con 0x7f8458100680 2024-04-25T01:52:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f84604b3640 1 -- 172.21.15.31:0/2776429096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8458068930 con 0x7f8458100680 2024-04-25T01:52:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f844f7fe640 1 -- 172.21.15.31:0/2776429096 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8448003ab0 con 0x7f8458100680 2024-04-25T01:52:34.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.372+0000 7f84604b3640 1 -- 172.21.15.31:0/2776429096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8458068e80 con 0x7f8458100680 2024-04-25T01:52:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.373+0000 7f84604b3640 1 -- 172.21.15.31:0/2776429096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8458068560 con 0x7f8458100680 2024-04-25T01:52:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.374+0000 7f844f7fe640 1 -- 172.21.15.31:0/2776429096 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f84480041c0 con 0x7f8458100680 2024-04-25T01:52:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.375+0000 7f844f7fe640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f84440a2c20 0x7f84440a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.375+0000 7f844f7fe640 1 -- 172.21.15.31:0/2776429096 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f84480d0510 con 0x7f8458100680 2024-04-25T01:52:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.375+0000 7f845da4e640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f84440a2c20 0x7f84440a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.375+0000 7f845da4e640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f84440a2c20 0x7f84440a50e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f845402a930 tx=0x7f8454000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:34.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.379+0000 7f844f7fe640 1 -- 172.21.15.31:0/2776429096 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f844809d3c0 con 0x7f8458100680 2024-04-25T01:52:34.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.561+0000 7f84604b3640 1 -- 172.21.15.31:0/2776429096 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f84580690e0 con 0x7f84440a2c20 2024-04-25T01:52:34.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.568+0000 7f844f7fe640 1 -- 172.21.15.31:0/2776429096 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258938 (secure 0 0 0) 0x7f84580690e0 con 0x7f84440a2c20 2024-04-25T01:52:34.571 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:34.573 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T01:52:34.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 -- 172.21.15.31:0/2776429096 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f84440a2c20 msgr2=0x7f84440a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f84440a2c20 0x7f84440a50e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f845402a930 tx=0x7f8454000bc0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 -- 172.21.15.31:0/2776429096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8458100680 msgr2=0x7f84581968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8458100680 0x7f84581968f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8448009860 tx=0x7f8448009d20 comp rx=0 tx=0).stop 2024-04-25T01:52:34.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 -- 172.21.15.31:0/2776429096 shutdown_connections 2024-04-25T01:52:34.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f84440a2c20 0x7f84440a50e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8458102230 0x7f845819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8458101870 0x7f8458196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 --2- 172.21.15.31:0/2776429096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8458100680 0x7f84581968f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 -- 172.21.15.31:0/2776429096 >> 172.21.15.31:0/2776429096 conn(0x7f84580fbbf0 msgr2=0x7f8458104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:34.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 -- 172.21.15.31:0/2776429096 shutdown_connections 2024-04-25T01:52:34.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:34.571+0000 7f84604b3640 1 -- 172.21.15.31:0/2776429096 wait complete. 2024-04-25T01:52:34.582 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-04-25T01:52:33.732097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":613786,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":311,"num_write_kb":2995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368868,"kb_used_data":11376,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572956,"statfs":{"total":1151940427776,"available":1151562706944,"internally_reserved":0,"allocated":11649024,"data_stored":7200106,"data_compressed":106956,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23418,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.224344"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988531+0000","last_change":"2024-04-25T01:52:25.523199+0000","last_active":"2024-04-25T01:52:27.988531+0000","last_peered":"2024-04-25T01:52:27.988531+0000","last_clean":"2024-04-25T01:52:27.988531+0000","last_became_active":"2024-04-25T01:52:25.522052+0000","last_became_peered":"2024-04-25T01:52:25.522052+0000","last_unstale":"2024-04-25T01:52:27.988531+0000","last_undegraded":"2024-04-25T01:52:27.988531+0000","last_fullsized":"2024-04-25T01:52:27.988531+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:07:27.095370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487912+0000","last_change":"2024-04-25T01:52:25.528431+0000","last_active":"2024-04-25T01:52:27.487912+0000","last_peered":"2024-04-25T01:52:27.487912+0000","last_clean":"2024-04-25T01:52:27.487912+0000","last_became_active":"2024-04-25T01:52:25.528122+0000","last_became_peered":"2024-04-25T01:52:25.528122+0000","last_unstale":"2024-04-25T01:52:27.487912+0000","last_undegraded":"2024-04-25T01:52:27.487912+0000","last_fullsized":"2024-04-25T01:52:27.487912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:26:56.794582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487847+0000","last_change":"2024-04-25T01:52:25.532963+0000","last_active":"2024-04-25T01:52:27.487847+0000","last_peered":"2024-04-25T01:52:27.487847+0000","last_clean":"2024-04-25T01:52:27.487847+0000","last_became_active":"2024-04-25T01:52:25.532770+0000","last_became_peered":"2024-04-25T01:52:25.532770+0000","last_unstale":"2024-04-25T01:52:27.487847+0000","last_undegraded":"2024-04-25T01:52:27.487847+0000","last_fullsized":"2024-04-25T01:52:27.487847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:13.398868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793349+0000","last_change":"2024-04-25T01:52:25.533032+0000","last_active":"2024-04-25T01:52:27.793349+0000","last_peered":"2024-04-25T01:52:27.793349+0000","last_clean":"2024-04-25T01:52:27.793349+0000","last_became_active":"2024-04-25T01:52:25.530437+0000","last_became_peered":"2024-04-25T01:52:25.530437+0000","last_unstale":"2024-04-25T01:52:27.793349+0000","last_undegraded":"2024-04-25T01:52:27.793349+0000","last_fullsized":"2024-04-25T01:52:27.793349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:48.526222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.521921+0000","last_change":"2024-04-25T01:52:25.523624+0000","last_active":"2024-04-25T01:52:26.521921+0000","last_peered":"2024-04-25T01:52:26.521921+0000","last_clean":"2024-04-25T01:52:26.521921+0000","last_became_active":"2024-04-25T01:52:25.523289+0000","last_became_peered":"2024-04-25T01:52:25.523289+0000","last_unstale":"2024-04-25T01:52:26.521921+0000","last_undegraded":"2024-04-25T01:52:26.521921+0000","last_fullsized":"2024-04-25T01:52:26.521921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:09.688472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.007733+0000","last_change":"2024-04-25T01:52:25.527123+0000","last_active":"2024-04-25T01:52:27.007733+0000","last_peered":"2024-04-25T01:52:27.007733+0000","last_clean":"2024-04-25T01:52:27.007733+0000","last_became_active":"2024-04-25T01:52:25.526808+0000","last_became_peered":"2024-04-25T01:52:25.526808+0000","last_unstale":"2024-04-25T01:52:27.007733+0000","last_undegraded":"2024-04-25T01:52:27.007733+0000","last_fullsized":"2024-04-25T01:52:27.007733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:31.654857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377688+0000","last_change":"2024-04-25T01:52:25.517011+0000","last_active":"2024-04-25T01:52:27.377688+0000","last_peered":"2024-04-25T01:52:27.377688+0000","last_clean":"2024-04-25T01:52:27.377688+0000","last_became_active":"2024-04-25T01:52:25.516680+0000","last_became_peered":"2024-04-25T01:52:25.516680+0000","last_unstale":"2024-04-25T01:52:27.377688+0000","last_undegraded":"2024-04-25T01:52:27.377688+0000","last_fullsized":"2024-04-25T01:52:27.377688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:03.388576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732057+0000","last_change":"2024-04-25T01:52:25.531079+0000","last_active":"2024-04-25T01:52:27.732057+0000","last_peered":"2024-04-25T01:52:27.732057+0000","last_clean":"2024-04-25T01:52:27.732057+0000","last_became_active":"2024-04-25T01:52:25.529200+0000","last_became_peered":"2024-04-25T01:52:25.529200+0000","last_unstale":"2024-04-25T01:52:27.732057+0000","last_undegraded":"2024-04-25T01:52:27.732057+0000","last_fullsized":"2024-04-25T01:52:27.732057+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:30.484010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.520111+0000","last_change":"2024-04-25T01:52:25.533421+0000","last_active":"2024-04-25T01:52:26.520111+0000","last_peered":"2024-04-25T01:52:26.520111+0000","last_clean":"2024-04-25T01:52:26.520111+0000","last_became_active":"2024-04-25T01:52:25.531781+0000","last_became_peered":"2024-04-25T01:52:25.531781+0000","last_unstale":"2024-04-25T01:52:26.520111+0000","last_undegraded":"2024-04-25T01:52:26.520111+0000","last_fullsized":"2024-04-25T01:52:26.520111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:32.505793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.008019+0000","last_change":"2024-04-25T01:52:25.530223+0000","last_active":"2024-04-25T01:52:27.008019+0000","last_peered":"2024-04-25T01:52:27.008019+0000","last_clean":"2024-04-25T01:52:27.008019+0000","last_became_active":"2024-04-25T01:52:25.529909+0000","last_became_peered":"2024-04-25T01:52:25.529909+0000","last_unstale":"2024-04-25T01:52:27.008019+0000","last_undegraded":"2024-04-25T01:52:27.008019+0000","last_fullsized":"2024-04-25T01:52:27.008019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:07.429026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.378147+0000","last_change":"2024-04-25T01:52:25.524502+0000","last_active":"2024-04-25T01:52:27.378147+0000","last_peered":"2024-04-25T01:52:27.378147+0000","last_clean":"2024-04-25T01:52:27.378147+0000","last_became_active":"2024-04-25T01:52:25.524018+0000","last_became_peered":"2024-04-25T01:52:25.524018+0000","last_unstale":"2024-04-25T01:52:27.378147+0000","last_undegraded":"2024-04-25T01:52:27.378147+0000","last_fullsized":"2024-04-25T01:52:27.378147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:23.760389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793370+0000","last_change":"2024-04-25T01:52:25.533593+0000","last_active":"2024-04-25T01:52:27.793370+0000","last_peered":"2024-04-25T01:52:27.793370+0000","last_clean":"2024-04-25T01:52:27.793370+0000","last_became_active":"2024-04-25T01:52:25.532745+0000","last_became_peered":"2024-04-25T01:52:25.532745+0000","last_unstale":"2024-04-25T01:52:27.793370+0000","last_undegraded":"2024-04-25T01:52:27.793370+0000","last_fullsized":"2024-04-25T01:52:27.793370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:45.083267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517350+0000","last_change":"2024-04-25T01:52:25.528563+0000","last_active":"2024-04-25T01:52:26.517350+0000","last_peered":"2024-04-25T01:52:26.517350+0000","last_clean":"2024-04-25T01:52:26.517350+0000","last_became_active":"2024-04-25T01:52:25.528236+0000","last_became_peered":"2024-04-25T01:52:25.528236+0000","last_unstale":"2024-04-25T01:52:26.517350+0000","last_undegraded":"2024-04-25T01:52:26.517350+0000","last_fullsized":"2024-04-25T01:52:26.517350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:28:27.222799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487482+0000","last_change":"2024-04-25T01:52:25.524951+0000","last_active":"2024-04-25T01:52:27.487482+0000","last_peered":"2024-04-25T01:52:27.487482+0000","last_clean":"2024-04-25T01:52:27.487482+0000","last_became_active":"2024-04-25T01:52:25.524617+0000","last_became_peered":"2024-04-25T01:52:25.524617+0000","last_unstale":"2024-04-25T01:52:27.487482+0000","last_undegraded":"2024-04-25T01:52:27.487482+0000","last_fullsized":"2024-04-25T01:52:27.487482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:03.561883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487726+0000","last_change":"2024-04-25T01:52:25.532471+0000","last_active":"2024-04-25T01:52:27.487726+0000","last_peered":"2024-04-25T01:52:27.487726+0000","last_clean":"2024-04-25T01:52:27.487726+0000","last_became_active":"2024-04-25T01:52:25.532131+0000","last_became_peered":"2024-04-25T01:52:25.532131+0000","last_unstale":"2024-04-25T01:52:27.487726+0000","last_undegraded":"2024-04-25T01:52:27.487726+0000","last_fullsized":"2024-04-25T01:52:27.487726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:03.478816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.519636+0000","last_change":"2024-04-25T01:52:25.529160+0000","last_active":"2024-04-25T01:52:26.519636+0000","last_peered":"2024-04-25T01:52:26.519636+0000","last_clean":"2024-04-25T01:52:26.519636+0000","last_became_active":"2024-04-25T01:52:25.528854+0000","last_became_peered":"2024-04-25T01:52:25.528854+0000","last_unstale":"2024-04-25T01:52:26.519636+0000","last_undegraded":"2024-04-25T01:52:26.519636+0000","last_fullsized":"2024-04-25T01:52:26.519636+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:54.713715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988660+0000","last_change":"2024-04-25T01:52:25.523372+0000","last_active":"2024-04-25T01:52:27.988660+0000","last_peered":"2024-04-25T01:52:27.988660+0000","last_clean":"2024-04-25T01:52:27.988660+0000","last_became_active":"2024-04-25T01:52:25.522538+0000","last_became_peered":"2024-04-25T01:52:25.522538+0000","last_unstale":"2024-04-25T01:52:27.988660+0000","last_undegraded":"2024-04-25T01:52:27.988660+0000","last_fullsized":"2024-04-25T01:52:27.988660+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:47.213815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.730426+0000","last_change":"2024-04-25T01:52:25.522920+0000","last_active":"2024-04-25T01:52:27.730426+0000","last_peered":"2024-04-25T01:52:27.730426+0000","last_clean":"2024-04-25T01:52:27.730426+0000","last_became_active":"2024-04-25T01:52:25.522594+0000","last_became_peered":"2024-04-25T01:52:25.522594+0000","last_unstale":"2024-04-25T01:52:27.730426+0000","last_undegraded":"2024-04-25T01:52:27.730426+0000","last_fullsized":"2024-04-25T01:52:27.730426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:45.043667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793620+0000","last_change":"2024-04-25T01:52:25.532738+0000","last_active":"2024-04-25T01:52:27.793620+0000","last_peered":"2024-04-25T01:52:27.793620+0000","last_clean":"2024-04-25T01:52:27.793620+0000","last_became_active":"2024-04-25T01:52:25.532237+0000","last_became_peered":"2024-04-25T01:52:25.532237+0000","last_unstale":"2024-04-25T01:52:27.793620+0000","last_undegraded":"2024-04-25T01:52:27.793620+0000","last_fullsized":"2024-04-25T01:52:27.793620+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:54.456704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:29.736267+0000","last_change":"2024-04-25T01:52:24.515686+0000","last_active":"2024-04-25T01:52:29.736267+0000","last_peered":"2024-04-25T01:52:29.736267+0000","last_clean":"2024-04-25T01:52:29.736267+0000","last_became_active":"2024-04-25T01:52:24.515246+0000","last_became_peered":"2024-04-25T01:52:24.515246+0000","last_unstale":"2024-04-25T01:52:29.736267+0000","last_undegraded":"2024-04-25T01:52:29.736267+0000","last_fullsized":"2024-04-25T01:52:29.736267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:14.325747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732403+0000","last_change":"2024-04-25T01:52:25.528796+0000","last_active":"2024-04-25T01:52:27.732403+0000","last_peered":"2024-04-25T01:52:27.732403+0000","last_clean":"2024-04-25T01:52:27.732403+0000","last_became_active":"2024-04-25T01:52:25.528453+0000","last_became_peered":"2024-04-25T01:52:25.528453+0000","last_unstale":"2024-04-25T01:52:27.732403+0000","last_undegraded":"2024-04-25T01:52:27.732403+0000","last_fullsized":"2024-04-25T01:52:27.732403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:29.432519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732524+0000","last_change":"2024-04-25T01:52:24.520229+0000","last_active":"2024-04-25T01:52:27.732524+0000","last_peered":"2024-04-25T01:52:27.732524+0000","last_clean":"2024-04-25T01:52:27.732524+0000","last_became_active":"2024-04-25T01:52:24.519932+0000","last_became_peered":"2024-04-25T01:52:24.519932+0000","last_unstale":"2024-04-25T01:52:27.732524+0000","last_undegraded":"2024-04-25T01:52:27.732524+0000","last_fullsized":"2024-04-25T01:52:27.732524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.786949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.488038+0000","last_change":"2024-04-25T01:52:25.532822+0000","last_active":"2024-04-25T01:52:27.488038+0000","last_peered":"2024-04-25T01:52:27.488038+0000","last_clean":"2024-04-25T01:52:27.488038+0000","last_became_active":"2024-04-25T01:52:25.532462+0000","last_became_peered":"2024-04-25T01:52:25.532462+0000","last_unstale":"2024-04-25T01:52:27.488038+0000","last_undegraded":"2024-04-25T01:52:27.488038+0000","last_fullsized":"2024-04-25T01:52:27.488038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:55.569883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:29.734097+0000","last_change":"2024-04-25T01:52:24.512923+0000","last_active":"2024-04-25T01:52:29.734097+0000","last_peered":"2024-04-25T01:52:29.734097+0000","last_clean":"2024-04-25T01:52:29.734097+0000","last_became_active":"2024-04-25T01:52:24.512604+0000","last_became_peered":"2024-04-25T01:52:24.512604+0000","last_unstale":"2024-04-25T01:52:29.734097+0000","last_undegraded":"2024-04-25T01:52:29.734097+0000","last_fullsized":"2024-04-25T01:52:29.734097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:41.640234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.519837+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:52:26.519837+0000","last_peered":"2024-04-25T01:52:26.519837+0000","last_clean":"2024-04-25T01:52:26.519837+0000","last_became_active":"2024-04-25T01:52:25.526447+0000","last_became_peered":"2024-04-25T01:52:25.526447+0000","last_unstale":"2024-04-25T01:52:26.519837+0000","last_undegraded":"2024-04-25T01:52:26.519837+0000","last_fullsized":"2024-04-25T01:52:26.519837+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:10:18.313830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.522057+0000","last_change":"2024-04-25T01:52:24.511565+0000","last_active":"2024-04-25T01:52:26.522057+0000","last_peered":"2024-04-25T01:52:26.522057+0000","last_clean":"2024-04-25T01:52:26.522057+0000","last_became_active":"2024-04-25T01:52:24.511258+0000","last_became_peered":"2024-04-25T01:52:24.511258+0000","last_unstale":"2024-04-25T01:52:26.522057+0000","last_undegraded":"2024-04-25T01:52:26.522057+0000","last_fullsized":"2024-04-25T01:52:26.522057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:38.733338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793361+0000","last_change":"2024-04-25T01:52:25.527138+0000","last_active":"2024-04-25T01:52:27.793361+0000","last_peered":"2024-04-25T01:52:27.793361+0000","last_clean":"2024-04-25T01:52:27.793361+0000","last_became_active":"2024-04-25T01:52:25.526842+0000","last_became_peered":"2024-04-25T01:52:25.526842+0000","last_unstale":"2024-04-25T01:52:27.793361+0000","last_undegraded":"2024-04-25T01:52:27.793361+0000","last_fullsized":"2024-04-25T01:52:27.793361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:27.305773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.726803+0000","last_change":"2024-04-25T01:52:24.514854+0000","last_active":"2024-04-25T01:52:28.726803+0000","last_peered":"2024-04-25T01:52:28.726803+0000","last_clean":"2024-04-25T01:52:28.726803+0000","last_became_active":"2024-04-25T01:52:24.514548+0000","last_became_peered":"2024-04-25T01:52:24.514548+0000","last_unstale":"2024-04-25T01:52:28.726803+0000","last_undegraded":"2024-04-25T01:52:28.726803+0000","last_fullsized":"2024-04-25T01:52:28.726803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:38.539202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.522069+0000","last_change":"2024-04-25T01:52:25.523888+0000","last_active":"2024-04-25T01:52:26.522069+0000","last_peered":"2024-04-25T01:52:26.522069+0000","last_clean":"2024-04-25T01:52:26.522069+0000","last_became_active":"2024-04-25T01:52:25.523581+0000","last_became_peered":"2024-04-25T01:52:25.523581+0000","last_unstale":"2024-04-25T01:52:26.522069+0000","last_undegraded":"2024-04-25T01:52:26.522069+0000","last_fullsized":"2024-04-25T01:52:26.522069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:59.521757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:30.744219+0000","last_change":"2024-04-25T01:52:24.512774+0000","last_active":"2024-04-25T01:52:30.744219+0000","last_peered":"2024-04-25T01:52:30.744219+0000","last_clean":"2024-04-25T01:52:30.744219+0000","last_became_active":"2024-04-25T01:52:24.512393+0000","last_became_peered":"2024-04-25T01:52:24.512393+0000","last_unstale":"2024-04-25T01:52:30.744219+0000","last_undegraded":"2024-04-25T01:52:30.744219+0000","last_fullsized":"2024-04-25T01:52:30.744219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:41.639961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793080+0000","last_change":"2024-04-25T01:52:25.523381+0000","last_active":"2024-04-25T01:52:27.793080+0000","last_peered":"2024-04-25T01:52:27.793080+0000","last_clean":"2024-04-25T01:52:27.793080+0000","last_became_active":"2024-04-25T01:52:25.522978+0000","last_became_peered":"2024-04-25T01:52:25.522978+0000","last_unstale":"2024-04-25T01:52:27.793080+0000","last_undegraded":"2024-04-25T01:52:27.793080+0000","last_fullsized":"2024-04-25T01:52:27.793080+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:45.702788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517184+0000","last_change":"2024-04-25T01:52:24.521280+0000","last_active":"2024-04-25T01:52:26.517184+0000","last_peered":"2024-04-25T01:52:26.517184+0000","last_clean":"2024-04-25T01:52:26.517184+0000","last_became_active":"2024-04-25T01:52:24.520959+0000","last_became_peered":"2024-04-25T01:52:24.520959+0000","last_unstale":"2024-04-25T01:52:26.517184+0000","last_undegraded":"2024-04-25T01:52:26.517184+0000","last_fullsized":"2024-04-25T01:52:26.517184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:55.037678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.731936+0000","last_change":"2024-04-25T01:52:25.524331+0000","last_active":"2024-04-25T01:52:27.731936+0000","last_peered":"2024-04-25T01:52:27.731936+0000","last_clean":"2024-04-25T01:52:27.731936+0000","last_became_active":"2024-04-25T01:52:25.523431+0000","last_became_peered":"2024-04-25T01:52:25.523431+0000","last_unstale":"2024-04-25T01:52:27.731936+0000","last_undegraded":"2024-04-25T01:52:27.731936+0000","last_fullsized":"2024-04-25T01:52:27.731936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:55.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.792499+0000","last_change":"2024-04-25T01:52:24.515697+0000","last_active":"2024-04-25T01:52:27.792499+0000","last_peered":"2024-04-25T01:52:27.792499+0000","last_clean":"2024-04-25T01:52:27.792499+0000","last_became_active":"2024-04-25T01:52:24.515285+0000","last_became_peered":"2024-04-25T01:52:24.515285+0000","last_unstale":"2024-04-25T01:52:27.792499+0000","last_undegraded":"2024-04-25T01:52:27.792499+0000","last_fullsized":"2024-04-25T01:52:27.792499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:24.249203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.730154+0000","last_change":"2024-04-25T01:52:25.522415+0000","last_active":"2024-04-25T01:52:27.730154+0000","last_peered":"2024-04-25T01:52:27.730154+0000","last_clean":"2024-04-25T01:52:27.730154+0000","last_became_active":"2024-04-25T01:52:25.521270+0000","last_became_peered":"2024-04-25T01:52:25.521270+0000","last_unstale":"2024-04-25T01:52:27.730154+0000","last_undegraded":"2024-04-25T01:52:27.730154+0000","last_fullsized":"2024-04-25T01:52:27.730154+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:53.514169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.723787+0000","last_change":"2024-04-25T01:52:24.509525+0000","last_active":"2024-04-25T01:52:28.723787+0000","last_peered":"2024-04-25T01:52:28.723787+0000","last_clean":"2024-04-25T01:52:28.723787+0000","last_became_active":"2024-04-25T01:52:24.509184+0000","last_became_peered":"2024-04-25T01:52:24.509184+0000","last_unstale":"2024-04-25T01:52:28.723787+0000","last_undegraded":"2024-04-25T01:52:28.723787+0000","last_fullsized":"2024-04-25T01:52:28.723787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:16:43.301401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.520145+0000","last_change":"2024-04-25T01:52:25.525293+0000","last_active":"2024-04-25T01:52:26.520145+0000","last_peered":"2024-04-25T01:52:26.520145+0000","last_clean":"2024-04-25T01:52:26.520145+0000","last_became_active":"2024-04-25T01:52:25.524982+0000","last_became_peered":"2024-04-25T01:52:25.524982+0000","last_unstale":"2024-04-25T01:52:26.520145+0000","last_undegraded":"2024-04-25T01:52:26.520145+0000","last_fullsized":"2024-04-25T01:52:26.520145+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:22.222898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.724420+0000","last_change":"2024-04-25T01:52:24.519834+0000","last_active":"2024-04-25T01:52:28.724420+0000","last_peered":"2024-04-25T01:52:28.724420+0000","last_clean":"2024-04-25T01:52:28.724420+0000","last_became_active":"2024-04-25T01:52:24.519521+0000","last_became_peered":"2024-04-25T01:52:24.519521+0000","last_unstale":"2024-04-25T01:52:28.724420+0000","last_undegraded":"2024-04-25T01:52:28.724420+0000","last_fullsized":"2024-04-25T01:52:28.724420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:02.414527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.488009+0000","last_change":"2024-04-25T01:52:25.523809+0000","last_active":"2024-04-25T01:52:27.488009+0000","last_peered":"2024-04-25T01:52:27.488009+0000","last_clean":"2024-04-25T01:52:27.488009+0000","last_became_active":"2024-04-25T01:52:25.523406+0000","last_became_peered":"2024-04-25T01:52:25.523406+0000","last_unstale":"2024-04-25T01:52:27.488009+0000","last_undegraded":"2024-04-25T01:52:27.488009+0000","last_fullsized":"2024-04-25T01:52:27.488009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:22.169633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793424+0000","last_change":"2024-04-25T01:52:24.510226+0000","last_active":"2024-04-25T01:52:27.793424+0000","last_peered":"2024-04-25T01:52:27.793424+0000","last_clean":"2024-04-25T01:52:27.793424+0000","last_became_active":"2024-04-25T01:52:24.509667+0000","last_became_peered":"2024-04-25T01:52:24.509667+0000","last_unstale":"2024-04-25T01:52:27.793424+0000","last_undegraded":"2024-04-25T01:52:27.793424+0000","last_fullsized":"2024-04-25T01:52:27.793424+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:49.180675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.519894+0000","last_change":"2024-04-25T01:52:25.531567+0000","last_active":"2024-04-25T01:52:26.519894+0000","last_peered":"2024-04-25T01:52:26.519894+0000","last_clean":"2024-04-25T01:52:26.519894+0000","last_became_active":"2024-04-25T01:52:25.531200+0000","last_became_peered":"2024-04-25T01:52:25.531200+0000","last_unstale":"2024-04-25T01:52:26.519894+0000","last_undegraded":"2024-04-25T01:52:26.519894+0000","last_fullsized":"2024-04-25T01:52:26.519894+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:51:14.336413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'17","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:32.130105+0000","last_change":"2024-04-25T01:52:24.512679+0000","last_active":"2024-04-25T01:52:32.130105+0000","last_peered":"2024-04-25T01:52:32.130105+0000","last_clean":"2024-04-25T01:52:32.130105+0000","last_became_active":"2024-04-25T01:52:24.512283+0000","last_became_peered":"2024-04-25T01:52:24.512283+0000","last_unstale":"2024-04-25T01:52:32.130105+0000","last_undegraded":"2024-04-25T01:52:32.130105+0000","last_fullsized":"2024-04-25T01:52:32.130105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:28.045138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20895,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.522210+0000","last_change":"2024-04-25T01:52:25.525931+0000","last_active":"2024-04-25T01:52:26.522210+0000","last_peered":"2024-04-25T01:52:26.522210+0000","last_clean":"2024-04-25T01:52:26.522210+0000","last_became_active":"2024-04-25T01:52:25.525460+0000","last_became_peered":"2024-04-25T01:52:25.525460+0000","last_unstale":"2024-04-25T01:52:26.522210+0000","last_undegraded":"2024-04-25T01:52:26.522210+0000","last_fullsized":"2024-04-25T01:52:26.522210+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:46.569587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.724366+0000","last_change":"2024-04-25T01:52:24.505470+0000","last_active":"2024-04-25T01:52:28.724366+0000","last_peered":"2024-04-25T01:52:28.724366+0000","last_clean":"2024-04-25T01:52:28.724366+0000","last_became_active":"2024-04-25T01:52:24.504677+0000","last_became_peered":"2024-04-25T01:52:24.504677+0000","last_unstale":"2024-04-25T01:52:28.724366+0000","last_undegraded":"2024-04-25T01:52:28.724366+0000","last_fullsized":"2024-04-25T01:52:28.724366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:01.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517150+0000","last_change":"2024-04-25T01:52:25.526364+0000","last_active":"2024-04-25T01:52:26.517150+0000","last_peered":"2024-04-25T01:52:26.517150+0000","last_clean":"2024-04-25T01:52:26.517150+0000","last_became_active":"2024-04-25T01:52:25.525376+0000","last_became_peered":"2024-04-25T01:52:25.525376+0000","last_unstale":"2024-04-25T01:52:26.517150+0000","last_undegraded":"2024-04-25T01:52:26.517150+0000","last_fullsized":"2024-04-25T01:52:26.517150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:26.770994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.726244+0000","last_change":"2024-04-25T01:52:24.520287+0000","last_active":"2024-04-25T01:52:28.726244+0000","last_peered":"2024-04-25T01:52:28.726244+0000","last_clean":"2024-04-25T01:52:28.726244+0000","last_became_active":"2024-04-25T01:52:24.519967+0000","last_became_peered":"2024-04-25T01:52:24.519967+0000","last_unstale":"2024-04-25T01:52:28.726244+0000","last_undegraded":"2024-04-25T01:52:28.726244+0000","last_fullsized":"2024-04-25T01:52:28.726244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:45.078410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377949+0000","last_change":"2024-04-25T01:52:25.525766+0000","last_active":"2024-04-25T01:52:27.377949+0000","last_peered":"2024-04-25T01:52:27.377949+0000","last_clean":"2024-04-25T01:52:27.377949+0000","last_became_active":"2024-04-25T01:52:25.525350+0000","last_became_peered":"2024-04-25T01:52:25.525350+0000","last_unstale":"2024-04-25T01:52:27.377949+0000","last_undegraded":"2024-04-25T01:52:27.377949+0000","last_fullsized":"2024-04-25T01:52:27.377949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:05.387502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.792800+0000","last_change":"2024-04-25T01:52:24.513674+0000","last_active":"2024-04-25T01:52:27.792800+0000","last_peered":"2024-04-25T01:52:27.792800+0000","last_clean":"2024-04-25T01:52:27.792800+0000","last_became_active":"2024-04-25T01:52:24.513270+0000","last_became_peered":"2024-04-25T01:52:24.513270+0000","last_unstale":"2024-04-25T01:52:27.792800+0000","last_undegraded":"2024-04-25T01:52:27.792800+0000","last_fullsized":"2024-04-25T01:52:27.792800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:23.385928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377568+0000","last_change":"2024-04-25T01:52:25.517166+0000","last_active":"2024-04-25T01:52:27.377568+0000","last_peered":"2024-04-25T01:52:27.377568+0000","last_clean":"2024-04-25T01:52:27.377568+0000","last_became_active":"2024-04-25T01:52:25.516853+0000","last_became_peered":"2024-04-25T01:52:25.516853+0000","last_unstale":"2024-04-25T01:52:27.377568+0000","last_undegraded":"2024-04-25T01:52:27.377568+0000","last_fullsized":"2024-04-25T01:52:27.377568+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:18.502558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:29.736794+0000","last_change":"2024-04-25T01:52:24.514695+0000","last_active":"2024-04-25T01:52:29.736794+0000","last_peered":"2024-04-25T01:52:29.736794+0000","last_clean":"2024-04-25T01:52:29.736794+0000","last_became_active":"2024-04-25T01:52:24.514370+0000","last_became_peered":"2024-04-25T01:52:24.514370+0000","last_unstale":"2024-04-25T01:52:29.736794+0000","last_undegraded":"2024-04-25T01:52:29.736794+0000","last_fullsized":"2024-04-25T01:52:29.736794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:28:28.537307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.792696+0000","last_change":"2024-04-25T01:52:25.527062+0000","last_active":"2024-04-25T01:52:27.792696+0000","last_peered":"2024-04-25T01:52:27.792696+0000","last_clean":"2024-04-25T01:52:27.792696+0000","last_became_active":"2024-04-25T01:52:25.526757+0000","last_became_peered":"2024-04-25T01:52:25.526757+0000","last_unstale":"2024-04-25T01:52:27.792696+0000","last_undegraded":"2024-04-25T01:52:27.792696+0000","last_fullsized":"2024-04-25T01:52:27.792696+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:36.360030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.522155+0000","last_change":"2024-04-25T01:52:24.512235+0000","last_active":"2024-04-25T01:52:26.522155+0000","last_peered":"2024-04-25T01:52:26.522155+0000","last_clean":"2024-04-25T01:52:26.522155+0000","last_became_active":"2024-04-25T01:52:24.511910+0000","last_became_peered":"2024-04-25T01:52:24.511910+0000","last_unstale":"2024-04-25T01:52:26.522155+0000","last_undegraded":"2024-04-25T01:52:26.522155+0000","last_fullsized":"2024-04-25T01:52:26.522155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:38.435734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988402+0000","last_change":"2024-04-25T01:52:25.527854+0000","last_active":"2024-04-25T01:52:27.988402+0000","last_peered":"2024-04-25T01:52:27.988402+0000","last_clean":"2024-04-25T01:52:27.988402+0000","last_became_active":"2024-04-25T01:52:25.527481+0000","last_became_peered":"2024-04-25T01:52:25.527481+0000","last_unstale":"2024-04-25T01:52:27.988402+0000","last_undegraded":"2024-04-25T01:52:27.988402+0000","last_fullsized":"2024-04-25T01:52:27.988402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:09:11.281982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.730195+0000","last_change":"2024-04-25T01:52:24.513463+0000","last_active":"2024-04-25T01:52:27.730195+0000","last_peered":"2024-04-25T01:52:27.730195+0000","last_clean":"2024-04-25T01:52:27.730195+0000","last_became_active":"2024-04-25T01:52:24.513176+0000","last_became_peered":"2024-04-25T01:52:24.513176+0000","last_unstale":"2024-04-25T01:52:27.730195+0000","last_undegraded":"2024-04-25T01:52:27.730195+0000","last_fullsized":"2024-04-25T01:52:27.730195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:13:33.507559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.588800+0000","last_change":"2024-04-25T01:52:25.528316+0000","last_active":"2024-04-25T01:52:27.588800+0000","last_peered":"2024-04-25T01:52:27.588800+0000","last_clean":"2024-04-25T01:52:27.588800+0000","last_became_active":"2024-04-25T01:52:25.527991+0000","last_became_peered":"2024-04-25T01:52:25.527991+0000","last_unstale":"2024-04-25T01:52:27.588800+0000","last_undegraded":"2024-04-25T01:52:27.588800+0000","last_fullsized":"2024-04-25T01:52:27.588800+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:55.871663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377711+0000","last_change":"2024-04-25T01:52:24.519357+0000","last_active":"2024-04-25T01:52:27.377711+0000","last_peered":"2024-04-25T01:52:27.377711+0000","last_clean":"2024-04-25T01:52:27.377711+0000","last_became_active":"2024-04-25T01:52:24.519025+0000","last_became_peered":"2024-04-25T01:52:24.519025+0000","last_unstale":"2024-04-25T01:52:27.377711+0000","last_undegraded":"2024-04-25T01:52:27.377711+0000","last_fullsized":"2024-04-25T01:52:27.377711+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:03.476554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.007801+0000","last_change":"2024-04-25T01:52:25.527922+0000","last_active":"2024-04-25T01:52:27.007801+0000","last_peered":"2024-04-25T01:52:27.007801+0000","last_clean":"2024-04-25T01:52:27.007801+0000","last_became_active":"2024-04-25T01:52:25.527596+0000","last_became_peered":"2024-04-25T01:52:25.527596+0000","last_unstale":"2024-04-25T01:52:27.007801+0000","last_undegraded":"2024-04-25T01:52:27.007801+0000","last_fullsized":"2024-04-25T01:52:27.007801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:56.002208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.792833+0000","last_change":"2024-04-25T01:52:25.533800+0000","last_active":"2024-04-25T01:52:27.792833+0000","last_peered":"2024-04-25T01:52:27.792833+0000","last_clean":"2024-04-25T01:52:27.792833+0000","last_became_active":"2024-04-25T01:52:25.533484+0000","last_became_peered":"2024-04-25T01:52:25.533484+0000","last_unstale":"2024-04-25T01:52:27.792833+0000","last_undegraded":"2024-04-25T01:52:27.792833+0000","last_fullsized":"2024-04-25T01:52:27.792833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:13.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517111+0000","last_change":"2024-04-25T01:52:24.506068+0000","last_active":"2024-04-25T01:52:26.517111+0000","last_peered":"2024-04-25T01:52:26.517111+0000","last_clean":"2024-04-25T01:52:26.517111+0000","last_became_active":"2024-04-25T01:52:24.505667+0000","last_became_peered":"2024-04-25T01:52:24.505667+0000","last_unstale":"2024-04-25T01:52:26.517111+0000","last_undegraded":"2024-04-25T01:52:26.517111+0000","last_fullsized":"2024-04-25T01:52:26.517111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:30.494376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732361+0000","last_change":"2024-04-25T01:52:25.530681+0000","last_active":"2024-04-25T01:52:27.732361+0000","last_peered":"2024-04-25T01:52:27.732361+0000","last_clean":"2024-04-25T01:52:27.732361+0000","last_became_active":"2024-04-25T01:52:25.529035+0000","last_became_peered":"2024-04-25T01:52:25.529035+0000","last_unstale":"2024-04-25T01:52:27.732361+0000","last_undegraded":"2024-04-25T01:52:27.732361+0000","last_fullsized":"2024-04-25T01:52:27.732361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:28.787180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.724985+0000","last_change":"2024-04-25T01:52:24.508948+0000","last_active":"2024-04-25T01:52:28.724985+0000","last_peered":"2024-04-25T01:52:28.724985+0000","last_clean":"2024-04-25T01:52:28.724985+0000","last_became_active":"2024-04-25T01:52:24.508563+0000","last_became_peered":"2024-04-25T01:52:24.508563+0000","last_unstale":"2024-04-25T01:52:28.724985+0000","last_undegraded":"2024-04-25T01:52:28.724985+0000","last_fullsized":"2024-04-25T01:52:28.724985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:37.591783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377547+0000","last_change":"2024-04-25T01:51:15.621251+0000","last_active":"2024-04-25T01:52:27.377547+0000","last_peered":"2024-04-25T01:52:27.377547+0000","last_clean":"2024-04-25T01:52:27.377547+0000","last_became_active":"2024-04-25T01:51:15.620771+0000","last_became_peered":"2024-04-25T01:51:15.620771+0000","last_unstale":"2024-04-25T01:52:27.377547+0000","last_undegraded":"2024-04-25T01:52:27.377547+0000","last_fullsized":"2024-04-25T01:52:27.377547+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_clean_scrub_stamp":"2024-04-25T01:47:50.544829+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:44.048065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517069+0000","last_change":"2024-04-25T01:52:25.530216+0000","last_active":"2024-04-25T01:52:26.517069+0000","last_peered":"2024-04-25T01:52:26.517069+0000","last_clean":"2024-04-25T01:52:26.517069+0000","last_became_active":"2024-04-25T01:52:25.529813+0000","last_became_peered":"2024-04-25T01:52:25.529813+0000","last_unstale":"2024-04-25T01:52:26.517069+0000","last_undegraded":"2024-04-25T01:52:26.517069+0000","last_fullsized":"2024-04-25T01:52:26.517069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:57.656375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:29.735263+0000","last_change":"2024-04-25T01:52:24.517069+0000","last_active":"2024-04-25T01:52:29.735263+0000","last_peered":"2024-04-25T01:52:29.735263+0000","last_clean":"2024-04-25T01:52:29.735263+0000","last_became_active":"2024-04-25T01:52:24.516732+0000","last_became_peered":"2024-04-25T01:52:24.516732+0000","last_unstale":"2024-04-25T01:52:29.735263+0000","last_undegraded":"2024-04-25T01:52:29.735263+0000","last_fullsized":"2024-04-25T01:52:29.735263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:58.546314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.522221+0000","last_change":"2024-04-25T01:52:24.514786+0000","last_active":"2024-04-25T01:52:26.522221+0000","last_peered":"2024-04-25T01:52:26.522221+0000","last_clean":"2024-04-25T01:52:26.522221+0000","last_became_active":"2024-04-25T01:52:24.514486+0000","last_became_peered":"2024-04-25T01:52:24.514486+0000","last_unstale":"2024-04-25T01:52:26.522221+0000","last_undegraded":"2024-04-25T01:52:26.522221+0000","last_fullsized":"2024-04-25T01:52:26.522221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:42.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517462+0000","last_change":"2024-04-25T01:52:25.526518+0000","last_active":"2024-04-25T01:52:26.517462+0000","last_peered":"2024-04-25T01:52:26.517462+0000","last_clean":"2024-04-25T01:52:26.517462+0000","last_became_active":"2024-04-25T01:52:25.525917+0000","last_became_peered":"2024-04-25T01:52:25.525917+0000","last_unstale":"2024-04-25T01:52:26.517462+0000","last_undegraded":"2024-04-25T01:52:26.517462+0000","last_fullsized":"2024-04-25T01:52:26.517462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:20.776385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:29.734537+0000","last_change":"2024-04-25T01:52:24.511561+0000","last_active":"2024-04-25T01:52:29.734537+0000","last_peered":"2024-04-25T01:52:29.734537+0000","last_clean":"2024-04-25T01:52:29.734537+0000","last_became_active":"2024-04-25T01:52:24.511165+0000","last_became_peered":"2024-04-25T01:52:24.511165+0000","last_unstale":"2024-04-25T01:52:29.734537+0000","last_undegraded":"2024-04-25T01:52:29.734537+0000","last_fullsized":"2024-04-25T01:52:29.734537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:27:41.006137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732445+0000","last_change":"2024-04-25T01:52:25.527991+0000","last_active":"2024-04-25T01:52:27.732445+0000","last_peered":"2024-04-25T01:52:27.732445+0000","last_clean":"2024-04-25T01:52:27.732445+0000","last_became_active":"2024-04-25T01:52:25.527677+0000","last_became_peered":"2024-04-25T01:52:25.527677+0000","last_unstale":"2024-04-25T01:52:27.732445+0000","last_undegraded":"2024-04-25T01:52:27.732445+0000","last_fullsized":"2024-04-25T01:52:27.732445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:07:24.698774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.725108+0000","last_change":"2024-04-25T01:52:24.523536+0000","last_active":"2024-04-25T01:52:28.725108+0000","last_peered":"2024-04-25T01:52:28.725108+0000","last_clean":"2024-04-25T01:52:28.725108+0000","last_became_active":"2024-04-25T01:52:24.523233+0000","last_became_peered":"2024-04-25T01:52:24.523233+0000","last_unstale":"2024-04-25T01:52:28.725108+0000","last_undegraded":"2024-04-25T01:52:28.725108+0000","last_fullsized":"2024-04-25T01:52:28.725108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:49:44.444968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487949+0000","last_change":"2024-04-25T01:52:25.527090+0000","last_active":"2024-04-25T01:52:27.487949+0000","last_peered":"2024-04-25T01:52:27.487949+0000","last_clean":"2024-04-25T01:52:27.487949+0000","last_became_active":"2024-04-25T01:52:25.526771+0000","last_became_peered":"2024-04-25T01:52:25.526771+0000","last_unstale":"2024-04-25T01:52:27.487949+0000","last_undegraded":"2024-04-25T01:52:27.487949+0000","last_fullsized":"2024-04-25T01:52:27.487949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:58.758748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.725563+0000","last_change":"2024-04-25T01:52:24.516469+0000","last_active":"2024-04-25T01:52:28.725563+0000","last_peered":"2024-04-25T01:52:28.725563+0000","last_clean":"2024-04-25T01:52:28.725563+0000","last_became_active":"2024-04-25T01:52:24.515948+0000","last_became_peered":"2024-04-25T01:52:24.515948+0000","last_unstale":"2024-04-25T01:52:28.725563+0000","last_undegraded":"2024-04-25T01:52:28.725563+0000","last_fullsized":"2024-04-25T01:52:28.725563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:25.122153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.521898+0000","last_change":"2024-04-25T01:52:25.524967+0000","last_active":"2024-04-25T01:52:26.521898+0000","last_peered":"2024-04-25T01:52:26.521898+0000","last_clean":"2024-04-25T01:52:26.521898+0000","last_became_active":"2024-04-25T01:52:25.524670+0000","last_became_peered":"2024-04-25T01:52:25.524670+0000","last_unstale":"2024-04-25T01:52:26.521898+0000","last_undegraded":"2024-04-25T01:52:26.521898+0000","last_fullsized":"2024-04-25T01:52:26.521898+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:08.829052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:31.750420+0000","last_change":"2024-04-25T01:52:24.512187+0000","last_active":"2024-04-25T01:52:31.750420+0000","last_peered":"2024-04-25T01:52:31.750420+0000","last_clean":"2024-04-25T01:52:31.750420+0000","last_became_active":"2024-04-25T01:52:24.510303+0000","last_became_peered":"2024-04-25T01:52:24.510303+0000","last_unstale":"2024-04-25T01:52:31.750420+0000","last_undegraded":"2024-04-25T01:52:31.750420+0000","last_fullsized":"2024-04-25T01:52:31.750420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:42:31.596355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988098+0000","last_change":"2024-04-25T01:52:25.530386+0000","last_active":"2024-04-25T01:52:27.988098+0000","last_peered":"2024-04-25T01:52:27.988098+0000","last_clean":"2024-04-25T01:52:27.988098+0000","last_became_active":"2024-04-25T01:52:25.530034+0000","last_became_peered":"2024-04-25T01:52:25.530034+0000","last_unstale":"2024-04-25T01:52:27.988098+0000","last_undegraded":"2024-04-25T01:52:27.988098+0000","last_fullsized":"2024-04-25T01:52:27.988098+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:11.470123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793046+0000","last_change":"2024-04-25T01:52:24.511881+0000","last_active":"2024-04-25T01:52:27.793046+0000","last_peered":"2024-04-25T01:52:27.793046+0000","last_clean":"2024-04-25T01:52:27.793046+0000","last_became_active":"2024-04-25T01:52:24.511226+0000","last_became_peered":"2024-04-25T01:52:24.511226+0000","last_unstale":"2024-04-25T01:52:27.793046+0000","last_undegraded":"2024-04-25T01:52:27.793046+0000","last_fullsized":"2024-04-25T01:52:27.793046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:49:10.038555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732522+0000","last_change":"2024-04-25T01:52:25.525695+0000","last_active":"2024-04-25T01:52:27.732522+0000","last_peered":"2024-04-25T01:52:27.732522+0000","last_clean":"2024-04-25T01:52:27.732522+0000","last_became_active":"2024-04-25T01:52:25.524830+0000","last_became_peered":"2024-04-25T01:52:25.524830+0000","last_unstale":"2024-04-25T01:52:27.732522+0000","last_undegraded":"2024-04-25T01:52:27.732522+0000","last_fullsized":"2024-04-25T01:52:27.732522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:04:58.449350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:28.725260+0000","last_change":"2024-04-25T01:52:24.519232+0000","last_active":"2024-04-25T01:52:28.725260+0000","last_peered":"2024-04-25T01:52:28.725260+0000","last_clean":"2024-04-25T01:52:28.725260+0000","last_became_active":"2024-04-25T01:52:24.518902+0000","last_became_peered":"2024-04-25T01:52:24.518902+0000","last_unstale":"2024-04-25T01:52:28.725260+0000","last_undegraded":"2024-04-25T01:52:28.725260+0000","last_fullsized":"2024-04-25T01:52:28.725260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.173730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377940+0000","last_change":"2024-04-25T01:52:25.522955+0000","last_active":"2024-04-25T01:52:27.377940+0000","last_peered":"2024-04-25T01:52:27.377940+0000","last_clean":"2024-04-25T01:52:27.377940+0000","last_became_active":"2024-04-25T01:52:25.522626+0000","last_became_peered":"2024-04-25T01:52:25.522626+0000","last_unstale":"2024-04-25T01:52:27.377940+0000","last_undegraded":"2024-04-25T01:52:27.377940+0000","last_fullsized":"2024-04-25T01:52:27.377940+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:47.218461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793443+0000","last_change":"2024-04-25T01:52:24.516383+0000","last_active":"2024-04-25T01:52:27.793443+0000","last_peered":"2024-04-25T01:52:27.793443+0000","last_clean":"2024-04-25T01:52:27.793443+0000","last_became_active":"2024-04-25T01:52:24.515096+0000","last_became_peered":"2024-04-25T01:52:24.515096+0000","last_unstale":"2024-04-25T01:52:27.793443+0000","last_undegraded":"2024-04-25T01:52:27.793443+0000","last_fullsized":"2024-04-25T01:52:27.793443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:52.658161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.378280+0000","last_change":"2024-04-25T01:52:25.524874+0000","last_active":"2024-04-25T01:52:27.378280+0000","last_peered":"2024-04-25T01:52:27.378280+0000","last_clean":"2024-04-25T01:52:27.378280+0000","last_became_active":"2024-04-25T01:52:25.523532+0000","last_became_peered":"2024-04-25T01:52:25.523532+0000","last_unstale":"2024-04-25T01:52:27.378280+0000","last_undegraded":"2024-04-25T01:52:27.378280+0000","last_fullsized":"2024-04-25T01:52:27.378280+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:10.746446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793126+0000","last_change":"2024-04-25T01:52:24.526212+0000","last_active":"2024-04-25T01:52:27.793126+0000","last_peered":"2024-04-25T01:52:27.793126+0000","last_clean":"2024-04-25T01:52:27.793126+0000","last_became_active":"2024-04-25T01:52:24.525882+0000","last_became_peered":"2024-04-25T01:52:24.525882+0000","last_unstale":"2024-04-25T01:52:27.793126+0000","last_undegraded":"2024-04-25T01:52:27.793126+0000","last_fullsized":"2024-04-25T01:52:27.793126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:04.792560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.377867+0000","last_change":"2024-04-25T01:52:25.517387+0000","last_active":"2024-04-25T01:52:27.377867+0000","last_peered":"2024-04-25T01:52:27.377867+0000","last_clean":"2024-04-25T01:52:27.377867+0000","last_became_active":"2024-04-25T01:52:25.517047+0000","last_became_peered":"2024-04-25T01:52:25.517047+0000","last_unstale":"2024-04-25T01:52:27.377867+0000","last_undegraded":"2024-04-25T01:52:27.377867+0000","last_fullsized":"2024-04-25T01:52:27.377867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:39.462474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517458+0000","last_change":"2024-04-25T01:52:24.506510+0000","last_active":"2024-04-25T01:52:26.517458+0000","last_peered":"2024-04-25T01:52:26.517458+0000","last_clean":"2024-04-25T01:52:26.517458+0000","last_became_active":"2024-04-25T01:52:24.506184+0000","last_became_peered":"2024-04-25T01:52:24.506184+0000","last_unstale":"2024-04-25T01:52:26.517458+0000","last_undegraded":"2024-04-25T01:52:26.517458+0000","last_fullsized":"2024-04-25T01:52:26.517458+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:56.754386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.378115+0000","last_change":"2024-04-25T01:52:25.525197+0000","last_active":"2024-04-25T01:52:27.378115+0000","last_peered":"2024-04-25T01:52:27.378115+0000","last_clean":"2024-04-25T01:52:27.378115+0000","last_became_active":"2024-04-25T01:52:25.524692+0000","last_became_peered":"2024-04-25T01:52:25.524692+0000","last_unstale":"2024-04-25T01:52:27.378115+0000","last_undegraded":"2024-04-25T01:52:27.378115+0000","last_fullsized":"2024-04-25T01:52:27.378115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.264227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988215+0000","last_change":"2024-04-25T01:52:25.531060+0000","last_active":"2024-04-25T01:52:27.988215+0000","last_peered":"2024-04-25T01:52:27.988215+0000","last_clean":"2024-04-25T01:52:27.988215+0000","last_became_active":"2024-04-25T01:52:25.530774+0000","last_became_peered":"2024-04-25T01:52:25.530774+0000","last_unstale":"2024-04-25T01:52:27.988215+0000","last_undegraded":"2024-04-25T01:52:27.988215+0000","last_fullsized":"2024-04-25T01:52:27.988215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:32.168699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793070+0000","last_change":"2024-04-25T01:52:25.535381+0000","last_active":"2024-04-25T01:52:27.793070+0000","last_peered":"2024-04-25T01:52:27.793070+0000","last_clean":"2024-04-25T01:52:27.793070+0000","last_became_active":"2024-04-25T01:52:25.533623+0000","last_became_peered":"2024-04-25T01:52:25.533623+0000","last_unstale":"2024-04-25T01:52:27.793070+0000","last_undegraded":"2024-04-25T01:52:27.793070+0000","last_fullsized":"2024-04-25T01:52:27.793070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:14.032508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793201+0000","last_change":"2024-04-25T01:52:25.528803+0000","last_active":"2024-04-25T01:52:27.793201+0000","last_peered":"2024-04-25T01:52:27.793201+0000","last_clean":"2024-04-25T01:52:27.793201+0000","last_became_active":"2024-04-25T01:52:25.528505+0000","last_became_peered":"2024-04-25T01:52:25.528505+0000","last_unstale":"2024-04-25T01:52:27.793201+0000","last_undegraded":"2024-04-25T01:52:27.793201+0000","last_fullsized":"2024-04-25T01:52:27.793201+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:35.811181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.487769+0000","last_change":"2024-04-25T01:52:25.523620+0000","last_active":"2024-04-25T01:52:27.487769+0000","last_peered":"2024-04-25T01:52:27.487769+0000","last_clean":"2024-04-25T01:52:27.487769+0000","last_became_active":"2024-04-25T01:52:25.523007+0000","last_became_peered":"2024-04-25T01:52:25.523007+0000","last_unstale":"2024-04-25T01:52:27.487769+0000","last_undegraded":"2024-04-25T01:52:27.487769+0000","last_fullsized":"2024-04-25T01:52:27.487769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:23.587838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517315+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:52:26.517315+0000","last_peered":"2024-04-25T01:52:26.517315+0000","last_clean":"2024-04-25T01:52:26.517315+0000","last_became_active":"2024-04-25T01:52:25.526398+0000","last_became_peered":"2024-04-25T01:52:25.526398+0000","last_unstale":"2024-04-25T01:52:26.517315+0000","last_undegraded":"2024-04-25T01:52:26.517315+0000","last_fullsized":"2024-04-25T01:52:26.517315+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:01:37.286676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988266+0000","last_change":"2024-04-25T01:52:25.524999+0000","last_active":"2024-04-25T01:52:27.988266+0000","last_peered":"2024-04-25T01:52:27.988266+0000","last_clean":"2024-04-25T01:52:27.988266+0000","last_became_active":"2024-04-25T01:52:25.524652+0000","last_became_peered":"2024-04-25T01:52:25.524652+0000","last_unstale":"2024-04-25T01:52:27.988266+0000","last_undegraded":"2024-04-25T01:52:27.988266+0000","last_fullsized":"2024-04-25T01:52:27.988266+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:47:37.463701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.793710+0000","last_change":"2024-04-25T01:52:25.532996+0000","last_active":"2024-04-25T01:52:27.793710+0000","last_peered":"2024-04-25T01:52:27.793710+0000","last_clean":"2024-04-25T01:52:27.793710+0000","last_became_active":"2024-04-25T01:52:25.530230+0000","last_became_peered":"2024-04-25T01:52:25.530230+0000","last_unstale":"2024-04-25T01:52:27.793710+0000","last_undegraded":"2024-04-25T01:52:27.793710+0000","last_fullsized":"2024-04-25T01:52:27.793710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:08.996434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732256+0000","last_change":"2024-04-25T01:52:25.523838+0000","last_active":"2024-04-25T01:52:27.732256+0000","last_peered":"2024-04-25T01:52:27.732256+0000","last_clean":"2024-04-25T01:52:27.732256+0000","last_became_active":"2024-04-25T01:52:25.523297+0000","last_became_peered":"2024-04-25T01:52:25.523297+0000","last_unstale":"2024-04-25T01:52:27.732256+0000","last_undegraded":"2024-04-25T01:52:27.732256+0000","last_fullsized":"2024-04-25T01:52:27.732256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:52.938763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.792550+0000","last_change":"2024-04-25T01:52:25.519028+0000","last_active":"2024-04-25T01:52:27.792550+0000","last_peered":"2024-04-25T01:52:27.792550+0000","last_clean":"2024-04-25T01:52:27.792550+0000","last_became_active":"2024-04-25T01:52:25.518716+0000","last_became_peered":"2024-04-25T01:52:25.518716+0000","last_unstale":"2024-04-25T01:52:27.792550+0000","last_undegraded":"2024-04-25T01:52:27.792550+0000","last_fullsized":"2024-04-25T01:52:27.792550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:37:31.573403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.732042+0000","last_change":"2024-04-25T01:52:25.533894+0000","last_active":"2024-04-25T01:52:27.732042+0000","last_peered":"2024-04-25T01:52:27.732042+0000","last_clean":"2024-04-25T01:52:27.732042+0000","last_became_active":"2024-04-25T01:52:25.533510+0000","last_became_peered":"2024-04-25T01:52:25.533510+0000","last_unstale":"2024-04-25T01:52:27.732042+0000","last_undegraded":"2024-04-25T01:52:27.732042+0000","last_fullsized":"2024-04-25T01:52:27.732042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:07.128296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.519798+0000","last_change":"2024-04-25T01:52:25.524693+0000","last_active":"2024-04-25T01:52:26.519798+0000","last_peered":"2024-04-25T01:52:26.519798+0000","last_clean":"2024-04-25T01:52:26.519798+0000","last_became_active":"2024-04-25T01:52:25.524366+0000","last_became_peered":"2024-04-25T01:52:25.524366+0000","last_unstale":"2024-04-25T01:52:26.519798+0000","last_undegraded":"2024-04-25T01:52:26.519798+0000","last_fullsized":"2024-04-25T01:52:26.519798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:23.133722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:26.517377+0000","last_change":"2024-04-25T01:52:25.525936+0000","last_active":"2024-04-25T01:52:26.517377+0000","last_peered":"2024-04-25T01:52:26.517377+0000","last_clean":"2024-04-25T01:52:26.517377+0000","last_became_active":"2024-04-25T01:52:25.525586+0000","last_became_peered":"2024-04-25T01:52:25.525586+0000","last_unstale":"2024-04-25T01:52:26.517377+0000","last_undegraded":"2024-04-25T01:52:26.517377+0000","last_fullsized":"2024-04-25T01:52:26.517377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:34.756898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:27.988207+0000","last_change":"2024-04-25T01:52:25.528704+0000","last_active":"2024-04-25T01:52:27.988207+0000","last_peered":"2024-04-25T01:52:27.988207+0000","last_clean":"2024-04-25T01:52:27.988207+0000","last_became_active":"2024-04-25T01:52:25.528379+0000","last_became_peered":"2024-04-25T01:52:25.528379+0000","last_unstale":"2024-04-25T01:52:27.988207+0000","last_undegraded":"2024-04-25T01:52:27.988207+0000","last_fullsized":"2024-04-25T01:52:27.988207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:43.453698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280293,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":23301,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":36409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318285,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":451252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579923,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":1063176,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743446,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":450487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005085,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":450295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":999424,"data_stored":1041241,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":449669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":950272,"data_stored":451281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":451147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":451315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":450333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":449691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":999424,"data_stored":1040219,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":23022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:52:34.586 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T01:52:34.586 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T01:52:34.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T01:52:34.868 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:34.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:34 smithi031 ceph-mon[29204]: pgmap v292: 97 pgs: 97 active+clean; 599 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s wr, 15 op/s 2024-04-25T01:52:34.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:34 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2040775782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:34.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:34 smithi031 ceph-mon[29204]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:34.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:34 smithi031 ceph-mon[29204]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:35.089 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:34 smithi134 ceph-mon[33098]: pgmap v292: 97 pgs: 97 active+clean; 599 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s wr, 15 op/s 2024-04-25T01:52:35.089 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:34 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2040775782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:35.089 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:34 smithi134 ceph-mon[33098]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:35.089 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:34 smithi134 ceph-mon[33098]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:35.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:34 smithi040 ceph-mon[32418]: pgmap v292: 97 pgs: 97 active+clean; 599 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s wr, 15 op/s 2024-04-25T01:52:35.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:34 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2040775782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:35.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:34 smithi040 ceph-mon[32418]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:35.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:34 smithi040 ceph-mon[32418]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:36.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.560+0000 7f5b90faa700 1 -- 172.21.15.31:0/2983548140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c1034e0 msgr2=0x7f5b8c103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:36.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.560+0000 7f5b90faa700 1 --2- 172.21.15.31:0/2983548140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c1034e0 0x7f5b8c103940 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5b740099c0 tx=0x7f5b74009cd0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.560+0000 7f5b90faa700 1 -- 172.21.15.31:0/2983548140 shutdown_connections 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.560+0000 7f5b90faa700 1 --2- 172.21.15.31:0/2983548140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b8c103e80 0x7f5b8c10a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.560+0000 7f5b90faa700 1 --2- 172.21.15.31:0/2983548140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b8c1022e0 0x7f5b8c102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.560+0000 7f5b90faa700 1 --2- 172.21.15.31:0/2983548140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c1034e0 0x7f5b8c103940 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.560+0000 7f5b90faa700 1 -- 172.21.15.31:0/2983548140 >> 172.21.15.31:0/2983548140 conn(0x7f5b8c0fd880 msgr2=0x7f5b8c0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.560+0000 7f5b90faa700 1 -- 172.21.15.31:0/2983548140 shutdown_connections 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.560+0000 7f5b90faa700 1 -- 172.21.15.31:0/2983548140 wait complete. 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b90faa700 1 Processor -- start 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b90faa700 1 -- start start 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b90faa700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c1022e0 0x7f5b8c075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b90faa700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b8c1034e0 0x7f5b8c074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b90faa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b8c103e80 0x7f5b8c074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b90faa700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c10dd60 con 0x7f5b8c1022e0 2024-04-25T01:52:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b90faa700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c10dbe0 con 0x7f5b8c103e80 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b90faa700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c10dee0 con 0x7f5b8c1034e0 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b8a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c1022e0 0x7f5b8c075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b8a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c1022e0 0x7f5b8c075fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38096/0 (socket says 172.21.15.31:38096) 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2357682491 learned_addr learned my addr 172.21.15.31:0/2357682491 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b89d9b700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b8c1034e0 0x7f5b8c074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b8ad9d700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b8c103e80 0x7f5b8c074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2357682491 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b8c1034e0 msgr2=0x7f5b8c074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b8c1034e0 0x7f5b8c074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2357682491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b8c103e80 msgr2=0x7f5b8c074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b8c103e80 0x7f5b8c074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2357682491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8c0750a0 con 0x7f5b8c1022e0 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.561+0000 7f5b8ad9d700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b8c103e80 0x7f5b8c074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.562+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c1022e0 0x7f5b8c075fd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00d980 tx=0x7f5b7c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.562+0000 7f5b837fe700 1 -- 172.21.15.31:0/2357682491 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c017030 con 0x7f5b8c1022e0 2024-04-25T01:52:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.562+0000 7f5b837fe700 1 -- 172.21.15.31:0/2357682491 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b7c0044d0 con 0x7f5b8c1022e0 2024-04-25T01:52:36.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.562+0000 7f5b90faa700 1 -- 172.21.15.31:0/2357682491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b8c075390 con 0x7f5b8c1022e0 2024-04-25T01:52:36.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.562+0000 7f5b837fe700 1 -- 172.21.15.31:0/2357682491 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0039e0 con 0x7f5b8c1022e0 2024-04-25T01:52:36.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.562+0000 7f5b90faa700 1 -- 172.21.15.31:0/2357682491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b8c1a6de0 con 0x7f5b8c1022e0 2024-04-25T01:52:36.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.563+0000 7f5b90faa700 1 -- 172.21.15.31:0/2357682491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b6c005360 con 0x7f5b8c1022e0 2024-04-25T01:52:36.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.564+0000 7f5b837fe700 1 -- 172.21.15.31:0/2357682491 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5b7c01a070 con 0x7f5b8c1022e0 2024-04-25T01:52:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.564+0000 7f5b837fe700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b780a2de0 0x7f5b780a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.565+0000 7f5b89d9b700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b780a2de0 0x7f5b780a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.565+0000 7f5b837fe700 1 -- 172.21.15.31:0/2357682491 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f5b7c0ce860 con 0x7f5b8c1022e0 2024-04-25T01:52:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.565+0000 7f5b89d9b700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b780a2de0 0x7f5b780a52a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5b74000c00 tx=0x7f5b74005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:36.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.569+0000 7f5b837fe700 1 -- 172.21.15.31:0/2357682491 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5b7c09b810 con 0x7f5b8c1022e0 2024-04-25T01:52:36.783 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:36 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b[62588]: 2024-04-25T01:52:36.485+0000 7fa3082fa700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T01:52:36.783 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 01:52:36 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b[62588]: 2024-04-25T01:52:36.485+0000 7fa3082fa700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T01:52:36.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.861+0000 7f5b90faa700 1 -- 172.21.15.31:0/2357682491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f5b6c005650 con 0x7f5b8c1022e0 2024-04-25T01:52:36.861 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:36 smithi031 ceph-mon[29204]: pgmap v293: 97 pgs: 97 active+clean; 604 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8.9 KiB/s wr, 26 op/s 2024-04-25T01:52:36.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.865+0000 7f5b837fe700 1 -- 172.21.15.31:0/2357682491 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f5b7c09b110 con 0x7f5b8c1022e0 2024-04-25T01:52:36.865 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-04-25T01:52:36.865 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBktylmtrFrMxAAimuwa09beVX849TBoNOo9w== 2024-04-25T01:52:36.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 -- 172.21.15.31:0/2357682491 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b780a2de0 msgr2=0x7f5b780a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b780a2de0 0x7f5b780a52a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5b74000c00 tx=0x7f5b74005fb0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 -- 172.21.15.31:0/2357682491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c1022e0 msgr2=0x7f5b8c075fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c1022e0 0x7f5b8c075fd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00d980 tx=0x7f5b7c00dc90 comp rx=0 tx=0).stop 2024-04-25T01:52:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 -- 172.21.15.31:0/2357682491 shutdown_connections 2024-04-25T01:52:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b780a2de0 0x7f5b780a52a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b8c103e80 0x7f5b8c074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c1022e0 0x7f5b8c075fd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 --2- 172.21.15.31:0/2357682491 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b8c1034e0 0x7f5b8c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 -- 172.21.15.31:0/2357682491 >> 172.21.15.31:0/2357682491 conn(0x7f5b8c0fd880 msgr2=0x7f5b8c106780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 -- 172.21.15.31:0/2357682491 shutdown_connections 2024-04-25T01:52:36.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:36.867+0000 7f5b90faa700 1 -- 172.21.15.31:0/2357682491 wait complete. 2024-04-25T01:52:37.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:36 smithi040 ceph-mon[32418]: pgmap v293: 97 pgs: 97 active+clean; 604 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8.9 KiB/s wr, 26 op/s 2024-04-25T01:52:37.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:36 smithi134 ceph-mon[33098]: pgmap v293: 97 pgs: 97 active+clean; 604 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8.9 KiB/s wr, 26 op/s 2024-04-25T01:52:37.510 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:52:37.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T01:52:37.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T01:52:37.549 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T01:52:37.549 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T01:52:37.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T01:52:37.839 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:38.060 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:37 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2357682491' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:52:38.060 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:37 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2357682491' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:52:38.154 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:37 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2357682491' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:52:38.154 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:37 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2357682491' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:52:38.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:37 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2357682491' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:52:38.222 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:37 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2357682491' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:52:38.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:38 smithi031 ceph-mon[29204]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 KiB/s wr, 26 op/s 2024-04-25T01:52:39.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:38 smithi040 ceph-mon[32418]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 KiB/s wr, 26 op/s 2024-04-25T01:52:39.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:38 smithi134 ceph-mon[33098]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 KiB/s wr, 26 op/s 2024-04-25T01:52:39.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.597+0000 7fb572904700 1 -- 172.21.15.31:0/1503892013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb56c107590 msgr2=0x7fb56c10f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.597+0000 7fb572904700 1 --2- 172.21.15.31:0/1503892013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb56c107590 0x7fb56c10f160 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb56400b0a0 tx=0x7fb56400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.597+0000 7fb572904700 1 -- 172.21.15.31:0/1503892013 shutdown_connections 2024-04-25T01:52:39.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.597+0000 7fb572904700 1 --2- 172.21.15.31:0/1503892013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb56c107590 0x7fb56c10f160 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.597+0000 7fb572904700 1 --2- 172.21.15.31:0/1503892013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c102510 0x7fb56c1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.597+0000 7fb572904700 1 --2- 172.21.15.31:0/1503892013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb56c102ec0 0x7fb56c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.597+0000 7fb572904700 1 -- 172.21.15.31:0/1503892013 >> 172.21.15.31:0/1503892013 conn(0x7fb56c0793a0 msgr2=0x7fb56c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:39.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.597+0000 7fb572904700 1 -- 172.21.15.31:0/1503892013 shutdown_connections 2024-04-25T01:52:39.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.597+0000 7fb572904700 1 -- 172.21.15.31:0/1503892013 wait complete. 2024-04-25T01:52:39.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb572904700 1 Processor -- start 2024-04-25T01:52:39.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb572904700 1 -- start start 2024-04-25T01:52:39.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb572904700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb56c102510 0x7fb56c074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb572904700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb56c102ec0 0x7fb56c074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb572904700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c107590 0x7fb56c074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb572904700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c112480 con 0x7fb56c102510 2024-04-25T01:52:39.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb572904700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c112300 con 0x7fb56c107590 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb572904700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c112600 con 0x7fb56c102ec0 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb56bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb56c102510 0x7fb56c074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb56bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb56c102510 0x7fb56c074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38110/0 (socket says 172.21.15.31:38110) 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb56bfff700 1 -- 172.21.15.31:0/1637641253 learned_addr learned my addr 172.21.15.31:0/1637641253 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.598+0000 7fb570ea1700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c107590 0x7fb56c074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb56b7fe700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb56c102ec0 0x7fb56c074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb56bfff700 1 -- 172.21.15.31:0/1637641253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb56c102ec0 msgr2=0x7fb56c074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb56bfff700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb56c102ec0 0x7fb56c074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb56bfff700 1 -- 172.21.15.31:0/1637641253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c107590 msgr2=0x7fb56c074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb56bfff700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c107590 0x7fb56c074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb56bfff700 1 -- 172.21.15.31:0/1637641253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c1a6ea0 con 0x7fb56c102510 2024-04-25T01:52:39.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb56b7fe700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb56c102ec0 0x7fb56c074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:52:39.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb570ea1700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c107590 0x7fb56c074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:39.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb56bfff700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb56c102510 0x7fb56c074530 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00d980 tx=0x7fb55c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:39.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb5697fa700 1 -- 172.21.15.31:0/1637641253 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c016070 con 0x7fb56c102510 2024-04-25T01:52:39.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb5697fa700 1 -- 172.21.15.31:0/1637641253 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb55c004830 con 0x7fb56c102510 2024-04-25T01:52:39.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb572904700 1 -- 172.21.15.31:0/1637641253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56c1a7190 con 0x7fb56c102510 2024-04-25T01:52:39.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb5697fa700 1 -- 172.21.15.31:0/1637641253 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0054b0 con 0x7fb56c102510 2024-04-25T01:52:39.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.599+0000 7fb572904700 1 -- 172.21.15.31:0/1637641253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb56c1a76d0 con 0x7fb56c102510 2024-04-25T01:52:39.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.600+0000 7fb572904700 1 -- 172.21.15.31:0/1637641253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb56c068f20 con 0x7fb56c102510 2024-04-25T01:52:39.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.602+0000 7fb5697fa700 1 -- 172.21.15.31:0/1637641253 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb55c0049e0 con 0x7fb56c102510 2024-04-25T01:52:39.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.603+0000 7fb5697fa700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb5540a2cb0 0x7fb5540a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.603+0000 7fb56b7fe700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb5540a2cb0 0x7fb5540a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.603+0000 7fb5697fa700 1 -- 172.21.15.31:0/1637641253 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fb55c0cf2d0 con 0x7fb56c102510 2024-04-25T01:52:39.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.603+0000 7fb56b7fe700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb5540a2cb0 0x7fb5540a5170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb56c108ac0 tx=0x7fb560008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:39.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.606+0000 7fb5697fa700 1 -- 172.21.15.31:0/1637641253 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb55c01f020 con 0x7fb56c102510 2024-04-25T01:52:39.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.877+0000 7fb572904700 1 -- 172.21.15.31:0/1637641253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb56c075a90 con 0x7fb56c102510 2024-04-25T01:52:39.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.882+0000 7fb5697fa700 1 -- 172.21.15.31:0/1637641253 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263200 (secure 0 0 0) 0x7fb56c075a90 con 0x7fb56c102510 2024-04-25T01:52:39.887 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:39.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.885+0000 7fb572904700 1 -- 172.21.15.31:0/1637641253 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb5540a2cb0 msgr2=0x7fb5540a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.885+0000 7fb572904700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb5540a2cb0 0x7fb5540a5170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb56c108ac0 tx=0x7fb560008040 comp rx=0 tx=0).stop 2024-04-25T01:52:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.885+0000 7fb572904700 1 -- 172.21.15.31:0/1637641253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb56c102510 msgr2=0x7fb56c074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.885+0000 7fb572904700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb56c102510 0x7fb56c074530 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00d980 tx=0x7fb55c00dc90 comp rx=0 tx=0).stop 2024-04-25T01:52:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.886+0000 7fb572904700 1 -- 172.21.15.31:0/1637641253 shutdown_connections 2024-04-25T01:52:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.886+0000 7fb572904700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb5540a2cb0 0x7fb5540a5170 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.886+0000 7fb572904700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c107590 0x7fb56c074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.886+0000 7fb572904700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb56c102510 0x7fb56c074530 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.886+0000 7fb572904700 1 --2- 172.21.15.31:0/1637641253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb56c102ec0 0x7fb56c074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.886+0000 7fb572904700 1 -- 172.21.15.31:0/1637641253 >> 172.21.15.31:0/1637641253 conn(0x7fb56c0793a0 msgr2=0x7fb56c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.886+0000 7fb572904700 1 -- 172.21.15.31:0/1637641253 shutdown_connections 2024-04-25T01:52:39.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:39.886+0000 7fb572904700 1 -- 172.21.15.31:0/1637641253 wait complete. 2024-04-25T01:52:40.540 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":2306032998},{"type":"v1","addr":"172.21.15.31:6801","nonce":2306032998}]},"active_addr":"172.21.15.31:6801/2306032998","active_change":"2024-04-25T01:45:23.647306+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.31:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1358173279}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3913527371}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3045817976}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":686937010}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2512212301}]}]} 2024-04-25T01:52:40.547 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T01:52:40.547 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T01:52:40.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:52:40.805 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:41.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:40 smithi040 ceph-mon[32418]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.1 KiB/s wr, 25 op/s 2024-04-25T01:52:41.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:40 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1637641253' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:52:41.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:40 smithi031 ceph-mon[29204]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.1 KiB/s wr, 25 op/s 2024-04-25T01:52:41.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:40 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1637641253' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:52:41.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:40 smithi134 ceph-mon[33098]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.1 KiB/s wr, 25 op/s 2024-04-25T01:52:41.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:40 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1637641253' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:52:42.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.466+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2914668447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68bc1082c0 msgr2=0x7f68bc1086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:42.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.466+0000 7f68c1b8b700 1 --2- 172.21.15.31:0/2914668447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68bc1082c0 0x7f68bc1086a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009990 tx=0x7f68ac009ca0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.466+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2914668447 shutdown_connections 2024-04-25T01:52:42.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.466+0000 7f68c1b8b700 1 --2- 172.21.15.31:0/2914668447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68bc1011c0 0x7f68bc10f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.466+0000 7f68c1b8b700 1 --2- 172.21.15.31:0/2914668447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68bc1082c0 0x7f68bc1086a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.466+0000 7f68c1b8b700 1 --2- 172.21.15.31:0/2914668447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68bc100820 0x7f68bc100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.466+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2914668447 >> 172.21.15.31:0/2914668447 conn(0x7f68bc0777a0 msgr2=0x7f68bc0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:42.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.466+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2914668447 shutdown_connections 2024-04-25T01:52:42.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.466+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2914668447 wait complete. 2024-04-25T01:52:42.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.467+0000 7f68c1b8b700 1 Processor -- start 2024-04-25T01:52:42.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.467+0000 7f68c1b8b700 1 -- start start 2024-04-25T01:52:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.467+0000 7f68c1b8b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68bc100820 0x7f68bc101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.467+0000 7f68c1b8b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68bc1011c0 0x7f68bc101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68c1b8b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68bc1082c0 0x7f68bc102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68c1b8b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc112430 con 0x7f68bc1082c0 2024-04-25T01:52:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68c1b8b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc1122b0 con 0x7f68bc1011c0 2024-04-25T01:52:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68c1b8b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc1125b0 con 0x7f68bc100820 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68bbfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68bc1082c0 0x7f68bc102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68bbfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68bc1082c0 0x7f68bc102480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38138/0 (socket says 172.21.15.31:38138) 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68bbfff700 1 -- 172.21.15.31:0/2898112101 learned_addr learned my addr 172.21.15.31:0/2898112101 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68baffd700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68bc1011c0 0x7f68bc101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68bb7fe700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68bc100820 0x7f68bc101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68bbfff700 1 -- 172.21.15.31:0/2898112101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68bc100820 msgr2=0x7f68bc101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68bbfff700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68bc100820 0x7f68bc101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68bbfff700 1 -- 172.21.15.31:0/2898112101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68bc1011c0 msgr2=0x7f68bc101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68bbfff700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68bc1011c0 0x7f68bc101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68bbfff700 1 -- 172.21.15.31:0/2898112101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68bc1a6f50 con 0x7f68bc1082c0 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68bb7fe700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68bc100820 0x7f68bc101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68baffd700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68bc1011c0 0x7f68bc101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68bbfff700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68bc1082c0 0x7f68bc102480 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f68b400b970 tx=0x7f68b400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.468+0000 7f68b8ff9700 1 -- 172.21.15.31:0/2898112101 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b4019070 con 0x7f68bc1082c0 2024-04-25T01:52:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.469+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2898112101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68bc1a7240 con 0x7f68bc1082c0 2024-04-25T01:52:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.469+0000 7f68b8ff9700 1 -- 172.21.15.31:0/2898112101 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68b4014440 con 0x7f68bc1082c0 2024-04-25T01:52:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.469+0000 7f68b8ff9700 1 -- 172.21.15.31:0/2898112101 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b400ce60 con 0x7f68bc1082c0 2024-04-25T01:52:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.469+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2898112101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68bc1a7780 con 0x7f68bc1082c0 2024-04-25T01:52:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.470+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2898112101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68bc103b40 con 0x7f68bc1082c0 2024-04-25T01:52:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.471+0000 7f68b8ff9700 1 -- 172.21.15.31:0/2898112101 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f68b401c070 con 0x7f68bc1082c0 2024-04-25T01:52:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.471+0000 7f68b8ff9700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f68a40a2d30 0x7f68a40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.471+0000 7f68b8ff9700 1 -- 172.21.15.31:0/2898112101 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f68b40ce300 con 0x7f68bc1082c0 2024-04-25T01:52:42.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.471+0000 7f68bb7fe700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f68a40a2d30 0x7f68a40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:42.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.472+0000 7f68bb7fe700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f68a40a2d30 0x7f68a40a51f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00b5c0 tx=0x7f68ac005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:42.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.476+0000 7f68b8ff9700 1 -- 172.21.15.31:0/2898112101 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68b409b3c0 con 0x7f68bc1082c0 2024-04-25T01:52:42.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.683+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2898112101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68bc102f30 con 0x7f68bc1082c0 2024-04-25T01:52:42.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.684+0000 7f68b8ff9700 1 -- 172.21.15.31:0/2898112101 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22147 (secure 0 0 0) 0x7f68b409acc0 con 0x7f68bc1082c0 2024-04-25T01:52:42.684 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:52:26.507181+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:52:42.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.686+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2898112101 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f68a40a2d30 msgr2=0x7f68a40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:42.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.686+0000 7f68c1b8b700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f68a40a2d30 0x7f68a40a51f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00b5c0 tx=0x7f68ac005a90 comp rx=0 tx=0).stop 2024-04-25T01:52:42.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.686+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2898112101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68bc1082c0 msgr2=0x7f68bc102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:42.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.686+0000 7f68c1b8b700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68bc1082c0 0x7f68bc102480 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f68b400b970 tx=0x7f68b400bd30 comp rx=0 tx=0).stop 2024-04-25T01:52:42.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.686+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2898112101 shutdown_connections 2024-04-25T01:52:42.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.686+0000 7f68c1b8b700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f68a40a2d30 0x7f68a40a51f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.686+0000 7f68c1b8b700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68bc1082c0 0x7f68bc102480 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.686+0000 7f68c1b8b700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68bc100820 0x7f68bc101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.686+0000 7f68c1b8b700 1 --2- 172.21.15.31:0/2898112101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68bc1011c0 0x7f68bc101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.687+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2898112101 >> 172.21.15.31:0/2898112101 conn(0x7f68bc0777a0 msgr2=0x7f68bc0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:42.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.687+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2898112101 shutdown_connections 2024-04-25T01:52:42.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:42.687+0000 7f68c1b8b700 1 -- 172.21.15.31:0/2898112101 wait complete. 2024-04-25T01:52:42.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:42 smithi031 ceph-mon[29204]: pgmap v296: 97 pgs: 97 active+clean; 626 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.7 KiB/s wr, 24 op/s 2024-04-25T01:52:42.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:42 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2898112101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:43.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:42 smithi040 ceph-mon[32418]: pgmap v296: 97 pgs: 97 active+clean; 626 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.7 KiB/s wr, 24 op/s 2024-04-25T01:52:43.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:42 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2898112101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:43.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:42 smithi134 ceph-mon[33098]: pgmap v296: 97 pgs: 97 active+clean; 626 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.7 KiB/s wr, 24 op/s 2024-04-25T01:52:43.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:42 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2898112101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:43.344 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T01:52:43.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:52:43.594 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:44.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:44 smithi031 ceph-mon[29204]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.1 KiB/s wr, 21 op/s 2024-04-25T01:52:45.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:44 smithi040 ceph-mon[32418]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.1 KiB/s wr, 21 op/s 2024-04-25T01:52:45.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:44 smithi134 ceph-mon[33098]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.1 KiB/s wr, 21 op/s 2024-04-25T01:52:45.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.289+0000 7fb045541700 1 -- 172.21.15.31:0/3055084602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0401075a0 msgr2=0x7fb04010f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:45.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.289+0000 7fb045541700 1 --2- 172.21.15.31:0/3055084602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0401075a0 0x7fb04010f170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb03800b0a0 tx=0x7fb03800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.289+0000 7fb045541700 1 -- 172.21.15.31:0/3055084602 shutdown_connections 2024-04-25T01:52:45.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.289+0000 7fb045541700 1 --2- 172.21.15.31:0/3055084602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0401075a0 0x7fb04010f170 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.289+0000 7fb045541700 1 --2- 172.21.15.31:0/3055084602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0401024e0 0x7fb0401028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.289+0000 7fb045541700 1 --2- 172.21.15.31:0/3055084602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb040102e90 0x7fb040106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.289+0000 7fb045541700 1 -- 172.21.15.31:0/3055084602 >> 172.21.15.31:0/3055084602 conn(0x7fb0400793e0 msgr2=0x7fb0400797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:45.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.289+0000 7fb045541700 1 -- 172.21.15.31:0/3055084602 shutdown_connections 2024-04-25T01:52:45.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.289+0000 7fb045541700 1 -- 172.21.15.31:0/3055084602 wait complete. 2024-04-25T01:52:45.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.290+0000 7fb045541700 1 Processor -- start 2024-04-25T01:52:45.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.290+0000 7fb045541700 1 -- start start 2024-04-25T01:52:45.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.290+0000 7fb045541700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0401024e0 0x7fb04019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:45.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.290+0000 7fb045541700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb040102e90 0x7fb04019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:45.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.290+0000 7fb045541700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0401075a0 0x7fb040198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:45.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.290+0000 7fb045541700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb040112430 con 0x7fb0401024e0 2024-04-25T01:52:45.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.290+0000 7fb045541700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb0401122b0 con 0x7fb040102e90 2024-04-25T01:52:45.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.290+0000 7fb045541700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb0401125b0 con 0x7fb0401075a0 2024-04-25T01:52:45.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb037fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb040102e90 0x7fb04019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:45.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb03effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0401024e0 0x7fb04019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:45.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb03f7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0401075a0 0x7fb040198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb037fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb040102e90 0x7fb04019eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35754/0 (socket says 172.21.15.31:35754) 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb03f7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0401075a0 0x7fb040198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59696/0 (socket says 172.21.15.31:59696) 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb03effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0401024e0 0x7fb04019e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37136/0 (socket says 172.21.15.31:37136) 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb037fff700 1 -- 172.21.15.31:0/3780599632 learned_addr learned my addr 172.21.15.31:0/3780599632 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb03effd700 1 -- 172.21.15.31:0/3780599632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0401075a0 msgr2=0x7fb040198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb03effd700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0401075a0 0x7fb040198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb03effd700 1 -- 172.21.15.31:0/3780599632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb040102e90 msgr2=0x7fb04019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb03effd700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb040102e90 0x7fb04019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb03effd700 1 -- 172.21.15.31:0/3780599632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb040199010 con 0x7fb0401024e0 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb03f7fe700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0401075a0 0x7fb040198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.291+0000 7fb03effd700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0401024e0 0x7fb04019e660 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb03000d750 tx=0x7fb03000da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:45.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.292+0000 7fb03cff9700 1 -- 172.21.15.31:0/3780599632 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb030016070 con 0x7fb0401024e0 2024-04-25T01:52:45.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.292+0000 7fb03cff9700 1 -- 172.21.15.31:0/3780599632 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb030005020 con 0x7fb0401024e0 2024-04-25T01:52:45.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.292+0000 7fb045541700 1 -- 172.21.15.31:0/3780599632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb040199300 con 0x7fb0401024e0 2024-04-25T01:52:45.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.292+0000 7fb03cff9700 1 -- 172.21.15.31:0/3780599632 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03000f9f0 con 0x7fb0401024e0 2024-04-25T01:52:45.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.292+0000 7fb045541700 1 -- 172.21.15.31:0/3780599632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0401a55e0 con 0x7fb0401024e0 2024-04-25T01:52:45.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.294+0000 7fb045541700 1 -- 172.21.15.31:0/3780599632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0401039a0 con 0x7fb0401024e0 2024-04-25T01:52:45.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.295+0000 7fb03cff9700 1 -- 172.21.15.31:0/3780599632 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb0300051d0 con 0x7fb0401024e0 2024-04-25T01:52:45.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.295+0000 7fb03cff9700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb02c0a2d30 0x7fb02c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:45.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.296+0000 7fb03cff9700 1 -- 172.21.15.31:0/3780599632 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fb0300ce970 con 0x7fb0401024e0 2024-04-25T01:52:45.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.296+0000 7fb037fff700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb02c0a2d30 0x7fb02c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:45.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.296+0000 7fb037fff700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb02c0a2d30 0x7fb02c0a51f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb040199ef0 tx=0x7fb028009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:45.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.300+0000 7fb03cff9700 1 -- 172.21.15.31:0/3780599632 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb0300d0e20 con 0x7fb0401024e0 2024-04-25T01:52:45.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.507+0000 7fb045541700 1 -- 172.21.15.31:0/3780599632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb040199c60 con 0x7fb0401024e0 2024-04-25T01:52:45.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.508+0000 7fb03cff9700 1 -- 172.21.15.31:0/3780599632 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22147 (secure 0 0 0) 0x7fb03009b920 con 0x7fb0401024e0 2024-04-25T01:52:45.509 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:52:45.509 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:52:26.507181+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.510+0000 7fb045541700 1 -- 172.21.15.31:0/3780599632 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb02c0a2d30 msgr2=0x7fb02c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.510+0000 7fb045541700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb02c0a2d30 0x7fb02c0a51f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb040199ef0 tx=0x7fb028009260 comp rx=0 tx=0).stop 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.510+0000 7fb045541700 1 -- 172.21.15.31:0/3780599632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0401024e0 msgr2=0x7fb04019e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.510+0000 7fb045541700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0401024e0 0x7fb04019e660 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb03000d750 tx=0x7fb03000da60 comp rx=0 tx=0).stop 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.511+0000 7fb045541700 1 -- 172.21.15.31:0/3780599632 shutdown_connections 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.511+0000 7fb045541700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb02c0a2d30 0x7fb02c0a51f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.511+0000 7fb045541700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0401075a0 0x7fb040198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.511+0000 7fb045541700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0401024e0 0x7fb04019e660 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.511+0000 7fb045541700 1 --2- 172.21.15.31:0/3780599632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb040102e90 0x7fb04019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.511+0000 7fb045541700 1 -- 172.21.15.31:0/3780599632 >> 172.21.15.31:0/3780599632 conn(0x7fb0400793e0 msgr2=0x7fb040100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.511+0000 7fb045541700 1 -- 172.21.15.31:0/3780599632 shutdown_connections 2024-04-25T01:52:45.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:45.511+0000 7fb045541700 1 -- 172.21.15.31:0/3780599632 wait complete. 2024-04-25T01:52:46.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:45 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3780599632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:46.174 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:45 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3780599632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:46.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:45 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3780599632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:46.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T01:52:46.204 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T01:52:46.204 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T01:52:46.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T01:52:46.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T01:52:46.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T01:52:46.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T01:52:46.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T01:52:46.207 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T01:52:46.207 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T01:52:46.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T01:52:46.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T01:52:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:46.920 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:46.922 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:47.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:46 smithi040 ceph-mon[32418]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 KiB/s wr, 21 op/s 2024-04-25T01:52:47.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:46 smithi134 ceph-mon[33098]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 KiB/s wr, 21 op/s 2024-04-25T01:52:47.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:46 smithi031 ceph-mon[29204]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 KiB/s wr, 21 op/s 2024-04-25T01:52:48.014 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:48.014 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:48.014 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:48.015 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:48.015 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:48.015 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:48.309 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:48.310 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:49.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:48 smithi031 ceph-mon[29204]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-04-25T01:52:49.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:48 smithi040 ceph-mon[32418]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-04-25T01:52:49.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:48 smithi134 ceph-mon[33098]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-04-25T01:52:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.625+0000 7fcfb4d14700 1 -- 172.21.15.31:0/1696549792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfb0102f00 msgr2=0x7fcfb0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.625+0000 7fcfb4d14700 1 --2- 172.21.15.31:0/1696549792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfb0102f00 0x7fcfb0106e60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40099c0 tx=0x7fcfa4009cd0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.625+0000 7fcfb4d14700 1 -- 172.21.15.31:0/1696549792 shutdown_connections 2024-04-25T01:52:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.625+0000 7fcfb4d14700 1 --2- 172.21.15.31:0/1696549792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcfb0107540 0x7fcfb010f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.625+0000 7fcfb4d14700 1 --2- 172.21.15.31:0/1696549792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfb0102550 0x7fcfb0102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.625+0000 7fcfb4d14700 1 --2- 172.21.15.31:0/1696549792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfb0102f00 0x7fcfb0106e60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.625+0000 7fcfb4d14700 1 -- 172.21.15.31:0/1696549792 >> 172.21.15.31:0/1696549792 conn(0x7fcfb00793f0 msgr2=0x7fcfb0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:50.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.625+0000 7fcfb4d14700 1 -- 172.21.15.31:0/1696549792 shutdown_connections 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.625+0000 7fcfb4d14700 1 -- 172.21.15.31:0/1696549792 wait complete. 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfb4d14700 1 Processor -- start 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfb4d14700 1 -- start start 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfb4d14700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfb0102550 0x7fcfb019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfb4d14700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcfb0102f00 0x7fcfb019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfb4d14700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfb0107540 0x7fcfb0198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfb4d14700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb0112480 con 0x7fcfb0107540 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfb4d14700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb0112300 con 0x7fcfb0102550 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfb4d14700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb0112600 con 0x7fcfb0102f00 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfaed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfb0107540 0x7fcfb0198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfaed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfb0107540 0x7fcfb0198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37154/0 (socket says 172.21.15.31:37154) 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfaed9d700 1 -- 172.21.15.31:0/3759161182 learned_addr learned my addr 172.21.15.31:0/3759161182 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfadd9b700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcfb0102f00 0x7fcfb019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfae59c700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfb0102550 0x7fcfb019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfaed9d700 1 -- 172.21.15.31:0/3759161182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcfb0102f00 msgr2=0x7fcfb019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfaed9d700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcfb0102f00 0x7fcfb019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfaed9d700 1 -- 172.21.15.31:0/3759161182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfb0102550 msgr2=0x7fcfb019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfaed9d700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfb0102550 0x7fcfb019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.626+0000 7fcfaed9d700 1 -- 172.21.15.31:0/3759161182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb0199020 con 0x7fcfb0107540 2024-04-25T01:52:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.627+0000 7fcfaed9d700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfb0107540 0x7fcfb0198840 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800b970 tx=0x7fcfa800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.627+0000 7fcf9f7fe700 1 -- 172.21.15.31:0/3759161182 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8019070 con 0x7fcfb0107540 2024-04-25T01:52:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.627+0000 7fcf9f7fe700 1 -- 172.21.15.31:0/3759161182 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfa8014440 con 0x7fcfb0107540 2024-04-25T01:52:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.627+0000 7fcf9f7fe700 1 -- 172.21.15.31:0/3759161182 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa800ce60 con 0x7fcfb0107540 2024-04-25T01:52:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.627+0000 7fcfb4d14700 1 -- 172.21.15.31:0/3759161182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb0199310 con 0x7fcfb0107540 2024-04-25T01:52:50.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.627+0000 7fcfb4d14700 1 -- 172.21.15.31:0/3759161182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb01a54d0 con 0x7fcfb0107540 2024-04-25T01:52:50.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.628+0000 7fcfb4d14700 1 -- 172.21.15.31:0/3759161182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcfb0062980 con 0x7fcfb0107540 2024-04-25T01:52:50.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.629+0000 7fcf9f7fe700 1 -- 172.21.15.31:0/3759161182 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcfa801c070 con 0x7fcfb0107540 2024-04-25T01:52:50.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.630+0000 7fcf9f7fe700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fcf980a2cb0 0x7fcf980a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.630+0000 7fcfae59c700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fcf980a2cb0 0x7fcf980a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.630+0000 7fcf9f7fe700 1 -- 172.21.15.31:0/3759161182 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fcfa80ceb60 con 0x7fcfb0107540 2024-04-25T01:52:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.630+0000 7fcf9f7fe700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401] conn(0x7fcf980ac490 0x7fcf980ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.630+0000 7fcf9f7fe700 1 -- 172.21.15.31:0/3759161182 --> [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfa8004860 con 0x7fcf980ac490 2024-04-25T01:52:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.630+0000 7fcf9f7fe700 1 -- 172.21.15.31:0/3759161182 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fcfa80cf040 con 0x7fcfb0107540 2024-04-25T01:52:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.630+0000 7fcfadd9b700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401] conn(0x7fcf980ac490 0x7fcf980ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.630+0000 7fcfae59c700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fcf980a2cb0 0x7fcf980a5170 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0005950 tx=0x7fcfa00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.631+0000 7fcfadd9b700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401] conn(0x7fcf980ac490 0x7fcf980ae8b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:50.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.631+0000 7fcf9f7fe700 1 -- 172.21.15.31:0/3759161182 <== osd.0 v2:172.21.15.31:6802/268630401 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfa8004860 con 0x7fcf980ac490 2024-04-25T01:52:50.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.647+0000 7fcfb4d14700 1 -- 172.21.15.31:0/3759161182 --> [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcfb0062e00 con 0x7fcf980ac490 2024-04-25T01:52:50.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcf9f7fe700 1 -- 172.21.15.31:0/3759161182 <== osd.0 v2:172.21.15.31:6802/268630401 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcfb0062e00 con 0x7fcf980ac490 2024-04-25T01:52:50.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcfb4d14700 1 -- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401] conn(0x7fcf980ac490 msgr2=0x7fcf980ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcfb4d14700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401] conn(0x7fcf980ac490 0x7fcf980ae8b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcfb4d14700 1 -- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fcf980a2cb0 msgr2=0x7fcf980a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcfb4d14700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fcf980a2cb0 0x7fcf980a5170 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0005950 tx=0x7fcfa00058e0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcfb4d14700 1 -- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfb0107540 msgr2=0x7fcfb0198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcfb4d14700 1 --2- 172.21.15.31:0/3759161182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcfb0107540 0x7fcfb0198840 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800b970 tx=0x7fcfa800bd30 comp rx=0 tx=0).stop 2024-04-25T01:52:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcfaed9d700 1 -- 172.21.15.31:0/3759161182 reap_dead start 2024-04-25T01:52:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcfb4d14700 1 -- 172.21.15.31:0/3759161182 shutdown_connections 2024-04-25T01:52:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcfb4d14700 1 -- 172.21.15.31:0/3759161182 >> 172.21.15.31:0/3759161182 conn(0x7fcfb00793f0 msgr2=0x7fcfb0100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcfb4d14700 1 -- 172.21.15.31:0/3759161182 shutdown_connections 2024-04-25T01:52:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:50.648+0000 7fcfb4d14700 1 -- 172.21.15.31:0/3759161182 wait complete. 2024-04-25T01:52:51.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:50 smithi040 ceph-mon[32418]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T01:52:51.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:50 smithi134 ceph-mon[33098]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T01:52:51.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:50 smithi031 ceph-mon[29204]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T01:52:53.158 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:52 smithi040 ceph-mon[32418]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T01:52:53.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:52 smithi134 ceph-mon[33098]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T01:52:53.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:52 smithi031 ceph-mon[29204]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T01:52:53.405 INFO:teuthology.orchestra.run.smithi031.stdout:34359738441 2024-04-25T01:52:53.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T01:52:55.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:54 smithi040 ceph-mon[32418]: pgmap v302: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T01:52:55.161 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:54 smithi031 ceph-mon[29204]: pgmap v302: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T01:52:55.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:54 smithi134 ceph-mon[33098]: pgmap v302: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T01:52:57.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:56 smithi040 ceph-mon[32418]: pgmap v303: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T01:52:57.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:56 smithi134 ceph-mon[33098]: pgmap v303: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T01:52:57.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:56 smithi031 ceph-mon[29204]: pgmap v303: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T01:52:59.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.785+0000 7fd2d429a700 1 -- 172.21.15.31:0/3643159358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2cc1082c0 msgr2=0x7fd2cc1086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:59.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.785+0000 7fd2d429a700 1 --2- 172.21.15.31:0/3643159358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2cc1082c0 0x7fd2cc1086a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009990 tx=0x7fd2c8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:52:59.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.785+0000 7fd2d429a700 1 -- 172.21.15.31:0/3643159358 shutdown_connections 2024-04-25T01:52:59.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.785+0000 7fd2d429a700 1 --2- 172.21.15.31:0/3643159358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2cc1011c0 0x7fd2cc10f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:59.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.785+0000 7fd2d429a700 1 --2- 172.21.15.31:0/3643159358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2cc1082c0 0x7fd2cc1086a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:59.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.785+0000 7fd2d429a700 1 --2- 172.21.15.31:0/3643159358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2cc100820 0x7fd2cc100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:59.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.785+0000 7fd2d429a700 1 -- 172.21.15.31:0/3643159358 >> 172.21.15.31:0/3643159358 conn(0x7fd2cc0777a0 msgr2=0x7fd2cc0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:59.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.785+0000 7fd2d429a700 1 -- 172.21.15.31:0/3643159358 shutdown_connections 2024-04-25T01:52:59.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.785+0000 7fd2d429a700 1 -- 172.21.15.31:0/3643159358 wait complete. 2024-04-25T01:52:59.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.785+0000 7fd2d429a700 1 Processor -- start 2024-04-25T01:52:59.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d429a700 1 -- start start 2024-04-25T01:52:59.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d429a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2cc100820 0x7fd2cc19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:59.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d429a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2cc1011c0 0x7fd2cc19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:59.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d429a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2cc1082c0 0x7fd2cc1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:59.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d429a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc1123d0 con 0x7fd2cc100820 2024-04-25T01:52:59.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d429a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc112250 con 0x7fd2cc1082c0 2024-04-25T01:52:59.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d429a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc112550 con 0x7fd2cc1011c0 2024-04-25T01:52:59.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d2036700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2cc100820 0x7fd2cc19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:59.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d2036700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2cc100820 0x7fd2cc19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45404/0 (socket says 172.21.15.31:45404) 2024-04-25T01:52:59.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d2036700 1 -- 172.21.15.31:0/2572216964 learned_addr learned my addr 172.21.15.31:0/2572216964 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:52:59.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d2837700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2cc1082c0 0x7fd2cc1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:59.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d1835700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2cc1011c0 0x7fd2cc19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:59.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d2036700 1 -- 172.21.15.31:0/2572216964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2cc1011c0 msgr2=0x7fd2cc19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:59.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d2036700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd2cc1011c0 0x7fd2cc19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:59.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d2036700 1 -- 172.21.15.31:0/2572216964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2cc1082c0 msgr2=0x7fd2cc1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:59.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d2036700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2cc1082c0 0x7fd2cc1987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:59.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d2036700 1 -- 172.21.15.31:0/2572216964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2cc198fb0 con 0x7fd2cc100820 2024-04-25T01:52:59.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.786+0000 7fd2d2036700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2cc100820 0x7fd2cc19e650 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8000c00 tx=0x7fd2c80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:59.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.787+0000 7fd2c37fe700 1 -- 172.21.15.31:0/2572216964 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8027070 con 0x7fd2cc100820 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.787+0000 7fd2c37fe700 1 -- 172.21.15.31:0/2572216964 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2c8022470 con 0x7fd2cc100820 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.787+0000 7fd2d429a700 1 -- 172.21.15.31:0/2572216964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2cc199240 con 0x7fd2cc100820 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.787+0000 7fd2c37fe700 1 -- 172.21.15.31:0/2572216964 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8017650 con 0x7fd2cc100820 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.787+0000 7fd2d429a700 1 -- 172.21.15.31:0/2572216964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2cc1a54b0 con 0x7fd2cc100820 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.787+0000 7fd2d429a700 1 -- 172.21.15.31:0/2572216964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd2cc062940 con 0x7fd2cc100820 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.789+0000 7fd2c37fe700 1 -- 172.21.15.31:0/2572216964 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd2c802a070 con 0x7fd2cc100820 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.789+0000 7fd2c37fe700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd2b80a2cb0 0x7fd2b80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.790+0000 7fd2d1835700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd2b80a2cb0 0x7fd2b80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.790+0000 7fd2c37fe700 1 -- 172.21.15.31:0/2572216964 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fd2c80d02f0 con 0x7fd2cc100820 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.790+0000 7fd2c37fe700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525] conn(0x7fd2b80ac490 0x7fd2b80ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.790+0000 7fd2c37fe700 1 -- 172.21.15.31:0/2572216964 --> [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2c802ae50 con 0x7fd2b80ac490 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.790+0000 7fd2c37fe700 1 -- 172.21.15.31:0/2572216964 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fd2c80d06a0 con 0x7fd2cc100820 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.790+0000 7fd2d2837700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525] conn(0x7fd2b80ac490 0x7fd2b80ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:59.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.790+0000 7fd2d1835700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd2b80a2cb0 0x7fd2b80a5170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0fe8f0 tx=0x7fd2bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.790+0000 7fd2d2837700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525] conn(0x7fd2b80ac490 0x7fd2b80ae8b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.791+0000 7fd2c37fe700 1 -- 172.21.15.31:0/2572216964 <== osd.1 v2:172.21.15.40:6800/3593667525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2c802ae50 con 0x7fd2b80ac490 2024-04-25T01:52:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.808+0000 7fd2d429a700 1 -- 172.21.15.31:0/2572216964 --> [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd2cc062dc0 con 0x7fd2b80ac490 2024-04-25T01:52:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.808+0000 7fd2c37fe700 1 -- 172.21.15.31:0/2572216964 <== osd.1 v2:172.21.15.40:6800/3593667525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd2cc062dc0 con 0x7fd2b80ac490 2024-04-25T01:52:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.808+0000 7fd2d429a700 1 -- 172.21.15.31:0/2572216964 >> [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525] conn(0x7fd2b80ac490 msgr2=0x7fd2b80ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.808+0000 7fd2d429a700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525] conn(0x7fd2b80ac490 0x7fd2b80ae8b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.809+0000 7fd2d429a700 1 -- 172.21.15.31:0/2572216964 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd2b80a2cb0 msgr2=0x7fd2b80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.809+0000 7fd2d429a700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd2b80a2cb0 0x7fd2b80a5170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0fe8f0 tx=0x7fd2bc009260 comp rx=0 tx=0).stop 2024-04-25T01:52:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.809+0000 7fd2d429a700 1 -- 172.21.15.31:0/2572216964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2cc100820 msgr2=0x7fd2cc19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.809+0000 7fd2d429a700 1 --2- 172.21.15.31:0/2572216964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2cc100820 0x7fd2cc19e650 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8000c00 tx=0x7fd2c80047b0 comp rx=0 tx=0).stop 2024-04-25T01:52:59.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.809+0000 7fd2d2837700 1 -- 172.21.15.31:0/2572216964 reap_dead start 2024-04-25T01:52:59.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.809+0000 7fd2d429a700 1 -- 172.21.15.31:0/2572216964 shutdown_connections 2024-04-25T01:52:59.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.809+0000 7fd2d429a700 1 -- 172.21.15.31:0/2572216964 >> 172.21.15.31:0/2572216964 conn(0x7fd2cc0777a0 msgr2=0x7fd2cc0ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:59.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.809+0000 7fd2d429a700 1 -- 172.21.15.31:0/2572216964 shutdown_connections 2024-04-25T01:52:59.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:52:58.809+0000 7fd2d429a700 1 -- 172.21.15.31:0/2572216964 wait complete. 2024-04-25T01:52:59.125 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:52:58 smithi031 ceph-mon[29204]: pgmap v304: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T01:52:59.159 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:52:58 smithi040 ceph-mon[32418]: pgmap v304: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T01:52:59.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:52:58 smithi134 ceph-mon[33098]: pgmap v304: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T01:53:01.051 INFO:teuthology.orchestra.run.smithi031.stdout:55834574918 2024-04-25T01:53:01.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T01:53:01.168 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:00 smithi031 ceph-mon[29204]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T01:53:01.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:00 smithi134 ceph-mon[33098]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T01:53:01.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:00 smithi040 ceph-mon[32418]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T01:53:01.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.558+0000 7fdf1247d700 1 -- 172.21.15.31:0/4146262717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c1082a0 msgr2=0x7fdf0c108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.558+0000 7fdf1247d700 1 --2- 172.21.15.31:0/4146262717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c1082a0 0x7fdf0c108680 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009990 tx=0x7fdefc009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.558+0000 7fdf1247d700 1 -- 172.21.15.31:0/4146262717 shutdown_connections 2024-04-25T01:53:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.558+0000 7fdf1247d700 1 --2- 172.21.15.31:0/4146262717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf0c06b170 0x7fdf0c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.558+0000 7fdf1247d700 1 --2- 172.21.15.31:0/4146262717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c1082a0 0x7fdf0c108680 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.558+0000 7fdf1247d700 1 --2- 172.21.15.31:0/4146262717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf0c06a7d0 0x7fdf0c06ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.558+0000 7fdf1247d700 1 -- 172.21.15.31:0/4146262717 >> 172.21.15.31:0/4146262717 conn(0x7fdf0c0780a0 msgr2=0x7fdf0c0784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.558+0000 7fdf1247d700 1 -- 172.21.15.31:0/4146262717 shutdown_connections 2024-04-25T01:53:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.558+0000 7fdf1247d700 1 -- 172.21.15.31:0/4146262717 wait complete. 2024-04-25T01:53:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.558+0000 7fdf1247d700 1 Processor -- start 2024-04-25T01:53:01.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf1247d700 1 -- start start 2024-04-25T01:53:01.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf1247d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf0c06a7d0 0x7fdf0c1011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:01.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf1247d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf0c06b170 0x7fdf0c101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf1247d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c1082a0 0x7fdf0c101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf1247d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c112410 con 0x7fdf0c1082a0 2024-04-25T01:53:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf1247d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c112290 con 0x7fdf0c06a7d0 2024-04-25T01:53:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf1247d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c112590 con 0x7fdf0c06b170 2024-04-25T01:53:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf10a1a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c1082a0 0x7fdf0c101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf10a1a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c1082a0 0x7fdf0c101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45426/0 (socket says 172.21.15.31:45426) 2024-04-25T01:53:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf10a1a700 1 -- 172.21.15.31:0/202623075 learned_addr learned my addr 172.21.15.31:0/202623075 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf0bfff700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf0c06a7d0 0x7fdf0c1011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:01.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf0b7fe700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf0c06b170 0x7fdf0c101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf10a1a700 1 -- 172.21.15.31:0/202623075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf0c06b170 msgr2=0x7fdf0c101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf10a1a700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf0c06b170 0x7fdf0c101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf10a1a700 1 -- 172.21.15.31:0/202623075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf0c06a7d0 msgr2=0x7fdf0c1011f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf10a1a700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf0c06a7d0 0x7fdf0c1011f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.559+0000 7fdf10a1a700 1 -- 172.21.15.31:0/202623075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0c1a6ed0 con 0x7fdf0c1082a0 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.560+0000 7fdf10a1a700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c1082a0 0x7fdf0c101c70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400b970 tx=0x7fdf0400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.560+0000 7fdf097fa700 1 -- 172.21.15.31:0/202623075 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf04019070 con 0x7fdf0c1082a0 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.560+0000 7fdf097fa700 1 -- 172.21.15.31:0/202623075 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf0400cde0 con 0x7fdf0c1082a0 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.560+0000 7fdf1247d700 1 -- 172.21.15.31:0/202623075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf0c1a71c0 con 0x7fdf0c1082a0 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.560+0000 7fdf097fa700 1 -- 172.21.15.31:0/202623075 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf04007370 con 0x7fdf0c1082a0 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.560+0000 7fdf1247d700 1 -- 172.21.15.31:0/202623075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf0c1a7700 con 0x7fdf0c1082a0 2024-04-25T01:53:01.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.561+0000 7fdf1247d700 1 -- 172.21.15.31:0/202623075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdf0c062940 con 0x7fdf0c1082a0 2024-04-25T01:53:01.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.562+0000 7fdf097fa700 1 -- 172.21.15.31:0/202623075 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fdf0401c070 con 0x7fdf0c1082a0 2024-04-25T01:53:01.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.563+0000 7fdf097fa700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdef40a2cb0 0x7fdef40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:01.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.563+0000 7fdf0bfff700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdef40a2cb0 0x7fdef40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:01.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.563+0000 7fdf097fa700 1 -- 172.21.15.31:0/202623075 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fdf040cee30 con 0x7fdf0c1082a0 2024-04-25T01:53:01.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.563+0000 7fdf097fa700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088] conn(0x7fdef40ac490 0x7fdef40ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:01.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.563+0000 7fdf097fa700 1 -- 172.21.15.31:0/202623075 --> [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf0401ce50 con 0x7fdef40ac490 2024-04-25T01:53:01.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.563+0000 7fdf097fa700 1 -- 172.21.15.31:0/202623075 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fdf040cf310 con 0x7fdf0c1082a0 2024-04-25T01:53:01.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.563+0000 7fdf0b7fe700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088] conn(0x7fdef40ac490 0x7fdef40ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:01.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.563+0000 7fdf0bfff700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdef40a2cb0 0x7fdef40a5170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdefc00b5c0 tx=0x7fdefc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:01.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.563+0000 7fdf0b7fe700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088] conn(0x7fdef40ac490 0x7fdef40ae8b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:01.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.564+0000 7fdf097fa700 1 -- 172.21.15.31:0/202623075 <== osd.9 v2:172.21.15.31:6826/238962088 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdf0401ce50 con 0x7fdef40ac490 2024-04-25T01:53:01.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.580+0000 7fdf1247d700 1 -- 172.21.15.31:0/202623075 --> [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdf0c062d40 con 0x7fdef40ac490 2024-04-25T01:53:01.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf097fa700 1 -- 172.21.15.31:0/202623075 <== osd.9 v2:172.21.15.31:6826/238962088 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdf0c062d40 con 0x7fdef40ac490 2024-04-25T01:53:01.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf1247d700 1 -- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088] conn(0x7fdef40ac490 msgr2=0x7fdef40ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:01.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf1247d700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088] conn(0x7fdef40ac490 0x7fdef40ae8b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:01.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf1247d700 1 -- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdef40a2cb0 msgr2=0x7fdef40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:01.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf1247d700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdef40a2cb0 0x7fdef40a5170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdefc00b5c0 tx=0x7fdefc005960 comp rx=0 tx=0).stop 2024-04-25T01:53:01.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf1247d700 1 -- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c1082a0 msgr2=0x7fdf0c101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:01.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf1247d700 1 --2- 172.21.15.31:0/202623075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf0c1082a0 0x7fdf0c101c70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400b970 tx=0x7fdf0400bc80 comp rx=0 tx=0).stop 2024-04-25T01:53:01.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf10a1a700 1 -- 172.21.15.31:0/202623075 reap_dead start 2024-04-25T01:53:01.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf1247d700 1 -- 172.21.15.31:0/202623075 shutdown_connections 2024-04-25T01:53:01.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf1247d700 1 -- 172.21.15.31:0/202623075 >> 172.21.15.31:0/202623075 conn(0x7fdf0c0780a0 msgr2=0x7fdf0c0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:01.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf1247d700 1 -- 172.21.15.31:0/202623075 shutdown_connections 2024-04-25T01:53:01.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:01.581+0000 7fdf1247d700 1 -- 172.21.15.31:0/202623075 wait complete. 2024-04-25T01:53:03.168 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:02 smithi031 ceph-mon[29204]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-25T01:53:03.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:02 smithi134 ceph-mon[33098]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-25T01:53:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.284+0000 7fb98ca9b700 1 -- 172.21.15.31:0/3408676966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988101770 msgr2=0x7fb988101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.284+0000 7fb98ca9b700 1 --2- 172.21.15.31:0/3408676966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988101770 0x7fb988101b50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb97800b0d0 tx=0x7fb97800b3e0 comp rx=0 tx=0).stop 2024-04-25T01:53:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.284+0000 7fb98ca9b700 1 -- 172.21.15.31:0/3408676966 shutdown_connections 2024-04-25T01:53:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.284+0000 7fb98ca9b700 1 --2- 172.21.15.31:0/3408676966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9881064c0 0x7fb98810e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.284+0000 7fb98ca9b700 1 --2- 172.21.15.31:0/3408676966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988101770 0x7fb988101b50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.284+0000 7fb98ca9b700 1 --2- 172.21.15.31:0/3408676966 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb988102120 0x7fb988105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.284+0000 7fb98ca9b700 1 -- 172.21.15.31:0/3408676966 >> 172.21.15.31:0/3408676966 conn(0x7fb9880780e0 msgr2=0x7fb9880784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.284+0000 7fb98ca9b700 1 -- 172.21.15.31:0/3408676966 shutdown_connections 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.284+0000 7fb98ca9b700 1 -- 172.21.15.31:0/3408676966 wait complete. 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb98ca9b700 1 Processor -- start 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb98ca9b700 1 -- start start 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb98ca9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb988101770 0x7fb98819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb98ca9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988102120 0x7fb98819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb98ca9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9881064c0 0x7fb9881987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb98ca9b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb988111380 con 0x7fb988102120 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb98ca9b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb988111200 con 0x7fb988101770 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb98ca9b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb988111500 con 0x7fb9881064c0 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb985d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988102120 0x7fb98819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb985d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988102120 0x7fb98819eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47148/0 (socket says 172.21.15.31:47148) 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb985d9b700 1 -- 172.21.15.31:0/1807235861 learned_addr learned my addr 172.21.15.31:0/1807235861 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.285+0000 7fb98659c700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb988101770 0x7fb98819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb986d9d700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9881064c0 0x7fb9881987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb985d9b700 1 -- 172.21.15.31:0/1807235861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9881064c0 msgr2=0x7fb9881987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb985d9b700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9881064c0 0x7fb9881987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb985d9b700 1 -- 172.21.15.31:0/1807235861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb988101770 msgr2=0x7fb98819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb985d9b700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb988101770 0x7fb98819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb985d9b700 1 -- 172.21.15.31:0/1807235861 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb988198fc0 con 0x7fb988102120 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb986d9d700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9881064c0 0x7fb9881987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb985d9b700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988102120 0x7fb98819eba0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb97000b4d0 tx=0x7fb97000b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb97f7fe700 1 -- 172.21.15.31:0/1807235861 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb970015020 con 0x7fb988102120 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb98ca9b700 1 -- 172.21.15.31:0/1807235861 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9881992b0 con 0x7fb988102120 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb97f7fe700 1 -- 172.21.15.31:0/1807235861 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb970011ce0 con 0x7fb988102120 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb97f7fe700 1 -- 172.21.15.31:0/1807235861 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb970010cd0 con 0x7fb988102120 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.286+0000 7fb98ca9b700 1 -- 172.21.15.31:0/1807235861 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9881a54c0 con 0x7fb988102120 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.287+0000 7fb98ca9b700 1 -- 172.21.15.31:0/1807235861 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb9880629c0 con 0x7fb988102120 2024-04-25T01:53:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.288+0000 7fb97f7fe700 1 -- 172.21.15.31:0/1807235861 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb970018020 con 0x7fb988102120 2024-04-25T01:53:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.289+0000 7fb97f7fe700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb9740a2cb0 0x7fb9740a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.289+0000 7fb98659c700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb9740a2cb0 0x7fb9740a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.289+0000 7fb97f7fe700 1 -- 172.21.15.31:0/1807235861 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fb9700ced20 con 0x7fb988102120 2024-04-25T01:53:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.289+0000 7fb97f7fe700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802] conn(0x7fb9740ac490 0x7fb9740ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.289+0000 7fb97f7fe700 1 -- 172.21.15.31:0/1807235861 --> [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb970018ec0 con 0x7fb9740ac490 2024-04-25T01:53:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.289+0000 7fb97f7fe700 1 -- 172.21.15.31:0/1807235861 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fb9700cf200 con 0x7fb988102120 2024-04-25T01:53:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.290+0000 7fb98659c700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb9740a2cb0 0x7fb9740a5170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb978000f80 tx=0x7fb97800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.290+0000 7fb986d9d700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802] conn(0x7fb9740ac490 0x7fb9740ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.290+0000 7fb986d9d700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802] conn(0x7fb9740ac490 0x7fb9740ae8b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.291+0000 7fb97f7fe700 1 -- 172.21.15.31:0/1807235861 <== osd.2 v2:172.21.15.134:6800/414932802 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb970018ec0 con 0x7fb9740ac490 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.308+0000 7fb98ca9b700 1 -- 172.21.15.31:0/1807235861 --> [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb988062dc0 con 0x7fb9740ac490 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb97f7fe700 1 -- 172.21.15.31:0/1807235861 <== osd.2 v2:172.21.15.134:6800/414932802 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb988062dc0 con 0x7fb9740ac490 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb98ca9b700 1 -- 172.21.15.31:0/1807235861 >> [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802] conn(0x7fb9740ac490 msgr2=0x7fb9740ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb98ca9b700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802] conn(0x7fb9740ac490 0x7fb9740ae8b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb98ca9b700 1 -- 172.21.15.31:0/1807235861 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb9740a2cb0 msgr2=0x7fb9740a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb98ca9b700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb9740a2cb0 0x7fb9740a5170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb978000f80 tx=0x7fb97800b040 comp rx=0 tx=0).stop 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb98ca9b700 1 -- 172.21.15.31:0/1807235861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988102120 msgr2=0x7fb98819eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb98ca9b700 1 --2- 172.21.15.31:0/1807235861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb988102120 0x7fb98819eba0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb97000b4d0 tx=0x7fb97000b7e0 comp rx=0 tx=0).stop 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb986d9d700 1 -- 172.21.15.31:0/1807235861 reap_dead start 2024-04-25T01:53:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb98ca9b700 1 -- 172.21.15.31:0/1807235861 shutdown_connections 2024-04-25T01:53:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb98ca9b700 1 -- 172.21.15.31:0/1807235861 >> 172.21.15.31:0/1807235861 conn(0x7fb9880780e0 msgr2=0x7fb98806ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb98ca9b700 1 -- 172.21.15.31:0/1807235861 shutdown_connections 2024-04-25T01:53:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:03.309+0000 7fb98ca9b700 1 -- 172.21.15.31:0/1807235861 wait complete. 2024-04-25T01:53:03.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:02 smithi040 ceph-mon[32418]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-25T01:53:04.360 INFO:teuthology.orchestra.run.smithi031.stdout:296352743453 2024-04-25T01:53:04.361 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T01:53:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.779+0000 7fe6511c1700 1 -- 172.21.15.31:0/2303455816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe64c106490 msgr2=0x7fe64c10e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.779+0000 7fe6511c1700 1 --2- 172.21.15.31:0/2303455816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe64c106490 0x7fe64c10e060 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe64400b0d0 tx=0x7fe64400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.779+0000 7fe6511c1700 1 -- 172.21.15.31:0/2303455816 shutdown_connections 2024-04-25T01:53:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.779+0000 7fe6511c1700 1 --2- 172.21.15.31:0/2303455816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe64c106490 0x7fe64c10e060 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.779+0000 7fe6511c1700 1 --2- 172.21.15.31:0/2303455816 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe64c1013b0 0x7fe64c101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.779+0000 7fe6511c1700 1 --2- 172.21.15.31:0/2303455816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe64c101d60 0x7fe64c105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.780+0000 7fe6511c1700 1 -- 172.21.15.31:0/2303455816 >> 172.21.15.31:0/2303455816 conn(0x7fe64c0781d0 msgr2=0x7fe64c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.780+0000 7fe6511c1700 1 -- 172.21.15.31:0/2303455816 shutdown_connections 2024-04-25T01:53:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.780+0000 7fe6511c1700 1 -- 172.21.15.31:0/2303455816 wait complete. 2024-04-25T01:53:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.780+0000 7fe6511c1700 1 Processor -- start 2024-04-25T01:53:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe6511c1700 1 -- start start 2024-04-25T01:53:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe6511c1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe64c1013b0 0x7fe64c199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe6511c1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe64c101d60 0x7fe64c199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe6511c1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe64c106490 0x7fe64c1a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe6511c1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe64c111330 con 0x7fe64c106490 2024-04-25T01:53:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe6511c1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe64c1111b0 con 0x7fe64c101d60 2024-04-25T01:53:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe6511c1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe64c1114b0 con 0x7fe64c1013b0 2024-04-25T01:53:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64b59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe64c106490 0x7fe64c1a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64b59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe64c106490 0x7fe64c1a4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47166/0 (socket says 172.21.15.31:47166) 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64b59e700 1 -- 172.21.15.31:0/847634478 learned_addr learned my addr 172.21.15.31:0/847634478 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64ad9d700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe64c1013b0 0x7fe64c199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64a59c700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe64c101d60 0x7fe64c199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64b59e700 1 -- 172.21.15.31:0/847634478 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe64c1013b0 msgr2=0x7fe64c199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64b59e700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe64c1013b0 0x7fe64c199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64b59e700 1 -- 172.21.15.31:0/847634478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe64c101d60 msgr2=0x7fe64c199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64b59e700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe64c101d60 0x7fe64c199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64b59e700 1 -- 172.21.15.31:0/847634478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c197480 con 0x7fe64c106490 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64a59c700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe64c101d60 0x7fe64c199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.781+0000 7fe64b59e700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe64c106490 0x7fe64c1a4710 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe644009fd0 tx=0x7fe644007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.782+0000 7fe633fff700 1 -- 172.21.15.31:0/847634478 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64401f070 con 0x7fe64c106490 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.782+0000 7fe6511c1700 1 -- 172.21.15.31:0/847634478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64c197710 con 0x7fe64c106490 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.782+0000 7fe633fff700 1 -- 172.21.15.31:0/847634478 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe644003c00 con 0x7fe64c106490 2024-04-25T01:53:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.782+0000 7fe633fff700 1 -- 172.21.15.31:0/847634478 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64401b770 con 0x7fe64c106490 2024-04-25T01:53:04.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.782+0000 7fe6511c1700 1 -- 172.21.15.31:0/847634478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe64c197c40 con 0x7fe64c106490 2024-04-25T01:53:04.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.783+0000 7fe6511c1700 1 -- 172.21.15.31:0/847634478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe64c062940 con 0x7fe64c106490 2024-04-25T01:53:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.784+0000 7fe633fff700 1 -- 172.21.15.31:0/847634478 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe644022070 con 0x7fe64c106490 2024-04-25T01:53:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.784+0000 7fe633fff700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fe6340a2cb0 0x7fe6340a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.785+0000 7fe64ad9d700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fe6340a2cb0 0x7fe6340a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.785+0000 7fe633fff700 1 -- 172.21.15.31:0/847634478 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fe64400f020 con 0x7fe64c106490 2024-04-25T01:53:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.785+0000 7fe633fff700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774] conn(0x7fe6340ac490 0x7fe6340ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.785+0000 7fe633fff700 1 -- 172.21.15.31:0/847634478 --> [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6440020b0 con 0x7fe6340ac490 2024-04-25T01:53:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.785+0000 7fe633fff700 1 -- 172.21.15.31:0/847634478 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fe6440d0c60 con 0x7fe64c106490 2024-04-25T01:53:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.785+0000 7fe64ad9d700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fe6340a2cb0 0x7fe6340a5170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00ac50 tx=0x7fe63c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.785+0000 7fe64a59c700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774] conn(0x7fe6340ac490 0x7fe6340ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.785+0000 7fe64a59c700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774] conn(0x7fe6340ac490 0x7fe6340ae8b0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.786+0000 7fe633fff700 1 -- 172.21.15.31:0/847634478 <== osd.11 v2:172.21.15.134:6824/897924774 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe6440020b0 con 0x7fe6340ac490 2024-04-25T01:53:04.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.806+0000 7fe6511c1700 1 -- 172.21.15.31:0/847634478 --> [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe64c062d40 con 0x7fe6340ac490 2024-04-25T01:53:04.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.806+0000 7fe633fff700 1 -- 172.21.15.31:0/847634478 <== osd.11 v2:172.21.15.134:6824/897924774 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe64c062d40 con 0x7fe6340ac490 2024-04-25T01:53:04.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.807+0000 7fe6511c1700 1 -- 172.21.15.31:0/847634478 >> [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774] conn(0x7fe6340ac490 msgr2=0x7fe6340ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:04.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.807+0000 7fe6511c1700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774] conn(0x7fe6340ac490 0x7fe6340ae8b0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.807+0000 7fe6511c1700 1 -- 172.21.15.31:0/847634478 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fe6340a2cb0 msgr2=0x7fe6340a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:04.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.807+0000 7fe6511c1700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fe6340a2cb0 0x7fe6340a5170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00ac50 tx=0x7fe63c00a380 comp rx=0 tx=0).stop 2024-04-25T01:53:04.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.807+0000 7fe6511c1700 1 -- 172.21.15.31:0/847634478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe64c106490 msgr2=0x7fe64c1a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:04.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.807+0000 7fe6511c1700 1 --2- 172.21.15.31:0/847634478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe64c106490 0x7fe64c1a4710 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe644009fd0 tx=0x7fe644007a20 comp rx=0 tx=0).stop 2024-04-25T01:53:04.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.807+0000 7fe64b59e700 1 -- 172.21.15.31:0/847634478 reap_dead start 2024-04-25T01:53:04.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.807+0000 7fe6511c1700 1 -- 172.21.15.31:0/847634478 shutdown_connections 2024-04-25T01:53:04.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.807+0000 7fe6511c1700 1 -- 172.21.15.31:0/847634478 >> 172.21.15.31:0/847634478 conn(0x7fe64c0781d0 msgr2=0x7fe64c0ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:04.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.807+0000 7fe6511c1700 1 -- 172.21.15.31:0/847634478 shutdown_connections 2024-04-25T01:53:04.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:04.807+0000 7fe6511c1700 1 -- 172.21.15.31:0/847634478 wait complete. 2024-04-25T01:53:04.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:04 smithi031 ceph-mon[29204]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:53:05.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:04 smithi134 ceph-mon[33098]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:53:05.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:04 smithi040 ceph-mon[32418]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:53:06.715 INFO:teuthology.orchestra.run.smithi031.stdout:352187318292 2024-04-25T01:53:06.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T01:53:06.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.178+0000 7fae07558700 1 -- 172.21.15.31:0/264337740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae00107570 msgr2=0x7fae0010f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.178+0000 7fae07558700 1 --2- 172.21.15.31:0/264337740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae00107570 0x7fae0010f140 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fadf000b0a0 tx=0x7fadf000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.178+0000 7fae07558700 1 -- 172.21.15.31:0/264337740 shutdown_connections 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.178+0000 7fae07558700 1 --2- 172.21.15.31:0/264337740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae00107570 0x7fae0010f140 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.178+0000 7fae07558700 1 --2- 172.21.15.31:0/264337740 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae001024e0 0x7fae001028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.178+0000 7fae07558700 1 --2- 172.21.15.31:0/264337740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae00102e90 0x7fae00106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.178+0000 7fae07558700 1 -- 172.21.15.31:0/264337740 >> 172.21.15.31:0/264337740 conn(0x7fae00079390 msgr2=0x7fae000797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.178+0000 7fae07558700 1 -- 172.21.15.31:0/264337740 shutdown_connections 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.178+0000 7fae07558700 1 -- 172.21.15.31:0/264337740 wait complete. 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae07558700 1 Processor -- start 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae07558700 1 -- start start 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae07558700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae001024e0 0x7fae0019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae07558700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae00102e90 0x7fae0019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:06.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae07558700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae00107570 0x7fae00198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae07558700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fae00112400 con 0x7fae00107570 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae07558700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fae00112280 con 0x7fae00102e90 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae07558700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fae00112580 con 0x7fae001024e0 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae05af5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae00107570 0x7fae00198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae05af5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae00107570 0x7fae00198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47192/0 (socket says 172.21.15.31:47192) 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae05af5700 1 -- 172.21.15.31:0/3103976311 learned_addr learned my addr 172.21.15.31:0/3103976311 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae052f4700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae001024e0 0x7fae0019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae04af3700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae00102e90 0x7fae0019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae05af5700 1 -- 172.21.15.31:0/3103976311 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae001024e0 msgr2=0x7fae0019e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae05af5700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae001024e0 0x7fae0019e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae05af5700 1 -- 172.21.15.31:0/3103976311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae00102e90 msgr2=0x7fae0019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae05af5700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae00102e90 0x7fae0019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae05af5700 1 -- 172.21.15.31:0/3103976311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae00198f20 con 0x7fae00107570 2024-04-25T01:53:06.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.179+0000 7fae04af3700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae00102e90 0x7fae0019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.180+0000 7fae05af5700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae00107570 0x7fae00198780 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fadf000b830 tx=0x7fadf0004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.180+0000 7fadfe7fc700 1 -- 172.21.15.31:0/3103976311 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0021070 con 0x7fae00107570 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.180+0000 7fadfe7fc700 1 -- 172.21.15.31:0/3103976311 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadf0007e30 con 0x7fae00107570 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.180+0000 7fadfe7fc700 1 -- 172.21.15.31:0/3103976311 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf00047d0 con 0x7fae00107570 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.180+0000 7fae07558700 1 -- 172.21.15.31:0/3103976311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae001991b0 con 0x7fae00107570 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.180+0000 7fae07558700 1 -- 172.21.15.31:0/3103976311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae001a5440 con 0x7fae00107570 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.181+0000 7fae07558700 1 -- 172.21.15.31:0/3103976311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fae00062940 con 0x7fae00107570 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.182+0000 7fadfe7fc700 1 -- 172.21.15.31:0/3103976311 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fadf0007980 con 0x7fae00107570 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.183+0000 7fadfe7fc700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fade80a2cb0 0x7fade80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.183+0000 7fae052f4700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fade80a2cb0 0x7fade80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.183+0000 7fadfe7fc700 1 -- 172.21.15.31:0/3103976311 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fadf000f020 con 0x7fae00107570 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.183+0000 7fadfe7fc700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767] conn(0x7fade80ac490 0x7fade80ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.183+0000 7fadfe7fc700 1 -- 172.21.15.31:0/3103976311 --> [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fadf000aea0 con 0x7fade80ac490 2024-04-25T01:53:06.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.183+0000 7fadfe7fc700 1 -- 172.21.15.31:0/3103976311 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fadf00d0c30 con 0x7fae00107570 2024-04-25T01:53:06.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.183+0000 7fae052f4700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fade80a2cb0 0x7fade80a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fadf8004630 tx=0x7fadf800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:06.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.183+0000 7fae04af3700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767] conn(0x7fade80ac490 0x7fade80ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:06.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.184+0000 7fae04af3700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767] conn(0x7fade80ac490 0x7fade80ae8b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:06.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.184+0000 7fadfe7fc700 1 -- 172.21.15.31:0/3103976311 <== osd.5 v2:172.21.15.134:6808/1004304767 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fadf000aea0 con 0x7fade80ac490 2024-04-25T01:53:06.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.201+0000 7fae07558700 1 -- 172.21.15.31:0/3103976311 --> [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fae00062dc0 con 0x7fade80ac490 2024-04-25T01:53:06.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.201+0000 7fadfe7fc700 1 -- 172.21.15.31:0/3103976311 <== osd.5 v2:172.21.15.134:6808/1004304767 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fae00062dc0 con 0x7fade80ac490 2024-04-25T01:53:06.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.202+0000 7fae07558700 1 -- 172.21.15.31:0/3103976311 >> [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767] conn(0x7fade80ac490 msgr2=0x7fade80ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:06.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.202+0000 7fae07558700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767] conn(0x7fade80ac490 0x7fade80ae8b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:06.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.202+0000 7fae07558700 1 -- 172.21.15.31:0/3103976311 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fade80a2cb0 msgr2=0x7fade80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:06.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.202+0000 7fae07558700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fade80a2cb0 0x7fade80a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fadf8004630 tx=0x7fadf800b3c0 comp rx=0 tx=0).stop 2024-04-25T01:53:06.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.202+0000 7fae07558700 1 -- 172.21.15.31:0/3103976311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae00107570 msgr2=0x7fae00198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:06.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.202+0000 7fae07558700 1 --2- 172.21.15.31:0/3103976311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fae00107570 0x7fae00198780 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fadf000b830 tx=0x7fadf0004ce0 comp rx=0 tx=0).stop 2024-04-25T01:53:06.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.202+0000 7fae05af5700 1 -- 172.21.15.31:0/3103976311 reap_dead start 2024-04-25T01:53:06.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.202+0000 7fae07558700 1 -- 172.21.15.31:0/3103976311 shutdown_connections 2024-04-25T01:53:06.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.202+0000 7fae07558700 1 -- 172.21.15.31:0/3103976311 >> 172.21.15.31:0/3103976311 conn(0x7fae00079390 msgr2=0x7fae001086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:06.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.202+0000 7fae07558700 1 -- 172.21.15.31:0/3103976311 shutdown_connections 2024-04-25T01:53:06.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:06.202+0000 7fae07558700 1 -- 172.21.15.31:0/3103976311 wait complete. 2024-04-25T01:53:06.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:06 smithi031 ceph-mon[29204]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:53:07.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:06 smithi134 ceph-mon[33098]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:53:07.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:06 smithi040 ceph-mon[32418]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.598+0000 7f6dd2f56700 1 -- 172.21.15.31:0/4085861738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc101ae0 msgr2=0x7f6dcc10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.598+0000 7f6dd2f56700 1 --2- 172.21.15.31:0/4085861738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc101ae0 0x7f6dcc10dfd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6dc400b0a0 tx=0x7f6dc400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.598+0000 7f6dd2f56700 1 -- 172.21.15.31:0/4085861738 shutdown_connections 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.598+0000 7f6dd2f56700 1 --2- 172.21.15.31:0/4085861738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc101ae0 0x7f6dcc10dfd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.598+0000 7f6dd2f56700 1 --2- 172.21.15.31:0/4085861738 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6dcc107190 0x7f6dcc107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.598+0000 7f6dd2f56700 1 --2- 172.21.15.31:0/4085861738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dcc101120 0x7f6dcc1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.598+0000 7f6dd2f56700 1 -- 172.21.15.31:0/4085861738 >> 172.21.15.31:0/4085861738 conn(0x7f6dcc0780a0 msgr2=0x7f6dcc0784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.598+0000 7f6dd2f56700 1 -- 172.21.15.31:0/4085861738 shutdown_connections 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.598+0000 7f6dd2f56700 1 -- 172.21.15.31:0/4085861738 wait complete. 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.598+0000 7f6dd2f56700 1 Processor -- start 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd2f56700 1 -- start start 2024-04-25T01:53:07.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd2f56700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6dcc101120 0x7f6dcc19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd2f56700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dcc101ae0 0x7f6dcc19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd2f56700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc107190 0x7f6dcc198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd2f56700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6dcc1123d0 con 0x7f6dcc107190 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd2f56700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6dcc112250 con 0x7f6dcc101ae0 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd2f56700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6dcc112550 con 0x7f6dcc101120 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd14f3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc107190 0x7f6dcc198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd14f3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc107190 0x7f6dcc198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47200/0 (socket says 172.21.15.31:47200) 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd14f3700 1 -- 172.21.15.31:0/3166760550 learned_addr learned my addr 172.21.15.31:0/3166760550 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd0cf2700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6dcc101120 0x7f6dcc19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dcc101ae0 0x7f6dcc19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd14f3700 1 -- 172.21.15.31:0/3166760550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6dcc101120 msgr2=0x7f6dcc19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd14f3700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6dcc101120 0x7f6dcc19e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd14f3700 1 -- 172.21.15.31:0/3166760550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dcc101ae0 msgr2=0x7f6dcc19ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd14f3700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dcc101ae0 0x7f6dcc19ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd14f3700 1 -- 172.21.15.31:0/3166760550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dcc199030 con 0x7f6dcc107190 2024-04-25T01:53:07.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.599+0000 7f6dd14f3700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc107190 0x7f6dcc198750 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6dc40062a0 tx=0x7f6dc4004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:07.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.600+0000 7f6dc9ffb700 1 -- 172.21.15.31:0/3166760550 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4021070 con 0x7f6dcc107190 2024-04-25T01:53:07.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.600+0000 7f6dc9ffb700 1 -- 172.21.15.31:0/3166760550 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dc4007e30 con 0x7f6dcc107190 2024-04-25T01:53:07.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.600+0000 7f6dd2f56700 1 -- 172.21.15.31:0/3166760550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dcc1992c0 con 0x7f6dcc107190 2024-04-25T01:53:07.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.600+0000 7f6dc9ffb700 1 -- 172.21.15.31:0/3166760550 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4004780 con 0x7f6dcc107190 2024-04-25T01:53:07.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.600+0000 7f6dd2f56700 1 -- 172.21.15.31:0/3166760550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dcc1a5430 con 0x7f6dcc107190 2024-04-25T01:53:07.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.601+0000 7f6dd2f56700 1 -- 172.21.15.31:0/3166760550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6dcc062900 con 0x7f6dcc107190 2024-04-25T01:53:07.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.602+0000 7f6dc9ffb700 1 -- 172.21.15.31:0/3166760550 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6dc4007980 con 0x7f6dcc107190 2024-04-25T01:53:07.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.603+0000 7f6dc9ffb700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6dbc0a2d30 0x7f6dbc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:07.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.603+0000 7f6dd0cf2700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6dbc0a2d30 0x7f6dbc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:07.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.603+0000 7f6dc9ffb700 1 -- 172.21.15.31:0/3166760550 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f6dc400f020 con 0x7f6dcc107190 2024-04-25T01:53:07.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.603+0000 7f6dc9ffb700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732] conn(0x7f6dbc0ac510 0x7f6dbc0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:07.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.603+0000 7f6dc9ffb700 1 -- 172.21.15.31:0/3166760550 --> [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6dbc0aefe0 con 0x7f6dbc0ac510 2024-04-25T01:53:07.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.603+0000 7f6dc9ffb700 1 -- 172.21.15.31:0/3166760550 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f6dc40d7050 con 0x7f6dcc107190 2024-04-25T01:53:07.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.603+0000 7f6dd0cf2700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6dbc0a2d30 0x7f6dbc0a51f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0005950 tx=0x7f6dc000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:07.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.603+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732] conn(0x7f6dbc0ac510 0x7f6dbc0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:07.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.604+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732] conn(0x7f6dbc0ac510 0x7f6dbc0ae930 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:07.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.605+0000 7f6dc9ffb700 1 -- 172.21.15.31:0/3166760550 <== osd.7 v2:172.21.15.40:6816/1477675732 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6dbc0aefe0 con 0x7f6dbc0ac510 2024-04-25T01:53:07.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.621+0000 7f6dd2f56700 1 -- 172.21.15.31:0/3166760550 --> [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6dcc062d00 con 0x7f6dbc0ac510 2024-04-25T01:53:07.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.622+0000 7f6dc9ffb700 1 -- 172.21.15.31:0/3166760550 <== osd.7 v2:172.21.15.40:6816/1477675732 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6dcc062d00 con 0x7f6dbc0ac510 2024-04-25T01:53:07.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.622+0000 7f6dd2f56700 1 -- 172.21.15.31:0/3166760550 >> [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732] conn(0x7f6dbc0ac510 msgr2=0x7f6dbc0ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.622+0000 7f6dd2f56700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732] conn(0x7f6dbc0ac510 0x7f6dbc0ae930 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.622+0000 7f6dd2f56700 1 -- 172.21.15.31:0/3166760550 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6dbc0a2d30 msgr2=0x7f6dbc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.622+0000 7f6dd2f56700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6dbc0a2d30 0x7f6dbc0a51f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0005950 tx=0x7f6dc000b410 comp rx=0 tx=0).stop 2024-04-25T01:53:07.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.622+0000 7f6dd2f56700 1 -- 172.21.15.31:0/3166760550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc107190 msgr2=0x7f6dcc198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.622+0000 7f6dd2f56700 1 --2- 172.21.15.31:0/3166760550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc107190 0x7f6dcc198750 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6dc40062a0 tx=0x7f6dc4004c80 comp rx=0 tx=0).stop 2024-04-25T01:53:07.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.623+0000 7f6dd14f3700 1 -- 172.21.15.31:0/3166760550 reap_dead start 2024-04-25T01:53:07.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.623+0000 7f6dd2f56700 1 -- 172.21.15.31:0/3166760550 shutdown_connections 2024-04-25T01:53:07.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.623+0000 7f6dd2f56700 1 -- 172.21.15.31:0/3166760550 >> 172.21.15.31:0/3166760550 conn(0x7f6dcc0780a0 msgr2=0x7f6dcc0ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:07.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.623+0000 7f6dd2f56700 1 -- 172.21.15.31:0/3166760550 shutdown_connections 2024-04-25T01:53:07.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.623+0000 7f6dd2f56700 1 -- 172.21.15.31:0/3166760550 wait complete. 2024-04-25T01:53:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.852+0000 7f9ccee3e700 1 -- 172.21.15.31:0/3390079616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc8101ae0 msgr2=0x7f9cc810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.852+0000 7f9ccee3e700 1 --2- 172.21.15.31:0/3390079616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc8101ae0 0x7f9cc810dfd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000b0a0 tx=0x7f9cc000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.852+0000 7f9ccee3e700 1 -- 172.21.15.31:0/3390079616 shutdown_connections 2024-04-25T01:53:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.852+0000 7f9ccee3e700 1 --2- 172.21.15.31:0/3390079616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc8101ae0 0x7f9cc810dfd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.852+0000 7f9ccee3e700 1 --2- 172.21.15.31:0/3390079616 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cc8107190 0x7f9cc8107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.852+0000 7f9ccee3e700 1 --2- 172.21.15.31:0/3390079616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101120 0x7f9cc81015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.852+0000 7f9ccee3e700 1 -- 172.21.15.31:0/3390079616 >> 172.21.15.31:0/3390079616 conn(0x7f9cc80780a0 msgr2=0x7f9cc80784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.852+0000 7f9ccee3e700 1 -- 172.21.15.31:0/3390079616 shutdown_connections 2024-04-25T01:53:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.852+0000 7f9ccee3e700 1 -- 172.21.15.31:0/3390079616 wait complete. 2024-04-25T01:53:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccee3e700 1 Processor -- start 2024-04-25T01:53:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccee3e700 1 -- start start 2024-04-25T01:53:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccee3e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101120 0x7f9cc81a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccee3e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cc8101ae0 0x7f9cc81a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccee3e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc8107190 0x7f9cc819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccee3e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc81112d0 con 0x7f9cc8107190 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccee3e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc8111150 con 0x7f9cc8101120 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccee3e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc8111450 con 0x7f9cc8101ae0 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccd3db700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc8107190 0x7f9cc819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccd3db700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc8107190 0x7f9cc819a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47226/0 (socket says 172.21.15.31:47226) 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccd3db700 1 -- 172.21.15.31:0/2583070590 learned_addr learned my addr 172.21.15.31:0/2583070590 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9cccbda700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101120 0x7f9cc81a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9cc7fff700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cc8101ae0 0x7f9cc81a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccd3db700 1 -- 172.21.15.31:0/2583070590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cc8101ae0 msgr2=0x7f9cc81a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccd3db700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cc8101ae0 0x7f9cc81a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccd3db700 1 -- 172.21.15.31:0/2583070590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101120 msgr2=0x7f9cc81a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccd3db700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101120 0x7f9cc81a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9ccd3db700 1 -- 172.21.15.31:0/2583070590 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc819b020 con 0x7f9cc8107190 2024-04-25T01:53:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.853+0000 7f9cc7fff700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cc8101ae0 0x7f9cc81a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:07.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.854+0000 7f9ccd3db700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc8107190 0x7f9cc819a8c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00062a0 tx=0x7f9cc0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:07.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.854+0000 7f9cc5ffb700 1 -- 172.21.15.31:0/2583070590 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc0021070 con 0x7f9cc8107190 2024-04-25T01:53:07.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.854+0000 7f9cc5ffb700 1 -- 172.21.15.31:0/2583070590 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cc0007e30 con 0x7f9cc8107190 2024-04-25T01:53:07.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.854+0000 7f9ccee3e700 1 -- 172.21.15.31:0/2583070590 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc819b2b0 con 0x7f9cc8107190 2024-04-25T01:53:07.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.854+0000 7f9cc5ffb700 1 -- 172.21.15.31:0/2583070590 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc0004780 con 0x7f9cc8107190 2024-04-25T01:53:07.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.854+0000 7f9ccee3e700 1 -- 172.21.15.31:0/2583070590 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc81a75d0 con 0x7f9cc8107190 2024-04-25T01:53:07.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.855+0000 7f9ccee3e700 1 -- 172.21.15.31:0/2583070590 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9cc810dca0 con 0x7f9cc8107190 2024-04-25T01:53:07.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.856+0000 7f9cc5ffb700 1 -- 172.21.15.31:0/2583070590 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9cc0007980 con 0x7f9cc8107190 2024-04-25T01:53:07.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.857+0000 7f9cc5ffb700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9cb80a2cb0 0x7f9cb80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:07.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.857+0000 7f9cccbda700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9cb80a2cb0 0x7f9cb80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:07.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.857+0000 7f9cc5ffb700 1 -- 172.21.15.31:0/2583070590 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f9cc00d03f0 con 0x7f9cc8107190 2024-04-25T01:53:07.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.857+0000 7f9cc5ffb700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093] conn(0x7f9cb80ac360 0x7f9cb80ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:07.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.857+0000 7f9cc5ffb700 1 -- 172.21.15.31:0/2583070590 --> [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9cc00020b0 con 0x7f9cb80ac360 2024-04-25T01:53:07.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.857+0000 7f9cc5ffb700 1 -- 172.21.15.31:0/2583070590 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f9cc00d08d0 con 0x7f9cc8107190 2024-04-25T01:53:07.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.857+0000 7f9cccbda700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9cb80a2cb0 0x7f9cb80a5170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9cc819ba00 tx=0x7f9cbc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:07.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.858+0000 7f9cc7fff700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093] conn(0x7f9cb80ac360 0x7f9cb80ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:07.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.858+0000 7f9cc7fff700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093] conn(0x7f9cb80ac360 0x7f9cb80ae780 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:07.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.859+0000 7f9cc5ffb700 1 -- 172.21.15.31:0/2583070590 <== osd.4 v2:172.21.15.40:6808/3237694093 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cc00020b0 con 0x7f9cb80ac360 2024-04-25T01:53:07.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.875+0000 7f9ccee3e700 1 -- 172.21.15.31:0/2583070590 --> [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9cc8102840 con 0x7f9cb80ac360 2024-04-25T01:53:07.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9cc5ffb700 1 -- 172.21.15.31:0/2583070590 <== osd.4 v2:172.21.15.40:6808/3237694093 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9cc8102840 con 0x7f9cb80ac360 2024-04-25T01:53:07.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9ccee3e700 1 -- 172.21.15.31:0/2583070590 >> [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093] conn(0x7f9cb80ac360 msgr2=0x7f9cb80ae780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9ccee3e700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093] conn(0x7f9cb80ac360 0x7f9cb80ae780 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:07.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9ccee3e700 1 -- 172.21.15.31:0/2583070590 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9cb80a2cb0 msgr2=0x7f9cb80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9ccee3e700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9cb80a2cb0 0x7f9cb80a5170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9cc819ba00 tx=0x7f9cbc009260 comp rx=0 tx=0).stop 2024-04-25T01:53:07.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9ccee3e700 1 -- 172.21.15.31:0/2583070590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc8107190 msgr2=0x7f9cc819a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:07.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9ccee3e700 1 --2- 172.21.15.31:0/2583070590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc8107190 0x7f9cc819a8c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00062a0 tx=0x7f9cc0004c80 comp rx=0 tx=0).stop 2024-04-25T01:53:07.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9ccd3db700 1 -- 172.21.15.31:0/2583070590 reap_dead start 2024-04-25T01:53:07.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9ccee3e700 1 -- 172.21.15.31:0/2583070590 shutdown_connections 2024-04-25T01:53:07.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9ccee3e700 1 -- 172.21.15.31:0/2583070590 >> 172.21.15.31:0/2583070590 conn(0x7f9cc80780a0 msgr2=0x7f9cc810d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:07.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9ccee3e700 1 -- 172.21.15.31:0/2583070590 shutdown_connections 2024-04-25T01:53:07.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:07.876+0000 7f9ccee3e700 1 -- 172.21.15.31:0/2583070590 wait complete. 2024-04-25T01:53:08.916 INFO:teuthology.orchestra.run.smithi031.stdout:223338299434 2024-04-25T01:53:08.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T01:53:09.142 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:08 smithi031 ceph-mon[29204]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:09.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:08 smithi134 ceph-mon[33098]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:09.211 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:08 smithi040 ceph-mon[32418]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:09.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.257+0000 7fb616bb1700 1 -- 172.21.15.31:0/2144387241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6101064c0 msgr2=0x7fb61010e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.257+0000 7fb616bb1700 1 --2- 172.21.15.31:0/2144387241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6101064c0 0x7fb61010e060 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb60800b0d0 tx=0x7fb60800b3e0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.257+0000 7fb616bb1700 1 -- 172.21.15.31:0/2144387241 shutdown_connections 2024-04-25T01:53:09.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.257+0000 7fb616bb1700 1 --2- 172.21.15.31:0/2144387241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6101064c0 0x7fb61010e060 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.257+0000 7fb616bb1700 1 --2- 172.21.15.31:0/2144387241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb610101770 0x7fb610101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.257+0000 7fb616bb1700 1 --2- 172.21.15.31:0/2144387241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb610102120 0x7fb610105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.257+0000 7fb616bb1700 1 -- 172.21.15.31:0/2144387241 >> 172.21.15.31:0/2144387241 conn(0x7fb6100780e0 msgr2=0x7fb6100784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:09.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.257+0000 7fb616bb1700 1 -- 172.21.15.31:0/2144387241 shutdown_connections 2024-04-25T01:53:09.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.257+0000 7fb616bb1700 1 -- 172.21.15.31:0/2144387241 wait complete. 2024-04-25T01:53:09.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb616bb1700 1 Processor -- start 2024-04-25T01:53:09.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb616bb1700 1 -- start start 2024-04-25T01:53:09.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb616bb1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb610101770 0x7fb610199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:09.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb616bb1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb610102120 0x7fb610199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:09.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb616bb1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6101064c0 0x7fb6101a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:09.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb61514e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6101064c0 0x7fb6101a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:09.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb60ffff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb610102120 0x7fb610199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:09.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb60ffff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb610102120 0x7fb610199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:45098/0 (socket says 172.21.15.31:45098) 2024-04-25T01:53:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb60ffff700 1 -- 172.21.15.31:0/3436194768 learned_addr learned my addr 172.21.15.31:0/3436194768 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb616bb1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb6101113a0 con 0x7fb6101064c0 2024-04-25T01:53:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb610111220 con 0x7fb610101770 2024-04-25T01:53:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb610111520 con 0x7fb610102120 2024-04-25T01:53:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.258+0000 7fb61494d700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb610101770 0x7fb610199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb60ffff700 1 -- 172.21.15.31:0/3436194768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb610101770 msgr2=0x7fb610199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb60ffff700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb610101770 0x7fb610199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb60ffff700 1 -- 172.21.15.31:0/3436194768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6101064c0 msgr2=0x7fb6101a4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb60ffff700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6101064c0 0x7fb6101a4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb60ffff700 1 -- 172.21.15.31:0/3436194768 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6101a4d40 con 0x7fb610102120 2024-04-25T01:53:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb61514e700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6101064c0 0x7fb6101a4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb60ffff700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb610102120 0x7fb610199cc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e940 tx=0x7fb5fc00ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb60dffb700 1 -- 172.21.15.31:0/3436194768 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc00c9e0 con 0x7fb610102120 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb60dffb700 1 -- 172.21.15.31:0/3436194768 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5fc00cb80 con 0x7fb610102120 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6101a4f70 con 0x7fb610102120 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb60dffb700 1 -- 172.21.15.31:0/3436194768 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc018680 con 0x7fb610102120 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.259+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6101a5420 con 0x7fb610102120 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.260+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb6100628c0 con 0x7fb610102120 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.261+0000 7fb60dffb700 1 -- 172.21.15.31:0/3436194768 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb5fc010430 con 0x7fb610102120 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.262+0000 7fb60dffb700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb6000a2cb0 0x7fb6000a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.262+0000 7fb61494d700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb6000a2cb0 0x7fb6000a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.262+0000 7fb60dffb700 1 -- 172.21.15.31:0/3436194768 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fb5fc0cf720 con 0x7fb610102120 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.262+0000 7fb60dffb700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048] conn(0x7fb6000ac490 0x7fb6000ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.262+0000 7fb60dffb700 1 -- 172.21.15.31:0/3436194768 --> [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5fc01bec0 con 0x7fb6000ac490 2024-04-25T01:53:09.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.262+0000 7fb60dffb700 1 -- 172.21.15.31:0/3436194768 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fb5fc0cfad0 con 0x7fb610102120 2024-04-25T01:53:09.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.263+0000 7fb61494d700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb6000a2cb0 0x7fb6000a5170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb604009c60 tx=0x7fb604009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:09.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.263+0000 7fb61514e700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048] conn(0x7fb6000ac490 0x7fb6000ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:09.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.263+0000 7fb61514e700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048] conn(0x7fb6000ac490 0x7fb6000ae8b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:09.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.264+0000 7fb60dffb700 1 -- 172.21.15.31:0/3436194768 <== osd.8 v2:172.21.15.134:6816/3295606048 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5fc01bec0 con 0x7fb6000ac490 2024-04-25T01:53:09.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.288+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 --> [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb610062d40 con 0x7fb6000ac490 2024-04-25T01:53:09.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.288+0000 7fb60dffb700 1 -- 172.21.15.31:0/3436194768 <== osd.8 v2:172.21.15.134:6816/3295606048 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb610062d40 con 0x7fb6000ac490 2024-04-25T01:53:09.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.288+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 >> [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048] conn(0x7fb6000ac490 msgr2=0x7fb6000ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.288+0000 7fb616bb1700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048] conn(0x7fb6000ac490 0x7fb6000ae8b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.289+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb6000a2cb0 msgr2=0x7fb6000a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.289+0000 7fb616bb1700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb6000a2cb0 0x7fb6000a5170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb604009c60 tx=0x7fb604009260 comp rx=0 tx=0).stop 2024-04-25T01:53:09.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.289+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb610102120 msgr2=0x7fb610199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.289+0000 7fb616bb1700 1 --2- 172.21.15.31:0/3436194768 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb610102120 0x7fb610199cc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e940 tx=0x7fb5fc00ec50 comp rx=0 tx=0).stop 2024-04-25T01:53:09.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.289+0000 7fb61514e700 1 -- 172.21.15.31:0/3436194768 reap_dead start 2024-04-25T01:53:09.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.289+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 shutdown_connections 2024-04-25T01:53:09.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.289+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 >> 172.21.15.31:0/3436194768 conn(0x7fb6100780e0 msgr2=0x7fb6100ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:09.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.289+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 shutdown_connections 2024-04-25T01:53:09.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.289+0000 7fb616bb1700 1 -- 172.21.15.31:0/3436194768 wait complete. 2024-04-25T01:53:09.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.921+0000 7f9f06a33700 1 -- 172.21.15.31:0/1089706387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f00101130 msgr2=0x7f9f001015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.921+0000 7f9f06a33700 1 --2- 172.21.15.31:0/1089706387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f00101130 0x7f9f001015b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009990 tx=0x7f9ef4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.921+0000 7f9f06a33700 1 -- 172.21.15.31:0/1089706387 shutdown_connections 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.921+0000 7f9f06a33700 1 --2- 172.21.15.31:0/1089706387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f00101af0 0x7f9f0010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.921+0000 7f9f06a33700 1 --2- 172.21.15.31:0/1089706387 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f001071a0 0x7f9f00107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.921+0000 7f9f06a33700 1 --2- 172.21.15.31:0/1089706387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f00101130 0x7f9f001015b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.921+0000 7f9f06a33700 1 -- 172.21.15.31:0/1089706387 >> 172.21.15.31:0/1089706387 conn(0x7f9f000780e0 msgr2=0x7f9f000784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.921+0000 7f9f06a33700 1 -- 172.21.15.31:0/1089706387 shutdown_connections 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.921+0000 7f9f06a33700 1 -- 172.21.15.31:0/1089706387 wait complete. 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.922+0000 7f9f06a33700 1 Processor -- start 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.922+0000 7f9f06a33700 1 -- start start 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.922+0000 7f9f06a33700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f00101130 0x7f9f00199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:09.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.922+0000 7f9f06a33700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f00101af0 0x7f9f00199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:09.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.922+0000 7f9f06a33700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f001071a0 0x7f9f001a47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:09.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.922+0000 7f9f06a33700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9f001112e0 con 0x7f9f00101af0 2024-04-25T01:53:09.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.922+0000 7f9f06a33700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9f00111160 con 0x7f9f001071a0 2024-04-25T01:53:09.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.922+0000 7f9f06a33700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9f00111460 con 0x7f9f00101130 2024-04-25T01:53:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.923+0000 7f9eff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f00101af0 0x7f9f00199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.923+0000 7f9eff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f00101af0 0x7f9f00199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47252/0 (socket says 172.21.15.31:47252) 2024-04-25T01:53:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.923+0000 7f9eff7fe700 1 -- 172.21.15.31:0/3149095937 learned_addr learned my addr 172.21.15.31:0/3149095937 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.923+0000 7f9efffff700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f00101130 0x7f9f00199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.923+0000 7f9eff7fe700 1 -- 172.21.15.31:0/3149095937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f00101130 msgr2=0x7f9f00199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.923+0000 7f9eff7fe700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f00101130 0x7f9f00199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.923+0000 7f9eff7fe700 1 -- 172.21.15.31:0/3149095937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f001071a0 msgr2=0x7f9f001a47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.923+0000 7f9eff7fe700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f001071a0 0x7f9f001a47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.923+0000 7f9eff7fe700 1 -- 172.21.15.31:0/3149095937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f001a4d00 con 0x7f9f00101af0 2024-04-25T01:53:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.923+0000 7f9efffff700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f00101130 0x7f9f00199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:09.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.923+0000 7f9eff7fe700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f00101af0 0x7f9f00199c80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4000c00 tx=0x7f9ef40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:09.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.925+0000 7f9efd7fa700 1 -- 172.21.15.31:0/3149095937 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4027070 con 0x7f9f00101af0 2024-04-25T01:53:09.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.926+0000 7f9efd7fa700 1 -- 172.21.15.31:0/3149095937 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ef4022470 con 0x7f9f00101af0 2024-04-25T01:53:09.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.926+0000 7f9efd7fa700 1 -- 172.21.15.31:0/3149095937 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4017650 con 0x7f9f00101af0 2024-04-25T01:53:09.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.926+0000 7f9f06a33700 1 -- 172.21.15.31:0/3149095937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f001a4ed0 con 0x7f9f00101af0 2024-04-25T01:53:09.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.926+0000 7f9f06a33700 1 -- 172.21.15.31:0/3149095937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f001a5270 con 0x7f9f00101af0 2024-04-25T01:53:09.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.927+0000 7f9f06a33700 1 -- 172.21.15.31:0/3149095937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f000628c0 con 0x7f9f00101af0 2024-04-25T01:53:09.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.931+0000 7f9efd7fa700 1 -- 172.21.15.31:0/3149095937 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9ef4009d10 con 0x7f9f00101af0 2024-04-25T01:53:09.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.932+0000 7f9efd7fa700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9ee80a2db0 0x7f9ee80a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.932+0000 7f9efd7fa700 1 -- 172.21.15.31:0/3149095937 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f9ef40d0280 con 0x7f9f00101af0 2024-04-25T01:53:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.932+0000 7f9efffff700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9ee80a2db0 0x7f9ee80a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.932+0000 7f9efd7fa700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542] conn(0x7f9ee80ac590 0x7f9ee80ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.932+0000 7f9efd7fa700 1 -- 172.21.15.31:0/3149095937 --> [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ef4001d30 con 0x7f9ee80ac590 2024-04-25T01:53:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.932+0000 7f9efd7fa700 1 -- 172.21.15.31:0/3149095937 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f9ef40d0670 con 0x7f9f00101af0 2024-04-25T01:53:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.933+0000 7f9f04fd0700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542] conn(0x7f9ee80ac590 0x7f9ee80ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.933+0000 7f9efffff700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9ee80a2db0 0x7f9ee80a5270 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0005950 tx=0x7f9ef000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:09.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.933+0000 7f9f04fd0700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542] conn(0x7f9ee80ac590 0x7f9ee80ae9b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:09.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.934+0000 7f9efd7fa700 1 -- 172.21.15.31:0/3149095937 <== osd.6 v2:172.21.15.31:6818/2233839542 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ef4001d30 con 0x7f9ee80ac590 2024-04-25T01:53:09.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.952+0000 7f9f06a33700 1 -- 172.21.15.31:0/3149095937 --> [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9f00062d90 con 0x7f9ee80ac590 2024-04-25T01:53:09.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.952+0000 7f9efd7fa700 1 -- 172.21.15.31:0/3149095937 <== osd.6 v2:172.21.15.31:6818/2233839542 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9f00062d90 con 0x7f9ee80ac590 2024-04-25T01:53:09.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.953+0000 7f9f06a33700 1 -- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542] conn(0x7f9ee80ac590 msgr2=0x7f9ee80ae9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.953+0000 7f9f06a33700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542] conn(0x7f9ee80ac590 0x7f9ee80ae9b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.953+0000 7f9f06a33700 1 -- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9ee80a2db0 msgr2=0x7f9ee80a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.953+0000 7f9f06a33700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9ee80a2db0 0x7f9ee80a5270 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0005950 tx=0x7f9ef000a300 comp rx=0 tx=0).stop 2024-04-25T01:53:09.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.953+0000 7f9f06a33700 1 -- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f00101af0 msgr2=0x7f9f00199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:09.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.953+0000 7f9f06a33700 1 --2- 172.21.15.31:0/3149095937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f00101af0 0x7f9f00199c80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4000c00 tx=0x7f9ef40047b0 comp rx=0 tx=0).stop 2024-04-25T01:53:09.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.953+0000 7f9f04fd0700 1 -- 172.21.15.31:0/3149095937 reap_dead start 2024-04-25T01:53:09.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.953+0000 7f9f06a33700 1 -- 172.21.15.31:0/3149095937 shutdown_connections 2024-04-25T01:53:09.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.953+0000 7f9f06a33700 1 -- 172.21.15.31:0/3149095937 >> 172.21.15.31:0/3149095937 conn(0x7f9f000780e0 msgr2=0x7f9f000ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:09.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.953+0000 7f9f06a33700 1 -- 172.21.15.31:0/3149095937 shutdown_connections 2024-04-25T01:53:09.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:09.954+0000 7f9f06a33700 1 -- 172.21.15.31:0/3149095937 wait complete. 2024-04-25T01:53:10.513 INFO:teuthology.orchestra.run.smithi031.stdout:77309411394 2024-04-25T01:53:10.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T01:53:10.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.697+0000 7f3120f50700 1 -- 172.21.15.31:0/30306160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f311c06b2a0 msgr2=0x7f311c10f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:10.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.697+0000 7f3120f50700 1 --2- 172.21.15.31:0/30306160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f311c06b2a0 0x7f311c10f110 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f311400b0a0 tx=0x7f311400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:53:10.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.697+0000 7f3120f50700 1 -- 172.21.15.31:0/30306160 shutdown_connections 2024-04-25T01:53:10.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.697+0000 7f3120f50700 1 --2- 172.21.15.31:0/30306160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f311c06b2a0 0x7f311c10f110 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:10.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.697+0000 7f3120f50700 1 --2- 172.21.15.31:0/30306160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f311c1082d0 0x7f311c1086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:10.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.697+0000 7f3120f50700 1 --2- 172.21.15.31:0/30306160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f311c06a900 0x7f311c06ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:10.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.697+0000 7f3120f50700 1 -- 172.21.15.31:0/30306160 >> 172.21.15.31:0/30306160 conn(0x7f311c0780e0 msgr2=0x7f311c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:10.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.698+0000 7f3120f50700 1 -- 172.21.15.31:0/30306160 shutdown_connections 2024-04-25T01:53:10.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.698+0000 7f3120f50700 1 -- 172.21.15.31:0/30306160 wait complete. 2024-04-25T01:53:10.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3120f50700 1 Processor -- start 2024-04-25T01:53:10.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3120f50700 1 -- start start 2024-04-25T01:53:10.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3120f50700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f311c06a900 0x7f311c1015e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:10.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3120f50700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f311c06b2a0 0x7f311c101b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:10.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3120f50700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f311c1082d0 0x7f311c102060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:10.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3120f50700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f311c1123d0 con 0x7f311c06b2a0 2024-04-25T01:53:10.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3120f50700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f311c112250 con 0x7f311c06a900 2024-04-25T01:53:10.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3120f50700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f311c112550 con 0x7f311c1082d0 2024-04-25T01:53:10.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3119d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f311c06b2a0 0x7f311c101b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:10.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3119d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f311c06b2a0 0x7f311c101b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47276/0 (socket says 172.21.15.31:47276) 2024-04-25T01:53:10.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3119d9b700 1 -- 172.21.15.31:0/457026877 learned_addr learned my addr 172.21.15.31:0/457026877 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:10.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f311a59c700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f311c06a900 0x7f311c1015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:10.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f311ad9d700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f311c1082d0 0x7f311c102060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:10.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f311a59c700 1 -- 172.21.15.31:0/457026877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f311c1082d0 msgr2=0x7f311c102060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:10.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f311a59c700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f311c1082d0 0x7f311c102060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:10.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f311a59c700 1 -- 172.21.15.31:0/457026877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f311c06b2a0 msgr2=0x7f311c101b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:10.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f311a59c700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f311c06b2a0 0x7f311c101b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:10.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f311a59c700 1 -- 172.21.15.31:0/457026877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311c1a6df0 con 0x7f311c06a900 2024-04-25T01:53:10.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.699+0000 7f3119d9b700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f311c06b2a0 0x7f311c101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:10.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.700+0000 7f311a59c700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f311c06a900 0x7f311c1015e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f310c00e9f0 tx=0x7f310c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:10.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.700+0000 7f310b7fe700 1 -- 172.21.15.31:0/457026877 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c00c9c0 con 0x7f311c06a900 2024-04-25T01:53:10.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.700+0000 7f310b7fe700 1 -- 172.21.15.31:0/457026877 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f310c00cb60 con 0x7f311c06a900 2024-04-25T01:53:10.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.700+0000 7f3120f50700 1 -- 172.21.15.31:0/457026877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311c1a70e0 con 0x7f311c06a900 2024-04-25T01:53:10.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.700+0000 7f310b7fe700 1 -- 172.21.15.31:0/457026877 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c018680 con 0x7f311c06a900 2024-04-25T01:53:10.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.700+0000 7f3120f50700 1 -- 172.21.15.31:0/457026877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f311c1a7670 con 0x7f311c06a900 2024-04-25T01:53:10.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.701+0000 7f3120f50700 1 -- 172.21.15.31:0/457026877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f311c062900 con 0x7f311c06a900 2024-04-25T01:53:10.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.702+0000 7f310b7fe700 1 -- 172.21.15.31:0/457026877 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f310c010430 con 0x7f311c06a900 2024-04-25T01:53:10.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.703+0000 7f310b7fe700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f31040a2db0 0x7f31040a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:10.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.703+0000 7f3119d9b700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f31040a2db0 0x7f31040a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:10.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.703+0000 7f310b7fe700 1 -- 172.21.15.31:0/457026877 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f310c0cf6f0 con 0x7f311c06a900 2024-04-25T01:53:10.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.703+0000 7f310b7fe700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197] conn(0x7f31040ac590 0x7f31040ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:10.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.703+0000 7f310b7fe700 1 -- 172.21.15.31:0/457026877 --> [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31040af060 con 0x7f31040ac590 2024-04-25T01:53:10.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.703+0000 7f310b7fe700 1 -- 172.21.15.31:0/457026877 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f310c0cfae0 con 0x7f311c06a900 2024-04-25T01:53:10.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.703+0000 7f3119d9b700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f31040a2db0 0x7f31040a5270 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f311c078750 tx=0x7f3110007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:10.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.703+0000 7f311ad9d700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197] conn(0x7f31040ac590 0x7f31040ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:10.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.704+0000 7f311ad9d700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197] conn(0x7f31040ac590 0x7f31040ae9b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:10.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.705+0000 7f310b7fe700 1 -- 172.21.15.31:0/457026877 <== osd.10 v2:172.21.15.40:6824/2339369197 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31040af060 con 0x7f31040ac590 2024-04-25T01:53:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.721+0000 7f3120f50700 1 -- 172.21.15.31:0/457026877 --> [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f311c062d30 con 0x7f31040ac590 2024-04-25T01:53:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.721+0000 7f310b7fe700 1 -- 172.21.15.31:0/457026877 <== osd.10 v2:172.21.15.40:6824/2339369197 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f311c062d30 con 0x7f31040ac590 2024-04-25T01:53:10.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.721+0000 7f3120f50700 1 -- 172.21.15.31:0/457026877 >> [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197] conn(0x7f31040ac590 msgr2=0x7f31040ae9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:10.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.721+0000 7f3120f50700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197] conn(0x7f31040ac590 0x7f31040ae9b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:10.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.722+0000 7f3120f50700 1 -- 172.21.15.31:0/457026877 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f31040a2db0 msgr2=0x7f31040a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:10.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.722+0000 7f3120f50700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f31040a2db0 0x7f31040a5270 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f311c078750 tx=0x7f3110007480 comp rx=0 tx=0).stop 2024-04-25T01:53:10.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.722+0000 7f3120f50700 1 -- 172.21.15.31:0/457026877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f311c06a900 msgr2=0x7f311c1015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.722+0000 7f3120f50700 1 --2- 172.21.15.31:0/457026877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f311c06a900 0x7f311c1015e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f310c00e9f0 tx=0x7f310c00ed00 comp rx=0 tx=0).stop 2024-04-25T01:53:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.722+0000 7f311ad9d700 1 -- 172.21.15.31:0/457026877 reap_dead start 2024-04-25T01:53:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.722+0000 7f3120f50700 1 -- 172.21.15.31:0/457026877 shutdown_connections 2024-04-25T01:53:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.722+0000 7f3120f50700 1 -- 172.21.15.31:0/457026877 >> 172.21.15.31:0/457026877 conn(0x7f311c0780e0 msgr2=0x7f311c100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.722+0000 7f3120f50700 1 -- 172.21.15.31:0/457026877 shutdown_connections 2024-04-25T01:53:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:10.722+0000 7f3120f50700 1 -- 172.21.15.31:0/457026877 wait complete. 2024-04-25T01:53:11.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:10 smithi134 ceph-mon[33098]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:11.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:10 smithi031 ceph-mon[29204]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:11.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:10 smithi040 ceph-mon[32418]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:11.783 INFO:teuthology.orchestra.run.smithi031.stdout:197568495663 2024-04-25T01:53:11.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T01:53:12.542 INFO:teuthology.orchestra.run.smithi031.stdout:317827579931 2024-04-25T01:53:12.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T01:53:12.992 INFO:teuthology.orchestra.run.smithi031.stdout:261993005093 2024-04-25T01:53:12.993 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T01:53:13.122 INFO:teuthology.orchestra.run.smithi031.stdout:141733920825 2024-04-25T01:53:13.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T01:53:13.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:12 smithi134 ceph-mon[33098]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:13.203 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:12 smithi031 ceph-mon[29204]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:13.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:12 smithi040 ceph-mon[32418]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:13.640 INFO:teuthology.orchestra.run.smithi031.stdout:163208757299 2024-04-25T01:53:13.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T01:53:13.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.914+0000 7f44ee3ae700 1 -- 172.21.15.31:0/3876989622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44e81064c0 msgr2=0x7f44e810e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.914+0000 7f44ee3ae700 1 --2- 172.21.15.31:0/3876989622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44e81064c0 0x7f44e810e060 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f44e000b0d0 tx=0x7f44e000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:53:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.914+0000 7f44ee3ae700 1 -- 172.21.15.31:0/3876989622 shutdown_connections 2024-04-25T01:53:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.914+0000 7f44ee3ae700 1 --2- 172.21.15.31:0/3876989622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44e81064c0 0x7f44e810e060 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.914+0000 7f44ee3ae700 1 --2- 172.21.15.31:0/3876989622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44e8101770 0x7f44e8101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.914+0000 7f44ee3ae700 1 --2- 172.21.15.31:0/3876989622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44e8102120 0x7f44e8105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.914+0000 7f44ee3ae700 1 -- 172.21.15.31:0/3876989622 >> 172.21.15.31:0/3876989622 conn(0x7f44e80780e0 msgr2=0x7f44e80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.914+0000 7f44ee3ae700 1 -- 172.21.15.31:0/3876989622 shutdown_connections 2024-04-25T01:53:13.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.914+0000 7f44ee3ae700 1 -- 172.21.15.31:0/3876989622 wait complete. 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.915+0000 7f44ee3ae700 1 Processor -- start 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.915+0000 7f44ee3ae700 1 -- start start 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.915+0000 7f44ee3ae700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44e8101770 0x7f44e8102e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.915+0000 7f44ee3ae700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44e8102120 0x7f44e8103370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.915+0000 7f44ee3ae700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44e81064c0 0x7f44e81038b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44ee3ae700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f44e8111350 con 0x7f44e8101770 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44ee3ae700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f44e81111d0 con 0x7f44e8102120 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44ee3ae700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f44e81114d0 con 0x7f44e81064c0 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44e77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44e8102120 0x7f44e8103370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44e7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44e8101770 0x7f44e8102e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44ec94b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44e81064c0 0x7f44e81038b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44ec94b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44e81064c0 0x7f44e81038b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:50956/0 (socket says 172.21.15.31:50956) 2024-04-25T01:53:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44ec94b700 1 -- 172.21.15.31:0/834136185 learned_addr learned my addr 172.21.15.31:0/834136185 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:13.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44ec94b700 1 -- 172.21.15.31:0/834136185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44e8102120 msgr2=0x7f44e8103370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:13.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44ec94b700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44e8102120 0x7f44e8103370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:13.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44ec94b700 1 -- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44e8101770 msgr2=0x7f44e8102e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:13.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44ec94b700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44e8101770 0x7f44e8102e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:13.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.916+0000 7f44ec94b700 1 -- 172.21.15.31:0/834136185 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e81a6ee0 con 0x7f44e81064c0 2024-04-25T01:53:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.917+0000 7f44ec94b700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44e81064c0 0x7f44e81038b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f44e00095c0 tx=0x7f44e0007a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.918+0000 7f44e57fa700 1 -- 172.21.15.31:0/834136185 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e001f070 con 0x7f44e81064c0 2024-04-25T01:53:13.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.918+0000 7f44e57fa700 1 -- 172.21.15.31:0/834136185 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44e0007e20 con 0x7f44e81064c0 2024-04-25T01:53:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.918+0000 7f44e57fa700 1 -- 172.21.15.31:0/834136185 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e001b4a0 con 0x7f44e81064c0 2024-04-25T01:53:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.918+0000 7f44ee3ae700 1 -- 172.21.15.31:0/834136185 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e81a7170 con 0x7f44e81064c0 2024-04-25T01:53:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.918+0000 7f44ee3ae700 1 -- 172.21.15.31:0/834136185 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e81a7570 con 0x7f44e81064c0 2024-04-25T01:53:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.920+0000 7f44ee3ae700 1 -- 172.21.15.31:0/834136185 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f44e8105240 con 0x7f44e81064c0 2024-04-25T01:53:13.929 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:13.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.925+0000 7f44e57fa700 1 -- 172.21.15.31:0/834136185 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f44e0022070 con 0x7f44e81064c0 2024-04-25T01:53:13.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.926+0000 7f44e57fa700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f44d40a2db0 0x7f44d40a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:13.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.927+0000 7f44e57fa700 1 -- 172.21.15.31:0/834136185 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f44e000f070 con 0x7f44e81064c0 2024-04-25T01:53:13.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.927+0000 7f44e57fa700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424] conn(0x7f44d40ac460 0x7f44d40ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:13.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.927+0000 7f44e57fa700 1 -- 172.21.15.31:0/834136185 --> [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44e000cbc0 con 0x7f44d40ac460 2024-04-25T01:53:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.927+0000 7f44e57fa700 1 -- 172.21.15.31:0/834136185 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f44e00d0700 con 0x7f44e81064c0 2024-04-25T01:53:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.927+0000 7f44e7fff700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f44d40a2db0 0x7f44d40a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.927+0000 7f44e77fe700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424] conn(0x7f44d40ac460 0x7f44d40ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.928+0000 7f44e7fff700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f44d40a2db0 0x7f44d40a5270 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f44d8007be0 tx=0x7f44d80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:13.930 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.928+0000 7f44e77fe700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424] conn(0x7f44d40ac460 0x7f44d40ae880 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:13.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.929+0000 7f44e57fa700 1 -- 172.21.15.31:0/834136185 <== osd.3 v2:172.21.15.31:6810/513563424 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44e000cbc0 con 0x7f44d40ac460 2024-04-25T01:53:13.944 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:13.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.967+0000 7f44ee3ae700 1 -- 172.21.15.31:0/834136185 --> [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f44e8105450 con 0x7f44d40ac460 2024-04-25T01:53:13.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.968+0000 7f44e57fa700 1 -- 172.21.15.31:0/834136185 <== osd.3 v2:172.21.15.31:6810/513563424 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f44e8105450 con 0x7f44d40ac460 2024-04-25T01:53:13.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.971+0000 7f44ee3ae700 1 -- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424] conn(0x7f44d40ac460 msgr2=0x7f44d40ae880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:13.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.971+0000 7f44ee3ae700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424] conn(0x7f44d40ac460 0x7f44d40ae880 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.971+0000 7f44ee3ae700 1 -- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f44d40a2db0 msgr2=0x7f44d40a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.971+0000 7f44ee3ae700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f44d40a2db0 0x7f44d40a5270 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f44d8007be0 tx=0x7f44d80073d0 comp rx=0 tx=0).stop 2024-04-25T01:53:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.971+0000 7f44ee3ae700 1 -- 172.21.15.31:0/834136185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44e81064c0 msgr2=0x7f44e81038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.971+0000 7f44ee3ae700 1 --2- 172.21.15.31:0/834136185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44e81064c0 0x7f44e81038b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f44e00095c0 tx=0x7f44e0007a20 comp rx=0 tx=0).stop 2024-04-25T01:53:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.972+0000 7f44ec94b700 1 -- 172.21.15.31:0/834136185 reap_dead start 2024-04-25T01:53:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.972+0000 7f44ee3ae700 1 -- 172.21.15.31:0/834136185 shutdown_connections 2024-04-25T01:53:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.972+0000 7f44ee3ae700 1 -- 172.21.15.31:0/834136185 >> 172.21.15.31:0/834136185 conn(0x7f44e80780e0 msgr2=0x7f44e8100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.972+0000 7f44ee3ae700 1 -- 172.21.15.31:0/834136185 shutdown_connections 2024-04-25T01:53:13.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:13.972+0000 7f44ee3ae700 1 -- 172.21.15.31:0/834136185 wait complete. 2024-04-25T01:53:14.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:14 smithi031 ceph-mon[29204]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:15.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:14 smithi134 ceph-mon[33098]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:15.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:14 smithi040 ceph-mon[32418]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T01:53:16.107 INFO:teuthology.orchestra.run.smithi031.stdout:107374182464 2024-04-25T01:53:16.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T01:53:17.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:16 smithi134 ceph-mon[33098]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-25T01:53:17.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:16 smithi031 ceph-mon[29204]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-25T01:53:17.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:16 smithi040 ceph-mon[32418]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-25T01:53:17.565 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:17.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.576+0000 7f82e3cb6700 1 -- 172.21.15.31:0/2815035905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc102ec0 msgr2=0x7f82dc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:17.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.576+0000 7f82e3cb6700 1 --2- 172.21.15.31:0/2815035905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc102ec0 0x7f82dc106f00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f82cc0099c0 tx=0x7f82cc009cd0 comp rx=0 tx=0).stop 2024-04-25T01:53:17.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.576+0000 7f82e3cb6700 1 -- 172.21.15.31:0/2815035905 shutdown_connections 2024-04-25T01:53:17.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.576+0000 7f82e3cb6700 1 --2- 172.21.15.31:0/2815035905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82dc1075e0 0x7f82dc10f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:17.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.576+0000 7f82e3cb6700 1 --2- 172.21.15.31:0/2815035905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc102510 0x7f82dc1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:17.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.576+0000 7f82e3cb6700 1 --2- 172.21.15.31:0/2815035905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc102ec0 0x7f82dc106f00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:17.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.576+0000 7f82e3cb6700 1 -- 172.21.15.31:0/2815035905 >> 172.21.15.31:0/2815035905 conn(0x7f82dc0793e0 msgr2=0x7f82dc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:17.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.576+0000 7f82e3cb6700 1 -- 172.21.15.31:0/2815035905 shutdown_connections 2024-04-25T01:53:17.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.576+0000 7f82e3cb6700 1 -- 172.21.15.31:0/2815035905 wait complete. 2024-04-25T01:53:17.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.577+0000 7f82e3cb6700 1 Processor -- start 2024-04-25T01:53:17.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.577+0000 7f82e3cb6700 1 -- start start 2024-04-25T01:53:17.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e3cb6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc102510 0x7f82dc19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e3cb6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82dc102ec0 0x7f82dc19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e3cb6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc1075e0 0x7f82dc198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e3cb6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc112480 con 0x7f82dc102510 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e3cb6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc112300 con 0x7f82dc1075e0 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e3cb6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc112600 con 0x7f82dc102ec0 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e1a52700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc102510 0x7f82dc19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e1a52700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc102510 0x7f82dc19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53130/0 (socket says 172.21.15.31:53130) 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e1a52700 1 -- 172.21.15.31:0/1404523285 learned_addr learned my addr 172.21.15.31:0/1404523285 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e1251700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82dc102ec0 0x7f82dc19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e2253700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc1075e0 0x7f82dc198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e1a52700 1 -- 172.21.15.31:0/1404523285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82dc102ec0 msgr2=0x7f82dc19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e1a52700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82dc102ec0 0x7f82dc19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e1a52700 1 -- 172.21.15.31:0/1404523285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc1075e0 msgr2=0x7f82dc198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e1a52700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc1075e0 0x7f82dc198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e1a52700 1 -- 172.21.15.31:0/1404523285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82dc199040 con 0x7f82dc102510 2024-04-25T01:53:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e1251700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82dc102ec0 0x7f82dc19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.578+0000 7f82e1a52700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc102510 0x7f82dc19e690 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82d800c9b0 tx=0x7f82d800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.579+0000 7f82d2ffd700 1 -- 172.21.15.31:0/1404523285 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8007b00 con 0x7f82dc102510 2024-04-25T01:53:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.579+0000 7f82d2ffd700 1 -- 172.21.15.31:0/1404523285 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82d8007ca0 con 0x7f82dc102510 2024-04-25T01:53:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.579+0000 7f82e3cb6700 1 -- 172.21.15.31:0/1404523285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82dc199330 con 0x7f82dc102510 2024-04-25T01:53:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.579+0000 7f82d2ffd700 1 -- 172.21.15.31:0/1404523285 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8018730 con 0x7f82dc102510 2024-04-25T01:53:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.579+0000 7f82e3cb6700 1 -- 172.21.15.31:0/1404523285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82dc1a5610 con 0x7f82dc102510 2024-04-25T01:53:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.580+0000 7f82e3cb6700 1 -- 172.21.15.31:0/1404523285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82dc1039e0 con 0x7f82dc102510 2024-04-25T01:53:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.581+0000 7f82d2ffd700 1 -- 172.21.15.31:0/1404523285 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f82d800f450 con 0x7f82dc102510 2024-04-25T01:53:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.582+0000 7f82d2ffd700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f82c80a2cb0 0x7f82c80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.582+0000 7f82d2ffd700 1 -- 172.21.15.31:0/1404523285 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f82d80cf370 con 0x7f82dc102510 2024-04-25T01:53:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.582+0000 7f82e1251700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f82c80a2cb0 0x7f82c80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.583+0000 7f82e1251700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f82c80a2cb0 0x7f82c80a5170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f82cc000c00 tx=0x7f82cc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:17.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.589+0000 7f82d2ffd700 1 -- 172.21.15.31:0/1404523285 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f82d8014030 con 0x7f82dc102510 2024-04-25T01:53:17.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.797+0000 7f82e3cb6700 1 -- 172.21.15.31:0/1404523285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f82dc199c90 con 0x7f82dc102510 2024-04-25T01:53:17.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.797+0000 7f82d2ffd700 1 -- 172.21.15.31:0/1404523285 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f82d809c3b0 con 0x7f82dc102510 2024-04-25T01:53:17.798 INFO:teuthology.orchestra.run.smithi031.stdout:352187318294 2024-04-25T01:53:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.799+0000 7f82e3cb6700 1 -- 172.21.15.31:0/1404523285 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f82c80a2cb0 msgr2=0x7f82c80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.799+0000 7f82e3cb6700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f82c80a2cb0 0x7f82c80a5170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f82cc000c00 tx=0x7f82cc005fd0 comp rx=0 tx=0).stop 2024-04-25T01:53:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.799+0000 7f82e3cb6700 1 -- 172.21.15.31:0/1404523285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc102510 msgr2=0x7f82dc19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.799+0000 7f82e3cb6700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc102510 0x7f82dc19e690 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82d800c9b0 tx=0x7f82d800cd70 comp rx=0 tx=0).stop 2024-04-25T01:53:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.799+0000 7f82e3cb6700 1 -- 172.21.15.31:0/1404523285 shutdown_connections 2024-04-25T01:53:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.799+0000 7f82e3cb6700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f82c80a2cb0 0x7f82c80a5170 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.799+0000 7f82e3cb6700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc1075e0 0x7f82dc198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.799+0000 7f82e3cb6700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc102510 0x7f82dc19e690 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.799+0000 7f82e3cb6700 1 --2- 172.21.15.31:0/1404523285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82dc102ec0 0x7f82dc19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.799+0000 7f82e3cb6700 1 -- 172.21.15.31:0/1404523285 >> 172.21.15.31:0/1404523285 conn(0x7f82dc0793e0 msgr2=0x7f82dc1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.800+0000 7f82e3cb6700 1 -- 172.21.15.31:0/1404523285 shutdown_connections 2024-04-25T01:53:17.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:17.800+0000 7f82e3cb6700 1 -- 172.21.15.31:0/1404523285 wait complete. 2024-04-25T01:53:18.141 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:17 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1404523285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:53:18.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:17 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1404523285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:53:18.334 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:18.335 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:18.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:17 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1404523285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:53:19.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:18 smithi134 ceph-mon[33098]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:19.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:18 smithi031 ceph-mon[29204]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:19.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:18 smithi040 ceph-mon[32418]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.115+0000 7f3d10192700 1 -- 172.21.15.31:0/1273693563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d08101dd0 msgr2=0x7f3d08105d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.115+0000 7f3d10192700 1 --2- 172.21.15.31:0/1273693563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d08101dd0 0x7f3d08105d90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80099c0 tx=0x7f3cf8009cd0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.115+0000 7f3d10192700 1 -- 172.21.15.31:0/1273693563 shutdown_connections 2024-04-25T01:53:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.115+0000 7f3d10192700 1 --2- 172.21.15.31:0/1273693563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d08106470 0x7f3d0810e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.115+0000 7f3d10192700 1 --2- 172.21.15.31:0/1273693563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d08101420 0x7f3d08101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.115+0000 7f3d10192700 1 --2- 172.21.15.31:0/1273693563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d08101dd0 0x7f3d08105d90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.115+0000 7f3d10192700 1 -- 172.21.15.31:0/1273693563 >> 172.21.15.31:0/1273693563 conn(0x7f3d08077180 msgr2=0x7f3d08077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.115+0000 7f3d10192700 1 -- 172.21.15.31:0/1273693563 shutdown_connections 2024-04-25T01:53:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.115+0000 7f3d10192700 1 -- 172.21.15.31:0/1273693563 wait complete. 2024-04-25T01:53:20.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.115+0000 7f3d10192700 1 Processor -- start 2024-04-25T01:53:20.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d10192700 1 -- start start 2024-04-25T01:53:20.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d10192700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d08101420 0x7f3d08199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:20.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d10192700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d08101dd0 0x7f3d08199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d10192700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d08106470 0x7f3d081a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d10192700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3d081112e0 con 0x7f3d08101dd0 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d10192700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d08111160 con 0x7f3d08106470 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d10192700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3d08111460 con 0x7f3d08101420 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0df2e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d08101420 0x7f3d08199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0df2e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d08101420 0x7f3d08199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:50990/0 (socket says 172.21.15.31:50990) 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0df2e700 1 -- 172.21.15.31:0/4172987283 learned_addr learned my addr 172.21.15.31:0/4172987283 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0d72d700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d08101dd0 0x7f3d08199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0e72f700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d08106470 0x7f3d081a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0d72d700 1 -- 172.21.15.31:0/4172987283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d08101420 msgr2=0x7f3d08199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0d72d700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d08101420 0x7f3d08199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0d72d700 1 -- 172.21.15.31:0/4172987283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d08106470 msgr2=0x7f3d081a4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:20.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0d72d700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d08106470 0x7f3d081a4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0d72d700 1 -- 172.21.15.31:0/4172987283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d081a4cc0 con 0x7f3d08101dd0 2024-04-25T01:53:20.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0e72f700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d08106470 0x7f3d081a4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:20.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.116+0000 7f3d0d72d700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d08101dd0 0x7f3d08199c40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80098e0 tx=0x7f3cf80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:20.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.117+0000 7f3cfeffd700 1 -- 172.21.15.31:0/4172987283 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8026070 con 0x7f3d08101dd0 2024-04-25T01:53:20.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.117+0000 7f3cfeffd700 1 -- 172.21.15.31:0/4172987283 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3cf800f460 con 0x7f3d08101dd0 2024-04-25T01:53:20.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.117+0000 7f3cfeffd700 1 -- 172.21.15.31:0/4172987283 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8009e60 con 0x7f3d08101dd0 2024-04-25T01:53:20.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.117+0000 7f3d10192700 1 -- 172.21.15.31:0/4172987283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d081a4e90 con 0x7f3d08101dd0 2024-04-25T01:53:20.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.117+0000 7f3d10192700 1 -- 172.21.15.31:0/4172987283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d081a5230 con 0x7f3d08101dd0 2024-04-25T01:53:20.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.118+0000 7f3d10192700 1 -- 172.21.15.31:0/4172987283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d08068f20 con 0x7f3d08101dd0 2024-04-25T01:53:20.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.120+0000 7f3cfeffd700 1 -- 172.21.15.31:0/4172987283 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3cf8029070 con 0x7f3d08101dd0 2024-04-25T01:53:20.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.120+0000 7f3cfeffd700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3cf40a2d30 0x7f3cf40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:20.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.120+0000 7f3d0df2e700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3cf40a2d30 0x7f3cf40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:20.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.120+0000 7f3cfeffd700 1 -- 172.21.15.31:0/4172987283 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f3cf80cfa10 con 0x7f3d08101dd0 2024-04-25T01:53:20.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.121+0000 7f3d0df2e700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3cf40a2d30 0x7f3cf40a51f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400ad20 tx=0x7f3d04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.124+0000 7f3cfeffd700 1 -- 172.21.15.31:0/4172987283 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3cf8030080 con 0x7f3d08101dd0 2024-04-25T01:53:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.337+0000 7f3d10192700 1 -- 172.21.15.31:0/4172987283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f3d0802dd90 con 0x7f3d08101dd0 2024-04-25T01:53:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.337+0000 7f3cfeffd700 1 -- 172.21.15.31:0/4172987283 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3cf8013070 con 0x7f3d08101dd0 2024-04-25T01:53:20.338 INFO:teuthology.orchestra.run.smithi031.stdout:296352743456 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 -- 172.21.15.31:0/4172987283 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3cf40a2d30 msgr2=0x7f3cf40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3cf40a2d30 0x7f3cf40a51f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400ad20 tx=0x7f3d04005f90 comp rx=0 tx=0).stop 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 -- 172.21.15.31:0/4172987283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d08101dd0 msgr2=0x7f3d08199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d08101dd0 0x7f3d08199c40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80098e0 tx=0x7f3cf80045e0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 -- 172.21.15.31:0/4172987283 shutdown_connections 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3cf40a2d30 0x7f3cf40a51f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d08106470 0x7f3d081a4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d08101420 0x7f3d08199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 --2- 172.21.15.31:0/4172987283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d08101dd0 0x7f3d08199c40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 -- 172.21.15.31:0/4172987283 >> 172.21.15.31:0/4172987283 conn(0x7f3d08077180 msgr2=0x7f3d080ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 -- 172.21.15.31:0/4172987283 shutdown_connections 2024-04-25T01:53:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:20.339+0000 7f3d10192700 1 -- 172.21.15.31:0/4172987283 wait complete. 2024-04-25T01:53:20.575 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318292 got 352187318294 for osd.11 2024-04-25T01:53:20.575 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:20.779 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:20.779 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:20.781 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:20.781 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:20.796 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:21.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:20 smithi031 ceph-mon[29204]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:21.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:20 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/4172987283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:53:21.407 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743453 got 296352743456 for osd.9 2024-04-25T01:53:21.407 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:21.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:20 smithi040 ceph-mon[32418]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:21.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:20 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/4172987283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:53:21.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:20 smithi134 ceph-mon[33098]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:21.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:20 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/4172987283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:53:23.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:22 smithi031 ceph-mon[29204]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:53:23.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:22 smithi040 ceph-mon[32418]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:53:23.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:22 smithi134 ceph-mon[33098]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:53:24.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-25T01:53:24.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T01:53:24.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:53:24.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-25T01:53:24.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-25T01:53:24.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-25T01:53:24.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:23 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-25T01:53:24.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:23 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/10527723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:53:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.606+0000 7f8198e44700 1 -- 172.21.15.31:0/2190440685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f819406b5f0 msgr2=0x7f8194106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.606+0000 7f8198e44700 1 --2- 172.21.15.31:0/2190440685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f819406b5f0 0x7f8194106eb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f81880099c0 tx=0x7f8188009cd0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.606+0000 7f8198e44700 1 -- 172.21.15.31:0/2190440685 shutdown_connections 2024-04-25T01:53:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.606+0000 7f8198e44700 1 --2- 172.21.15.31:0/2190440685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8194107590 0x7f819410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.606+0000 7f8198e44700 1 --2- 172.21.15.31:0/2190440685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406ac40 0x7f819406b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.606+0000 7f8198e44700 1 --2- 172.21.15.31:0/2190440685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f819406b5f0 0x7f8194106eb0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.606+0000 7f8198e44700 1 -- 172.21.15.31:0/2190440685 >> 172.21.15.31:0/2190440685 conn(0x7f81940781d0 msgr2=0x7f81940785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.606+0000 7f8198e44700 1 -- 172.21.15.31:0/2190440685 shutdown_connections 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.607+0000 7f8198e44700 1 -- 172.21.15.31:0/2190440685 wait complete. 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.607+0000 7f8198e44700 1 Processor -- start 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.607+0000 7f8198e44700 1 -- start start 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.607+0000 7f8198e44700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f819406ac40 0x7f81941011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.607+0000 7f8198e44700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f819406b5f0 0x7f81941016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.607+0000 7f8198e44700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8194107590 0x7f8194101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.607+0000 7f8198e44700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8194112400 con 0x7f819406b5f0 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.607+0000 7f8198e44700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8194112280 con 0x7f8194107590 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.607+0000 7f8198e44700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8194112580 con 0x7f819406ac40 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8191d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f819406b5f0 0x7f81941016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8191d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f819406b5f0 0x7f81941016e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55044/0 (socket says 172.21.15.31:55044) 2024-04-25T01:53:24.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8191d9b700 1 -- 172.21.15.31:0/10527723 learned_addr learned my addr 172.21.15.31:0/10527723 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8192d9d700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8194107590 0x7f8194101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f819259c700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f819406ac40 0x7f81941011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8191d9b700 1 -- 172.21.15.31:0/10527723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f819406ac40 msgr2=0x7f81941011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8191d9b700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f819406ac40 0x7f81941011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8191d9b700 1 -- 172.21.15.31:0/10527723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8194107590 msgr2=0x7f8194101c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8191d9b700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8194107590 0x7f8194101c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8191d9b700 1 -- 172.21.15.31:0/10527723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81941a6e80 con 0x7f819406b5f0 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8192d9d700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8194107590 0x7f8194101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8191d9b700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f819406b5f0 0x7f81941016e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f818800c010 tx=0x7f81880047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f81837fe700 1 -- 172.21.15.31:0/10527723 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188004830 con 0x7f819406b5f0 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8198e44700 1 -- 172.21.15.31:0/10527723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81941a7110 con 0x7f819406b5f0 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f81837fe700 1 -- 172.21.15.31:0/10527723 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8188005020 con 0x7f819406b5f0 2024-04-25T01:53:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f81837fe700 1 -- 172.21.15.31:0/10527723 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188003dd0 con 0x7f819406b5f0 2024-04-25T01:53:24.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.608+0000 7f8198e44700 1 -- 172.21.15.31:0/10527723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81941a7500 con 0x7f819406b5f0 2024-04-25T01:53:24.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.609+0000 7f8198e44700 1 -- 172.21.15.31:0/10527723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8194068f20 con 0x7f819406b5f0 2024-04-25T01:53:24.247 INFO:teuthology.orchestra.run.smithi031.stdout:34359738448 2024-04-25T01:53:24.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.611+0000 7f81837fe700 1 -- 172.21.15.31:0/10527723 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f81880051d0 con 0x7f819406b5f0 2024-04-25T01:53:24.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.611+0000 7f81837fe700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f817c0a2cb0 0x7f817c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:24.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.611+0000 7f819259c700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f817c0a2cb0 0x7f817c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:24.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.611+0000 7f81837fe700 1 -- 172.21.15.31:0/10527723 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f81880d0500 con 0x7f819406b5f0 2024-04-25T01:53:24.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.612+0000 7f819259c700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f817c0a2cb0 0x7f817c0a5170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8184005950 tx=0x7f81840058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.615+0000 7f81837fe700 1 -- 172.21.15.31:0/10527723 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8188013070 con 0x7f819406b5f0 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.831+0000 7f8198e44700 1 -- 172.21.15.31:0/10527723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f819402dcf0 con 0x7f819406b5f0 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.831+0000 7f81837fe700 1 -- 172.21.15.31:0/10527723 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f818809d660 con 0x7f819406b5f0 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 -- 172.21.15.31:0/10527723 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f817c0a2cb0 msgr2=0x7f817c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f817c0a2cb0 0x7f817c0a5170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8184005950 tx=0x7f81840058e0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 -- 172.21.15.31:0/10527723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f819406b5f0 msgr2=0x7f81941016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f819406b5f0 0x7f81941016e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f818800c010 tx=0x7f81880047b0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 -- 172.21.15.31:0/10527723 shutdown_connections 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f817c0a2cb0 0x7f817c0a5170 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8194107590 0x7f8194101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f819406ac40 0x7f81941011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 --2- 172.21.15.31:0/10527723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f819406b5f0 0x7f81941016e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 -- 172.21.15.31:0/10527723 >> 172.21.15.31:0/10527723 conn(0x7f81940781d0 msgr2=0x7f8194100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 -- 172.21.15.31:0/10527723 shutdown_connections 2024-04-25T01:53:24.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:23.833+0000 7f8198e44700 1 -- 172.21.15.31:0/10527723 wait complete. 2024-04-25T01:53:24.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-25T01:53:24.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T01:53:24.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:53:24.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-25T01:53:24.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-25T01:53:24.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-25T01:53:24.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:23 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-25T01:53:24.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:23 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/10527723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:53:24.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-25T01:53:24.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T01:53:24.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:53:24.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-25T01:53:24.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-25T01:53:24.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-25T01:53:24.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:23 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-25T01:53:24.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:23 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/10527723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:53:25.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:24 smithi031 ceph-mon[29204]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:53:25.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-25T01:53:25.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T01:53:25.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:53:25.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-25T01:53:25.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-25T01:53:25.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-25T01:53:25.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:24 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-25T01:53:25.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:24 smithi031 ceph-mon[29204]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:53:25.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:24 smithi040 ceph-mon[32418]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:53:25.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-25T01:53:25.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T01:53:25.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:53:25.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-25T01:53:25.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-25T01:53:25.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-25T01:53:25.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:24 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-25T01:53:25.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:24 smithi040 ceph-mon[32418]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:53:25.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:24 smithi134 ceph-mon[33098]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:53:25.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-25T01:53:25.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T01:53:25.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:53:25.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-25T01:53:25.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-25T01:53:25.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-25T01:53:25.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:24 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-25T01:53:25.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:24 smithi134 ceph-mon[33098]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:53:26.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:25 smithi031 ceph-mon[29204]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:53:26.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:25 smithi040 ceph-mon[32418]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:53:26.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:25 smithi134 ceph-mon[33098]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:53:26.910 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738441 got 34359738448 for osd.0 2024-04-25T01:53:26.910 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:27.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:26 smithi031 ceph-mon[29204]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-25T01:53:27.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:26 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:53:27.233 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:26 smithi134 ceph-mon[33098]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-25T01:53:27.234 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:26 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:53:27.260 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:26 smithi040 ceph-mon[32418]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-25T01:53:27.260 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:26 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:53:28.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.200+0000 7fd4e8d71700 1 -- 172.21.15.31:0/2213571685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4e4102540 msgr2=0x7fd4e4102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:28.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.200+0000 7fd4e8d71700 1 --2- 172.21.15.31:0/2213571685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4e4102540 0x7fd4e4102920 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00b0a0 tx=0x7fd4dc00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:53:28.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.200+0000 7fd4e8d71700 1 -- 172.21.15.31:0/2213571685 shutdown_connections 2024-04-25T01:53:28.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.200+0000 7fd4e8d71700 1 --2- 172.21.15.31:0/2213571685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4e41075a0 0x7fd4e410f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:28.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.200+0000 7fd4e8d71700 1 --2- 172.21.15.31:0/2213571685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4e4102540 0x7fd4e4102920 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:28.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.200+0000 7fd4e8d71700 1 --2- 172.21.15.31:0/2213571685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4e4102ef0 0x7fd4e4106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:28.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.200+0000 7fd4e8d71700 1 -- 172.21.15.31:0/2213571685 >> 172.21.15.31:0/2213571685 conn(0x7fd4e40792c0 msgr2=0x7fd4e40796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:28.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.200+0000 7fd4e8d71700 1 -- 172.21.15.31:0/2213571685 shutdown_connections 2024-04-25T01:53:28.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.200+0000 7fd4e8d71700 1 -- 172.21.15.31:0/2213571685 wait complete. 2024-04-25T01:53:28.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e8d71700 1 Processor -- start 2024-04-25T01:53:28.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e8d71700 1 -- start start 2024-04-25T01:53:28.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e8d71700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4e4102540 0x7fd4e419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:28.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e8d71700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4e4102ef0 0x7fd4e419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:28.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e8d71700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4e41075a0 0x7fd4e4198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:28.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e8d71700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e41124b0 con 0x7fd4e4102540 2024-04-25T01:53:28.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e8d71700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e4112330 con 0x7fd4e41075a0 2024-04-25T01:53:28.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e8d71700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e4112630 con 0x7fd4e4102ef0 2024-04-25T01:53:28.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4e4102540 0x7fd4e419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:28.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4e4102540 0x7fd4e419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55066/0 (socket says 172.21.15.31:55066) 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e259c700 1 -- 172.21.15.31:0/944338544 learned_addr learned my addr 172.21.15.31:0/944338544 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.201+0000 7fd4e1d9b700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4e4102ef0 0x7fd4e419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4e2d9d700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4e41075a0 0x7fd4e4198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4e1d9b700 1 -- 172.21.15.31:0/944338544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4e41075a0 msgr2=0x7fd4e4198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4e1d9b700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4e41075a0 0x7fd4e4198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4e1d9b700 1 -- 172.21.15.31:0/944338544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4e4102540 msgr2=0x7fd4e419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4e1d9b700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4e4102540 0x7fd4e419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4e1d9b700 1 -- 172.21.15.31:0/944338544 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e4199050 con 0x7fd4e4102ef0 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4e2d9d700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4e41075a0 0x7fd4e4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4e1d9b700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4e4102ef0 0x7fd4e419ebe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00e9c0 tx=0x7fd4cc00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4db7fe700 1 -- 172.21.15.31:0/944338544 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc00cbd0 con 0x7fd4e4102ef0 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4e8d71700 1 -- 172.21.15.31:0/944338544 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4e4199340 con 0x7fd4e4102ef0 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4db7fe700 1 -- 172.21.15.31:0/944338544 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4cc0044d0 con 0x7fd4e4102ef0 2024-04-25T01:53:28.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4db7fe700 1 -- 172.21.15.31:0/944338544 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc005020 con 0x7fd4e4102ef0 2024-04-25T01:53:28.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.202+0000 7fd4e8d71700 1 -- 172.21.15.31:0/944338544 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4e41a5620 con 0x7fd4e4102ef0 2024-04-25T01:53:28.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.203+0000 7fd4e8d71700 1 -- 172.21.15.31:0/944338544 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4e410a7c0 con 0x7fd4e4102ef0 2024-04-25T01:53:28.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.204+0000 7fd4db7fe700 1 -- 172.21.15.31:0/944338544 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd4cc042b10 con 0x7fd4e4102ef0 2024-04-25T01:53:28.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.205+0000 7fd4db7fe700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd4d00a2cb0 0x7fd4d00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:28.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.205+0000 7fd4e259c700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd4d00a2cb0 0x7fd4d00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:28.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.205+0000 7fd4db7fe700 1 -- 172.21.15.31:0/944338544 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fd4cc0cf1e0 con 0x7fd4e4102ef0 2024-04-25T01:53:28.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.205+0000 7fd4e259c700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd4d00a2cb0 0x7fd4d00a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009e60 tx=0x7fd4dc00b740 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:28.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.208+0000 7fd4db7fe700 1 -- 172.21.15.31:0/944338544 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd4cc01e300 con 0x7fd4e4102ef0 2024-04-25T01:53:28.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.417+0000 7fd4e8d71700 1 -- 172.21.15.31:0/944338544 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fd4e4199ca0 con 0x7fd4e4102ef0 2024-04-25T01:53:28.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.418+0000 7fd4db7fe700 1 -- 172.21.15.31:0/944338544 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd4cc09bfb0 con 0x7fd4e4102ef0 2024-04-25T01:53:28.418 INFO:teuthology.orchestra.run.smithi031.stdout:317827579935 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 -- 172.21.15.31:0/944338544 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd4d00a2cb0 msgr2=0x7fd4d00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd4d00a2cb0 0x7fd4d00a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009e60 tx=0x7fd4dc00b740 comp rx=0 tx=0).stop 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 -- 172.21.15.31:0/944338544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4e4102ef0 msgr2=0x7fd4e419ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4e4102ef0 0x7fd4e419ebe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00e9c0 tx=0x7fd4cc00ecd0 comp rx=0 tx=0).stop 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 -- 172.21.15.31:0/944338544 shutdown_connections 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd4d00a2cb0 0x7fd4d00a5170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4e41075a0 0x7fd4e4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4e4102540 0x7fd4e419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 --2- 172.21.15.31:0/944338544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4e4102ef0 0x7fd4e419ebe0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 -- 172.21.15.31:0/944338544 >> 172.21.15.31:0/944338544 conn(0x7fd4e40792c0 msgr2=0x7fd4e4100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 -- 172.21.15.31:0/944338544 shutdown_connections 2024-04-25T01:53:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:28.420+0000 7fd4e8d71700 1 -- 172.21.15.31:0/944338544 wait complete. 2024-04-25T01:53:29.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.149+0000 7ff373e85700 1 -- 172.21.15.31:0/916162541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff36c1086f0 msgr2=0x7ff36c10f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:29.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.149+0000 7ff373e85700 1 --2- 172.21.15.31:0/916162541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff36c1086f0 0x7ff36c10f120 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff36400b0d0 tx=0x7ff36400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:53:29.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.150+0000 7ff373e85700 1 -- 172.21.15.31:0/916162541 shutdown_connections 2024-04-25T01:53:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.150+0000 7ff373e85700 1 --2- 172.21.15.31:0/916162541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff36c1086f0 0x7ff36c10f120 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.150+0000 7ff373e85700 1 --2- 172.21.15.31:0/916162541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff36c100610 0x7ff36c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.150+0000 7ff373e85700 1 --2- 172.21.15.31:0/916162541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff36c100fc0 0x7ff36c108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.150+0000 7ff373e85700 1 -- 172.21.15.31:0/916162541 >> 172.21.15.31:0/916162541 conn(0x7ff36c077330 msgr2=0x7ff36c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.150+0000 7ff373e85700 1 -- 172.21.15.31:0/916162541 shutdown_connections 2024-04-25T01:53:29.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.150+0000 7ff373e85700 1 -- 172.21.15.31:0/916162541 wait complete. 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.150+0000 7ff373e85700 1 Processor -- start 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.150+0000 7ff373e85700 1 -- start start 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff373e85700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff36c100610 0x7ff36c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff373e85700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff36c100fc0 0x7ff36c19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff373e85700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff36c1086f0 0x7ff36c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff373e85700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c112400 con 0x7ff36c100fc0 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff373e85700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c112280 con 0x7ff36c100610 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff373e85700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c112580 con 0x7ff36c1086f0 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff371420700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff36c100fc0 0x7ff36c19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff371420700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff36c100fc0 0x7ff36c19eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55088/0 (socket says 172.21.15.31:55088) 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff371420700 1 -- 172.21.15.31:0/3286384206 learned_addr learned my addr 172.21.15.31:0/3286384206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff371c21700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff36c100610 0x7ff36c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff372422700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff36c1086f0 0x7ff36c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff371420700 1 -- 172.21.15.31:0/3286384206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff36c1086f0 msgr2=0x7ff36c198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff371420700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff36c1086f0 0x7ff36c198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff371420700 1 -- 172.21.15.31:0/3286384206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff36c100610 msgr2=0x7ff36c19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff371420700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff36c100610 0x7ff36c19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff371420700 1 -- 172.21.15.31:0/3286384206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36c198f20 con 0x7ff36c100fc0 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff371420700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff36c100fc0 0x7ff36c19eb60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00d950 tx=0x7ff35c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff362ffd700 1 -- 172.21.15.31:0/3286384206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c016070 con 0x7ff36c100fc0 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff362ffd700 1 -- 172.21.15.31:0/3286384206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff35c004830 con 0x7ff36c100fc0 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff373e85700 1 -- 172.21.15.31:0/3286384206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff36c199210 con 0x7ff36c100fc0 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff362ffd700 1 -- 172.21.15.31:0/3286384206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0054f0 con 0x7ff36c100fc0 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.151+0000 7ff373e85700 1 -- 172.21.15.31:0/3286384206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff36c1a5440 con 0x7ff36c100fc0 2024-04-25T01:53:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.152+0000 7ff373e85700 1 -- 172.21.15.31:0/3286384206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff36c101740 con 0x7ff36c100fc0 2024-04-25T01:53:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.154+0000 7ff362ffd700 1 -- 172.21.15.31:0/3286384206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff35c004220 con 0x7ff36c100fc0 2024-04-25T01:53:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.154+0000 7ff362ffd700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7ff3580a2cb0 0x7ff3580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.154+0000 7ff371c21700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7ff3580a2cb0 0x7ff3580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.154+0000 7ff362ffd700 1 -- 172.21.15.31:0/3286384206 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7ff35c0cf390 con 0x7ff36c100fc0 2024-04-25T01:53:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.155+0000 7ff371c21700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7ff3580a2cb0 0x7ff3580a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff368005950 tx=0x7ff36800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.158+0000 7ff362ffd700 1 -- 172.21.15.31:0/3286384206 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff35c01f020 con 0x7ff36c100fc0 2024-04-25T01:53:29.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:29 smithi031 ceph-mon[29204]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s; 7.4 KiB/s, 1 objects/s recovering 2024-04-25T01:53:29.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.354 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:29 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.354 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:29 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/944338544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:53:29.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.368+0000 7ff373e85700 1 -- 172.21.15.31:0/3286384206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7ff36c02db30 con 0x7ff36c100fc0 2024-04-25T01:53:29.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.368+0000 7ff362ffd700 1 -- 172.21.15.31:0/3286384206 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff35c09c1c0 con 0x7ff36c100fc0 2024-04-25T01:53:29.369 INFO:teuthology.orchestra.run.smithi031.stdout:261993005097 2024-04-25T01:53:29.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 -- 172.21.15.31:0/3286384206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7ff3580a2cb0 msgr2=0x7ff3580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:29.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7ff3580a2cb0 0x7ff3580a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff368005950 tx=0x7ff36800b410 comp rx=0 tx=0).stop 2024-04-25T01:53:29.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 -- 172.21.15.31:0/3286384206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff36c100fc0 msgr2=0x7ff36c19eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:29.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff36c100fc0 0x7ff36c19eb60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00d950 tx=0x7ff35c00dc60 comp rx=0 tx=0).stop 2024-04-25T01:53:29.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 -- 172.21.15.31:0/3286384206 shutdown_connections 2024-04-25T01:53:29.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7ff3580a2cb0 0x7ff3580a5170 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:29.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff36c1086f0 0x7ff36c198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:29.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff36c100610 0x7ff36c19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:29.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 --2- 172.21.15.31:0/3286384206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff36c100fc0 0x7ff36c19eb60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:29.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 -- 172.21.15.31:0/3286384206 >> 172.21.15.31:0/3286384206 conn(0x7ff36c077330 msgr2=0x7ff36c0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:29.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 -- 172.21.15.31:0/3286384206 shutdown_connections 2024-04-25T01:53:29.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:29.371+0000 7ff373e85700 1 -- 172.21.15.31:0/3286384206 wait complete. 2024-04-25T01:53:29.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:29 smithi040 ceph-mon[32418]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s; 7.4 KiB/s, 1 objects/s recovering 2024-04-25T01:53:29.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:29 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:29 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/944338544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:53:29.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:29 smithi134 ceph-mon[33098]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s; 7.4 KiB/s, 1 objects/s recovering 2024-04-25T01:53:29.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:29 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:29.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:29 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/944338544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:53:30.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:30 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3286384206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:53:30.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:30 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3286384206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:53:30.486 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:30 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3286384206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:53:31.183 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005093 got 261993005097 for osd.8 2024-04-25T01:53:31.183 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:31.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:31 smithi040 ceph-mon[32418]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 7.4 KiB/s, 1 objects/s recovering 2024-04-25T01:53:31.423 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:31 smithi031 ceph-mon[29204]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 7.4 KiB/s, 1 objects/s recovering 2024-04-25T01:53:31.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:31 smithi134 ceph-mon[33098]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 7.4 KiB/s, 1 objects/s recovering 2024-04-25T01:53:32.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.326+0000 7f8d17166700 1 -- 172.21.15.31:0/1361028656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d101024e0 msgr2=0x7f8d101028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.326+0000 7f8d17166700 1 --2- 172.21.15.31:0/1361028656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d101024e0 0x7f8d101028c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8d00009990 tx=0x7f8d00009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.326+0000 7f8d17166700 1 -- 172.21.15.31:0/1361028656 shutdown_connections 2024-04-25T01:53:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.326+0000 7f8d17166700 1 --2- 172.21.15.31:0/1361028656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d10107570 0x7f8d1010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.326+0000 7f8d17166700 1 --2- 172.21.15.31:0/1361028656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d101024e0 0x7f8d101028c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.326+0000 7f8d17166700 1 --2- 172.21.15.31:0/1361028656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d10102e90 0x7f8d10106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.326+0000 7f8d17166700 1 -- 172.21.15.31:0/1361028656 >> 172.21.15.31:0/1361028656 conn(0x7f8d10079390 msgr2=0x7f8d100797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.326+0000 7f8d17166700 1 -- 172.21.15.31:0/1361028656 shutdown_connections 2024-04-25T01:53:32.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.326+0000 7f8d17166700 1 -- 172.21.15.31:0/1361028656 wait complete. 2024-04-25T01:53:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.326+0000 7f8d17166700 1 Processor -- start 2024-04-25T01:53:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.326+0000 7f8d17166700 1 -- start start 2024-04-25T01:53:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d17166700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d101024e0 0x7f8d10074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d17166700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d10102e90 0x7f8d10074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d17166700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d10107570 0x7f8d10074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d17166700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10112410 con 0x7f8d101024e0 2024-04-25T01:53:32.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d17166700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10112290 con 0x7f8d10102e90 2024-04-25T01:53:32.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d17166700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10112590 con 0x7f8d10107570 2024-04-25T01:53:32.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d14f02700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d101024e0 0x7f8d10074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d14f02700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d101024e0 0x7f8d10074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55120/0 (socket says 172.21.15.31:55120) 2024-04-25T01:53:32.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d14f02700 1 -- 172.21.15.31:0/1336470125 learned_addr learned my addr 172.21.15.31:0/1336470125 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:32.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d0ffff700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d10102e90 0x7f8d10074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d15703700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d10107570 0x7f8d10074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d14f02700 1 -- 172.21.15.31:0/1336470125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d10107570 msgr2=0x7f8d10074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d14f02700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d10107570 0x7f8d10074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d14f02700 1 -- 172.21.15.31:0/1336470125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d10102e90 msgr2=0x7f8d10074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d14f02700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d10102e90 0x7f8d10074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d14f02700 1 -- 172.21.15.31:0/1336470125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d101a6e50 con 0x7f8d101024e0 2024-04-25T01:53:32.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.327+0000 7f8d14f02700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d101024e0 0x7f8d10074530 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8d00000c00 tx=0x7f8d00003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:32.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.328+0000 7f8d0dffb700 1 -- 172.21.15.31:0/1336470125 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00027070 con 0x7f8d101024e0 2024-04-25T01:53:32.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.328+0000 7f8d0dffb700 1 -- 172.21.15.31:0/1336470125 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d0000bdc0 con 0x7f8d101024e0 2024-04-25T01:53:32.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.328+0000 7f8d17166700 1 -- 172.21.15.31:0/1336470125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d101a70e0 con 0x7f8d101024e0 2024-04-25T01:53:32.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.328+0000 7f8d0dffb700 1 -- 172.21.15.31:0/1336470125 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00022d30 con 0x7f8d101024e0 2024-04-25T01:53:32.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.328+0000 7f8d17166700 1 -- 172.21.15.31:0/1336470125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d101a7530 con 0x7f8d101024e0 2024-04-25T01:53:32.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.329+0000 7f8d17166700 1 -- 172.21.15.31:0/1336470125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d10068f20 con 0x7f8d101024e0 2024-04-25T01:53:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.330+0000 7f8d0dffb700 1 -- 172.21.15.31:0/1336470125 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8d0002a070 con 0x7f8d101024e0 2024-04-25T01:53:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.331+0000 7f8d0dffb700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8cf80a2cb0 0x7f8cf80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.331+0000 7f8d0ffff700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8cf80a2cb0 0x7f8cf80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.331+0000 7f8d0dffb700 1 -- 172.21.15.31:0/1336470125 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f8d000d02a0 con 0x7f8d101024e0 2024-04-25T01:53:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.331+0000 7f8d0ffff700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8cf80a2cb0 0x7f8cf80a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8d10101eb0 tx=0x7f8d040073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:32.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.334+0000 7f8d0dffb700 1 -- 172.21.15.31:0/1336470125 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8d00013070 con 0x7f8d101024e0 2024-04-25T01:53:32.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.534+0000 7fa061b3a700 1 -- 172.21.15.31:0/3302273065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa05c100820 msgr2=0x7fa05c100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.534+0000 7fa061b3a700 1 --2- 172.21.15.31:0/3302273065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa05c100820 0x7fa05c100c80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa044009990 tx=0x7fa044009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.534+0000 7fa061b3a700 1 -- 172.21.15.31:0/3302273065 shutdown_connections 2024-04-25T01:53:32.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.534+0000 7fa061b3a700 1 --2- 172.21.15.31:0/3302273065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa05c1011c0 0x7fa05c10f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.534+0000 7fa061b3a700 1 --2- 172.21.15.31:0/3302273065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa05c1082c0 0x7fa05c1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.534+0000 7fa061b3a700 1 --2- 172.21.15.31:0/3302273065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa05c100820 0x7fa05c100c80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.534+0000 7fa061b3a700 1 -- 172.21.15.31:0/3302273065 >> 172.21.15.31:0/3302273065 conn(0x7fa05c0777a0 msgr2=0x7fa05c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:32.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.535+0000 7fa061b3a700 1 -- 172.21.15.31:0/3302273065 shutdown_connections 2024-04-25T01:53:32.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.535+0000 7fa061b3a700 1 -- 172.21.15.31:0/3302273065 wait complete. 2024-04-25T01:53:32.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.535+0000 7fa061b3a700 1 Processor -- start 2024-04-25T01:53:32.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.535+0000 7fa061b3a700 1 -- start start 2024-04-25T01:53:32.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa061b3a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa05c100820 0x7fa05c101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa061b3a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa05c1011c0 0x7fa05c101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa061b3a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa05c1082c0 0x7fa05c102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa061b3a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa05c1123d0 con 0x7fa05c100820 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa061b3a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa05c112250 con 0x7fa05c1011c0 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa061b3a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa05c112550 con 0x7fa05c1082c0 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa05b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa05c100820 0x7fa05c101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa05b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa05c100820 0x7fa05c101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55142/0 (socket says 172.21.15.31:55142) 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa05b7fe700 1 -- 172.21.15.31:0/4125799424 learned_addr learned my addr 172.21.15.31:0/4125799424 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa052ffd700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa05c1011c0 0x7fa05c101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa05bfff700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa05c1082c0 0x7fa05c102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa05b7fe700 1 -- 172.21.15.31:0/4125799424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa05c1082c0 msgr2=0x7fa05c102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa05b7fe700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa05c1082c0 0x7fa05c102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa05b7fe700 1 -- 172.21.15.31:0/4125799424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa05c1011c0 msgr2=0x7fa05c101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa05b7fe700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa05c1011c0 0x7fa05c101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa05b7fe700 1 -- 172.21.15.31:0/4125799424 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05c1a6df0 con 0x7fa05c100820 2024-04-25T01:53:32.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa05bfff700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa05c1082c0 0x7fa05c102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:32.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.536+0000 7fa05b7fe700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa05c100820 0x7fa05c101a00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00d950 tx=0x7fa04c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:32.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.537+0000 7fa0597fa700 1 -- 172.21.15.31:0/4125799424 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c016070 con 0x7fa05c100820 2024-04-25T01:53:32.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.537+0000 7fa0597fa700 1 -- 172.21.15.31:0/4125799424 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa04c009df0 con 0x7fa05c100820 2024-04-25T01:53:32.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.537+0000 7fa061b3a700 1 -- 172.21.15.31:0/4125799424 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa05c1a70e0 con 0x7fa05c100820 2024-04-25T01:53:32.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.537+0000 7fa0597fa700 1 -- 172.21.15.31:0/4125799424 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c00f3c0 con 0x7fa05c100820 2024-04-25T01:53:32.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.537+0000 7fa061b3a700 1 -- 172.21.15.31:0/4125799424 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa05c1a75f0 con 0x7fa05c100820 2024-04-25T01:53:32.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.538+0000 7fa061b3a700 1 -- 172.21.15.31:0/4125799424 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa05c103b40 con 0x7fa05c100820 2024-04-25T01:53:32.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.539+0000 7fa0597fa700 1 -- 172.21.15.31:0/4125799424 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa04c019070 con 0x7fa05c100820 2024-04-25T01:53:32.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.539+0000 7fa0597fa700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa0480a2cb0 0x7fa0480a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.539+0000 7fa052ffd700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa0480a2cb0 0x7fa0480a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.540+0000 7fa0597fa700 1 -- 172.21.15.31:0/4125799424 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fa04c0cecf0 con 0x7fa05c100820 2024-04-25T01:53:32.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.540+0000 7fa052ffd700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa0480a2cb0 0x7fa0480a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa0440098e0 tx=0x7fa044005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:32.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.544+0000 7fa0597fa700 1 -- 172.21.15.31:0/4125799424 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa04c09bb20 con 0x7fa05c100820 2024-04-25T01:53:32.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.546+0000 7f8d17166700 1 -- 172.21.15.31:0/1336470125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f8d1002dcf0 con 0x7f8d101024e0 2024-04-25T01:53:32.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.547+0000 7f8d0dffb700 1 -- 172.21.15.31:0/1336470125 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8d0009d160 con 0x7f8d101024e0 2024-04-25T01:53:32.547 INFO:teuthology.orchestra.run.smithi031.stdout:197568495667 2024-04-25T01:53:32.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.549+0000 7f8d17166700 1 -- 172.21.15.31:0/1336470125 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8cf80a2cb0 msgr2=0x7f8cf80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.549+0000 7f8d17166700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8cf80a2cb0 0x7f8cf80a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8d10101eb0 tx=0x7f8d040073d0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.549+0000 7f8d17166700 1 -- 172.21.15.31:0/1336470125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d101024e0 msgr2=0x7f8d10074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.549+0000 7f8d17166700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d101024e0 0x7f8d10074530 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8d00000c00 tx=0x7f8d00003a80 comp rx=0 tx=0).stop 2024-04-25T01:53:32.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.550+0000 7f8d17166700 1 -- 172.21.15.31:0/1336470125 shutdown_connections 2024-04-25T01:53:32.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.550+0000 7f8d17166700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8cf80a2cb0 0x7f8cf80a5170 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.550+0000 7f8d17166700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d10107570 0x7f8d10074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.550+0000 7f8d17166700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d101024e0 0x7f8d10074530 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.550+0000 7f8d17166700 1 --2- 172.21.15.31:0/1336470125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d10102e90 0x7f8d10074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.550+0000 7f8d17166700 1 -- 172.21.15.31:0/1336470125 >> 172.21.15.31:0/1336470125 conn(0x7f8d10079390 msgr2=0x7f8d101005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:32.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.550+0000 7f8d17166700 1 -- 172.21.15.31:0/1336470125 shutdown_connections 2024-04-25T01:53:32.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.550+0000 7f8d17166700 1 -- 172.21.15.31:0/1336470125 wait complete. 2024-04-25T01:53:32.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.756+0000 7fa061b3a700 1 -- 172.21.15.31:0/4125799424 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fa05c02db30 con 0x7fa05c100820 2024-04-25T01:53:32.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.756+0000 7fa0597fa700 1 -- 172.21.15.31:0/4125799424 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa04c09b420 con 0x7fa05c100820 2024-04-25T01:53:32.757 INFO:teuthology.orchestra.run.smithi031.stdout:55834574925 2024-04-25T01:53:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.758+0000 7fa061b3a700 1 -- 172.21.15.31:0/4125799424 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa0480a2cb0 msgr2=0x7fa0480a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.758+0000 7fa061b3a700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa0480a2cb0 0x7fa0480a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa0440098e0 tx=0x7fa044005960 comp rx=0 tx=0).stop 2024-04-25T01:53:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.758+0000 7fa061b3a700 1 -- 172.21.15.31:0/4125799424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa05c100820 msgr2=0x7fa05c101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.758+0000 7fa061b3a700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa05c100820 0x7fa05c101a00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00d950 tx=0x7fa04c00dc60 comp rx=0 tx=0).stop 2024-04-25T01:53:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.758+0000 7fa061b3a700 1 -- 172.21.15.31:0/4125799424 shutdown_connections 2024-04-25T01:53:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.758+0000 7fa061b3a700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fa0480a2cb0 0x7fa0480a5170 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.758+0000 7fa061b3a700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa05c1082c0 0x7fa05c102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.758+0000 7fa061b3a700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa05c100820 0x7fa05c101a00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.758+0000 7fa061b3a700 1 --2- 172.21.15.31:0/4125799424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa05c1011c0 0x7fa05c101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.758+0000 7fa061b3a700 1 -- 172.21.15.31:0/4125799424 >> 172.21.15.31:0/4125799424 conn(0x7fa05c0777a0 msgr2=0x7fa05c0fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.759+0000 7fa061b3a700 1 -- 172.21.15.31:0/4125799424 shutdown_connections 2024-04-25T01:53:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.759+0000 7fa061b3a700 1 -- 172.21.15.31:0/4125799424 wait complete. 2024-04-25T01:53:32.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.764+0000 7f25bde2f700 1 -- 172.21.15.31:0/708657720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8101130 msgr2=0x7f25b81015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.764+0000 7f25bde2f700 1 --2- 172.21.15.31:0/708657720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8101130 0x7f25b81015b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009990 tx=0x7f25a8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.764+0000 7f25bde2f700 1 -- 172.21.15.31:0/708657720 shutdown_connections 2024-04-25T01:53:32.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.764+0000 7f25bde2f700 1 --2- 172.21.15.31:0/708657720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25b8101af0 0x7f25b810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.764+0000 7f25bde2f700 1 --2- 172.21.15.31:0/708657720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81071a0 0x7f25b8107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.764+0000 7f25bde2f700 1 --2- 172.21.15.31:0/708657720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8101130 0x7f25b81015b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.764+0000 7f25bde2f700 1 -- 172.21.15.31:0/708657720 >> 172.21.15.31:0/708657720 conn(0x7f25b80780e0 msgr2=0x7f25b80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:32.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.764+0000 7f25bde2f700 1 -- 172.21.15.31:0/708657720 shutdown_connections 2024-04-25T01:53:32.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.764+0000 7f25bde2f700 1 -- 172.21.15.31:0/708657720 wait complete. 2024-04-25T01:53:32.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.765+0000 7f25bde2f700 1 Processor -- start 2024-04-25T01:53:32.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.765+0000 7f25bde2f700 1 -- start start 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.765+0000 7f25bde2f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8101130 0x7f25b819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.765+0000 7f25bde2f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25b8101af0 0x7f25b819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.765+0000 7f25bde2f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81071a0 0x7f25b8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.765+0000 7f25bde2f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f25b81112f0 con 0x7f25b8101130 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.765+0000 7f25bde2f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25b8111170 con 0x7f25b81071a0 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.765+0000 7f25bde2f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f25b8111470 con 0x7f25b8101af0 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.765+0000 7f25b77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8101130 0x7f25b819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.765+0000 7f25b77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8101130 0x7f25b819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38234/0 (socket says 172.21.15.31:38234) 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.765+0000 7f25b77fe700 1 -- 172.21.15.31:0/50956472 learned_addr learned my addr 172.21.15.31:0/50956472 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b7fff700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81071a0 0x7f25b8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b6ffd700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25b8101af0 0x7f25b819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b77fe700 1 -- 172.21.15.31:0/50956472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25b8101af0 msgr2=0x7f25b819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b77fe700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25b8101af0 0x7f25b819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b77fe700 1 -- 172.21.15.31:0/50956472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81071a0 msgr2=0x7f25b8198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b77fe700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81071a0 0x7f25b8198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b77fe700 1 -- 172.21.15.31:0/50956472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b8199000 con 0x7f25b8101130 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b6ffd700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25b8101af0 0x7f25b819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b77fe700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8101130 0x7f25b819e650 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f25a000c9b0 tx=0x7f25a000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b4ff9700 1 -- 172.21.15.31:0/50956472 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0007b00 con 0x7f25b8101130 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b4ff9700 1 -- 172.21.15.31:0/50956472 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25a0007ca0 con 0x7f25b8101130 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25bde2f700 1 -- 172.21.15.31:0/50956472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b81992f0 con 0x7f25b8101130 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25b4ff9700 1 -- 172.21.15.31:0/50956472 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0018730 con 0x7f25b8101130 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.766+0000 7f25bde2f700 1 -- 172.21.15.31:0/50956472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b81a55d0 con 0x7f25b8101130 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.767+0000 7f25bde2f700 1 -- 172.21.15.31:0/50956472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b8102850 con 0x7f25b8101130 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.769+0000 7f25b4ff9700 1 -- 172.21.15.31:0/50956472 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f25a000f450 con 0x7f25b8101130 2024-04-25T01:53:32.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.769+0000 7f25b4ff9700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25a40a2cb0 0x7f25a40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:32.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.769+0000 7f25b6ffd700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25a40a2cb0 0x7f25a40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:32.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.769+0000 7f25b4ff9700 1 -- 172.21.15.31:0/50956472 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f25a00cf510 con 0x7f25b8101130 2024-04-25T01:53:32.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.770+0000 7f25b6ffd700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25a40a2cb0 0x7f25a40a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f25a8006010 tx=0x7f25a800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:32.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.775+0000 7f25b4ff9700 1 -- 172.21.15.31:0/50956472 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f25a0014030 con 0x7f25b8101130 2024-04-25T01:53:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.984+0000 7f25bde2f700 1 -- 172.21.15.31:0/50956472 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f25b81a57a0 con 0x7f25b8101130 2024-04-25T01:53:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.984+0000 7f25b4ff9700 1 -- 172.21.15.31:0/50956472 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f25a00a2090 con 0x7f25b8101130 2024-04-25T01:53:32.985 INFO:teuthology.orchestra.run.smithi031.stdout:77309411400 2024-04-25T01:53:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 -- 172.21.15.31:0/50956472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25a40a2cb0 msgr2=0x7f25a40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25a40a2cb0 0x7f25a40a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f25a8006010 tx=0x7f25a800b540 comp rx=0 tx=0).stop 2024-04-25T01:53:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 -- 172.21.15.31:0/50956472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8101130 msgr2=0x7f25b819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8101130 0x7f25b819e650 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f25a000c9b0 tx=0x7f25a000cd70 comp rx=0 tx=0).stop 2024-04-25T01:53:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 -- 172.21.15.31:0/50956472 shutdown_connections 2024-04-25T01:53:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25a40a2cb0 0x7f25a40a5170 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81071a0 0x7f25b8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25b8101130 0x7f25b819e650 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 --2- 172.21.15.31:0/50956472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25b8101af0 0x7f25b819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 -- 172.21.15.31:0/50956472 >> 172.21.15.31:0/50956472 conn(0x7f25b80780e0 msgr2=0x7f25b81003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:32.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 -- 172.21.15.31:0/50956472 shutdown_connections 2024-04-25T01:53:32.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:32.986+0000 7f25bde2f700 1 -- 172.21.15.31:0/50956472 wait complete. 2024-04-25T01:53:33.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:33 smithi031 ceph-mon[29204]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s; 7.4 KiB/s, 1 objects/s recovering 2024-04-25T01:53:33.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:33 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1336470125' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:53:33.237 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:33 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/4125799424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:53:33.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:33 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/50956472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:53:33.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:33 smithi040 ceph-mon[32418]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s; 7.4 KiB/s, 1 objects/s recovering 2024-04-25T01:53:33.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:33 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1336470125' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:53:33.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:33 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/4125799424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:53:33.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:33 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/50956472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:53:33.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:33 smithi134 ceph-mon[33098]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s; 7.4 KiB/s, 1 objects/s recovering 2024-04-25T01:53:33.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:33 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1336470125' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:53:33.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:33 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/4125799424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:53:33.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:33 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/50956472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:53:33.785 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411394 got 77309411400 for osd.2 2024-04-25T01:53:33.785 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:34.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.593+0000 7f9fe59c2700 1 -- 172.21.15.31:0/4204672440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fe010ec60 msgr2=0x7f9fe0111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:34.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.593+0000 7f9fe59c2700 1 --2- 172.21.15.31:0/4204672440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fe010ec60 0x7f9fe0111050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9fd800b0a0 tx=0x7f9fd800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.593+0000 7f9fe59c2700 1 -- 172.21.15.31:0/4204672440 shutdown_connections 2024-04-25T01:53:34.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.593+0000 7f9fe59c2700 1 --2- 172.21.15.31:0/4204672440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fe010ec60 0x7f9fe0111050 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.593+0000 7f9fe59c2700 1 --2- 172.21.15.31:0/4204672440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9fe0074460 0x7f9fe0074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.593+0000 7f9fe59c2700 1 --2- 172.21.15.31:0/4204672440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9fe0074d80 0x7f9fe010e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.593+0000 7f9fe59c2700 1 -- 172.21.15.31:0/4204672440 >> 172.21.15.31:0/4204672440 conn(0x7f9fe00793e0 msgr2=0x7f9fe00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:34.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.593+0000 7f9fe59c2700 1 -- 172.21.15.31:0/4204672440 shutdown_connections 2024-04-25T01:53:34.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.593+0000 7f9fe59c2700 1 -- 172.21.15.31:0/4204672440 wait complete. 2024-04-25T01:53:34.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.594+0000 7f9fe59c2700 1 Processor -- start 2024-04-25T01:53:34.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.594+0000 7f9fe59c2700 1 -- start start 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.594+0000 7f9fe59c2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9fe0074460 0x7f9fe01a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.594+0000 7f9fe59c2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fe0074d80 0x7f9fe01a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fe59c2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9fe010ec60 0x7f9fe019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fe59c2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe0114690 con 0x7f9fe0074d80 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fe59c2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe0114510 con 0x7f9fe0074460 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fe59c2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe0114810 con 0x7f9fe010ec60 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fde7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fe0074d80 0x7f9fe01a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fde7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fe0074d80 0x7f9fe01a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38258/0 (socket says 172.21.15.31:38258) 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fde7fc700 1 -- 172.21.15.31:0/418165095 learned_addr learned my addr 172.21.15.31:0/418165095 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fdeffd700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9fe0074460 0x7f9fe01a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fdf7fe700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9fe010ec60 0x7f9fe019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fde7fc700 1 -- 172.21.15.31:0/418165095 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9fe010ec60 msgr2=0x7f9fe019a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fde7fc700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9fe010ec60 0x7f9fe019a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fde7fc700 1 -- 172.21.15.31:0/418165095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9fe0074460 msgr2=0x7f9fe01a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:34.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fde7fc700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9fe0074460 0x7f9fe01a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fde7fc700 1 -- 172.21.15.31:0/418165095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe019b070 con 0x7f9fe0074d80 2024-04-25T01:53:34.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.595+0000 7f9fde7fc700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fe0074d80 0x7f9fe01a0c90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400c9b0 tx=0x7f9fd400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:34.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.596+0000 7f9fe49c0700 1 -- 172.21.15.31:0/418165095 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4007b00 con 0x7f9fe0074d80 2024-04-25T01:53:34.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.596+0000 7f9fe49c0700 1 -- 172.21.15.31:0/418165095 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9fd400ce50 con 0x7f9fe0074d80 2024-04-25T01:53:34.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.596+0000 7f9fe59c2700 1 -- 172.21.15.31:0/418165095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe019b360 con 0x7f9fe0074d80 2024-04-25T01:53:34.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.596+0000 7f9fe49c0700 1 -- 172.21.15.31:0/418165095 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4018660 con 0x7f9fe0074d80 2024-04-25T01:53:34.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.596+0000 7f9fe59c2700 1 -- 172.21.15.31:0/418165095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe01a7620 con 0x7f9fe0074d80 2024-04-25T01:53:34.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.597+0000 7f9fe59c2700 1 -- 172.21.15.31:0/418165095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe0068f20 con 0x7f9fe0074d80 2024-04-25T01:53:34.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.598+0000 7f9fe49c0700 1 -- 172.21.15.31:0/418165095 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9fd400f450 con 0x7f9fe0074d80 2024-04-25T01:53:34.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.599+0000 7f9fe49c0700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9fc80a2cb0 0x7f9fc80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:34.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.599+0000 7f9fdeffd700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9fc80a2cb0 0x7f9fc80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:34.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.599+0000 7f9fe49c0700 1 -- 172.21.15.31:0/418165095 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f9fd40cefa0 con 0x7f9fe0074d80 2024-04-25T01:53:34.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.600+0000 7f9fdeffd700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9fc80a2cb0 0x7f9fc80a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9fe019ba50 tx=0x7f9fd00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:34.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.604+0000 7f9fe49c0700 1 -- 172.21.15.31:0/418165095 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9fd4014030 con 0x7f9fe0074d80 2024-04-25T01:53:34.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.810+0000 7f9fe59c2700 1 -- 172.21.15.31:0/418165095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f9fe01a78b0 con 0x7f9fe0074d80 2024-04-25T01:53:34.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.811+0000 7f9fe49c0700 1 -- 172.21.15.31:0/418165095 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9fd409be70 con 0x7f9fe0074d80 2024-04-25T01:53:34.812 INFO:teuthology.orchestra.run.smithi031.stdout:223338299439 2024-04-25T01:53:34.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 -- 172.21.15.31:0/418165095 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9fc80a2cb0 msgr2=0x7f9fc80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:34.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9fc80a2cb0 0x7f9fc80a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9fe019ba50 tx=0x7f9fd00073d0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 -- 172.21.15.31:0/418165095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fe0074d80 msgr2=0x7f9fe01a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fe0074d80 0x7f9fe01a0c90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400c9b0 tx=0x7f9fd400ccc0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 -- 172.21.15.31:0/418165095 shutdown_connections 2024-04-25T01:53:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9fc80a2cb0 0x7f9fc80a5170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9fe010ec60 0x7f9fe019a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9fe0074460 0x7f9fe01a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 --2- 172.21.15.31:0/418165095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fe0074d80 0x7f9fe01a0c90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 -- 172.21.15.31:0/418165095 >> 172.21.15.31:0/418165095 conn(0x7f9fe00793e0 msgr2=0x7f9fe0100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 -- 172.21.15.31:0/418165095 shutdown_connections 2024-04-25T01:53:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:34.813+0000 7f9fe59c2700 1 -- 172.21.15.31:0/418165095 wait complete. 2024-04-25T01:53:35.147 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574918 got 55834574925 for osd.1 2024-04-25T01:53:35.147 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:35.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:35 smithi040 ceph-mon[32418]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 6.1 KiB/s, 1 objects/s recovering 2024-04-25T01:53:35.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:35 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/418165095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:53:35.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:35 smithi134 ceph-mon[33098]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 6.1 KiB/s, 1 objects/s recovering 2024-04-25T01:53:35.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:35 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/418165095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:53:35.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:35 smithi031 ceph-mon[29204]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 6.1 KiB/s, 1 objects/s recovering 2024-04-25T01:53:35.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:35 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/418165095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:53:35.851 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579931 got 317827579935 for osd.10 2024-04-25T01:53:35.852 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:36.140 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299434 got 223338299439 for osd.7 2024-04-25T01:53:36.140 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:36.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.932+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2847633519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed4107670 msgr2=0x7f2ed410f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:36.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.932+0000 7f2edb6ba700 1 --2- 172.21.15.31:0/2847633519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed4107670 0x7f2ed410f1d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc00b0d0 tx=0x7f2ecc00b3e0 comp rx=0 tx=0).stop 2024-04-25T01:53:36.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.932+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2847633519 shutdown_connections 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.932+0000 7f2edb6ba700 1 --2- 172.21.15.31:0/2847633519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed4107670 0x7f2ed410f1d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.932+0000 7f2edb6ba700 1 --2- 172.21.15.31:0/2847633519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ed4100e50 0x7f2ed4101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.932+0000 7f2edb6ba700 1 --2- 172.21.15.31:0/2847633519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed4101800 0x7f2ed4106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.932+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2847633519 >> 172.21.15.31:0/2847633519 conn(0x7f2ed4076450 msgr2=0x7f2ed4076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.932+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2847633519 shutdown_connections 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.932+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2847633519 wait complete. 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2edb6ba700 1 Processor -- start 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2edb6ba700 1 -- start start 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2edb6ba700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed4100e50 0x7f2ed419e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2edb6ba700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed4101800 0x7f2ed419eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2edb6ba700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ed4107670 0x7f2ed4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:36.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2edb6ba700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed41124a0 con 0x7f2ed4100e50 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2edb6ba700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed4112320 con 0x7f2ed4101800 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2edb6ba700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed4112620 con 0x7f2ed4107670 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2ed9456700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed4100e50 0x7f2ed419e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2ed9456700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed4100e50 0x7f2ed419e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38278/0 (socket says 172.21.15.31:38278) 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2ed9456700 1 -- 172.21.15.31:0/2755847393 learned_addr learned my addr 172.21.15.31:0/2755847393 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2ed8c55700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed4101800 0x7f2ed419eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2ed9c57700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ed4107670 0x7f2ed4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2ed9456700 1 -- 172.21.15.31:0/2755847393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ed4107670 msgr2=0x7f2ed4198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2ed9456700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ed4107670 0x7f2ed4198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2ed9456700 1 -- 172.21.15.31:0/2755847393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed4101800 msgr2=0x7f2ed419eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2ed9456700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed4101800 0x7f2ed419eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2ed9456700 1 -- 172.21.15.31:0/2755847393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed4198fe0 con 0x7f2ed4100e50 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.933+0000 7f2ed9c57700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ed4107670 0x7f2ed4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:36.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.934+0000 7f2ed9456700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed4100e50 0x7f2ed419e5e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000b530 tx=0x7f2ed000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.934+0000 7f2eca7fc700 1 -- 172.21.15.31:0/2755847393 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0015020 con 0x7f2ed4100e50 2024-04-25T01:53:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.934+0000 7f2eca7fc700 1 -- 172.21.15.31:0/2755847393 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ed0011ce0 con 0x7f2ed4100e50 2024-04-25T01:53:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.934+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2755847393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ed41992d0 con 0x7f2ed4100e50 2024-04-25T01:53:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.934+0000 7f2eca7fc700 1 -- 172.21.15.31:0/2755847393 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0010e40 con 0x7f2ed4100e50 2024-04-25T01:53:36.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.934+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2755847393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ed41a5560 con 0x7f2ed4100e50 2024-04-25T01:53:36.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.935+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2755847393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ed406a400 con 0x7f2ed4100e50 2024-04-25T01:53:36.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.936+0000 7f2eca7fc700 1 -- 172.21.15.31:0/2755847393 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2ed0018020 con 0x7f2ed4100e50 2024-04-25T01:53:36.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.937+0000 7f2eca7fc700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ec00a2d30 0x7f2ec00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:36.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.937+0000 7f2ed8c55700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ec00a2d30 0x7f2ec00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:36.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.937+0000 7f2eca7fc700 1 -- 172.21.15.31:0/2755847393 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f2ed00cedc0 con 0x7f2ed4100e50 2024-04-25T01:53:36.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.937+0000 7f2ed8c55700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ec00a2d30 0x7f2ec00a51f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4199e70 tx=0x7f2ec400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:36.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:36.940+0000 7f2eca7fc700 1 -- 172.21.15.31:0/2755847393 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2ed001f020 con 0x7f2ed4100e50 2024-04-25T01:53:36.947 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:37.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.158+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2755847393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2ed402db30 con 0x7f2ed4100e50 2024-04-25T01:53:37.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.159+0000 7f2eca7fc700 1 -- 172.21.15.31:0/2755847393 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2ed0010380 con 0x7f2ed4100e50 2024-04-25T01:53:37.160 INFO:teuthology.orchestra.run.smithi031.stdout:141733920831 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2755847393 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ec00a2d30 msgr2=0x7f2ec00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ec00a2d30 0x7f2ec00a51f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4199e70 tx=0x7f2ec400b410 comp rx=0 tx=0).stop 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2755847393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed4100e50 msgr2=0x7f2ed419e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed4100e50 0x7f2ed419e5e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000b530 tx=0x7f2ed000b840 comp rx=0 tx=0).stop 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2755847393 shutdown_connections 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ec00a2d30 0x7f2ec00a51f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ed4107670 0x7f2ed4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ed4100e50 0x7f2ed419e5e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 --2- 172.21.15.31:0/2755847393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed4101800 0x7f2ed419eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2755847393 >> 172.21.15.31:0/2755847393 conn(0x7f2ed4076450 msgr2=0x7f2ed40fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2755847393 shutdown_connections 2024-04-25T01:53:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:37.161+0000 7f2edb6ba700 1 -- 172.21.15.31:0/2755847393 wait complete. 2024-04-25T01:53:37.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:37 smithi031 ceph-mon[29204]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 5.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:37.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:37 smithi040 ceph-mon[32418]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 5.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:37.417 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495663 got 197568495667 for osd.6 2024-04-25T01:53:37.418 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:37.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:37 smithi134 ceph-mon[33098]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 5.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:38.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:38 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2755847393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:53:38.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:38 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2755847393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:53:38.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:38 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2755847393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:53:38.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.270+0000 7f553a8ff700 1 -- 172.21.15.31:0/2978288299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f553410e860 msgr2=0x7f553410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:38.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.270+0000 7f553a8ff700 1 --2- 172.21.15.31:0/2978288299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f553410e860 0x7f553410ec40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5524009990 tx=0x7f5524009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.270+0000 7f553a8ff700 1 -- 172.21.15.31:0/2978288299 shutdown_connections 2024-04-25T01:53:38.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.270+0000 7f553a8ff700 1 --2- 172.21.15.31:0/2978288299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5534109220 0x7f5534115730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.270+0000 7f553a8ff700 1 --2- 172.21.15.31:0/2978288299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f553410e860 0x7f553410ec40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.270+0000 7f553a8ff700 1 --2- 172.21.15.31:0/2978288299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5534108860 0x7f5534108ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.270+0000 7f553a8ff700 1 -- 172.21.15.31:0/2978288299 >> 172.21.15.31:0/2978288299 conn(0x7f5534075a00 msgr2=0x7f5534107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.270+0000 7f553a8ff700 1 -- 172.21.15.31:0/2978288299 shutdown_connections 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.270+0000 7f553a8ff700 1 -- 172.21.15.31:0/2978288299 wait complete. 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f553a8ff700 1 Processor -- start 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f553a8ff700 1 -- start start 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f553a8ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5534108860 0x7f553419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f553a8ff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5534109220 0x7f553419f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f553a8ff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f553410e860 0x7f55341a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f553a8ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5534118a60 con 0x7f5534108860 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f553a8ff700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f55341188e0 con 0x7f5534109220 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f553a8ff700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5534118be0 con 0x7f553410e860 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f5533fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5534108860 0x7f553419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f5533fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5534108860 0x7f553419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38288/0 (socket says 172.21.15.31:38288) 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f5533fff700 1 -- 172.21.15.31:0/428507257 learned_addr learned my addr 172.21.15.31:0/428507257 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:38.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.271+0000 7f55337fe700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5534109220 0x7f553419f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f5538e9c700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f553410e860 0x7f55341a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f5533fff700 1 -- 172.21.15.31:0/428507257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f553410e860 msgr2=0x7f55341a38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f5533fff700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f553410e860 0x7f55341a38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f5533fff700 1 -- 172.21.15.31:0/428507257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5534109220 msgr2=0x7f553419f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f5533fff700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5534109220 0x7f553419f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f5533fff700 1 -- 172.21.15.31:0/428507257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55341a3fe0 con 0x7f5534108860 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f5533fff700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5534108860 0x7f553419efd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5524000c00 tx=0x7f55240047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f55317fa700 1 -- 172.21.15.31:0/428507257 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524027070 con 0x7f5534108860 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f55317fa700 1 -- 172.21.15.31:0/428507257 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5524022470 con 0x7f5534108860 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f553a8ff700 1 -- 172.21.15.31:0/428507257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55341a4270 con 0x7f5534108860 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f55317fa700 1 -- 172.21.15.31:0/428507257 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524017650 con 0x7f5534108860 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.272+0000 7f553a8ff700 1 -- 172.21.15.31:0/428507257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55341abb50 con 0x7f5534108860 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.273+0000 7f553a8ff700 1 -- 172.21.15.31:0/428507257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5534110df0 con 0x7f5534108860 2024-04-25T01:53:38.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.275+0000 7f55317fa700 1 -- 172.21.15.31:0/428507257 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f552402a070 con 0x7f5534108860 2024-04-25T01:53:38.687 INFO:teuthology.orchestra.run.smithi031.stdout:163208757305 2024-04-25T01:53:38.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.276+0000 7f55317fa700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f551c0a2d30 0x7f551c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:38.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.276+0000 7f55337fe700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f551c0a2d30 0x7f551c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:38.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.276+0000 7f55317fa700 1 -- 172.21.15.31:0/428507257 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f55240cfee0 con 0x7f5534108860 2024-04-25T01:53:38.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.277+0000 7f55337fe700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f551c0a2d30 0x7f551c0a51f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5534077a50 tx=0x7f552800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:38.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.281+0000 7f55317fa700 1 -- 172.21.15.31:0/428507257 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5524013070 con 0x7f5534108860 2024-04-25T01:53:38.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.507+0000 7f553a8ff700 1 -- 172.21.15.31:0/428507257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f55341a0330 con 0x7f5534108860 2024-04-25T01:53:38.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.507+0000 7f55317fa700 1 -- 172.21.15.31:0/428507257 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f552409cda0 con 0x7f5534108860 2024-04-25T01:53:38.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.509+0000 7f553a8ff700 1 -- 172.21.15.31:0/428507257 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f551c0a2d30 msgr2=0x7f551c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:38.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.509+0000 7f553a8ff700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f551c0a2d30 0x7f551c0a51f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5534077a50 tx=0x7f552800b3c0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.509+0000 7f553a8ff700 1 -- 172.21.15.31:0/428507257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5534108860 msgr2=0x7f553419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:38.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.509+0000 7f553a8ff700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5534108860 0x7f553419efd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5524000c00 tx=0x7f55240047b0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.509+0000 7f553a8ff700 1 -- 172.21.15.31:0/428507257 shutdown_connections 2024-04-25T01:53:38.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.509+0000 7f553a8ff700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f551c0a2d30 0x7f551c0a51f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.509+0000 7f553a8ff700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f553410e860 0x7f55341a38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.509+0000 7f553a8ff700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5534108860 0x7f553419efd0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.509+0000 7f553a8ff700 1 --2- 172.21.15.31:0/428507257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5534109220 0x7f553419f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.510+0000 7f553a8ff700 1 -- 172.21.15.31:0/428507257 >> 172.21.15.31:0/428507257 conn(0x7f5534075a00 msgr2=0x7f55340760e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:38.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.510+0000 7f553a8ff700 1 -- 172.21.15.31:0/428507257 shutdown_connections 2024-04-25T01:53:38.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:38.510+0000 7f553a8ff700 1 -- 172.21.15.31:0/428507257 wait complete. 2024-04-25T01:53:39.310 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757299 got 163208757305 for osd.5 2024-04-25T01:53:39.311 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:39 smithi040 ceph-mon[32418]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 5.0 KiB/s, 0 objects/s recovering 2024-04-25T01:53:39.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:39 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/428507257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:53:39.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:39 smithi134 ceph-mon[33098]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 5.0 KiB/s, 0 objects/s recovering 2024-04-25T01:53:39.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:39 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/428507257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:53:39.458 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:39 smithi031 ceph-mon[29204]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 5.0 KiB/s, 0 objects/s recovering 2024-04-25T01:53:39.458 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:39 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/428507257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:53:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.310+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/3271970397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cb0101130 msgr2=0x7f0cb01015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.310+0000 7f0cb6e0e700 1 --2- 172.21.15.31:0/3271970397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cb0101130 0x7f0cb01015b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009990 tx=0x7f0ca4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.310+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/3271970397 shutdown_connections 2024-04-25T01:53:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.310+0000 7f0cb6e0e700 1 --2- 172.21.15.31:0/3271970397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cb0101af0 0x7f0cb010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.310+0000 7f0cb6e0e700 1 --2- 172.21.15.31:0/3271970397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cb01071a0 0x7f0cb0107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.310+0000 7f0cb6e0e700 1 --2- 172.21.15.31:0/3271970397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cb0101130 0x7f0cb01015b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.310+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/3271970397 >> 172.21.15.31:0/3271970397 conn(0x7f0cb00780e0 msgr2=0x7f0cb00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.310+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/3271970397 shutdown_connections 2024-04-25T01:53:40.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.310+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/3271970397 wait complete. 2024-04-25T01:53:40.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.310+0000 7f0cb6e0e700 1 Processor -- start 2024-04-25T01:53:40.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.310+0000 7f0cb6e0e700 1 -- start start 2024-04-25T01:53:40.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb6e0e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cb0101130 0x7f0cb019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:40.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb6e0e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cb0101af0 0x7f0cb019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb6e0e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cb01071a0 0x7f0cb0198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb6e0e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb0111410 con 0x7f0cb01071a0 2024-04-25T01:53:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb6e0e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb0111290 con 0x7f0cb0101af0 2024-04-25T01:53:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb6e0e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb0111590 con 0x7f0cb0101130 2024-04-25T01:53:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb53ab700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cb01071a0 0x7f0cb0198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb53ab700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cb01071a0 0x7f0cb0198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38312/0 (socket says 172.21.15.31:38312) 2024-04-25T01:53:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb53ab700 1 -- 172.21.15.31:0/2036052068 learned_addr learned my addr 172.21.15.31:0/2036052068 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:40.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb4baa700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cb0101130 0x7f0cb019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0caffff700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cb0101af0 0x7f0cb019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb53ab700 1 -- 172.21.15.31:0/2036052068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cb0101130 msgr2=0x7f0cb019e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb53ab700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cb0101130 0x7f0cb019e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb53ab700 1 -- 172.21.15.31:0/2036052068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cb0101af0 msgr2=0x7f0cb019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb53ab700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cb0101af0 0x7f0cb019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb53ab700 1 -- 172.21.15.31:0/2036052068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb0198fe0 con 0x7f0cb01071a0 2024-04-25T01:53:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb4baa700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cb0101130 0x7f0cb019e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:40.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.311+0000 7f0cb53ab700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cb01071a0 0x7f0cb0198800 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800b970 tx=0x7f0ca800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:40.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.312+0000 7f0cadffb700 1 -- 172.21.15.31:0/2036052068 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8019070 con 0x7f0cb01071a0 2024-04-25T01:53:40.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.312+0000 7f0cadffb700 1 -- 172.21.15.31:0/2036052068 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ca8014440 con 0x7f0cb01071a0 2024-04-25T01:53:40.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.312+0000 7f0cadffb700 1 -- 172.21.15.31:0/2036052068 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca800ce60 con 0x7f0cb01071a0 2024-04-25T01:53:40.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.312+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/2036052068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb01992d0 con 0x7f0cb01071a0 2024-04-25T01:53:40.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.312+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/2036052068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb01a54e0 con 0x7f0cb01071a0 2024-04-25T01:53:40.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.313+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/2036052068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cb0102850 con 0x7f0cb01071a0 2024-04-25T01:53:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.314+0000 7f0cadffb700 1 -- 172.21.15.31:0/2036052068 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0ca801c070 con 0x7f0cb01071a0 2024-04-25T01:53:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.314+0000 7f0cadffb700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0c980a2cb0 0x7f0c980a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.314+0000 7f0cb4baa700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0c980a2cb0 0x7f0c980a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.315+0000 7f0cadffb700 1 -- 172.21.15.31:0/2036052068 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f0ca80ce490 con 0x7f0cb01071a0 2024-04-25T01:53:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.315+0000 7f0cb4baa700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0c980a2cb0 0x7f0c980a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00099d0 tx=0x7f0ca0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:40.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.319+0000 7f0cadffb700 1 -- 172.21.15.31:0/2036052068 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0ca809b3d0 con 0x7f0cb01071a0 2024-04-25T01:53:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.527+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/2036052068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0cb01a5710 con 0x7f0cb01071a0 2024-04-25T01:53:40.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.528+0000 7f0cadffb700 1 -- 172.21.15.31:0/2036052068 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0ca809acd0 con 0x7f0cb01071a0 2024-04-25T01:53:40.528 INFO:teuthology.orchestra.run.smithi031.stdout:107374182469 2024-04-25T01:53:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/2036052068 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0c980a2cb0 msgr2=0x7f0c980a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0c980a2cb0 0x7f0c980a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00099d0 tx=0x7f0ca0008040 comp rx=0 tx=0).stop 2024-04-25T01:53:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/2036052068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cb01071a0 msgr2=0x7f0cb0198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cb01071a0 0x7f0cb0198800 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800b970 tx=0x7f0ca800bd30 comp rx=0 tx=0).stop 2024-04-25T01:53:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/2036052068 shutdown_connections 2024-04-25T01:53:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0c980a2cb0 0x7f0c980a5170 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cb01071a0 0x7f0cb0198800 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cb0101130 0x7f0cb019e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 --2- 172.21.15.31:0/2036052068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cb0101af0 0x7f0cb019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:40.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/2036052068 >> 172.21.15.31:0/2036052068 conn(0x7f0cb00780e0 msgr2=0x7f0cb0100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:40.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/2036052068 shutdown_connections 2024-04-25T01:53:40.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:40.530+0000 7f0cb6e0e700 1 -- 172.21.15.31:0/2036052068 wait complete. 2024-04-25T01:53:41.250 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182464 got 107374182469 for osd.3 2024-04-25T01:53:41.250 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:41.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:41 smithi040 ceph-mon[32418]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T01:53:41.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:41 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2036052068' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:53:41.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:41 smithi134 ceph-mon[33098]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T01:53:41.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:41 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2036052068' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:53:41.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:41 smithi031 ceph-mon[29204]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T01:53:41.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:41 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2036052068' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:53:41.665 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920825 got 141733920831 for osd.4 2024-04-25T01:53:41.666 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:41.666 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T01:53:41.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T01:53:42.114 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:43.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:43 smithi040 ceph-mon[32418]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:43.418 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:43 smithi031 ceph-mon[29204]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:43.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:43 smithi134 ceph-mon[33098]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:43.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.823+0000 7f8087486700 1 -- 172.21.15.31:0/2311009268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80801024e0 msgr2=0x7f80801028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:43.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.823+0000 7f8087486700 1 --2- 172.21.15.31:0/2311009268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80801024e0 0x7f80801028c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f807c009990 tx=0x7f807c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:43.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.823+0000 7f8087486700 1 -- 172.21.15.31:0/2311009268 shutdown_connections 2024-04-25T01:53:43.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.823+0000 7f8087486700 1 --2- 172.21.15.31:0/2311009268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f80801075a0 0x7f808010f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:43.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.823+0000 7f8087486700 1 --2- 172.21.15.31:0/2311009268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80801024e0 0x7f80801028c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:43.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.823+0000 7f8087486700 1 --2- 172.21.15.31:0/2311009268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8080102e90 0x7f8080106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:43.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.823+0000 7f8087486700 1 -- 172.21.15.31:0/2311009268 >> 172.21.15.31:0/2311009268 conn(0x7f80800793e0 msgr2=0x7f80800797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:43.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.823+0000 7f8087486700 1 -- 172.21.15.31:0/2311009268 shutdown_connections 2024-04-25T01:53:43.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.823+0000 7f8087486700 1 -- 172.21.15.31:0/2311009268 wait complete. 2024-04-25T01:53:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.824+0000 7f8087486700 1 Processor -- start 2024-04-25T01:53:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.824+0000 7f8087486700 1 -- start start 2024-04-25T01:53:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.824+0000 7f8087486700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80801024e0 0x7f8080074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.824+0000 7f8087486700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8080102e90 0x7f8080074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.824+0000 7f8087486700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80801075a0 0x7f8080074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.824+0000 7f8087486700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8080112490 con 0x7f80801075a0 2024-04-25T01:53:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8087486700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8080112310 con 0x7f80801024e0 2024-04-25T01:53:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8087486700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8080112610 con 0x7f8080102e90 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8085a23700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80801075a0 0x7f8080074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8085222700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80801024e0 0x7f8080074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8085a23700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80801075a0 0x7f8080074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39836/0 (socket says 172.21.15.31:39836) 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8085a23700 1 -- 172.21.15.31:0/448325997 learned_addr learned my addr 172.21.15.31:0/448325997 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8084a21700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8080102e90 0x7f8080074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8085a23700 1 -- 172.21.15.31:0/448325997 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8080102e90 msgr2=0x7f8080074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8085a23700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8080102e90 0x7f8080074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8085a23700 1 -- 172.21.15.31:0/448325997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80801024e0 msgr2=0x7f8080074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8085a23700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80801024e0 0x7f8080074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8085a23700 1 -- 172.21.15.31:0/448325997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80801a6e90 con 0x7f80801075a0 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8085a23700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80801075a0 0x7f8080074fb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f807800b9e0 tx=0x7f807800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f80767fc700 1 -- 172.21.15.31:0/448325997 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807800d820 con 0x7f80801075a0 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f80767fc700 1 -- 172.21.15.31:0/448325997 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8078015420 con 0x7f80801075a0 2024-04-25T01:53:43.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.825+0000 7f8087486700 1 -- 172.21.15.31:0/448325997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80801a7180 con 0x7f80801075a0 2024-04-25T01:53:43.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.826+0000 7f8087486700 1 -- 172.21.15.31:0/448325997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80801a7710 con 0x7f80801075a0 2024-04-25T01:53:43.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.826+0000 7f80767fc700 1 -- 172.21.15.31:0/448325997 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8078014610 con 0x7f80801075a0 2024-04-25T01:53:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.826+0000 7f8087486700 1 -- 172.21.15.31:0/448325997 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80801039a0 con 0x7f80801075a0 2024-04-25T01:53:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.828+0000 7f80767fc700 1 -- 172.21.15.31:0/448325997 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f80780155d0 con 0x7f80801075a0 2024-04-25T01:53:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.828+0000 7f80767fc700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f806c0a2c10 0x7f806c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.829+0000 7f8085222700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f806c0a2c10 0x7f806c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.829+0000 7f80767fc700 1 -- 172.21.15.31:0/448325997 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f80780cedc0 con 0x7f80801075a0 2024-04-25T01:53:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.829+0000 7f8085222700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f806c0a2c10 0x7f806c0a50d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f807c00b5c0 tx=0x7f807c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:43.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:43.834+0000 7f80767fc700 1 -- 172.21.15.31:0/448325997 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f807809bbf0 con 0x7f80801075a0 2024-04-25T01:53:44.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.079+0000 7f8087486700 1 -- 172.21.15.31:0/448325997 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8080068f20 con 0x7f806c0a2c10 2024-04-25T01:53:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.087+0000 7f80767fc700 1 -- 172.21.15.31:0/448325997 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258997 (secure 0 0 0) 0x7f8080068f20 con 0x7f806c0a2c10 2024-04-25T01:53:44.090 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:53:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 -- 172.21.15.31:0/448325997 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f806c0a2c10 msgr2=0x7f806c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f806c0a2c10 0x7f806c0a50d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f807c00b5c0 tx=0x7f807c005960 comp rx=0 tx=0).stop 2024-04-25T01:53:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 -- 172.21.15.31:0/448325997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80801075a0 msgr2=0x7f8080074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80801075a0 0x7f8080074fb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f807800b9e0 tx=0x7f807800bcf0 comp rx=0 tx=0).stop 2024-04-25T01:53:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 -- 172.21.15.31:0/448325997 shutdown_connections 2024-04-25T01:53:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f806c0a2c10 0x7f806c0a50d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80801075a0 0x7f8080074fb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80801024e0 0x7f8080074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 --2- 172.21.15.31:0/448325997 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8080102e90 0x7f8080074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 -- 172.21.15.31:0/448325997 >> 172.21.15.31:0/448325997 conn(0x7f80800793e0 msgr2=0x7f8080108720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 -- 172.21.15.31:0/448325997 shutdown_connections 2024-04-25T01:53:44.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:44.091+0000 7f8087486700 1 -- 172.21.15.31:0/448325997 wait complete. 2024-04-25T01:53:44.093 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T01:53:44.799 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-25T01:53:43.750196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658852,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":849,"num_read_kb":553,"num_write":421,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1241782,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":346,"ondisk_log_size":346,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369504,"kb_used_data":11948,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572320,"statfs":{"total":1151940427776,"available":1151562055680,"internally_reserved":0,"allocated":12234752,"data_stored":7576476,"data_compressed":120255,"data_compressed_allocated":155648,"data_compressed_original":1843200,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002983"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.968589+0000","last_change":"2024-04-25T01:52:25.523199+0000","last_active":"2024-04-25T01:53:24.968589+0000","last_peered":"2024-04-25T01:53:24.968589+0000","last_clean":"2024-04-25T01:53:24.968589+0000","last_became_active":"2024-04-25T01:52:25.522052+0000","last_became_peered":"2024-04-25T01:52:25.522052+0000","last_unstale":"2024-04-25T01:53:24.968589+0000","last_undegraded":"2024-04-25T01:53:24.968589+0000","last_fullsized":"2024-04-25T01:53:24.968589+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:07:27.095370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296619+0000","last_change":"2024-04-25T01:52:25.528431+0000","last_active":"2024-04-25T01:53:25.296619+0000","last_peered":"2024-04-25T01:53:25.296619+0000","last_clean":"2024-04-25T01:53:25.296619+0000","last_became_active":"2024-04-25T01:52:25.528122+0000","last_became_peered":"2024-04-25T01:52:25.528122+0000","last_unstale":"2024-04-25T01:53:25.296619+0000","last_undegraded":"2024-04-25T01:53:25.296619+0000","last_fullsized":"2024-04-25T01:53:25.296619+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:26:56.794582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296554+0000","last_change":"2024-04-25T01:52:25.532963+0000","last_active":"2024-04-25T01:53:25.296554+0000","last_peered":"2024-04-25T01:53:25.296554+0000","last_clean":"2024-04-25T01:53:25.296554+0000","last_became_active":"2024-04-25T01:52:25.532770+0000","last_became_peered":"2024-04-25T01:52:25.532770+0000","last_unstale":"2024-04-25T01:53:25.296554+0000","last_undegraded":"2024-04-25T01:53:25.296554+0000","last_fullsized":"2024-04-25T01:53:25.296554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:13.398868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.968478+0000","last_change":"2024-04-25T01:53:24.968017+0000","last_active":"2024-04-25T01:53:24.968478+0000","last_peered":"2024-04-25T01:53:24.968478+0000","last_clean":"2024-04-25T01:53:24.968478+0000","last_became_active":"2024-04-25T01:53:24.967740+0000","last_became_peered":"2024-04-25T01:53:24.967740+0000","last_unstale":"2024-04-25T01:53:24.968478+0000","last_undegraded":"2024-04-25T01:53:24.968478+0000","last_fullsized":"2024-04-25T01:53:24.968478+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:45.219797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965326+0000","last_change":"2024-04-25T01:52:25.523624+0000","last_active":"2024-04-25T01:53:24.965326+0000","last_peered":"2024-04-25T01:53:24.965326+0000","last_clean":"2024-04-25T01:53:24.965326+0000","last_became_active":"2024-04-25T01:52:25.523289+0000","last_became_peered":"2024-04-25T01:52:25.523289+0000","last_unstale":"2024-04-25T01:53:24.965326+0000","last_undegraded":"2024-04-25T01:53:24.965326+0000","last_fullsized":"2024-04-25T01:53:24.965326+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:09.688472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961924+0000","last_change":"2024-04-25T01:52:25.527123+0000","last_active":"2024-04-25T01:53:24.961924+0000","last_peered":"2024-04-25T01:53:24.961924+0000","last_clean":"2024-04-25T01:53:24.961924+0000","last_became_active":"2024-04-25T01:52:25.526808+0000","last_became_peered":"2024-04-25T01:52:25.526808+0000","last_unstale":"2024-04-25T01:53:24.961924+0000","last_undegraded":"2024-04-25T01:53:24.961924+0000","last_fullsized":"2024-04-25T01:53:24.961924+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:31.654857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970613+0000","last_change":"2024-04-25T01:52:25.517011+0000","last_active":"2024-04-25T01:53:24.970613+0000","last_peered":"2024-04-25T01:53:24.970613+0000","last_clean":"2024-04-25T01:53:24.970613+0000","last_became_active":"2024-04-25T01:52:25.516680+0000","last_became_peered":"2024-04-25T01:52:25.516680+0000","last_unstale":"2024-04-25T01:53:24.970613+0000","last_undegraded":"2024-04-25T01:53:24.970613+0000","last_fullsized":"2024-04-25T01:53:24.970613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:03.388576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969756+0000","last_change":"2024-04-25T01:52:25.531079+0000","last_active":"2024-04-25T01:53:24.969756+0000","last_peered":"2024-04-25T01:53:24.969756+0000","last_clean":"2024-04-25T01:53:24.969756+0000","last_became_active":"2024-04-25T01:52:25.529200+0000","last_became_peered":"2024-04-25T01:52:25.529200+0000","last_unstale":"2024-04-25T01:53:24.969756+0000","last_undegraded":"2024-04-25T01:53:24.969756+0000","last_fullsized":"2024-04-25T01:53:24.969756+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:30.484010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965280+0000","last_change":"2024-04-25T01:52:25.533421+0000","last_active":"2024-04-25T01:53:24.965280+0000","last_peered":"2024-04-25T01:53:24.965280+0000","last_clean":"2024-04-25T01:53:24.965280+0000","last_became_active":"2024-04-25T01:52:25.531781+0000","last_became_peered":"2024-04-25T01:52:25.531781+0000","last_unstale":"2024-04-25T01:53:24.965280+0000","last_undegraded":"2024-04-25T01:53:24.965280+0000","last_fullsized":"2024-04-25T01:53:24.965280+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:32.505793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.962511+0000","last_change":"2024-04-25T01:52:25.530223+0000","last_active":"2024-04-25T01:53:24.962511+0000","last_peered":"2024-04-25T01:53:24.962511+0000","last_clean":"2024-04-25T01:53:24.962511+0000","last_became_active":"2024-04-25T01:52:25.529909+0000","last_became_peered":"2024-04-25T01:52:25.529909+0000","last_unstale":"2024-04-25T01:53:24.962511+0000","last_undegraded":"2024-04-25T01:53:24.962511+0000","last_fullsized":"2024-04-25T01:53:24.962511+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:07.429026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971132+0000","last_change":"2024-04-25T01:52:25.524502+0000","last_active":"2024-04-25T01:53:24.971132+0000","last_peered":"2024-04-25T01:53:24.971132+0000","last_clean":"2024-04-25T01:53:24.971132+0000","last_became_active":"2024-04-25T01:52:25.524018+0000","last_became_peered":"2024-04-25T01:52:25.524018+0000","last_unstale":"2024-04-25T01:53:24.971132+0000","last_undegraded":"2024-04-25T01:53:24.971132+0000","last_fullsized":"2024-04-25T01:53:24.971132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:23.760389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.360412+0000","last_change":"2024-04-25T01:52:25.533593+0000","last_active":"2024-04-25T01:53:25.360412+0000","last_peered":"2024-04-25T01:53:25.360412+0000","last_clean":"2024-04-25T01:53:25.360412+0000","last_became_active":"2024-04-25T01:52:25.532745+0000","last_became_peered":"2024-04-25T01:52:25.532745+0000","last_unstale":"2024-04-25T01:53:25.360412+0000","last_undegraded":"2024-04-25T01:53:25.360412+0000","last_fullsized":"2024-04-25T01:53:25.360412+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:45.083267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961801+0000","last_change":"2024-04-25T01:52:25.528563+0000","last_active":"2024-04-25T01:53:24.961801+0000","last_peered":"2024-04-25T01:53:24.961801+0000","last_clean":"2024-04-25T01:53:24.961801+0000","last_became_active":"2024-04-25T01:52:25.528236+0000","last_became_peered":"2024-04-25T01:52:25.528236+0000","last_unstale":"2024-04-25T01:53:24.961801+0000","last_undegraded":"2024-04-25T01:53:24.961801+0000","last_fullsized":"2024-04-25T01:53:24.961801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:28:27.222799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296333+0000","last_change":"2024-04-25T01:52:25.524951+0000","last_active":"2024-04-25T01:53:25.296333+0000","last_peered":"2024-04-25T01:53:25.296333+0000","last_clean":"2024-04-25T01:53:25.296333+0000","last_became_active":"2024-04-25T01:52:25.524617+0000","last_became_peered":"2024-04-25T01:52:25.524617+0000","last_unstale":"2024-04-25T01:53:25.296333+0000","last_undegraded":"2024-04-25T01:53:25.296333+0000","last_fullsized":"2024-04-25T01:53:25.296333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:03.561883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296227+0000","last_change":"2024-04-25T01:52:25.532471+0000","last_active":"2024-04-25T01:53:25.296227+0000","last_peered":"2024-04-25T01:53:25.296227+0000","last_clean":"2024-04-25T01:53:25.296227+0000","last_became_active":"2024-04-25T01:52:25.532131+0000","last_became_peered":"2024-04-25T01:52:25.532131+0000","last_unstale":"2024-04-25T01:53:25.296227+0000","last_undegraded":"2024-04-25T01:53:25.296227+0000","last_fullsized":"2024-04-25T01:53:25.296227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:03.478816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964769+0000","last_change":"2024-04-25T01:52:25.529160+0000","last_active":"2024-04-25T01:53:24.964769+0000","last_peered":"2024-04-25T01:53:24.964769+0000","last_clean":"2024-04-25T01:53:24.964769+0000","last_became_active":"2024-04-25T01:52:25.528854+0000","last_became_peered":"2024-04-25T01:52:25.528854+0000","last_unstale":"2024-04-25T01:53:24.964769+0000","last_undegraded":"2024-04-25T01:53:24.964769+0000","last_fullsized":"2024-04-25T01:53:24.964769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:54.713715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.968773+0000","last_change":"2024-04-25T01:52:25.523372+0000","last_active":"2024-04-25T01:53:24.968773+0000","last_peered":"2024-04-25T01:53:24.968773+0000","last_clean":"2024-04-25T01:53:24.968773+0000","last_became_active":"2024-04-25T01:52:25.522538+0000","last_became_peered":"2024-04-25T01:52:25.522538+0000","last_unstale":"2024-04-25T01:53:24.968773+0000","last_undegraded":"2024-04-25T01:53:24.968773+0000","last_fullsized":"2024-04-25T01:53:24.968773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:47.213815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964875+0000","last_change":"2024-04-25T01:52:25.522920+0000","last_active":"2024-04-25T01:53:24.964875+0000","last_peered":"2024-04-25T01:53:24.964875+0000","last_clean":"2024-04-25T01:53:24.964875+0000","last_became_active":"2024-04-25T01:52:25.522594+0000","last_became_peered":"2024-04-25T01:52:25.522594+0000","last_unstale":"2024-04-25T01:53:24.964875+0000","last_undegraded":"2024-04-25T01:53:24.964875+0000","last_fullsized":"2024-04-25T01:53:24.964875+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:45.043667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965577+0000","last_change":"2024-04-25T01:52:25.532738+0000","last_active":"2024-04-25T01:53:24.965577+0000","last_peered":"2024-04-25T01:53:24.965577+0000","last_clean":"2024-04-25T01:53:24.965577+0000","last_became_active":"2024-04-25T01:52:25.532237+0000","last_became_peered":"2024-04-25T01:52:25.532237+0000","last_unstale":"2024-04-25T01:53:24.965577+0000","last_undegraded":"2024-04-25T01:53:24.965577+0000","last_fullsized":"2024-04-25T01:53:24.965577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:54.456704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'13","reported_seq":101,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:38.870790+0000","last_change":"2024-04-25T01:52:24.515686+0000","last_active":"2024-04-25T01:53:38.870790+0000","last_peered":"2024-04-25T01:53:38.870790+0000","last_clean":"2024-04-25T01:53:38.870790+0000","last_became_active":"2024-04-25T01:52:24.515246+0000","last_became_peered":"2024-04-25T01:52:24.515246+0000","last_unstale":"2024-04-25T01:53:38.870790+0000","last_undegraded":"2024-04-25T01:53:38.870790+0000","last_fullsized":"2024-04-25T01:53:38.870790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:14.325747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970110+0000","last_change":"2024-04-25T01:52:25.528796+0000","last_active":"2024-04-25T01:53:24.970110+0000","last_peered":"2024-04-25T01:53:24.970110+0000","last_clean":"2024-04-25T01:53:24.970110+0000","last_became_active":"2024-04-25T01:52:25.528453+0000","last_became_peered":"2024-04-25T01:52:25.528453+0000","last_unstale":"2024-04-25T01:53:24.970110+0000","last_undegraded":"2024-04-25T01:53:24.970110+0000","last_fullsized":"2024-04-25T01:53:24.970110+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:29.432519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970313+0000","last_change":"2024-04-25T01:52:24.520229+0000","last_active":"2024-04-25T01:53:24.970313+0000","last_peered":"2024-04-25T01:53:24.970313+0000","last_clean":"2024-04-25T01:53:24.970313+0000","last_became_active":"2024-04-25T01:52:24.519932+0000","last_became_peered":"2024-04-25T01:52:24.519932+0000","last_unstale":"2024-04-25T01:53:24.970313+0000","last_undegraded":"2024-04-25T01:53:24.970313+0000","last_fullsized":"2024-04-25T01:53:24.970313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.786949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296681+0000","last_change":"2024-04-25T01:52:25.532822+0000","last_active":"2024-04-25T01:53:25.296681+0000","last_peered":"2024-04-25T01:53:25.296681+0000","last_clean":"2024-04-25T01:53:25.296681+0000","last_became_active":"2024-04-25T01:52:25.532462+0000","last_became_peered":"2024-04-25T01:52:25.532462+0000","last_unstale":"2024-04-25T01:53:25.296681+0000","last_undegraded":"2024-04-25T01:53:25.296681+0000","last_fullsized":"2024-04-25T01:53:25.296681+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:55.569883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965899+0000","last_change":"2024-04-25T01:52:24.512923+0000","last_active":"2024-04-25T01:53:24.965899+0000","last_peered":"2024-04-25T01:53:24.965899+0000","last_clean":"2024-04-25T01:53:24.965899+0000","last_became_active":"2024-04-25T01:52:24.512604+0000","last_became_peered":"2024-04-25T01:52:24.512604+0000","last_unstale":"2024-04-25T01:53:24.965899+0000","last_undegraded":"2024-04-25T01:53:24.965899+0000","last_fullsized":"2024-04-25T01:53:24.965899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:41.640234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964962+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:53:24.964962+0000","last_peered":"2024-04-25T01:53:24.964962+0000","last_clean":"2024-04-25T01:53:24.964962+0000","last_became_active":"2024-04-25T01:52:25.526447+0000","last_became_peered":"2024-04-25T01:52:25.526447+0000","last_unstale":"2024-04-25T01:53:24.964962+0000","last_undegraded":"2024-04-25T01:53:24.964962+0000","last_fullsized":"2024-04-25T01:53:24.964962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:10:18.313830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965805+0000","last_change":"2024-04-25T01:52:24.511565+0000","last_active":"2024-04-25T01:53:24.965805+0000","last_peered":"2024-04-25T01:53:24.965805+0000","last_clean":"2024-04-25T01:53:24.965805+0000","last_became_active":"2024-04-25T01:52:24.511258+0000","last_became_peered":"2024-04-25T01:52:24.511258+0000","last_unstale":"2024-04-25T01:53:24.965805+0000","last_undegraded":"2024-04-25T01:53:24.965805+0000","last_fullsized":"2024-04-25T01:53:24.965805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:38.733338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.361468+0000","last_change":"2024-04-25T01:53:25.360965+0000","last_active":"2024-04-25T01:53:25.361468+0000","last_peered":"2024-04-25T01:53:25.361468+0000","last_clean":"2024-04-25T01:53:25.361468+0000","last_became_active":"2024-04-25T01:53:25.360482+0000","last_became_peered":"2024-04-25T01:53:25.360482+0000","last_unstale":"2024-04-25T01:53:25.361468+0000","last_undegraded":"2024-04-25T01:53:25.361468+0000","last_fullsized":"2024-04-25T01:53:25.361468+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:05:01.122687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":93,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:37.872156+0000","last_change":"2024-04-25T01:52:24.514854+0000","last_active":"2024-04-25T01:53:37.872156+0000","last_peered":"2024-04-25T01:53:37.872156+0000","last_clean":"2024-04-25T01:53:37.872156+0000","last_became_active":"2024-04-25T01:52:24.514548+0000","last_became_peered":"2024-04-25T01:52:24.514548+0000","last_unstale":"2024-04-25T01:53:37.872156+0000","last_undegraded":"2024-04-25T01:53:37.872156+0000","last_fullsized":"2024-04-25T01:53:37.872156+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:38.539202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965307+0000","last_change":"2024-04-25T01:52:25.523888+0000","last_active":"2024-04-25T01:53:24.965307+0000","last_peered":"2024-04-25T01:53:24.965307+0000","last_clean":"2024-04-25T01:53:24.965307+0000","last_became_active":"2024-04-25T01:52:25.523581+0000","last_became_peered":"2024-04-25T01:52:25.523581+0000","last_unstale":"2024-04-25T01:53:24.965307+0000","last_undegraded":"2024-04-25T01:53:24.965307+0000","last_fullsized":"2024-04-25T01:53:24.965307+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:59.521757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.972169+0000","last_change":"2024-04-25T01:53:24.972169+0000","last_active":"2024-04-25T01:53:24.972169+0000","last_peered":"2024-04-25T01:53:24.972169+0000","last_clean":"2024-04-25T01:53:24.972169+0000","last_became_active":"2024-04-25T01:53:24.966425+0000","last_became_peered":"2024-04-25T01:53:24.966425+0000","last_unstale":"2024-04-25T01:53:24.972169+0000","last_undegraded":"2024-04-25T01:53:24.972169+0000","last_fullsized":"2024-04-25T01:53:24.972169+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:31.052476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971398+0000","last_change":"2024-04-25T01:52:25.523381+0000","last_active":"2024-04-25T01:53:24.971398+0000","last_peered":"2024-04-25T01:53:24.971398+0000","last_clean":"2024-04-25T01:53:24.971398+0000","last_became_active":"2024-04-25T01:52:25.522978+0000","last_became_peered":"2024-04-25T01:52:25.522978+0000","last_unstale":"2024-04-25T01:53:24.971398+0000","last_undegraded":"2024-04-25T01:53:24.971398+0000","last_fullsized":"2024-04-25T01:53:24.971398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:45.702788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961591+0000","last_change":"2024-04-25T01:52:24.521280+0000","last_active":"2024-04-25T01:53:24.961591+0000","last_peered":"2024-04-25T01:53:24.961591+0000","last_clean":"2024-04-25T01:53:24.961591+0000","last_became_active":"2024-04-25T01:52:24.520959+0000","last_became_peered":"2024-04-25T01:52:24.520959+0000","last_unstale":"2024-04-25T01:53:24.961591+0000","last_undegraded":"2024-04-25T01:53:24.961591+0000","last_fullsized":"2024-04-25T01:53:24.961591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:55.037678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969675+0000","last_change":"2024-04-25T01:52:25.524331+0000","last_active":"2024-04-25T01:53:24.969675+0000","last_peered":"2024-04-25T01:53:24.969675+0000","last_clean":"2024-04-25T01:53:24.969675+0000","last_became_active":"2024-04-25T01:52:25.523431+0000","last_became_peered":"2024-04-25T01:52:25.523431+0000","last_unstale":"2024-04-25T01:53:24.969675+0000","last_undegraded":"2024-04-25T01:53:24.969675+0000","last_fullsized":"2024-04-25T01:53:24.969675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:55.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970795+0000","last_change":"2024-04-25T01:52:24.515697+0000","last_active":"2024-04-25T01:53:24.970795+0000","last_peered":"2024-04-25T01:53:24.970795+0000","last_clean":"2024-04-25T01:53:24.970795+0000","last_became_active":"2024-04-25T01:52:24.515285+0000","last_became_peered":"2024-04-25T01:52:24.515285+0000","last_unstale":"2024-04-25T01:53:24.970795+0000","last_undegraded":"2024-04-25T01:53:24.970795+0000","last_fullsized":"2024-04-25T01:53:24.970795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:24.249203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964509+0000","last_change":"2024-04-25T01:52:25.522415+0000","last_active":"2024-04-25T01:53:24.964509+0000","last_peered":"2024-04-25T01:53:24.964509+0000","last_clean":"2024-04-25T01:53:24.964509+0000","last_became_active":"2024-04-25T01:52:25.521270+0000","last_became_peered":"2024-04-25T01:52:25.521270+0000","last_unstale":"2024-04-25T01:53:24.964509+0000","last_undegraded":"2024-04-25T01:53:24.964509+0000","last_fullsized":"2024-04-25T01:53:24.964509+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:53.514169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":163,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:41.877082+0000","last_change":"2024-04-25T01:52:24.509525+0000","last_active":"2024-04-25T01:53:41.877082+0000","last_peered":"2024-04-25T01:53:41.877082+0000","last_clean":"2024-04-25T01:53:41.877082+0000","last_became_active":"2024-04-25T01:52:24.509184+0000","last_became_peered":"2024-04-25T01:52:24.509184+0000","last_unstale":"2024-04-25T01:53:41.877082+0000","last_undegraded":"2024-04-25T01:53:41.877082+0000","last_fullsized":"2024-04-25T01:53:41.877082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:16:43.301401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965228+0000","last_change":"2024-04-25T01:52:25.525293+0000","last_active":"2024-04-25T01:53:24.965228+0000","last_peered":"2024-04-25T01:53:24.965228+0000","last_clean":"2024-04-25T01:53:24.965228+0000","last_became_active":"2024-04-25T01:52:25.524982+0000","last_became_peered":"2024-04-25T01:52:25.524982+0000","last_unstale":"2024-04-25T01:53:24.965228+0000","last_undegraded":"2024-04-25T01:53:24.965228+0000","last_fullsized":"2024-04-25T01:53:24.965228+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:22.222898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971062+0000","last_change":"2024-04-25T01:52:24.519834+0000","last_active":"2024-04-25T01:53:24.971062+0000","last_peered":"2024-04-25T01:53:24.971062+0000","last_clean":"2024-04-25T01:53:24.971062+0000","last_became_active":"2024-04-25T01:52:24.519521+0000","last_became_peered":"2024-04-25T01:52:24.519521+0000","last_unstale":"2024-04-25T01:53:24.971062+0000","last_undegraded":"2024-04-25T01:53:24.971062+0000","last_fullsized":"2024-04-25T01:53:24.971062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:02.414527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296642+0000","last_change":"2024-04-25T01:52:25.523809+0000","last_active":"2024-04-25T01:53:25.296642+0000","last_peered":"2024-04-25T01:53:25.296642+0000","last_clean":"2024-04-25T01:53:25.296642+0000","last_became_active":"2024-04-25T01:52:25.523406+0000","last_became_peered":"2024-04-25T01:52:25.523406+0000","last_unstale":"2024-04-25T01:53:25.296642+0000","last_undegraded":"2024-04-25T01:53:25.296642+0000","last_fullsized":"2024-04-25T01:53:25.296642+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:22.169633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971618+0000","last_change":"2024-04-25T01:52:24.510226+0000","last_active":"2024-04-25T01:53:24.971618+0000","last_peered":"2024-04-25T01:53:24.971618+0000","last_clean":"2024-04-25T01:53:24.971618+0000","last_became_active":"2024-04-25T01:52:24.509667+0000","last_became_peered":"2024-04-25T01:52:24.509667+0000","last_unstale":"2024-04-25T01:53:24.971618+0000","last_undegraded":"2024-04-25T01:53:24.971618+0000","last_fullsized":"2024-04-25T01:53:24.971618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:49.180675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965130+0000","last_change":"2024-04-25T01:52:25.531567+0000","last_active":"2024-04-25T01:53:24.965130+0000","last_peered":"2024-04-25T01:53:24.965130+0000","last_clean":"2024-04-25T01:53:24.965130+0000","last_became_active":"2024-04-25T01:52:25.531200+0000","last_became_peered":"2024-04-25T01:52:25.531200+0000","last_unstale":"2024-04-25T01:53:24.965130+0000","last_undegraded":"2024-04-25T01:53:24.965130+0000","last_fullsized":"2024-04-25T01:53:24.965130+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:51:14.336413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":217,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:42.390462+0000","last_change":"2024-04-25T01:53:25.369844+0000","last_active":"2024-04-25T01:53:42.390462+0000","last_peered":"2024-04-25T01:53:42.390462+0000","last_clean":"2024-04-25T01:53:42.390462+0000","last_became_active":"2024-04-25T01:53:25.362162+0000","last_became_peered":"2024-04-25T01:53:25.362162+0000","last_unstale":"2024-04-25T01:53:42.390462+0000","last_undegraded":"2024-04-25T01:53:42.390462+0000","last_fullsized":"2024-04-25T01:53:42.390462+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:27:01.730381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":139,"num_read_kb":129,"num_write":33,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965621+0000","last_change":"2024-04-25T01:52:25.525931+0000","last_active":"2024-04-25T01:53:24.965621+0000","last_peered":"2024-04-25T01:53:24.965621+0000","last_clean":"2024-04-25T01:53:24.965621+0000","last_became_active":"2024-04-25T01:52:25.525460+0000","last_became_peered":"2024-04-25T01:52:25.525460+0000","last_unstale":"2024-04-25T01:53:24.965621+0000","last_undegraded":"2024-04-25T01:53:24.965621+0000","last_fullsized":"2024-04-25T01:53:24.965621+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:46.569587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'9","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965434+0000","last_change":"2024-04-25T01:52:24.505470+0000","last_active":"2024-04-25T01:53:24.965434+0000","last_peered":"2024-04-25T01:53:24.965434+0000","last_clean":"2024-04-25T01:53:24.965434+0000","last_became_active":"2024-04-25T01:52:24.504677+0000","last_became_peered":"2024-04-25T01:52:24.504677+0000","last_unstale":"2024-04-25T01:53:24.965434+0000","last_undegraded":"2024-04-25T01:53:24.965434+0000","last_fullsized":"2024-04-25T01:53:24.965434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:01.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961591+0000","last_change":"2024-04-25T01:52:25.526364+0000","last_active":"2024-04-25T01:53:24.961591+0000","last_peered":"2024-04-25T01:53:24.961591+0000","last_clean":"2024-04-25T01:53:24.961591+0000","last_became_active":"2024-04-25T01:52:25.525376+0000","last_became_peered":"2024-04-25T01:52:25.525376+0000","last_unstale":"2024-04-25T01:53:24.961591+0000","last_undegraded":"2024-04-25T01:53:24.961591+0000","last_fullsized":"2024-04-25T01:53:24.961591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:26.770994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971031+0000","last_change":"2024-04-25T01:52:24.520287+0000","last_active":"2024-04-25T01:53:24.971031+0000","last_peered":"2024-04-25T01:53:24.971031+0000","last_clean":"2024-04-25T01:53:24.971031+0000","last_became_active":"2024-04-25T01:52:24.519967+0000","last_became_peered":"2024-04-25T01:52:24.519967+0000","last_unstale":"2024-04-25T01:53:24.971031+0000","last_undegraded":"2024-04-25T01:53:24.971031+0000","last_fullsized":"2024-04-25T01:53:24.971031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:45.078410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970939+0000","last_change":"2024-04-25T01:52:25.525766+0000","last_active":"2024-04-25T01:53:24.970939+0000","last_peered":"2024-04-25T01:53:24.970939+0000","last_clean":"2024-04-25T01:53:24.970939+0000","last_became_active":"2024-04-25T01:52:25.525350+0000","last_became_peered":"2024-04-25T01:52:25.525350+0000","last_unstale":"2024-04-25T01:53:24.970939+0000","last_undegraded":"2024-04-25T01:53:24.970939+0000","last_fullsized":"2024-04-25T01:53:24.970939+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:05.387502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'6","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:37.870352+0000","last_change":"2024-04-25T01:52:24.513674+0000","last_active":"2024-04-25T01:53:37.870352+0000","last_peered":"2024-04-25T01:53:37.870352+0000","last_clean":"2024-04-25T01:53:37.870352+0000","last_became_active":"2024-04-25T01:52:24.513270+0000","last_became_peered":"2024-04-25T01:52:24.513270+0000","last_unstale":"2024-04-25T01:53:37.870352+0000","last_undegraded":"2024-04-25T01:53:37.870352+0000","last_fullsized":"2024-04-25T01:53:37.870352+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:23.385928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970706+0000","last_change":"2024-04-25T01:52:25.517166+0000","last_active":"2024-04-25T01:53:24.970706+0000","last_peered":"2024-04-25T01:53:24.970706+0000","last_clean":"2024-04-25T01:53:24.970706+0000","last_became_active":"2024-04-25T01:52:25.516853+0000","last_became_peered":"2024-04-25T01:52:25.516853+0000","last_unstale":"2024-04-25T01:53:24.970706+0000","last_undegraded":"2024-04-25T01:53:24.970706+0000","last_fullsized":"2024-04-25T01:53:24.970706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:18.502558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'17","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965158+0000","last_change":"2024-04-25T01:52:24.514695+0000","last_active":"2024-04-25T01:53:24.965158+0000","last_peered":"2024-04-25T01:53:24.965158+0000","last_clean":"2024-04-25T01:53:24.965158+0000","last_became_active":"2024-04-25T01:52:24.514370+0000","last_became_peered":"2024-04-25T01:52:24.514370+0000","last_unstale":"2024-04-25T01:53:24.965158+0000","last_undegraded":"2024-04-25T01:53:24.965158+0000","last_fullsized":"2024-04-25T01:53:24.965158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:28:28.537307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971033+0000","last_change":"2024-04-25T01:52:25.527062+0000","last_active":"2024-04-25T01:53:24.971033+0000","last_peered":"2024-04-25T01:53:24.971033+0000","last_clean":"2024-04-25T01:53:24.971033+0000","last_became_active":"2024-04-25T01:52:25.526757+0000","last_became_peered":"2024-04-25T01:52:25.526757+0000","last_unstale":"2024-04-25T01:53:24.971033+0000","last_undegraded":"2024-04-25T01:53:24.971033+0000","last_fullsized":"2024-04-25T01:53:24.971033+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:36.360030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965888+0000","last_change":"2024-04-25T01:52:24.512235+0000","last_active":"2024-04-25T01:53:24.965888+0000","last_peered":"2024-04-25T01:53:24.965888+0000","last_clean":"2024-04-25T01:53:24.965888+0000","last_became_active":"2024-04-25T01:52:24.511910+0000","last_became_peered":"2024-04-25T01:52:24.511910+0000","last_unstale":"2024-04-25T01:53:24.965888+0000","last_undegraded":"2024-04-25T01:53:24.965888+0000","last_fullsized":"2024-04-25T01:53:24.965888+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:38.435734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.968104+0000","last_change":"2024-04-25T01:52:25.527854+0000","last_active":"2024-04-25T01:53:24.968104+0000","last_peered":"2024-04-25T01:53:24.968104+0000","last_clean":"2024-04-25T01:53:24.968104+0000","last_became_active":"2024-04-25T01:52:25.527481+0000","last_became_peered":"2024-04-25T01:52:25.527481+0000","last_unstale":"2024-04-25T01:53:24.968104+0000","last_undegraded":"2024-04-25T01:53:24.968104+0000","last_fullsized":"2024-04-25T01:53:24.968104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:09:11.281982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964620+0000","last_change":"2024-04-25T01:52:24.513463+0000","last_active":"2024-04-25T01:53:24.964620+0000","last_peered":"2024-04-25T01:53:24.964620+0000","last_clean":"2024-04-25T01:53:24.964620+0000","last_became_active":"2024-04-25T01:52:24.513176+0000","last_became_peered":"2024-04-25T01:52:24.513176+0000","last_unstale":"2024-04-25T01:53:24.964620+0000","last_undegraded":"2024-04-25T01:53:24.964620+0000","last_fullsized":"2024-04-25T01:53:24.964620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:13:33.507559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.962527+0000","last_change":"2024-04-25T01:52:25.528316+0000","last_active":"2024-04-25T01:53:24.962527+0000","last_peered":"2024-04-25T01:53:24.962527+0000","last_clean":"2024-04-25T01:53:24.962527+0000","last_became_active":"2024-04-25T01:52:25.527991+0000","last_became_peered":"2024-04-25T01:52:25.527991+0000","last_unstale":"2024-04-25T01:53:24.962527+0000","last_undegraded":"2024-04-25T01:53:24.962527+0000","last_fullsized":"2024-04-25T01:53:24.962527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:55.871663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'5","reported_seq":93,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:38.871278+0000","last_change":"2024-04-25T01:52:24.519357+0000","last_active":"2024-04-25T01:53:38.871278+0000","last_peered":"2024-04-25T01:53:38.871278+0000","last_clean":"2024-04-25T01:53:38.871278+0000","last_became_active":"2024-04-25T01:52:24.519025+0000","last_became_peered":"2024-04-25T01:52:24.519025+0000","last_unstale":"2024-04-25T01:53:38.871278+0000","last_undegraded":"2024-04-25T01:53:38.871278+0000","last_fullsized":"2024-04-25T01:53:38.871278+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:03.476554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.962232+0000","last_change":"2024-04-25T01:52:25.527922+0000","last_active":"2024-04-25T01:53:24.962232+0000","last_peered":"2024-04-25T01:53:24.962232+0000","last_clean":"2024-04-25T01:53:24.962232+0000","last_became_active":"2024-04-25T01:52:25.527596+0000","last_became_peered":"2024-04-25T01:52:25.527596+0000","last_unstale":"2024-04-25T01:53:24.962232+0000","last_undegraded":"2024-04-25T01:53:24.962232+0000","last_fullsized":"2024-04-25T01:53:24.962232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:56.002208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971471+0000","last_change":"2024-04-25T01:52:25.533800+0000","last_active":"2024-04-25T01:53:24.971471+0000","last_peered":"2024-04-25T01:53:24.971471+0000","last_clean":"2024-04-25T01:53:24.971471+0000","last_became_active":"2024-04-25T01:52:25.533484+0000","last_became_peered":"2024-04-25T01:52:25.533484+0000","last_unstale":"2024-04-25T01:53:24.971471+0000","last_undegraded":"2024-04-25T01:53:24.971471+0000","last_fullsized":"2024-04-25T01:53:24.971471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:13.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:37.869822+0000","last_change":"2024-04-25T01:52:24.506068+0000","last_active":"2024-04-25T01:53:37.869822+0000","last_peered":"2024-04-25T01:53:37.869822+0000","last_clean":"2024-04-25T01:53:37.869822+0000","last_became_active":"2024-04-25T01:52:24.505667+0000","last_became_peered":"2024-04-25T01:52:24.505667+0000","last_unstale":"2024-04-25T01:53:37.869822+0000","last_undegraded":"2024-04-25T01:53:37.869822+0000","last_fullsized":"2024-04-25T01:53:37.869822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:30.494376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970062+0000","last_change":"2024-04-25T01:52:25.530681+0000","last_active":"2024-04-25T01:53:24.970062+0000","last_peered":"2024-04-25T01:53:24.970062+0000","last_clean":"2024-04-25T01:53:24.970062+0000","last_became_active":"2024-04-25T01:52:25.529035+0000","last_became_peered":"2024-04-25T01:52:25.529035+0000","last_unstale":"2024-04-25T01:53:24.970062+0000","last_undegraded":"2024-04-25T01:53:24.970062+0000","last_fullsized":"2024-04-25T01:53:24.970062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:28.787180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'6","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.359690+0000","last_change":"2024-04-25T01:52:24.508948+0000","last_active":"2024-04-25T01:53:25.359690+0000","last_peered":"2024-04-25T01:53:25.359690+0000","last_clean":"2024-04-25T01:53:25.359690+0000","last_became_active":"2024-04-25T01:52:24.508563+0000","last_became_peered":"2024-04-25T01:52:24.508563+0000","last_unstale":"2024-04-25T01:53:25.359690+0000","last_undegraded":"2024-04-25T01:53:25.359690+0000","last_fullsized":"2024-04-25T01:53:25.359690+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:37.591783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970795+0000","last_change":"2024-04-25T01:51:15.621251+0000","last_active":"2024-04-25T01:53:24.970795+0000","last_peered":"2024-04-25T01:53:24.970795+0000","last_clean":"2024-04-25T01:53:24.970795+0000","last_became_active":"2024-04-25T01:51:15.620771+0000","last_became_peered":"2024-04-25T01:51:15.620771+0000","last_unstale":"2024-04-25T01:53:24.970795+0000","last_undegraded":"2024-04-25T01:53:24.970795+0000","last_fullsized":"2024-04-25T01:53:24.970795+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_clean_scrub_stamp":"2024-04-25T01:47:50.544829+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:44.048065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961446+0000","last_change":"2024-04-25T01:52:25.530216+0000","last_active":"2024-04-25T01:53:24.961446+0000","last_peered":"2024-04-25T01:53:24.961446+0000","last_clean":"2024-04-25T01:53:24.961446+0000","last_became_active":"2024-04-25T01:52:25.529813+0000","last_became_peered":"2024-04-25T01:52:25.529813+0000","last_unstale":"2024-04-25T01:53:24.961446+0000","last_undegraded":"2024-04-25T01:53:24.961446+0000","last_fullsized":"2024-04-25T01:53:24.961446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:57.656375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'4","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.979645+0000","last_change":"2024-04-25T01:53:24.979114+0000","last_active":"2024-04-25T01:53:24.979645+0000","last_peered":"2024-04-25T01:53:24.979645+0000","last_clean":"2024-04-25T01:53:24.979645+0000","last_became_active":"2024-04-25T01:53:24.972591+0000","last_became_peered":"2024-04-25T01:53:24.972591+0000","last_unstale":"2024-04-25T01:53:24.979645+0000","last_undegraded":"2024-04-25T01:53:24.979645+0000","last_fullsized":"2024-04-25T01:53:24.979645+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:22.792173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965734+0000","last_change":"2024-04-25T01:52:24.514786+0000","last_active":"2024-04-25T01:53:24.965734+0000","last_peered":"2024-04-25T01:53:24.965734+0000","last_clean":"2024-04-25T01:53:24.965734+0000","last_became_active":"2024-04-25T01:52:24.514486+0000","last_became_peered":"2024-04-25T01:52:24.514486+0000","last_unstale":"2024-04-25T01:53:24.965734+0000","last_undegraded":"2024-04-25T01:53:24.965734+0000","last_fullsized":"2024-04-25T01:53:24.965734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:42.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961791+0000","last_change":"2024-04-25T01:52:25.526518+0000","last_active":"2024-04-25T01:53:24.961791+0000","last_peered":"2024-04-25T01:53:24.961791+0000","last_clean":"2024-04-25T01:53:24.961791+0000","last_became_active":"2024-04-25T01:52:25.525917+0000","last_became_peered":"2024-04-25T01:52:25.525917+0000","last_unstale":"2024-04-25T01:53:24.961791+0000","last_undegraded":"2024-04-25T01:53:24.961791+0000","last_fullsized":"2024-04-25T01:53:24.961791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:20.776385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'12","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964935+0000","last_change":"2024-04-25T01:52:24.511561+0000","last_active":"2024-04-25T01:53:24.964935+0000","last_peered":"2024-04-25T01:53:24.964935+0000","last_clean":"2024-04-25T01:53:24.964935+0000","last_became_active":"2024-04-25T01:52:24.511165+0000","last_became_peered":"2024-04-25T01:52:24.511165+0000","last_unstale":"2024-04-25T01:53:24.964935+0000","last_undegraded":"2024-04-25T01:53:24.964935+0000","last_fullsized":"2024-04-25T01:53:24.964935+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:27:41.006137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970218+0000","last_change":"2024-04-25T01:52:25.527991+0000","last_active":"2024-04-25T01:53:24.970218+0000","last_peered":"2024-04-25T01:53:24.970218+0000","last_clean":"2024-04-25T01:53:24.970218+0000","last_became_active":"2024-04-25T01:52:25.527677+0000","last_became_peered":"2024-04-25T01:52:25.527677+0000","last_unstale":"2024-04-25T01:53:24.970218+0000","last_undegraded":"2024-04-25T01:53:24.970218+0000","last_fullsized":"2024-04-25T01:53:24.970218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:07:24.698774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.360291+0000","last_change":"2024-04-25T01:52:24.523536+0000","last_active":"2024-04-25T01:53:25.360291+0000","last_peered":"2024-04-25T01:53:25.360291+0000","last_clean":"2024-04-25T01:53:25.360291+0000","last_became_active":"2024-04-25T01:52:24.523233+0000","last_became_peered":"2024-04-25T01:52:24.523233+0000","last_unstale":"2024-04-25T01:53:25.360291+0000","last_undegraded":"2024-04-25T01:53:25.360291+0000","last_fullsized":"2024-04-25T01:53:25.360291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:49:44.444968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296496+0000","last_change":"2024-04-25T01:52:25.527090+0000","last_active":"2024-04-25T01:53:25.296496+0000","last_peered":"2024-04-25T01:53:25.296496+0000","last_clean":"2024-04-25T01:53:25.296496+0000","last_became_active":"2024-04-25T01:52:25.526771+0000","last_became_peered":"2024-04-25T01:52:25.526771+0000","last_unstale":"2024-04-25T01:53:25.296496+0000","last_undegraded":"2024-04-25T01:53:25.296496+0000","last_fullsized":"2024-04-25T01:53:25.296496+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:58.758748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964881+0000","last_change":"2024-04-25T01:52:24.516469+0000","last_active":"2024-04-25T01:53:24.964881+0000","last_peered":"2024-04-25T01:53:24.964881+0000","last_clean":"2024-04-25T01:53:24.964881+0000","last_became_active":"2024-04-25T01:52:24.515948+0000","last_became_peered":"2024-04-25T01:52:24.515948+0000","last_unstale":"2024-04-25T01:53:24.964881+0000","last_undegraded":"2024-04-25T01:53:24.964881+0000","last_fullsized":"2024-04-25T01:53:24.964881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:25.122153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965327+0000","last_change":"2024-04-25T01:52:25.524967+0000","last_active":"2024-04-25T01:53:24.965327+0000","last_peered":"2024-04-25T01:53:24.965327+0000","last_clean":"2024-04-25T01:53:24.965327+0000","last_became_active":"2024-04-25T01:52:25.524670+0000","last_became_peered":"2024-04-25T01:52:25.524670+0000","last_unstale":"2024-04-25T01:53:24.965327+0000","last_undegraded":"2024-04-25T01:53:24.965327+0000","last_fullsized":"2024-04-25T01:53:24.965327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:08.829052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969883+0000","last_change":"2024-04-25T01:52:24.512187+0000","last_active":"2024-04-25T01:53:24.969883+0000","last_peered":"2024-04-25T01:53:24.969883+0000","last_clean":"2024-04-25T01:53:24.969883+0000","last_became_active":"2024-04-25T01:52:24.510303+0000","last_became_peered":"2024-04-25T01:52:24.510303+0000","last_unstale":"2024-04-25T01:53:24.969883+0000","last_undegraded":"2024-04-25T01:53:24.969883+0000","last_fullsized":"2024-04-25T01:53:24.969883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:42:31.596355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969298+0000","last_change":"2024-04-25T01:52:25.530386+0000","last_active":"2024-04-25T01:53:24.969298+0000","last_peered":"2024-04-25T01:53:24.969298+0000","last_clean":"2024-04-25T01:53:24.969298+0000","last_became_active":"2024-04-25T01:52:25.530034+0000","last_became_peered":"2024-04-25T01:52:25.530034+0000","last_unstale":"2024-04-25T01:53:24.969298+0000","last_undegraded":"2024-04-25T01:53:24.969298+0000","last_fullsized":"2024-04-25T01:53:24.969298+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:11.470123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.976907+0000","last_change":"2024-04-25T01:53:24.976373+0000","last_active":"2024-04-25T01:53:24.976907+0000","last_peered":"2024-04-25T01:53:24.976907+0000","last_clean":"2024-04-25T01:53:24.976907+0000","last_became_active":"2024-04-25T01:53:24.971146+0000","last_became_peered":"2024-04-25T01:53:24.971146+0000","last_unstale":"2024-04-25T01:53:24.976907+0000","last_undegraded":"2024-04-25T01:53:24.976907+0000","last_fullsized":"2024-04-25T01:53:24.976907+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:33.079071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970054+0000","last_change":"2024-04-25T01:52:25.525695+0000","last_active":"2024-04-25T01:53:24.970054+0000","last_peered":"2024-04-25T01:53:24.970054+0000","last_clean":"2024-04-25T01:53:24.970054+0000","last_became_active":"2024-04-25T01:52:25.524830+0000","last_became_peered":"2024-04-25T01:52:25.524830+0000","last_unstale":"2024-04-25T01:53:24.970054+0000","last_undegraded":"2024-04-25T01:53:24.970054+0000","last_fullsized":"2024-04-25T01:53:24.970054+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:04:58.449350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965440+0000","last_change":"2024-04-25T01:52:24.519232+0000","last_active":"2024-04-25T01:53:24.965440+0000","last_peered":"2024-04-25T01:53:24.965440+0000","last_clean":"2024-04-25T01:53:24.965440+0000","last_became_active":"2024-04-25T01:52:24.518902+0000","last_became_peered":"2024-04-25T01:52:24.518902+0000","last_unstale":"2024-04-25T01:53:24.965440+0000","last_undegraded":"2024-04-25T01:53:24.965440+0000","last_fullsized":"2024-04-25T01:53:24.965440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.173730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970889+0000","last_change":"2024-04-25T01:52:25.522955+0000","last_active":"2024-04-25T01:53:24.970889+0000","last_peered":"2024-04-25T01:53:24.970889+0000","last_clean":"2024-04-25T01:53:24.970889+0000","last_became_active":"2024-04-25T01:52:25.522626+0000","last_became_peered":"2024-04-25T01:52:25.522626+0000","last_unstale":"2024-04-25T01:53:24.970889+0000","last_undegraded":"2024-04-25T01:53:24.970889+0000","last_fullsized":"2024-04-25T01:53:24.970889+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:47.218461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'6","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:37.871977+0000","last_change":"2024-04-25T01:52:24.516383+0000","last_active":"2024-04-25T01:53:37.871977+0000","last_peered":"2024-04-25T01:53:37.871977+0000","last_clean":"2024-04-25T01:53:37.871977+0000","last_became_active":"2024-04-25T01:52:24.515096+0000","last_became_peered":"2024-04-25T01:52:24.515096+0000","last_unstale":"2024-04-25T01:53:37.871977+0000","last_undegraded":"2024-04-25T01:53:37.871977+0000","last_fullsized":"2024-04-25T01:53:37.871977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:52.658161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971131+0000","last_change":"2024-04-25T01:52:25.524874+0000","last_active":"2024-04-25T01:53:24.971131+0000","last_peered":"2024-04-25T01:53:24.971131+0000","last_clean":"2024-04-25T01:53:24.971131+0000","last_became_active":"2024-04-25T01:52:25.523532+0000","last_became_peered":"2024-04-25T01:52:25.523532+0000","last_unstale":"2024-04-25T01:53:24.971131+0000","last_undegraded":"2024-04-25T01:53:24.971131+0000","last_fullsized":"2024-04-25T01:53:24.971131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:10.746446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971336+0000","last_change":"2024-04-25T01:52:24.526212+0000","last_active":"2024-04-25T01:53:24.971336+0000","last_peered":"2024-04-25T01:53:24.971336+0000","last_clean":"2024-04-25T01:53:24.971336+0000","last_became_active":"2024-04-25T01:52:24.525882+0000","last_became_peered":"2024-04-25T01:52:24.525882+0000","last_unstale":"2024-04-25T01:53:24.971336+0000","last_undegraded":"2024-04-25T01:53:24.971336+0000","last_fullsized":"2024-04-25T01:53:24.971336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:04.792560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970974+0000","last_change":"2024-04-25T01:52:25.517387+0000","last_active":"2024-04-25T01:53:24.970974+0000","last_peered":"2024-04-25T01:53:24.970974+0000","last_clean":"2024-04-25T01:53:24.970974+0000","last_became_active":"2024-04-25T01:52:25.517047+0000","last_became_peered":"2024-04-25T01:52:25.517047+0000","last_unstale":"2024-04-25T01:53:24.970974+0000","last_undegraded":"2024-04-25T01:53:24.970974+0000","last_fullsized":"2024-04-25T01:53:24.970974+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:39.462474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961908+0000","last_change":"2024-04-25T01:52:24.506510+0000","last_active":"2024-04-25T01:53:24.961908+0000","last_peered":"2024-04-25T01:53:24.961908+0000","last_clean":"2024-04-25T01:53:24.961908+0000","last_became_active":"2024-04-25T01:52:24.506184+0000","last_became_peered":"2024-04-25T01:52:24.506184+0000","last_unstale":"2024-04-25T01:53:24.961908+0000","last_undegraded":"2024-04-25T01:53:24.961908+0000","last_fullsized":"2024-04-25T01:53:24.961908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:56.754386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971089+0000","last_change":"2024-04-25T01:52:25.525197+0000","last_active":"2024-04-25T01:53:24.971089+0000","last_peered":"2024-04-25T01:53:24.971089+0000","last_clean":"2024-04-25T01:53:24.971089+0000","last_became_active":"2024-04-25T01:52:25.524692+0000","last_became_peered":"2024-04-25T01:52:25.524692+0000","last_unstale":"2024-04-25T01:53:24.971089+0000","last_undegraded":"2024-04-25T01:53:24.971089+0000","last_fullsized":"2024-04-25T01:53:24.971089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.264227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969334+0000","last_change":"2024-04-25T01:52:25.531060+0000","last_active":"2024-04-25T01:53:24.969334+0000","last_peered":"2024-04-25T01:53:24.969334+0000","last_clean":"2024-04-25T01:53:24.969334+0000","last_became_active":"2024-04-25T01:52:25.530774+0000","last_became_peered":"2024-04-25T01:52:25.530774+0000","last_unstale":"2024-04-25T01:53:24.969334+0000","last_undegraded":"2024-04-25T01:53:24.969334+0000","last_fullsized":"2024-04-25T01:53:24.969334+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:32.168699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965113+0000","last_change":"2024-04-25T01:52:25.535381+0000","last_active":"2024-04-25T01:53:24.965113+0000","last_peered":"2024-04-25T01:53:24.965113+0000","last_clean":"2024-04-25T01:53:24.965113+0000","last_became_active":"2024-04-25T01:52:25.533623+0000","last_became_peered":"2024-04-25T01:52:25.533623+0000","last_unstale":"2024-04-25T01:53:24.965113+0000","last_undegraded":"2024-04-25T01:53:24.965113+0000","last_fullsized":"2024-04-25T01:53:24.965113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:14.032508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965111+0000","last_change":"2024-04-25T01:52:25.528803+0000","last_active":"2024-04-25T01:53:24.965111+0000","last_peered":"2024-04-25T01:53:24.965111+0000","last_clean":"2024-04-25T01:53:24.965111+0000","last_became_active":"2024-04-25T01:52:25.528505+0000","last_became_peered":"2024-04-25T01:52:25.528505+0000","last_unstale":"2024-04-25T01:53:24.965111+0000","last_undegraded":"2024-04-25T01:53:24.965111+0000","last_fullsized":"2024-04-25T01:53:24.965111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:35.811181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296340+0000","last_change":"2024-04-25T01:52:25.523620+0000","last_active":"2024-04-25T01:53:25.296340+0000","last_peered":"2024-04-25T01:53:25.296340+0000","last_clean":"2024-04-25T01:53:25.296340+0000","last_became_active":"2024-04-25T01:52:25.523007+0000","last_became_peered":"2024-04-25T01:52:25.523007+0000","last_unstale":"2024-04-25T01:53:25.296340+0000","last_undegraded":"2024-04-25T01:53:25.296340+0000","last_fullsized":"2024-04-25T01:53:25.296340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:23.587838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961801+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:53:24.961801+0000","last_peered":"2024-04-25T01:53:24.961801+0000","last_clean":"2024-04-25T01:53:24.961801+0000","last_became_active":"2024-04-25T01:52:25.526398+0000","last_became_peered":"2024-04-25T01:52:25.526398+0000","last_unstale":"2024-04-25T01:53:24.961801+0000","last_undegraded":"2024-04-25T01:53:24.961801+0000","last_fullsized":"2024-04-25T01:53:24.961801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:01:37.286676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.968105+0000","last_change":"2024-04-25T01:52:25.524999+0000","last_active":"2024-04-25T01:53:24.968105+0000","last_peered":"2024-04-25T01:53:24.968105+0000","last_clean":"2024-04-25T01:53:24.968105+0000","last_became_active":"2024-04-25T01:52:25.524652+0000","last_became_peered":"2024-04-25T01:52:25.524652+0000","last_unstale":"2024-04-25T01:53:24.968105+0000","last_undegraded":"2024-04-25T01:53:24.968105+0000","last_fullsized":"2024-04-25T01:53:24.968105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:47:37.463701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.966221+0000","last_change":"2024-04-25T01:52:25.532996+0000","last_active":"2024-04-25T01:53:24.966221+0000","last_peered":"2024-04-25T01:53:24.966221+0000","last_clean":"2024-04-25T01:53:24.966221+0000","last_became_active":"2024-04-25T01:52:25.530230+0000","last_became_peered":"2024-04-25T01:52:25.530230+0000","last_unstale":"2024-04-25T01:53:24.966221+0000","last_undegraded":"2024-04-25T01:53:24.966221+0000","last_fullsized":"2024-04-25T01:53:24.966221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:08.996434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970181+0000","last_change":"2024-04-25T01:52:25.523838+0000","last_active":"2024-04-25T01:53:24.970181+0000","last_peered":"2024-04-25T01:53:24.970181+0000","last_clean":"2024-04-25T01:53:24.970181+0000","last_became_active":"2024-04-25T01:52:25.523297+0000","last_became_peered":"2024-04-25T01:52:25.523297+0000","last_unstale":"2024-04-25T01:53:24.970181+0000","last_undegraded":"2024-04-25T01:53:24.970181+0000","last_fullsized":"2024-04-25T01:53:24.970181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:52.938763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971135+0000","last_change":"2024-04-25T01:52:25.519028+0000","last_active":"2024-04-25T01:53:24.971135+0000","last_peered":"2024-04-25T01:53:24.971135+0000","last_clean":"2024-04-25T01:53:24.971135+0000","last_became_active":"2024-04-25T01:52:25.518716+0000","last_became_peered":"2024-04-25T01:52:25.518716+0000","last_unstale":"2024-04-25T01:53:24.971135+0000","last_undegraded":"2024-04-25T01:53:24.971135+0000","last_fullsized":"2024-04-25T01:53:24.971135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:37:31.573403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969714+0000","last_change":"2024-04-25T01:52:25.533894+0000","last_active":"2024-04-25T01:53:24.969714+0000","last_peered":"2024-04-25T01:53:24.969714+0000","last_clean":"2024-04-25T01:53:24.969714+0000","last_became_active":"2024-04-25T01:52:25.533510+0000","last_became_peered":"2024-04-25T01:52:25.533510+0000","last_unstale":"2024-04-25T01:53:24.969714+0000","last_undegraded":"2024-04-25T01:53:24.969714+0000","last_fullsized":"2024-04-25T01:53:24.969714+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:07.128296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965735+0000","last_change":"2024-04-25T01:52:25.524693+0000","last_active":"2024-04-25T01:53:24.965735+0000","last_peered":"2024-04-25T01:53:24.965735+0000","last_clean":"2024-04-25T01:53:24.965735+0000","last_became_active":"2024-04-25T01:52:25.524366+0000","last_became_peered":"2024-04-25T01:52:25.524366+0000","last_unstale":"2024-04-25T01:53:24.965735+0000","last_undegraded":"2024-04-25T01:53:24.965735+0000","last_fullsized":"2024-04-25T01:53:24.965735+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:23.133722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961791+0000","last_change":"2024-04-25T01:52:25.525936+0000","last_active":"2024-04-25T01:53:24.961791+0000","last_peered":"2024-04-25T01:53:24.961791+0000","last_clean":"2024-04-25T01:53:24.961791+0000","last_became_active":"2024-04-25T01:52:25.525586+0000","last_became_peered":"2024-04-25T01:52:25.525586+0000","last_unstale":"2024-04-25T01:53:24.961791+0000","last_undegraded":"2024-04-25T01:53:24.961791+0000","last_fullsized":"2024-04-25T01:53:24.961791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:34.756898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970621+0000","last_change":"2024-04-25T01:53:24.970621+0000","last_active":"2024-04-25T01:53:24.970621+0000","last_peered":"2024-04-25T01:53:24.970621+0000","last_clean":"2024-04-25T01:53:24.970621+0000","last_became_active":"2024-04-25T01:53:24.970184+0000","last_became_peered":"2024-04-25T01:53:24.970184+0000","last_unstale":"2024-04-25T01:53:24.970621+0000","last_undegraded":"2024-04-25T01:53:24.970621+0000","last_fullsized":"2024-04-25T01:53:24.970621+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:52:43.072630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280293,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68250,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":713,"num_read_kb":434,"num_write":238,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":204750,"data_compressed":13299,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318300,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":468243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579938,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32284,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712868,"statfs":{"total":95995035648,"available":95961976832,"internally_reserved":0,"allocated":1126400,"data_stored":1120318,"data_compressed":37503,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743461,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":467782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005100,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":468160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":1060678,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495669,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":467499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":529458,"data_compressed":1851,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":469229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":469023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":467324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920832,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":528797,"data_compressed":9597,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757306,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":1059965,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62798,"data_compressed":1851,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":62137,"data_compressed":9597,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":63173,"data_compressed":1851,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:53:44.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T01:53:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:45.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:45 smithi031 ceph-mon[29204]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:45.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:45 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:45.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:45 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:45.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:45 smithi031 ceph-mon[29204]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:53:45.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:45 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:53:45.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:45 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:53:45.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:45 smithi031 ceph-mon[29204]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:45.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:45 smithi040 ceph-mon[32418]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:45.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:45 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:45.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:45 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:45.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:45 smithi040 ceph-mon[32418]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:53:45.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:45 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:53:45.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:45 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:53:45.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:45 smithi040 ceph-mon[32418]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:45.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:45 smithi134 ceph-mon[33098]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:45.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:45 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:45.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:45 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:45.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:45 smithi134 ceph-mon[33098]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:53:45.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:45 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:53:45.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:45 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:53:45.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:45 smithi134 ceph-mon[33098]: from='mgr.14152 172.21.15.31:0/506058614' entity='mgr.x' 2024-04-25T01:53:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.780+0000 7f422bc99700 1 -- 172.21.15.31:0/3212213171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42241082c0 msgr2=0x7f42241086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.780+0000 7f422bc99700 1 --2- 172.21.15.31:0/3212213171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42241082c0 0x7f42241086a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4220009990 tx=0x7f4220009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.780+0000 7f422bc99700 1 -- 172.21.15.31:0/3212213171 shutdown_connections 2024-04-25T01:53:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.780+0000 7f422bc99700 1 --2- 172.21.15.31:0/3212213171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42241011c0 0x7f422410f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.780+0000 7f422bc99700 1 --2- 172.21.15.31:0/3212213171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42241082c0 0x7f42241086a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.780+0000 7f422bc99700 1 --2- 172.21.15.31:0/3212213171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4224100820 0x7f4224100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.780+0000 7f422bc99700 1 -- 172.21.15.31:0/3212213171 >> 172.21.15.31:0/3212213171 conn(0x7f42240777a0 msgr2=0x7f42240feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.780+0000 7f422bc99700 1 -- 172.21.15.31:0/3212213171 shutdown_connections 2024-04-25T01:53:46.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.780+0000 7f422bc99700 1 -- 172.21.15.31:0/3212213171 wait complete. 2024-04-25T01:53:46.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.780+0000 7f422bc99700 1 Processor -- start 2024-04-25T01:53:46.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f422bc99700 1 -- start start 2024-04-25T01:53:46.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f422bc99700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4224100820 0x7f4224101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:46.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f422bc99700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42241011c0 0x7f4224101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:46.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f422bc99700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f42241082c0 0x7f4224102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:46.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f422bc99700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f42241123b0 con 0x7f4224100820 2024-04-25T01:53:46.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f422bc99700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4224112230 con 0x7f42241011c0 2024-04-25T01:53:46.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f422bc99700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4224112530 con 0x7f42241082c0 2024-04-25T01:53:46.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f4229a35700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4224100820 0x7f4224101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:46.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f4229a35700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4224100820 0x7f4224101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39850/0 (socket says 172.21.15.31:39850) 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f4229a35700 1 -- 172.21.15.31:0/4268642995 learned_addr learned my addr 172.21.15.31:0/4268642995 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f422a236700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f42241082c0 0x7f4224102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f4229234700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42241011c0 0x7f4224101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f4229a35700 1 -- 172.21.15.31:0/4268642995 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f42241082c0 msgr2=0x7f4224102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f4229a35700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f42241082c0 0x7f4224102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f4229a35700 1 -- 172.21.15.31:0/4268642995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42241011c0 msgr2=0x7f4224101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f4229a35700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42241011c0 0x7f4224101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f4229a35700 1 -- 172.21.15.31:0/4268642995 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42241a6d90 con 0x7f4224100820 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.781+0000 7f4229a35700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4224100820 0x7f4224101a00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4220003ab0 tx=0x7f4220003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.782+0000 7f421affd700 1 -- 172.21.15.31:0/4268642995 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4220027070 con 0x7f4224100820 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.782+0000 7f421affd700 1 -- 172.21.15.31:0/4268642995 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f422000bdc0 con 0x7f4224100820 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.782+0000 7f422bc99700 1 -- 172.21.15.31:0/4268642995 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42241a7020 con 0x7f4224100820 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.782+0000 7f421affd700 1 -- 172.21.15.31:0/4268642995 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4220022bf0 con 0x7f4224100820 2024-04-25T01:53:46.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.782+0000 7f422bc99700 1 -- 172.21.15.31:0/4268642995 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42241a7500 con 0x7f4224100820 2024-04-25T01:53:46.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.783+0000 7f422bc99700 1 -- 172.21.15.31:0/4268642995 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4224103b40 con 0x7f4224100820 2024-04-25T01:53:46.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.785+0000 7f421affd700 1 -- 172.21.15.31:0/4268642995 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f422002a070 con 0x7f4224100820 2024-04-25T01:53:46.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.785+0000 7f421affd700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f42100a2cb0 0x7f42100a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:46.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.785+0000 7f4229234700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f42100a2cb0 0x7f42100a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:46.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.785+0000 7f421affd700 1 -- 172.21.15.31:0/4268642995 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f42200d02d0 con 0x7f4224100820 2024-04-25T01:53:46.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.786+0000 7f4229234700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f42100a2cb0 0x7f42100a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4224107a00 tx=0x7f421400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:46.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.789+0000 7f421affd700 1 -- 172.21.15.31:0/4268642995 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4220031080 con 0x7f4224100820 2024-04-25T01:53:46.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:46.996+0000 7f422bc99700 1 -- 172.21.15.31:0/4268642995 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4224062670 con 0x7f42100a2cb0 2024-04-25T01:53:47.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.003+0000 7f421affd700 1 -- 172.21.15.31:0/4268642995 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258997 (secure 0 0 0) 0x7f4224062670 con 0x7f42100a2cb0 2024-04-25T01:53:47.008 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:53:47.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 -- 172.21.15.31:0/4268642995 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f42100a2cb0 msgr2=0x7f42100a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:47.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f42100a2cb0 0x7f42100a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4224107a00 tx=0x7f421400a380 comp rx=0 tx=0).stop 2024-04-25T01:53:47.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 -- 172.21.15.31:0/4268642995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4224100820 msgr2=0x7f4224101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:47.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4224100820 0x7f4224101a00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4220003ab0 tx=0x7f4220003ae0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 -- 172.21.15.31:0/4268642995 shutdown_connections 2024-04-25T01:53:47.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f42100a2cb0 0x7f42100a5170 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f42241082c0 0x7f4224102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4224100820 0x7f4224101a00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 --2- 172.21.15.31:0/4268642995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42241011c0 0x7f4224101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 -- 172.21.15.31:0/4268642995 >> 172.21.15.31:0/4268642995 conn(0x7f42240777a0 msgr2=0x7f42240fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:47.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 -- 172.21.15.31:0/4268642995 shutdown_connections 2024-04-25T01:53:47.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:47.006+0000 7f422bc99700 1 -- 172.21.15.31:0/4268642995 wait complete. 2024-04-25T01:53:47.011 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T01:53:47.352 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:47 smithi031 ceph-mon[29204]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T01:53:47.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:47 smithi040 ceph-mon[32418]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T01:53:47.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:47 smithi134 ceph-mon[33098]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T01:53:47.684 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-04-25T01:53:45.750666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658852,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":879,"num_read_kb":563,"num_write":421,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1241782,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":346,"ondisk_log_size":346,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369504,"kb_used_data":11948,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572320,"statfs":{"total":1151940427776,"available":1151562055680,"internally_reserved":0,"allocated":12234752,"data_stored":7576476,"data_compressed":120255,"data_compressed_allocated":155648,"data_compressed_original":1843200,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":130,"num_read_kb":60,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002891"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.968589+0000","last_change":"2024-04-25T01:52:25.523199+0000","last_active":"2024-04-25T01:53:24.968589+0000","last_peered":"2024-04-25T01:53:24.968589+0000","last_clean":"2024-04-25T01:53:24.968589+0000","last_became_active":"2024-04-25T01:52:25.522052+0000","last_became_peered":"2024-04-25T01:52:25.522052+0000","last_unstale":"2024-04-25T01:53:24.968589+0000","last_undegraded":"2024-04-25T01:53:24.968589+0000","last_fullsized":"2024-04-25T01:53:24.968589+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:07:27.095370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296619+0000","last_change":"2024-04-25T01:52:25.528431+0000","last_active":"2024-04-25T01:53:25.296619+0000","last_peered":"2024-04-25T01:53:25.296619+0000","last_clean":"2024-04-25T01:53:25.296619+0000","last_became_active":"2024-04-25T01:52:25.528122+0000","last_became_peered":"2024-04-25T01:52:25.528122+0000","last_unstale":"2024-04-25T01:53:25.296619+0000","last_undegraded":"2024-04-25T01:53:25.296619+0000","last_fullsized":"2024-04-25T01:53:25.296619+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:26:56.794582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296554+0000","last_change":"2024-04-25T01:52:25.532963+0000","last_active":"2024-04-25T01:53:25.296554+0000","last_peered":"2024-04-25T01:53:25.296554+0000","last_clean":"2024-04-25T01:53:25.296554+0000","last_became_active":"2024-04-25T01:52:25.532770+0000","last_became_peered":"2024-04-25T01:52:25.532770+0000","last_unstale":"2024-04-25T01:53:25.296554+0000","last_undegraded":"2024-04-25T01:53:25.296554+0000","last_fullsized":"2024-04-25T01:53:25.296554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:13.398868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.968478+0000","last_change":"2024-04-25T01:53:24.968017+0000","last_active":"2024-04-25T01:53:24.968478+0000","last_peered":"2024-04-25T01:53:24.968478+0000","last_clean":"2024-04-25T01:53:24.968478+0000","last_became_active":"2024-04-25T01:53:24.967740+0000","last_became_peered":"2024-04-25T01:53:24.967740+0000","last_unstale":"2024-04-25T01:53:24.968478+0000","last_undegraded":"2024-04-25T01:53:24.968478+0000","last_fullsized":"2024-04-25T01:53:24.968478+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:45.219797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965326+0000","last_change":"2024-04-25T01:52:25.523624+0000","last_active":"2024-04-25T01:53:24.965326+0000","last_peered":"2024-04-25T01:53:24.965326+0000","last_clean":"2024-04-25T01:53:24.965326+0000","last_became_active":"2024-04-25T01:52:25.523289+0000","last_became_peered":"2024-04-25T01:52:25.523289+0000","last_unstale":"2024-04-25T01:53:24.965326+0000","last_undegraded":"2024-04-25T01:53:24.965326+0000","last_fullsized":"2024-04-25T01:53:24.965326+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:09.688472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961924+0000","last_change":"2024-04-25T01:52:25.527123+0000","last_active":"2024-04-25T01:53:24.961924+0000","last_peered":"2024-04-25T01:53:24.961924+0000","last_clean":"2024-04-25T01:53:24.961924+0000","last_became_active":"2024-04-25T01:52:25.526808+0000","last_became_peered":"2024-04-25T01:52:25.526808+0000","last_unstale":"2024-04-25T01:53:24.961924+0000","last_undegraded":"2024-04-25T01:53:24.961924+0000","last_fullsized":"2024-04-25T01:53:24.961924+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:31.654857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970613+0000","last_change":"2024-04-25T01:52:25.517011+0000","last_active":"2024-04-25T01:53:24.970613+0000","last_peered":"2024-04-25T01:53:24.970613+0000","last_clean":"2024-04-25T01:53:24.970613+0000","last_became_active":"2024-04-25T01:52:25.516680+0000","last_became_peered":"2024-04-25T01:52:25.516680+0000","last_unstale":"2024-04-25T01:53:24.970613+0000","last_undegraded":"2024-04-25T01:53:24.970613+0000","last_fullsized":"2024-04-25T01:53:24.970613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:03.388576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969756+0000","last_change":"2024-04-25T01:52:25.531079+0000","last_active":"2024-04-25T01:53:24.969756+0000","last_peered":"2024-04-25T01:53:24.969756+0000","last_clean":"2024-04-25T01:53:24.969756+0000","last_became_active":"2024-04-25T01:52:25.529200+0000","last_became_peered":"2024-04-25T01:52:25.529200+0000","last_unstale":"2024-04-25T01:53:24.969756+0000","last_undegraded":"2024-04-25T01:53:24.969756+0000","last_fullsized":"2024-04-25T01:53:24.969756+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:30.484010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965280+0000","last_change":"2024-04-25T01:52:25.533421+0000","last_active":"2024-04-25T01:53:24.965280+0000","last_peered":"2024-04-25T01:53:24.965280+0000","last_clean":"2024-04-25T01:53:24.965280+0000","last_became_active":"2024-04-25T01:52:25.531781+0000","last_became_peered":"2024-04-25T01:52:25.531781+0000","last_unstale":"2024-04-25T01:53:24.965280+0000","last_undegraded":"2024-04-25T01:53:24.965280+0000","last_fullsized":"2024-04-25T01:53:24.965280+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:32.505793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.962511+0000","last_change":"2024-04-25T01:52:25.530223+0000","last_active":"2024-04-25T01:53:24.962511+0000","last_peered":"2024-04-25T01:53:24.962511+0000","last_clean":"2024-04-25T01:53:24.962511+0000","last_became_active":"2024-04-25T01:52:25.529909+0000","last_became_peered":"2024-04-25T01:52:25.529909+0000","last_unstale":"2024-04-25T01:53:24.962511+0000","last_undegraded":"2024-04-25T01:53:24.962511+0000","last_fullsized":"2024-04-25T01:53:24.962511+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:07.429026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971132+0000","last_change":"2024-04-25T01:52:25.524502+0000","last_active":"2024-04-25T01:53:24.971132+0000","last_peered":"2024-04-25T01:53:24.971132+0000","last_clean":"2024-04-25T01:53:24.971132+0000","last_became_active":"2024-04-25T01:52:25.524018+0000","last_became_peered":"2024-04-25T01:52:25.524018+0000","last_unstale":"2024-04-25T01:53:24.971132+0000","last_undegraded":"2024-04-25T01:53:24.971132+0000","last_fullsized":"2024-04-25T01:53:24.971132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:23.760389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.360412+0000","last_change":"2024-04-25T01:52:25.533593+0000","last_active":"2024-04-25T01:53:25.360412+0000","last_peered":"2024-04-25T01:53:25.360412+0000","last_clean":"2024-04-25T01:53:25.360412+0000","last_became_active":"2024-04-25T01:52:25.532745+0000","last_became_peered":"2024-04-25T01:52:25.532745+0000","last_unstale":"2024-04-25T01:53:25.360412+0000","last_undegraded":"2024-04-25T01:53:25.360412+0000","last_fullsized":"2024-04-25T01:53:25.360412+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:45.083267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961801+0000","last_change":"2024-04-25T01:52:25.528563+0000","last_active":"2024-04-25T01:53:24.961801+0000","last_peered":"2024-04-25T01:53:24.961801+0000","last_clean":"2024-04-25T01:53:24.961801+0000","last_became_active":"2024-04-25T01:52:25.528236+0000","last_became_peered":"2024-04-25T01:52:25.528236+0000","last_unstale":"2024-04-25T01:53:24.961801+0000","last_undegraded":"2024-04-25T01:53:24.961801+0000","last_fullsized":"2024-04-25T01:53:24.961801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:28:27.222799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296333+0000","last_change":"2024-04-25T01:52:25.524951+0000","last_active":"2024-04-25T01:53:25.296333+0000","last_peered":"2024-04-25T01:53:25.296333+0000","last_clean":"2024-04-25T01:53:25.296333+0000","last_became_active":"2024-04-25T01:52:25.524617+0000","last_became_peered":"2024-04-25T01:52:25.524617+0000","last_unstale":"2024-04-25T01:53:25.296333+0000","last_undegraded":"2024-04-25T01:53:25.296333+0000","last_fullsized":"2024-04-25T01:53:25.296333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:03.561883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296227+0000","last_change":"2024-04-25T01:52:25.532471+0000","last_active":"2024-04-25T01:53:25.296227+0000","last_peered":"2024-04-25T01:53:25.296227+0000","last_clean":"2024-04-25T01:53:25.296227+0000","last_became_active":"2024-04-25T01:52:25.532131+0000","last_became_peered":"2024-04-25T01:52:25.532131+0000","last_unstale":"2024-04-25T01:53:25.296227+0000","last_undegraded":"2024-04-25T01:53:25.296227+0000","last_fullsized":"2024-04-25T01:53:25.296227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:03.478816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964769+0000","last_change":"2024-04-25T01:52:25.529160+0000","last_active":"2024-04-25T01:53:24.964769+0000","last_peered":"2024-04-25T01:53:24.964769+0000","last_clean":"2024-04-25T01:53:24.964769+0000","last_became_active":"2024-04-25T01:52:25.528854+0000","last_became_peered":"2024-04-25T01:52:25.528854+0000","last_unstale":"2024-04-25T01:53:24.964769+0000","last_undegraded":"2024-04-25T01:53:24.964769+0000","last_fullsized":"2024-04-25T01:53:24.964769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:54.713715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.968773+0000","last_change":"2024-04-25T01:52:25.523372+0000","last_active":"2024-04-25T01:53:24.968773+0000","last_peered":"2024-04-25T01:53:24.968773+0000","last_clean":"2024-04-25T01:53:24.968773+0000","last_became_active":"2024-04-25T01:52:25.522538+0000","last_became_peered":"2024-04-25T01:52:25.522538+0000","last_unstale":"2024-04-25T01:53:24.968773+0000","last_undegraded":"2024-04-25T01:53:24.968773+0000","last_fullsized":"2024-04-25T01:53:24.968773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:47.213815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964875+0000","last_change":"2024-04-25T01:52:25.522920+0000","last_active":"2024-04-25T01:53:24.964875+0000","last_peered":"2024-04-25T01:53:24.964875+0000","last_clean":"2024-04-25T01:53:24.964875+0000","last_became_active":"2024-04-25T01:52:25.522594+0000","last_became_peered":"2024-04-25T01:52:25.522594+0000","last_unstale":"2024-04-25T01:53:24.964875+0000","last_undegraded":"2024-04-25T01:53:24.964875+0000","last_fullsized":"2024-04-25T01:53:24.964875+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:45.043667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965577+0000","last_change":"2024-04-25T01:52:25.532738+0000","last_active":"2024-04-25T01:53:24.965577+0000","last_peered":"2024-04-25T01:53:24.965577+0000","last_clean":"2024-04-25T01:53:24.965577+0000","last_became_active":"2024-04-25T01:52:25.532237+0000","last_became_peered":"2024-04-25T01:52:25.532237+0000","last_unstale":"2024-04-25T01:53:24.965577+0000","last_undegraded":"2024-04-25T01:53:24.965577+0000","last_fullsized":"2024-04-25T01:53:24.965577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:54.456704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'13","reported_seq":106,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.877625+0000","last_change":"2024-04-25T01:52:24.515686+0000","last_active":"2024-04-25T01:53:43.877625+0000","last_peered":"2024-04-25T01:53:43.877625+0000","last_clean":"2024-04-25T01:53:43.877625+0000","last_became_active":"2024-04-25T01:52:24.515246+0000","last_became_peered":"2024-04-25T01:52:24.515246+0000","last_unstale":"2024-04-25T01:53:43.877625+0000","last_undegraded":"2024-04-25T01:53:43.877625+0000","last_fullsized":"2024-04-25T01:53:43.877625+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:14.325747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970110+0000","last_change":"2024-04-25T01:52:25.528796+0000","last_active":"2024-04-25T01:53:24.970110+0000","last_peered":"2024-04-25T01:53:24.970110+0000","last_clean":"2024-04-25T01:53:24.970110+0000","last_became_active":"2024-04-25T01:52:25.528453+0000","last_became_peered":"2024-04-25T01:52:25.528453+0000","last_unstale":"2024-04-25T01:53:24.970110+0000","last_undegraded":"2024-04-25T01:53:24.970110+0000","last_fullsized":"2024-04-25T01:53:24.970110+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:29.432519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970313+0000","last_change":"2024-04-25T01:52:24.520229+0000","last_active":"2024-04-25T01:53:24.970313+0000","last_peered":"2024-04-25T01:53:24.970313+0000","last_clean":"2024-04-25T01:53:24.970313+0000","last_became_active":"2024-04-25T01:52:24.519932+0000","last_became_peered":"2024-04-25T01:52:24.519932+0000","last_unstale":"2024-04-25T01:53:24.970313+0000","last_undegraded":"2024-04-25T01:53:24.970313+0000","last_fullsized":"2024-04-25T01:53:24.970313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.786949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296681+0000","last_change":"2024-04-25T01:52:25.532822+0000","last_active":"2024-04-25T01:53:25.296681+0000","last_peered":"2024-04-25T01:53:25.296681+0000","last_clean":"2024-04-25T01:53:25.296681+0000","last_became_active":"2024-04-25T01:52:25.532462+0000","last_became_peered":"2024-04-25T01:52:25.532462+0000","last_unstale":"2024-04-25T01:53:25.296681+0000","last_undegraded":"2024-04-25T01:53:25.296681+0000","last_fullsized":"2024-04-25T01:53:25.296681+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:55.569883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965899+0000","last_change":"2024-04-25T01:52:24.512923+0000","last_active":"2024-04-25T01:53:24.965899+0000","last_peered":"2024-04-25T01:53:24.965899+0000","last_clean":"2024-04-25T01:53:24.965899+0000","last_became_active":"2024-04-25T01:52:24.512604+0000","last_became_peered":"2024-04-25T01:52:24.512604+0000","last_unstale":"2024-04-25T01:53:24.965899+0000","last_undegraded":"2024-04-25T01:53:24.965899+0000","last_fullsized":"2024-04-25T01:53:24.965899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:41.640234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964962+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:53:24.964962+0000","last_peered":"2024-04-25T01:53:24.964962+0000","last_clean":"2024-04-25T01:53:24.964962+0000","last_became_active":"2024-04-25T01:52:25.526447+0000","last_became_peered":"2024-04-25T01:52:25.526447+0000","last_unstale":"2024-04-25T01:53:24.964962+0000","last_undegraded":"2024-04-25T01:53:24.964962+0000","last_fullsized":"2024-04-25T01:53:24.964962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:10:18.313830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965805+0000","last_change":"2024-04-25T01:52:24.511565+0000","last_active":"2024-04-25T01:53:24.965805+0000","last_peered":"2024-04-25T01:53:24.965805+0000","last_clean":"2024-04-25T01:53:24.965805+0000","last_became_active":"2024-04-25T01:52:24.511258+0000","last_became_peered":"2024-04-25T01:52:24.511258+0000","last_unstale":"2024-04-25T01:53:24.965805+0000","last_undegraded":"2024-04-25T01:53:24.965805+0000","last_fullsized":"2024-04-25T01:53:24.965805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:38.733338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.361468+0000","last_change":"2024-04-25T01:53:25.360965+0000","last_active":"2024-04-25T01:53:25.361468+0000","last_peered":"2024-04-25T01:53:25.361468+0000","last_clean":"2024-04-25T01:53:25.361468+0000","last_became_active":"2024-04-25T01:53:25.360482+0000","last_became_peered":"2024-04-25T01:53:25.360482+0000","last_unstale":"2024-04-25T01:53:25.361468+0000","last_undegraded":"2024-04-25T01:53:25.361468+0000","last_fullsized":"2024-04-25T01:53:25.361468+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:05:01.122687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":98,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:42.879312+0000","last_change":"2024-04-25T01:52:24.514854+0000","last_active":"2024-04-25T01:53:42.879312+0000","last_peered":"2024-04-25T01:53:42.879312+0000","last_clean":"2024-04-25T01:53:42.879312+0000","last_became_active":"2024-04-25T01:52:24.514548+0000","last_became_peered":"2024-04-25T01:52:24.514548+0000","last_unstale":"2024-04-25T01:53:42.879312+0000","last_undegraded":"2024-04-25T01:53:42.879312+0000","last_fullsized":"2024-04-25T01:53:42.879312+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:38.539202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965307+0000","last_change":"2024-04-25T01:52:25.523888+0000","last_active":"2024-04-25T01:53:24.965307+0000","last_peered":"2024-04-25T01:53:24.965307+0000","last_clean":"2024-04-25T01:53:24.965307+0000","last_became_active":"2024-04-25T01:52:25.523581+0000","last_became_peered":"2024-04-25T01:52:25.523581+0000","last_unstale":"2024-04-25T01:53:24.965307+0000","last_undegraded":"2024-04-25T01:53:24.965307+0000","last_fullsized":"2024-04-25T01:53:24.965307+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:59.521757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.972169+0000","last_change":"2024-04-25T01:53:24.972169+0000","last_active":"2024-04-25T01:53:24.972169+0000","last_peered":"2024-04-25T01:53:24.972169+0000","last_clean":"2024-04-25T01:53:24.972169+0000","last_became_active":"2024-04-25T01:53:24.966425+0000","last_became_peered":"2024-04-25T01:53:24.966425+0000","last_unstale":"2024-04-25T01:53:24.972169+0000","last_undegraded":"2024-04-25T01:53:24.972169+0000","last_fullsized":"2024-04-25T01:53:24.972169+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:31.052476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971398+0000","last_change":"2024-04-25T01:52:25.523381+0000","last_active":"2024-04-25T01:53:24.971398+0000","last_peered":"2024-04-25T01:53:24.971398+0000","last_clean":"2024-04-25T01:53:24.971398+0000","last_became_active":"2024-04-25T01:52:25.522978+0000","last_became_peered":"2024-04-25T01:52:25.522978+0000","last_unstale":"2024-04-25T01:53:24.971398+0000","last_undegraded":"2024-04-25T01:53:24.971398+0000","last_fullsized":"2024-04-25T01:53:24.971398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:45.702788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961591+0000","last_change":"2024-04-25T01:52:24.521280+0000","last_active":"2024-04-25T01:53:24.961591+0000","last_peered":"2024-04-25T01:53:24.961591+0000","last_clean":"2024-04-25T01:53:24.961591+0000","last_became_active":"2024-04-25T01:52:24.520959+0000","last_became_peered":"2024-04-25T01:52:24.520959+0000","last_unstale":"2024-04-25T01:53:24.961591+0000","last_undegraded":"2024-04-25T01:53:24.961591+0000","last_fullsized":"2024-04-25T01:53:24.961591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:55.037678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969675+0000","last_change":"2024-04-25T01:52:25.524331+0000","last_active":"2024-04-25T01:53:24.969675+0000","last_peered":"2024-04-25T01:53:24.969675+0000","last_clean":"2024-04-25T01:53:24.969675+0000","last_became_active":"2024-04-25T01:52:25.523431+0000","last_became_peered":"2024-04-25T01:52:25.523431+0000","last_unstale":"2024-04-25T01:53:24.969675+0000","last_undegraded":"2024-04-25T01:53:24.969675+0000","last_fullsized":"2024-04-25T01:53:24.969675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:55.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970795+0000","last_change":"2024-04-25T01:52:24.515697+0000","last_active":"2024-04-25T01:53:24.970795+0000","last_peered":"2024-04-25T01:53:24.970795+0000","last_clean":"2024-04-25T01:53:24.970795+0000","last_became_active":"2024-04-25T01:52:24.515285+0000","last_became_peered":"2024-04-25T01:52:24.515285+0000","last_unstale":"2024-04-25T01:53:24.970795+0000","last_undegraded":"2024-04-25T01:53:24.970795+0000","last_fullsized":"2024-04-25T01:53:24.970795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:24.249203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964509+0000","last_change":"2024-04-25T01:52:25.522415+0000","last_active":"2024-04-25T01:53:24.964509+0000","last_peered":"2024-04-25T01:53:24.964509+0000","last_clean":"2024-04-25T01:53:24.964509+0000","last_became_active":"2024-04-25T01:52:25.521270+0000","last_became_peered":"2024-04-25T01:52:25.521270+0000","last_unstale":"2024-04-25T01:53:24.964509+0000","last_undegraded":"2024-04-25T01:53:24.964509+0000","last_fullsized":"2024-04-25T01:53:24.964509+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:53.514169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":163,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:41.877082+0000","last_change":"2024-04-25T01:52:24.509525+0000","last_active":"2024-04-25T01:53:41.877082+0000","last_peered":"2024-04-25T01:53:41.877082+0000","last_clean":"2024-04-25T01:53:41.877082+0000","last_became_active":"2024-04-25T01:52:24.509184+0000","last_became_peered":"2024-04-25T01:52:24.509184+0000","last_unstale":"2024-04-25T01:53:41.877082+0000","last_undegraded":"2024-04-25T01:53:41.877082+0000","last_fullsized":"2024-04-25T01:53:41.877082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:16:43.301401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965228+0000","last_change":"2024-04-25T01:52:25.525293+0000","last_active":"2024-04-25T01:53:24.965228+0000","last_peered":"2024-04-25T01:53:24.965228+0000","last_clean":"2024-04-25T01:53:24.965228+0000","last_became_active":"2024-04-25T01:52:25.524982+0000","last_became_peered":"2024-04-25T01:52:25.524982+0000","last_unstale":"2024-04-25T01:53:24.965228+0000","last_undegraded":"2024-04-25T01:53:24.965228+0000","last_fullsized":"2024-04-25T01:53:24.965228+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:22.222898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971062+0000","last_change":"2024-04-25T01:52:24.519834+0000","last_active":"2024-04-25T01:53:24.971062+0000","last_peered":"2024-04-25T01:53:24.971062+0000","last_clean":"2024-04-25T01:53:24.971062+0000","last_became_active":"2024-04-25T01:52:24.519521+0000","last_became_peered":"2024-04-25T01:52:24.519521+0000","last_unstale":"2024-04-25T01:53:24.971062+0000","last_undegraded":"2024-04-25T01:53:24.971062+0000","last_fullsized":"2024-04-25T01:53:24.971062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:02.414527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296642+0000","last_change":"2024-04-25T01:52:25.523809+0000","last_active":"2024-04-25T01:53:25.296642+0000","last_peered":"2024-04-25T01:53:25.296642+0000","last_clean":"2024-04-25T01:53:25.296642+0000","last_became_active":"2024-04-25T01:52:25.523406+0000","last_became_peered":"2024-04-25T01:52:25.523406+0000","last_unstale":"2024-04-25T01:53:25.296642+0000","last_undegraded":"2024-04-25T01:53:25.296642+0000","last_fullsized":"2024-04-25T01:53:25.296642+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:22.169633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971618+0000","last_change":"2024-04-25T01:52:24.510226+0000","last_active":"2024-04-25T01:53:24.971618+0000","last_peered":"2024-04-25T01:53:24.971618+0000","last_clean":"2024-04-25T01:53:24.971618+0000","last_became_active":"2024-04-25T01:52:24.509667+0000","last_became_peered":"2024-04-25T01:52:24.509667+0000","last_unstale":"2024-04-25T01:53:24.971618+0000","last_undegraded":"2024-04-25T01:53:24.971618+0000","last_fullsized":"2024-04-25T01:53:24.971618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:49.180675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965130+0000","last_change":"2024-04-25T01:52:25.531567+0000","last_active":"2024-04-25T01:53:24.965130+0000","last_peered":"2024-04-25T01:53:24.965130+0000","last_clean":"2024-04-25T01:53:24.965130+0000","last_became_active":"2024-04-25T01:52:25.531200+0000","last_became_peered":"2024-04-25T01:52:25.531200+0000","last_unstale":"2024-04-25T01:53:24.965130+0000","last_undegraded":"2024-04-25T01:53:24.965130+0000","last_fullsized":"2024-04-25T01:53:24.965130+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:51:14.336413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":217,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:42.390462+0000","last_change":"2024-04-25T01:53:25.369844+0000","last_active":"2024-04-25T01:53:42.390462+0000","last_peered":"2024-04-25T01:53:42.390462+0000","last_clean":"2024-04-25T01:53:42.390462+0000","last_became_active":"2024-04-25T01:53:25.362162+0000","last_became_peered":"2024-04-25T01:53:25.362162+0000","last_unstale":"2024-04-25T01:53:42.390462+0000","last_undegraded":"2024-04-25T01:53:42.390462+0000","last_fullsized":"2024-04-25T01:53:42.390462+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:27:01.730381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":139,"num_read_kb":129,"num_write":33,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965621+0000","last_change":"2024-04-25T01:52:25.525931+0000","last_active":"2024-04-25T01:53:24.965621+0000","last_peered":"2024-04-25T01:53:24.965621+0000","last_clean":"2024-04-25T01:53:24.965621+0000","last_became_active":"2024-04-25T01:52:25.525460+0000","last_became_peered":"2024-04-25T01:52:25.525460+0000","last_unstale":"2024-04-25T01:53:24.965621+0000","last_undegraded":"2024-04-25T01:53:24.965621+0000","last_fullsized":"2024-04-25T01:53:24.965621+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:46.569587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'9","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965434+0000","last_change":"2024-04-25T01:52:24.505470+0000","last_active":"2024-04-25T01:53:24.965434+0000","last_peered":"2024-04-25T01:53:24.965434+0000","last_clean":"2024-04-25T01:53:24.965434+0000","last_became_active":"2024-04-25T01:52:24.504677+0000","last_became_peered":"2024-04-25T01:52:24.504677+0000","last_unstale":"2024-04-25T01:53:24.965434+0000","last_undegraded":"2024-04-25T01:53:24.965434+0000","last_fullsized":"2024-04-25T01:53:24.965434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:01.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961591+0000","last_change":"2024-04-25T01:52:25.526364+0000","last_active":"2024-04-25T01:53:24.961591+0000","last_peered":"2024-04-25T01:53:24.961591+0000","last_clean":"2024-04-25T01:53:24.961591+0000","last_became_active":"2024-04-25T01:52:25.525376+0000","last_became_peered":"2024-04-25T01:52:25.525376+0000","last_unstale":"2024-04-25T01:53:24.961591+0000","last_undegraded":"2024-04-25T01:53:24.961591+0000","last_fullsized":"2024-04-25T01:53:24.961591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:26.770994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971031+0000","last_change":"2024-04-25T01:52:24.520287+0000","last_active":"2024-04-25T01:53:24.971031+0000","last_peered":"2024-04-25T01:53:24.971031+0000","last_clean":"2024-04-25T01:53:24.971031+0000","last_became_active":"2024-04-25T01:52:24.519967+0000","last_became_peered":"2024-04-25T01:52:24.519967+0000","last_unstale":"2024-04-25T01:53:24.971031+0000","last_undegraded":"2024-04-25T01:53:24.971031+0000","last_fullsized":"2024-04-25T01:53:24.971031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:45.078410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970939+0000","last_change":"2024-04-25T01:52:25.525766+0000","last_active":"2024-04-25T01:53:24.970939+0000","last_peered":"2024-04-25T01:53:24.970939+0000","last_clean":"2024-04-25T01:53:24.970939+0000","last_became_active":"2024-04-25T01:52:25.525350+0000","last_became_peered":"2024-04-25T01:52:25.525350+0000","last_unstale":"2024-04-25T01:53:24.970939+0000","last_undegraded":"2024-04-25T01:53:24.970939+0000","last_fullsized":"2024-04-25T01:53:24.970939+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:05.387502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'6","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:42.876968+0000","last_change":"2024-04-25T01:52:24.513674+0000","last_active":"2024-04-25T01:53:42.876968+0000","last_peered":"2024-04-25T01:53:42.876968+0000","last_clean":"2024-04-25T01:53:42.876968+0000","last_became_active":"2024-04-25T01:52:24.513270+0000","last_became_peered":"2024-04-25T01:52:24.513270+0000","last_unstale":"2024-04-25T01:53:42.876968+0000","last_undegraded":"2024-04-25T01:53:42.876968+0000","last_fullsized":"2024-04-25T01:53:42.876968+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:23.385928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970706+0000","last_change":"2024-04-25T01:52:25.517166+0000","last_active":"2024-04-25T01:53:24.970706+0000","last_peered":"2024-04-25T01:53:24.970706+0000","last_clean":"2024-04-25T01:53:24.970706+0000","last_became_active":"2024-04-25T01:52:25.516853+0000","last_became_peered":"2024-04-25T01:52:25.516853+0000","last_unstale":"2024-04-25T01:53:24.970706+0000","last_undegraded":"2024-04-25T01:53:24.970706+0000","last_fullsized":"2024-04-25T01:53:24.970706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:18.502558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'17","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965158+0000","last_change":"2024-04-25T01:52:24.514695+0000","last_active":"2024-04-25T01:53:24.965158+0000","last_peered":"2024-04-25T01:53:24.965158+0000","last_clean":"2024-04-25T01:53:24.965158+0000","last_became_active":"2024-04-25T01:52:24.514370+0000","last_became_peered":"2024-04-25T01:52:24.514370+0000","last_unstale":"2024-04-25T01:53:24.965158+0000","last_undegraded":"2024-04-25T01:53:24.965158+0000","last_fullsized":"2024-04-25T01:53:24.965158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:28:28.537307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971033+0000","last_change":"2024-04-25T01:52:25.527062+0000","last_active":"2024-04-25T01:53:24.971033+0000","last_peered":"2024-04-25T01:53:24.971033+0000","last_clean":"2024-04-25T01:53:24.971033+0000","last_became_active":"2024-04-25T01:52:25.526757+0000","last_became_peered":"2024-04-25T01:52:25.526757+0000","last_unstale":"2024-04-25T01:53:24.971033+0000","last_undegraded":"2024-04-25T01:53:24.971033+0000","last_fullsized":"2024-04-25T01:53:24.971033+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:36.360030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965888+0000","last_change":"2024-04-25T01:52:24.512235+0000","last_active":"2024-04-25T01:53:24.965888+0000","last_peered":"2024-04-25T01:53:24.965888+0000","last_clean":"2024-04-25T01:53:24.965888+0000","last_became_active":"2024-04-25T01:52:24.511910+0000","last_became_peered":"2024-04-25T01:52:24.511910+0000","last_unstale":"2024-04-25T01:53:24.965888+0000","last_undegraded":"2024-04-25T01:53:24.965888+0000","last_fullsized":"2024-04-25T01:53:24.965888+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:38.435734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.968104+0000","last_change":"2024-04-25T01:52:25.527854+0000","last_active":"2024-04-25T01:53:24.968104+0000","last_peered":"2024-04-25T01:53:24.968104+0000","last_clean":"2024-04-25T01:53:24.968104+0000","last_became_active":"2024-04-25T01:52:25.527481+0000","last_became_peered":"2024-04-25T01:52:25.527481+0000","last_unstale":"2024-04-25T01:53:24.968104+0000","last_undegraded":"2024-04-25T01:53:24.968104+0000","last_fullsized":"2024-04-25T01:53:24.968104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:09:11.281982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964620+0000","last_change":"2024-04-25T01:52:24.513463+0000","last_active":"2024-04-25T01:53:24.964620+0000","last_peered":"2024-04-25T01:53:24.964620+0000","last_clean":"2024-04-25T01:53:24.964620+0000","last_became_active":"2024-04-25T01:52:24.513176+0000","last_became_peered":"2024-04-25T01:52:24.513176+0000","last_unstale":"2024-04-25T01:53:24.964620+0000","last_undegraded":"2024-04-25T01:53:24.964620+0000","last_fullsized":"2024-04-25T01:53:24.964620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:13:33.507559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.962527+0000","last_change":"2024-04-25T01:52:25.528316+0000","last_active":"2024-04-25T01:53:24.962527+0000","last_peered":"2024-04-25T01:53:24.962527+0000","last_clean":"2024-04-25T01:53:24.962527+0000","last_became_active":"2024-04-25T01:52:25.527991+0000","last_became_peered":"2024-04-25T01:52:25.527991+0000","last_unstale":"2024-04-25T01:53:24.962527+0000","last_undegraded":"2024-04-25T01:53:24.962527+0000","last_fullsized":"2024-04-25T01:53:24.962527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:55.871663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'5","reported_seq":98,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.878150+0000","last_change":"2024-04-25T01:52:24.519357+0000","last_active":"2024-04-25T01:53:43.878150+0000","last_peered":"2024-04-25T01:53:43.878150+0000","last_clean":"2024-04-25T01:53:43.878150+0000","last_became_active":"2024-04-25T01:52:24.519025+0000","last_became_peered":"2024-04-25T01:52:24.519025+0000","last_unstale":"2024-04-25T01:53:43.878150+0000","last_undegraded":"2024-04-25T01:53:43.878150+0000","last_fullsized":"2024-04-25T01:53:43.878150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:03.476554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.962232+0000","last_change":"2024-04-25T01:52:25.527922+0000","last_active":"2024-04-25T01:53:24.962232+0000","last_peered":"2024-04-25T01:53:24.962232+0000","last_clean":"2024-04-25T01:53:24.962232+0000","last_became_active":"2024-04-25T01:52:25.527596+0000","last_became_peered":"2024-04-25T01:52:25.527596+0000","last_unstale":"2024-04-25T01:53:24.962232+0000","last_undegraded":"2024-04-25T01:53:24.962232+0000","last_fullsized":"2024-04-25T01:53:24.962232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:56.002208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971471+0000","last_change":"2024-04-25T01:52:25.533800+0000","last_active":"2024-04-25T01:53:24.971471+0000","last_peered":"2024-04-25T01:53:24.971471+0000","last_clean":"2024-04-25T01:53:24.971471+0000","last_became_active":"2024-04-25T01:52:25.533484+0000","last_became_peered":"2024-04-25T01:52:25.533484+0000","last_unstale":"2024-04-25T01:53:24.971471+0000","last_undegraded":"2024-04-25T01:53:24.971471+0000","last_fullsized":"2024-04-25T01:53:24.971471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:13.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":97,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:42.876140+0000","last_change":"2024-04-25T01:52:24.506068+0000","last_active":"2024-04-25T01:53:42.876140+0000","last_peered":"2024-04-25T01:53:42.876140+0000","last_clean":"2024-04-25T01:53:42.876140+0000","last_became_active":"2024-04-25T01:52:24.505667+0000","last_became_peered":"2024-04-25T01:52:24.505667+0000","last_unstale":"2024-04-25T01:53:42.876140+0000","last_undegraded":"2024-04-25T01:53:42.876140+0000","last_fullsized":"2024-04-25T01:53:42.876140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:30.494376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970062+0000","last_change":"2024-04-25T01:52:25.530681+0000","last_active":"2024-04-25T01:53:24.970062+0000","last_peered":"2024-04-25T01:53:24.970062+0000","last_clean":"2024-04-25T01:53:24.970062+0000","last_became_active":"2024-04-25T01:52:25.529035+0000","last_became_peered":"2024-04-25T01:52:25.529035+0000","last_unstale":"2024-04-25T01:53:24.970062+0000","last_undegraded":"2024-04-25T01:53:24.970062+0000","last_fullsized":"2024-04-25T01:53:24.970062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:28.787180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'6","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.359690+0000","last_change":"2024-04-25T01:52:24.508948+0000","last_active":"2024-04-25T01:53:25.359690+0000","last_peered":"2024-04-25T01:53:25.359690+0000","last_clean":"2024-04-25T01:53:25.359690+0000","last_became_active":"2024-04-25T01:52:24.508563+0000","last_became_peered":"2024-04-25T01:52:24.508563+0000","last_unstale":"2024-04-25T01:53:25.359690+0000","last_undegraded":"2024-04-25T01:53:25.359690+0000","last_fullsized":"2024-04-25T01:53:25.359690+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:37.591783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970795+0000","last_change":"2024-04-25T01:51:15.621251+0000","last_active":"2024-04-25T01:53:24.970795+0000","last_peered":"2024-04-25T01:53:24.970795+0000","last_clean":"2024-04-25T01:53:24.970795+0000","last_became_active":"2024-04-25T01:51:15.620771+0000","last_became_peered":"2024-04-25T01:51:15.620771+0000","last_unstale":"2024-04-25T01:53:24.970795+0000","last_undegraded":"2024-04-25T01:53:24.970795+0000","last_fullsized":"2024-04-25T01:53:24.970795+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_clean_scrub_stamp":"2024-04-25T01:47:50.544829+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:44.048065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961446+0000","last_change":"2024-04-25T01:52:25.530216+0000","last_active":"2024-04-25T01:53:24.961446+0000","last_peered":"2024-04-25T01:53:24.961446+0000","last_clean":"2024-04-25T01:53:24.961446+0000","last_became_active":"2024-04-25T01:52:25.529813+0000","last_became_peered":"2024-04-25T01:52:25.529813+0000","last_unstale":"2024-04-25T01:53:24.961446+0000","last_undegraded":"2024-04-25T01:53:24.961446+0000","last_fullsized":"2024-04-25T01:53:24.961446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:57.656375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'4","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.979645+0000","last_change":"2024-04-25T01:53:24.979114+0000","last_active":"2024-04-25T01:53:24.979645+0000","last_peered":"2024-04-25T01:53:24.979645+0000","last_clean":"2024-04-25T01:53:24.979645+0000","last_became_active":"2024-04-25T01:53:24.972591+0000","last_became_peered":"2024-04-25T01:53:24.972591+0000","last_unstale":"2024-04-25T01:53:24.979645+0000","last_undegraded":"2024-04-25T01:53:24.979645+0000","last_fullsized":"2024-04-25T01:53:24.979645+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:22.792173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965734+0000","last_change":"2024-04-25T01:52:24.514786+0000","last_active":"2024-04-25T01:53:24.965734+0000","last_peered":"2024-04-25T01:53:24.965734+0000","last_clean":"2024-04-25T01:53:24.965734+0000","last_became_active":"2024-04-25T01:52:24.514486+0000","last_became_peered":"2024-04-25T01:52:24.514486+0000","last_unstale":"2024-04-25T01:53:24.965734+0000","last_undegraded":"2024-04-25T01:53:24.965734+0000","last_fullsized":"2024-04-25T01:53:24.965734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:42.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961791+0000","last_change":"2024-04-25T01:52:25.526518+0000","last_active":"2024-04-25T01:53:24.961791+0000","last_peered":"2024-04-25T01:53:24.961791+0000","last_clean":"2024-04-25T01:53:24.961791+0000","last_became_active":"2024-04-25T01:52:25.525917+0000","last_became_peered":"2024-04-25T01:52:25.525917+0000","last_unstale":"2024-04-25T01:53:24.961791+0000","last_undegraded":"2024-04-25T01:53:24.961791+0000","last_fullsized":"2024-04-25T01:53:24.961791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:20.776385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'12","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964935+0000","last_change":"2024-04-25T01:52:24.511561+0000","last_active":"2024-04-25T01:53:24.964935+0000","last_peered":"2024-04-25T01:53:24.964935+0000","last_clean":"2024-04-25T01:53:24.964935+0000","last_became_active":"2024-04-25T01:52:24.511165+0000","last_became_peered":"2024-04-25T01:52:24.511165+0000","last_unstale":"2024-04-25T01:53:24.964935+0000","last_undegraded":"2024-04-25T01:53:24.964935+0000","last_fullsized":"2024-04-25T01:53:24.964935+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:27:41.006137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970218+0000","last_change":"2024-04-25T01:52:25.527991+0000","last_active":"2024-04-25T01:53:24.970218+0000","last_peered":"2024-04-25T01:53:24.970218+0000","last_clean":"2024-04-25T01:53:24.970218+0000","last_became_active":"2024-04-25T01:52:25.527677+0000","last_became_peered":"2024-04-25T01:52:25.527677+0000","last_unstale":"2024-04-25T01:53:24.970218+0000","last_undegraded":"2024-04-25T01:53:24.970218+0000","last_fullsized":"2024-04-25T01:53:24.970218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:07:24.698774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.360291+0000","last_change":"2024-04-25T01:52:24.523536+0000","last_active":"2024-04-25T01:53:25.360291+0000","last_peered":"2024-04-25T01:53:25.360291+0000","last_clean":"2024-04-25T01:53:25.360291+0000","last_became_active":"2024-04-25T01:52:24.523233+0000","last_became_peered":"2024-04-25T01:52:24.523233+0000","last_unstale":"2024-04-25T01:53:25.360291+0000","last_undegraded":"2024-04-25T01:53:25.360291+0000","last_fullsized":"2024-04-25T01:53:25.360291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:49:44.444968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296496+0000","last_change":"2024-04-25T01:52:25.527090+0000","last_active":"2024-04-25T01:53:25.296496+0000","last_peered":"2024-04-25T01:53:25.296496+0000","last_clean":"2024-04-25T01:53:25.296496+0000","last_became_active":"2024-04-25T01:52:25.526771+0000","last_became_peered":"2024-04-25T01:52:25.526771+0000","last_unstale":"2024-04-25T01:53:25.296496+0000","last_undegraded":"2024-04-25T01:53:25.296496+0000","last_fullsized":"2024-04-25T01:53:25.296496+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:58.758748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.964881+0000","last_change":"2024-04-25T01:52:24.516469+0000","last_active":"2024-04-25T01:53:24.964881+0000","last_peered":"2024-04-25T01:53:24.964881+0000","last_clean":"2024-04-25T01:53:24.964881+0000","last_became_active":"2024-04-25T01:52:24.515948+0000","last_became_peered":"2024-04-25T01:52:24.515948+0000","last_unstale":"2024-04-25T01:53:24.964881+0000","last_undegraded":"2024-04-25T01:53:24.964881+0000","last_fullsized":"2024-04-25T01:53:24.964881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:25.122153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965327+0000","last_change":"2024-04-25T01:52:25.524967+0000","last_active":"2024-04-25T01:53:24.965327+0000","last_peered":"2024-04-25T01:53:24.965327+0000","last_clean":"2024-04-25T01:53:24.965327+0000","last_became_active":"2024-04-25T01:52:25.524670+0000","last_became_peered":"2024-04-25T01:52:25.524670+0000","last_unstale":"2024-04-25T01:53:24.965327+0000","last_undegraded":"2024-04-25T01:53:24.965327+0000","last_fullsized":"2024-04-25T01:53:24.965327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:08.829052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969883+0000","last_change":"2024-04-25T01:52:24.512187+0000","last_active":"2024-04-25T01:53:24.969883+0000","last_peered":"2024-04-25T01:53:24.969883+0000","last_clean":"2024-04-25T01:53:24.969883+0000","last_became_active":"2024-04-25T01:52:24.510303+0000","last_became_peered":"2024-04-25T01:52:24.510303+0000","last_unstale":"2024-04-25T01:53:24.969883+0000","last_undegraded":"2024-04-25T01:53:24.969883+0000","last_fullsized":"2024-04-25T01:53:24.969883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:42:31.596355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969298+0000","last_change":"2024-04-25T01:52:25.530386+0000","last_active":"2024-04-25T01:53:24.969298+0000","last_peered":"2024-04-25T01:53:24.969298+0000","last_clean":"2024-04-25T01:53:24.969298+0000","last_became_active":"2024-04-25T01:52:25.530034+0000","last_became_peered":"2024-04-25T01:52:25.530034+0000","last_unstale":"2024-04-25T01:53:24.969298+0000","last_undegraded":"2024-04-25T01:53:24.969298+0000","last_fullsized":"2024-04-25T01:53:24.969298+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:11.470123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.976907+0000","last_change":"2024-04-25T01:53:24.976373+0000","last_active":"2024-04-25T01:53:24.976907+0000","last_peered":"2024-04-25T01:53:24.976907+0000","last_clean":"2024-04-25T01:53:24.976907+0000","last_became_active":"2024-04-25T01:53:24.971146+0000","last_became_peered":"2024-04-25T01:53:24.971146+0000","last_unstale":"2024-04-25T01:53:24.976907+0000","last_undegraded":"2024-04-25T01:53:24.976907+0000","last_fullsized":"2024-04-25T01:53:24.976907+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:33.079071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970054+0000","last_change":"2024-04-25T01:52:25.525695+0000","last_active":"2024-04-25T01:53:24.970054+0000","last_peered":"2024-04-25T01:53:24.970054+0000","last_clean":"2024-04-25T01:53:24.970054+0000","last_became_active":"2024-04-25T01:52:25.524830+0000","last_became_peered":"2024-04-25T01:52:25.524830+0000","last_unstale":"2024-04-25T01:53:24.970054+0000","last_undegraded":"2024-04-25T01:53:24.970054+0000","last_fullsized":"2024-04-25T01:53:24.970054+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:04:58.449350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965440+0000","last_change":"2024-04-25T01:52:24.519232+0000","last_active":"2024-04-25T01:53:24.965440+0000","last_peered":"2024-04-25T01:53:24.965440+0000","last_clean":"2024-04-25T01:53:24.965440+0000","last_became_active":"2024-04-25T01:52:24.518902+0000","last_became_peered":"2024-04-25T01:52:24.518902+0000","last_unstale":"2024-04-25T01:53:24.965440+0000","last_undegraded":"2024-04-25T01:53:24.965440+0000","last_fullsized":"2024-04-25T01:53:24.965440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.173730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970889+0000","last_change":"2024-04-25T01:52:25.522955+0000","last_active":"2024-04-25T01:53:24.970889+0000","last_peered":"2024-04-25T01:53:24.970889+0000","last_clean":"2024-04-25T01:53:24.970889+0000","last_became_active":"2024-04-25T01:52:25.522626+0000","last_became_peered":"2024-04-25T01:52:25.522626+0000","last_unstale":"2024-04-25T01:53:24.970889+0000","last_undegraded":"2024-04-25T01:53:24.970889+0000","last_fullsized":"2024-04-25T01:53:24.970889+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:47.218461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'6","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:42.879420+0000","last_change":"2024-04-25T01:52:24.516383+0000","last_active":"2024-04-25T01:53:42.879420+0000","last_peered":"2024-04-25T01:53:42.879420+0000","last_clean":"2024-04-25T01:53:42.879420+0000","last_became_active":"2024-04-25T01:52:24.515096+0000","last_became_peered":"2024-04-25T01:52:24.515096+0000","last_unstale":"2024-04-25T01:53:42.879420+0000","last_undegraded":"2024-04-25T01:53:42.879420+0000","last_fullsized":"2024-04-25T01:53:42.879420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:52.658161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971131+0000","last_change":"2024-04-25T01:52:25.524874+0000","last_active":"2024-04-25T01:53:24.971131+0000","last_peered":"2024-04-25T01:53:24.971131+0000","last_clean":"2024-04-25T01:53:24.971131+0000","last_became_active":"2024-04-25T01:52:25.523532+0000","last_became_peered":"2024-04-25T01:52:25.523532+0000","last_unstale":"2024-04-25T01:53:24.971131+0000","last_undegraded":"2024-04-25T01:53:24.971131+0000","last_fullsized":"2024-04-25T01:53:24.971131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:10.746446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971336+0000","last_change":"2024-04-25T01:52:24.526212+0000","last_active":"2024-04-25T01:53:24.971336+0000","last_peered":"2024-04-25T01:53:24.971336+0000","last_clean":"2024-04-25T01:53:24.971336+0000","last_became_active":"2024-04-25T01:52:24.525882+0000","last_became_peered":"2024-04-25T01:52:24.525882+0000","last_unstale":"2024-04-25T01:53:24.971336+0000","last_undegraded":"2024-04-25T01:53:24.971336+0000","last_fullsized":"2024-04-25T01:53:24.971336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:04.792560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970974+0000","last_change":"2024-04-25T01:52:25.517387+0000","last_active":"2024-04-25T01:53:24.970974+0000","last_peered":"2024-04-25T01:53:24.970974+0000","last_clean":"2024-04-25T01:53:24.970974+0000","last_became_active":"2024-04-25T01:52:25.517047+0000","last_became_peered":"2024-04-25T01:52:25.517047+0000","last_unstale":"2024-04-25T01:53:24.970974+0000","last_undegraded":"2024-04-25T01:53:24.970974+0000","last_fullsized":"2024-04-25T01:53:24.970974+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:39.462474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961908+0000","last_change":"2024-04-25T01:52:24.506510+0000","last_active":"2024-04-25T01:53:24.961908+0000","last_peered":"2024-04-25T01:53:24.961908+0000","last_clean":"2024-04-25T01:53:24.961908+0000","last_became_active":"2024-04-25T01:52:24.506184+0000","last_became_peered":"2024-04-25T01:52:24.506184+0000","last_unstale":"2024-04-25T01:53:24.961908+0000","last_undegraded":"2024-04-25T01:53:24.961908+0000","last_fullsized":"2024-04-25T01:53:24.961908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:56.754386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971089+0000","last_change":"2024-04-25T01:52:25.525197+0000","last_active":"2024-04-25T01:53:24.971089+0000","last_peered":"2024-04-25T01:53:24.971089+0000","last_clean":"2024-04-25T01:53:24.971089+0000","last_became_active":"2024-04-25T01:52:25.524692+0000","last_became_peered":"2024-04-25T01:52:25.524692+0000","last_unstale":"2024-04-25T01:53:24.971089+0000","last_undegraded":"2024-04-25T01:53:24.971089+0000","last_fullsized":"2024-04-25T01:53:24.971089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.264227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969334+0000","last_change":"2024-04-25T01:52:25.531060+0000","last_active":"2024-04-25T01:53:24.969334+0000","last_peered":"2024-04-25T01:53:24.969334+0000","last_clean":"2024-04-25T01:53:24.969334+0000","last_became_active":"2024-04-25T01:52:25.530774+0000","last_became_peered":"2024-04-25T01:52:25.530774+0000","last_unstale":"2024-04-25T01:53:24.969334+0000","last_undegraded":"2024-04-25T01:53:24.969334+0000","last_fullsized":"2024-04-25T01:53:24.969334+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:32.168699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965113+0000","last_change":"2024-04-25T01:52:25.535381+0000","last_active":"2024-04-25T01:53:24.965113+0000","last_peered":"2024-04-25T01:53:24.965113+0000","last_clean":"2024-04-25T01:53:24.965113+0000","last_became_active":"2024-04-25T01:52:25.533623+0000","last_became_peered":"2024-04-25T01:52:25.533623+0000","last_unstale":"2024-04-25T01:53:24.965113+0000","last_undegraded":"2024-04-25T01:53:24.965113+0000","last_fullsized":"2024-04-25T01:53:24.965113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:14.032508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965111+0000","last_change":"2024-04-25T01:52:25.528803+0000","last_active":"2024-04-25T01:53:24.965111+0000","last_peered":"2024-04-25T01:53:24.965111+0000","last_clean":"2024-04-25T01:53:24.965111+0000","last_became_active":"2024-04-25T01:52:25.528505+0000","last_became_peered":"2024-04-25T01:52:25.528505+0000","last_unstale":"2024-04-25T01:53:24.965111+0000","last_undegraded":"2024-04-25T01:53:24.965111+0000","last_fullsized":"2024-04-25T01:53:24.965111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:35.811181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:25.296340+0000","last_change":"2024-04-25T01:52:25.523620+0000","last_active":"2024-04-25T01:53:25.296340+0000","last_peered":"2024-04-25T01:53:25.296340+0000","last_clean":"2024-04-25T01:53:25.296340+0000","last_became_active":"2024-04-25T01:52:25.523007+0000","last_became_peered":"2024-04-25T01:52:25.523007+0000","last_unstale":"2024-04-25T01:53:25.296340+0000","last_undegraded":"2024-04-25T01:53:25.296340+0000","last_fullsized":"2024-04-25T01:53:25.296340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:23.587838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961801+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:53:24.961801+0000","last_peered":"2024-04-25T01:53:24.961801+0000","last_clean":"2024-04-25T01:53:24.961801+0000","last_became_active":"2024-04-25T01:52:25.526398+0000","last_became_peered":"2024-04-25T01:52:25.526398+0000","last_unstale":"2024-04-25T01:53:24.961801+0000","last_undegraded":"2024-04-25T01:53:24.961801+0000","last_fullsized":"2024-04-25T01:53:24.961801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:01:37.286676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.968105+0000","last_change":"2024-04-25T01:52:25.524999+0000","last_active":"2024-04-25T01:53:24.968105+0000","last_peered":"2024-04-25T01:53:24.968105+0000","last_clean":"2024-04-25T01:53:24.968105+0000","last_became_active":"2024-04-25T01:52:25.524652+0000","last_became_peered":"2024-04-25T01:52:25.524652+0000","last_unstale":"2024-04-25T01:53:24.968105+0000","last_undegraded":"2024-04-25T01:53:24.968105+0000","last_fullsized":"2024-04-25T01:53:24.968105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:47:37.463701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.966221+0000","last_change":"2024-04-25T01:52:25.532996+0000","last_active":"2024-04-25T01:53:24.966221+0000","last_peered":"2024-04-25T01:53:24.966221+0000","last_clean":"2024-04-25T01:53:24.966221+0000","last_became_active":"2024-04-25T01:52:25.530230+0000","last_became_peered":"2024-04-25T01:52:25.530230+0000","last_unstale":"2024-04-25T01:53:24.966221+0000","last_undegraded":"2024-04-25T01:53:24.966221+0000","last_fullsized":"2024-04-25T01:53:24.966221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:08.996434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970181+0000","last_change":"2024-04-25T01:52:25.523838+0000","last_active":"2024-04-25T01:53:24.970181+0000","last_peered":"2024-04-25T01:53:24.970181+0000","last_clean":"2024-04-25T01:53:24.970181+0000","last_became_active":"2024-04-25T01:52:25.523297+0000","last_became_peered":"2024-04-25T01:52:25.523297+0000","last_unstale":"2024-04-25T01:53:24.970181+0000","last_undegraded":"2024-04-25T01:53:24.970181+0000","last_fullsized":"2024-04-25T01:53:24.970181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:52.938763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.971135+0000","last_change":"2024-04-25T01:52:25.519028+0000","last_active":"2024-04-25T01:53:24.971135+0000","last_peered":"2024-04-25T01:53:24.971135+0000","last_clean":"2024-04-25T01:53:24.971135+0000","last_became_active":"2024-04-25T01:52:25.518716+0000","last_became_peered":"2024-04-25T01:52:25.518716+0000","last_unstale":"2024-04-25T01:53:24.971135+0000","last_undegraded":"2024-04-25T01:53:24.971135+0000","last_fullsized":"2024-04-25T01:53:24.971135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:37:31.573403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.969714+0000","last_change":"2024-04-25T01:52:25.533894+0000","last_active":"2024-04-25T01:53:24.969714+0000","last_peered":"2024-04-25T01:53:24.969714+0000","last_clean":"2024-04-25T01:53:24.969714+0000","last_became_active":"2024-04-25T01:52:25.533510+0000","last_became_peered":"2024-04-25T01:52:25.533510+0000","last_unstale":"2024-04-25T01:53:24.969714+0000","last_undegraded":"2024-04-25T01:53:24.969714+0000","last_fullsized":"2024-04-25T01:53:24.969714+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:07.128296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.965735+0000","last_change":"2024-04-25T01:52:25.524693+0000","last_active":"2024-04-25T01:53:24.965735+0000","last_peered":"2024-04-25T01:53:24.965735+0000","last_clean":"2024-04-25T01:53:24.965735+0000","last_became_active":"2024-04-25T01:52:25.524366+0000","last_became_peered":"2024-04-25T01:52:25.524366+0000","last_unstale":"2024-04-25T01:53:24.965735+0000","last_undegraded":"2024-04-25T01:53:24.965735+0000","last_fullsized":"2024-04-25T01:53:24.965735+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:23.133722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.961791+0000","last_change":"2024-04-25T01:52:25.525936+0000","last_active":"2024-04-25T01:53:24.961791+0000","last_peered":"2024-04-25T01:53:24.961791+0000","last_clean":"2024-04-25T01:53:24.961791+0000","last_became_active":"2024-04-25T01:52:25.525586+0000","last_became_peered":"2024-04-25T01:52:25.525586+0000","last_unstale":"2024-04-25T01:53:24.961791+0000","last_undegraded":"2024-04-25T01:53:24.961791+0000","last_fullsized":"2024-04-25T01:53:24.961791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:34.756898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:24.970621+0000","last_change":"2024-04-25T01:53:24.970621+0000","last_active":"2024-04-25T01:53:24.970621+0000","last_peered":"2024-04-25T01:53:24.970621+0000","last_clean":"2024-04-25T01:53:24.970621+0000","last_became_active":"2024-04-25T01:53:24.970184+0000","last_became_peered":"2024-04-25T01:53:24.970184+0000","last_unstale":"2024-04-25T01:53:24.970621+0000","last_undegraded":"2024-04-25T01:53:24.970621+0000","last_fullsized":"2024-04-25T01:53:24.970621+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:52:43.072630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280293,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68250,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":743,"num_read_kb":444,"num_write":238,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":204750,"data_compressed":13299,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318300,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":468243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579938,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32284,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712868,"statfs":{"total":95995035648,"available":95961976832,"internally_reserved":0,"allocated":1126400,"data_stored":1120318,"data_compressed":37503,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743462,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":467782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005101,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":468160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299442,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":1060678,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495670,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":467499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":529458,"data_compressed":1851,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":469229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":469023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":467324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920833,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":528797,"data_compressed":9597,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757307,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":1059965,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34011,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62798,"data_compressed":1851,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":62137,"data_compressed":9597,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":63173,"data_compressed":1851,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:53:47.688 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T01:53:47.688 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T01:53:47.688 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T01:53:47.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T01:53:47.950 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:48.217 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:48 smithi031 ceph-mon[29204]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:53:48.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:48 smithi040 ceph-mon[32418]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:53:48.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:48 smithi134 ceph-mon[33098]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:53:49.383 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:49 smithi031 ceph-mon[29204]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:49.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:49 smithi040 ceph-mon[32418]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:49.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:49 smithi134 ceph-mon[33098]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:49.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.617+0000 7f39a637f700 1 -- 172.21.15.31:0/3596497917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a0100e50 msgr2=0x7f39a0101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:49.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.617+0000 7f39a637f700 1 --2- 172.21.15.31:0/3596497917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a0100e50 0x7f39a0101230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f39940099c0 tx=0x7f3994009cd0 comp rx=0 tx=0).stop 2024-04-25T01:53:49.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.617+0000 7f39a637f700 1 -- 172.21.15.31:0/3596497917 shutdown_connections 2024-04-25T01:53:49.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.617+0000 7f39a637f700 1 --2- 172.21.15.31:0/3596497917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0107670 0x7f39a010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:49.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.617+0000 7f39a637f700 1 --2- 172.21.15.31:0/3596497917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a0100e50 0x7f39a0101230 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:49.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.617+0000 7f39a637f700 1 --2- 172.21.15.31:0/3596497917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39a0101800 0x7f39a0106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:49.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.617+0000 7f39a637f700 1 -- 172.21.15.31:0/3596497917 >> 172.21.15.31:0/3596497917 conn(0x7f39a0076450 msgr2=0x7f39a0076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:49.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.617+0000 7f39a637f700 1 -- 172.21.15.31:0/3596497917 shutdown_connections 2024-04-25T01:53:49.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.617+0000 7f39a637f700 1 -- 172.21.15.31:0/3596497917 wait complete. 2024-04-25T01:53:49.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f39a637f700 1 Processor -- start 2024-04-25T01:53:49.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f39a637f700 1 -- start start 2024-04-25T01:53:49.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f39a637f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a0100e50 0x7f39a01a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:49.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f39a637f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0101800 0x7f39a01a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:49.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f39a637f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39a0107670 0x7f39a019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:49.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f39a637f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39a01124f0 con 0x7f39a0100e50 2024-04-25T01:53:49.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f39a637f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f39a0112370 con 0x7f39a0101800 2024-04-25T01:53:49.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f39a637f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f39a0112670 con 0x7f39a0107670 2024-04-25T01:53:49.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f399ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a0100e50 0x7f39a01a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:49.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f399ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a0100e50 0x7f39a01a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39880/0 (socket says 172.21.15.31:39880) 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f399ffff700 1 -- 172.21.15.31:0/175524369 learned_addr learned my addr 172.21.15.31:0/175524369 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.618+0000 7f39a491c700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39a0107670 0x7f39a019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f399f7fe700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0101800 0x7f39a01a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f399ffff700 1 -- 172.21.15.31:0/175524369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39a0107670 msgr2=0x7f39a019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f399ffff700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39a0107670 0x7f39a019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f399ffff700 1 -- 172.21.15.31:0/175524369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0101800 msgr2=0x7f39a01a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f399ffff700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0101800 0x7f39a01a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f399ffff700 1 -- 172.21.15.31:0/175524369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a019b0d0 con 0x7f39a0100e50 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f399f7fe700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0101800 0x7f39a01a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f39a491c700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39a0107670 0x7f39a019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f399ffff700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a0100e50 0x7f39a01a07b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3994009990 tx=0x7f3994009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f399d7fa700 1 -- 172.21.15.31:0/175524369 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994027070 con 0x7f39a0100e50 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f399d7fa700 1 -- 172.21.15.31:0/175524369 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3994003fd0 con 0x7f39a0100e50 2024-04-25T01:53:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f39a637f700 1 -- 172.21.15.31:0/175524369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a019b360 con 0x7f39a0100e50 2024-04-25T01:53:49.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f399d7fa700 1 -- 172.21.15.31:0/175524369 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994022c70 con 0x7f39a0100e50 2024-04-25T01:53:49.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.619+0000 7f39a637f700 1 -- 172.21.15.31:0/175524369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a01a7680 con 0x7f39a0100e50 2024-04-25T01:53:49.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.620+0000 7f39a637f700 1 -- 172.21.15.31:0/175524369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a0068f20 con 0x7f39a0100e50 2024-04-25T01:53:49.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.621+0000 7f399d7fa700 1 -- 172.21.15.31:0/175524369 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f399402a070 con 0x7f39a0100e50 2024-04-25T01:53:49.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.622+0000 7f399d7fa700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39880a2cb0 0x7f39880a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:49.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.622+0000 7f399f7fe700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39880a2cb0 0x7f39880a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:49.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.622+0000 7f399d7fa700 1 -- 172.21.15.31:0/175524369 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f39940d0890 con 0x7f39a0100e50 2024-04-25T01:53:49.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.623+0000 7f399f7fe700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39880a2cb0 0x7f39880a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3990005fd0 tx=0x7f3990005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:49.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.626+0000 7f399d7fa700 1 -- 172.21.15.31:0/175524369 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3994031070 con 0x7f39a0100e50 2024-04-25T01:53:49.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.880+0000 7f39a637f700 1 -- 172.21.15.31:0/175524369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f39a002db30 con 0x7f39a0100e50 2024-04-25T01:53:49.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.880+0000 7f399d7fa700 1 -- 172.21.15.31:0/175524369 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f3994013070 con 0x7f39a0100e50 2024-04-25T01:53:49.881 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:53:49.881 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T01:53:49.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 -- 172.21.15.31:0/175524369 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39880a2cb0 msgr2=0x7f39880a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:49.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39880a2cb0 0x7f39880a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3990005fd0 tx=0x7f3990005ee0 comp rx=0 tx=0).stop 2024-04-25T01:53:49.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 -- 172.21.15.31:0/175524369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a0100e50 msgr2=0x7f39a01a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:49.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a0100e50 0x7f39a01a07b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3994009990 tx=0x7f3994009f80 comp rx=0 tx=0).stop 2024-04-25T01:53:49.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 -- 172.21.15.31:0/175524369 shutdown_connections 2024-04-25T01:53:49.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f39880a2cb0 0x7f39880a5170 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:49.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39a0107670 0x7f39a019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:49.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a0100e50 0x7f39a01a07b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:49.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 --2- 172.21.15.31:0/175524369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39a0101800 0x7f39a01a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:49.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 -- 172.21.15.31:0/175524369 >> 172.21.15.31:0/175524369 conn(0x7f39a0076450 msgr2=0x7f39a00fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:49.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 -- 172.21.15.31:0/175524369 shutdown_connections 2024-04-25T01:53:49.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:49.882+0000 7f39a637f700 1 -- 172.21.15.31:0/175524369 wait complete. 2024-04-25T01:53:50.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:50 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/175524369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:53:50.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:50 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/175524369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:53:50.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:50 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/175524369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:53:50.523 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T01:53:50.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:53:50.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.649+0000 7f4d57161640 1 -- 172.21.15.31:0/3159950434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d50101870 msgr2=0x7f4d50101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:50.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.649+0000 7f4d57161640 1 --2- 172.21.15.31:0/3159950434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d50101870 0x7f4d50101cf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009860 tx=0x7f4d3c02a340 comp rx=0 tx=0).stop 2024-04-25T01:53:50.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.649+0000 7f4d57161640 1 -- 172.21.15.31:0/3159950434 shutdown_connections 2024-04-25T01:53:50.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.649+0000 7f4d57161640 1 --2- 172.21.15.31:0/3159950434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d50102230 0x7f4d50108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.649+0000 7f4d57161640 1 --2- 172.21.15.31:0/3159950434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d50101870 0x7f4d50101cf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.649+0000 7f4d57161640 1 --2- 172.21.15.31:0/3159950434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d50100680 0x7f4d50100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.649+0000 7f4d57161640 1 -- 172.21.15.31:0/3159950434 >> 172.21.15.31:0/3159950434 conn(0x7f4d500fbbf0 msgr2=0x7f4d500fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:50.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.649+0000 7f4d57161640 1 -- 172.21.15.31:0/3159950434 shutdown_connections 2024-04-25T01:53:50.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.649+0000 7f4d57161640 1 -- 172.21.15.31:0/3159950434 wait complete. 2024-04-25T01:53:50.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d57161640 1 Processor -- start 2024-04-25T01:53:50.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d57161640 1 -- start start 2024-04-25T01:53:50.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d57161640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d50100680 0x7f4d500694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:50.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d57161640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d50101870 0x7f4d5019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:50.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d57161640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d50102230 0x7f4d5019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:50.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d57161640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5010c190 con 0x7f4d50100680 2024-04-25T01:53:50.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d57161640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5010c010 con 0x7f4d50101870 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d57161640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5010c310 con 0x7f4d50102230 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d54efd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d50100680 0x7f4d500694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d54efd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d50100680 0x7f4d500694a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39900/0 (socket says 172.21.15.31:39900) 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d54efd640 1 -- 172.21.15.31:0/578259687 learned_addr learned my addr 172.21.15.31:0/578259687 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d4ffff640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d50101870 0x7f4d5019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.650+0000 7f4d556fe640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d50102230 0x7f4d5019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d54efd640 1 -- 172.21.15.31:0/578259687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d50102230 msgr2=0x7f4d5019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d54efd640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d50102230 0x7f4d5019ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d54efd640 1 -- 172.21.15.31:0/578259687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d50101870 msgr2=0x7f4d5019c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d54efd640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d50101870 0x7f4d5019c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d54efd640 1 -- 172.21.15.31:0/578259687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5019f610 con 0x7f4d50100680 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d556fe640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d50102230 0x7f4d5019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:50.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d54efd640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d50100680 0x7f4d500694a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400d950 tx=0x7f4d4400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:50.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d4dffb640 1 -- 172.21.15.31:0/578259687 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44017020 con 0x7f4d50100680 2024-04-25T01:53:50.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d57161640 1 -- 172.21.15.31:0/578259687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5019f900 con 0x7f4d50100680 2024-04-25T01:53:50.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d4dffb640 1 -- 172.21.15.31:0/578259687 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d44009c30 con 0x7f4d50100680 2024-04-25T01:53:50.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d4dffb640 1 -- 172.21.15.31:0/578259687 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4400f6b0 con 0x7f4d50100680 2024-04-25T01:53:50.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.651+0000 7f4d57161640 1 -- 172.21.15.31:0/578259687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d5019fe20 con 0x7f4d50100680 2024-04-25T01:53:50.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.652+0000 7f4d57161640 1 -- 172.21.15.31:0/578259687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d50107a70 con 0x7f4d50100680 2024-04-25T01:53:50.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.653+0000 7f4d4dffb640 1 -- 172.21.15.31:0/578259687 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4d4401a020 con 0x7f4d50100680 2024-04-25T01:53:50.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.654+0000 7f4d4dffb640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d400a2c20 0x7f4d400a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:50.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.654+0000 7f4d4ffff640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d400a2c20 0x7f4d400a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:50.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.654+0000 7f4d4dffb640 1 -- 172.21.15.31:0/578259687 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f4d440cf260 con 0x7f4d50100680 2024-04-25T01:53:50.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.655+0000 7f4d4ffff640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d400a2c20 0x7f4d400a50e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c000c00 tx=0x7f4d3c030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:50.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.658+0000 7f4d4dffb640 1 -- 172.21.15.31:0/578259687 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4d44009de0 con 0x7f4d50100680 2024-04-25T01:53:50.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.846+0000 7f4d57161640 1 -- 172.21.15.31:0/578259687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d5002d170 con 0x7f4d50100680 2024-04-25T01:53:50.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.847+0000 7f4d4dffb640 1 -- 172.21.15.31:0/578259687 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22478 (secure 0 0 0) 0x7f4d4409c030 con 0x7f4d50100680 2024-04-25T01:53:50.849 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:53:50.849 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:53:24.952281+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:53:50.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.849+0000 7f4d57161640 1 -- 172.21.15.31:0/578259687 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d400a2c20 msgr2=0x7f4d400a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:50.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.849+0000 7f4d57161640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d400a2c20 0x7f4d400a50e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c000c00 tx=0x7f4d3c030470 comp rx=0 tx=0).stop 2024-04-25T01:53:50.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.849+0000 7f4d57161640 1 -- 172.21.15.31:0/578259687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d50100680 msgr2=0x7f4d500694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:50.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.849+0000 7f4d57161640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d50100680 0x7f4d500694a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400d950 tx=0x7f4d4400de10 comp rx=0 tx=0).stop 2024-04-25T01:53:50.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.849+0000 7f4d57161640 1 -- 172.21.15.31:0/578259687 shutdown_connections 2024-04-25T01:53:50.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.849+0000 7f4d57161640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4d400a2c20 0x7f4d400a50e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.849+0000 7f4d57161640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d50102230 0x7f4d5019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.849+0000 7f4d57161640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d50101870 0x7f4d5019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.849+0000 7f4d57161640 1 --2- 172.21.15.31:0/578259687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d50100680 0x7f4d500694a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.849+0000 7f4d57161640 1 -- 172.21.15.31:0/578259687 >> 172.21.15.31:0/578259687 conn(0x7f4d500fbbf0 msgr2=0x7f4d50104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:50.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.850+0000 7f4d57161640 1 -- 172.21.15.31:0/578259687 shutdown_connections 2024-04-25T01:53:50.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.850+0000 7f4d57161640 1 -- 172.21.15.31:0/578259687 wait complete. 2024-04-25T01:53:50.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:23.432566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:24.255373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:53:50.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:53:50.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.978+0000 7f1ffae1c640 1 -- 172.21.15.31:0/1152074938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ff4100680 msgr2=0x7f1ff4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:50.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.978+0000 7f1ffae1c640 1 --2- 172.21.15.31:0/1152074938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ff4100680 0x7f1ff4100aa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009900 tx=0x7f1ff002a410 comp rx=0 tx=0).stop 2024-04-25T01:53:50.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.978+0000 7f1ffae1c640 1 -- 172.21.15.31:0/1152074938 shutdown_connections 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.978+0000 7f1ffae1c640 1 --2- 172.21.15.31:0/1152074938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ff4102230 0x7f1ff4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.978+0000 7f1ffae1c640 1 --2- 172.21.15.31:0/1152074938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff4101870 0x7f1ff4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.978+0000 7f1ffae1c640 1 --2- 172.21.15.31:0/1152074938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ff4100680 0x7f1ff4100aa0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.978+0000 7f1ffae1c640 1 -- 172.21.15.31:0/1152074938 >> 172.21.15.31:0/1152074938 conn(0x7f1ff40fbbf0 msgr2=0x7f1ff40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.978+0000 7f1ffae1c640 1 -- 172.21.15.31:0/1152074938 shutdown_connections 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.978+0000 7f1ffae1c640 1 -- 172.21.15.31:0/1152074938 wait complete. 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1ffae1c640 1 Processor -- start 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1ffae1c640 1 -- start start 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1ffae1c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ff4100680 0x7f1ff4196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1ffae1c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ff4101870 0x7f1ff4196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1ffae1c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff4102230 0x7f1ff419de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:50.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1ffae1c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff410c180 con 0x7f1ff4101870 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1ffae1c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff410c000 con 0x7f1ff4100680 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1ffae1c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff410c300 con 0x7f1ff4102230 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1febfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ff4101870 0x7f1ff4196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1febfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ff4101870 0x7f1ff4196dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39914/0 (socket says 172.21.15.31:39914) 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1febfff640 1 -- 172.21.15.31:0/829429267 learned_addr learned my addr 172.21.15.31:0/829429267 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1ff8bb8640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ff4100680 0x7f1ff4196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1ff93b9640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff4102230 0x7f1ff419de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1febfff640 1 -- 172.21.15.31:0/829429267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff4102230 msgr2=0x7f1ff419de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1febfff640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff4102230 0x7f1ff419de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1febfff640 1 -- 172.21.15.31:0/829429267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ff4100680 msgr2=0x7f1ff4196880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1febfff640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ff4100680 0x7f1ff4196880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1febfff640 1 -- 172.21.15.31:0/829429267 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff419e4a0 con 0x7f1ff4101870 2024-04-25T01:53:50.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1ff93b9640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff4102230 0x7f1ff419de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:50.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.979+0000 7f1febfff640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ff4101870 0x7f1ff4196dc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400c920 tx=0x7f1fe400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:50.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.980+0000 7f1fe9ffb640 1 -- 172.21.15.31:0/829429267 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4007c70 con 0x7f1ff4101870 2024-04-25T01:53:50.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.980+0000 7f1fe9ffb640 1 -- 172.21.15.31:0/829429267 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1fe4007e10 con 0x7f1ff4101870 2024-04-25T01:53:50.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.980+0000 7f1ffae1c640 1 -- 172.21.15.31:0/829429267 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ff419e790 con 0x7f1ff4101870 2024-04-25T01:53:50.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.980+0000 7f1fe9ffb640 1 -- 172.21.15.31:0/829429267 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4018820 con 0x7f1ff4101870 2024-04-25T01:53:50.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.980+0000 7f1ffae1c640 1 -- 172.21.15.31:0/829429267 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff419ece0 con 0x7f1ff4101870 2024-04-25T01:53:50.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.981+0000 7f1ffae1c640 1 -- 172.21.15.31:0/829429267 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ff4107a40 con 0x7f1ff4101870 2024-04-25T01:53:50.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.982+0000 7f1fe9ffb640 1 -- 172.21.15.31:0/829429267 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1fe40189c0 con 0x7f1ff4101870 2024-04-25T01:53:50.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.983+0000 7f1fe9ffb640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1fdc0a2c20 0x7f1fdc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:50.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.983+0000 7f1fe9ffb640 1 -- 172.21.15.31:0/829429267 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f1fe40cf330 con 0x7f1ff4101870 2024-04-25T01:53:50.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.983+0000 7f1ff8bb8640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1fdc0a2c20 0x7f1fdc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:50.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.983+0000 7f1ff8bb8640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1fdc0a2c20 0x7f1fdc0a50e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0004770 tx=0x7f1ff0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:50.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:50.987+0000 7f1fe9ffb640 1 -- 172.21.15.31:0/829429267 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1fe4014030 con 0x7f1ff4101870 2024-04-25T01:53:51.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.203+0000 7f1ffae1c640 1 -- 172.21.15.31:0/829429267 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1ff402d3c0 con 0x7f1ff4101870 2024-04-25T01:53:51.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.203+0000 7f1fe9ffb640 1 -- 172.21.15.31:0/829429267 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f1fe409c100 con 0x7f1ff4101870 2024-04-25T01:53:51.204 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T01:53:51.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.205+0000 7f1ffae1c640 1 -- 172.21.15.31:0/829429267 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1fdc0a2c20 msgr2=0x7f1fdc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.205+0000 7f1ffae1c640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1fdc0a2c20 0x7f1fdc0a50e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0004770 tx=0x7f1ff0000bc0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.205+0000 7f1ffae1c640 1 -- 172.21.15.31:0/829429267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ff4101870 msgr2=0x7f1ff4196dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.205+0000 7f1ffae1c640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ff4101870 0x7f1ff4196dc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400c920 tx=0x7f1fe400cde0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.205+0000 7f1ffae1c640 1 -- 172.21.15.31:0/829429267 shutdown_connections 2024-04-25T01:53:51.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.205+0000 7f1ffae1c640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1fdc0a2c20 0x7f1fdc0a50e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.205+0000 7f1ffae1c640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff4102230 0x7f1ff419de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.205+0000 7f1ffae1c640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ff4101870 0x7f1ff4196dc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.205+0000 7f1ffae1c640 1 --2- 172.21.15.31:0/829429267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ff4100680 0x7f1ff4196880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.205+0000 7f1ffae1c640 1 -- 172.21.15.31:0/829429267 >> 172.21.15.31:0/829429267 conn(0x7f1ff40fbbf0 msgr2=0x7f1ff4104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:51.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.206+0000 7f1ffae1c640 1 -- 172.21.15.31:0/829429267 shutdown_connections 2024-04-25T01:53:51.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.206+0000 7f1ffae1c640 1 -- 172.21.15.31:0/829429267 wait complete. 2024-04-25T01:53:51.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:53:51.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.324+0000 7eff6e4a5640 1 -- 172.21.15.31:0/3894473057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff68104b30 msgr2=0x7eff68106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.324+0000 7eff6e4a5640 1 --2- 172.21.15.31:0/3894473057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff68104b30 0x7eff68106f20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009900 tx=0x7eff5c02a410 comp rx=0 tx=0).stop 2024-04-25T01:53:51.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.325+0000 7eff6e4a5640 1 -- 172.21.15.31:0/3894473057 shutdown_connections 2024-04-25T01:53:51.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.325+0000 7eff6e4a5640 1 --2- 172.21.15.31:0/3894473057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff68107460 0x7eff681098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.325+0000 7eff6e4a5640 1 --2- 172.21.15.31:0/3894473057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff68104b30 0x7eff68106f20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.325+0000 7eff6e4a5640 1 --2- 172.21.15.31:0/3894473057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff68102200 0x7eff681045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.325+0000 7eff6e4a5640 1 -- 172.21.15.31:0/3894473057 >> 172.21.15.31:0/3894473057 conn(0x7eff680fbbf0 msgr2=0x7eff680fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:51.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.325+0000 7eff6e4a5640 1 -- 172.21.15.31:0/3894473057 shutdown_connections 2024-04-25T01:53:51.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.325+0000 7eff6e4a5640 1 -- 172.21.15.31:0/3894473057 wait complete. 2024-04-25T01:53:51.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.325+0000 7eff6e4a5640 1 Processor -- start 2024-04-25T01:53:51.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.325+0000 7eff6e4a5640 1 -- start start 2024-04-25T01:53:51.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff6e4a5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff68102200 0x7eff68197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff6e4a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff68104b30 0x7eff68197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff6e4a5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff68107460 0x7eff6819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff6e4a5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7eff6810d2a0 con 0x7eff68104b30 2024-04-25T01:53:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff6e4a5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7eff6810d120 con 0x7eff68107460 2024-04-25T01:53:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff6e4a5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7eff6810d420 con 0x7eff68102200 2024-04-25T01:53:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff677fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff68104b30 0x7eff68197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff677fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff68104b30 0x7eff68197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39928/0 (socket says 172.21.15.31:39928) 2024-04-25T01:53:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff677fe640 1 -- 172.21.15.31:0/4229508622 learned_addr learned my addr 172.21.15.31:0/4229508622 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff67fff640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff68102200 0x7eff68197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff6ca42640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff68107460 0x7eff6819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff677fe640 1 -- 172.21.15.31:0/4229508622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff68102200 msgr2=0x7eff68197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff677fe640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff68102200 0x7eff68197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff677fe640 1 -- 172.21.15.31:0/4229508622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff68107460 msgr2=0x7eff6819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff677fe640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff68107460 0x7eff6819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff677fe640 1 -- 172.21.15.31:0/4229508622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6819f5a0 con 0x7eff68104b30 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff6ca42640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff68107460 0x7eff6819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.326+0000 7eff677fe640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff68104b30 0x7eff68197ec0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0314e0 tx=0x7eff5c031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.327+0000 7eff657fa640 1 -- 172.21.15.31:0/4229508622 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c03f030 con 0x7eff68104b30 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.327+0000 7eff6e4a5640 1 -- 172.21.15.31:0/4229508622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6819f830 con 0x7eff68104b30 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.327+0000 7eff657fa640 1 -- 172.21.15.31:0/4229508622 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff5c003aa0 con 0x7eff68104b30 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.327+0000 7eff657fa640 1 -- 172.21.15.31:0/4229508622 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c035070 con 0x7eff68104b30 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.327+0000 7eff6e4a5640 1 -- 172.21.15.31:0/4229508622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff6819fd20 con 0x7eff68104b30 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.328+0000 7eff6e4a5640 1 -- 172.21.15.31:0/4229508622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff680fd6e0 con 0x7eff68104b30 2024-04-25T01:53:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.329+0000 7eff657fa640 1 -- 172.21.15.31:0/4229508622 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7eff5c042030 con 0x7eff68104b30 2024-04-25T01:53:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.330+0000 7eff657fa640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7eff500a2c20 0x7eff500a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.330+0000 7eff67fff640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7eff500a2c20 0x7eff500a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.330+0000 7eff657fa640 1 -- 172.21.15.31:0/4229508622 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7eff5c0efbc0 con 0x7eff68104b30 2024-04-25T01:53:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.330+0000 7eff67fff640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7eff500a2c20 0x7eff500a50e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7eff5800abe0 tx=0x7eff5800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:51.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.337+0000 7eff657fa640 1 -- 172.21.15.31:0/4229508622 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7eff5c0bc990 con 0x7eff68104b30 2024-04-25T01:53:51.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:51 smithi040 ceph-mon[32418]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:51.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:51 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/578259687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:51.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:51 smithi134 ceph-mon[33098]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:51.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:51 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/578259687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:51.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:51 smithi031 ceph-mon[29204]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:51.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:51 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/578259687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:51.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.523+0000 7eff6e4a5640 1 -- 172.21.15.31:0/4229508622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7eff6802d3c0 con 0x7eff68104b30 2024-04-25T01:53:51.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.524+0000 7eff657fa640 1 -- 172.21.15.31:0/4229508622 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7eff5c0bc290 con 0x7eff68104b30 2024-04-25T01:53:51.525 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T01:53:51.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 -- 172.21.15.31:0/4229508622 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7eff500a2c20 msgr2=0x7eff500a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7eff500a2c20 0x7eff500a50e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7eff5800abe0 tx=0x7eff5800a2d0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 -- 172.21.15.31:0/4229508622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff68104b30 msgr2=0x7eff68197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff68104b30 0x7eff68197ec0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0314e0 tx=0x7eff5c031a30 comp rx=0 tx=0).stop 2024-04-25T01:53:51.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 -- 172.21.15.31:0/4229508622 shutdown_connections 2024-04-25T01:53:51.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7eff500a2c20 0x7eff500a50e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff68107460 0x7eff6819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff68104b30 0x7eff68197ec0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 --2- 172.21.15.31:0/4229508622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff68102200 0x7eff68197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 -- 172.21.15.31:0/4229508622 >> 172.21.15.31:0/4229508622 conn(0x7eff680fbbf0 msgr2=0x7eff68102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:51.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 -- 172.21.15.31:0/4229508622 shutdown_connections 2024-04-25T01:53:51.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.526+0000 7eff6e4a5640 1 -- 172.21.15.31:0/4229508622 wait complete. 2024-04-25T01:53:51.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:53:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.642+0000 7efddead6640 1 -- 172.21.15.31:0/187981457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd8102230 msgr2=0x7efdd8108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.642+0000 7efddead6640 1 --2- 172.21.15.31:0/187981457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd8102230 0x7efdd8108ac0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009ca0 tx=0x7efdd002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 -- 172.21.15.31:0/187981457 shutdown_connections 2024-04-25T01:53:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 --2- 172.21.15.31:0/187981457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd8102230 0x7efdd8108ac0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 --2- 172.21.15.31:0/187981457 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd8101870 0x7efdd8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 --2- 172.21.15.31:0/187981457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdd8100680 0x7efdd8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 -- 172.21.15.31:0/187981457 >> 172.21.15.31:0/187981457 conn(0x7efdd80fbbf0 msgr2=0x7efdd80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 -- 172.21.15.31:0/187981457 shutdown_connections 2024-04-25T01:53:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 -- 172.21.15.31:0/187981457 wait complete. 2024-04-25T01:53:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 Processor -- start 2024-04-25T01:53:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 -- start start 2024-04-25T01:53:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd8100680 0x7efdd81968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdd8101870 0x7efdd8196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd8102230 0x7efdd819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efdd810c1b0 con 0x7efdd8102230 2024-04-25T01:53:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efdd810c030 con 0x7efdd8101870 2024-04-25T01:53:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.643+0000 7efddead6640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efdd810c330 con 0x7efdd8100680 2024-04-25T01:53:51.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddd073640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd8102230 0x7efdd819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddc872640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd8100680 0x7efdd81968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddc872640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd8100680 0x7efdd81968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:34766/0 (socket says 172.21.15.31:34766) 2024-04-25T01:53:51.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddd073640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd8102230 0x7efdd819de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39954/0 (socket says 172.21.15.31:39954) 2024-04-25T01:53:51.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddc872640 1 -- 172.21.15.31:0/3447033759 learned_addr learned my addr 172.21.15.31:0/3447033759 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:51.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efdd7fff640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdd8101870 0x7efdd8196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddd073640 1 -- 172.21.15.31:0/3447033759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd8100680 msgr2=0x7efdd81968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddd073640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd8100680 0x7efdd81968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddd073640 1 -- 172.21.15.31:0/3447033759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdd8101870 msgr2=0x7efdd8196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddd073640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdd8101870 0x7efdd8196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddd073640 1 -- 172.21.15.31:0/3447033759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd819e4d0 con 0x7efdd8102230 2024-04-25T01:53:51.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddd073640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd8102230 0x7efdd819de60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7efdd0031970 tx=0x7efdd0003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:51.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efdd5ffb640 1 -- 172.21.15.31:0/3447033759 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd0032ec0 con 0x7efdd8102230 2024-04-25T01:53:51.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddead6640 1 -- 172.21.15.31:0/3447033759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd80688d0 con 0x7efdd8102230 2024-04-25T01:53:51.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efdd5ffb640 1 -- 172.21.15.31:0/3447033759 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdd00044f0 con 0x7efdd8102230 2024-04-25T01:53:51.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efdd5ffb640 1 -- 172.21.15.31:0/3447033759 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd0030ce0 con 0x7efdd8102230 2024-04-25T01:53:51.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.644+0000 7efddead6640 1 -- 172.21.15.31:0/3447033759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd8068e70 con 0x7efdd8102230 2024-04-25T01:53:51.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.645+0000 7efddead6640 1 -- 172.21.15.31:0/3447033759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd81074e0 con 0x7efdd8102230 2024-04-25T01:53:51.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.647+0000 7efdd5ffb640 1 -- 172.21.15.31:0/3447033759 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7efdd00046a0 con 0x7efdd8102230 2024-04-25T01:53:51.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.647+0000 7efdd5ffb640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efdc80a4fd0 0x7efdc80a7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.647+0000 7efddc872640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efdc80a4fd0 0x7efdc80a7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.647+0000 7efdd5ffb640 1 -- 172.21.15.31:0/3447033759 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7efdd00f06d0 con 0x7efdd8102230 2024-04-25T01:53:51.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.648+0000 7efddc872640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efdc80a4fd0 0x7efdc80a7490 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efdcc00bc80 tx=0x7efdcc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:51.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.655+0000 7efdd5ffb640 1 -- 172.21.15.31:0/3447033759 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efdd00bd610 con 0x7efdd8102230 2024-04-25T01:53:51.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.841+0000 7efddead6640 1 -- 172.21.15.31:0/3447033759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efdd81076e0 con 0x7efdd8102230 2024-04-25T01:53:51.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.842+0000 7efdd5ffb640 1 -- 172.21.15.31:0/3447033759 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7efdd00bcf10 con 0x7efdd8102230 2024-04-25T01:53:51.843 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T01:53:51.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 -- 172.21.15.31:0/3447033759 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efdc80a4fd0 msgr2=0x7efdc80a7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efdc80a4fd0 0x7efdc80a7490 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efdcc00bc80 tx=0x7efdcc00b410 comp rx=0 tx=0).stop 2024-04-25T01:53:51.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 -- 172.21.15.31:0/3447033759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd8102230 msgr2=0x7efdd819de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd8102230 0x7efdd819de60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7efdd0031970 tx=0x7efdd0003ba0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 -- 172.21.15.31:0/3447033759 shutdown_connections 2024-04-25T01:53:51.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efdc80a4fd0 0x7efdc80a7490 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdd8102230 0x7efdd819de60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdd8101870 0x7efdd8196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 --2- 172.21.15.31:0/3447033759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdd8100680 0x7efdd81968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 -- 172.21.15.31:0/3447033759 >> 172.21.15.31:0/3447033759 conn(0x7efdd80fbbf0 msgr2=0x7efdd8104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:51.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 -- 172.21.15.31:0/3447033759 shutdown_connections 2024-04-25T01:53:51.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.844+0000 7efddead6640 1 -- 172.21.15.31:0/3447033759 wait complete. 2024-04-25T01:53:51.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:53:51.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.957+0000 7fc557412640 1 -- 172.21.15.31:0/3517511983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc550101870 msgr2=0x7fc550101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.957+0000 7fc557412640 1 --2- 172.21.15.31:0/3517511983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc550101870 0x7fc550101cf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc540009900 tx=0x7fc54002a410 comp rx=0 tx=0).stop 2024-04-25T01:53:51.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.957+0000 7fc557412640 1 -- 172.21.15.31:0/3517511983 shutdown_connections 2024-04-25T01:53:51.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.957+0000 7fc557412640 1 --2- 172.21.15.31:0/3517511983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc550102230 0x7fc550108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.957+0000 7fc557412640 1 --2- 172.21.15.31:0/3517511983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc550101870 0x7fc550101cf0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.957+0000 7fc557412640 1 --2- 172.21.15.31:0/3517511983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc550100680 0x7fc550100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.957+0000 7fc557412640 1 -- 172.21.15.31:0/3517511983 >> 172.21.15.31:0/3517511983 conn(0x7fc5500fbc30 msgr2=0x7fc5500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.957+0000 7fc557412640 1 -- 172.21.15.31:0/3517511983 shutdown_connections 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.957+0000 7fc557412640 1 -- 172.21.15.31:0/3517511983 wait complete. 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc557412640 1 Processor -- start 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc557412640 1 -- start start 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc557412640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc550100680 0x7fc550104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc557412640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc550101870 0x7fc550102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc557412640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc550102230 0x7fc550103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc557412640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc55010c1b0 con 0x7fc550101870 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc557412640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc55010c030 con 0x7fc550100680 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc557412640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc55010c330 con 0x7fc550102230 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc5549ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc550101870 0x7fc550102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc5549ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc550101870 0x7fc550102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39980/0 (socket says 172.21.15.31:39980) 2024-04-25T01:53:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc5549ad640 1 -- 172.21.15.31:0/4092084658 learned_addr learned my addr 172.21.15.31:0/4092084658 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc5551ae640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc550100680 0x7fc550104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.958+0000 7fc5559af640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc550102230 0x7fc550103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.959+0000 7fc5549ad640 1 -- 172.21.15.31:0/4092084658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc550102230 msgr2=0x7fc550103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.959+0000 7fc5549ad640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc550102230 0x7fc550103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.959+0000 7fc5549ad640 1 -- 172.21.15.31:0/4092084658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc550100680 msgr2=0x7fc550104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.959+0000 7fc5549ad640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc550100680 0x7fc550104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.959+0000 7fc5549ad640 1 -- 172.21.15.31:0/4092084658 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc550103a20 con 0x7fc550101870 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.959+0000 7fc5549ad640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc550101870 0x7fc550102c50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc540005940 tx=0x7fc5400042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.959+0000 7fc5467fc640 1 -- 172.21.15.31:0/4092084658 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54003f030 con 0x7fc550101870 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.959+0000 7fc557412640 1 -- 172.21.15.31:0/4092084658 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5501a1ac0 con 0x7fc550101870 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.959+0000 7fc5467fc640 1 -- 172.21.15.31:0/4092084658 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc540003cb0 con 0x7fc550101870 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.959+0000 7fc5467fc640 1 -- 172.21.15.31:0/4092084658 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc540035070 con 0x7fc550101870 2024-04-25T01:53:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.959+0000 7fc557412640 1 -- 172.21.15.31:0/4092084658 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5501a2010 con 0x7fc550101870 2024-04-25T01:53:51.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.960+0000 7fc557412640 1 -- 172.21.15.31:0/4092084658 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5501a21e0 con 0x7fc550101870 2024-04-25T01:53:51.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.961+0000 7fc5467fc640 1 -- 172.21.15.31:0/4092084658 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc540042030 con 0x7fc550101870 2024-04-25T01:53:51.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.962+0000 7fc5467fc640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc53c0a2c20 0x7fc53c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.962+0000 7fc5551ae640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc53c0a2c20 0x7fc53c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.962+0000 7fc5467fc640 1 -- 172.21.15.31:0/4092084658 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fc5400ef690 con 0x7fc550101870 2024-04-25T01:53:51.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.962+0000 7fc5551ae640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc53c0a2c20 0x7fc53c0a50e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc54c005fd0 tx=0x7fc54c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:51.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:51.966+0000 7fc5467fc640 1 -- 172.21.15.31:0/4092084658 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc5400bbe70 con 0x7fc550101870 2024-04-25T01:53:52.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.191+0000 7fc557412640 1 -- 172.21.15.31:0/4092084658 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5501a23b0 con 0x7fc550101870 2024-04-25T01:53:52.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.192+0000 7fc5467fc640 1 -- 172.21.15.31:0/4092084658 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11923 (secure 0 0 0) 0x7fc5400c0170 con 0x7fc550101870 2024-04-25T01:53:52.193 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:53:52.193 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:52:32.772296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24487,24473,24503,24463,24451]},"id":1}]} 2024-04-25T01:53:52.193 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 -- 172.21.15.31:0/4092084658 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc53c0a2c20 msgr2=0x7fc53c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc53c0a2c20 0x7fc53c0a50e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc54c005fd0 tx=0x7fc54c00b390 comp rx=0 tx=0).stop 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 -- 172.21.15.31:0/4092084658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc550101870 msgr2=0x7fc550102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc550101870 0x7fc550102c50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc540005940 tx=0x7fc5400042f0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 -- 172.21.15.31:0/4092084658 shutdown_connections 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc53c0a2c20 0x7fc53c0a50e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc550102230 0x7fc550103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc550101870 0x7fc550102c50 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 --2- 172.21.15.31:0/4092084658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc550100680 0x7fc550104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 -- 172.21.15.31:0/4092084658 >> 172.21.15.31:0/4092084658 conn(0x7fc5500fbc30 msgr2=0x7fc5500fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 -- 172.21.15.31:0/4092084658 shutdown_connections 2024-04-25T01:53:52.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.194+0000 7fc557412640 1 -- 172.21.15.31:0/4092084658 wait complete. 2024-04-25T01:53:52.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:53:52.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.307+0000 7f1bf3658640 1 -- 172.21.15.31:0/3246794795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bec1034b0 msgr2=0x7f1bec109d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.307+0000 7f1bf3658640 1 --2- 172.21.15.31:0/3246794795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bec1034b0 0x7f1bec109d40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009cd0 tx=0x7f1be402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.307+0000 7f1bf3658640 1 -- 172.21.15.31:0/3246794795 shutdown_connections 2024-04-25T01:53:52.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.307+0000 7f1bf3658640 1 --2- 172.21.15.31:0/3246794795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bec1034b0 0x7f1bec109d40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.307+0000 7f1bf3658640 1 --2- 172.21.15.31:0/3246794795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bec102af0 0x7f1bec102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.307+0000 7f1bf3658640 1 --2- 172.21.15.31:0/3246794795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bec0ffa80 0x7f1bec0ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.307+0000 7f1bf3658640 1 -- 172.21.15.31:0/3246794795 >> 172.21.15.31:0/3246794795 conn(0x7f1bec0fb3b0 msgr2=0x7f1bec0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.307+0000 7f1bf3658640 1 -- 172.21.15.31:0/3246794795 shutdown_connections 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.307+0000 7f1bf3658640 1 -- 172.21.15.31:0/3246794795 wait complete. 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf3658640 1 Processor -- start 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf3658640 1 -- start start 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf3658640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bec0ffa80 0x7f1bec102460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf3658640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bec102af0 0x7f1bec100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf3658640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bec1034b0 0x7f1bec100fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf3658640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec10d2d0 con 0x7f1bec102af0 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf3658640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec10d150 con 0x7f1bec0ffa80 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf3658640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec10d450 con 0x7f1bec1034b0 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf0bf3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bec102af0 0x7f1bec100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf0bf3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bec102af0 0x7f1bec100aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39990/0 (socket says 172.21.15.31:39990) 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf0bf3640 1 -- 172.21.15.31:0/1858589788 learned_addr learned my addr 172.21.15.31:0/1858589788 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf13f4640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bec0ffa80 0x7f1bec102460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf1bf5640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bec1034b0 0x7f1bec100fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf0bf3640 1 -- 172.21.15.31:0/1858589788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bec1034b0 msgr2=0x7f1bec100fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf0bf3640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bec1034b0 0x7f1bec100fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf0bf3640 1 -- 172.21.15.31:0/1858589788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bec0ffa80 msgr2=0x7f1bec102460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf0bf3640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bec0ffa80 0x7f1bec102460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf0bf3640 1 -- 172.21.15.31:0/1858589788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec101870 con 0x7f1bec102af0 2024-04-25T01:53:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf1bf5640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bec1034b0 0x7f1bec100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.308+0000 7f1bf0bf3640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bec102af0 0x7f1bec100aa0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00c920 tx=0x7f1bdc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.309+0000 7f1be27fc640 1 -- 172.21.15.31:0/1858589788 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc007c70 con 0x7f1bec102af0 2024-04-25T01:53:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.309+0000 7f1be27fc640 1 -- 172.21.15.31:0/1858589788 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1bdc007e10 con 0x7f1bec102af0 2024-04-25T01:53:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.309+0000 7f1bf3658640 1 -- 172.21.15.31:0/1858589788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bec1a1a30 con 0x7f1bec102af0 2024-04-25T01:53:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.309+0000 7f1be27fc640 1 -- 172.21.15.31:0/1858589788 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc018820 con 0x7f1bec102af0 2024-04-25T01:53:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.309+0000 7f1bf3658640 1 -- 172.21.15.31:0/1858589788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bec1a1f80 con 0x7f1bec102af0 2024-04-25T01:53:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.310+0000 7f1bf3658640 1 -- 172.21.15.31:0/1858589788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bec108bc0 con 0x7f1bec102af0 2024-04-25T01:53:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.311+0000 7f1be27fc640 1 -- 172.21.15.31:0/1858589788 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1bdc0189c0 con 0x7f1bec102af0 2024-04-25T01:53:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.311+0000 7f1be27fc640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1bd80a2c20 0x7f1bd80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.312+0000 7f1bf13f4640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1bd80a2c20 0x7f1bd80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.312+0000 7f1be27fc640 1 -- 172.21.15.31:0/1858589788 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f1bdc0cf3d0 con 0x7f1bec102af0 2024-04-25T01:53:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.312+0000 7f1bf13f4640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1bd80a2c20 0x7f1bd80a50e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1be8005fd0 tx=0x7f1be800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:52.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.316+0000 7f1be27fc640 1 -- 172.21.15.31:0/1858589788 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1bdc014030 con 0x7f1bec102af0 2024-04-25T01:53:52.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:52 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/829429267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:53:52.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:52 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/4229508622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:53:52.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:52 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3447033759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:53:52.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:52 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/829429267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:53:52.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:52 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/4229508622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:53:52.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:52 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3447033759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:53:52.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:52 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/829429267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:53:52.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:52 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/4229508622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:53:52.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:52 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3447033759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:53:52.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.504+0000 7f1bf3658640 1 -- 172.21.15.31:0/1858589788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1bec02d3c0 con 0x7f1bec102af0 2024-04-25T01:53:52.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.505+0000 7f1be27fc640 1 -- 172.21.15.31:0/1858589788 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22478 (secure 0 0 0) 0x7f1bdc09c1a0 con 0x7f1bec102af0 2024-04-25T01:53:52.507 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:53:52.507 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:53:24.952281+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:53:52.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 -- 172.21.15.31:0/1858589788 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1bd80a2c20 msgr2=0x7f1bd80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1bd80a2c20 0x7f1bd80a50e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1be8005fd0 tx=0x7f1be800b390 comp rx=0 tx=0).stop 2024-04-25T01:53:52.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 -- 172.21.15.31:0/1858589788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bec102af0 msgr2=0x7f1bec100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bec102af0 0x7f1bec100aa0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00c920 tx=0x7f1bdc00cde0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 -- 172.21.15.31:0/1858589788 shutdown_connections 2024-04-25T01:53:52.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f1bd80a2c20 0x7f1bd80a50e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bec1034b0 0x7f1bec100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bec102af0 0x7f1bec100aa0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 --2- 172.21.15.31:0/1858589788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bec0ffa80 0x7f1bec102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 -- 172.21.15.31:0/1858589788 >> 172.21.15.31:0/1858589788 conn(0x7f1bec0fb3b0 msgr2=0x7f1bec105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:52.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 -- 172.21.15.31:0/1858589788 shutdown_connections 2024-04-25T01:53:52.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.508+0000 7f1bf3658640 1 -- 172.21.15.31:0/1858589788 wait complete. 2024-04-25T01:53:52.520 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:23.432566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:24.255373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:53:52.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:53:52.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.624+0000 7fb1b159e640 1 -- 172.21.15.31:0/2151034777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100890 msgr2=0x7fb1ac100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.624+0000 7fb1b159e640 1 --2- 172.21.15.31:0/2151034777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100890 0x7fb1ac100cf0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb19c009900 tx=0x7fb19c02a410 comp rx=0 tx=0).stop 2024-04-25T01:53:52.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.624+0000 7fb1b159e640 1 -- 172.21.15.31:0/2151034777 shutdown_connections 2024-04-25T01:53:52.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.624+0000 7fb1b159e640 1 --2- 172.21.15.31:0/2151034777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1ac101230 0x7fb1ac1016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.624+0000 7fb1b159e640 1 --2- 172.21.15.31:0/2151034777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100890 0x7fb1ac100cf0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.624+0000 7fb1b159e640 1 --2- 172.21.15.31:0/2151034777 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1ac0fff30 0x7fb1ac100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.624+0000 7fb1b159e640 1 -- 172.21.15.31:0/2151034777 >> 172.21.15.31:0/2151034777 conn(0x7fb1ac0fbaf0 msgr2=0x7fb1ac0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:52.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.624+0000 7fb1b159e640 1 -- 172.21.15.31:0/2151034777 shutdown_connections 2024-04-25T01:53:52.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.624+0000 7fb1b159e640 1 -- 172.21.15.31:0/2151034777 wait complete. 2024-04-25T01:53:52.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.625+0000 7fb1b159e640 1 Processor -- start 2024-04-25T01:53:52.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.625+0000 7fb1b159e640 1 -- start start 2024-04-25T01:53:52.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.625+0000 7fb1b159e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1ac0fff30 0x7fb1ac199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.625+0000 7fb1b159e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100890 0x7fb1ac199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.625+0000 7fb1b159e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1ac101230 0x7fb1ac1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.625+0000 7fb1b159e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac10f370 con 0x7fb1ac100890 2024-04-25T01:53:52.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.625+0000 7fb1b159e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac10f1f0 con 0x7fb1ac0fff30 2024-04-25T01:53:52.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.625+0000 7fb1b159e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac10f4f0 con 0x7fb1ac101230 2024-04-25T01:53:52.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.625+0000 7fb1ab7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100890 0x7fb1ac199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1ab7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100890 0x7fb1ac199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40018/0 (socket says 172.21.15.31:40018) 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1ab7fe640 1 -- 172.21.15.31:0/320687907 learned_addr learned my addr 172.21.15.31:0/320687907 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1abfff640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1ac0fff30 0x7fb1ac199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1b0d9d640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1ac101230 0x7fb1ac1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1ab7fe640 1 -- 172.21.15.31:0/320687907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1ac101230 msgr2=0x7fb1ac1a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1ab7fe640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1ac101230 0x7fb1ac1a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1ab7fe640 1 -- 172.21.15.31:0/320687907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1ac0fff30 msgr2=0x7fb1ac199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1ab7fe640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1ac0fff30 0x7fb1ac199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1ab7fe640 1 -- 172.21.15.31:0/320687907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ac1a1640 con 0x7fb1ac100890 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1ab7fe640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100890 0x7fb1ac199f60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb19c004770 tx=0x7fb19c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1a97fa640 1 -- 172.21.15.31:0/320687907 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c03f030 con 0x7fb1ac100890 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1b159e640 1 -- 172.21.15.31:0/320687907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ac1a18d0 con 0x7fb1ac100890 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1a97fa640 1 -- 172.21.15.31:0/320687907 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb19c003cb0 con 0x7fb1ac100890 2024-04-25T01:53:52.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1a97fa640 1 -- 172.21.15.31:0/320687907 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c035070 con 0x7fb1ac100890 2024-04-25T01:53:52.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.626+0000 7fb1b159e640 1 -- 172.21.15.31:0/320687907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ac1a1dc0 con 0x7fb1ac100890 2024-04-25T01:53:52.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.627+0000 7fb1b159e640 1 -- 172.21.15.31:0/320687907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1ac10ae00 con 0x7fb1ac100890 2024-04-25T01:53:52.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.628+0000 7fb1a97fa640 1 -- 172.21.15.31:0/320687907 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb19c042030 con 0x7fb1ac100890 2024-04-25T01:53:52.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.629+0000 7fb1a97fa640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1980a2c20 0x7fb1980a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.629+0000 7fb1abfff640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1980a2c20 0x7fb1980a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.629+0000 7fb1a97fa640 1 -- 172.21.15.31:0/320687907 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fb19c0efad0 con 0x7fb1ac100890 2024-04-25T01:53:52.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.630+0000 7fb1abfff640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1980a2c20 0x7fb1980a50e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb194005fd0 tx=0x7fb19400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:52.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.634+0000 7fb1a97fa640 1 -- 172.21.15.31:0/320687907 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb19c0bc2b0 con 0x7fb1ac100890 2024-04-25T01:53:52.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.819+0000 7fb1b159e640 1 -- 172.21.15.31:0/320687907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb1ac02d350 con 0x7fb1ac100890 2024-04-25T01:53:52.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.820+0000 7fb1a97fa640 1 -- 172.21.15.31:0/320687907 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fb19c0b6e20 con 0x7fb1ac100890 2024-04-25T01:53:52.820 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T01:53:52.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.821+0000 7fb1b159e640 1 -- 172.21.15.31:0/320687907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1980a2c20 msgr2=0x7fb1980a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.821+0000 7fb1b159e640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1980a2c20 0x7fb1980a50e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb194005fd0 tx=0x7fb19400b390 comp rx=0 tx=0).stop 2024-04-25T01:53:52.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.821+0000 7fb1b159e640 1 -- 172.21.15.31:0/320687907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100890 msgr2=0x7fb1ac199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.821+0000 7fb1b159e640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100890 0x7fb1ac199f60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb19c004770 tx=0x7fb19c0042f0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.821+0000 7fb1b159e640 1 -- 172.21.15.31:0/320687907 shutdown_connections 2024-04-25T01:53:52.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.821+0000 7fb1b159e640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1980a2c20 0x7fb1980a50e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.821+0000 7fb1b159e640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1ac101230 0x7fb1ac1a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.821+0000 7fb1b159e640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100890 0x7fb1ac199f60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.822+0000 7fb1b159e640 1 --2- 172.21.15.31:0/320687907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1ac0fff30 0x7fb1ac199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.822+0000 7fb1b159e640 1 -- 172.21.15.31:0/320687907 >> 172.21.15.31:0/320687907 conn(0x7fb1ac0fbaf0 msgr2=0x7fb1ac0fd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:52.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.822+0000 7fb1b159e640 1 -- 172.21.15.31:0/320687907 shutdown_connections 2024-04-25T01:53:52.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.822+0000 7fb1b159e640 1 -- 172.21.15.31:0/320687907 wait complete. 2024-04-25T01:53:52.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:53:52.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.938+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/973263189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c18104b30 msgr2=0x7f3c18106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.938+0000 7f3c1d1f9640 1 --2- 172.21.15.31:0/973263189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c18104b30 0x7f3c18106f20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009900 tx=0x7f3c0c02a410 comp rx=0 tx=0).stop 2024-04-25T01:53:52.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.938+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/973263189 shutdown_connections 2024-04-25T01:53:52.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.938+0000 7f3c1d1f9640 1 --2- 172.21.15.31:0/973263189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c18107460 0x7f3c181098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.938+0000 7f3c1d1f9640 1 --2- 172.21.15.31:0/973263189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c18104b30 0x7f3c18106f20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.938+0000 7f3c1d1f9640 1 --2- 172.21.15.31:0/973263189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c18102200 0x7f3c181045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.938+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/973263189 >> 172.21.15.31:0/973263189 conn(0x7f3c180fbbf0 msgr2=0x7f3c180fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:52.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.938+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/973263189 shutdown_connections 2024-04-25T01:53:52.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.938+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/973263189 wait complete. 2024-04-25T01:53:52.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1d1f9640 1 Processor -- start 2024-04-25T01:53:52.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1d1f9640 1 -- start start 2024-04-25T01:53:52.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1d1f9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c18102200 0x7f3c18197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1d1f9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c18104b30 0x7f3c18197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1d1f9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c18107460 0x7f3c1819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1d1f9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1810d360 con 0x7f3c18107460 2024-04-25T01:53:52.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1d1f9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1810d1e0 con 0x7f3c18104b30 2024-04-25T01:53:52.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1d1f9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1810d4e0 con 0x7f3c18102200 2024-04-25T01:53:52.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c18107460 0x7f3c1819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c18107460 0x7f3c1819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46148/0 (socket says 172.21.15.31:46148) 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1759e640 1 -- 172.21.15.31:0/1045780223 learned_addr learned my addr 172.21.15.31:0/1045780223 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c1659c640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c18104b30 0x7f3c18197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.939+0000 7f3c16d9d640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c18102200 0x7f3c18197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3c1759e640 1 -- 172.21.15.31:0/1045780223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c18102200 msgr2=0x7f3c18197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3c1759e640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c18102200 0x7f3c18197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3c1759e640 1 -- 172.21.15.31:0/1045780223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c18104b30 msgr2=0x7f3c18197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3c1759e640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c18104b30 0x7f3c18197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3c1759e640 1 -- 172.21.15.31:0/1045780223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1819f680 con 0x7f3c18107460 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3c16d9d640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c18102200 0x7f3c18197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3c1659c640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c18104b30 0x7f3c18197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3c1759e640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c18107460 0x7f3c1819f010 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800c970 tx=0x7f3c0800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3bfffff640 1 -- 172.21.15.31:0/1045780223 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08007bb0 con 0x7f3c18107460 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/1045780223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c1819f970 con 0x7f3c18107460 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3bfffff640 1 -- 172.21.15.31:0/1045780223 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3c08007d50 con 0x7f3c18107460 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3bfffff640 1 -- 172.21.15.31:0/1045780223 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08018700 con 0x7f3c18107460 2024-04-25T01:53:52.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.940+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/1045780223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c1819fec0 con 0x7f3c18107460 2024-04-25T01:53:52.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.941+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/1045780223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c180fd6e0 con 0x7f3c18107460 2024-04-25T01:53:52.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.942+0000 7f3bfffff640 1 -- 172.21.15.31:0/1045780223 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3c0800fa20 con 0x7f3c18107460 2024-04-25T01:53:52.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.943+0000 7f3bfffff640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3c000a2c20 0x7f3c000a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:52.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.943+0000 7f3c16d9d640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3c000a2c20 0x7f3c000a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:52.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.943+0000 7f3bfffff640 1 -- 172.21.15.31:0/1045780223 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f3c080cf340 con 0x7f3c18107460 2024-04-25T01:53:52.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.943+0000 7f3c16d9d640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3c000a2c20 0x7f3c000a50e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009fa0 tx=0x7f3c10009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:52.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:52.947+0000 7f3bfffff640 1 -- 172.21.15.31:0/1045780223 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3c08014030 con 0x7f3c18107460 2024-04-25T01:53:53.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.131+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/1045780223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3c1802d320 con 0x7f3c18107460 2024-04-25T01:53:53.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.132+0000 7f3bfffff640 1 -- 172.21.15.31:0/1045780223 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f3c0809c170 con 0x7f3c18107460 2024-04-25T01:53:53.133 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T01:53:53.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/1045780223 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3c000a2c20 msgr2=0x7f3c000a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3c000a2c20 0x7f3c000a50e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009fa0 tx=0x7f3c10009570 comp rx=0 tx=0).stop 2024-04-25T01:53:53.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/1045780223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c18107460 msgr2=0x7f3c1819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c18107460 0x7f3c1819f010 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800c970 tx=0x7f3c0800ce30 comp rx=0 tx=0).stop 2024-04-25T01:53:53.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/1045780223 shutdown_connections 2024-04-25T01:53:53.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f3c000a2c20 0x7f3c000a50e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c18107460 0x7f3c1819f010 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c18104b30 0x7f3c18197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 --2- 172.21.15.31:0/1045780223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c18102200 0x7f3c18197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/1045780223 >> 172.21.15.31:0/1045780223 conn(0x7f3c180fbbf0 msgr2=0x7f3c18102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:53.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/1045780223 shutdown_connections 2024-04-25T01:53:53.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.134+0000 7f3c1d1f9640 1 -- 172.21.15.31:0/1045780223 wait complete. 2024-04-25T01:53:53.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:53:53.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.250+0000 7f07682bd640 1 -- 172.21.15.31:0/888420610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07601017a0 msgr2=0x7f0760101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.250+0000 7f07682bd640 1 --2- 172.21.15.31:0/888420610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07601017a0 0x7f0760101bc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0758009900 tx=0x7f075802a410 comp rx=0 tx=0).stop 2024-04-25T01:53:53.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.250+0000 7f07682bd640 1 -- 172.21.15.31:0/888420610 shutdown_connections 2024-04-25T01:53:53.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.250+0000 7f07682bd640 1 --2- 172.21.15.31:0/888420610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0760103350 0x7f0760109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.250+0000 7f07682bd640 1 --2- 172.21.15.31:0/888420610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0760102990 0x7f0760102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.250+0000 7f07682bd640 1 --2- 172.21.15.31:0/888420610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07601017a0 0x7f0760101bc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.250+0000 7f07682bd640 1 -- 172.21.15.31:0/888420610 >> 172.21.15.31:0/888420610 conn(0x7f07600fcd10 msgr2=0x7f07600ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:53.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.250+0000 7f07682bd640 1 -- 172.21.15.31:0/888420610 shutdown_connections 2024-04-25T01:53:53.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.250+0000 7f07682bd640 1 -- 172.21.15.31:0/888420610 wait complete. 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f07682bd640 1 Processor -- start 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f07682bd640 1 -- start start 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f07682bd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07601017a0 0x7f07601979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f07682bd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0760102990 0x7f0760197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f07682bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0760103350 0x7f076019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f07682bd640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f076010d2d0 con 0x7f0760102990 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f07682bd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f076010d150 con 0x7f0760103350 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f07682bd640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f076010d450 con 0x7f07601017a0 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f0765858640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0760102990 0x7f0760197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f0765858640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0760102990 0x7f0760197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46172/0 (socket says 172.21.15.31:46172) 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f0765858640 1 -- 172.21.15.31:0/1043100141 learned_addr learned my addr 172.21.15.31:0/1043100141 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f0766059640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07601017a0 0x7f07601979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f076685a640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0760103350 0x7f076019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f0765858640 1 -- 172.21.15.31:0/1043100141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07601017a0 msgr2=0x7f07601979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f0765858640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07601017a0 0x7f07601979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f0765858640 1 -- 172.21.15.31:0/1043100141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0760103350 msgr2=0x7f076019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f0765858640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0760103350 0x7f076019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.251+0000 7f0765858640 1 -- 172.21.15.31:0/1043100141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076019f610 con 0x7f0760102990 2024-04-25T01:53:53.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.252+0000 7f0765858640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0760102990 0x7f0760197f30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f075400d950 tx=0x7f075400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:53.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.252+0000 7f075f7fe640 1 -- 172.21.15.31:0/1043100141 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0754017070 con 0x7f0760102990 2024-04-25T01:53:53.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.252+0000 7f075f7fe640 1 -- 172.21.15.31:0/1043100141 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0754009c30 con 0x7f0760102990 2024-04-25T01:53:53.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.252+0000 7f07682bd640 1 -- 172.21.15.31:0/1043100141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076019f900 con 0x7f0760102990 2024-04-25T01:53:53.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.252+0000 7f075f7fe640 1 -- 172.21.15.31:0/1043100141 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075400f620 con 0x7f0760102990 2024-04-25T01:53:53.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.252+0000 7f07682bd640 1 -- 172.21.15.31:0/1043100141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f076019fea0 con 0x7f0760102990 2024-04-25T01:53:53.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.253+0000 7f07682bd640 1 -- 172.21.15.31:0/1043100141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0760108be0 con 0x7f0760102990 2024-04-25T01:53:53.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.254+0000 7f075f7fe640 1 -- 172.21.15.31:0/1043100141 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f07540109a0 con 0x7f0760102990 2024-04-25T01:53:53.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.255+0000 7f075f7fe640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f07480a2ca0 0x7f07480a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.255+0000 7f0766059640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f07480a2ca0 0x7f07480a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.255+0000 7f075f7fe640 1 -- 172.21.15.31:0/1043100141 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f07540cecc0 con 0x7f0760102990 2024-04-25T01:53:53.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.255+0000 7f0766059640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f07480a2ca0 0x7f07480a5160 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0758004770 tx=0x7f0758000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:53.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.258+0000 7f075f7fe640 1 -- 172.21.15.31:0/1043100141 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0754009de0 con 0x7f0760102990 2024-04-25T01:53:53.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:53 smithi040 ceph-mon[32418]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:53.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:53 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/4092084658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:53:53.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:53 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1858589788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:53.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:53 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/320687907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:53:53.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:53 smithi134 ceph-mon[33098]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:53.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:53 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/4092084658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:53:53.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:53 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1858589788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:53.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:53 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/320687907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:53:53.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.443+0000 7f07682bd640 1 -- 172.21.15.31:0/1043100141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f076002d3c0 con 0x7f0760102990 2024-04-25T01:53:53.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.444+0000 7f075f7fe640 1 -- 172.21.15.31:0/1043100141 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f075409b4c0 con 0x7f0760102990 2024-04-25T01:53:53.444 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T01:53:53.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.445+0000 7f07682bd640 1 -- 172.21.15.31:0/1043100141 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f07480a2ca0 msgr2=0x7f07480a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.445+0000 7f07682bd640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f07480a2ca0 0x7f07480a5160 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0758004770 tx=0x7f0758000bc0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.445+0000 7f07682bd640 1 -- 172.21.15.31:0/1043100141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0760102990 msgr2=0x7f0760197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.445+0000 7f07682bd640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0760102990 0x7f0760197f30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f075400d950 tx=0x7f075400de10 comp rx=0 tx=0).stop 2024-04-25T01:53:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.446+0000 7f07682bd640 1 -- 172.21.15.31:0/1043100141 shutdown_connections 2024-04-25T01:53:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.446+0000 7f07682bd640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f07480a2ca0 0x7f07480a5160 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.446+0000 7f07682bd640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0760103350 0x7f076019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.446+0000 7f07682bd640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0760102990 0x7f0760197f30 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.446+0000 7f07682bd640 1 --2- 172.21.15.31:0/1043100141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07601017a0 0x7f07601979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.446+0000 7f07682bd640 1 -- 172.21.15.31:0/1043100141 >> 172.21.15.31:0/1043100141 conn(0x7f07600fcd10 msgr2=0x7f0760105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.446+0000 7f07682bd640 1 -- 172.21.15.31:0/1043100141 shutdown_connections 2024-04-25T01:53:53.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.446+0000 7f07682bd640 1 -- 172.21.15.31:0/1043100141 wait complete. 2024-04-25T01:53:53.456 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:53 smithi031 ceph-mon[29204]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:53:53.456 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:53 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/4092084658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:53:53.456 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:53 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1858589788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:53.456 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:53 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/320687907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:53:53.457 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:53:53.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.561+0000 7fb18b2df640 1 -- 172.21.15.31:0/3138275061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101870 msgr2=0x7fb184101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.561+0000 7fb18b2df640 1 --2- 172.21.15.31:0/3138275061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101870 0x7fb184101cf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb174009900 tx=0x7fb17402a410 comp rx=0 tx=0).stop 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.561+0000 7fb18b2df640 1 -- 172.21.15.31:0/3138275061 shutdown_connections 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.561+0000 7fb18b2df640 1 --2- 172.21.15.31:0/3138275061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb184102230 0x7fb184108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.561+0000 7fb18b2df640 1 --2- 172.21.15.31:0/3138275061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101870 0x7fb184101cf0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.561+0000 7fb18b2df640 1 --2- 172.21.15.31:0/3138275061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb184100680 0x7fb184100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.561+0000 7fb18b2df640 1 -- 172.21.15.31:0/3138275061 >> 172.21.15.31:0/3138275061 conn(0x7fb1840fbbf0 msgr2=0x7fb1840fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.561+0000 7fb18b2df640 1 -- 172.21.15.31:0/3138275061 shutdown_connections 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.561+0000 7fb18b2df640 1 -- 172.21.15.31:0/3138275061 wait complete. 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.561+0000 7fb18b2df640 1 Processor -- start 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.561+0000 7fb18b2df640 1 -- start start 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18b2df640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb184100680 0x7fb184197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18b2df640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101870 0x7fb184197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18b2df640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb184102230 0x7fb18419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18b2df640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb18410c1b0 con 0x7fb184101870 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18b2df640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb18410c030 con 0x7fb184100680 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18b2df640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb18410c330 con 0x7fb184102230 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18887a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101870 0x7fb184197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18887a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101870 0x7fb184197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46188/0 (socket says 172.21.15.31:46188) 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18887a640 1 -- 172.21.15.31:0/3299133377 learned_addr learned my addr 172.21.15.31:0/3299133377 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18987c640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb184102230 0x7fb18419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18907b640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb184100680 0x7fb184197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18887a640 1 -- 172.21.15.31:0/3299133377 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb184102230 msgr2=0x7fb18419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18887a640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb184102230 0x7fb18419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18887a640 1 -- 172.21.15.31:0/3299133377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb184100680 msgr2=0x7fb184197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18887a640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb184100680 0x7fb184197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18887a640 1 -- 172.21.15.31:0/3299133377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18419f620 con 0x7fb184101870 2024-04-25T01:53:53.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18887a640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101870 0x7fb184197f40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb174004770 tx=0x7fb1740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:53.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb17a7fc640 1 -- 172.21.15.31:0/3299133377 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17403f030 con 0x7fb184101870 2024-04-25T01:53:53.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18b2df640 1 -- 172.21.15.31:0/3299133377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18419f8b0 con 0x7fb184101870 2024-04-25T01:53:53.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb17a7fc640 1 -- 172.21.15.31:0/3299133377 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb174003cb0 con 0x7fb184101870 2024-04-25T01:53:53.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb17a7fc640 1 -- 172.21.15.31:0/3299133377 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb174035070 con 0x7fb184101870 2024-04-25T01:53:53.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.562+0000 7fb18b2df640 1 -- 172.21.15.31:0/3299133377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18419fda0 con 0x7fb184101870 2024-04-25T01:53:53.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.563+0000 7fb18b2df640 1 -- 172.21.15.31:0/3299133377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb18419ffa0 con 0x7fb184101870 2024-04-25T01:53:53.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.564+0000 7fb17a7fc640 1 -- 172.21.15.31:0/3299133377 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb174042030 con 0x7fb184101870 2024-04-25T01:53:53.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.565+0000 7fb17a7fc640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1700a2c20 0x7fb1700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.565+0000 7fb18907b640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1700a2c20 0x7fb1700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.566+0000 7fb17a7fc640 1 -- 172.21.15.31:0/3299133377 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fb1740efe50 con 0x7fb184101870 2024-04-25T01:53:53.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.566+0000 7fb18907b640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1700a2c20 0x7fb1700a50e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb180005fd0 tx=0x7fb180005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:53.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.569+0000 7fb17a7fc640 1 -- 172.21.15.31:0/3299133377 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb1740bcc20 con 0x7fb184101870 2024-04-25T01:53:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.824+0000 7fb18b2df640 1 -- 172.21.15.31:0/3299133377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1841a0290 con 0x7fb184101870 2024-04-25T01:53:53.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.825+0000 7fb17a7fc640 1 -- 172.21.15.31:0/3299133377 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11923 (secure 0 0 0) 0x7fb1740bc520 con 0x7fb184101870 2024-04-25T01:53:53.826 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:53:53.826 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:52:32.772296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24487,24473,24503,24463,24451]},"id":1}]} 2024-04-25T01:53:53.826 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T01:53:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 -- 172.21.15.31:0/3299133377 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1700a2c20 msgr2=0x7fb1700a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1700a2c20 0x7fb1700a50e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb180005fd0 tx=0x7fb180005850 comp rx=0 tx=0).stop 2024-04-25T01:53:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 -- 172.21.15.31:0/3299133377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101870 msgr2=0x7fb184197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101870 0x7fb184197f40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb174004770 tx=0x7fb1740042f0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 -- 172.21.15.31:0/3299133377 shutdown_connections 2024-04-25T01:53:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb1700a2c20 0x7fb1700a50e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb184102230 0x7fb18419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb184101870 0x7fb184197f40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 --2- 172.21.15.31:0/3299133377 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb184100680 0x7fb184197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 -- 172.21.15.31:0/3299133377 >> 172.21.15.31:0/3299133377 conn(0x7fb1840fbbf0 msgr2=0x7fb184068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:53.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 -- 172.21.15.31:0/3299133377 shutdown_connections 2024-04-25T01:53:53.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.827+0000 7fb18b2df640 1 -- 172.21.15.31:0/3299133377 wait complete. 2024-04-25T01:53:53.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:53:53.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.941+0000 7f08a78a5640 1 -- 172.21.15.31:0/444528530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a01029a0 msgr2=0x7f08a0102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.941+0000 7f08a78a5640 1 --2- 172.21.15.31:0/444528530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a01029a0 0x7f08a0102e20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0890009900 tx=0x7f089002a410 comp rx=0 tx=0).stop 2024-04-25T01:53:53.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.941+0000 7f08a78a5640 1 -- 172.21.15.31:0/444528530 shutdown_connections 2024-04-25T01:53:53.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.941+0000 7f08a78a5640 1 --2- 172.21.15.31:0/444528530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a0103360 0x7f08a0109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.941+0000 7f08a78a5640 1 --2- 172.21.15.31:0/444528530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a01029a0 0x7f08a0102e20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.941+0000 7f08a78a5640 1 --2- 172.21.15.31:0/444528530 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a01017b0 0x7f08a0101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.941+0000 7f08a78a5640 1 -- 172.21.15.31:0/444528530 >> 172.21.15.31:0/444528530 conn(0x7f08a00fcd40 msgr2=0x7f08a00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:53.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.941+0000 7f08a78a5640 1 -- 172.21.15.31:0/444528530 shutdown_connections 2024-04-25T01:53:53.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.941+0000 7f08a78a5640 1 -- 172.21.15.31:0/444528530 wait complete. 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a78a5640 1 Processor -- start 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a78a5640 1 -- start start 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a78a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a01017b0 0x7f08a00754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a78a5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a01029a0 0x7f08a0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a78a5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0103360 0x7f08a0074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a78a5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f08a010d2c0 con 0x7f08a01017b0 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a78a5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f08a010d140 con 0x7f08a01029a0 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a78a5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f08a010d440 con 0x7f08a0103360 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a5641640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a01017b0 0x7f08a00754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a5641640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a01017b0 0x7f08a00754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46204/0 (socket says 172.21.15.31:46204) 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a5641640 1 -- 172.21.15.31:0/3488208907 learned_addr learned my addr 172.21.15.31:0/3488208907 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a5e42640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0103360 0x7f08a0074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a4e40640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a01029a0 0x7f08a0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a5641640 1 -- 172.21.15.31:0/3488208907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0103360 msgr2=0x7f08a0074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a5641640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0103360 0x7f08a0074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a5641640 1 -- 172.21.15.31:0/3488208907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a01029a0 msgr2=0x7f08a0073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:53.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a5641640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a01029a0 0x7f08a0073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:53.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a5641640 1 -- 172.21.15.31:0/3488208907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a0074900 con 0x7f08a01017b0 2024-04-25T01:53:53.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.942+0000 7f08a5641640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a01017b0 0x7f08a00754f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f089c00c970 tx=0x7f089c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:53.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.943+0000 7f08967fc640 1 -- 172.21.15.31:0/3488208907 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c007c10 con 0x7f08a01017b0 2024-04-25T01:53:53.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.943+0000 7f08967fc640 1 -- 172.21.15.31:0/3488208907 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f089c007db0 con 0x7f08a01017b0 2024-04-25T01:53:53.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.943+0000 7f08a78a5640 1 -- 172.21.15.31:0/3488208907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a01a1a90 con 0x7f08a01017b0 2024-04-25T01:53:53.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.943+0000 7f08967fc640 1 -- 172.21.15.31:0/3488208907 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c00f870 con 0x7f08a01017b0 2024-04-25T01:53:53.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.943+0000 7f08a78a5640 1 -- 172.21.15.31:0/3488208907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a01a1fb0 con 0x7f08a01017b0 2024-04-25T01:53:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.943+0000 7f08a78a5640 1 -- 172.21.15.31:0/3488208907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a00689b0 con 0x7f08a01017b0 2024-04-25T01:53:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.945+0000 7f08967fc640 1 -- 172.21.15.31:0/3488208907 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f089c00fa10 con 0x7f08a01017b0 2024-04-25T01:53:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.945+0000 7f08967fc640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f088c0a2c20 0x7f088c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.945+0000 7f08a4e40640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f088c0a2c20 0x7f088c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.945+0000 7f08967fc640 1 -- 172.21.15.31:0/3488208907 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f089c0cf780 con 0x7f08a01017b0 2024-04-25T01:53:53.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.946+0000 7f08a4e40640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f088c0a2c20 0x7f088c0a50e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0890006010 tx=0x7f0890000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:53.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:53.949+0000 7f08967fc640 1 -- 172.21.15.31:0/3488208907 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f089c09c6d0 con 0x7f08a01017b0 2024-04-25T01:53:54.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.107+0000 7f08967fc640 1 -- 172.21.15.31:0/3488208907 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f089c098640 con 0x7f08a01017b0 2024-04-25T01:53:54.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.141+0000 7f08a78a5640 1 -- 172.21.15.31:0/3488208907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08a0108ad0 con 0x7f08a01017b0 2024-04-25T01:53:54.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.142+0000 7f08967fc640 1 -- 172.21.15.31:0/3488208907 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22478 (secure 0 0 0) 0x7f089c0ccdf0 con 0x7f08a01017b0 2024-04-25T01:53:54.144 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:53:54.144 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:53:24.952281+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 -- 172.21.15.31:0/3488208907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f088c0a2c20 msgr2=0x7f088c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f088c0a2c20 0x7f088c0a50e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0890006010 tx=0x7f0890000bc0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 -- 172.21.15.31:0/3488208907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a01017b0 msgr2=0x7f08a00754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a01017b0 0x7f08a00754f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f089c00c970 tx=0x7f089c00ce30 comp rx=0 tx=0).stop 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 -- 172.21.15.31:0/3488208907 shutdown_connections 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f088c0a2c20 0x7f088c0a50e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08a0103360 0x7f08a0074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a01029a0 0x7f08a0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 --2- 172.21.15.31:0/3488208907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a01017b0 0x7f08a00754f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 -- 172.21.15.31:0/3488208907 >> 172.21.15.31:0/3488208907 conn(0x7f08a00fcd40 msgr2=0x7f08a0105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 -- 172.21.15.31:0/3488208907 shutdown_connections 2024-04-25T01:53:54.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.145+0000 7f08a78a5640 1 -- 172.21.15.31:0/3488208907 wait complete. 2024-04-25T01:53:54.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:53:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.258+0000 7fc769a2c640 1 -- 172.21.15.31:0/2108588395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc764101790 msgr2=0x7fc764101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.258+0000 7fc769a2c640 1 --2- 172.21.15.31:0/2108588395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc764101790 0x7fc764101bb0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009900 tx=0x7fc75c02a410 comp rx=0 tx=0).stop 2024-04-25T01:53:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.258+0000 7fc769a2c640 1 -- 172.21.15.31:0/2108588395 shutdown_connections 2024-04-25T01:53:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.258+0000 7fc769a2c640 1 --2- 172.21.15.31:0/2108588395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc764103340 0x7fc764109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.258+0000 7fc769a2c640 1 --2- 172.21.15.31:0/2108588395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc764102980 0x7fc764102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.258+0000 7fc769a2c640 1 --2- 172.21.15.31:0/2108588395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc764101790 0x7fc764101bb0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.258+0000 7fc769a2c640 1 -- 172.21.15.31:0/2108588395 >> 172.21.15.31:0/2108588395 conn(0x7fc7640fcd20 msgr2=0x7fc7640ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.258+0000 7fc769a2c640 1 -- 172.21.15.31:0/2108588395 shutdown_connections 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.258+0000 7fc769a2c640 1 -- 172.21.15.31:0/2108588395 wait complete. 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc769a2c640 1 Processor -- start 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc769a2c640 1 -- start start 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc769a2c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc764101790 0x7fc7641979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc769a2c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc764102980 0x7fc764197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc769a2c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc764103340 0x7fc76419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc769a2c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc76410d2b0 con 0x7fc764103340 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc769a2c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc76410d130 con 0x7fc764101790 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc769a2c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc76410d430 con 0x7fc764102980 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc7637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc764103340 0x7fc76419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc7637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc764103340 0x7fc76419ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46230/0 (socket says 172.21.15.31:46230) 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc7637fe640 1 -- 172.21.15.31:0/52892037 learned_addr learned my addr 172.21.15.31:0/52892037 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc762ffd640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc764101790 0x7fc7641979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.259+0000 7fc7627fc640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc764102980 0x7fc764197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.260+0000 7fc7637fe640 1 -- 172.21.15.31:0/52892037 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc764102980 msgr2=0x7fc764197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.260+0000 7fc7637fe640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc764102980 0x7fc764197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.260+0000 7fc7637fe640 1 -- 172.21.15.31:0/52892037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc764101790 msgr2=0x7fc7641979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.260+0000 7fc7637fe640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc764101790 0x7fc7641979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.260+0000 7fc7637fe640 1 -- 172.21.15.31:0/52892037 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76419f600 con 0x7fc764103340 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.260+0000 7fc7637fe640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc764103340 0x7fc76419ef90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc75400eb10 tx=0x7fc75400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.260+0000 7fc768a2a640 1 -- 172.21.15.31:0/52892037 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc754004b20 con 0x7fc764103340 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.260+0000 7fc768a2a640 1 -- 172.21.15.31:0/52892037 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc754004cc0 con 0x7fc764103340 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.260+0000 7fc768a2a640 1 -- 172.21.15.31:0/52892037 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc754010690 con 0x7fc764103340 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.260+0000 7fc769a2c640 1 -- 172.21.15.31:0/52892037 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc76419f8f0 con 0x7fc764103340 2024-04-25T01:53:54.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.260+0000 7fc769a2c640 1 -- 172.21.15.31:0/52892037 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76419fe40 con 0x7fc764103340 2024-04-25T01:53:54.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.261+0000 7fc769a2c640 1 -- 172.21.15.31:0/52892037 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc764108b30 con 0x7fc764103340 2024-04-25T01:53:54.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.262+0000 7fc768a2a640 1 -- 172.21.15.31:0/52892037 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc754005020 con 0x7fc764103340 2024-04-25T01:53:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.262+0000 7fc768a2a640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc74c0a2c10 0x7fc74c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.263+0000 7fc762ffd640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc74c0a2c10 0x7fc74c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.263+0000 7fc768a2a640 1 -- 172.21.15.31:0/52892037 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fc7540cf6f0 con 0x7fc764103340 2024-04-25T01:53:54.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.263+0000 7fc762ffd640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc74c0a2c10 0x7fc74c0a50d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc75c000c00 tx=0x7fc75c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.267+0000 7fc768a2a640 1 -- 172.21.15.31:0/52892037 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc75409c620 con 0x7fc764103340 2024-04-25T01:53:54.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:54 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1045780223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:53:54.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:54 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1043100141' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:53:54.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:54 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3299133377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:53:54.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:54 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1045780223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:53:54.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:54 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1043100141' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:53:54.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:54 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3299133377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:53:54.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:54 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1045780223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:53:54.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:54 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1043100141' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:53:54.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:54 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3299133377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:53:54.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.493+0000 7fc769a2c640 1 -- 172.21.15.31:0/52892037 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc76402d170 con 0x7fc764103340 2024-04-25T01:53:54.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.494+0000 7fc768a2a640 1 -- 172.21.15.31:0/52892037 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11923 (secure 0 0 0) 0x7fc75409bf20 con 0x7fc764103340 2024-04-25T01:53:54.496 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:53:54.496 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:52:32.772296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24487,24473,24503,24463,24451]},"id":1}]} 2024-04-25T01:53:54.496 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T01:53:54.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.496+0000 7fc769a2c640 1 -- 172.21.15.31:0/52892037 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc74c0a2c10 msgr2=0x7fc74c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.496+0000 7fc769a2c640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc74c0a2c10 0x7fc74c0a50d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc75c000c00 tx=0x7fc75c0314a0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.496+0000 7fc769a2c640 1 -- 172.21.15.31:0/52892037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc764103340 msgr2=0x7fc76419ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.496+0000 7fc769a2c640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc764103340 0x7fc76419ef90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc75400eb10 tx=0x7fc75400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.496+0000 7fc769a2c640 1 -- 172.21.15.31:0/52892037 shutdown_connections 2024-04-25T01:53:54.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.496+0000 7fc769a2c640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fc74c0a2c10 0x7fc74c0a50d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.496+0000 7fc769a2c640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc764103340 0x7fc76419ef90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.496+0000 7fc769a2c640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc764102980 0x7fc764197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.496+0000 7fc769a2c640 1 --2- 172.21.15.31:0/52892037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc764101790 0x7fc7641979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.496+0000 7fc769a2c640 1 -- 172.21.15.31:0/52892037 >> 172.21.15.31:0/52892037 conn(0x7fc7640fcd20 msgr2=0x7fc764105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.497+0000 7fc769a2c640 1 -- 172.21.15.31:0/52892037 shutdown_connections 2024-04-25T01:53:54.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.497+0000 7fc769a2c640 1 -- 172.21.15.31:0/52892037 wait complete. 2024-04-25T01:53:54.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T01:52:24.898441+0000', 'modified': '2024-04-25T01:52:32.772296+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24503, 'mds_1': 24487, 'mds_2': 24473, 'mds_3': 24463, 'mds_4': 24451}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14622': {'gid': 14622, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.31:6839/3704800465', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6838', 'nonce': 3704800465}, {'type': 'v1', 'addr': '172.21.15.31:6839', 'nonce': 3704800465}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14634': {'gid': 14634, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.31:6841/4277792607', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6840', 'nonce': 4277792607}, {'type': 'v1', 'addr': '172.21.15.31:6841', 'nonce': 4277792607}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24407': {'gid': 24407, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6833/3875547004', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 3875547004}, {'type': 'v1', 'addr': '172.21.15.40:6833', 'nonce': 3875547004}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24412': {'gid': 24412, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.134:6833/400138993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6832', 'nonce': 400138993}, {'type': 'v1', 'addr': '172.21.15.134:6833', 'nonce': 400138993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6835/538218297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 538218297}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 538218297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.134:6835/1266939870', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6834', 'nonce': 1266939870}, {'type': 'v1', 'addr': '172.21.15.134:6835', 'nonce': 1266939870}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.134:6837/2058394187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6836', 'nonce': 2058394187}, {'type': 'v1', 'addr': '172.21.15.134:6837', 'nonce': 2058394187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.40:6837/1901184087', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6836', 'nonce': 1901184087}, {'type': 'v1', 'addr': '172.21.15.40:6837', 'nonce': 1901184087}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.134:6839/450925094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6838', 'nonce': 450925094}, {'type': 'v1', 'addr': '172.21.15.134:6839', 'nonce': 450925094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6839/3236110559', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 3236110559}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 3236110559}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24503, 'qdb_cluster': [24487, 24473, 24503, 24463, 24451]} 2024-04-25T01:53:54.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T01:53:54.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-25T01:53:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.611+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/4093057697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fd0103330 msgr2=0x7f5fd0109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.611+0000 7f5fd4bdc640 1 --2- 172.21.15.31:0/4093057697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fd0103330 0x7f5fd0109bc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009be0 tx=0x7f5fc002a6c0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.611+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/4093057697 shutdown_connections 2024-04-25T01:53:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.611+0000 7f5fd4bdc640 1 --2- 172.21.15.31:0/4093057697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fd0103330 0x7f5fd0109bc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 --2- 172.21.15.31:0/4093057697 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fd0102970 0x7f5fd0102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 --2- 172.21.15.31:0/4093057697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fd0101780 0x7f5fd0101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/4093057697 >> 172.21.15.31:0/4093057697 conn(0x7f5fd00fccf0 msgr2=0x7f5fd00ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/4093057697 shutdown_connections 2024-04-25T01:53:54.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/4093057697 wait complete. 2024-04-25T01:53:54.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 Processor -- start 2024-04-25T01:53:54.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 -- start start 2024-04-25T01:53:54.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fd0101780 0x7f5fd01978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fd0102970 0x7f5fd0197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fd0103330 0x7f5fd019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010d290 con 0x7f5fd0103330 2024-04-25T01:53:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010d110 con 0x7f5fd0101780 2024-04-25T01:53:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.612+0000 7f5fd4bdc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010d410 con 0x7f5fd0102970 2024-04-25T01:53:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fd0103330 0x7f5fd019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fd0103330 0x7f5fd019ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46246/0 (socket says 172.21.15.31:46246) 2024-04-25T01:53:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fced9d640 1 -- 172.21.15.31:0/3516997792 learned_addr learned my addr 172.21.15.31:0/3516997792 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fce59c640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fd0101780 0x7f5fd01978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fcdd9b640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fd0102970 0x7f5fd0197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fced9d640 1 -- 172.21.15.31:0/3516997792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fd0102970 msgr2=0x7f5fd0197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fced9d640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fd0102970 0x7f5fd0197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fced9d640 1 -- 172.21.15.31:0/3516997792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fd0101780 msgr2=0x7f5fd01978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fced9d640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fd0101780 0x7f5fd01978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fced9d640 1 -- 172.21.15.31:0/3516997792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd019f4f0 con 0x7f5fd0103330 2024-04-25T01:53:54.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fcdd9b640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fd0102970 0x7f5fd0197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:54.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fced9d640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fd0103330 0x7f5fd019ee80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0034040 tx=0x7f5fc0031dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fbf7fe640 1 -- 172.21.15.31:0/3516997792 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0041020 con 0x7f5fd0103330 2024-04-25T01:53:54.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fbf7fe640 1 -- 172.21.15.31:0/3516997792 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fc0037040 con 0x7f5fd0103330 2024-04-25T01:53:54.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/3516997792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd019f780 con 0x7f5fd0103330 2024-04-25T01:53:54.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fbf7fe640 1 -- 172.21.15.31:0/3516997792 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0036070 con 0x7f5fd0103330 2024-04-25T01:53:54.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.613+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/3516997792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd019fcc0 con 0x7f5fd0103330 2024-04-25T01:53:54.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.615+0000 7f5fbf7fe640 1 -- 172.21.15.31:0/3516997792 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5fc0044020 con 0x7f5fd0103330 2024-04-25T01:53:54.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fd79c9740 1 Processor -- start 2024-04-25T01:53:54.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fd79c9740 1 -- start start 2024-04-25T01:53:54.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fbf7fe640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5fb80a2c10 0x7f5fb80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce90850 0x55dfcce90c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce91240 0x55dfcce95d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfcce98840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dfccdeff00 con 0x55dfcce963b0 2024-04-25T01:53:54.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dfccd98750 con 0x55dfcce90850 2024-04-25T01:53:54.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dfccdebec0 con 0x55dfcce91240 2024-04-25T01:53:54.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fcdd9b640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5fb80a2c10 0x7f5fb80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfcce98840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfcce98840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46248/0 (socket says 172.21.15.31:46248) 2024-04-25T01:53:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fced9d640 1 -- 172.21.15.31:0/3339365947 learned_addr learned my addr 172.21.15.31:0/3339365947 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fce59c640 1 --2- 172.21.15.31:0/3339365947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce90850 0x55dfcce90c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.616+0000 7f5fbf7fe640 1 -- 172.21.15.31:0/3516997792 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f5fc00efaa0 con 0x7f5fd0103330 2024-04-25T01:53:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.617+0000 7f5fced9d640 1 -- 172.21.15.31:0/3339365947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce91240 msgr2=0x55dfcce95d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.617+0000 7f5fced9d640 1 --2- 172.21.15.31:0/3339365947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce91240 0x55dfcce95d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.617+0000 7f5fced9d640 1 -- 172.21.15.31:0/3339365947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce90850 msgr2=0x55dfcce90c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.617+0000 7f5fced9d640 1 --2- 172.21.15.31:0/3339365947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce90850 0x55dfcce90c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.617+0000 7f5fced9d640 1 -- 172.21.15.31:0/3339365947 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfccb042d0 con 0x55dfcce963b0 2024-04-25T01:53:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.617+0000 7f5fcdd9b640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5fb80a2c10 0x7f5fb80a50d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0198de0 tx=0x7f5fc4008540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.617+0000 7f5fced9d640 1 --2- 172.21.15.31:0/3339365947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfcce98840 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00bc260 tx=0x7f5fc002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd91dac50f53bfa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.617+0000 7f5fbcff9640 1 -- 172.21.15.31:0/3339365947 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc00d54d0 con 0x55dfcce963b0 2024-04-25T01:53:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.617+0000 7f5fbcff9640 1 -- 172.21.15.31:0/3339365947 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fc0004040 con 0x55dfcce963b0 2024-04-25T01:53:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.617+0000 7f5fbcff9640 1 -- 172.21.15.31:0/3339365947 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc00c4f40 con 0x55dfcce963b0 2024-04-25T01:53:54.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 -- 172.21.15.31:0/3339365947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 msgr2=0x55dfcce98840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/3339365947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfcce98840 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00bc260 tx=0x7f5fc002a6c0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 -- 172.21.15.31:0/3339365947 shutdown_connections 2024-04-25T01:53:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/3339365947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfcce98840 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/3339365947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce91240 0x55dfcce95d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/3339365947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce90850 0x55dfcce90c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 -- 172.21.15.31:0/3339365947 >> 172.21.15.31:0/3339365947 conn(0x55dfcce87ab0 msgr2=0x55dfcce87eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 -- 172.21.15.31:0/3339365947 shutdown_connections 2024-04-25T01:53:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 -- 172.21.15.31:0/3339365947 wait complete. 2024-04-25T01:53:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 -- start start 2024-04-25T01:53:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 Processor -- start 2024-04-25T01:53:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce90850 0x55dfcce9aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce91240 0x55dfcce9af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dfcce99120 con 0x55dfcce963b0 2024-04-25T01:53:54.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dfccdabcf0 con 0x55dfcce91240 2024-04-25T01:53:54.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.618+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dfcce992a0 con 0x55dfcce90850 2024-04-25T01:53:54.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fd79c9740 10 client.0 _ll_register_callbacks cb 0x55dfccdab6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:53:54.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46262/0 (socket says 172.21.15.31:46262) 2024-04-25T01:53:54.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fced9d640 1 -- 172.21.15.31:0/617693540 learned_addr learned my addr 172.21.15.31:0/617693540 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fcdd9b640 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce91240 0x55dfcce9af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fce59c640 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce90850 0x55dfcce9aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fced9d640 1 -- 172.21.15.31:0/617693540 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce90850 msgr2=0x55dfcce9aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fced9d640 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce90850 0x55dfcce9aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fced9d640 1 -- 172.21.15.31:0/617693540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce91240 msgr2=0x55dfcce9af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fced9d640 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce91240 0x55dfcce9af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fced9d640 1 -- 172.21.15.31:0/617693540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfccbce6b0 con 0x55dfcce963b0 2024-04-25T01:53:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fce59c640 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce90850 0x55dfcce9aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fced9d640 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1f40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00c4a10 tx=0x7f5fc0005c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4e9ffd3499577e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:53:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/617693540 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc00bc9d0 con 0x55dfcce963b0 2024-04-25T01:53:54.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/617693540 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f5fc00ed680 con 0x55dfcce963b0 2024-04-25T01:53:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fd79c9740 1 -- 172.21.15.31:0/617693540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dfccdfd440 con 0x55dfcce963b0 2024-04-25T01:53:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/617693540 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5fc0003e00 con 0x55dfcce963b0 2024-04-25T01:53:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fd79c9740 10 client.15051 fetch_fsmap learned FSMap version 26 2024-04-25T01:53:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fd79c9740 1 -- 172.21.15.31:0/617693540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dfcce99120 con 0x55dfcce963b0 2024-04-25T01:53:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.619+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/617693540 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55dfccdfd440 con 0x55dfcce963b0 2024-04-25T01:53:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.620+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/617693540 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5fc00d07f0 con 0x55dfcce963b0 2024-04-25T01:53:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.620+0000 7f5fd79c9740 10 client.15051 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:53:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.620+0000 7f5fd79c9740 10 client.15051 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T01:53:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.620+0000 7f5fd79c9740 20 client.15051 populate_metadata read hostname 'smithi031' 2024-04-25T01:53:54.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.620+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55dfccea7370 0x55dfccea87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.620+0000 7f5fd79c9740 4 client.15051 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.620+0000 7f5fd79c9740 1 -- 172.21.15.31:0/617693540 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfccdf7790 con 0x55dfccea7370 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.620+0000 7f5fce59c640 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55dfccea7370 0x55dfccea87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.620+0000 7f5fce59c640 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55dfccea7370 0x55dfccea87e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ba3eb76bee001573 server_cookie=5a02781b482e2d60 in_seq=0 out_seq=0 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.620+0000 7f5fb5ffb640 10 client.15051 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/617693540 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dfccdf7790 con 0x55dfccea7370 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fb5ffb640 10 client.15051 handle_command_reply: tid=0 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fd79c9740 1 client.15051 shutdown 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fd79c9740 1 -- 172.21.15.31:0/617693540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 msgr2=0x55dfccea1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1f40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00c4a10 tx=0x7f5fc0005c40 comp rx=0 tx=0).stop 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fd79c9740 1 -- 172.21.15.31:0/617693540 shutdown_connections 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55dfccea7370 0x55dfccea87e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1f40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce91240 0x55dfcce9af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fb5ffb640 0 client.15051 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:53:54.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/617693540 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce90850 0x55dfcce9aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fd79c9740 1 -- 172.21.15.31:0/617693540 >> 172.21.15.31:0/617693540 conn(0x55dfcce87ab0 msgr2=0x55dfcce86ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.621+0000 7f5fd79c9740 1 -- 172.21.15.31:0/617693540 shutdown_connections 2024-04-25T01:53:54.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.622+0000 7f5fd79c9740 1 -- 172.21.15.31:0/617693540 wait complete. 2024-04-25T01:53:54.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.622+0000 7f5fd79c9740 20 client.15051 trim_cache size 0 max 16384 2024-04-25T01:53:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fd79c9740 1 Processor -- start 2024-04-25T01:53:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fd79c9740 1 -- start start 2024-04-25T01:53:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce90850 0x55dfcce99d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce91240 0x55dfccea1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce963b0 0x55dfcce9a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dfccea6b20 con 0x55dfcce90850 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dfccea71f0 con 0x55dfcce963b0 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dfccea78c0 con 0x55dfcce91240 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fce59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce90850 0x55dfcce99d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fce59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce90850 0x55dfcce99d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46278/0 (socket says 172.21.15.31:46278) 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fce59c640 1 -- 172.21.15.31:0/3717328852 learned_addr learned my addr 172.21.15.31:0/3717328852 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fced9d640 1 --2- 172.21.15.31:0/3717328852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce963b0 0x55dfcce9a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fcdd9b640 1 --2- 172.21.15.31:0/3717328852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce91240 0x55dfccea1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fce59c640 1 -- 172.21.15.31:0/3717328852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce91240 msgr2=0x55dfccea1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fce59c640 1 --2- 172.21.15.31:0/3717328852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce91240 0x55dfccea1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fce59c640 1 -- 172.21.15.31:0/3717328852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce963b0 msgr2=0x55dfcce9a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fce59c640 1 --2- 172.21.15.31:0/3717328852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce963b0 0x55dfcce9a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fce59c640 1 -- 172.21.15.31:0/3717328852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfcce878e0 con 0x55dfcce90850 2024-04-25T01:53:54.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.640+0000 7f5fcdd9b640 1 --2- 172.21.15.31:0/3717328852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce91240 0x55dfccea1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fce59c640 1 --2- 172.21.15.31:0/3717328852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce90850 0x55dfcce99d20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55dfcce91aa0 tx=0x7f5fc8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44c57172ea5be5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fb7fff640 1 -- 172.21.15.31:0/3717328852 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8026070 con 0x55dfcce90850 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fb7fff640 1 -- 172.21.15.31:0/3717328852 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fc8004310 con 0x55dfcce90850 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fb7fff640 1 -- 172.21.15.31:0/3717328852 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8003a80 con 0x55dfcce90850 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fd79c9740 1 -- 172.21.15.31:0/3717328852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce90850 msgr2=0x55dfcce99d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/3717328852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce90850 0x55dfcce99d20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55dfcce91aa0 tx=0x7f5fc8009260 comp rx=0 tx=0).stop 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fd79c9740 1 -- 172.21.15.31:0/3717328852 shutdown_connections 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/3717328852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce963b0 0x55dfcce9a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/3717328852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce91240 0x55dfccea1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/3717328852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce90850 0x55dfcce99d20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fd79c9740 1 -- 172.21.15.31:0/3717328852 >> 172.21.15.31:0/3717328852 conn(0x55dfcce87ab0 msgr2=0x55dfcce9a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fd79c9740 1 -- 172.21.15.31:0/3717328852 shutdown_connections 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.641+0000 7f5fd79c9740 1 -- 172.21.15.31:0/3717328852 wait complete. 2024-04-25T01:53:54.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fd79c9740 1 -- start start 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fd79c9740 1 Processor -- start 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce90850 0x55dfcce9e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce91240 0x55dfcce9e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fd79c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dfccea8660 con 0x55dfcce963b0 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dfccea7f90 con 0x55dfcce91240 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fd79c9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dfccea8d30 con 0x55dfcce90850 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fd79c9740 10 client.0 _ll_register_callbacks cb 0x7f5fc00c0760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46294/0 (socket says 172.21.15.31:46294) 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fced9d640 1 -- 172.21.15.31:0/1831994945 learned_addr learned my addr 172.21.15.31:0/1831994945 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fcdd9b640 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce91240 0x55dfcce9e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fce59c640 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce90850 0x55dfcce9e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fced9d640 1 -- 172.21.15.31:0/1831994945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce90850 msgr2=0x55dfcce9e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fced9d640 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce90850 0x55dfcce9e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fced9d640 1 -- 172.21.15.31:0/1831994945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce91240 msgr2=0x55dfcce9e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fced9d640 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce91240 0x55dfcce9e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fced9d640 1 -- 172.21.15.31:0/1831994945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfccea6950 con 0x55dfcce963b0 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fced9d640 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1420 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00bbda0 tx=0x7f5fc0009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1dfc6be6d66b8ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.642+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/1831994945 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc00bc9d0 con 0x55dfcce963b0 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/1831994945 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f5fc00b8940 con 0x55dfcce963b0 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fd79c9740 1 -- 172.21.15.31:0/1831994945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fc8004310 con 0x55dfcce963b0 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/1831994945 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5fc00c4f40 con 0x55dfcce963b0 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fd79c9740 10 client.15057 fetch_fsmap learned FSMap version 26 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fd79c9740 1 -- 172.21.15.31:0/1831994945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dfccea6950 con 0x55dfcce963b0 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/1831994945 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fc8004310 con 0x55dfcce963b0 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/1831994945 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5fc00c9070 con 0x55dfcce963b0 2024-04-25T01:53:54.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fd79c9740 10 client.15057 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fd79c9740 10 client.15057 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fd79c9740 20 client.15057 populate_metadata read hostname 'smithi031' 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55dfccf12070 0x55dfccf0e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fd79c9740 4 client.15057 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fd79c9740 1 -- 172.21.15.31:0/1831994945 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "status"}) -- 0x55dfccd75b80 con 0x55dfccf12070 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.643+0000 7f5fce59c640 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55dfccf12070 0x55dfccf0e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fce59c640 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55dfccf12070 0x55dfccf0e4b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d344d8814cb5082 server_cookie=65e52628b6c571f8 in_seq=0 out_seq=0 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fb5ffb640 10 client.15057 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fb5ffb640 1 -- 172.21.15.31:0/1831994945 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55dfccd75b80 con 0x55dfccf12070 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fb5ffb640 10 client.15057 handle_command_reply: tid=0 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 1 client.15057 shutdown 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 1 -- 172.21.15.31:0/1831994945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 msgr2=0x55dfccea1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1420 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00bbda0 tx=0x7f5fc0009500 comp rx=0 tx=0).stop 2024-04-25T01:53:54.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 1 -- 172.21.15.31:0/1831994945 shutdown_connections 2024-04-25T01:53:54.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55dfccf12070 0x55dfccf0e4b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfcce963b0 0x55dfccea1420 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dfcce91240 0x55dfcce9e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fb5ffb640 0 client.15057 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:53:54.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 1 --2- 172.21.15.31:0/1831994945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dfcce90850 0x55dfcce9e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 1 -- 172.21.15.31:0/1831994945 >> 172.21.15.31:0/1831994945 conn(0x55dfcce87ab0 msgr2=0x55dfcce99d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 1 -- 172.21.15.31:0/1831994945 shutdown_connections 2024-04-25T01:53:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 1 -- 172.21.15.31:0/1831994945 wait complete. 2024-04-25T01:53:54.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.644+0000 7f5fd79c9740 20 client.15057 trim_cache size 0 max 16384 2024-04-25T01:53:54.659 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 4, 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24451, 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 82.904364177000005, 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 110.48200474799999 2024-04-25T01:53:54.660 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/3516997792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5fb80a2c10 msgr2=0x7f5fb80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5fb80a2c10 0x7f5fb80a50d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0198de0 tx=0x7f5fc4008540 comp rx=0 tx=0).stop 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/3516997792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fd0103330 msgr2=0x7f5fd019ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fd0103330 0x7f5fd019ee80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0034040 tx=0x7f5fc0031dc0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/3516997792 shutdown_connections 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5fb80a2c10 0x7f5fb80a50d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fd0103330 0x7f5fd019ee80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fd0102970 0x7f5fd0197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 --2- 172.21.15.31:0/3516997792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fd0101780 0x7f5fd01978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/3516997792 >> 172.21.15.31:0/3516997792 conn(0x7f5fd00fccf0 msgr2=0x7f5fd0105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/3516997792 shutdown_connections 2024-04-25T01:53:54.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.645+0000 7f5fd4bdc640 1 -- 172.21.15.31:0/3516997792 wait complete. 2024-04-25T01:53:54.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T01:53:54.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.765+0000 7f0d129c8640 1 -- 172.21.15.31:0/366345318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d0c104c10 msgr2=0x7f0d0c107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.765+0000 7f0d129c8640 1 --2- 172.21.15.31:0/366345318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d0c104c10 0x7f0d0c107000 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0d00009900 tx=0x7f0d0002a410 comp rx=0 tx=0).stop 2024-04-25T01:53:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.765+0000 7f0d129c8640 1 -- 172.21.15.31:0/366345318 shutdown_connections 2024-04-25T01:53:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.765+0000 7f0d129c8640 1 --2- 172.21.15.31:0/366345318 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d0c107540 0x7f0d0c109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.765+0000 7f0d129c8640 1 --2- 172.21.15.31:0/366345318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d0c104c10 0x7f0d0c107000 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.765+0000 7f0d129c8640 1 --2- 172.21.15.31:0/366345318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c1022e0 0x7f0d0c1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.765+0000 7f0d129c8640 1 -- 172.21.15.31:0/366345318 >> 172.21.15.31:0/366345318 conn(0x7f0d0c0fbc30 msgr2=0x7f0d0c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.765+0000 7f0d129c8640 1 -- 172.21.15.31:0/366345318 shutdown_connections 2024-04-25T01:53:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.765+0000 7f0d129c8640 1 -- 172.21.15.31:0/366345318 wait complete. 2024-04-25T01:53:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.765+0000 7f0d129c8640 1 Processor -- start 2024-04-25T01:53:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d129c8640 1 -- start start 2024-04-25T01:53:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d129c8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c1022e0 0x7f0d0c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d129c8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d0c104c10 0x7f0d0c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d129c8640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d0c107540 0x7f0d0c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d129c8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c10d2e0 con 0x7f0d0c104c10 2024-04-25T01:53:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d129c8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c10d160 con 0x7f0d0c1022e0 2024-04-25T01:53:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d129c8640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c10d460 con 0x7f0d0c107540 2024-04-25T01:53:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d0c104c10 0x7f0d0c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d0c104c10 0x7f0d0c197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46312/0 (socket says 172.21.15.31:46312) 2024-04-25T01:53:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3142965579 learned_addr learned my addr 172.21.15.31:0/3142965579 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d0bfff640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c1022e0 0x7f0d0c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d10f65640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d0c107540 0x7f0d0c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3142965579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d0c107540 msgr2=0x7f0d0c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d0c107540 0x7f0d0c19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3142965579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c1022e0 msgr2=0x7f0d0c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c1022e0 0x7f0d0c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3142965579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0c19f670 con 0x7f0d0c104c10 2024-04-25T01:53:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.766+0000 7f0d0bfff640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c1022e0 0x7f0d0c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.767+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d0c104c10 0x7f0d0c197f90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0d000314e0 tx=0x7f0d00031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.767+0000 7f0d097fa640 1 -- 172.21.15.31:0/3142965579 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0003f030 con 0x7f0d0c104c10 2024-04-25T01:53:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.767+0000 7f0d129c8640 1 -- 172.21.15.31:0/3142965579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d0c19f900 con 0x7f0d0c104c10 2024-04-25T01:53:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.767+0000 7f0d097fa640 1 -- 172.21.15.31:0/3142965579 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d000044b0 con 0x7f0d0c104c10 2024-04-25T01:53:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.767+0000 7f0d097fa640 1 -- 172.21.15.31:0/3142965579 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00035070 con 0x7f0d0c104c10 2024-04-25T01:53:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.767+0000 7f0d129c8640 1 -- 172.21.15.31:0/3142965579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d0c19fe40 con 0x7f0d0c104c10 2024-04-25T01:53:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.769+0000 7f0d097fa640 1 -- 172.21.15.31:0/3142965579 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0d00042030 con 0x7f0d0c104c10 2024-04-25T01:53:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d157b2740 1 Processor -- start 2024-04-25T01:53:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d157b2740 1 -- start start 2024-04-25T01:53:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481a440 0x55b20481a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481ae30 0x55b20481f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ffa0 0x55b204822430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d157b2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b20477df00 con 0x55b20481ffa0 2024-04-25T01:53:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d157b2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b204726750 con 0x55b20481ae30 2024-04-25T01:53:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d157b2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b204779ec0 con 0x55b20481a440 2024-04-25T01:53:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ffa0 0x55b204822430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ffa0 0x55b204822430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46324/0 (socket says 172.21.15.31:46324) 2024-04-25T01:53:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3817732240 learned_addr learned my addr 172.21.15.31:0/3817732240 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d097fa640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0cf40a2b90 0x7f0cf40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d10f65640 1 --2- 172.21.15.31:0/3817732240 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481a440 0x55b20481a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d0bfff640 1 --2- 172.21.15.31:0/3817732240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481ae30 0x55b20481f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d10f65640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0cf40a2b90 0x7f0cf40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3817732240 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481a440 msgr2=0x55b20481a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3817732240 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481a440 0x55b20481a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3817732240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481ae30 msgr2=0x55b20481f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3817732240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481ae30 0x55b20481f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.770+0000 7f0d097fa640 1 -- 172.21.15.31:0/3142965579 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f0d000ef690 con 0x7f0d0c104c10 2024-04-25T01:53:54.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.771+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3817732240 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2044922d0 con 0x55b20481ffa0 2024-04-25T01:53:54.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.771+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3817732240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ffa0 0x55b204822430 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0d000b8320 tx=0x7f0d0002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0d34ab49c765a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.771+0000 7f0d10f65640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0cf40a2b90 0x7f0cf40a5050 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55b20481b690 tx=0x7f0d04004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.771+0000 7f0cf27fc640 1 -- 172.21.15.31:0/3817732240 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00049070 con 0x55b20481ffa0 2024-04-25T01:53:54.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.771+0000 7f0cf27fc640 1 -- 172.21.15.31:0/3817732240 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d00003e20 con 0x55b20481ffa0 2024-04-25T01:53:54.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.771+0000 7f0cf27fc640 1 -- 172.21.15.31:0/3817732240 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0003b460 con 0x55b20481ffa0 2024-04-25T01:53:54.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.771+0000 7f0d157b2740 1 -- 172.21.15.31:0/3817732240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ffa0 msgr2=0x55b204822430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.771+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3817732240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ffa0 0x55b204822430 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0d000b8320 tx=0x7f0d0002a410 comp rx=0 tx=0).stop 2024-04-25T01:53:54.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 -- 172.21.15.31:0/3817732240 shutdown_connections 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3817732240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ffa0 0x55b204822430 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3817732240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481ae30 0x55b20481f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3817732240 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481a440 0x55b20481a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 -- 172.21.15.31:0/3817732240 >> 172.21.15.31:0/3817732240 conn(0x55b204815ab0 msgr2=0x55b204815eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 -- 172.21.15.31:0/3817732240 shutdown_connections 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 -- 172.21.15.31:0/3817732240 wait complete. 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 -- start start 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 Processor -- start 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481a440 0x55b2048244f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481ae30 0x55b204824a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b20482ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b20477df00 con 0x55b20481a440 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b204739cf0 con 0x55b20481ae30 2024-04-25T01:53:54.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b204822d10 con 0x55b20481ffa0 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d157b2740 10 client.0 _ll_register_callbacks cb 0x55b2047396e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d0bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481a440 0x55b2048244f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d0bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481a440 0x55b2048244f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46326/0 (socket says 172.21.15.31:46326) 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.772+0000 7f0d0bfff640 1 -- 172.21.15.31:0/3634968358 learned_addr learned my addr 172.21.15.31:0/3634968358 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481ae30 0x55b204824a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0d10f65640 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b20482ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0d0bfff640 1 -- 172.21.15.31:0/3634968358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 msgr2=0x55b20482ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0d0bfff640 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b20482ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0d0bfff640 1 -- 172.21.15.31:0/3634968358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481ae30 msgr2=0x55b204824a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0d0bfff640 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481ae30 0x55b204824a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0d0bfff640 1 -- 172.21.15.31:0/3634968358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b20455c6b0 con 0x55b20481a440 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0d0bfff640 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481a440 0x55b2048244f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc012470 tx=0x7f0cfc012930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec85b400c9dda61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0ceb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:53:54.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3634968358 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc01f060 con 0x55b20481a440 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3634968358 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f0cfc004460 con 0x55b20481a440 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0d157b2740 1 -- 172.21.15.31:0/3634968358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b20478b440 con 0x55b20481a440 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3634968358 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0cfc0035f0 con 0x55b20481a440 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0d157b2740 10 client.15069 fetch_fsmap learned FSMap version 26 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0d157b2740 1 -- 172.21.15.31:0/3634968358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b20482eab0 con 0x55b20481a440 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.773+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3634968358 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b20478b440 con 0x55b20481a440 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.774+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3634968358 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f0cfc019030 con 0x55b20481a440 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.774+0000 7f0d157b2740 10 client.15069 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.774+0000 7f0d157b2740 10 client.15069 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.774+0000 7f0d157b2740 20 client.15069 populate_metadata read hostname 'smithi031' 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.774+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55b2048311b0 0x55b204832620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.774+0000 7f0d157b2740 4 client.15069 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.774+0000 7f0d157b2740 1 -- 172.21.15.31:0/3634968358 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b204785790 con 0x55b2048311b0 2024-04-25T01:53:54.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.774+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55b2048311b0 0x55b204832620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.774+0000 7f0ceb7fe640 10 client.15069 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.774+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55b2048311b0 0x55b204832620 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5b03522982026073 server_cookie=b63f5f509c1a86e6 in_seq=0 out_seq=0 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3634968358 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b204785790 con 0x55b2048311b0 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0ceb7fe640 10 client.15069 handle_command_reply: tid=0 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0d157b2740 1 client.15069 shutdown 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0d157b2740 1 -- 172.21.15.31:0/3634968358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481a440 msgr2=0x55b2048244f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481a440 0x55b2048244f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc012470 tx=0x7f0cfc012930 comp rx=0 tx=0).stop 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0d157b2740 1 -- 172.21.15.31:0/3634968358 shutdown_connections 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55b2048311b0 0x55b204832620 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b20482ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0ceb7fe640 0 client.15069 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:53:54.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481ae30 0x55b204824a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3634968358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481a440 0x55b2048244f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0d157b2740 1 -- 172.21.15.31:0/3634968358 >> 172.21.15.31:0/3634968358 conn(0x55b204815ab0 msgr2=0x55b204814ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0d157b2740 1 -- 172.21.15.31:0/3634968358 shutdown_connections 2024-04-25T01:53:54.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.775+0000 7f0d157b2740 1 -- 172.21.15.31:0/3634968358 wait complete. 2024-04-25T01:53:54.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.776+0000 7f0d157b2740 20 client.15069 trim_cache size 0 max 16384 2024-04-25T01:53:54.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d157b2740 1 Processor -- start 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d157b2740 1 -- start start 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481a440 0x55b204818790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204818cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b204819210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d157b2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b204830960 con 0x55b20481ae30 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d157b2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b204831030 con 0x55b20481a440 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d157b2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b204831700 con 0x55b20481ffa0 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204818cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204818cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46342/0 (socket says 172.21.15.31:46342) 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/669734248 learned_addr learned my addr 172.21.15.31:0/669734248 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/669734248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 msgr2=0x55b204819210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.794+0000 7f0d10f65640 1 --2- 172.21.15.31:0/669734248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b204819210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d0bfff640 1 --2- 172.21.15.31:0/669734248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481a440 0x55b204818790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/669734248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b204819210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/669734248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481a440 msgr2=0x55b204818790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/669734248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481a440 0x55b204818790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/669734248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b20481b210 con 0x55b20481ae30 2024-04-25T01:53:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d10f65640 1 --2- 172.21.15.31:0/669734248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b204819210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d0bfff640 1 --2- 172.21.15.31:0/669734248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481a440 0x55b204818790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/669734248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204818cd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0d000bbe50 tx=0x7f0d000dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7183d6a9c6f6c7d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0cf1ffb640 1 -- 172.21.15.31:0/669734248 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00049040 con 0x55b20481ae30 2024-04-25T01:53:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0cf1ffb640 1 -- 172.21.15.31:0/669734248 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d000bc5c0 con 0x55b20481ae30 2024-04-25T01:53:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0cf1ffb640 1 -- 172.21.15.31:0/669734248 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d000f5120 con 0x55b20481ae30 2024-04-25T01:53:54.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d157b2740 1 -- 172.21.15.31:0/669734248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 msgr2=0x55b204818cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d157b2740 1 --2- 172.21.15.31:0/669734248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204818cd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0d000bbe50 tx=0x7f0d000dc040 comp rx=0 tx=0).stop 2024-04-25T01:53:54.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d157b2740 1 -- 172.21.15.31:0/669734248 shutdown_connections 2024-04-25T01:53:54.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d157b2740 1 --2- 172.21.15.31:0/669734248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b204819210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d157b2740 1 --2- 172.21.15.31:0/669734248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204818cd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d157b2740 1 --2- 172.21.15.31:0/669734248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481a440 0x55b204818790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d157b2740 1 -- 172.21.15.31:0/669734248 >> 172.21.15.31:0/669734248 conn(0x55b204815ab0 msgr2=0x55b20481c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.795+0000 7f0d157b2740 1 -- 172.21.15.31:0/669734248 shutdown_connections 2024-04-25T01:53:54.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d157b2740 1 -- 172.21.15.31:0/669734248 wait complete. 2024-04-25T01:53:54.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d157b2740 1 -- start start 2024-04-25T01:53:54.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d157b2740 1 Processor -- start 2024-04-25T01:53:54.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481a440 0x55b20481ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204897160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d157b2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b2048996f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d157b2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b2048324a0 con 0x55b20481ae30 2024-04-25T01:53:54.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d157b2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b204831dd0 con 0x55b20481a440 2024-04-25T01:53:54.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d157b2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b204832b70 con 0x55b20481ffa0 2024-04-25T01:53:54.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d157b2740 10 client.0 _ll_register_callbacks cb 0x7f0cfc009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:53:54.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204897160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204897160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46350/0 (socket says 172.21.15.31:46350) 2024-04-25T01:53:54.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3675324541 learned_addr learned my addr 172.21.15.31:0/3675324541 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d10f65640 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b2048996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.796+0000 7f0d0bfff640 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481a440 0x55b20481ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3675324541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 msgr2=0x55b2048996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b2048996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3675324541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481a440 msgr2=0x55b20481ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481a440 0x55b20481ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d0b7fe640 1 -- 172.21.15.31:0/3675324541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b204899f20 con 0x55b20481ae30 2024-04-25T01:53:54.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d10f65640 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b2048996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:54.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d0b7fe640 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204897160 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0d000ba170 tx=0x7f0d0000a3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b486ec893abd2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0ceb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:53:54.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3675324541 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00086060 con 0x55b20481ae30 2024-04-25T01:53:54.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3675324541 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f0d000bc120 con 0x55b20481ae30 2024-04-25T01:53:54.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d157b2740 1 -- 172.21.15.31:0/3675324541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d000bc5c0 con 0x55b20481ae30 2024-04-25T01:53:54.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3675324541 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0d00049040 con 0x55b20481ae30 2024-04-25T01:53:54.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d157b2740 10 client.15075 fetch_fsmap learned FSMap version 26 2024-04-25T01:53:54.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d157b2740 1 -- 172.21.15.31:0/3675324541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b20489b190 con 0x55b20481ae30 2024-04-25T01:53:54.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3675324541 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d000bc5c0 con 0x55b20481ae30 2024-04-25T01:53:54.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3675324541 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f0d000da030 con 0x55b20481ae30 2024-04-25T01:53:54.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d157b2740 10 client.15075 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:53:54.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d157b2740 10 client.15075 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:53:54.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.797+0000 7f0d157b2740 20 client.15075 populate_metadata read hostname 'smithi031' 2024-04-25T01:53:54.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.798+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55b2048311b0 0x55b20489e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.798+0000 7f0d157b2740 4 client.15075 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:53:54.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.798+0000 7f0d157b2740 1 -- 172.21.15.31:0/3675324541 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "status"}) -- 0x55b204703b80 con 0x55b2048311b0 2024-04-25T01:53:54.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.798+0000 7f0d0bfff640 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55b2048311b0 0x55b20489e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.798+0000 7f0ceb7fe640 10 client.15075 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:53:54.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.798+0000 7f0d0bfff640 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55b2048311b0 0x55b20489e6f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6c0eafa640226f11 server_cookie=f118ca3d60abb7b8 in_seq=0 out_seq=0 2024-04-25T01:53:54.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0ceb7fe640 1 -- 172.21.15.31:0/3675324541 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b204703b80 con 0x55b2048311b0 2024-04-25T01:53:54.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0ceb7fe640 10 client.15075 handle_command_reply: tid=0 2024-04-25T01:53:54.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 1 client.15075 shutdown 2024-04-25T01:53:54.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 1 -- 172.21.15.31:0/3675324541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 msgr2=0x55b204897160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204897160 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0d000ba170 tx=0x7f0d0000a3e0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 1 -- 172.21.15.31:0/3675324541 shutdown_connections 2024-04-25T01:53:54.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55b2048311b0 0x55b20489e6f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b20481ffa0 0x55b2048996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b20481ae30 0x55b204897160 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0ceb7fe640 0 client.15075 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:53:54.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 1 --2- 172.21.15.31:0/3675324541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b20481a440 0x55b20481ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 1 -- 172.21.15.31:0/3675324541 >> 172.21.15.31:0/3675324541 conn(0x55b204815ab0 msgr2=0x55b204814250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 1 -- 172.21.15.31:0/3675324541 shutdown_connections 2024-04-25T01:53:54.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 1 -- 172.21.15.31:0/3675324541 wait complete. 2024-04-25T01:53:54.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d157b2740 20 client.15075 trim_cache size 0 max 16384 2024-04-25T01:53:54.811 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:53:54.811 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:53:54.811 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 3, 2024-04-25T01:53:54.811 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24463, 2024-04-25T01:53:54.811 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T01:53:54.811 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T01:53:54.811 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T01:53:54.811 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 84.066295827999994, 2024-04-25T01:53:54.812 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T01:53:54.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2024-04-25T01:53:54.812 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:53:54.812 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 107.79302276200001 2024-04-25T01:53:54.812 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:53:54.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d129c8640 1 -- 172.21.15.31:0/3142965579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0cf40a2b90 msgr2=0x7f0cf40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d129c8640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0cf40a2b90 0x7f0cf40a5050 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55b20481b690 tx=0x7f0d04004a40 comp rx=0 tx=0).stop 2024-04-25T01:53:54.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d129c8640 1 -- 172.21.15.31:0/3142965579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d0c104c10 msgr2=0x7f0d0c197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.799+0000 7f0d129c8640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d0c104c10 0x7f0d0c197f90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0d000314e0 tx=0x7f0d00031a30 comp rx=0 tx=0).stop 2024-04-25T01:53:54.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.800+0000 7f0d129c8640 1 -- 172.21.15.31:0/3142965579 shutdown_connections 2024-04-25T01:53:54.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.800+0000 7f0d129c8640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0cf40a2b90 0x7f0cf40a5050 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.800+0000 7f0d129c8640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d0c107540 0x7f0d0c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.800+0000 7f0d129c8640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d0c104c10 0x7f0d0c197f90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.800+0000 7f0d129c8640 1 --2- 172.21.15.31:0/3142965579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c1022e0 0x7f0d0c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.800+0000 7f0d129c8640 1 -- 172.21.15.31:0/3142965579 >> 172.21.15.31:0/3142965579 conn(0x7f0d0c0fbc30 msgr2=0x7f0d0c102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.800+0000 7f0d129c8640 1 -- 172.21.15.31:0/3142965579 shutdown_connections 2024-04-25T01:53:54.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.800+0000 7f0d129c8640 1 -- 172.21.15.31:0/3142965579 wait complete. 2024-04-25T01:53:54.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T01:53:54.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.916+0000 7f6609377640 1 -- 172.21.15.31:0/2872283461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6604101350 msgr2=0x7f66041017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.916+0000 7f6609377640 1 --2- 172.21.15.31:0/2872283461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6604101350 0x7f66041017d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009ca0 tx=0x7f65fc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.916+0000 7f6609377640 1 -- 172.21.15.31:0/2872283461 shutdown_connections 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.916+0000 7f6609377640 1 --2- 172.21.15.31:0/2872283461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6604101350 0x7f66041017d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.916+0000 7f6609377640 1 --2- 172.21.15.31:0/2872283461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66041009b0 0x7f6604100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.916+0000 7f6609377640 1 --2- 172.21.15.31:0/2872283461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6604100050 0x7f6604100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.916+0000 7f6609377640 1 -- 172.21.15.31:0/2872283461 >> 172.21.15.31:0/2872283461 conn(0x7f66040fbbf0 msgr2=0x7f66040fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.916+0000 7f6609377640 1 -- 172.21.15.31:0/2872283461 shutdown_connections 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.916+0000 7f6609377640 1 -- 172.21.15.31:0/2872283461 wait complete. 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.917+0000 7f6609377640 1 Processor -- start 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.917+0000 7f6609377640 1 -- start start 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.917+0000 7f6609377640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6604100050 0x7f6604199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.917+0000 7f6609377640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66041009b0 0x7f660419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.917+0000 7f6609377640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6604101350 0x7f66041a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.917+0000 7f6609377640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f660410f4a0 con 0x7f6604100050 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.917+0000 7f6609377640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f660410f320 con 0x7f6604101350 2024-04-25T01:53:54.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.917+0000 7f6609377640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f660410f620 con 0x7f66041009b0 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.917+0000 7f6602ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6604100050 0x7f6604199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f6602ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6604100050 0x7f6604199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46372/0 (socket says 172.21.15.31:46372) 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f6602ffd640 1 -- 172.21.15.31:0/1396487842 learned_addr learned my addr 172.21.15.31:0/1396487842 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f66027fc640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66041009b0 0x7f660419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f66037fe640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6604101350 0x7f66041a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f6602ffd640 1 -- 172.21.15.31:0/1396487842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66041009b0 msgr2=0x7f660419a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f6602ffd640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66041009b0 0x7f660419a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f6602ffd640 1 -- 172.21.15.31:0/1396487842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6604101350 msgr2=0x7f66041a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f6602ffd640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6604101350 0x7f66041a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f6602ffd640 1 -- 172.21.15.31:0/1396487842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66041a17d0 con 0x7f6604100050 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f66027fc640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66041009b0 0x7f660419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:54.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f6602ffd640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6604100050 0x7f6604199bb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f65f400d950 tx=0x7f65f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f65ebfff640 1 -- 172.21.15.31:0/1396487842 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4017030 con 0x7f6604100050 2024-04-25T01:53:54.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f65ebfff640 1 -- 172.21.15.31:0/1396487842 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65f4004020 con 0x7f6604100050 2024-04-25T01:53:54.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f6609377640 1 -- 172.21.15.31:0/1396487842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66041a1ac0 con 0x7f6604100050 2024-04-25T01:53:54.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f65ebfff640 1 -- 172.21.15.31:0/1396487842 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f40047a0 con 0x7f6604100050 2024-04-25T01:53:54.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.918+0000 7f6609377640 1 -- 172.21.15.31:0/1396487842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66041a2060 con 0x7f6604100050 2024-04-25T01:53:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.921+0000 7f65ebfff640 1 -- 172.21.15.31:0/1396487842 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f65f40041d0 con 0x7f6604100050 2024-04-25T01:53:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.921+0000 7f660c166740 1 Processor -- start 2024-04-25T01:53:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.921+0000 7f660c166740 1 -- start start 2024-04-25T01:53:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.921+0000 7f660c166740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490e440 0x55dbb490e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.921+0000 7f660c166740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb4913930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.921+0000 7f660c166740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb4916430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.921+0000 7f660c166740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dbb4871f00 con 0x55dbb490ee30 2024-04-25T01:53:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.921+0000 7f660c166740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbb481a750 con 0x55dbb490e440 2024-04-25T01:53:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.921+0000 7f660c166740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dbb486dec0 con 0x55dbb4913fa0 2024-04-25T01:53:54.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.921+0000 7f6602ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb4913930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f66037fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490e440 0x55dbb490e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f6602ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb4913930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46386/0 (socket says 172.21.15.31:46386) 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f6602ffd640 1 -- 172.21.15.31:0/765669132 learned_addr learned my addr 172.21.15.31:0/765669132 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f65ebfff640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65ec0a2c10 0x7f65ec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f66027fc640 1 --2- 172.21.15.31:0/765669132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb4916430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f6602ffd640 1 -- 172.21.15.31:0/765669132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 msgr2=0x55dbb4916430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f6602ffd640 1 --2- 172.21.15.31:0/765669132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb4916430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f6602ffd640 1 -- 172.21.15.31:0/765669132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490e440 msgr2=0x55dbb490e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f6602ffd640 1 --2- 172.21.15.31:0/765669132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490e440 0x55dbb490e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f6602ffd640 1 -- 172.21.15.31:0/765669132 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbb45862d0 con 0x55dbb490ee30 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f66027fc640 1 --2- 172.21.15.31:0/765669132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb4916430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f66037fe640 1 --2- 172.21.15.31:0/765669132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490e440 0x55dbb490e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f66037fe640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65ec0a2c10 0x7f65ec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f65ebfff640 1 -- 172.21.15.31:0/1396487842 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f65f40ceba0 con 0x7f6604100050 2024-04-25T01:53:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f6602ffd640 1 --2- 172.21.15.31:0/765669132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb4913930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f65f4088930 tx=0x7f65f40908b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9620a5ea1e3943 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f65e97fa640 1 -- 172.21.15.31:0/765669132 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f409b1a0 con 0x55dbb490ee30 2024-04-25T01:53:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f66037fe640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65ec0a2c10 0x7f65ec0a50d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x55dbb490f690 tx=0x7f65fc0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f65e97fa640 1 -- 172.21.15.31:0/765669132 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65f40664b0 con 0x55dbb490ee30 2024-04-25T01:53:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.922+0000 7f65e97fa640 1 -- 172.21.15.31:0/765669132 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4088790 con 0x55dbb490ee30 2024-04-25T01:53:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.923+0000 7f660c166740 1 -- 172.21.15.31:0/765669132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 msgr2=0x55dbb4913930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.923+0000 7f660c166740 1 --2- 172.21.15.31:0/765669132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb4913930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f65f4088930 tx=0x7f65f40908b0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.923+0000 7f660c166740 1 -- 172.21.15.31:0/765669132 shutdown_connections 2024-04-25T01:53:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.923+0000 7f660c166740 1 --2- 172.21.15.31:0/765669132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb4916430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.923+0000 7f660c166740 1 --2- 172.21.15.31:0/765669132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb4913930 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.923+0000 7f660c166740 1 --2- 172.21.15.31:0/765669132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490e440 0x55dbb490e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.923+0000 7f660c166740 1 -- 172.21.15.31:0/765669132 >> 172.21.15.31:0/765669132 conn(0x55dbb4909ab0 msgr2=0x55dbb4909eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.923+0000 7f660c166740 1 -- 172.21.15.31:0/765669132 shutdown_connections 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.923+0000 7f660c166740 1 -- 172.21.15.31:0/765669132 wait complete. 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.923+0000 7f660c166740 1 -- start start 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.923+0000 7f660c166740 1 Processor -- start 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f660c166740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490e440 0x55dbb49187d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f660c166740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 0x55dbb4918d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f660c166740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb491fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f660c166740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dbb4916ef0 con 0x55dbb490e440 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f660c166740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbb482dcf0 con 0x55dbb490ee30 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f660c166740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dbb4917070 con 0x55dbb4913fa0 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f660c166740 10 client.0 _ll_register_callbacks cb 0x55dbb482d6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:53:54.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f6602ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490e440 0x55dbb49187d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f6602ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490e440 0x55dbb49187d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46396/0 (socket says 172.21.15.31:46396) 2024-04-25T01:53:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f6602ffd640 1 -- 172.21.15.31:0/66164218 learned_addr learned my addr 172.21.15.31:0/66164218 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f66037fe640 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb491fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f66027fc640 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 0x55dbb4918d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f6602ffd640 1 -- 172.21.15.31:0/66164218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 msgr2=0x55dbb491fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f6602ffd640 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb491fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f6602ffd640 1 -- 172.21.15.31:0/66164218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 msgr2=0x55dbb4918d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f6602ffd640 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 0x55dbb4918d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f6602ffd640 1 -- 172.21.15.31:0/66164218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbb46506b0 con 0x55dbb490e440 2024-04-25T01:53:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f66037fe640 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb491fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:54.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f66027fc640 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 0x55dbb4918d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f6602ffd640 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490e440 0x55dbb49187d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f65f409b0d0 tx=0x7f65f409f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fadf0a4926c06937 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f65e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f65e6ff5640 1 -- 172.21.15.31:0/66164218 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4066010 con 0x55dbb490e440 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f65e6ff5640 1 -- 172.21.15.31:0/66164218 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f65f40661b0 con 0x55dbb490e440 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f660c166740 1 -- 172.21.15.31:0/66164218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65f40664b0 con 0x55dbb490e440 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f65e6ff5640 1 -- 172.21.15.31:0/66164218 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f65f409ad30 con 0x55dbb490e440 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f660c166740 10 client.15087 fetch_fsmap learned FSMap version 26 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.924+0000 7f660c166740 1 -- 172.21.15.31:0/66164218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dbb4922ca0 con 0x55dbb490e440 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.925+0000 7f65e6ff5640 1 -- 172.21.15.31:0/66164218 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65f40d4070 con 0x55dbb490e440 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.925+0000 7f65e6ff5640 1 -- 172.21.15.31:0/66164218 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f65f406c070 con 0x55dbb490e440 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.925+0000 7f660c166740 10 client.15087 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.925+0000 7f660c166740 10 client.15087 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.925+0000 7f660c166740 20 client.15087 populate_metadata read hostname 'smithi031' 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.925+0000 7f660c166740 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55dbb4924e30 0x55dbb49262a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.925+0000 7f660c166740 4 client.15087 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:53:54.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.925+0000 7f660c166740 1 -- 172.21.15.31:0/66164218 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbb487f440 con 0x55dbb4924e30 2024-04-25T01:53:54.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.925+0000 7f66027fc640 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55dbb4924e30 0x55dbb49262a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.926+0000 7f65e6ff5640 10 client.15087 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:53:54.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.926+0000 7f66027fc640 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55dbb4924e30 0x55dbb49262a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c83c9924e3eeccf server_cookie=3a8ea50eebb322b3 in_seq=0 out_seq=0 2024-04-25T01:53:54.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.926+0000 7f65e6ff5640 1 -- 172.21.15.31:0/66164218 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dbb487f440 con 0x55dbb4924e30 2024-04-25T01:53:54.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.926+0000 7f65e6ff5640 10 client.15087 handle_command_reply: tid=0 2024-04-25T01:53:54.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.926+0000 7f660c166740 1 client.15087 shutdown 2024-04-25T01:53:54.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f660c166740 1 -- 172.21.15.31:0/66164218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490e440 msgr2=0x55dbb49187d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f660c166740 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490e440 0x55dbb49187d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f65f409b0d0 tx=0x7f65f409f7f0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f660c166740 1 -- 172.21.15.31:0/66164218 shutdown_connections 2024-04-25T01:53:54.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f660c166740 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55dbb4924e30 0x55dbb49262a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f660c166740 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb491fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f660c166740 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 0x55dbb4918d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f65e6ff5640 0 client.15087 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:53:54.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f660c166740 1 --2- 172.21.15.31:0/66164218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490e440 0x55dbb49187d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f660c166740 1 -- 172.21.15.31:0/66164218 >> 172.21.15.31:0/66164218 conn(0x55dbb4909ab0 msgr2=0x55dbb4908ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f660c166740 1 -- 172.21.15.31:0/66164218 shutdown_connections 2024-04-25T01:53:54.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f660c166740 1 -- 172.21.15.31:0/66164218 wait complete. 2024-04-25T01:53:54.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.927+0000 7f660c166740 20 client.15087 trim_cache size 0 max 16384 2024-04-25T01:53:54.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f660c166740 1 Processor -- start 2024-04-25T01:53:54.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f660c166740 1 -- start start 2024-04-25T01:53:54.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f660c166740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490e440 0x55dbb4908f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f660c166740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb49203e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f660c166740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb4913860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f660c166740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dbb49245e0 con 0x55dbb490ee30 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f660c166740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbb4924cb0 con 0x55dbb490e440 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f660c166740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dbb4925380 con 0x55dbb4913fa0 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f66027fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb49203e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f66027fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb49203e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46408/0 (socket says 172.21.15.31:46408) 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f66027fc640 1 -- 172.21.15.31:0/1532063020 learned_addr learned my addr 172.21.15.31:0/1532063020 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f6602ffd640 1 --2- 172.21.15.31:0/1532063020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490e440 0x55dbb4908f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f66027fc640 1 -- 172.21.15.31:0/1532063020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 msgr2=0x55dbb4913860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f66027fc640 1 --2- 172.21.15.31:0/1532063020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb4913860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f66027fc640 1 -- 172.21.15.31:0/1532063020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490e440 msgr2=0x55dbb4908f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f66027fc640 1 --2- 172.21.15.31:0/1532063020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490e440 0x55dbb4908f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f66027fc640 1 -- 172.21.15.31:0/1532063020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbb490f210 con 0x55dbb490ee30 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.945+0000 7f66027fc640 1 --2- 172.21.15.31:0/1532063020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb49203e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55dbb490a980 tx=0x7f65f800f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d14c9088b1d124 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f65e8ff9640 1 -- 172.21.15.31:0/1532063020 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f800fe30 con 0x55dbb490ee30 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f65e8ff9640 1 -- 172.21.15.31:0/1532063020 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65f801c920 con 0x55dbb490ee30 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f65e8ff9640 1 -- 172.21.15.31:0/1532063020 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8004b20 con 0x55dbb490ee30 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f660c166740 1 -- 172.21.15.31:0/1532063020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 msgr2=0x55dbb49203e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f660c166740 1 --2- 172.21.15.31:0/1532063020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb49203e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55dbb490a980 tx=0x7f65f800f7f0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f660c166740 1 -- 172.21.15.31:0/1532063020 shutdown_connections 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f660c166740 1 --2- 172.21.15.31:0/1532063020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb4913fa0 0x55dbb4913860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f660c166740 1 --2- 172.21.15.31:0/1532063020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb490ee30 0x55dbb49203e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f660c166740 1 --2- 172.21.15.31:0/1532063020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490e440 0x55dbb4908f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f660c166740 1 -- 172.21.15.31:0/1532063020 >> 172.21.15.31:0/1532063020 conn(0x55dbb4909ab0 msgr2=0x55dbb4909e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f660c166740 1 -- 172.21.15.31:0/1532063020 shutdown_connections 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f660c166740 1 -- 172.21.15.31:0/1532063020 wait complete. 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.946+0000 7f660c166740 1 -- start start 2024-04-25T01:53:54.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f660c166740 1 Processor -- start 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f660c166740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb490e440 0x55dbb4912be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f660c166740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 0x55dbb491dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f660c166740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb4913fa0 0x55dbb491e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f660c166740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dbb4926120 con 0x55dbb4913fa0 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f660c166740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dbb4925a50 con 0x55dbb490ee30 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f660c166740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dbb49267f0 con 0x55dbb490e440 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f660c166740 10 client.0 _ll_register_callbacks cb 0x7f65f40050c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f66037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb4913fa0 0x55dbb491e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f66037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb4913fa0 0x55dbb491e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46410/0 (socket says 172.21.15.31:46410) 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f66037fe640 1 -- 172.21.15.31:0/814052191 learned_addr learned my addr 172.21.15.31:0/814052191 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f66027fc640 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 0x55dbb491dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f66037fe640 1 -- 172.21.15.31:0/814052191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb490e440 msgr2=0x55dbb4912be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f66037fe640 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb490e440 0x55dbb4912be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f66037fe640 1 -- 172.21.15.31:0/814052191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 msgr2=0x55dbb491dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:54.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f66037fe640 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 0x55dbb491dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f66037fe640 1 -- 172.21.15.31:0/814052191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbb491ea60 con 0x55dbb4913fa0 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f66027fc640 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 0x55dbb491dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f66037fe640 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb4913fa0 0x55dbb491e230 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009ca0 tx=0x7f65fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48de0c2cb0a8e5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f65e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.947+0000 7f65e6ff5640 1 -- 172.21.15.31:0/814052191 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc004040 con 0x55dbb4913fa0 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f660c166740 1 -- 172.21.15.31:0/814052191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dbb4879790 con 0x55dbb4913fa0 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f65e6ff5640 1 -- 172.21.15.31:0/814052191 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f65fc0041e0 con 0x55dbb4913fa0 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f65e6ff5640 1 -- 172.21.15.31:0/814052191 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f65fc0043e0 con 0x55dbb4913fa0 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f660c166740 10 client.15093 fetch_fsmap learned FSMap version 26 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f660c166740 1 -- 172.21.15.31:0/814052191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dbb498b380 con 0x55dbb4913fa0 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f65e6ff5640 1 -- 172.21.15.31:0/814052191 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55dbb4879790 con 0x55dbb4913fa0 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f65e6ff5640 1 -- 172.21.15.31:0/814052191 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f65fc02f4c0 con 0x55dbb4913fa0 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f660c166740 10 client.15093 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:53:54.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f660c166740 10 client.15093 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f660c166740 20 client.15093 populate_metadata read hostname 'smithi031' 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f660c166740 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55dbb4990020 0x55dbb498c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f660c166740 4 client.15093 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f660c166740 1 -- 172.21.15.31:0/814052191 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "status"}) -- 0x55dbb47f7b80 con 0x55dbb4990020 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.948+0000 7f6602ffd640 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55dbb4990020 0x55dbb498c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f65e6ff5640 10 client.15093 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f6602ffd640 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55dbb4990020 0x55dbb498c460 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=be05165407e84062 server_cookie=a3fb602308de6883 in_seq=0 out_seq=0 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f65e6ff5640 1 -- 172.21.15.31:0/814052191 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55dbb47f7b80 con 0x55dbb4990020 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f65e6ff5640 10 client.15093 handle_command_reply: tid=0 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 1 client.15093 shutdown 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 1 -- 172.21.15.31:0/814052191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb4913fa0 msgr2=0x55dbb491e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb4913fa0 0x55dbb491e230 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009ca0 tx=0x7f65fc004a40 comp rx=0 tx=0).stop 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 1 -- 172.21.15.31:0/814052191 shutdown_connections 2024-04-25T01:53:54.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55dbb4990020 0x55dbb498c460 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dbb4913fa0 0x55dbb491e230 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f65e6ff5640 0 client.15093 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:53:54.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dbb490ee30 0x55dbb491dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 1 --2- 172.21.15.31:0/814052191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dbb490e440 0x55dbb4912be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 1 -- 172.21.15.31:0/814052191 >> 172.21.15.31:0/814052191 conn(0x55dbb4909ab0 msgr2=0x55dbb4904690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 1 -- 172.21.15.31:0/814052191 shutdown_connections 2024-04-25T01:53:54.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 1 -- 172.21.15.31:0/814052191 wait complete. 2024-04-25T01:53:54.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.949+0000 7f660c166740 20 client.15093 trim_cache size 0 max 16384 2024-04-25T01:53:54.956 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:53:54.956 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:53:54.956 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2024-04-25T01:53:54.957 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24473, 2024-04-25T01:53:54.957 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T01:53:54.957 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T01:53:54.957 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T01:53:54.957 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 85.224187306000005, 2024-04-25T01:53:54.957 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T01:53:54.957 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2024-04-25T01:53:54.957 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:53:54.957 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 104.944849811 2024-04-25T01:53:54.957 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:53:54.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 -- 172.21.15.31:0/1396487842 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65ec0a2c10 msgr2=0x7f65ec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65ec0a2c10 0x7f65ec0a50d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x55dbb490f690 tx=0x7f65fc0095c0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 -- 172.21.15.31:0/1396487842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6604100050 msgr2=0x7f6604199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:54.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6604100050 0x7f6604199bb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f65f400d950 tx=0x7f65f400de10 comp rx=0 tx=0).stop 2024-04-25T01:53:54.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 -- 172.21.15.31:0/1396487842 shutdown_connections 2024-04-25T01:53:54.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f65ec0a2c10 0x7f65ec0a50d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6604101350 0x7f66041a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66041009b0 0x7f660419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 --2- 172.21.15.31:0/1396487842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6604100050 0x7f6604199bb0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:54.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 -- 172.21.15.31:0/1396487842 >> 172.21.15.31:0/1396487842 conn(0x7f66040fbbf0 msgr2=0x7f66040fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:54.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 -- 172.21.15.31:0/1396487842 shutdown_connections 2024-04-25T01:53:54.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:54.950+0000 7f6609377640 1 -- 172.21.15.31:0/1396487842 wait complete. 2024-04-25T01:53:54.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T01:53:55.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.062+0000 7fd702b4f640 1 -- 172.21.15.31:0/4018478310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc101870 msgr2=0x7fd6fc101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.062+0000 7fd702b4f640 1 --2- 172.21.15.31:0/4018478310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc101870 0x7fd6fc101cf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009900 tx=0x7fd6e802a410 comp rx=0 tx=0).stop 2024-04-25T01:53:55.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.062+0000 7fd702b4f640 1 -- 172.21.15.31:0/4018478310 shutdown_connections 2024-04-25T01:53:55.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.062+0000 7fd702b4f640 1 --2- 172.21.15.31:0/4018478310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6fc102230 0x7fd6fc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.062+0000 7fd702b4f640 1 --2- 172.21.15.31:0/4018478310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc101870 0x7fd6fc101cf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.062+0000 7fd702b4f640 1 --2- 172.21.15.31:0/4018478310 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6fc100680 0x7fd6fc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.062+0000 7fd702b4f640 1 -- 172.21.15.31:0/4018478310 >> 172.21.15.31:0/4018478310 conn(0x7fd6fc0fbbf0 msgr2=0x7fd6fc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.062+0000 7fd702b4f640 1 -- 172.21.15.31:0/4018478310 shutdown_connections 2024-04-25T01:53:55.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.062+0000 7fd702b4f640 1 -- 172.21.15.31:0/4018478310 wait complete. 2024-04-25T01:53:55.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd702b4f640 1 Processor -- start 2024-04-25T01:53:55.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd702b4f640 1 -- start start 2024-04-25T01:53:55.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd702b4f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6fc100680 0x7fd6fc069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd702b4f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6fc101870 0x7fd6fc19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd702b4f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc102230 0x7fd6fc19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd702b4f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd6fc10c190 con 0x7fd6fc102230 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd702b4f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd6fc10c010 con 0x7fd6fc100680 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd702b4f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd6fc10c310 con 0x7fd6fc101870 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd7010ec640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc102230 0x7fd6fc19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd7010ec640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc102230 0x7fd6fc19ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46424/0 (socket says 172.21.15.31:46424) 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd7010ec640 1 -- 172.21.15.31:0/3925415392 learned_addr learned my addr 172.21.15.31:0/3925415392 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd7008eb640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6fc100680 0x7fd6fc069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd7010ec640 1 -- 172.21.15.31:0/3925415392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6fc101870 msgr2=0x7fd6fc19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd7010ec640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6fc101870 0x7fd6fc19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd7010ec640 1 -- 172.21.15.31:0/3925415392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6fc100680 msgr2=0x7fd6fc069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd7010ec640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6fc100680 0x7fd6fc069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd7010ec640 1 -- 172.21.15.31:0/3925415392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6fc19f600 con 0x7fd6fc102230 2024-04-25T01:53:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.063+0000 7fd7008eb640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6fc100680 0x7fd6fc069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.064+0000 7fd7010ec640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc102230 0x7fd6fc19ed40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd6f400eb10 tx=0x7fd6f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.064+0000 7fd6f9ffb640 1 -- 172.21.15.31:0/3925415392 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4004b20 con 0x7fd6fc102230 2024-04-25T01:53:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.064+0000 7fd6f9ffb640 1 -- 172.21.15.31:0/3925415392 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6f4004cc0 con 0x7fd6fc102230 2024-04-25T01:53:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.064+0000 7fd702b4f640 1 -- 172.21.15.31:0/3925415392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6fc19f8f0 con 0x7fd6fc102230 2024-04-25T01:53:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.064+0000 7fd6f9ffb640 1 -- 172.21.15.31:0/3925415392 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4010690 con 0x7fd6fc102230 2024-04-25T01:53:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.064+0000 7fd702b4f640 1 -- 172.21.15.31:0/3925415392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6fc19fe40 con 0x7fd6fc102230 2024-04-25T01:53:55.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.066+0000 7fd6f9ffb640 1 -- 172.21.15.31:0/3925415392 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd6f4005020 con 0x7fd6fc102230 2024-04-25T01:53:55.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd70585b740 1 Processor -- start 2024-04-25T01:53:55.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd70585b740 1 -- start start 2024-04-25T01:53:55.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd70585b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7ad440 0x55c9eb7ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd70585b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ade30 0x55c9eb7b2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd70585b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7b5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd70585b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb710f00 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd70585b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb6b9750 con 0x55c9eb7ad440 2024-04-25T01:53:55.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd70585b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb70cec0 con 0x55c9eb7ade30 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd6fbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd6f9ffb640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd6ec0a2c10 0x7fd6ec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd6fbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46438/0 (socket says 172.21.15.31:46438) 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd6fbfff640 1 -- 172.21.15.31:0/527979712 learned_addr learned my addr 172.21.15.31:0/527979712 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd7008eb640 1 --2- 172.21.15.31:0/527979712 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ade30 0x55c9eb7b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd7010ec640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd6ec0a2c10 0x7fd6ec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/527979712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7ad440 0x55c9eb7ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd6f9ffb640 1 -- 172.21.15.31:0/3925415392 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fd6f40cf690 con 0x7fd6fc102230 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd6fbfff640 1 -- 172.21.15.31:0/527979712 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ade30 msgr2=0x55c9eb7b2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/527979712 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ade30 0x55c9eb7b2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd6fbfff640 1 -- 172.21.15.31:0/527979712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7ad440 msgr2=0x55c9eb7ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/527979712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7ad440 0x55c9eb7ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.067+0000 7fd6fbfff640 1 -- 172.21.15.31:0/527979712 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9eb4252d0 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/527979712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7b5430 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8005ab0 tx=0x7fd6e8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c991f213f46178 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd7010ec640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd6ec0a2c10 0x7fd6ec0a50d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd6f409beb0 tx=0x7fd6f40c6ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd6e6ffd640 1 -- 172.21.15.31:0/527979712 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8044070 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd6e6ffd640 1 -- 172.21.15.31:0/527979712 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6e8031780 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd6e6ffd640 1 -- 172.21.15.31:0/527979712 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8035070 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd70585b740 1 -- 172.21.15.31:0/527979712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 msgr2=0x55c9eb7b5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd70585b740 1 --2- 172.21.15.31:0/527979712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7b5430 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8005ab0 tx=0x7fd6e8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd70585b740 1 -- 172.21.15.31:0/527979712 shutdown_connections 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd70585b740 1 --2- 172.21.15.31:0/527979712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7b5430 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd70585b740 1 --2- 172.21.15.31:0/527979712 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ade30 0x55c9eb7b2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd70585b740 1 --2- 172.21.15.31:0/527979712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7ad440 0x55c9eb7ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.068+0000 7fd70585b740 1 -- 172.21.15.31:0/527979712 >> 172.21.15.31:0/527979712 conn(0x55c9eb7a8ab0 msgr2=0x55c9eb7a8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd70585b740 1 -- 172.21.15.31:0/527979712 shutdown_connections 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd70585b740 1 -- 172.21.15.31:0/527979712 wait complete. 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd70585b740 1 -- start start 2024-04-25T01:53:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd70585b740 1 Processor -- start 2024-04-25T01:53:55.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd70585b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 0x55c9eb7b7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd70585b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd70585b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7bee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd70585b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb7b5ef0 con 0x55c9eb7ade30 2024-04-25T01:53:55.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd70585b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb6cccf0 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd70585b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb7b6070 con 0x55c9eb7ad440 2024-04-25T01:53:55.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd70585b740 10 client.0 _ll_register_callbacks cb 0x55c9eb6cc6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:53:55.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd7008eb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd7008eb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7b7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46452/0 (socket says 172.21.15.31:46452) 2024-04-25T01:53:55.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd7008eb640 1 -- 172.21.15.31:0/1760348917 learned_addr learned my addr 172.21.15.31:0/1760348917 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7bee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.069+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 0x55c9eb7b7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd7008eb640 1 -- 172.21.15.31:0/1760348917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 msgr2=0x55c9eb7b7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd7008eb640 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 0x55c9eb7b7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd7008eb640 1 -- 172.21.15.31:0/1760348917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7b2fa0 msgr2=0x55c9eb7bee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd7008eb640 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7bee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd7008eb640 1 -- 172.21.15.31:0/1760348917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9eb4ef6b0 con 0x55c9eb7ade30 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd7008eb640 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7b7dd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000cda0 tx=0x7fd6f00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51229319a545ebb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd6dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd6dbfff640 1 -- 172.21.15.31:0/1760348917 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f00119d0 con 0x55c9eb7ade30 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd6dbfff640 1 -- 172.21.15.31:0/1760348917 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fd6f0011b70 con 0x55c9eb7ade30 2024-04-25T01:53:55.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd70585b740 1 -- 172.21.15.31:0/1760348917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6e8031780 con 0x55c9eb7ade30 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd6dbfff640 1 -- 172.21.15.31:0/1760348917 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd6f0011d70 con 0x55c9eb7ade30 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd70585b740 10 client.15105 fetch_fsmap learned FSMap version 26 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd70585b740 1 -- 172.21.15.31:0/1760348917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c9eb7c1ec0 con 0x55c9eb7ade30 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.070+0000 7fd6dbfff640 1 -- 172.21.15.31:0/1760348917 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6e8031780 con 0x55c9eb7ade30 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.071+0000 7fd6dbfff640 1 -- 172.21.15.31:0/1760348917 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd6f0018070 con 0x55c9eb7ade30 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.071+0000 7fd70585b740 10 client.15105 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.071+0000 7fd70585b740 10 client.15105 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.071+0000 7fd70585b740 20 client.15105 populate_metadata read hostname 'smithi031' 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.071+0000 7fd70585b740 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55c9eb7c45c0 0x55c9eb7c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.071+0000 7fd70585b740 4 client.15105 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.071+0000 7fd70585b740 1 -- 172.21.15.31:0/1760348917 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9eb71e440 con 0x55c9eb7c45c0 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.071+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55c9eb7c45c0 0x55c9eb7c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.071+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55c9eb7c45c0 0x55c9eb7c5a30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=350f5014ac689ecc server_cookie=914d1b06f520c59c in_seq=0 out_seq=0 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.071+0000 7fd6dbfff640 10 client.15105 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd6dbfff640 1 -- 172.21.15.31:0/1760348917 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c9eb71e440 con 0x55c9eb7c45c0 2024-04-25T01:53:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd6dbfff640 10 client.15105 handle_command_reply: tid=0 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd70585b740 1 client.15105 shutdown 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd70585b740 1 -- 172.21.15.31:0/1760348917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 msgr2=0x55c9eb7b7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd70585b740 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7b7dd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000cda0 tx=0x7fd6f00039c0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd70585b740 1 -- 172.21.15.31:0/1760348917 shutdown_connections 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd70585b740 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55c9eb7c45c0 0x55c9eb7c5a30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd70585b740 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7bee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd6dbfff640 0 client.15105 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd70585b740 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7b7dd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd70585b740 1 --2- 172.21.15.31:0/1760348917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 0x55c9eb7b7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd70585b740 1 -- 172.21.15.31:0/1760348917 >> 172.21.15.31:0/1760348917 conn(0x55c9eb7a8ab0 msgr2=0x55c9eb7a7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd70585b740 1 -- 172.21.15.31:0/1760348917 shutdown_connections 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.072+0000 7fd70585b740 1 -- 172.21.15.31:0/1760348917 wait complete. 2024-04-25T01:53:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.073+0000 7fd70585b740 20 client.15105 trim_cache size 0 max 16384 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd70585b740 1 Processor -- start 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd70585b740 1 -- start start 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd70585b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 0x55c9eb7af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd70585b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd70585b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7a7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd70585b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb7c3d70 con 0x55c9eb7ade30 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd70585b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb7c4440 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd70585b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb7c4b10 con 0x55c9eb7ad440 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd7008eb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd7008eb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7af7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46454/0 (socket says 172.21.15.31:46454) 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd7008eb640 1 -- 172.21.15.31:0/154255632 learned_addr learned my addr 172.21.15.31:0/154255632 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd7008eb640 1 -- 172.21.15.31:0/154255632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 msgr2=0x55c9eb7af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd7008eb640 1 --2- 172.21.15.31:0/154255632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 0x55c9eb7af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd7008eb640 1 -- 172.21.15.31:0/154255632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7b2fa0 msgr2=0x55c9eb7a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd7008eb640 1 --2- 172.21.15.31:0/154255632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7a7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd7008eb640 1 -- 172.21.15.31:0/154255632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9eb7ae210 con 0x55c9eb7ade30 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd7008eb640 1 --2- 172.21.15.31:0/154255632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7af7d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0025590 tx=0x7fd6f000c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab140f5eb7b0624 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd6e67fc640 1 -- 172.21.15.31:0/154255632 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f00119d0 con 0x55c9eb7ade30 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd6e67fc640 1 -- 172.21.15.31:0/154255632 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6f0003f70 con 0x55c9eb7ade30 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.091+0000 7fd6e67fc640 1 -- 172.21.15.31:0/154255632 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0025b60 con 0x55c9eb7ade30 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.092+0000 7fd70585b740 1 -- 172.21.15.31:0/154255632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 msgr2=0x55c9eb7af7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.092+0000 7fd70585b740 1 --2- 172.21.15.31:0/154255632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7af7d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0025590 tx=0x7fd6f000c540 comp rx=0 tx=0).stop 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.092+0000 7fd70585b740 1 -- 172.21.15.31:0/154255632 shutdown_connections 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.092+0000 7fd70585b740 1 --2- 172.21.15.31:0/154255632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb7a7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.092+0000 7fd70585b740 1 --2- 172.21.15.31:0/154255632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7ade30 0x55c9eb7af7d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.092+0000 7fd70585b740 1 --2- 172.21.15.31:0/154255632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 0x55c9eb7af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.092+0000 7fd70585b740 1 -- 172.21.15.31:0/154255632 >> 172.21.15.31:0/154255632 conn(0x55c9eb7a8ab0 msgr2=0x55c9eb7a3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.092+0000 7fd70585b740 1 -- 172.21.15.31:0/154255632 shutdown_connections 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.092+0000 7fd70585b740 1 -- 172.21.15.31:0/154255632 wait complete. 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.092+0000 7fd70585b740 1 -- start start 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.092+0000 7fd70585b740 1 Processor -- start 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd70585b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 0x55c9eb8267a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd70585b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7ade30 0x55c9eb826ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd70585b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb82d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd70585b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb7c58b0 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd70585b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb7c51e0 con 0x55c9eb7ade30 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd70585b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c9eb7c5f80 con 0x55c9eb7ad440 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd70585b740 10 client.0 _ll_register_callbacks cb 0x7fd6f000c9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6fbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb82d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6fbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb82d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46462/0 (socket says 172.21.15.31:46462) 2024-04-25T01:53:55.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6fbfff640 1 -- 172.21.15.31:0/2590079892 learned_addr learned my addr 172.21.15.31:0/2590079892 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 0x55c9eb8267a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd7008eb640 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7ade30 0x55c9eb826ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6fbfff640 1 -- 172.21.15.31:0/2590079892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 msgr2=0x55c9eb8267a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 0x55c9eb8267a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6fbfff640 1 -- 172.21.15.31:0/2590079892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7ade30 msgr2=0x55c9eb826ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7ade30 0x55c9eb826ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6fbfff640 1 -- 172.21.15.31:0/2590079892 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9eb82dc80 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd7008eb640 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7ade30 0x55c9eb826ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:55.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6fbfff640 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb82d6c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80096c0 tx=0x7fd6e8040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa31834da8af015 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6dbfff640 1 -- 172.21.15.31:0/2590079892 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8035040 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd6dbfff640 1 -- 172.21.15.31:0/2590079892 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fd6e803d480 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.093+0000 7fd70585b740 1 -- 172.21.15.31:0/2590079892 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6f0003f70 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd6dbfff640 1 -- 172.21.15.31:0/2590079892 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd6e8062070 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd70585b740 10 client.15111 fetch_fsmap learned FSMap version 26 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd70585b740 1 -- 172.21.15.31:0/2590079892 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c9eb82ef80 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd6dbfff640 1 -- 172.21.15.31:0/2590079892 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6f0003f70 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd6dbfff640 1 -- 172.21.15.31:0/2590079892 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd6e804a050 con 0x55c9eb7b2fa0 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd70585b740 10 client.15111 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd70585b740 10 client.15111 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd70585b740 20 client.15111 populate_metadata read hostname 'smithi031' 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd70585b740 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55c9eb833650 0x55c9eb832360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd70585b740 4 client.15111 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd70585b740 1 -- 172.21.15.31:0/2590079892 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "status"}) -- 0x55c9eb718790 con 0x55c9eb833650 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.094+0000 7fd7008eb640 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55c9eb833650 0x55c9eb832360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd6dbfff640 10 client.15111 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd7008eb640 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55c9eb833650 0x55c9eb832360 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c466d205633dd9c server_cookie=6ad2ce9ec0ac8b22 in_seq=0 out_seq=0 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd6dbfff640 1 -- 172.21.15.31:0/2590079892 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c9eb718790 con 0x55c9eb833650 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd6dbfff640 10 client.15111 handle_command_reply: tid=0 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 1 client.15111 shutdown 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 1 -- 172.21.15.31:0/2590079892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 msgr2=0x55c9eb82d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb82d6c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80096c0 tx=0x7fd6e8040040 comp rx=0 tx=0).stop 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 1 -- 172.21.15.31:0/2590079892 shutdown_connections 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55c9eb833650 0x55c9eb832360 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9eb7b2fa0 0x55c9eb82d6c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9eb7ade30 0x55c9eb826ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd6dbfff640 0 client.15111 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:53:55.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 1 --2- 172.21.15.31:0/2590079892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9eb7ad440 0x55c9eb8267a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 1 -- 172.21.15.31:0/2590079892 >> 172.21.15.31:0/2590079892 conn(0x55c9eb7a8ab0 msgr2=0x55c9eb7af290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 1 -- 172.21.15.31:0/2590079892 shutdown_connections 2024-04-25T01:53:55.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 1 -- 172.21.15.31:0/2590079892 wait complete. 2024-04-25T01:53:55.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.095+0000 7fd70585b740 20 client.15111 trim_cache size 0 max 16384 2024-04-25T01:53:55.103 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:53:55.103 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:53:55.103 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2024-04-25T01:53:55.103 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24487, 2024-04-25T01:53:55.103 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T01:53:55.103 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T01:53:55.104 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T01:53:55.104 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 86.382999910999999, 2024-04-25T01:53:55.104 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T01:53:55.104 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2024-04-25T01:53:55.104 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:53:55.104 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 102.417629538 2024-04-25T01:53:55.104 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 -- 172.21.15.31:0/3925415392 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd6ec0a2c10 msgr2=0x7fd6ec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd6ec0a2c10 0x7fd6ec0a50d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd6f409beb0 tx=0x7fd6f40c6ab0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 -- 172.21.15.31:0/3925415392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc102230 msgr2=0x7fd6fc19ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc102230 0x7fd6fc19ed40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd6f400eb10 tx=0x7fd6f400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 -- 172.21.15.31:0/3925415392 shutdown_connections 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fd6ec0a2c10 0x7fd6ec0a50d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6fc102230 0x7fd6fc19ed40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6fc101870 0x7fd6fc19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 --2- 172.21.15.31:0/3925415392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6fc100680 0x7fd6fc069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 -- 172.21.15.31:0/3925415392 >> 172.21.15.31:0/3925415392 conn(0x7fd6fc0fbbf0 msgr2=0x7fd6fc104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 -- 172.21.15.31:0/3925415392 shutdown_connections 2024-04-25T01:53:55.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.096+0000 7fd702b4f640 1 -- 172.21.15.31:0/3925415392 wait complete. 2024-04-25T01:53:55.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:53:55.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.209+0000 7f2acb6bd640 1 -- 172.21.15.31:0/3309920002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ac4102230 msgr2=0x7f2ac4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.209+0000 7f2acb6bd640 1 --2- 172.21.15.31:0/3309920002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ac4102230 0x7f2ac4108ac0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009ca0 tx=0x7f2abc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.209+0000 7f2acb6bd640 1 -- 172.21.15.31:0/3309920002 shutdown_connections 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.209+0000 7f2acb6bd640 1 --2- 172.21.15.31:0/3309920002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ac4102230 0x7f2ac4108ac0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.209+0000 7f2acb6bd640 1 --2- 172.21.15.31:0/3309920002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac4101870 0x7f2ac4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.209+0000 7f2acb6bd640 1 --2- 172.21.15.31:0/3309920002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ac4100680 0x7f2ac4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.209+0000 7f2acb6bd640 1 -- 172.21.15.31:0/3309920002 >> 172.21.15.31:0/3309920002 conn(0x7f2ac40fbbf0 msgr2=0x7f2ac40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2acb6bd640 1 -- 172.21.15.31:0/3309920002 shutdown_connections 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2acb6bd640 1 -- 172.21.15.31:0/3309920002 wait complete. 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2acb6bd640 1 Processor -- start 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2acb6bd640 1 -- start start 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2acb6bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ac4100680 0x7f2ac4196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2acb6bd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac4101870 0x7f2ac4196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2acb6bd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ac4102230 0x7f2ac419ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2acb6bd640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac410c260 con 0x7f2ac4102230 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2acb6bd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac410c0e0 con 0x7f2ac4100680 2024-04-25T01:53:55.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2acb6bd640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac410c3e0 con 0x7f2ac4101870 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2ac9c5a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ac4102230 0x7f2ac419ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2ac9c5a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ac4102230 0x7f2ac419ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46486/0 (socket says 172.21.15.31:46486) 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.210+0000 7f2ac9c5a640 1 -- 172.21.15.31:0/2613305479 learned_addr learned my addr 172.21.15.31:0/2613305479 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2ac9459640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ac4100680 0x7f2ac4196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac4101870 0x7f2ac4196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2ac9c5a640 1 -- 172.21.15.31:0/2613305479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac4101870 msgr2=0x7f2ac4196e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2ac9c5a640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac4101870 0x7f2ac4196e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2ac9c5a640 1 -- 172.21.15.31:0/2613305479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ac4100680 msgr2=0x7f2ac4196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2ac9c5a640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ac4100680 0x7f2ac4196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2ac9c5a640 1 -- 172.21.15.31:0/2613305479 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac419e540 con 0x7f2ac4102230 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac4101870 0x7f2ac4196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2ac9c5a640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ac4102230 0x7f2ac419ded0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2abc031040 tx=0x7f2abc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2aba7fc640 1 -- 172.21.15.31:0/2613305479 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc04a070 con 0x7f2ac4102230 2024-04-25T01:53:55.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2aba7fc640 1 -- 172.21.15.31:0/2613305479 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2abc032ec0 con 0x7f2ac4102230 2024-04-25T01:53:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2acb6bd640 1 -- 172.21.15.31:0/2613305479 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac40688d0 con 0x7f2ac4102230 2024-04-25T01:53:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2aba7fc640 1 -- 172.21.15.31:0/2613305479 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc030b20 con 0x7f2ac4102230 2024-04-25T01:53:55.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.211+0000 7f2acb6bd640 1 -- 172.21.15.31:0/2613305479 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac4068e20 con 0x7f2ac4102230 2024-04-25T01:53:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.213+0000 7f2aba7fc640 1 -- 172.21.15.31:0/2613305479 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2abc04d070 con 0x7f2ac4102230 2024-04-25T01:53:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ace3d3740 1 Processor -- start 2024-04-25T01:53:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ace3d3740 1 -- start start 2024-04-25T01:53:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c12440 0x560833c12860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c17930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c17fa0 0x560833c1a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560833b75f00 con 0x560833c12e30 2024-04-25T01:53:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560833b1e750 con 0x560833c12440 2024-04-25T01:53:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560833b71ec0 con 0x560833c17fa0 2024-04-25T01:53:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2aba7fc640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ab00a2b90 0x7f2ab00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ac9459640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c17930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ac9459640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c17930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46490/0 (socket says 172.21.15.31:46490) 2024-04-25T01:53:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ac9459640 1 -- 172.21.15.31:0/544835919 learned_addr learned my addr 172.21.15.31:0/544835919 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/544835919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c17fa0 0x560833c1a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ac9c5a640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ab00a2b90 0x7f2ab00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/544835919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c12440 0x560833c12860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.214+0000 7f2aba7fc640 1 -- 172.21.15.31:0/2613305479 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f2abc0f1970 con 0x7f2ac4102230 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.215+0000 7f2ac9459640 1 -- 172.21.15.31:0/544835919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c17fa0 msgr2=0x560833c1a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.215+0000 7f2ac9459640 1 --2- 172.21.15.31:0/544835919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c17fa0 0x560833c1a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.215+0000 7f2ac9459640 1 -- 172.21.15.31:0/544835919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c12440 msgr2=0x560833c12860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.215+0000 7f2ac9459640 1 --2- 172.21.15.31:0/544835919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c12440 0x560833c12860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.215+0000 7f2ac9459640 1 -- 172.21.15.31:0/544835919 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56083388a2d0 con 0x560833c12e30 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.215+0000 7f2ac9459640 1 --2- 172.21.15.31:0/544835919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c17930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000bc90 tx=0x7f2ac000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390c479cc632c833 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.215+0000 7f2ac9c5a640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ab00a2b90 0x7f2ab00a5050 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0049d0 tx=0x7f2abc089330 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.215+0000 7f2aaf7fe640 1 -- 172.21.15.31:0/544835919 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0015070 con 0x560833c12e30 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.215+0000 7f2aaf7fe640 1 -- 172.21.15.31:0/544835919 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ac00035f0 con 0x560833c12e30 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.215+0000 7f2aaf7fe640 1 -- 172.21.15.31:0/544835919 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0011680 con 0x560833c12e30 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 -- 172.21.15.31:0/544835919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 msgr2=0x560833c17930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/544835919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c17930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000bc90 tx=0x7f2ac000f730 comp rx=0 tx=0).stop 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 -- 172.21.15.31:0/544835919 shutdown_connections 2024-04-25T01:53:55.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/544835919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c17fa0 0x560833c1a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/544835919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c17930 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/544835919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c12440 0x560833c12860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 -- 172.21.15.31:0/544835919 >> 172.21.15.31:0/544835919 conn(0x560833c0dab0 msgr2=0x560833c0deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 -- 172.21.15.31:0/544835919 shutdown_connections 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 -- 172.21.15.31:0/544835919 wait complete. 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 -- start start 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 Processor -- start 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12440 0x560833c1c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c1cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c17fa0 0x560833c23e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560833c1af50 con 0x560833c12e30 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560833b31cf0 con 0x560833c17fa0 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560833c1b0d0 con 0x560833c12440 2024-04-25T01:53:55.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.216+0000 7f2ace3d3740 10 client.0 _ll_register_callbacks cb 0x560833b316e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ac9459640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c1cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ac9459640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c1cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46504/0 (socket says 172.21.15.31:46504) 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ac8c58640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c17fa0 0x560833c23e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ac9459640 1 -- 172.21.15.31:0/2880624731 learned_addr learned my addr 172.21.15.31:0/2880624731 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12440 0x560833c1c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ac9459640 1 -- 172.21.15.31:0/2880624731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12440 msgr2=0x560833c1c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ac9459640 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12440 0x560833c1c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ac9459640 1 -- 172.21.15.31:0/2880624731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c17fa0 msgr2=0x560833c23e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ac9459640 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c17fa0 0x560833c23e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ac9459640 1 -- 172.21.15.31:0/2880624731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608339546b0 con 0x560833c12e30 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ac9459640 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c1cdd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0018000 tx=0x7f2ac00039d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c63d2f7a9d48e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2aacff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2aacff9640 1 -- 172.21.15.31:0/2880624731 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0015030 con 0x560833c12e30 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2aacff9640 1 -- 172.21.15.31:0/2880624731 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f2ac000bd80 con 0x560833c12e30 2024-04-25T01:53:55.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2880624731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ac00035f0 con 0x560833c12e30 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2aacff9640 1 -- 172.21.15.31:0/2880624731 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2ac002b070 con 0x560833c12e30 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ace3d3740 10 client.15123 fetch_fsmap learned FSMap version 26 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2880624731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560833c26df0 con 0x560833c12e30 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.217+0000 7f2aacff9640 1 -- 172.21.15.31:0/2880624731 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ac00035f0 con 0x560833c12e30 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.218+0000 7f2aacff9640 1 -- 172.21.15.31:0/2880624731 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2ac0004750 con 0x560833c12e30 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.218+0000 7f2ace3d3740 10 client.15123 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.218+0000 7f2ace3d3740 10 client.15123 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.218+0000 7f2ace3d3740 20 client.15123 populate_metadata read hostname 'smithi031' 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.218+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560833c29030 0x560833c2a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.218+0000 7f2ace3d3740 4 client.15123 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.218+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2880624731 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560833b83440 con 0x560833c29030 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.218+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560833c29030 0x560833c2a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.218+0000 7f2aacff9640 10 client.15123 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.218+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560833c29030 0x560833c2a4a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f1a4ac9f1e0648a server_cookie=39f53f0a40275749 in_seq=0 out_seq=0 2024-04-25T01:53:55.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2aacff9640 1 -- 172.21.15.31:0/2880624731 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560833b83440 con 0x560833c29030 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2aacff9640 10 client.15123 handle_command_reply: tid=0 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2ace3d3740 1 client.15123 shutdown 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2880624731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 msgr2=0x560833c1cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c1cdd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0018000 tx=0x7f2ac00039d0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2880624731 shutdown_connections 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560833c29030 0x560833c2a4a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c17fa0 0x560833c23e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12e30 0x560833c1cdd0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2aacff9640 0 client.15123 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2880624731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12440 0x560833c1c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2880624731 >> 172.21.15.31:0/2880624731 conn(0x560833c0dab0 msgr2=0x560833c0cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2880624731 shutdown_connections 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.219+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2880624731 wait complete. 2024-04-25T01:53:55.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.220+0000 7f2ace3d3740 20 client.15123 trim_cache size 0 max 16384 2024-04-25T01:53:55.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.238+0000 7f2ace3d3740 1 Processor -- start 2024-04-25T01:53:55.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.238+0000 7f2ace3d3740 1 -- start start 2024-04-25T01:53:55.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.238+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12440 0x560833c136c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.238+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c12e30 0x560833c0c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.238+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c17fa0 0x560833c0c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.238+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560833c287e0 con 0x560833c17fa0 2024-04-25T01:53:55.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.238+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560833c28eb0 con 0x560833c12e30 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.238+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560833c29580 con 0x560833c12440 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.238+0000 7f2ac8c58640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c17fa0 0x560833c0c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.238+0000 7f2ac8c58640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c17fa0 0x560833c0c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46508/0 (socket says 172.21.15.31:46508) 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.238+0000 7f2ac8c58640 1 -- 172.21.15.31:0/884250906 learned_addr learned my addr 172.21.15.31:0/884250906 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2ac9459640 1 --2- 172.21.15.31:0/884250906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c12e30 0x560833c0c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2ac8c58640 1 -- 172.21.15.31:0/884250906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12440 msgr2=0x560833c136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/884250906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12440 0x560833c136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2ac8c58640 1 -- 172.21.15.31:0/884250906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c12e30 msgr2=0x560833c0c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/884250906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c12e30 0x560833c0c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2ac8c58640 1 -- 172.21.15.31:0/884250906 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560833c0e780 con 0x560833c17fa0 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/884250906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c17fa0 0x560833c0c790 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400cfd0 tx=0x7f2ab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d6ff84d4e74380 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2aaeffd640 1 -- 172.21.15.31:0/884250906 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab400f040 con 0x560833c17fa0 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2aaeffd640 1 -- 172.21.15.31:0/884250906 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ab4023030 con 0x560833c17fa0 2024-04-25T01:53:55.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2aaeffd640 1 -- 172.21.15.31:0/884250906 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab40166a0 con 0x560833c17fa0 2024-04-25T01:53:55.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2ace3d3740 1 -- 172.21.15.31:0/884250906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c17fa0 msgr2=0x560833c0c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.239+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/884250906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c17fa0 0x560833c0c790 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400cfd0 tx=0x7f2ab40046c0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 -- 172.21.15.31:0/884250906 shutdown_connections 2024-04-25T01:53:55.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/884250906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c17fa0 0x560833c0c790 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/884250906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c12e30 0x560833c0c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/884250906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12440 0x560833c136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 -- 172.21.15.31:0/884250906 >> 172.21.15.31:0/884250906 conn(0x560833c0dab0 msgr2=0x560833c08690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 -- 172.21.15.31:0/884250906 shutdown_connections 2024-04-25T01:53:55.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 -- 172.21.15.31:0/884250906 wait complete. 2024-04-25T01:53:55.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 -- start start 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 Processor -- start 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12440 0x560833c1f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12e30 0x560833c1f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c17fa0 0x560833c22680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560833c2a320 con 0x560833c12440 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560833c29c50 con 0x560833c17fa0 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560833c2a9f0 con 0x560833c12e30 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ace3d3740 10 client.0 _ll_register_callbacks cb 0x7f2ac000bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ac8c58640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12440 0x560833c1f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ac8c58640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12440 0x560833c1f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46512/0 (socket says 172.21.15.31:46512) 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.240+0000 7f2ac8c58640 1 -- 172.21.15.31:0/2471550818 learned_addr learned my addr 172.21.15.31:0/2471550818 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ac9459640 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12e30 0x560833c1f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ac8c58640 1 -- 172.21.15.31:0/2471550818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12e30 msgr2=0x560833c1f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12e30 0x560833c1f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ac8c58640 1 -- 172.21.15.31:0/2471550818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c17fa0 msgr2=0x560833c22680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c17fa0 0x560833c22680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ac8c58640 1 -- 172.21.15.31:0/2471550818 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560833c22c70 con 0x560833c12440 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ac8c58640 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12440 0x560833c1f170 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400cba0 tx=0x7f2ab4018ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748c3ff8f548a8c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2aacff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2aacff9640 1 -- 172.21.15.31:0/2471550818 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab401d050 con 0x560833c12440 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2aacff9640 1 -- 172.21.15.31:0/2471550818 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f2ab4005c80 con 0x560833c12440 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2471550818 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ab4023030 con 0x560833c12440 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2aacff9640 1 -- 172.21.15.31:0/2471550818 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2ab400f040 con 0x560833c12440 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ace3d3740 10 client.15129 fetch_fsmap learned FSMap version 26 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2471550818 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560833c8f6a0 con 0x560833c12440 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2aacff9640 1 -- 172.21.15.31:0/2471550818 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ab4023030 con 0x560833c12440 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2aacff9640 1 -- 172.21.15.31:0/2471550818 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2ab4015040 con 0x560833c12440 2024-04-25T01:53:55.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ace3d3740 10 client.15129 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.241+0000 7f2ace3d3740 10 client.15129 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.242+0000 7f2ace3d3740 20 client.15129 populate_metadata read hostname 'smithi031' 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.242+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560833c94070 0x560833c92d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.242+0000 7f2ace3d3740 4 client.15129 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.242+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2471550818 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "status"}) -- 0x560833b7d790 con 0x560833c94070 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.242+0000 7f2ac9459640 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560833c94070 0x560833c92d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.242+0000 7f2ac9459640 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560833c94070 0x560833c92d80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=651762d5b0578227 server_cookie=7441c4e8c2c7e5d9 in_seq=0 out_seq=0 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.242+0000 7f2aacff9640 10 client.15129 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2aacff9640 1 -- 172.21.15.31:0/2471550818 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560833b7d790 con 0x560833c94070 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2aacff9640 10 client.15129 handle_command_reply: tid=0 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 1 client.15129 shutdown 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2471550818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12440 msgr2=0x560833c1f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12440 0x560833c1f170 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400cba0 tx=0x7f2ab4018ac0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2471550818 shutdown_connections 2024-04-25T01:53:55.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560833c94070 0x560833c92d80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560833c17fa0 0x560833c22680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2aacff9640 0 client.15129 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:53:55.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560833c12e30 0x560833c1f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 1 --2- 172.21.15.31:0/2471550818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560833c12440 0x560833c1f170 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2471550818 >> 172.21.15.31:0/2471550818 conn(0x560833c0dab0 msgr2=0x560833c136c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2471550818 shutdown_connections 2024-04-25T01:53:55.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 1 -- 172.21.15.31:0/2471550818 wait complete. 2024-04-25T01:53:55.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2ace3d3740 20 client.15129 trim_cache size 0 max 16384 2024-04-25T01:53:55.253 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:53:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:53:55.253 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-04-25T01:53:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24503, 2024-04-25T01:53:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T01:53:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T01:53:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T01:53:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 89.518500150999998, 2024-04-25T01:53:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T01:53:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2024-04-25T01:53:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:53:55.254 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 93.957904647999996 2024-04-25T01:53:55.254 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:53:55.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2acb6bd640 1 -- 172.21.15.31:0/2613305479 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ab00a2b90 msgr2=0x7f2ab00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2acb6bd640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ab00a2b90 0x7f2ab00a5050 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0049d0 tx=0x7f2abc089330 comp rx=0 tx=0).stop 2024-04-25T01:53:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2acb6bd640 1 -- 172.21.15.31:0/2613305479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ac4102230 msgr2=0x7f2ac419ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2acb6bd640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ac4102230 0x7f2ac419ded0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2abc031040 tx=0x7f2abc003ba0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2acb6bd640 1 -- 172.21.15.31:0/2613305479 shutdown_connections 2024-04-25T01:53:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2acb6bd640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f2ab00a2b90 0x7f2ab00a5050 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2acb6bd640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ac4102230 0x7f2ac419ded0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2acb6bd640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac4101870 0x7f2ac4196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2acb6bd640 1 --2- 172.21.15.31:0/2613305479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ac4100680 0x7f2ac4196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.243+0000 7f2acb6bd640 1 -- 172.21.15.31:0/2613305479 >> 172.21.15.31:0/2613305479 conn(0x7f2ac40fbbf0 msgr2=0x7f2ac4104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.244+0000 7f2acb6bd640 1 -- 172.21.15.31:0/2613305479 shutdown_connections 2024-04-25T01:53:55.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:55.244+0000 7f2acb6bd640 1 -- 172.21.15.31:0/2613305479 wait complete. 2024-04-25T01:53:55.257 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T01:53:55.257 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T01:53:55.270 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T01:53:55.270 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T01:53:55.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:55 smithi040 ceph-mon[32418]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:55.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:55 smithi040 ceph-mon[32418]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T01:53:55.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:55 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3488208907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:55.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:55 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/52892037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:53:55.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:55 smithi134 ceph-mon[33098]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:55.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:55 smithi134 ceph-mon[33098]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T01:53:55.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:55 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3488208907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:55.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:55 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/52892037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:53:55.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:55 smithi031 ceph-mon[29204]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:53:55.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:55 smithi031 ceph-mon[29204]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T01:53:55.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:55 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3488208907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:55.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:55 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/52892037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:53:55.516 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:57.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 -- 172.21.15.31:0/3823508813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f196c102290 msgr2=0x7f196c1026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:57.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 --2- 172.21.15.31:0/3823508813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f196c102290 0x7f196c1026b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f19680099c0 tx=0x7f1968009cd0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 -- 172.21.15.31:0/3823508813 shutdown_connections 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 --2- 172.21.15.31:0/3823508813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c103e30 0x7f196c10a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 --2- 172.21.15.31:0/3823508813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f196c102290 0x7f196c1026b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 --2- 172.21.15.31:0/3823508813 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f196c103490 0x7f196c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 -- 172.21.15.31:0/3823508813 >> 172.21.15.31:0/3823508813 conn(0x7f196c0fd830 msgr2=0x7f196c0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 -- 172.21.15.31:0/3823508813 shutdown_connections 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 -- 172.21.15.31:0/3823508813 wait complete. 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 Processor -- start 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 -- start start 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f196c102290 0x7f196c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.231+0000 7f1973e22700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f196c103490 0x7f196c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f1973e22700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c103e30 0x7f196c19fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f1973e22700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f196c10dd20 con 0x7f196c103490 2024-04-25T01:53:57.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f1973e22700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f196c10dba0 con 0x7f196c103e30 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f1973e22700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f196c10dea0 con 0x7f196c102290 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f19713bd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f196c103490 0x7f196c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f19713bd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f196c103490 0x7f196c198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46536/0 (socket says 172.21.15.31:46536) 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f19713bd700 1 -- 172.21.15.31:0/3851987961 learned_addr learned my addr 172.21.15.31:0/3851987961 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f1971bbe700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f196c102290 0x7f196c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f19723bf700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c103e30 0x7f196c19fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f19713bd700 1 -- 172.21.15.31:0/3851987961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f196c102290 msgr2=0x7f196c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f19713bd700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f196c102290 0x7f196c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f19713bd700 1 -- 172.21.15.31:0/3851987961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c103e30 msgr2=0x7f196c19fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f19713bd700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c103e30 0x7f196c19fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f19713bd700 1 -- 172.21.15.31:0/3851987961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196c1a00b0 con 0x7f196c103490 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f19713bd700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f196c103490 0x7f196c198990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f195c00d9b0 tx=0x7f195c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f1962ffd700 1 -- 172.21.15.31:0/3851987961 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c017070 con 0x7f196c103490 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f1962ffd700 1 -- 172.21.15.31:0/3851987961 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f195c005020 con 0x7f196c103490 2024-04-25T01:53:57.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f1973e22700 1 -- 172.21.15.31:0/3851987961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f196c1a03a0 con 0x7f196c103490 2024-04-25T01:53:57.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.232+0000 7f1962ffd700 1 -- 172.21.15.31:0/3851987961 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c009b50 con 0x7f196c103490 2024-04-25T01:53:57.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.233+0000 7f1973e22700 1 -- 172.21.15.31:0/3851987961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f196c1a0890 con 0x7f196c103490 2024-04-25T01:53:57.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.233+0000 7f1973e22700 1 -- 172.21.15.31:0/3851987961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f196c068f20 con 0x7f196c103490 2024-04-25T01:53:57.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.235+0000 7f1962ffd700 1 -- 172.21.15.31:0/3851987961 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f195c009cf0 con 0x7f196c103490 2024-04-25T01:53:57.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.235+0000 7f1962ffd700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f19580a2c20 0x7f19580a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:57.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.235+0000 7f1971bbe700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f19580a2c20 0x7f19580a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:57.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.235+0000 7f1962ffd700 1 -- 172.21.15.31:0/3851987961 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f195c0cf330 con 0x7f196c103490 2024-04-25T01:53:57.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.236+0000 7f1971bbe700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f19580a2c20 0x7f19580a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1968000c00 tx=0x7f1968005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:57.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.239+0000 7f1962ffd700 1 -- 172.21.15.31:0/3851987961 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f195c020020 con 0x7f196c103490 2024-04-25T01:53:57.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:57 smithi040 ceph-mon[32418]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:53:57.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:57 smithi134 ceph-mon[33098]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:53:57.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.465+0000 7f1973e22700 1 -- 172.21.15.31:0/3851987961 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f196c062670 con 0x7f19580a2c20 2024-04-25T01:53:57.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:57 smithi031 ceph-mon[29204]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:53:57.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.466+0000 7f1962ffd700 1 -- 172.21.15.31:0/3851987961 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f196c062670 con 0x7f19580a2c20 2024-04-25T01:53:57.467 INFO:teuthology.orchestra.run.smithi031.stdout:Backend: cephadm 2024-04-25T01:53:57.467 INFO:teuthology.orchestra.run.smithi031.stdout:Available: Yes 2024-04-25T01:53:57.467 INFO:teuthology.orchestra.run.smithi031.stdout:Paused: No 2024-04-25T01:53:57.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 -- 172.21.15.31:0/3851987961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f19580a2c20 msgr2=0x7f19580a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:57.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f19580a2c20 0x7f19580a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1968000c00 tx=0x7f1968005fb0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 -- 172.21.15.31:0/3851987961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f196c103490 msgr2=0x7f196c198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:57.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f196c103490 0x7f196c198990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f195c00d9b0 tx=0x7f195c00dcc0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 -- 172.21.15.31:0/3851987961 shutdown_connections 2024-04-25T01:53:57.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f19580a2c20 0x7f19580a50e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c103e30 0x7f196c19fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f196c102290 0x7f196c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 --2- 172.21.15.31:0/3851987961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f196c103490 0x7f196c198990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:57.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 -- 172.21.15.31:0/3851987961 >> 172.21.15.31:0/3851987961 conn(0x7f196c0fd830 msgr2=0x7f196c1068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:57.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 -- 172.21.15.31:0/3851987961 shutdown_connections 2024-04-25T01:53:57.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:53:57.468+0000 7f1973e22700 1 -- 172.21.15.31:0/3851987961 wait complete. 2024-04-25T01:53:58.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T01:53:58.374 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:58.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:58 smithi040 ceph-mon[32418]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:53:58.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:58 smithi134 ceph-mon[33098]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:53:58.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:58 smithi031 ceph-mon[29204]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:53:59.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:53:59 smithi040 ceph-mon[32418]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:53:59.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:53:59 smithi134 ceph-mon[33098]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:53:59.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:53:59 smithi031 ceph-mon[29204]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:54:00.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.085+0000 7efd5c217700 1 -- 172.21.15.31:0/3884425699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd541011b0 msgr2=0x7efd541015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.085+0000 7efd5c217700 1 --2- 172.21.15.31:0/3884425699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd541011b0 0x7efd541015d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efd500099c0 tx=0x7efd50009cd0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.086+0000 7efd5c217700 1 -- 172.21.15.31:0/3884425699 shutdown_connections 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.086+0000 7efd5c217700 1 --2- 172.21.15.31:0/3884425699 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd54102d50 0x7efd54109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.086+0000 7efd5c217700 1 --2- 172.21.15.31:0/3884425699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd541011b0 0x7efd541015d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.086+0000 7efd5c217700 1 --2- 172.21.15.31:0/3884425699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd541023b0 0x7efd54102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.086+0000 7efd5c217700 1 -- 172.21.15.31:0/3884425699 >> 172.21.15.31:0/3884425699 conn(0x7efd540fc790 msgr2=0x7efd540febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.086+0000 7efd5c217700 1 -- 172.21.15.31:0/3884425699 shutdown_connections 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.086+0000 7efd5c217700 1 -- 172.21.15.31:0/3884425699 wait complete. 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.086+0000 7efd5c217700 1 Processor -- start 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.086+0000 7efd5c217700 1 -- start start 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd5c217700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd541011b0 0x7efd54105290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd5c217700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd541023b0 0x7efd54103930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd5c217700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd54102d50 0x7efd54103e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd5c217700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efd5410cca0 con 0x7efd541023b0 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd5c217700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efd5410cb20 con 0x7efd54102d50 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd5c217700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efd5410ce20 con 0x7efd541011b0 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd597b2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd541023b0 0x7efd54103930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd597b2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd541023b0 0x7efd54103930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46560/0 (socket says 172.21.15.31:46560) 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd597b2700 1 -- 172.21.15.31:0/1746206726 learned_addr learned my addr 172.21.15.31:0/1746206726 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd59fb3700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd541011b0 0x7efd54105290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd5a7b4700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd54102d50 0x7efd54103e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd597b2700 1 -- 172.21.15.31:0/1746206726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd541011b0 msgr2=0x7efd54105290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd597b2700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd541011b0 0x7efd54105290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd597b2700 1 -- 172.21.15.31:0/1746206726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd54102d50 msgr2=0x7efd54103e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd597b2700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd54102d50 0x7efd54103e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd597b2700 1 -- 172.21.15.31:0/1746206726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd54104740 con 0x7efd541023b0 2024-04-25T01:54:00.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd59fb3700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd541011b0 0x7efd54105290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:00.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd5a7b4700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd54102d50 0x7efd54103e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:00.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.087+0000 7efd597b2700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd541023b0 0x7efd54103930 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efd4400e9a0 tx=0x7efd4400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:00.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.088+0000 7efd4affd700 1 -- 172.21.15.31:0/1746206726 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4400cbd0 con 0x7efd541023b0 2024-04-25T01:54:00.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.088+0000 7efd4affd700 1 -- 172.21.15.31:0/1746206726 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd4400cd70 con 0x7efd541023b0 2024-04-25T01:54:00.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.088+0000 7efd5c217700 1 -- 172.21.15.31:0/1746206726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd541a25c0 con 0x7efd541023b0 2024-04-25T01:54:00.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.088+0000 7efd4affd700 1 -- 172.21.15.31:0/1746206726 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd44018680 con 0x7efd541023b0 2024-04-25T01:54:00.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.088+0000 7efd5c217700 1 -- 172.21.15.31:0/1746206726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd541a2ad0 con 0x7efd541023b0 2024-04-25T01:54:00.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.089+0000 7efd5c217700 1 -- 172.21.15.31:0/1746206726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd54068f20 con 0x7efd541023b0 2024-04-25T01:54:00.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.090+0000 7efd4affd700 1 -- 172.21.15.31:0/1746206726 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efd44010430 con 0x7efd541023b0 2024-04-25T01:54:00.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.090+0000 7efd4affd700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd400a2c20 0x7efd400a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:00.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.091+0000 7efd59fb3700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd400a2c20 0x7efd400a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:00.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.091+0000 7efd4affd700 1 -- 172.21.15.31:0/1746206726 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7efd440cf720 con 0x7efd541023b0 2024-04-25T01:54:00.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.091+0000 7efd59fb3700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd400a2c20 0x7efd400a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efd5000b580 tx=0x7efd50005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:00.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.095+0000 7efd4affd700 1 -- 172.21.15.31:0/1746206726 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efd4409c6b0 con 0x7efd541023b0 2024-04-25T01:54:00.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.309+0000 7efd5c217700 1 -- 172.21.15.31:0/1746206726 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7efd5406b0b0 con 0x7efd400a2c20 2024-04-25T01:54:00.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.323+0000 7efd4affd700 1 -- 172.21.15.31:0/1746206726 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7efd5406b0b0 con 0x7efd400a2c20 2024-04-25T01:54:00.324 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T01:54:00.324 INFO:teuthology.orchestra.run.smithi031.stdout:mds.a smithi031 running (102s) 16s ago 101s 17.2M - 19.0.0-2445-gd2c45d97 f68e37b807ef 375c20a58b87 2024-04-25T01:54:00.324 INFO:teuthology.orchestra.run.smithi031.stdout:mds.b smithi040 running (99s) 32s ago 98s 26.2M - 19.0.0-2445-gd2c45d97 f68e37b807ef ac5fc400b968 2024-04-25T01:54:00.324 INFO:teuthology.orchestra.run.smithi031.stdout:mds.c smithi134 running (113s) 32s ago 113s 19.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef 1498fd4b1901 2024-04-25T01:54:00.324 INFO:teuthology.orchestra.run.smithi031.stdout:mds.d smithi031 running (2m) 16s ago 2m 15.2M - 19.0.0-2445-gd2c45d97 f68e37b807ef cd5a56bb5567 2024-04-25T01:54:00.324 INFO:teuthology.orchestra.run.smithi031.stdout:mds.e smithi040 running (119s) 32s ago 118s 20.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef 71f06a6cc78c 2024-04-25T01:54:00.324 INFO:teuthology.orchestra.run.smithi031.stdout:mds.f smithi134 running (116s) 32s ago 116s 17.5M - 19.0.0-2445-gd2c45d97 f68e37b807ef f69574034f9f 2024-04-25T01:54:00.324 INFO:teuthology.orchestra.run.smithi031.stdout:mds.g smithi031 running (105s) 16s ago 104s 19.5M - 19.0.0-2445-gd2c45d97 f68e37b807ef 5c78e932ce5f 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:mds.h smithi040 running (110s) 32s ago 110s 20.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef 46c3665ae4d9 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:mds.i smithi134 running (108s) 32s ago 107s 18.0M - 19.0.0-2445-gd2c45d97 f68e37b807ef 4aaa4ddc19e6 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:mds.j smithi031 running (2m) 16s ago 2m 15.3M - 19.0.0-2445-gd2c45d97 f68e37b807ef 14c1d8036640 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:mds.k smithi040 running (2m) 32s ago 2m 19.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef 7a908d565424 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:mds.l smithi134 running (2m) 32s ago 2m 16.6M - 19.0.0-2445-gd2c45d97 f68e37b807ef 7dcfa65053c5 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.x smithi031 *:9283,8765 running (9m) 16s ago 9m 513M - 19.0.0-2445-gd2c45d97 f68e37b807ef 8a3e25b0c23c 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.y smithi040 *:8443,8765 running (7m) 32s ago 7m 453M - 19.0.0-2445-gd2c45d97 f68e37b807ef b66e1f6b463d 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.z smithi134 *:8443,8765 running (7m) 32s ago 7m 451M - 19.0.0-2445-gd2c45d97 f68e37b807ef 9bee8c413a8c 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:mon.a smithi031 running (9m) 16s ago 9m 51.5M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 9b79660ad9d9 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:mon.b smithi040 running (7m) 32s ago 7m 39.4M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 6695eaeb0133 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:mon.c smithi134 running (7m) 32s ago 7m 42.1M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 5ff0d2aaefdc 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (7m) 16s ago 7m 47.3M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 385638b09d1c 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi040 running (6m) 32s ago 6m 57.1M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef b712b4be767e 2024-04-25T01:54:00.325 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi134 running (6m) 32s ago 6m 58.9M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 33a3e491f4f7 2024-04-25T01:54:00.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (5m) 16s ago 5m 43.4M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef cd83febc388c 2024-04-25T01:54:00.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi040 running (5m) 32s ago 5m 41.6M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef efa15a4fb0d5 2024-04-25T01:54:00.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi134 running (5m) 32s ago 5m 53.8M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef c11954d35b55 2024-04-25T01:54:00.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi031 running (4m) 16s ago 4m 53.7M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 88d17f63c892 2024-04-25T01:54:00.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi040 running (4m) 32s ago 4m 56.3M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 1dbabda55a08 2024-04-25T01:54:00.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8 smithi134 running (3m) 32s ago 3m 54.2M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 4b5931644b54 2024-04-25T01:54:00.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 smithi031 running (3m) 16s ago 3m 52.4M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 38ae8224f902 2024-04-25T01:54:00.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10 smithi040 running (2m) 32s ago 2m 53.6M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef baa6d1043199 2024-04-25T01:54:00.326 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11 smithi134 running (2m) 32s ago 2m 43.4M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 70d4659b9b1e 2024-04-25T01:54:00.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 -- 172.21.15.31:0/1746206726 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd400a2c20 msgr2=0x7efd400a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:00.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd400a2c20 0x7efd400a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efd5000b580 tx=0x7efd50005fb0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 -- 172.21.15.31:0/1746206726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd541023b0 msgr2=0x7efd54103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:00.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd541023b0 0x7efd54103930 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efd4400e9a0 tx=0x7efd4400ecb0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 -- 172.21.15.31:0/1746206726 shutdown_connections 2024-04-25T01:54:00.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efd400a2c20 0x7efd400a50e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd54102d50 0x7efd54103e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd541011b0 0x7efd54105290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 --2- 172.21.15.31:0/1746206726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd541023b0 0x7efd54103930 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 -- 172.21.15.31:0/1746206726 >> 172.21.15.31:0/1746206726 conn(0x7efd540fc790 msgr2=0x7efd5406a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:00.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 -- 172.21.15.31:0/1746206726 shutdown_connections 2024-04-25T01:54:00.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:00.325+0000 7efd5c217700 1 -- 172.21.15.31:0/1746206726 wait complete. 2024-04-25T01:54:01.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T01:54:01.287 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:01 smithi031 ceph-mon[29204]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:54:01.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:01 smithi031 ceph-mon[29204]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:01.302 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:01.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:01 smithi040 ceph-mon[32418]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:54:01.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:01 smithi040 ceph-mon[32418]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:01.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:01 smithi134 ceph-mon[33098]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:54:01.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:01 smithi134 ceph-mon[33098]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:03.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.006+0000 7f07195e8700 1 -- 172.21.15.31:0/2498205338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0714102d40 msgr2=0x7f0714105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:03.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.006+0000 7f07195e8700 1 --2- 172.21.15.31:0/2498205338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0714102d40 0x7f0714105130 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0704009990 tx=0x7f0704009ca0 comp rx=0 tx=0).stop 2024-04-25T01:54:03.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.006+0000 7f07195e8700 1 -- 172.21.15.31:0/2498205338 shutdown_connections 2024-04-25T01:54:03.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.006+0000 7f07195e8700 1 --2- 172.21.15.31:0/2498205338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0714107fa0 0x7f071410a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:03.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.006+0000 7f07195e8700 1 --2- 172.21.15.31:0/2498205338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0714102d40 0x7f0714105130 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:03.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.006+0000 7f07195e8700 1 --2- 172.21.15.31:0/2498205338 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0714105670 0x7f0714107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:03.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.006+0000 7f07195e8700 1 -- 172.21.15.31:0/2498205338 >> 172.21.15.31:0/2498205338 conn(0x7f07140fc750 msgr2=0x7f07140feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:03.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.006+0000 7f07195e8700 1 -- 172.21.15.31:0/2498205338 shutdown_connections 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.006+0000 7f07195e8700 1 -- 172.21.15.31:0/2498205338 wait complete. 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.006+0000 7f07195e8700 1 Processor -- start 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07195e8700 1 -- start start 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07195e8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0714102d40 0x7f07141983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07195e8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0714105670 0x7f07141988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07195e8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0714107fa0 0x7f071419f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07195e8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f071410dd40 con 0x7f0714107fa0 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07195e8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f071410dbc0 con 0x7f0714105670 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07195e8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f071410dec0 con 0x7f0714102d40 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0714107fa0 0x7f071419f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f0712ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0714102d40 0x7f07141983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0714107fa0 0x7f071419f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49344/0 (socket says 172.21.15.31:49344) 2024-04-25T01:54:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07137fe700 1 -- 172.21.15.31:0/4283718899 learned_addr learned my addr 172.21.15.31:0/4283718899 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07127fc700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0714105670 0x7f07141988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07137fe700 1 -- 172.21.15.31:0/4283718899 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0714102d40 msgr2=0x7f07141983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07137fe700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0714102d40 0x7f07141983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07137fe700 1 -- 172.21.15.31:0/4283718899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0714105670 msgr2=0x7f07141988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07137fe700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0714105670 0x7f07141988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07137fe700 1 -- 172.21.15.31:0/4283718899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071419ffc0 con 0x7f0714107fa0 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07127fc700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0714105670 0x7f07141988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.007+0000 7f07137fe700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0714107fa0 0x7f071419f920 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f070c00add0 tx=0x7f070c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.008+0000 7f06fbfff700 1 -- 172.21.15.31:0/4283718899 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c017020 con 0x7f0714107fa0 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.008+0000 7f07195e8700 1 -- 172.21.15.31:0/4283718899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07141a02b0 con 0x7f0714107fa0 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.008+0000 7f06fbfff700 1 -- 172.21.15.31:0/4283718899 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f070c00ae00 con 0x7f0714107fa0 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.008+0000 7f06fbfff700 1 -- 172.21.15.31:0/4283718899 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c005310 con 0x7f0714107fa0 2024-04-25T01:54:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.008+0000 7f07195e8700 1 -- 172.21.15.31:0/4283718899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07141a07f0 con 0x7f0714107fa0 2024-04-25T01:54:03.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.009+0000 7f07195e8700 1 -- 172.21.15.31:0/4283718899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0714068f20 con 0x7f0714107fa0 2024-04-25T01:54:03.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.010+0000 7f06fbfff700 1 -- 172.21.15.31:0/4283718899 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f070c00c940 con 0x7f0714107fa0 2024-04-25T01:54:03.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.011+0000 7f06fbfff700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f06fc0a2c20 0x7f06fc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:03.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.011+0000 7f0712ffd700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f06fc0a2c20 0x7f06fc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:03.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.011+0000 7f06fbfff700 1 -- 172.21.15.31:0/4283718899 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f070c0ceda0 con 0x7f0714107fa0 2024-04-25T01:54:03.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.011+0000 7f0712ffd700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f06fc0a2c20 0x7f06fc0a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0704003780 tx=0x7f0704003850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:03.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.014+0000 7f06fbfff700 1 -- 172.21.15.31:0/4283718899 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f070c021020 con 0x7f0714107fa0 2024-04-25T01:54:03.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.228+0000 7f07195e8700 1 -- 172.21.15.31:0/4283718899 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f0714062670 con 0x7f06fc0a2c20 2024-04-25T01:54:03.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.234+0000 7f06fbfff700 1 -- 172.21.15.31:0/4283718899 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f0714062670 con 0x7f06fc0a2c20 2024-04-25T01:54:03.235 INFO:teuthology.orchestra.run.smithi031.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T01:54:03.235 INFO:teuthology.orchestra.run.smithi031.stdout:mds.all 12/12 35s ago 2m smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi040=b;smithi040=e;smithi040=h;smithi040=k;smithi134=c;smithi134=f;smithi134=i;smithi134=l;count:12 2024-04-25T01:54:03.235 INFO:teuthology.orchestra.run.smithi031.stdout:mgr 3/3 35s ago 7m smithi031=x;smithi040=y;smithi134=z;count:3 2024-04-25T01:54:03.236 INFO:teuthology.orchestra.run.smithi031.stdout:mon 3/3 35s ago 7m smithi031:172.21.15.31=a;smithi040:172.21.15.40=b;smithi134:172.21.15.134=c;count:3 2024-04-25T01:54:03.236 INFO:teuthology.orchestra.run.smithi031.stdout:osd 12 35s ago - 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.236+0000 7f07195e8700 1 -- 172.21.15.31:0/4283718899 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f06fc0a2c20 msgr2=0x7f06fc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.236+0000 7f07195e8700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f06fc0a2c20 0x7f06fc0a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0704003780 tx=0x7f0704003850 comp rx=0 tx=0).stop 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.236+0000 7f07195e8700 1 -- 172.21.15.31:0/4283718899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0714107fa0 msgr2=0x7f071419f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.236+0000 7f07195e8700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0714107fa0 0x7f071419f920 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f070c00add0 tx=0x7f070c00c330 comp rx=0 tx=0).stop 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.236+0000 7f07195e8700 1 -- 172.21.15.31:0/4283718899 shutdown_connections 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.236+0000 7f07195e8700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f06fc0a2c20 0x7f06fc0a50e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.236+0000 7f07195e8700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0714107fa0 0x7f071419f920 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.236+0000 7f07195e8700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0714102d40 0x7f07141983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.236+0000 7f07195e8700 1 --2- 172.21.15.31:0/4283718899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0714105670 0x7f07141988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.236+0000 7f07195e8700 1 -- 172.21.15.31:0/4283718899 >> 172.21.15.31:0/4283718899 conn(0x7f07140fc750 msgr2=0x7f07141036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.237+0000 7f07195e8700 1 -- 172.21.15.31:0/4283718899 shutdown_connections 2024-04-25T01:54:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:03.237+0000 7f07195e8700 1 -- 172.21.15.31:0/4283718899 wait complete. 2024-04-25T01:54:03.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:03 smithi040 ceph-mon[32418]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:54:03.438 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:03 smithi134 ceph-mon[33098]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:54:03.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:03 smithi031 ceph-mon[29204]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:54:04.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T01:54:04.297 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:04.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:04 smithi040 ceph-mon[32418]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:04.419 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:04 smithi031 ceph-mon[29204]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:04.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:04 smithi134 ceph-mon[33098]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:05.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:05 smithi040 ceph-mon[32418]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:54:05.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:05 smithi134 ceph-mon[33098]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:54:05.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:05 smithi031 ceph-mon[29204]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:54:06.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.065+0000 7f4959377700 1 -- 172.21.15.31:0/1230612958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4954103e40 msgr2=0x7f495410a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:06.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.065+0000 7f4959377700 1 --2- 172.21.15.31:0/1230612958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4954103e40 0x7f495410a6f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f494400b0d0 tx=0x7f494400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.065+0000 7f4959377700 1 -- 172.21.15.31:0/1230612958 shutdown_connections 2024-04-25T01:54:06.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.065+0000 7f4959377700 1 --2- 172.21.15.31:0/1230612958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4954103e40 0x7f495410a6f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.065+0000 7f4959377700 1 --2- 172.21.15.31:0/1230612958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49541022a0 0x7f49541026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.065+0000 7f4959377700 1 --2- 172.21.15.31:0/1230612958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49541034a0 0x7f4954103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.065+0000 7f4959377700 1 -- 172.21.15.31:0/1230612958 >> 172.21.15.31:0/1230612958 conn(0x7f49540fd840 msgr2=0x7f49540ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.065+0000 7f4959377700 1 -- 172.21.15.31:0/1230612958 shutdown_connections 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.065+0000 7f4959377700 1 -- 172.21.15.31:0/1230612958 wait complete. 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.065+0000 7f4959377700 1 Processor -- start 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4959377700 1 -- start start 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4959377700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49541022a0 0x7f4954075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4959377700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49541034a0 0x7f4954074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4959377700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4954103e40 0x7f4954074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4959377700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f495410dd10 con 0x7f49541022a0 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4959377700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f495410db90 con 0x7f4954103e40 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4959377700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f495410de90 con 0x7f49541034a0 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4952ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49541022a0 0x7f4954075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:06.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4952ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49541022a0 0x7f4954075fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49360/0 (socket says 172.21.15.31:49360) 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4952ffd700 1 -- 172.21.15.31:0/725281763 learned_addr learned my addr 172.21.15.31:0/725281763 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f49527fc700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49541034a0 0x7f4954074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f49537fe700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4954103e40 0x7f4954074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4952ffd700 1 -- 172.21.15.31:0/725281763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49541034a0 msgr2=0x7f4954074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4952ffd700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49541034a0 0x7f4954074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4952ffd700 1 -- 172.21.15.31:0/725281763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4954103e40 msgr2=0x7f4954074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4952ffd700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4954103e40 0x7f4954074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f4952ffd700 1 -- 172.21.15.31:0/725281763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4954075410 con 0x7f49541022a0 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f49537fe700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4954103e40 0x7f4954074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.066+0000 7f49527fc700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49541034a0 0x7f4954074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.067+0000 7f4952ffd700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49541022a0 0x7f4954075fd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f494c00c990 tx=0x7f494c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:06.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.067+0000 7f494bfff700 1 -- 172.21.15.31:0/725281763 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c007b00 con 0x7f49541022a0 2024-04-25T01:54:06.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.067+0000 7f4959377700 1 -- 172.21.15.31:0/725281763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49541a2360 con 0x7f49541022a0 2024-04-25T01:54:06.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.067+0000 7f494bfff700 1 -- 172.21.15.31:0/725281763 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f494c007ca0 con 0x7f49541022a0 2024-04-25T01:54:06.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.067+0000 7f494bfff700 1 -- 172.21.15.31:0/725281763 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c018660 con 0x7f49541022a0 2024-04-25T01:54:06.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.067+0000 7f4959377700 1 -- 172.21.15.31:0/725281763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49541a28a0 con 0x7f49541022a0 2024-04-25T01:54:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.068+0000 7f4959377700 1 -- 172.21.15.31:0/725281763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4954068f20 con 0x7f49541022a0 2024-04-25T01:54:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.069+0000 7f494bfff700 1 -- 172.21.15.31:0/725281763 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f494c00f450 con 0x7f49541022a0 2024-04-25T01:54:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.070+0000 7f494bfff700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f49400a2c20 0x7f49400a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.070+0000 7f49527fc700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f49400a2c20 0x7f49400a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.070+0000 7f494bfff700 1 -- 172.21.15.31:0/725281763 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f494c0cef70 con 0x7f49541022a0 2024-04-25T01:54:06.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.070+0000 7f49527fc700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f49400a2c20 0x7f49400a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f493c005fd0 tx=0x7f493c005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:06.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.074+0000 7f494bfff700 1 -- 172.21.15.31:0/725281763 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f494c014030 con 0x7f49541022a0 2024-04-25T01:54:06.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.287+0000 7f4959377700 1 -- 172.21.15.31:0/725281763 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f4954062670 con 0x7f49400a2c20 2024-04-25T01:54:06.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.288+0000 7f494bfff700 1 -- 172.21.15.31:0/725281763 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f4954062670 con 0x7f49400a2c20 2024-04-25T01:54:06.289 INFO:teuthology.orchestra.run.smithi031.stdout:HOST ADDR LABELS STATUS 2024-04-25T01:54:06.289 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 172.21.15.31 2024-04-25T01:54:06.289 INFO:teuthology.orchestra.run.smithi031.stdout:smithi040 172.21.15.40 2024-04-25T01:54:06.289 INFO:teuthology.orchestra.run.smithi031.stdout:smithi134 172.21.15.134 2024-04-25T01:54:06.290 INFO:teuthology.orchestra.run.smithi031.stdout:3 hosts in cluster 2024-04-25T01:54:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.290+0000 7f4959377700 1 -- 172.21.15.31:0/725281763 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f49400a2c20 msgr2=0x7f49400a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.290+0000 7f4959377700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f49400a2c20 0x7f49400a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f493c005fd0 tx=0x7f493c005e20 comp rx=0 tx=0).stop 2024-04-25T01:54:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.290+0000 7f4959377700 1 -- 172.21.15.31:0/725281763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49541022a0 msgr2=0x7f4954075fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.290+0000 7f4959377700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49541022a0 0x7f4954075fd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f494c00c990 tx=0x7f494c00cd50 comp rx=0 tx=0).stop 2024-04-25T01:54:06.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.290+0000 7f4959377700 1 -- 172.21.15.31:0/725281763 shutdown_connections 2024-04-25T01:54:06.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.290+0000 7f4959377700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f49400a2c20 0x7f49400a50e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.290+0000 7f4959377700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4954103e40 0x7f4954074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.290+0000 7f4959377700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49541022a0 0x7f4954075fd0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.290+0000 7f4959377700 1 --2- 172.21.15.31:0/725281763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49541034a0 0x7f4954074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.290+0000 7f4959377700 1 -- 172.21.15.31:0/725281763 >> 172.21.15.31:0/725281763 conn(0x7f49540fd840 msgr2=0x7f49541066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:06.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.291+0000 7f4959377700 1 -- 172.21.15.31:0/725281763 shutdown_connections 2024-04-25T01:54:06.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:06.291+0000 7f4959377700 1 -- 172.21.15.31:0/725281763 wait complete. 2024-04-25T01:54:07.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T01:54:07.313 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:07.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:07 smithi040 ceph-mon[32418]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T01:54:07.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:07 smithi040 ceph-mon[32418]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:07.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:07 smithi031 ceph-mon[29204]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T01:54:07.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:07 smithi031 ceph-mon[29204]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:07.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:07 smithi134 ceph-mon[33098]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T01:54:07.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:07 smithi134 ceph-mon[33098]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:08.999+0000 7f694239e700 1 -- 172.21.15.31:0/3112698935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f693c103ec0 msgr2=0x7f693c10a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:08.999+0000 7f694239e700 1 --2- 172.21.15.31:0/3112698935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f693c103ec0 0x7f693c10a770 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f693400b0a0 tx=0x7f693400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:54:09.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:08.999+0000 7f694239e700 1 -- 172.21.15.31:0/3112698935 shutdown_connections 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:08.999+0000 7f694239e700 1 --2- 172.21.15.31:0/3112698935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f693c103ec0 0x7f693c10a770 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:08.999+0000 7f694239e700 1 --2- 172.21.15.31:0/3112698935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693c100540 0x7f693c100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:08.999+0000 7f694239e700 1 --2- 172.21.15.31:0/3112698935 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f693c103520 0x7f693c103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:08.999+0000 7f694239e700 1 -- 172.21.15.31:0/3112698935 >> 172.21.15.31:0/3112698935 conn(0x7f693c0fbec0 msgr2=0x7f693c0fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:08.999+0000 7f694239e700 1 -- 172.21.15.31:0/3112698935 shutdown_connections 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:08.999+0000 7f694239e700 1 -- 172.21.15.31:0/3112698935 wait complete. 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f694239e700 1 Processor -- start 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f694239e700 1 -- start start 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f694239e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693c100540 0x7f693c198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f694239e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f693c103520 0x7f693c198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f694239e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f693c103ec0 0x7f693c19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f694239e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f693c10dce0 con 0x7f693c103ec0 2024-04-25T01:54:09.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f694239e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f693c10db60 con 0x7f693c100540 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f694239e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f693c10de60 con 0x7f693c103520 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f694093b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f693c103ec0 0x7f693c19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f694093b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f693c103ec0 0x7f693c19f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49382/0 (socket says 172.21.15.31:49382) 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f694093b700 1 -- 172.21.15.31:0/2931644271 learned_addr learned my addr 172.21.15.31:0/2931644271 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f693bfff700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693c100540 0x7f693c198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.000+0000 7f693b7fe700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f693c103520 0x7f693c198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f694093b700 1 -- 172.21.15.31:0/2931644271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f693c103520 msgr2=0x7f693c198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f694093b700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f693c103520 0x7f693c198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f694093b700 1 -- 172.21.15.31:0/2931644271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693c100540 msgr2=0x7f693c198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f694093b700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693c100540 0x7f693c198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f694093b700 1 -- 172.21.15.31:0/2931644271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693c19ff80 con 0x7f693c103ec0 2024-04-25T01:54:09.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f693b7fe700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f693c103520 0x7f693c198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:09.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f694093b700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f693c103ec0 0x7f693c19f8e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f693400b070 tx=0x7f6934004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:09.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f69397fa700 1 -- 172.21.15.31:0/2931644271 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6934021070 con 0x7f693c103ec0 2024-04-25T01:54:09.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f694239e700 1 -- 172.21.15.31:0/2931644271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f693c1a0210 con 0x7f693c103ec0 2024-04-25T01:54:09.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f694239e700 1 -- 172.21.15.31:0/2931644271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f693c1a07a0 con 0x7f693c103ec0 2024-04-25T01:54:09.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f69397fa700 1 -- 172.21.15.31:0/2931644271 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6934007d00 con 0x7f693c103ec0 2024-04-25T01:54:09.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.001+0000 7f69397fa700 1 -- 172.21.15.31:0/2931644271 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693401d450 con 0x7f693c103ec0 2024-04-25T01:54:09.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.002+0000 7f694239e700 1 -- 172.21.15.31:0/2931644271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f693c068f20 con 0x7f693c103ec0 2024-04-25T01:54:09.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.003+0000 7f69397fa700 1 -- 172.21.15.31:0/2931644271 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6934003fb0 con 0x7f693c103ec0 2024-04-25T01:54:09.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.004+0000 7f69397fa700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69240a70b0 0x7f69240a9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:09.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.004+0000 7f693bfff700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69240a70b0 0x7f69240a9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:09.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.004+0000 7f69397fa700 1 -- 172.21.15.31:0/2931644271 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f693400f020 con 0x7f693c103ec0 2024-04-25T01:54:09.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.005+0000 7f693bfff700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69240a70b0 0x7f69240a9570 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f693c069d10 tx=0x7f692c007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:09.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.013+0000 7f69397fa700 1 -- 172.21.15.31:0/2931644271 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f693402a020 con 0x7f693c103ec0 2024-04-25T01:54:09.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.229+0000 7f694239e700 1 -- 172.21.15.31:0/2931644271 --> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f693c062670 con 0x7f69240a70b0 2024-04-25T01:54:09.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.231+0000 7f69397fa700 1 -- 172.21.15.31:0/2931644271 <== mgr.14152 v2:172.21.15.31:6800/2306032998 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f693c062670 con 0x7f69240a70b0 2024-04-25T01:54:09.232 INFO:teuthology.orchestra.run.smithi031.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T01:54:09.232 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53310008400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T01:54:09.232 INFO:teuthology.orchestra.run.smithi031.stdout:smithi040 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009J400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-25T01:54:09.232 INFO:teuthology.orchestra.run.smithi031.stdout:smithi134 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62420002400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-25T01:54:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 -- 172.21.15.31:0/2931644271 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69240a70b0 msgr2=0x7f69240a9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69240a70b0 0x7f69240a9570 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f693c069d10 tx=0x7f692c007400 comp rx=0 tx=0).stop 2024-04-25T01:54:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 -- 172.21.15.31:0/2931644271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f693c103ec0 msgr2=0x7f693c19f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f693c103ec0 0x7f693c19f8e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f693400b070 tx=0x7f6934004e40 comp rx=0 tx=0).stop 2024-04-25T01:54:09.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 -- 172.21.15.31:0/2931644271 shutdown_connections 2024-04-25T01:54:09.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f69240a70b0 0x7f69240a9570 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:09.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f693c103ec0 0x7f693c19f8e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:09.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f693c100540 0x7f693c198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:09.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 --2- 172.21.15.31:0/2931644271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f693c103520 0x7f693c198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:09.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 -- 172.21.15.31:0/2931644271 >> 172.21.15.31:0/2931644271 conn(0x7f693c0fbec0 msgr2=0x7f693c1068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:09.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 -- 172.21.15.31:0/2931644271 shutdown_connections 2024-04-25T01:54:09.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:09.233+0000 7f694239e700 1 -- 172.21.15.31:0/2931644271 wait complete. 2024-04-25T01:54:09.246 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:09 smithi031 ceph-mon[29204]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T01:54:09.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:09 smithi040 ceph-mon[32418]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T01:54:09.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:09 smithi134 ceph-mon[33098]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T01:54:09.865 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T01:54:09.876 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T01:54:09.876 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T01:54:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:10.399 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:10 smithi031 ceph-mon[29204]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:10.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:10 smithi040 ceph-mon[32418]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:10.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:10 smithi134 ceph-mon[33098]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:11.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:11 smithi040 ceph-mon[32418]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:54:11.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:11 smithi134 ceph-mon[33098]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:54:11.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:11 smithi031 ceph-mon[29204]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:54:11.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.824+0000 7faed7c6b700 1 -- 172.21.15.31:0/95798176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faed0101180 msgr2=0x7faed01015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:11.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.824+0000 7faed7c6b700 1 --2- 172.21.15.31:0/95798176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faed0101180 0x7faed01015a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faecc0099c0 tx=0x7faecc009cd0 comp rx=0 tx=0).stop 2024-04-25T01:54:11.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.824+0000 7faed7c6b700 1 -- 172.21.15.31:0/95798176 shutdown_connections 2024-04-25T01:54:11.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.824+0000 7faed7c6b700 1 --2- 172.21.15.31:0/95798176 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faed0102d20 0x7faed01095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:11.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.824+0000 7faed7c6b700 1 --2- 172.21.15.31:0/95798176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faed0101180 0x7faed01015a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:11.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.824+0000 7faed7c6b700 1 --2- 172.21.15.31:0/95798176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed0102380 0x7faed01027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:11.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.824+0000 7faed7c6b700 1 -- 172.21.15.31:0/95798176 >> 172.21.15.31:0/95798176 conn(0x7faed00fc720 msgr2=0x7faed00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:11.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.824+0000 7faed7c6b700 1 -- 172.21.15.31:0/95798176 shutdown_connections 2024-04-25T01:54:11.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.824+0000 7faed7c6b700 1 -- 172.21.15.31:0/95798176 wait complete. 2024-04-25T01:54:11.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed7c6b700 1 Processor -- start 2024-04-25T01:54:11.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed7c6b700 1 -- start start 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed7c6b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faed0101180 0x7faed01050c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed7c6b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faed0102380 0x7faed0103760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed7c6b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed0102d20 0x7faed0103ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed7c6b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faed010ccd0 con 0x7faed0102380 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed7c6b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faed010cb50 con 0x7faed0102d20 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed7c6b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7faed010ce50 con 0x7faed0101180 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed5206700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faed0102380 0x7faed0103760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed5206700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faed0102380 0x7faed0103760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49402/0 (socket says 172.21.15.31:49402) 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed5206700 1 -- 172.21.15.31:0/816132592 learned_addr learned my addr 172.21.15.31:0/816132592 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed5a07700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faed0101180 0x7faed01050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed6208700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed0102d20 0x7faed0103ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed5206700 1 -- 172.21.15.31:0/816132592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faed0101180 msgr2=0x7faed01050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed5206700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faed0101180 0x7faed01050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:11.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed5206700 1 -- 172.21.15.31:0/816132592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed0102d20 msgr2=0x7faed0103ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:11.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed5206700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed0102d20 0x7faed0103ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:11.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed5206700 1 -- 172.21.15.31:0/816132592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed0104570 con 0x7faed0102380 2024-04-25T01:54:11.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.825+0000 7faed6208700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed0102d20 0x7faed0103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:11.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.826+0000 7faed5206700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faed0102380 0x7faed0103760 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7faec000d980 tx=0x7faec000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:11.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.826+0000 7faec6ffd700 1 -- 172.21.15.31:0/816132592 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0016070 con 0x7faed0102380 2024-04-25T01:54:11.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.826+0000 7faec6ffd700 1 -- 172.21.15.31:0/816132592 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faec0004830 con 0x7faed0102380 2024-04-25T01:54:11.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.826+0000 7faed7c6b700 1 -- 172.21.15.31:0/816132592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed01a2640 con 0x7faed0102380 2024-04-25T01:54:11.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.826+0000 7faec6ffd700 1 -- 172.21.15.31:0/816132592 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0005570 con 0x7faed0102380 2024-04-25T01:54:11.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.826+0000 7faed7c6b700 1 -- 172.21.15.31:0/816132592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed01a2b50 con 0x7faed0102380 2024-04-25T01:54:11.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.827+0000 7faed7c6b700 1 -- 172.21.15.31:0/816132592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed0068f20 con 0x7faed0102380 2024-04-25T01:54:11.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.828+0000 7faec6ffd700 1 -- 172.21.15.31:0/816132592 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7faec00049e0 con 0x7faed0102380 2024-04-25T01:54:11.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.829+0000 7faec6ffd700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7faebc0a2c20 0x7faebc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:11.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.829+0000 7faed5a07700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7faebc0a2c20 0x7faebc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:11.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.829+0000 7faec6ffd700 1 -- 172.21.15.31:0/816132592 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7faec00cee90 con 0x7faed0102380 2024-04-25T01:54:11.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.829+0000 7faed5a07700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7faebc0a2c20 0x7faebc0a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7faecc000c00 tx=0x7faecc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:11.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:11.834+0000 7faec6ffd700 1 -- 172.21.15.31:0/816132592 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faec001f020 con 0x7faed0102380 2024-04-25T01:54:12.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.086+0000 7faed7c6b700 1 -- 172.21.15.31:0/816132592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7faed01a3480 con 0x7faed0102380 2024-04-25T01:54:12.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.087+0000 7faec6ffd700 1 -- 172.21.15.31:0/816132592 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) ==== 76+0+3084 (secure 0 0 0) 0x7faec000f3c0 con 0x7faed0102380 2024-04-25T01:54:12.088 INFO:teuthology.orchestra.run.smithi031.stdout:e26 2024-04-25T01:54:12.088 INFO:teuthology.orchestra.run.smithi031.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:54:12.088 INFO:teuthology.orchestra.run.smithi031.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:54:12.088 INFO:teuthology.orchestra.run.smithi031.stdout:legacy client fscid: 1 2024-04-25T01:54:12.088 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:12.088 INFO:teuthology.orchestra.run.smithi031.stdout:Filesystem 'cephfs' (1) 2024-04-25T01:54:12.088 INFO:teuthology.orchestra.run.smithi031.stdout:fs_name cephfs 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 26 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-04-25T01:52:24.898441+0000 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2024-04-25T01:52:32.772296+0000 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:tableserver 0 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:root 0 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:session_timeout 300 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:session_autoclose 300 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:max_file_size 1099511627776 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:max_xattr_size 65536 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:required_client_features {} 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:last_failure 0 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:last_failure_osd_epoch 0 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:54:12.089 INFO:teuthology.orchestra.run.smithi031.stdout:max_mds 5 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:in 0,1,2,3,4 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:up {0=24503,1=24487,2=24473,3=24463,4=24451} 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:failed 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:damaged 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:stopped 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:data_pools [3] 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:metadata_pool 2 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:inline_data disabled 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:balancer 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:bal_rank_mask -1 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:standby_count_wanted 0 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:qdb_cluster leader: 24503 members: 24487,24473,24503,24463,24451 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.b{0:24503} state up:active seq 3 addr [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.e{0:24443} state up:standby-replay seq 1 addr [v2:172.21.15.40:6834/538218297,v1:172.21.15.40:6835/538218297] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.090 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.i{1:24487} state up:active seq 6 addr [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.l{1:24412} state up:standby-replay seq 1 addr [v2:172.21.15.134:6832/400138993,v1:172.21.15.134:6833/400138993] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.h{2:24473} state up:active seq 6 addr [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.k{2:24407} state up:standby-replay seq 1 addr [v2:172.21.15.40:6832/3875547004,v1:172.21.15.40:6833/3875547004] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.c{3:24463} state up:active seq 7 addr [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.a{3:14634} state up:standby-replay seq 1 addr [v2:172.21.15.31:6840/4277792607,v1:172.21.15.31:6841/4277792607] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.f{4:24451} state up:active seq 8 addr [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.g{4:14622} state up:standby-replay seq 1 addr [v2:172.21.15.31:6838/3704800465,v1:172.21.15.31:6839/3704800465] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout:Standby daemons: 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.j{-1:14544} state up:standby seq 1 addr [v2:172.21.15.31:6834/3566811410,v1:172.21.15.31:6835/3566811410] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.091 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.d{-1:14562} state up:standby seq 1 addr [v2:172.21.15.31:6836/2948364489,v1:172.21.15.31:6837/2948364489] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:12.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 -- 172.21.15.31:0/816132592 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7faebc0a2c20 msgr2=0x7faebc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:12.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7faebc0a2c20 0x7faebc0a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7faecc000c00 tx=0x7faecc005fb0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 -- 172.21.15.31:0/816132592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faed0102380 msgr2=0x7faed0103760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:12.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faed0102380 0x7faed0103760 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7faec000d980 tx=0x7faec000dc90 comp rx=0 tx=0).stop 2024-04-25T01:54:12.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 -- 172.21.15.31:0/816132592 shutdown_connections 2024-04-25T01:54:12.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7faebc0a2c20 0x7faebc0a50e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faed0102d20 0x7faed0103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faed0101180 0x7faed01050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 --2- 172.21.15.31:0/816132592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faed0102380 0x7faed0103760 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 -- 172.21.15.31:0/816132592 >> 172.21.15.31:0/816132592 conn(0x7faed00fc720 msgr2=0x7faed00fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:12.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 -- 172.21.15.31:0/816132592 shutdown_connections 2024-04-25T01:54:12.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:12.089+0000 7faed7c6b700 1 -- 172.21.15.31:0/816132592 wait complete. 2024-04-25T01:54:12.093 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T01:54:12.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:12 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/816132592' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:54:12.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:12 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/816132592' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:54:12.445 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:12 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/816132592' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:54:12.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T01:54:13.151 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:13.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:13 smithi031 ceph-mon[29204]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:54:13.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:13 smithi040 ceph-mon[32418]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:54:13.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:13 smithi134 ceph-mon[33098]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:54:14.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.873+0000 7f991075d700 1 -- 172.21.15.31:0/2272458412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9908103f00 msgr2=0x7f990810a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.873+0000 7f991075d700 1 --2- 172.21.15.31:0/2272458412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9908103f00 0x7f990810a7b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f990000b0a0 tx=0x7f990000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.873+0000 7f991075d700 1 -- 172.21.15.31:0/2272458412 shutdown_connections 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.873+0000 7f991075d700 1 --2- 172.21.15.31:0/2272458412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9908103f00 0x7f990810a7b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.873+0000 7f991075d700 1 --2- 172.21.15.31:0/2272458412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9908100580 0x7f99081009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.873+0000 7f991075d700 1 --2- 172.21.15.31:0/2272458412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908103560 0x7f99081039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.873+0000 7f991075d700 1 -- 172.21.15.31:0/2272458412 >> 172.21.15.31:0/2272458412 conn(0x7f99080fbee0 msgr2=0x7f99080fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.873+0000 7f991075d700 1 -- 172.21.15.31:0/2272458412 shutdown_connections 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.873+0000 7f991075d700 1 -- 172.21.15.31:0/2272458412 wait complete. 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f991075d700 1 Processor -- start 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f991075d700 1 -- start start 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f991075d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9908100580 0x7f9908198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:14.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f991075d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9908103560 0x7f9908198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f991075d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908103f00 0x7f990819f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f991075d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f990810dd20 con 0x7f9908103560 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f991075d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f990810dba0 con 0x7f9908103f00 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f991075d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f990810dea0 con 0x7f9908100580 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f990dcf8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9908103560 0x7f9908198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f990dcf8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9908103560 0x7f9908198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60506/0 (socket says 172.21.15.31:60506) 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f990dcf8700 1 -- 172.21.15.31:0/38698395 learned_addr learned my addr 172.21.15.31:0/38698395 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f990e4f9700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9908100580 0x7f9908198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f990ecfa700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908103f00 0x7f990819f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f990dcf8700 1 -- 172.21.15.31:0/38698395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9908100580 msgr2=0x7f9908198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f990dcf8700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9908100580 0x7f9908198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f990dcf8700 1 -- 172.21.15.31:0/38698395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908103f00 msgr2=0x7f990819f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f990dcf8700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908103f00 0x7f990819f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.874+0000 7f990dcf8700 1 -- 172.21.15.31:0/38698395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990819ff60 con 0x7f9908103560 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.875+0000 7f990dcf8700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9908103560 0x7f9908198840 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f98f800d980 tx=0x7f98f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.875+0000 7f98ff7fe700 1 -- 172.21.15.31:0/38698395 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8016070 con 0x7f9908103560 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.875+0000 7f98ff7fe700 1 -- 172.21.15.31:0/38698395 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98f8009d80 con 0x7f9908103560 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.875+0000 7f98ff7fe700 1 -- 172.21.15.31:0/38698395 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f800f3c0 con 0x7f9908103560 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.875+0000 7f991075d700 1 -- 172.21.15.31:0/38698395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99081a0250 con 0x7f9908103560 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.875+0000 7f991075d700 1 -- 172.21.15.31:0/38698395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99081a0790 con 0x7f9908103560 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.876+0000 7f991075d700 1 -- 172.21.15.31:0/38698395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9908068f20 con 0x7f9908103560 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.877+0000 7f98ff7fe700 1 -- 172.21.15.31:0/38698395 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f98f8019070 con 0x7f9908103560 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.878+0000 7f98ff7fe700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f98f40a2c20 0x7f98f40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.878+0000 7f990e4f9700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f98f40a2c20 0x7f98f40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.878+0000 7f98ff7fe700 1 -- 172.21.15.31:0/38698395 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f98f80ceb20 con 0x7f9908103560 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.878+0000 7f990e4f9700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f98f40a2c20 0x7f98f40a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f990806a7a0 tx=0x7f990400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:14.882+0000 7f98ff7fe700 1 -- 172.21.15.31:0/38698395 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f98f809ba50 con 0x7f9908103560 2024-04-25T01:54:15.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.091+0000 7f991075d700 1 -- 172.21.15.31:0/38698395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f9908062e40 con 0x7f9908103560 2024-04-25T01:54:15.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.092+0000 7f98ff7fe700 1 -- 172.21.15.31:0/38698395 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v89) ==== 56+0+5234 (secure 0 0 0) 0x7f98f809b350 con 0x7f9908103560 2024-04-25T01:54:15.093 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 89 2024-04-25T01:54:15.093 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T01:54:15.093 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-04-25T01:44:19.899313+0000 2024-04-25T01:54:15.093 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2024-04-25T01:53:24.952281+0000 2024-04-25T01:54:15.093 INFO:teuthology.orchestra.run.smithi031.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T01:54:15.093 INFO:teuthology.orchestra.run.smithi031.stdout:crush_version 26 2024-04-25T01:54:15.093 INFO:teuthology.orchestra.run.smithi031.stdout:full_ratio 0.95 2024-04-25T01:54:15.093 INFO:teuthology.orchestra.run.smithi031.stdout:backfillfull_ratio 0.9 2024-04-25T01:54:15.093 INFO:teuthology.orchestra.run.smithi031.stdout:nearfull_ratio 0.85 2024-04-25T01:54:15.093 INFO:teuthology.orchestra.run.smithi031.stdout:require_min_compat_client luminous 2024-04-25T01:54:15.093 INFO:teuthology.orchestra.run.smithi031.stdout:min_compat_client luminous 2024-04-25T01:54:15.094 INFO:teuthology.orchestra.run.smithi031.stdout:require_osd_release squid 2024-04-25T01:54:15.094 INFO:teuthology.orchestra.run.smithi031.stdout:stretch_mode_enabled false 2024-04-25T01:54:15.094 INFO:teuthology.orchestra.run.smithi031.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T01:54:15.094 INFO:teuthology.orchestra.run.smithi031.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-25T01:54:15.094 INFO:teuthology.orchestra.run.smithi031.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-25T01:54:15.094 INFO:teuthology.orchestra.run.smithi031.stdout:max_osd 12 2024-04-25T01:54:15.094 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 up in weight 1 up_from 8 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6802/268630401,v1:172.21.15.31:6803/268630401] [v2:172.21.15.31:6804/268630401,v1:172.21.15.31:6805/268630401] exists,up 4b99beda-ae8b-4e4b-8569-1c3b5f44654b 2024-04-25T01:54:15.094 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6800/3593667525,v1:172.21.15.40:6801/3593667525] [v2:172.21.15.40:6802/3593667525,v1:172.21.15.40:6803/3593667525] exists,up 285db63c-f115-4517-894f-01b97208ba8d 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 up in weight 1 up_from 18 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6800/414932802,v1:172.21.15.134:6801/414932802] [v2:172.21.15.134:6802/414932802,v1:172.21.15.134:6803/414932802] exists,up af88e288-e53a-46a6-93a0-576c53c16b71 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6810/513563424,v1:172.21.15.31:6811/513563424] [v2:172.21.15.31:6812/513563424,v1:172.21.15.31:6813/513563424] exists,up bc257cfc-3310-4d0f-bb39-ec5aed0f18f0 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 up in weight 1 up_from 33 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6808/3237694093,v1:172.21.15.40:6809/3237694093] [v2:172.21.15.40:6810/3237694093,v1:172.21.15.40:6811/3237694093] exists,up 0dd91e15-4315-40e0-a93a-aa3e7d5ebed3 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 up in weight 1 up_from 38 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6808/1004304767,v1:172.21.15.134:6809/1004304767] [v2:172.21.15.134:6810/1004304767,v1:172.21.15.134:6811/1004304767] exists,up e992053c-b5dd-41cc-a72f-c371eadc508d 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 up in weight 1 up_from 46 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6818/2233839542,v1:172.21.15.31:6819/2233839542] [v2:172.21.15.31:6820/2233839542,v1:172.21.15.31:6821/2233839542] exists,up 1c03259a-375f-482d-be6e-410eed785c7d 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 up in weight 1 up_from 52 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6816/1477675732,v1:172.21.15.40:6817/1477675732] [v2:172.21.15.40:6818/1477675732,v1:172.21.15.40:6819/1477675732] exists,up d408419f-02da-4563-ae2a-e6b436cc4f6b 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8 up in weight 1 up_from 61 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6816/3295606048,v1:172.21.15.134:6817/3295606048] [v2:172.21.15.134:6818/3295606048,v1:172.21.15.134:6819/3295606048] exists,up 71eaf5df-507d-4ea3-b5e6-fbc65d839f10 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 up in weight 1 up_from 69 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6826/238962088,v1:172.21.15.31:6827/238962088] [v2:172.21.15.31:6828/238962088,v1:172.21.15.31:6829/238962088] exists,up eda04a7b-4e18-470d-9678-e21ffeb1f7ed 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10 up in weight 1 up_from 74 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6824/2339369197,v1:172.21.15.40:6825/2339369197] [v2:172.21.15.40:6826/2339369197,v1:172.21.15.40:6827/2339369197] exists,up 16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11 up in weight 1 up_from 82 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6824/897924774,v1:172.21.15.134:6825/897924774] [v2:172.21.15.134:6826/897924774,v1:172.21.15.134:6827/897924774] exists,up 5113a81e-7097-4d97-8998-b59224bb6b28 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.0 [7,11] 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.a [2,11] 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.14 [2,4] 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.1a [8,1] 2024-04-25T01:54:15.095 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.1d [5,9] 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.2c [4,9] 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.3c [4,9] 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/363866598 expires 2024-04-26T01:45:23.647161+0000 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/2433894874 expires 2024-04-26T01:45:23.647161+0000 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/3170545029 expires 2024-04-26T01:45:23.647161+0000 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/3170545029 expires 2024-04-26T01:45:23.647161+0000 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/196728475 expires 2024-04-26T01:44:56.414005+0000 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/2254757453 expires 2024-04-26T01:44:56.414005+0000 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/3153405418 expires 2024-04-26T01:44:56.414005+0000 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/3153405418 expires 2024-04-26T01:44:56.414005+0000 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/1967729027 expires 2024-04-26T01:45:23.647161+0000 2024-04-25T01:54:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/1543547115 expires 2024-04-26T01:44:56.414005+0000 2024-04-25T01:54:15.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 -- 172.21.15.31:0/38698395 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f98f40a2c20 msgr2=0x7f98f40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:15.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f98f40a2c20 0x7f98f40a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f990806a7a0 tx=0x7f990400b410 comp rx=0 tx=0).stop 2024-04-25T01:54:15.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 -- 172.21.15.31:0/38698395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9908103560 msgr2=0x7f9908198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:15.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9908103560 0x7f9908198840 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f98f800d980 tx=0x7f98f800dc90 comp rx=0 tx=0).stop 2024-04-25T01:54:15.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 -- 172.21.15.31:0/38698395 shutdown_connections 2024-04-25T01:54:15.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f98f40a2c20 0x7f98f40a50e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908103f00 0x7f990819f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9908100580 0x7f9908198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 --2- 172.21.15.31:0/38698395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9908103560 0x7f9908198840 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 -- 172.21.15.31:0/38698395 >> 172.21.15.31:0/38698395 conn(0x7f99080fbee0 msgr2=0x7f9908106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 -- 172.21.15.31:0/38698395 shutdown_connections 2024-04-25T01:54:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:15.094+0000 7f991075d700 1 -- 172.21.15.31:0/38698395 wait complete. 2024-04-25T01:54:15.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:15 smithi040 ceph-mon[32418]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:54:15.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:15 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/38698395' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:54:15.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:15 smithi134 ceph-mon[33098]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:54:15.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:15 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/38698395' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:54:15.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:15 smithi031 ceph-mon[29204]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:54:15.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:15 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/38698395' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:54:15.931 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T01:54:15.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:54:16.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.050+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/1066255997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fa0101890 msgr2=0x7f4fa0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.050+0000 7f4fa6e9e640 1 --2- 172.21.15.31:0/1066255997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fa0101890 0x7f4fa0101d10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009900 tx=0x7f4f8802a410 comp rx=0 tx=0).stop 2024-04-25T01:54:16.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.050+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/1066255997 shutdown_connections 2024-04-25T01:54:16.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.050+0000 7f4fa6e9e640 1 --2- 172.21.15.31:0/1066255997 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fa0102250 0x7f4fa0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.050+0000 7f4fa6e9e640 1 --2- 172.21.15.31:0/1066255997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fa0101890 0x7f4fa0101d10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.050+0000 7f4fa6e9e640 1 --2- 172.21.15.31:0/1066255997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fa01006a0 0x7f4fa0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.050+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/1066255997 >> 172.21.15.31:0/1066255997 conn(0x7f4fa00fbc30 msgr2=0x7f4fa00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:16.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.050+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/1066255997 shutdown_connections 2024-04-25T01:54:16.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.050+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/1066255997 wait complete. 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa6e9e640 1 Processor -- start 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa6e9e640 1 -- start start 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa6e9e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fa01006a0 0x7f4fa006a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa6e9e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fa0101890 0x7f4fa019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa6e9e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fa0102250 0x7f4fa019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa6e9e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa010c1d0 con 0x7f4fa01006a0 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa6e9e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa010c050 con 0x7f4fa0101890 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa6e9e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa010c350 con 0x7f4fa0102250 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa4c3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fa01006a0 0x7f4fa006a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa4c3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fa01006a0 0x7f4fa006a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60518/0 (socket says 172.21.15.31:60518) 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa4c3a640 1 -- 172.21.15.31:0/3483088458 learned_addr learned my addr 172.21.15.31:0/3483088458 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa543b640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fa0102250 0x7f4fa019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa4c3a640 1 -- 172.21.15.31:0/3483088458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fa0102250 msgr2=0x7f4fa019ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa4c3a640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fa0102250 0x7f4fa019ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa4c3a640 1 -- 172.21.15.31:0/3483088458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fa0101890 msgr2=0x7f4fa019c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa4c3a640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fa0101890 0x7f4fa019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa4c3a640 1 -- 172.21.15.31:0/3483088458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa019f570 con 0x7f4fa01006a0 2024-04-25T01:54:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.051+0000 7f4fa4c3a640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fa01006a0 0x7f4fa006a930 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000c970 tx=0x7f4f9000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.052+0000 7f4f9dffb640 1 -- 172.21.15.31:0/3483088458 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90007c10 con 0x7f4fa01006a0 2024-04-25T01:54:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.052+0000 7f4f9dffb640 1 -- 172.21.15.31:0/3483088458 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f90007db0 con 0x7f4fa01006a0 2024-04-25T01:54:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.052+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/3483088458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa019f860 con 0x7f4fa01006a0 2024-04-25T01:54:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.052+0000 7f4f9dffb640 1 -- 172.21.15.31:0/3483088458 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9000f870 con 0x7f4fa01006a0 2024-04-25T01:54:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.052+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/3483088458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa019fe00 con 0x7f4fa01006a0 2024-04-25T01:54:16.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.053+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/3483088458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fa0107a40 con 0x7f4fa01006a0 2024-04-25T01:54:16.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.054+0000 7f4f9dffb640 1 -- 172.21.15.31:0/3483088458 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4f9000fa10 con 0x7f4fa01006a0 2024-04-25T01:54:16.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.054+0000 7f4f9dffb640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4f8c0a2c10 0x7f4f8c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.055+0000 7f4f9dffb640 1 -- 172.21.15.31:0/3483088458 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f4f900cf3d0 con 0x7f4fa01006a0 2024-04-25T01:54:16.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.055+0000 7f4f9ffff640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4f8c0a2c10 0x7f4f8c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.055+0000 7f4f9ffff640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4f8c0a2c10 0x7f4f8c0a50d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4f8802a930 tx=0x7f4f880314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:16.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.059+0000 7f4f9dffb640 1 -- 172.21.15.31:0/3483088458 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4f9009c3f0 con 0x7f4fa01006a0 2024-04-25T01:54:16.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.247+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/3483088458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4fa0062340 con 0x7f4fa01006a0 2024-04-25T01:54:16.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.248+0000 7f4f9dffb640 1 -- 172.21.15.31:0/3483088458 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22478 (secure 0 0 0) 0x7f4f9009bcf0 con 0x7f4fa01006a0 2024-04-25T01:54:16.250 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:16.253 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:53:24.952281+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.250+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/3483088458 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4f8c0a2c10 msgr2=0x7f4f8c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.250+0000 7f4fa6e9e640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4f8c0a2c10 0x7f4f8c0a50d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4f8802a930 tx=0x7f4f880314a0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.250+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/3483088458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fa01006a0 msgr2=0x7f4fa006a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.250+0000 7f4fa6e9e640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fa01006a0 0x7f4fa006a930 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000c970 tx=0x7f4f9000ce30 comp rx=0 tx=0).stop 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.251+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/3483088458 shutdown_connections 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.251+0000 7f4fa6e9e640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4f8c0a2c10 0x7f4f8c0a50d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.251+0000 7f4fa6e9e640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4fa0102250 0x7f4fa019ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.251+0000 7f4fa6e9e640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fa0101890 0x7f4fa019c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.251+0000 7f4fa6e9e640 1 --2- 172.21.15.31:0/3483088458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fa01006a0 0x7f4fa006a930 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.251+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/3483088458 >> 172.21.15.31:0/3483088458 conn(0x7f4fa00fbc30 msgr2=0x7f4fa0104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.251+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/3483088458 shutdown_connections 2024-04-25T01:54:16.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.251+0000 7f4fa6e9e640 1 -- 172.21.15.31:0/3483088458 wait complete. 2024-04-25T01:54:16.264 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:23.432566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:24.255373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:54:16.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:54:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.371+0000 7fed30da2640 1 -- 172.21.15.31:0/2887282358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed2c102280 msgr2=0x7fed2c104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.371+0000 7fed30da2640 1 --2- 172.21.15.31:0/2887282358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed2c102280 0x7fed2c104670 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fed24009900 tx=0x7fed2402a410 comp rx=0 tx=0).stop 2024-04-25T01:54:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.371+0000 7fed30da2640 1 -- 172.21.15.31:0/2887282358 shutdown_connections 2024-04-25T01:54:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.371+0000 7fed30da2640 1 --2- 172.21.15.31:0/2887282358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2c1074e0 0x7fed2c1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.371+0000 7fed30da2640 1 --2- 172.21.15.31:0/2887282358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed2c104bb0 0x7fed2c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.371+0000 7fed30da2640 1 --2- 172.21.15.31:0/2887282358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed2c102280 0x7fed2c104670 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.371+0000 7fed30da2640 1 -- 172.21.15.31:0/2887282358 >> 172.21.15.31:0/2887282358 conn(0x7fed2c0fbc30 msgr2=0x7fed2c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.371+0000 7fed30da2640 1 -- 172.21.15.31:0/2887282358 shutdown_connections 2024-04-25T01:54:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.371+0000 7fed30da2640 1 -- 172.21.15.31:0/2887282358 wait complete. 2024-04-25T01:54:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed30da2640 1 Processor -- start 2024-04-25T01:54:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed30da2640 1 -- start start 2024-04-25T01:54:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed30da2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2c102280 0x7fed2c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed30da2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed2c104bb0 0x7fed2c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed30da2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed2c1074e0 0x7fed2c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed30da2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c10d2f0 con 0x7fed2c104bb0 2024-04-25T01:54:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed30da2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c10d170 con 0x7fed2c102280 2024-04-25T01:54:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed30da2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c10d470 con 0x7fed2c1074e0 2024-04-25T01:54:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed29d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed2c104bb0 0x7fed2c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed29d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed2c104bb0 0x7fed2c197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60530/0 (socket says 172.21.15.31:60530) 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed29d9b640 1 -- 172.21.15.31:0/2925310789 learned_addr learned my addr 172.21.15.31:0/2925310789 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed2ad9d640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed2c1074e0 0x7fed2c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed2a59c640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2c102280 0x7fed2c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed29d9b640 1 -- 172.21.15.31:0/2925310789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed2c1074e0 msgr2=0x7fed2c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed29d9b640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed2c1074e0 0x7fed2c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed29d9b640 1 -- 172.21.15.31:0/2925310789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2c102280 msgr2=0x7fed2c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed29d9b640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2c102280 0x7fed2c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.372+0000 7fed29d9b640 1 -- 172.21.15.31:0/2925310789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2c19f660 con 0x7fed2c104bb0 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.373+0000 7fed29d9b640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed2c104bb0 0x7fed2c197fd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fed1400e9f0 tx=0x7fed1400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.373+0000 7fed237fe640 1 -- 172.21.15.31:0/2925310789 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1400ccc0 con 0x7fed2c104bb0 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.373+0000 7fed237fe640 1 -- 172.21.15.31:0/2925310789 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed140054d0 con 0x7fed2c104bb0 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.373+0000 7fed30da2640 1 -- 172.21.15.31:0/2925310789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed2c19f950 con 0x7fed2c104bb0 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.373+0000 7fed237fe640 1 -- 172.21.15.31:0/2925310789 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed14010430 con 0x7fed2c104bb0 2024-04-25T01:54:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.373+0000 7fed30da2640 1 -- 172.21.15.31:0/2925310789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed2c19fe50 con 0x7fed2c104bb0 2024-04-25T01:54:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.374+0000 7fed30da2640 1 -- 172.21.15.31:0/2925310789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed2c0fd720 con 0x7fed2c104bb0 2024-04-25T01:54:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.375+0000 7fed237fe640 1 -- 172.21.15.31:0/2925310789 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fed14005020 con 0x7fed2c104bb0 2024-04-25T01:54:16.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.375+0000 7fed237fe640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fed180a2b90 0x7fed180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.376+0000 7fed2a59c640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fed180a2b90 0x7fed180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.376+0000 7fed237fe640 1 -- 172.21.15.31:0/2925310789 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fed140cf4e0 con 0x7fed2c104bb0 2024-04-25T01:54:16.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.376+0000 7fed2a59c640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fed180a2b90 0x7fed180a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fed24000c00 tx=0x7fed240314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:16.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.380+0000 7fed237fe640 1 -- 172.21.15.31:0/2925310789 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fed1409c410 con 0x7fed2c104bb0 2024-04-25T01:54:16.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.566+0000 7fed30da2640 1 -- 172.21.15.31:0/2925310789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fed2c062340 con 0x7fed2c104bb0 2024-04-25T01:54:16.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.566+0000 7fed237fe640 1 -- 172.21.15.31:0/2925310789 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fed1409bd10 con 0x7fed2c104bb0 2024-04-25T01:54:16.567 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T01:54:16.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 -- 172.21.15.31:0/2925310789 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fed180a2b90 msgr2=0x7fed180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fed180a2b90 0x7fed180a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fed24000c00 tx=0x7fed240314a0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 -- 172.21.15.31:0/2925310789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed2c104bb0 msgr2=0x7fed2c197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed2c104bb0 0x7fed2c197fd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fed1400e9f0 tx=0x7fed1400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 -- 172.21.15.31:0/2925310789 shutdown_connections 2024-04-25T01:54:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fed180a2b90 0x7fed180a5050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed2c1074e0 0x7fed2c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed2c104bb0 0x7fed2c197fd0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 --2- 172.21.15.31:0/2925310789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed2c102280 0x7fed2c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 -- 172.21.15.31:0/2925310789 >> 172.21.15.31:0/2925310789 conn(0x7fed2c0fbc30 msgr2=0x7fed2c102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 -- 172.21.15.31:0/2925310789 shutdown_connections 2024-04-25T01:54:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.568+0000 7fed30da2640 1 -- 172.21.15.31:0/2925310789 wait complete. 2024-04-25T01:54:16.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:54:16.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.684+0000 7f2127f1b640 1 -- 172.21.15.31:0/4050559658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2120100010 msgr2=0x7f2120102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.684+0000 7f2127f1b640 1 --2- 172.21.15.31:0/4050559658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2120100010 0x7f2120102440 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f211c009900 tx=0x7f211c02a410 comp rx=0 tx=0).stop 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.684+0000 7f2127f1b640 1 -- 172.21.15.31:0/4050559658 shutdown_connections 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.684+0000 7f2127f1b640 1 --2- 172.21.15.31:0/4050559658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2120105330 0x7f21201057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.684+0000 7f2127f1b640 1 --2- 172.21.15.31:0/4050559658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2120102980 0x7f2120104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.684+0000 7f2127f1b640 1 --2- 172.21.15.31:0/4050559658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2120100010 0x7f2120102440 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.684+0000 7f2127f1b640 1 -- 172.21.15.31:0/4050559658 >> 172.21.15.31:0/4050559658 conn(0x7f21200fbc30 msgr2=0x7f21200fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.684+0000 7f2127f1b640 1 -- 172.21.15.31:0/4050559658 shutdown_connections 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.684+0000 7f2127f1b640 1 -- 172.21.15.31:0/4050559658 wait complete. 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f2127f1b640 1 Processor -- start 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f2127f1b640 1 -- start start 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f2127f1b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2120100010 0x7f212019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f2127f1b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2120102980 0x7f212019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f2127f1b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2120105330 0x7f21201a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f2127f1b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f21201138a0 con 0x7f2120102980 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f2127f1b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2120113720 con 0x7f2120105330 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f2127f1b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2120113a20 con 0x7f2120100010 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f21254b6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2120102980 0x7f212019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f21254b6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2120102980 0x7f212019e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60540/0 (socket says 172.21.15.31:60540) 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f21254b6640 1 -- 172.21.15.31:0/877141901 learned_addr learned my addr 172.21.15.31:0/877141901 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f2125cb7640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2120100010 0x7f212019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.685+0000 7f21264b8640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2120105330 0x7f21201a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f21254b6640 1 -- 172.21.15.31:0/877141901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2120100010 msgr2=0x7f212019e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f21254b6640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2120100010 0x7f212019e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f21254b6640 1 -- 172.21.15.31:0/877141901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2120105330 msgr2=0x7f21201a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f21254b6640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2120105330 0x7f21201a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f21254b6640 1 -- 172.21.15.31:0/877141901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21201a5c20 con 0x7f2120102980 2024-04-25T01:54:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f21264b8640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2120105330 0x7f21201a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f21254b6640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2120102980 0x7f212019e540 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f211000d950 tx=0x7f211000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f2116ffd640 1 -- 172.21.15.31:0/877141901 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110017070 con 0x7f2120102980 2024-04-25T01:54:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f2116ffd640 1 -- 172.21.15.31:0/877141901 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2110009c30 con 0x7f2120102980 2024-04-25T01:54:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f2127f1b640 1 -- 172.21.15.31:0/877141901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21201a5f10 con 0x7f2120102980 2024-04-25T01:54:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f2116ffd640 1 -- 172.21.15.31:0/877141901 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211000f620 con 0x7f2120102980 2024-04-25T01:54:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.686+0000 7f2127f1b640 1 -- 172.21.15.31:0/877141901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21201a6420 con 0x7f2120102980 2024-04-25T01:54:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.687+0000 7f2127f1b640 1 -- 172.21.15.31:0/877141901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21200fd720 con 0x7f2120102980 2024-04-25T01:54:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.688+0000 7f2116ffd640 1 -- 172.21.15.31:0/877141901 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f21100109a0 con 0x7f2120102980 2024-04-25T01:54:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.689+0000 7f2116ffd640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f210c0a2c10 0x7f210c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.689+0000 7f2125cb7640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f210c0a2c10 0x7f210c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.689+0000 7f2116ffd640 1 -- 172.21.15.31:0/877141901 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f21100cebc0 con 0x7f2120102980 2024-04-25T01:54:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.690+0000 7f2125cb7640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f210c0a2c10 0x7f210c0a50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f211c006010 tx=0x7f211c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.694+0000 7f2116ffd640 1 -- 172.21.15.31:0/877141901 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2110009de0 con 0x7f2120102980 2024-04-25T01:54:16.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.878+0000 7f2127f1b640 1 -- 172.21.15.31:0/877141901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f21200623e0 con 0x7f2120102980 2024-04-25T01:54:16.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.879+0000 7f2116ffd640 1 -- 172.21.15.31:0/877141901 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f211009bb50 con 0x7f2120102980 2024-04-25T01:54:16.880 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.880+0000 7f2127f1b640 1 -- 172.21.15.31:0/877141901 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f210c0a2c10 msgr2=0x7f210c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.880+0000 7f2127f1b640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f210c0a2c10 0x7f210c0a50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f211c006010 tx=0x7f211c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.880+0000 7f2127f1b640 1 -- 172.21.15.31:0/877141901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2120102980 msgr2=0x7f212019e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.880+0000 7f2127f1b640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2120102980 0x7f212019e540 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f211000d950 tx=0x7f211000de10 comp rx=0 tx=0).stop 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.880+0000 7f2127f1b640 1 -- 172.21.15.31:0/877141901 shutdown_connections 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.880+0000 7f2127f1b640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f210c0a2c10 0x7f210c0a50d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.880+0000 7f2127f1b640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2120105330 0x7f21201a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.880+0000 7f2127f1b640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2120102980 0x7f212019e540 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.880+0000 7f2127f1b640 1 --2- 172.21.15.31:0/877141901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2120100010 0x7f212019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.881+0000 7f2127f1b640 1 -- 172.21.15.31:0/877141901 >> 172.21.15.31:0/877141901 conn(0x7f21200fbc30 msgr2=0x7f2120103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.881+0000 7f2127f1b640 1 -- 172.21.15.31:0/877141901 shutdown_connections 2024-04-25T01:54:16.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.881+0000 7f2127f1b640 1 -- 172.21.15.31:0/877141901 wait complete. 2024-04-25T01:54:16.895 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:54:17.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.998+0000 7f0b23efc640 1 -- 172.21.15.31:0/2209300579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1c102250 msgr2=0x7f0b1c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.998+0000 7f0b23efc640 1 --2- 172.21.15.31:0/2209300579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1c102250 0x7f0b1c108ae0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0b14009ca0 tx=0x7f0b1402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 -- 172.21.15.31:0/2209300579 shutdown_connections 2024-04-25T01:54:17.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 --2- 172.21.15.31:0/2209300579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1c102250 0x7f0b1c108ae0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 --2- 172.21.15.31:0/2209300579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b1c101890 0x7f0b1c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 --2- 172.21.15.31:0/2209300579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c1006a0 0x7f0b1c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 -- 172.21.15.31:0/2209300579 >> 172.21.15.31:0/2209300579 conn(0x7f0b1c0fbc30 msgr2=0x7f0b1c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:17.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 -- 172.21.15.31:0/2209300579 shutdown_connections 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 -- 172.21.15.31:0/2209300579 wait complete. 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 Processor -- start 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 -- start start 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1c1006a0 0x7f0b1c1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b1c101890 0x7f0b1c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c102250 0x7f0b1c19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c10d2b0 con 0x7f0b1c1006a0 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c10d130 con 0x7f0b1c102250 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b23efc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c10d430 con 0x7f0b1c101890 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:16.999+0000 7f0b21c98640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1c1006a0 0x7f0b1c1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b21c98640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1c1006a0 0x7f0b1c1979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60554/0 (socket says 172.21.15.31:60554) 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b21c98640 1 -- 172.21.15.31:0/45959256 learned_addr learned my addr 172.21.15.31:0/45959256 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b21497640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b1c101890 0x7f0b1c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b21c98640 1 -- 172.21.15.31:0/45959256 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b1c101890 msgr2=0x7f0b1c197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b21c98640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b1c101890 0x7f0b1c197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b21c98640 1 -- 172.21.15.31:0/45959256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c102250 msgr2=0x7f0b1c19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b22499640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c102250 0x7f0b1c19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b21c98640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c102250 0x7f0b1c19ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b21c98640 1 -- 172.21.15.31:0/45959256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1c19f5c0 con 0x7f0b1c1006a0 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b21497640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b1c101890 0x7f0b1c197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b21c98640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1c1006a0 0x7f0b1c1979a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800e9c0 tx=0x7f0b1800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b12ffd640 1 -- 172.21.15.31:0/45959256 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1800cd80 con 0x7f0b1c1006a0 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b12ffd640 1 -- 172.21.15.31:0/45959256 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b180035f0 con 0x7f0b1c1006a0 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b23efc640 1 -- 172.21.15.31:0/45959256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b1c19f8b0 con 0x7f0b1c1006a0 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b12ffd640 1 -- 172.21.15.31:0/45959256 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b18010690 con 0x7f0b1c1006a0 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.000+0000 7f0b23efc640 1 -- 172.21.15.31:0/45959256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1c19fe00 con 0x7f0b1c1006a0 2024-04-25T01:54:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.001+0000 7f0b23efc640 1 -- 172.21.15.31:0/45959256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b1c0689b0 con 0x7f0b1c1006a0 2024-04-25T01:54:17.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.002+0000 7f0b12ffd640 1 -- 172.21.15.31:0/45959256 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0b180037a0 con 0x7f0b1c1006a0 2024-04-25T01:54:17.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.003+0000 7f0b12ffd640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0b080a2b90 0x7f0b080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.003+0000 7f0b12ffd640 1 -- 172.21.15.31:0/45959256 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f0b180cf740 con 0x7f0b1c1006a0 2024-04-25T01:54:17.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.003+0000 7f0b21497640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0b080a2b90 0x7f0b080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.003+0000 7f0b21497640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0b080a2b90 0x7f0b080a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c198eb0 tx=0x7f0b0c005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.006+0000 7f0b12ffd640 1 -- 172.21.15.31:0/45959256 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0b18053740 con 0x7f0b1c1006a0 2024-04-25T01:54:17.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.193+0000 7f0b23efc640 1 -- 172.21.15.31:0/45959256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0b1c02d250 con 0x7f0b1c1006a0 2024-04-25T01:54:17.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.194+0000 7f0b12ffd640 1 -- 172.21.15.31:0/45959256 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f0b1809c6d0 con 0x7f0b1c1006a0 2024-04-25T01:54:17.195 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T01:54:17.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 -- 172.21.15.31:0/45959256 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0b080a2b90 msgr2=0x7f0b080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0b080a2b90 0x7f0b080a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c198eb0 tx=0x7f0b0c005e50 comp rx=0 tx=0).stop 2024-04-25T01:54:17.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 -- 172.21.15.31:0/45959256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1c1006a0 msgr2=0x7f0b1c1979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1c1006a0 0x7f0b1c1979a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800e9c0 tx=0x7f0b1800ee80 comp rx=0 tx=0).stop 2024-04-25T01:54:17.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 -- 172.21.15.31:0/45959256 shutdown_connections 2024-04-25T01:54:17.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f0b080a2b90 0x7f0b080a5050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b1c102250 0x7f0b1c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b1c101890 0x7f0b1c197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 --2- 172.21.15.31:0/45959256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b1c1006a0 0x7f0b1c1979a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 -- 172.21.15.31:0/45959256 >> 172.21.15.31:0/45959256 conn(0x7f0b1c0fbc30 msgr2=0x7f0b1c104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:17.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 -- 172.21.15.31:0/45959256 shutdown_connections 2024-04-25T01:54:17.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.196+0000 7f0b23efc640 1 -- 172.21.15.31:0/45959256 wait complete. 2024-04-25T01:54:17.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:54:17.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.308+0000 7f61cf3ce640 1 -- 172.21.15.31:0/383059863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8101180 msgr2=0x7f61c8101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.308+0000 7f61cf3ce640 1 --2- 172.21.15.31:0/383059863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8101180 0x7f61c8101a00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f61b8009ca0 tx=0x7f61b802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 -- 172.21.15.31:0/383059863 shutdown_connections 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 --2- 172.21.15.31:0/383059863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c8101180 0x7f61c8101a00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 --2- 172.21.15.31:0/383059863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61c81007c0 0x7f61c8100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 --2- 172.21.15.31:0/383059863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c80ffa60 0x7f61c80ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 -- 172.21.15.31:0/383059863 >> 172.21.15.31:0/383059863 conn(0x7f61c80fb370 msgr2=0x7f61c80fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 -- 172.21.15.31:0/383059863 shutdown_connections 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 -- 172.21.15.31:0/383059863 wait complete. 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 Processor -- start 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 -- start start 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c80ffa60 0x7f61c8199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61c81007c0 0x7f61c819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c8101180 0x7f61c81a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f61c810f4f0 con 0x7f61c80ffa60 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f61c810f370 con 0x7f61c8101180 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cf3ce640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f61c810f670 con 0x7f61c81007c0 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cd16a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c80ffa60 0x7f61c8199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cd16a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c80ffa60 0x7f61c8199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60586/0 (socket says 172.21.15.31:60586) 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.309+0000 7f61cd16a640 1 -- 172.21.15.31:0/1339226159 learned_addr learned my addr 172.21.15.31:0/1339226159 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61cd96b640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c8101180 0x7f61c81a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61cc969640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61c81007c0 0x7f61c819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61cd16a640 1 -- 172.21.15.31:0/1339226159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61c81007c0 msgr2=0x7f61c819a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61cd16a640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61c81007c0 0x7f61c819a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61cd16a640 1 -- 172.21.15.31:0/1339226159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c8101180 msgr2=0x7f61c81a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61cd16a640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c8101180 0x7f61c81a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61cd16a640 1 -- 172.21.15.31:0/1339226159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c81a1750 con 0x7f61c80ffa60 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61cd16a640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c80ffa60 0x7f61c8199b30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f61c000c920 tx=0x7f61c000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:17.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61c67fc640 1 -- 172.21.15.31:0/1339226159 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0007c70 con 0x7f61c80ffa60 2024-04-25T01:54:17.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61c67fc640 1 -- 172.21.15.31:0/1339226159 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61c0007e10 con 0x7f61c80ffa60 2024-04-25T01:54:17.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61cf3ce640 1 -- 172.21.15.31:0/1339226159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c81a1a40 con 0x7f61c80ffa60 2024-04-25T01:54:17.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61c67fc640 1 -- 172.21.15.31:0/1339226159 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0018820 con 0x7f61c80ffa60 2024-04-25T01:54:17.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.310+0000 7f61cf3ce640 1 -- 172.21.15.31:0/1339226159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c81a1fe0 con 0x7f61c80ffa60 2024-04-25T01:54:17.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.311+0000 7f61cf3ce640 1 -- 172.21.15.31:0/1339226159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61c810b570 con 0x7f61c80ffa60 2024-04-25T01:54:17.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.312+0000 7f61c67fc640 1 -- 172.21.15.31:0/1339226159 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f61c00189c0 con 0x7f61c80ffa60 2024-04-25T01:54:17.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.313+0000 7f61c67fc640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f61b00a2c10 0x7f61b00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.313+0000 7f61cc969640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f61b00a2c10 0x7f61b00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.313+0000 7f61c67fc640 1 -- 172.21.15.31:0/1339226159 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f61c00cf090 con 0x7f61c80ffa60 2024-04-25T01:54:17.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.313+0000 7f61cc969640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f61b00a2c10 0x7f61b00a50d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f61c819b040 tx=0x7f61bc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:17.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.317+0000 7f61c67fc640 1 -- 172.21.15.31:0/1339226159 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f61c0014030 con 0x7f61c80ffa60 2024-04-25T01:54:17.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:17 smithi040 ceph-mon[32418]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T01:54:17.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:17 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3483088458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:17.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:17 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2925310789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:17.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:17 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/877141901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:17.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:17 smithi134 ceph-mon[33098]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T01:54:17.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:17 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3483088458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:17.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:17 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2925310789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:17.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:17 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/877141901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:17.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:17 smithi031 ceph-mon[29204]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T01:54:17.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:17 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3483088458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:17.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:17 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2925310789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:17.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:17 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/877141901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:17.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.544+0000 7f61cf3ce640 1 -- 172.21.15.31:0/1339226159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61c802d170 con 0x7f61c80ffa60 2024-04-25T01:54:17.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.544+0000 7f61c67fc640 1 -- 172.21.15.31:0/1339226159 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11923 (secure 0 0 0) 0x7f61c009bfc0 con 0x7f61c80ffa60 2024-04-25T01:54:17.545 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:17.545 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:52:32.772296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24487,24473,24503,24463,24451]},"id":1}]} 2024-04-25T01:54:17.546 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.546+0000 7f61cf3ce640 1 -- 172.21.15.31:0/1339226159 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f61b00a2c10 msgr2=0x7f61b00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.546+0000 7f61cf3ce640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f61b00a2c10 0x7f61b00a50d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f61c819b040 tx=0x7f61bc00b390 comp rx=0 tx=0).stop 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.546+0000 7f61cf3ce640 1 -- 172.21.15.31:0/1339226159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c80ffa60 msgr2=0x7f61c8199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.546+0000 7f61cf3ce640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c80ffa60 0x7f61c8199b30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f61c000c920 tx=0x7f61c000cde0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.547+0000 7f61cf3ce640 1 -- 172.21.15.31:0/1339226159 shutdown_connections 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.547+0000 7f61cf3ce640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f61b00a2c10 0x7f61b00a50d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.547+0000 7f61cf3ce640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61c8101180 0x7f61c81a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.547+0000 7f61cf3ce640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61c81007c0 0x7f61c819a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.547+0000 7f61cf3ce640 1 --2- 172.21.15.31:0/1339226159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61c80ffa60 0x7f61c8199b30 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.547+0000 7f61cf3ce640 1 -- 172.21.15.31:0/1339226159 >> 172.21.15.31:0/1339226159 conn(0x7f61c80fb370 msgr2=0x7f61c80fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.547+0000 7f61cf3ce640 1 -- 172.21.15.31:0/1339226159 shutdown_connections 2024-04-25T01:54:17.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.547+0000 7f61cf3ce640 1 -- 172.21.15.31:0/1339226159 wait complete. 2024-04-25T01:54:17.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:54:17.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.660+0000 7f26069d5640 1 -- 172.21.15.31:0/1190966528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2600104b30 msgr2=0x7f2600106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.660+0000 7f26069d5640 1 --2- 172.21.15.31:0/1190966528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2600104b30 0x7f2600106f20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f25f0009900 tx=0x7f25f002a410 comp rx=0 tx=0).stop 2024-04-25T01:54:17.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.660+0000 7f26069d5640 1 -- 172.21.15.31:0/1190966528 shutdown_connections 2024-04-25T01:54:17.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.660+0000 7f26069d5640 1 --2- 172.21.15.31:0/1190966528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2600107460 0x7f26001098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.660+0000 7f26069d5640 1 --2- 172.21.15.31:0/1190966528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2600104b30 0x7f2600106f20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.660+0000 7f26069d5640 1 --2- 172.21.15.31:0/1190966528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2600102200 0x7f26001045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.660+0000 7f26069d5640 1 -- 172.21.15.31:0/1190966528 >> 172.21.15.31:0/1190966528 conn(0x7f26000fbbf0 msgr2=0x7f26000fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:17.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.660+0000 7f26069d5640 1 -- 172.21.15.31:0/1190966528 shutdown_connections 2024-04-25T01:54:17.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.660+0000 7f26069d5640 1 -- 172.21.15.31:0/1190966528 wait complete. 2024-04-25T01:54:17.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.661+0000 7f26069d5640 1 Processor -- start 2024-04-25T01:54:17.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.661+0000 7f26069d5640 1 -- start start 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.661+0000 7f26069d5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2600102200 0x7f2600101b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.661+0000 7f26069d5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2600104b30 0x7f26001001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.661+0000 7f26069d5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2600107460 0x7f26001006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.661+0000 7f26069d5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f260010d2c0 con 0x7f2600104b30 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.661+0000 7f26069d5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f260010d140 con 0x7f2600102200 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.661+0000 7f26069d5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f260010d440 con 0x7f2600107460 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.661+0000 7f25ff7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2600104b30 0x7f26001001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25ff7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2600104b30 0x7f26001001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60610/0 (socket says 172.21.15.31:60610) 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25ff7fe640 1 -- 172.21.15.31:0/1244054741 learned_addr learned my addr 172.21.15.31:0/1244054741 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f2604f72640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2600107460 0x7f26001006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25fffff640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2600102200 0x7f2600101b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25ff7fe640 1 -- 172.21.15.31:0/1244054741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2600107460 msgr2=0x7f26001006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25ff7fe640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2600107460 0x7f26001006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25ff7fe640 1 -- 172.21.15.31:0/1244054741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2600102200 msgr2=0x7f2600101b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25ff7fe640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2600102200 0x7f2600101b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25ff7fe640 1 -- 172.21.15.31:0/1244054741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2600100fb0 con 0x7f2600104b30 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25fffff640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2600102200 0x7f2600101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25ff7fe640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2600104b30 0x7f26001001b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f25f00314e0 tx=0x7f25f0031a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25fd7fa640 1 -- 172.21.15.31:0/1244054741 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f003f030 con 0x7f2600104b30 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f26069d5640 1 -- 172.21.15.31:0/1244054741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26001a1b20 con 0x7f2600104b30 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25fd7fa640 1 -- 172.21.15.31:0/1244054741 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25f00044b0 con 0x7f2600104b30 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f25fd7fa640 1 -- 172.21.15.31:0/1244054741 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0035070 con 0x7f2600104b30 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.662+0000 7f26069d5640 1 -- 172.21.15.31:0/1244054741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26001a1fe0 con 0x7f2600104b30 2024-04-25T01:54:17.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.663+0000 7f26069d5640 1 -- 172.21.15.31:0/1244054741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26000fd6e0 con 0x7f2600104b30 2024-04-25T01:54:17.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.664+0000 7f25fd7fa640 1 -- 172.21.15.31:0/1244054741 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f25f0042030 con 0x7f2600104b30 2024-04-25T01:54:17.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.665+0000 7f25fd7fa640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25ec0a2b90 0x7f25ec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.665+0000 7f25fffff640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25ec0a2b90 0x7f25ec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.665+0000 7f25fd7fa640 1 -- 172.21.15.31:0/1244054741 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f25f00ef5b0 con 0x7f2600104b30 2024-04-25T01:54:17.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.665+0000 7f25fffff640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25ec0a2b90 0x7f25ec0a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f25e80058c0 tx=0x7f25e8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:17.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.669+0000 7f25fd7fa640 1 -- 172.21.15.31:0/1244054741 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f25f00bc4e0 con 0x7f2600104b30 2024-04-25T01:54:17.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.860+0000 7f26069d5640 1 -- 172.21.15.31:0/1244054741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2600062340 con 0x7f2600104b30 2024-04-25T01:54:17.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.861+0000 7f25fd7fa640 1 -- 172.21.15.31:0/1244054741 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22478 (secure 0 0 0) 0x7f25f00bbde0 con 0x7f2600104b30 2024-04-25T01:54:17.863 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:17.863 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:53:24.952281+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 -- 172.21.15.31:0/1244054741 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25ec0a2b90 msgr2=0x7f25ec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25ec0a2b90 0x7f25ec0a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f25e80058c0 tx=0x7f25e8005850 comp rx=0 tx=0).stop 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 -- 172.21.15.31:0/1244054741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2600104b30 msgr2=0x7f26001001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2600104b30 0x7f26001001b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f25f00314e0 tx=0x7f25f0031a30 comp rx=0 tx=0).stop 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 -- 172.21.15.31:0/1244054741 shutdown_connections 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f25ec0a2b90 0x7f25ec0a5050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2600107460 0x7f26001006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2600104b30 0x7f26001001b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 --2- 172.21.15.31:0/1244054741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2600102200 0x7f2600101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 -- 172.21.15.31:0/1244054741 >> 172.21.15.31:0/1244054741 conn(0x7f26000fbbf0 msgr2=0x7f2600102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 -- 172.21.15.31:0/1244054741 shutdown_connections 2024-04-25T01:54:17.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.864+0000 7f26069d5640 1 -- 172.21.15.31:0/1244054741 wait complete. 2024-04-25T01:54:17.876 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:23.432566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:24.255373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:54:17.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:54:17.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.981+0000 7f9a92f22640 1 -- 172.21.15.31:0/2692273171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a8c107540 msgr2=0x7f9a8c109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.981+0000 7f9a92f22640 1 --2- 172.21.15.31:0/2692273171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a8c107540 0x7f9a8c109930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009ca0 tx=0x7f9a8402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.981+0000 7f9a92f22640 1 -- 172.21.15.31:0/2692273171 shutdown_connections 2024-04-25T01:54:17.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.981+0000 7f9a92f22640 1 --2- 172.21.15.31:0/2692273171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a8c107540 0x7f9a8c109930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.981+0000 7f9a92f22640 1 --2- 172.21.15.31:0/2692273171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a8c104c10 0x7f9a8c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.981+0000 7f9a92f22640 1 --2- 172.21.15.31:0/2692273171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8c1022e0 0x7f9a8c1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.981+0000 7f9a92f22640 1 -- 172.21.15.31:0/2692273171 >> 172.21.15.31:0/2692273171 conn(0x7f9a8c0fbc10 msgr2=0x7f9a8c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.981+0000 7f9a92f22640 1 -- 172.21.15.31:0/2692273171 shutdown_connections 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.981+0000 7f9a92f22640 1 -- 172.21.15.31:0/2692273171 wait complete. 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a92f22640 1 Processor -- start 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a92f22640 1 -- start start 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a92f22640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a8c1022e0 0x7f9a8c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a92f22640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8c104c10 0x7f9a8c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a92f22640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a8c107540 0x7f9a8c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a92f22640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c10d300 con 0x7f9a8c107540 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a92f22640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c10d180 con 0x7f9a8c104c10 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a92f22640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c10d480 con 0x7f9a8c1022e0 2024-04-25T01:54:17.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a914bf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a8c107540 0x7f9a8c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a914bf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a8c107540 0x7f9a8c19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60638/0 (socket says 172.21.15.31:60638) 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a914bf640 1 -- 172.21.15.31:0/498268346 learned_addr learned my addr 172.21.15.31:0/498268346 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a8bfff640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8c104c10 0x7f9a8c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a90cbe640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a8c1022e0 0x7f9a8c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a914bf640 1 -- 172.21.15.31:0/498268346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a8c1022e0 msgr2=0x7f9a8c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a914bf640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a8c1022e0 0x7f9a8c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a914bf640 1 -- 172.21.15.31:0/498268346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8c104c10 msgr2=0x7f9a8c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a914bf640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8c104c10 0x7f9a8c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a914bf640 1 -- 172.21.15.31:0/498268346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c19f650 con 0x7f9a8c107540 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.982+0000 7f9a914bf640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a8c107540 0x7f9a8c19efe0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9a84031970 tx=0x7f9a84003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.983+0000 7f9a89ffb640 1 -- 172.21.15.31:0/498268346 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84004040 con 0x7f9a8c107540 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.983+0000 7f9a92f22640 1 -- 172.21.15.31:0/498268346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8c19f8e0 con 0x7f9a8c107540 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.983+0000 7f9a89ffb640 1 -- 172.21.15.31:0/498268346 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a84004690 con 0x7f9a8c107540 2024-04-25T01:54:17.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.983+0000 7f9a89ffb640 1 -- 172.21.15.31:0/498268346 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84030c60 con 0x7f9a8c107540 2024-04-25T01:54:17.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.983+0000 7f9a92f22640 1 -- 172.21.15.31:0/498268346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8c19fe80 con 0x7f9a8c107540 2024-04-25T01:54:17.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.984+0000 7f9a92f22640 1 -- 172.21.15.31:0/498268346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a8c0fd700 con 0x7f9a8c107540 2024-04-25T01:54:17.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.985+0000 7f9a89ffb640 1 -- 172.21.15.31:0/498268346 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9a840041e0 con 0x7f9a8c107540 2024-04-25T01:54:17.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.986+0000 7f9a89ffb640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9a7c0a2b90 0x7f9a7c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:17.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.986+0000 7f9a90cbe640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9a7c0a2b90 0x7f9a7c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:17.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.986+0000 7f9a89ffb640 1 -- 172.21.15.31:0/498268346 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f9a840f0530 con 0x7f9a8c107540 2024-04-25T01:54:17.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.986+0000 7f9a90cbe640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9a7c0a2b90 0x7f9a7c0a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009b60 tx=0x7f9a80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:17.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:17.990+0000 7f9a89ffb640 1 -- 172.21.15.31:0/498268346 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9a840bd460 con 0x7f9a8c107540 2024-04-25T01:54:18.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.175+0000 7f9a92f22640 1 -- 172.21.15.31:0/498268346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9a8c062340 con 0x7f9a8c107540 2024-04-25T01:54:18.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.175+0000 7f9a89ffb640 1 -- 172.21.15.31:0/498268346 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f9a840bcd60 con 0x7f9a8c107540 2024-04-25T01:54:18.176 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.177+0000 7f9a92f22640 1 -- 172.21.15.31:0/498268346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9a7c0a2b90 msgr2=0x7f9a7c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.177+0000 7f9a92f22640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9a7c0a2b90 0x7f9a7c0a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009b60 tx=0x7f9a80005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.177+0000 7f9a92f22640 1 -- 172.21.15.31:0/498268346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a8c107540 msgr2=0x7f9a8c19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.177+0000 7f9a92f22640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a8c107540 0x7f9a8c19efe0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9a84031970 tx=0x7f9a84003ba0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.177+0000 7f9a92f22640 1 -- 172.21.15.31:0/498268346 shutdown_connections 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.177+0000 7f9a92f22640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f9a7c0a2b90 0x7f9a7c0a5050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.177+0000 7f9a92f22640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a8c107540 0x7f9a8c19efe0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.177+0000 7f9a92f22640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8c104c10 0x7f9a8c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.177+0000 7f9a92f22640 1 --2- 172.21.15.31:0/498268346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a8c1022e0 0x7f9a8c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.177+0000 7f9a92f22640 1 -- 172.21.15.31:0/498268346 >> 172.21.15.31:0/498268346 conn(0x7f9a8c0fbc10 msgr2=0x7f9a8c102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.177+0000 7f9a92f22640 1 -- 172.21.15.31:0/498268346 shutdown_connections 2024-04-25T01:54:18.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.178+0000 7f9a92f22640 1 -- 172.21.15.31:0/498268346 wait complete. 2024-04-25T01:54:18.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:54:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.293+0000 7f7393c31640 1 -- 172.21.15.31:0/1926997621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c103350 msgr2=0x7f738c109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.293+0000 7f7393c31640 1 --2- 172.21.15.31:0/1926997621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c103350 0x7f738c109be0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7384009ca0 tx=0x7f738402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.293+0000 7f7393c31640 1 -- 172.21.15.31:0/1926997621 shutdown_connections 2024-04-25T01:54:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.293+0000 7f7393c31640 1 --2- 172.21.15.31:0/1926997621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c103350 0x7f738c109be0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.293+0000 7f7393c31640 1 --2- 172.21.15.31:0/1926997621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f738c102990 0x7f738c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.293+0000 7f7393c31640 1 --2- 172.21.15.31:0/1926997621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f738c1017a0 0x7f738c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.293+0000 7f7393c31640 1 -- 172.21.15.31:0/1926997621 >> 172.21.15.31:0/1926997621 conn(0x7f738c0fcd10 msgr2=0x7f738c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.293+0000 7f7393c31640 1 -- 172.21.15.31:0/1926997621 shutdown_connections 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.293+0000 7f7393c31640 1 -- 172.21.15.31:0/1926997621 wait complete. 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f7393c31640 1 Processor -- start 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f7393c31640 1 -- start start 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f7393c31640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f738c1017a0 0x7f738c197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f7393c31640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c102990 0x7f738c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f7393c31640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f738c103350 0x7f738c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f7393c31640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f738c10d2d0 con 0x7f738c102990 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f7393c31640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f738c10d150 con 0x7f738c103350 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f7393c31640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f738c10d450 con 0x7f738c1017a0 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f73911cc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c102990 0x7f738c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f73911cc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c102990 0x7f738c197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60652/0 (socket says 172.21.15.31:60652) 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f73911cc640 1 -- 172.21.15.31:0/1661200761 learned_addr learned my addr 172.21.15.31:0/1661200761 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f73919cd640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f738c1017a0 0x7f738c197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f73911cc640 1 -- 172.21.15.31:0/1661200761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f738c1017a0 msgr2=0x7f738c197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f73911cc640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f738c1017a0 0x7f738c197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f73911cc640 1 -- 172.21.15.31:0/1661200761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f738c103350 msgr2=0x7f738c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f73911cc640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f738c103350 0x7f738c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f73911cc640 1 -- 172.21.15.31:0/1661200761 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738c19f5a0 con 0x7f738c102990 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f73919cd640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f738c1017a0 0x7f738c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.294+0000 7f73911cc640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c102990 0x7f738c197ec0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f737c00d950 tx=0x7f737c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.295+0000 7f7382ffd640 1 -- 172.21.15.31:0/1661200761 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c017030 con 0x7f738c102990 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.295+0000 7f7393c31640 1 -- 172.21.15.31:0/1661200761 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f738c19f890 con 0x7f738c102990 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.295+0000 7f7382ffd640 1 -- 172.21.15.31:0/1661200761 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f737c004020 con 0x7f738c102990 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.295+0000 7f7382ffd640 1 -- 172.21.15.31:0/1661200761 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c0047a0 con 0x7f738c102990 2024-04-25T01:54:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.295+0000 7f7393c31640 1 -- 172.21.15.31:0/1661200761 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f738c19fe30 con 0x7f738c102990 2024-04-25T01:54:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.296+0000 7f7393c31640 1 -- 172.21.15.31:0/1661200761 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f738c108ac0 con 0x7f738c102990 2024-04-25T01:54:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.297+0000 7f7382ffd640 1 -- 172.21.15.31:0/1661200761 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f737c0041d0 con 0x7f738c102990 2024-04-25T01:54:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.297+0000 7f7382ffd640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f73780a2b90 0x7f73780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.298+0000 7f73919cd640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f73780a2b90 0x7f73780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.298+0000 7f7382ffd640 1 -- 172.21.15.31:0/1661200761 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f737c0cebb0 con 0x7f738c102990 2024-04-25T01:54:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.298+0000 7f73919cd640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f73780a2b90 0x7f73780a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f73880058c0 tx=0x7f7388005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.302+0000 7f7382ffd640 1 -- 172.21.15.31:0/1661200761 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f737c09bae0 con 0x7f738c102990 2024-04-25T01:54:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:18 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/45959256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:18 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1339226159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:18.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:18 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1244054741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:18.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:18 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/45959256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:18.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:18 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1339226159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:18.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:18 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1244054741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:18.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:18 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/45959256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:18.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:18 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1339226159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:18.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:18 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1244054741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.487+0000 7f7393c31640 1 -- 172.21.15.31:0/1661200761 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f738c062340 con 0x7f738c102990 2024-04-25T01:54:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.488+0000 7f7382ffd640 1 -- 172.21.15.31:0/1661200761 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f737c09b3e0 con 0x7f738c102990 2024-04-25T01:54:18.489 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T01:54:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 -- 172.21.15.31:0/1661200761 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f73780a2b90 msgr2=0x7f73780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f73780a2b90 0x7f73780a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f73880058c0 tx=0x7f7388005850 comp rx=0 tx=0).stop 2024-04-25T01:54:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 -- 172.21.15.31:0/1661200761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c102990 msgr2=0x7f738c197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c102990 0x7f738c197ec0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f737c00d950 tx=0x7f737c00de10 comp rx=0 tx=0).stop 2024-04-25T01:54:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 -- 172.21.15.31:0/1661200761 shutdown_connections 2024-04-25T01:54:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f73780a2b90 0x7f73780a5050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f738c103350 0x7f738c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f738c102990 0x7f738c197ec0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 --2- 172.21.15.31:0/1661200761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f738c1017a0 0x7f738c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 -- 172.21.15.31:0/1661200761 >> 172.21.15.31:0/1661200761 conn(0x7f738c0fcd10 msgr2=0x7f738c105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 -- 172.21.15.31:0/1661200761 shutdown_connections 2024-04-25T01:54:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.490+0000 7f7393c31640 1 -- 172.21.15.31:0/1661200761 wait complete. 2024-04-25T01:54:18.502 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:54:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 -- 172.21.15.31:0/735906234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca01006a0 msgr2=0x7efca0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 --2- 172.21.15.31:0/735906234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca01006a0 0x7efca0100ac0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009900 tx=0x7efc9c02a410 comp rx=0 tx=0).stop 2024-04-25T01:54:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 -- 172.21.15.31:0/735906234 shutdown_connections 2024-04-25T01:54:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 --2- 172.21.15.31:0/735906234 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efca0102250 0x7efca0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 --2- 172.21.15.31:0/735906234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca0101890 0x7efca0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 --2- 172.21.15.31:0/735906234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca01006a0 0x7efca0100ac0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 -- 172.21.15.31:0/735906234 >> 172.21.15.31:0/735906234 conn(0x7efca00fbc30 msgr2=0x7efca00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 -- 172.21.15.31:0/735906234 shutdown_connections 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 -- 172.21.15.31:0/735906234 wait complete. 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 Processor -- start 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 -- start start 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efca01006a0 0x7efca0197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca0101890 0x7efca0197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca0102250 0x7efca019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efca010d2f0 con 0x7efca0101890 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efca010d170 con 0x7efca0102250 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.607+0000 7efca7914640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efca010d470 con 0x7efca01006a0 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca4eaf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca0101890 0x7efca0197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca4eaf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca0101890 0x7efca0197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60670/0 (socket says 172.21.15.31:60670) 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca4eaf640 1 -- 172.21.15.31:0/1102242343 learned_addr learned my addr 172.21.15.31:0/1102242343 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca5eb1640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca0102250 0x7efca019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca56b0640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efca01006a0 0x7efca0197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca4eaf640 1 -- 172.21.15.31:0/1102242343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efca01006a0 msgr2=0x7efca0197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca4eaf640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efca01006a0 0x7efca0197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca4eaf640 1 -- 172.21.15.31:0/1102242343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca0102250 msgr2=0x7efca019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca4eaf640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca0102250 0x7efca019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca4eaf640 1 -- 172.21.15.31:0/1102242343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca019f620 con 0x7efca0101890 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca4eaf640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca0101890 0x7efca0197f40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efc9000e9f0 tx=0x7efc9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efc967fc640 1 -- 172.21.15.31:0/1102242343 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9000cd80 con 0x7efca0101890 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efc967fc640 1 -- 172.21.15.31:0/1102242343 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc900108e0 con 0x7efca0101890 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca7914640 1 -- 172.21.15.31:0/1102242343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca019f910 con 0x7efca0101890 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efc967fc640 1 -- 172.21.15.31:0/1102242343 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90018920 con 0x7efca0101890 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.608+0000 7efca7914640 1 -- 172.21.15.31:0/1102242343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca019feb0 con 0x7efca0101890 2024-04-25T01:54:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.609+0000 7efca7914640 1 -- 172.21.15.31:0/1102242343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca0107b00 con 0x7efca0101890 2024-04-25T01:54:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.610+0000 7efc967fc640 1 -- 172.21.15.31:0/1102242343 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efc90010a90 con 0x7efca0101890 2024-04-25T01:54:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.611+0000 7efc967fc640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efc8c0a2c10 0x7efc8c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.611+0000 7efca56b0640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efc8c0a2c10 0x7efc8c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.611+0000 7efc967fc640 1 -- 172.21.15.31:0/1102242343 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7efc900cf350 con 0x7efca0101890 2024-04-25T01:54:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.611+0000 7efca56b0640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efc8c0a2c10 0x7efc8c0a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efc9c000c00 tx=0x7efc9c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:18.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.615+0000 7efc967fc640 1 -- 172.21.15.31:0/1102242343 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efc9009c280 con 0x7efca0101890 2024-04-25T01:54:18.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.803+0000 7efca7914640 1 -- 172.21.15.31:0/1102242343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efca002d390 con 0x7efca0101890 2024-04-25T01:54:18.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.803+0000 7efc967fc640 1 -- 172.21.15.31:0/1102242343 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7efc9009bb80 con 0x7efca0101890 2024-04-25T01:54:18.804 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T01:54:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 -- 172.21.15.31:0/1102242343 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efc8c0a2c10 msgr2=0x7efc8c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efc8c0a2c10 0x7efc8c0a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efc9c000c00 tx=0x7efc9c0314a0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 -- 172.21.15.31:0/1102242343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca0101890 msgr2=0x7efca0197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca0101890 0x7efca0197f40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efc9000e9f0 tx=0x7efc9000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 -- 172.21.15.31:0/1102242343 shutdown_connections 2024-04-25T01:54:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7efc8c0a2c10 0x7efc8c0a50d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca0102250 0x7efca019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efca0101890 0x7efca0197f40 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 --2- 172.21.15.31:0/1102242343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efca01006a0 0x7efca0197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 -- 172.21.15.31:0/1102242343 >> 172.21.15.31:0/1102242343 conn(0x7efca00fbc30 msgr2=0x7efca0104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 -- 172.21.15.31:0/1102242343 shutdown_connections 2024-04-25T01:54:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.805+0000 7efca7914640 1 -- 172.21.15.31:0/1102242343 wait complete. 2024-04-25T01:54:18.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:54:18.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.918+0000 7f6d105f9640 1 -- 172.21.15.31:0/1951479693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08102250 msgr2=0x7f6d08108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.918+0000 7f6d105f9640 1 --2- 172.21.15.31:0/1951479693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08102250 0x7f6d08108ae0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009ca0 tx=0x7f6d0002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.918+0000 7f6d105f9640 1 -- 172.21.15.31:0/1951479693 shutdown_connections 2024-04-25T01:54:18.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.918+0000 7f6d105f9640 1 --2- 172.21.15.31:0/1951479693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08102250 0x7f6d08108ae0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.918+0000 7f6d105f9640 1 --2- 172.21.15.31:0/1951479693 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d08101890 0x7f6d08101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.918+0000 7f6d105f9640 1 --2- 172.21.15.31:0/1951479693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d081006a0 0x7f6d08100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.918+0000 7f6d105f9640 1 -- 172.21.15.31:0/1951479693 >> 172.21.15.31:0/1951479693 conn(0x7f6d080fbc30 msgr2=0x7f6d080fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:18.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.918+0000 7f6d105f9640 1 -- 172.21.15.31:0/1951479693 shutdown_connections 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.918+0000 7f6d105f9640 1 -- 172.21.15.31:0/1951479693 wait complete. 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d105f9640 1 Processor -- start 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d105f9640 1 -- start start 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d105f9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d081006a0 0x7f6d0806a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d105f9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08101890 0x7f6d0819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d105f9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08102250 0x7f6d0819ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d105f9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0810c230 con 0x7f6d08101890 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d105f9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0810c0b0 con 0x7f6d08102250 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d105f9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0810c3b0 con 0x7f6d081006a0 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0db94640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08101890 0x7f6d0819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0db94640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08101890 0x7f6d0819c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60680/0 (socket says 172.21.15.31:60680) 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0db94640 1 -- 172.21.15.31:0/1962906454 learned_addr learned my addr 172.21.15.31:0/1962906454 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:18.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0e395640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d081006a0 0x7f6d0806a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:18.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0eb96640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08102250 0x7f6d0819ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:18.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0db94640 1 -- 172.21.15.31:0/1962906454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d081006a0 msgr2=0x7f6d0806a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0db94640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d081006a0 0x7f6d0806a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0db94640 1 -- 172.21.15.31:0/1962906454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08102250 msgr2=0x7f6d0819ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:18.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0db94640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08102250 0x7f6d0819ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:18.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0db94640 1 -- 172.21.15.31:0/1962906454 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0819f610 con 0x7f6d08101890 2024-04-25T01:54:18.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0eb96640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08102250 0x7f6d0819ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:18.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6d0db94640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08101890 0x7f6d0819c6e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400b4a0 tx=0x7f6d0400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:18.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.919+0000 7f6cff7fe640 1 -- 172.21.15.31:0/1962906454 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04014070 con 0x7f6d08101890 2024-04-25T01:54:18.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.920+0000 7f6d105f9640 1 -- 172.21.15.31:0/1962906454 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0819f900 con 0x7f6d08101890 2024-04-25T01:54:18.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.920+0000 7f6cff7fe640 1 -- 172.21.15.31:0/1962906454 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d04010ac0 con 0x7f6d08101890 2024-04-25T01:54:18.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.920+0000 7f6cff7fe640 1 -- 172.21.15.31:0/1962906454 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04017c30 con 0x7f6d08101890 2024-04-25T01:54:18.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.920+0000 7f6d105f9640 1 -- 172.21.15.31:0/1962906454 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0819fe00 con 0x7f6d08101890 2024-04-25T01:54:18.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.921+0000 7f6d105f9640 1 -- 172.21.15.31:0/1962906454 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d081079c0 con 0x7f6d08101890 2024-04-25T01:54:18.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.922+0000 7f6cff7fe640 1 -- 172.21.15.31:0/1962906454 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6d04017070 con 0x7f6d08101890 2024-04-25T01:54:18.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.923+0000 7f6cff7fe640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6cf40a2b90 0x7f6cf40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:18.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.923+0000 7f6d0e395640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6cf40a2b90 0x7f6cf40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:18.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.923+0000 7f6cff7fe640 1 -- 172.21.15.31:0/1962906454 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f6d040ce3c0 con 0x7f6d08101890 2024-04-25T01:54:18.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.923+0000 7f6d0e395640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6cf40a2b90 0x7f6cf40a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6d081016f0 tx=0x7f6cf800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:18.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:18.927+0000 7f6cff7fe640 1 -- 172.21.15.31:0/1962906454 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d0409b350 con 0x7f6d08101890 2024-04-25T01:54:19.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.149+0000 7f6d105f9640 1 -- 172.21.15.31:0/1962906454 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d0802d170 con 0x7f6d08101890 2024-04-25T01:54:19.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.150+0000 7f6cff7fe640 1 -- 172.21.15.31:0/1962906454 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11923 (secure 0 0 0) 0x7f6d0409ac50 con 0x7f6d08101890 2024-04-25T01:54:19.151 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:19.151 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:52:32.772296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24487,24473,24503,24463,24451]},"id":1}]} 2024-04-25T01:54:19.152 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T01:54:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 -- 172.21.15.31:0/1962906454 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6cf40a2b90 msgr2=0x7f6cf40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6cf40a2b90 0x7f6cf40a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6d081016f0 tx=0x7f6cf800a2d0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 -- 172.21.15.31:0/1962906454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08101890 msgr2=0x7f6d0819c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08101890 0x7f6d0819c6e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400b4a0 tx=0x7f6d0400b960 comp rx=0 tx=0).stop 2024-04-25T01:54:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 -- 172.21.15.31:0/1962906454 shutdown_connections 2024-04-25T01:54:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f6cf40a2b90 0x7f6cf40a5050 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08102250 0x7f6d0819ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08101890 0x7f6d0819c6e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 --2- 172.21.15.31:0/1962906454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d081006a0 0x7f6d0806a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 -- 172.21.15.31:0/1962906454 >> 172.21.15.31:0/1962906454 conn(0x7f6d080fbc30 msgr2=0x7f6d08104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 -- 172.21.15.31:0/1962906454 shutdown_connections 2024-04-25T01:54:19.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.152+0000 7f6d105f9640 1 -- 172.21.15.31:0/1962906454 wait complete. 2024-04-25T01:54:19.165 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:54:19.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.265+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/3473043364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b341006a0 msgr2=0x7f7b34100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.265+0000 7f7b3ae4d640 1 --2- 172.21.15.31:0/3473043364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b341006a0 0x7f7b34100ac0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009900 tx=0x7f7b2802a410 comp rx=0 tx=0).stop 2024-04-25T01:54:19.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.265+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/3473043364 shutdown_connections 2024-04-25T01:54:19.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.265+0000 7f7b3ae4d640 1 --2- 172.21.15.31:0/3473043364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b34102250 0x7f7b34108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.265+0000 7f7b3ae4d640 1 --2- 172.21.15.31:0/3473043364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b34101890 0x7f7b34101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.265+0000 7f7b3ae4d640 1 --2- 172.21.15.31:0/3473043364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b341006a0 0x7f7b34100ac0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.265+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/3473043364 >> 172.21.15.31:0/3473043364 conn(0x7f7b340fbc30 msgr2=0x7f7b340fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.265+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/3473043364 shutdown_connections 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.265+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/3473043364 wait complete. 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b3ae4d640 1 Processor -- start 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b3ae4d640 1 -- start start 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b3ae4d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b341006a0 0x7f7b3406a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b3ae4d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b34101890 0x7f7b3419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b3ae4d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b34102250 0x7f7b341a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b3ae4d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3410c220 con 0x7f7b34101890 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b3ae4d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3410c0a0 con 0x7f7b341006a0 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b3ae4d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3410c3a0 con 0x7f7b34102250 2024-04-25T01:54:19.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b34101890 0x7f7b3419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b34101890 0x7f7b3419eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60688/0 (socket says 172.21.15.31:60688) 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b33fff640 1 -- 172.21.15.31:0/2115868831 learned_addr learned my addr 172.21.15.31:0/2115868831 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b393ea640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b34102250 0x7f7b341a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b33fff640 1 -- 172.21.15.31:0/2115868831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b34102250 msgr2=0x7f7b341a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b33fff640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b34102250 0x7f7b341a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b38be9640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b341006a0 0x7f7b3406a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b33fff640 1 -- 172.21.15.31:0/2115868831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b341006a0 msgr2=0x7f7b3406a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b33fff640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b341006a0 0x7f7b3406a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b33fff640 1 -- 172.21.15.31:0/2115868831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b341a1820 con 0x7f7b34101890 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.266+0000 7f7b33fff640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b34101890 0x7f7b3419eae0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000e9f0 tx=0x7f7b2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.267+0000 7f7b31ffb640 1 -- 172.21.15.31:0/2115868831 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2000cd80 con 0x7f7b34101890 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.267+0000 7f7b31ffb640 1 -- 172.21.15.31:0/2115868831 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b200108e0 con 0x7f7b34101890 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.267+0000 7f7b31ffb640 1 -- 172.21.15.31:0/2115868831 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b20018920 con 0x7f7b34101890 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.267+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/2115868831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b341a1ab0 con 0x7f7b34101890 2024-04-25T01:54:19.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.267+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/2115868831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b341a2050 con 0x7f7b34101890 2024-04-25T01:54:19.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.268+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/2115868831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b34107a40 con 0x7f7b34101890 2024-04-25T01:54:19.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.269+0000 7f7b31ffb640 1 -- 172.21.15.31:0/2115868831 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7b20010a90 con 0x7f7b34101890 2024-04-25T01:54:19.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.270+0000 7f7b31ffb640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7b240a2c40 0x7f7b240a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.270+0000 7f7b38be9640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7b240a2c40 0x7f7b240a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.270+0000 7f7b31ffb640 1 -- 172.21.15.31:0/2115868831 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f7b200cf3c0 con 0x7f7b34101890 2024-04-25T01:54:19.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.270+0000 7f7b38be9640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7b240a2c40 0x7f7b240a5100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7b2802a930 tx=0x7f7b28005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.274+0000 7f7b31ffb640 1 -- 172.21.15.31:0/2115868831 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b2009c2f0 con 0x7f7b34101890 2024-04-25T01:54:19.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:19 smithi040 ceph-mon[32418]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:54:19.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:19 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/498268346' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:19.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:19 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1661200761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:19.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:19 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1102242343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:19.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:19 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1962906454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:19.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:19 smithi134 ceph-mon[33098]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:54:19.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:19 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/498268346' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:19.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:19 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1661200761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:19.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:19 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1102242343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:19.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:19 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1962906454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:19.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.459+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/2115868831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b340623e0 con 0x7f7b34101890 2024-04-25T01:54:19.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.461+0000 7f7b31ffb640 1 -- 172.21.15.31:0/2115868831 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22478 (secure 0 0 0) 0x7f7b2009bbf0 con 0x7f7b34101890 2024-04-25T01:54:19.462 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:19.462 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:53:24.952281+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:54:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.463+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/2115868831 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7b240a2c40 msgr2=0x7f7b240a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.463+0000 7f7b3ae4d640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7b240a2c40 0x7f7b240a5100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7b2802a930 tx=0x7f7b28005fd0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.463+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/2115868831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b34101890 msgr2=0x7f7b3419eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.463+0000 7f7b3ae4d640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b34101890 0x7f7b3419eae0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000e9f0 tx=0x7f7b2000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.463+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/2115868831 shutdown_connections 2024-04-25T01:54:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.463+0000 7f7b3ae4d640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f7b240a2c40 0x7f7b240a5100 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.463+0000 7f7b3ae4d640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b34102250 0x7f7b341a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.463+0000 7f7b3ae4d640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b34101890 0x7f7b3419eae0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.463+0000 7f7b3ae4d640 1 --2- 172.21.15.31:0/2115868831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b341006a0 0x7f7b3406a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.463+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/2115868831 >> 172.21.15.31:0/2115868831 conn(0x7f7b340fbc30 msgr2=0x7f7b34104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.463+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/2115868831 shutdown_connections 2024-04-25T01:54:19.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.464+0000 7f7b3ae4d640 1 -- 172.21.15.31:0/2115868831 wait complete. 2024-04-25T01:54:19.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:19 smithi031 ceph-mon[29204]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T01:54:19.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:19 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/498268346' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:19.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:19 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1661200761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:19.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:19 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1102242343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:19.466 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:19 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1962906454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:19.477 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T01:52:24.898441+0000', 'modified': '2024-04-25T01:52:32.772296+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24503, 'mds_1': 24487, 'mds_2': 24473, 'mds_3': 24463, 'mds_4': 24451}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14622': {'gid': 14622, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.31:6839/3704800465', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6838', 'nonce': 3704800465}, {'type': 'v1', 'addr': '172.21.15.31:6839', 'nonce': 3704800465}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14634': {'gid': 14634, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.31:6841/4277792607', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6840', 'nonce': 4277792607}, {'type': 'v1', 'addr': '172.21.15.31:6841', 'nonce': 4277792607}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24407': {'gid': 24407, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6833/3875547004', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 3875547004}, {'type': 'v1', 'addr': '172.21.15.40:6833', 'nonce': 3875547004}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24412': {'gid': 24412, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.134:6833/400138993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6832', 'nonce': 400138993}, {'type': 'v1', 'addr': '172.21.15.134:6833', 'nonce': 400138993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6835/538218297', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 538218297}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 538218297}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.134:6835/1266939870', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6834', 'nonce': 1266939870}, {'type': 'v1', 'addr': '172.21.15.134:6835', 'nonce': 1266939870}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.134:6837/2058394187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6836', 'nonce': 2058394187}, {'type': 'v1', 'addr': '172.21.15.134:6837', 'nonce': 2058394187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.40:6837/1901184087', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6836', 'nonce': 1901184087}, {'type': 'v1', 'addr': '172.21.15.40:6837', 'nonce': 1901184087}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.134:6839/450925094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6838', 'nonce': 450925094}, {'type': 'v1', 'addr': '172.21.15.134:6839', 'nonce': 450925094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6839/3236110559', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 3236110559}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 3236110559}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24503, 'qdb_cluster': [24487, 24473, 24503, 24463, 24451]} 2024-04-25T01:54:19.477 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T01:54:19.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-25T01:54:19.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.579+0000 7f8b89d90640 1 -- 172.21.15.31:0/1916295046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b84102990 msgr2=0x7f8b84102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.579+0000 7f8b89d90640 1 --2- 172.21.15.31:0/1916295046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b84102990 0x7f8b84102e10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8b78009900 tx=0x7f8b7802a410 comp rx=0 tx=0).stop 2024-04-25T01:54:19.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.579+0000 7f8b89d90640 1 -- 172.21.15.31:0/1916295046 shutdown_connections 2024-04-25T01:54:19.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.579+0000 7f8b89d90640 1 --2- 172.21.15.31:0/1916295046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b84103350 0x7f8b84109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.579+0000 7f8b89d90640 1 --2- 172.21.15.31:0/1916295046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b84102990 0x7f8b84102e10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.579+0000 7f8b89d90640 1 --2- 172.21.15.31:0/1916295046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b841017a0 0x7f8b84101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.579+0000 7f8b89d90640 1 -- 172.21.15.31:0/1916295046 >> 172.21.15.31:0/1916295046 conn(0x7f8b840fcd10 msgr2=0x7f8b840ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.579+0000 7f8b89d90640 1 -- 172.21.15.31:0/1916295046 shutdown_connections 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.579+0000 7f8b89d90640 1 -- 172.21.15.31:0/1916295046 wait complete. 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.579+0000 7f8b89d90640 1 Processor -- start 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.579+0000 7f8b89d90640 1 -- start start 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b89d90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b841017a0 0x7f8b840754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b89d90640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b84102990 0x7f8b84073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b89d90640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b84103350 0x7f8b84074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b89d90640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8410d2d0 con 0x7f8b84102990 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b89d90640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8410d150 con 0x7f8b841017a0 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b89d90640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8410d450 con 0x7f8b84103350 2024-04-25T01:54:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b82ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b84102990 0x7f8b84073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b82ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b84102990 0x7f8b84073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60708/0 (socket says 172.21.15.31:60708) 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b82ffd640 1 -- 172.21.15.31:0/2405061620 learned_addr learned my addr 172.21.15.31:0/2405061620 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b841017a0 0x7f8b840754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b83fff640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b84103350 0x7f8b84074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b82ffd640 1 -- 172.21.15.31:0/2405061620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b84103350 msgr2=0x7f8b84074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b82ffd640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b84103350 0x7f8b84074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b82ffd640 1 -- 172.21.15.31:0/2405061620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b841017a0 msgr2=0x7f8b840754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b82ffd640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b841017a0 0x7f8b840754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b82ffd640 1 -- 172.21.15.31:0/2405061620 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b84074900 con 0x7f8b84102990 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b82ffd640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b84102990 0x7f8b84073b30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8b78005940 tx=0x7f8b780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b80ff9640 1 -- 172.21.15.31:0/2405061620 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7803f030 con 0x7f8b84102990 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b80ff9640 1 -- 172.21.15.31:0/2405061620 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b78003cb0 con 0x7f8b84102990 2024-04-25T01:54:19.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.580+0000 7f8b89d90640 1 -- 172.21.15.31:0/2405061620 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b841a1a50 con 0x7f8b84102990 2024-04-25T01:54:19.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.581+0000 7f8b80ff9640 1 -- 172.21.15.31:0/2405061620 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78035070 con 0x7f8b84102990 2024-04-25T01:54:19.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.581+0000 7f8b89d90640 1 -- 172.21.15.31:0/2405061620 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b841a1fa0 con 0x7f8b84102990 2024-04-25T01:54:19.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b80ff9640 1 -- 172.21.15.31:0/2405061620 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8b78042030 con 0x7f8b84102990 2024-04-25T01:54:19.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b8cb7e740 1 Processor -- start 2024-04-25T01:54:19.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b8cb7e740 1 -- start start 2024-04-25T01:54:19.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb95fa860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fae30 0x561fb95ff930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fffa0 0x561fb9602430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561fb955df00 con 0x561fb95fa440 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561fb9506750 con 0x561fb95fae30 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561fb9559ec0 con 0x561fb95fffa0 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b83fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb95fa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b83fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb95fa860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60710/0 (socket says 172.21.15.31:60710) 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b83fff640 1 -- 172.21.15.31:0/2682837556 learned_addr learned my addr 172.21.15.31:0/2682837556 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b80ff9640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8b6c0a2b90 0x7f8b6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.583+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2682837556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fae30 0x561fb95ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b82ffd640 1 --2- 172.21.15.31:0/2682837556 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fffa0 0x561fb9602430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b83fff640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8b6c0a2b90 0x7f8b6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b83fff640 1 -- 172.21.15.31:0/2682837556 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fffa0 msgr2=0x561fb9602430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b83fff640 1 --2- 172.21.15.31:0/2682837556 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fffa0 0x561fb9602430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b83fff640 1 -- 172.21.15.31:0/2682837556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fae30 msgr2=0x561fb95ff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b83fff640 1 --2- 172.21.15.31:0/2682837556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fae30 0x561fb95ff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b83fff640 1 -- 172.21.15.31:0/2682837556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fb92722d0 con 0x561fb95fa440 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b80ff9640 1 -- 172.21.15.31:0/2405061620 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f8b780ef550 con 0x7f8b84102990 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2682837556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fae30 0x561fb95ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b83fff640 1 --2- 172.21.15.31:0/2682837556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb95fa860 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c013a50 tx=0x7f8b7c013f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aea384942824f6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b69ffb640 1 -- 172.21.15.31:0/2682837556 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c013050 con 0x561fb95fa440 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b69ffb640 1 -- 172.21.15.31:0/2682837556 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b7c022070 con 0x561fb95fa440 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.584+0000 7f8b69ffb640 1 -- 172.21.15.31:0/2682837556 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c012650 con 0x561fb95fa440 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2682837556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 msgr2=0x561fb95fa860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b83fff640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8b6c0a2b90 0x7f8b6c0a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c021000 tx=0x7f8b7c01f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2682837556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb95fa860 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c013a50 tx=0x7f8b7c013f10 comp rx=0 tx=0).stop 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2682837556 shutdown_connections 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2682837556 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fffa0 0x561fb9602430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2682837556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fae30 0x561fb95ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2682837556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb95fa860 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2682837556 >> 172.21.15.31:0/2682837556 conn(0x561fb95f5ab0 msgr2=0x561fb95f5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2682837556 shutdown_connections 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2682837556 wait complete. 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b8cb7e740 1 -- start start 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.585+0000 7f8b8cb7e740 1 Processor -- start 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fa440 0x561fb9604860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fae30 0x561fb9604da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fffa0 0x561fb960bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561fb9602f50 con 0x561fb95fae30 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561fb9519cf0 con 0x561fb95fa440 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561fb96030d0 con 0x561fb95fffa0 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b8cb7e740 10 client.0 _ll_register_callbacks cb 0x561fb95196e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b82ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fae30 0x561fb9604da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b82ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fae30 0x561fb9604da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60712/0 (socket says 172.21.15.31:60712) 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b82ffd640 1 -- 172.21.15.31:0/2738677803 learned_addr learned my addr 172.21.15.31:0/2738677803 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fa440 0x561fb9604860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b83fff640 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fffa0 0x561fb960bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b82ffd640 1 -- 172.21.15.31:0/2738677803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fffa0 msgr2=0x561fb960bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b82ffd640 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fffa0 0x561fb960bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b82ffd640 1 -- 172.21.15.31:0/2738677803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fa440 msgr2=0x561fb9604860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b82ffd640 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fa440 0x561fb9604860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b82ffd640 1 -- 172.21.15.31:0/2738677803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fb933c6b0 con 0x561fb95fae30 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b83fff640 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fffa0 0x561fb960bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b82ffd640 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fae30 0x561fb9604da0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8b780314e0 tx=0x7f8b7808ae10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7099d22243002138 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2738677803 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78049040 con 0x561fb95fae30 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2738677803 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f8b780b8500 con 0x561fb95fae30 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2738677803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b7c022070 con 0x561fb95fae30 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2738677803 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8b780bb9d0 con 0x561fb95fae30 2024-04-25T01:54:19.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b8cb7e740 10 client.15249 fetch_fsmap learned FSMap version 26 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.586+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2738677803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561fb960edc0 con 0x561fb95fae30 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.587+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2738677803 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b7c022070 con 0x561fb95fae30 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.587+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2738677803 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8b780bc0f0 con 0x561fb95fae30 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.587+0000 7f8b8cb7e740 10 client.15249 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.587+0000 7f8b8cb7e740 10 client.15249 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.587+0000 7f8b8cb7e740 20 client.15249 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.587+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x561fb96114b0 0x561fb9612920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.587+0000 7f8b8cb7e740 4 client.15249 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.587+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2738677803 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fb956b440 con 0x561fb96114b0 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.587+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x561fb96114b0 0x561fb9612920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.588+0000 7f8b62ffd640 10 client.15249 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.588+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x561fb96114b0 0x561fb9612920 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=11f3469fad765cab server_cookie=bd2a6ee565064e11 in_seq=0 out_seq=0 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.588+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2738677803 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561fb956b440 con 0x561fb96114b0 2024-04-25T01:54:19.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.588+0000 7f8b62ffd640 10 client.15249 handle_command_reply: tid=0 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.588+0000 7f8b8cb7e740 1 client.15249 shutdown 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2738677803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fae30 msgr2=0x561fb9604da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fae30 0x561fb9604da0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8b780314e0 tx=0x7f8b7808ae10 comp rx=0 tx=0).stop 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2738677803 shutdown_connections 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x561fb96114b0 0x561fb9612920 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fffa0 0x561fb960bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fae30 0x561fb9604da0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b62ffd640 0 client.15249 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2738677803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fa440 0x561fb9604860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2738677803 >> 172.21.15.31:0/2738677803 conn(0x561fb95f5ab0 msgr2=0x561fb95f4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2738677803 shutdown_connections 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2738677803 wait complete. 2024-04-25T01:54:19.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.589+0000 7f8b8cb7e740 20 client.15249 trim_cache size 0 max 16384 2024-04-25T01:54:19.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.607+0000 7f8b8cb7e740 1 Processor -- start 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.607+0000 7f8b8cb7e740 1 -- start start 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.607+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9602390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.607+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fae30 0x561fb960bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.607+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fffa0 0x561fb95f4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.607+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561fb9610c60 con 0x561fb95fa440 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.607+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561fb9611330 con 0x561fb95fffa0 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.607+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561fb9611a00 con 0x561fb95fae30 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9602390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9602390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60714/0 (socket says 172.21.15.31:60714) 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b837fe640 1 -- 172.21.15.31:0/2985261911 learned_addr learned my addr 172.21.15.31:0/2985261911 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b83fff640 1 --2- 172.21.15.31:0/2985261911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fffa0 0x561fb95f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b837fe640 1 -- 172.21.15.31:0/2985261911 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fae30 msgr2=0x561fb960bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2985261911 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fae30 0x561fb960bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b837fe640 1 -- 172.21.15.31:0/2985261911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fffa0 msgr2=0x561fb95f4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2985261911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fffa0 0x561fb95f4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b837fe640 1 -- 172.21.15.31:0/2985261911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fb95fb6c0 con 0x561fb95fa440 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2985261911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9602390 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x561fb95fb8a0 tx=0x7f8b74005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be19a86014aa194 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b697fa640 1 -- 172.21.15.31:0/2985261911 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b740200f0 con 0x561fb95fa440 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b697fa640 1 -- 172.21.15.31:0/2985261911 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b74020740 con 0x561fb95fa440 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.608+0000 7f8b697fa640 1 -- 172.21.15.31:0/2985261911 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74010d00 con 0x561fb95fa440 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.609+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2985261911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 msgr2=0x561fb9602390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.609+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2985261911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9602390 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x561fb95fb8a0 tx=0x7f8b74005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.609+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2985261911 shutdown_connections 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.609+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2985261911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fffa0 0x561fb95f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.609+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2985261911 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fae30 0x561fb960bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.609+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2985261911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9602390 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.609+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2985261911 >> 172.21.15.31:0/2985261911 conn(0x561fb95f5ab0 msgr2=0x561fb95f0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.609+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2985261911 shutdown_connections 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.609+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2985261911 wait complete. 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.609+0000 7f8b8cb7e740 1 -- start start 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.609+0000 7f8b8cb7e740 1 Processor -- start 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9609fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fae30 0x561fb960a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b8cb7e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fffa0 0x561fb960aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561fb96127a0 con 0x561fb95fa440 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561fb96120d0 con 0x561fb95fffa0 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b8cb7e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561fb9612e70 con 0x561fb95fae30 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b8cb7e740 10 client.0 _ll_register_callbacks cb 0x7f8b78035f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9609fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9609fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60718/0 (socket says 172.21.15.31:60718) 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b837fe640 1 -- 172.21.15.31:0/2457095030 learned_addr learned my addr 172.21.15.31:0/2457095030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b83fff640 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fffa0 0x561fb960aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b82ffd640 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fae30 0x561fb960a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b837fe640 1 -- 172.21.15.31:0/2457095030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fae30 msgr2=0x561fb960a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fae30 0x561fb960a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b837fe640 1 -- 172.21.15.31:0/2457095030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fffa0 msgr2=0x561fb960aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fffa0 0x561fb960aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b837fe640 1 -- 172.21.15.31:0/2457095030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fb960b260 con 0x561fb95fa440 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b83fff640 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fffa0 0x561fb960aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b82ffd640 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fae30 0x561fb960a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b837fe640 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9609fe0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009c90 tx=0x7f8b740208f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7926cfa9617d8803 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2457095030 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74030070 con 0x561fb95fa440 2024-04-25T01:54:19.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2457095030 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f8b74010a80 con 0x561fb95fa440 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2457095030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b74020740 con 0x561fb95fa440 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.610+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2457095030 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8b7403b050 con 0x561fb95fa440 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b8cb7e740 10 client.15255 fetch_fsmap learned FSMap version 26 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2457095030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561fb9677670 con 0x561fb95fa440 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2457095030 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b74020740 con 0x561fb95fa440 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2457095030 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f8b74005250 con 0x561fb95fa440 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b8cb7e740 10 client.15255 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b8cb7e740 10 client.15255 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b8cb7e740 20 client.15255 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x561fb967bf20 0x561fb967ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b8cb7e740 4 client.15255 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2457095030 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "status"}) -- 0x561fb9565790 con 0x561fb967bf20 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b82ffd640 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x561fb967bf20 0x561fb967ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b62ffd640 10 client.15255 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.611+0000 7f8b82ffd640 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x561fb967bf20 0x561fb967ac30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b91beeebcd589f49 server_cookie=8356ab35d0ac303a in_seq=0 out_seq=0 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b62ffd640 1 -- 172.21.15.31:0/2457095030 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x561fb9565790 con 0x561fb967bf20 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b62ffd640 10 client.15255 handle_command_reply: tid=0 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 1 client.15255 shutdown 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2457095030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 msgr2=0x561fb9609fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9609fe0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009c90 tx=0x7f8b740208f0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2457095030 shutdown_connections 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x561fb967bf20 0x561fb967ac30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb95fffa0 0x561fb960aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561fb95fae30 0x561fb960a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b62ffd640 0 client.15255 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 1 --2- 172.21.15.31:0/2457095030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561fb95fa440 0x561fb9609fe0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2457095030 >> 172.21.15.31:0/2457095030 conn(0x561fb95f5ab0 msgr2=0x561fb95f8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2457095030 shutdown_connections 2024-04-25T01:54:19.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 1 -- 172.21.15.31:0/2457095030 wait complete. 2024-04-25T01:54:19.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.612+0000 7f8b8cb7e740 20 client.15255 trim_cache size 0 max 16384 2024-04-25T01:54:19.621 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:54:19.621 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:54:19.621 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 4, 2024-04-25T01:54:19.621 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24451, 2024-04-25T01:54:19.621 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T01:54:19.621 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T01:54:19.621 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T01:54:19.621 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 107.87211408, 2024-04-25T01:54:19.621 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T01:54:19.621 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2024-04-25T01:54:19.622 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:54:19.622 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 135.44975480100001 2024-04-25T01:54:19.622 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:54:19.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 -- 172.21.15.31:0/2405061620 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8b6c0a2b90 msgr2=0x7f8b6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8b6c0a2b90 0x7f8b6c0a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c021000 tx=0x7f8b7c01f000 comp rx=0 tx=0).stop 2024-04-25T01:54:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 -- 172.21.15.31:0/2405061620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b84102990 msgr2=0x7f8b84073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b84102990 0x7f8b84073b30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8b78005940 tx=0x7f8b780042f0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 -- 172.21.15.31:0/2405061620 shutdown_connections 2024-04-25T01:54:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f8b6c0a2b90 0x7f8b6c0a5050 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b84103350 0x7f8b84074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b84102990 0x7f8b84073b30 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 --2- 172.21.15.31:0/2405061620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b841017a0 0x7f8b840754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 -- 172.21.15.31:0/2405061620 >> 172.21.15.31:0/2405061620 conn(0x7f8b840fcd10 msgr2=0x7f8b84105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 -- 172.21.15.31:0/2405061620 shutdown_connections 2024-04-25T01:54:19.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.613+0000 7f8b89d90640 1 -- 172.21.15.31:0/2405061620 wait complete. 2024-04-25T01:54:19.625 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T01:54:19.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.726+0000 7fec91461640 1 -- 172.21.15.31:0/1171146158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec8c102980 msgr2=0x7fec8c102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.726+0000 7fec91461640 1 --2- 172.21.15.31:0/1171146158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec8c102980 0x7fec8c102e00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fec80009900 tx=0x7fec8002a410 comp rx=0 tx=0).stop 2024-04-25T01:54:19.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.726+0000 7fec91461640 1 -- 172.21.15.31:0/1171146158 shutdown_connections 2024-04-25T01:54:19.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.726+0000 7fec91461640 1 --2- 172.21.15.31:0/1171146158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec8c103340 0x7fec8c109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.726+0000 7fec91461640 1 --2- 172.21.15.31:0/1171146158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec8c102980 0x7fec8c102e00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.726+0000 7fec91461640 1 --2- 172.21.15.31:0/1171146158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec8c101790 0x7fec8c101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.726+0000 7fec91461640 1 -- 172.21.15.31:0/1171146158 >> 172.21.15.31:0/1171146158 conn(0x7fec8c0fcd20 msgr2=0x7fec8c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.726+0000 7fec91461640 1 -- 172.21.15.31:0/1171146158 shutdown_connections 2024-04-25T01:54:19.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.726+0000 7fec91461640 1 -- 172.21.15.31:0/1171146158 wait complete. 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec91461640 1 Processor -- start 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec91461640 1 -- start start 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec91461640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec8c101790 0x7fec8c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec91461640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec8c102980 0x7fec8c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec91461640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec8c103340 0x7fec8c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec91461640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c10d310 con 0x7fec8c102980 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec91461640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c10d190 con 0x7fec8c101790 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec91461640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c10d490 con 0x7fec8c103340 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec8c102980 0x7fec8c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec8c102980 0x7fec8c197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60724/0 (socket says 172.21.15.31:60724) 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8a7fc640 1 -- 172.21.15.31:0/1115474646 learned_addr learned my addr 172.21.15.31:0/1115474646 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8b7fe640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec8c103340 0x7fec8c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8affd640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec8c101790 0x7fec8c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8a7fc640 1 -- 172.21.15.31:0/1115474646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec8c103340 msgr2=0x7fec8c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec8c103340 0x7fec8c19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8a7fc640 1 -- 172.21.15.31:0/1115474646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec8c101790 msgr2=0x7fec8c197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec8c101790 0x7fec8c197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8a7fc640 1 -- 172.21.15.31:0/1115474646 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8c19f630 con 0x7fec8c102980 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8b7fe640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec8c103340 0x7fec8c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.727+0000 7fec8affd640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec8c101790 0x7fec8c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.728+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec8c102980 0x7fec8c197f50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fec800314e0 tx=0x7fec80003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.728+0000 7fec73fff640 1 -- 172.21.15.31:0/1115474646 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8003f070 con 0x7fec8c102980 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.728+0000 7fec73fff640 1 -- 172.21.15.31:0/1115474646 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec80031e40 con 0x7fec8c102980 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.728+0000 7fec91461640 1 -- 172.21.15.31:0/1115474646 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8c19f8c0 con 0x7fec8c102980 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.728+0000 7fec73fff640 1 -- 172.21.15.31:0/1115474646 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80035030 con 0x7fec8c102980 2024-04-25T01:54:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.728+0000 7fec91461640 1 -- 172.21.15.31:0/1115474646 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec8c19fdb0 con 0x7fec8c102980 2024-04-25T01:54:19.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.730+0000 7fec73fff640 1 -- 172.21.15.31:0/1115474646 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fec80042070 con 0x7fec8c102980 2024-04-25T01:54:19.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.730+0000 7fec94168740 1 Processor -- start 2024-04-25T01:54:19.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.730+0000 7fec94168740 1 -- start start 2024-04-25T01:54:19.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.730+0000 7fec94168740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ed5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.730+0000 7fec94168740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963ed5e30 0x557963eda930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.730+0000 7fec94168740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963edd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.730+0000 7fec94168740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557963e38f00 con 0x557963edafa0 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.730+0000 7fec94168740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557963de1750 con 0x557963ed5440 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.730+0000 7fec94168740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557963e34ec0 con 0x557963ed5e30 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec73fff640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fec740a2b90 0x7fec740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963edd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963edd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60728/0 (socket says 172.21.15.31:60728) 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8a7fc640 1 -- 172.21.15.31:0/317651436 learned_addr learned my addr 172.21.15.31:0/317651436 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8b7fe640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fec740a2b90 0x7fec740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8b7fe640 1 --2- 172.21.15.31:0/317651436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ed5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8affd640 1 --2- 172.21.15.31:0/317651436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963ed5e30 0x557963eda930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec73fff640 1 -- 172.21.15.31:0/1115474646 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fec800ef9d0 con 0x7fec8c102980 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8a7fc640 1 -- 172.21.15.31:0/317651436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963ed5e30 msgr2=0x557963eda930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/317651436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963ed5e30 0x557963eda930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8a7fc640 1 -- 172.21.15.31:0/317651436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 msgr2=0x557963ed5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/317651436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ed5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8a7fc640 1 -- 172.21.15.31:0/317651436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557963b4d2d0 con 0x557963edafa0 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8b7fe640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fec740a2b90 0x7fec740a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fec7c012a20 tx=0x7fec7c00ac70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/317651436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963edd430 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fec800bc1f0 tx=0x7fec80034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957d8ba874c3fd17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec717fa640 1 -- 172.21.15.31:0/317651436 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80049050 con 0x557963edafa0 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec717fa640 1 -- 172.21.15.31:0/317651436 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec800bc960 con 0x557963edafa0 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.731+0000 7fec717fa640 1 -- 172.21.15.31:0/317651436 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec800bc4c0 con 0x557963edafa0 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.732+0000 7fec94168740 1 -- 172.21.15.31:0/317651436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 msgr2=0x557963edd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.732+0000 7fec94168740 1 --2- 172.21.15.31:0/317651436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963edd430 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fec800bc1f0 tx=0x7fec80034000 comp rx=0 tx=0).stop 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.732+0000 7fec94168740 1 -- 172.21.15.31:0/317651436 shutdown_connections 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.732+0000 7fec94168740 1 --2- 172.21.15.31:0/317651436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963edd430 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.732+0000 7fec94168740 1 --2- 172.21.15.31:0/317651436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963ed5e30 0x557963eda930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.732+0000 7fec94168740 1 --2- 172.21.15.31:0/317651436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ed5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.732+0000 7fec94168740 1 -- 172.21.15.31:0/317651436 >> 172.21.15.31:0/317651436 conn(0x557963ed0ab0 msgr2=0x557963ed0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.732+0000 7fec94168740 1 -- 172.21.15.31:0/317651436 shutdown_connections 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.732+0000 7fec94168740 1 -- 172.21.15.31:0/317651436 wait complete. 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.732+0000 7fec94168740 1 -- start start 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.732+0000 7fec94168740 1 Processor -- start 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec94168740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963edf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec94168740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963ed5e30 0x557963edf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec94168740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963ee6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec94168740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557963e38f00 con 0x557963edafa0 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec94168740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557963df4cf0 con 0x557963ed5440 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec94168740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557963eddd10 con 0x557963ed5e30 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec94168740 10 client.0 _ll_register_callbacks cb 0x557963df46e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963ee6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963ee6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60730/0 (socket says 172.21.15.31:60730) 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8b7fe640 1 -- 172.21.15.31:0/912778964 learned_addr learned my addr 172.21.15.31:0/912778964 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963ed5e30 0x557963edf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8affd640 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963edf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8b7fe640 1 -- 172.21.15.31:0/912778964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963ed5e30 msgr2=0x557963edf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8b7fe640 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963ed5e30 0x557963edf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8b7fe640 1 -- 172.21.15.31:0/912778964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 msgr2=0x557963edf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8b7fe640 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963edf380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8b7fe640 1 -- 172.21.15.31:0/912778964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557963c176b0 con 0x557963edafa0 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8affd640 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963edf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec8b7fe640 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963ee6900 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00fe70 tx=0x7fec7c005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2544e2fb0610b715 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec6eff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec6eff5640 1 -- 172.21.15.31:0/912778964 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c028430 con 0x557963edafa0 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec6eff5640 1 -- 172.21.15.31:0/912778964 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fec7c01e3f0 con 0x557963edafa0 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.733+0000 7fec94168740 1 -- 172.21.15.31:0/912778964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec800bc960 con 0x557963edafa0 2024-04-25T01:54:19.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec6eff5640 1 -- 172.21.15.31:0/912778964 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fec7c01e5f0 con 0x557963edafa0 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec94168740 10 client.15267 fetch_fsmap learned FSMap version 26 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec94168740 1 -- 172.21.15.31:0/912778964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557963ee9950 con 0x557963edafa0 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec6eff5640 1 -- 172.21.15.31:0/912778964 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec800bc960 con 0x557963edafa0 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec6eff5640 1 -- 172.21.15.31:0/912778964 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fec7c028860 con 0x557963edafa0 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec94168740 10 client.15267 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec94168740 10 client.15267 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec94168740 20 client.15267 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec94168740 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x557963eec080 0x557963eed4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec94168740 4 client.15267 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec94168740 1 -- 172.21.15.31:0/912778964 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557963e46440 con 0x557963eec080 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.734+0000 7fec8affd640 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x557963eec080 0x557963eed4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.735+0000 7fec6eff5640 10 client.15267 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.735+0000 7fec8affd640 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x557963eec080 0x557963eed4f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e278cb5246f97624 server_cookie=6a726b9b5225213c in_seq=0 out_seq=0 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.735+0000 7fec6eff5640 1 -- 172.21.15.31:0/912778964 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557963e46440 con 0x557963eec080 2024-04-25T01:54:19.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.735+0000 7fec6eff5640 10 client.15267 handle_command_reply: tid=0 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.735+0000 7fec94168740 1 client.15267 shutdown 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec94168740 1 -- 172.21.15.31:0/912778964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 msgr2=0x557963ee6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec94168740 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963ee6900 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00fe70 tx=0x7fec7c005eb0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec94168740 1 -- 172.21.15.31:0/912778964 shutdown_connections 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec94168740 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x557963eec080 0x557963eed4f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec94168740 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963edafa0 0x557963ee6900 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec94168740 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963ed5e30 0x557963edf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec6eff5640 0 client.15267 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec94168740 1 --2- 172.21.15.31:0/912778964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963edf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec94168740 1 -- 172.21.15.31:0/912778964 >> 172.21.15.31:0/912778964 conn(0x557963ed0ab0 msgr2=0x557963ecfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec94168740 1 -- 172.21.15.31:0/912778964 shutdown_connections 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec94168740 1 -- 172.21.15.31:0/912778964 wait complete. 2024-04-25T01:54:19.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.736+0000 7fec94168740 20 client.15267 trim_cache size 0 max 16384 2024-04-25T01:54:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec94168740 1 Processor -- start 2024-04-25T01:54:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec94168740 1 -- start start 2024-04-25T01:54:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec94168740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ecb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec94168740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec94168740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963edafa0 0x557963edf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec94168740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557963eeb830 con 0x557963ed5e30 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec94168740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557963eebf00 con 0x557963ed5440 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec94168740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557963eec5d0 con 0x557963edafa0 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec8a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec8a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60736/0 (socket says 172.21.15.31:60736) 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec8a7fc640 1 -- 172.21.15.31:0/3071896576 learned_addr learned my addr 172.21.15.31:0/3071896576 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec8a7fc640 1 -- 172.21.15.31:0/3071896576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963edafa0 msgr2=0x557963edf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/3071896576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963edafa0 0x557963edf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec8a7fc640 1 -- 172.21.15.31:0/3071896576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 msgr2=0x557963ecb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/3071896576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ecb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.754+0000 7fec8a7fc640 1 -- 172.21.15.31:0/3071896576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557963eddcb0 con 0x557963ed5e30 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/3071896576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fec800ed870 tx=0x7fec8002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae507f609475f363 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec70ff9640 1 -- 172.21.15.31:0/3071896576 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec800bbbc0 con 0x557963ed5e30 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec70ff9640 1 -- 172.21.15.31:0/3071896576 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec800bc960 con 0x557963ed5e30 2024-04-25T01:54:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec70ff9640 1 -- 172.21.15.31:0/3071896576 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec800bc720 con 0x557963ed5e30 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec94168740 1 -- 172.21.15.31:0/3071896576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 msgr2=0x557963ee6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec94168740 1 --2- 172.21.15.31:0/3071896576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fec800ed870 tx=0x7fec8002a410 comp rx=0 tx=0).stop 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec94168740 1 -- 172.21.15.31:0/3071896576 shutdown_connections 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec94168740 1 --2- 172.21.15.31:0/3071896576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963edafa0 0x557963edf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec94168740 1 --2- 172.21.15.31:0/3071896576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec94168740 1 --2- 172.21.15.31:0/3071896576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ecb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec94168740 1 -- 172.21.15.31:0/3071896576 >> 172.21.15.31:0/3071896576 conn(0x557963ed0ab0 msgr2=0x557963ed0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec94168740 1 -- 172.21.15.31:0/3071896576 shutdown_connections 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.755+0000 7fec94168740 1 -- 172.21.15.31:0/3071896576 wait complete. 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec94168740 1 -- start start 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec94168740 1 Processor -- start 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec94168740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ed9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec94168740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec94168740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963edafa0 0x557963f50020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec94168740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557963eed370 con 0x557963ed5e30 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec94168740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557963eecca0 con 0x557963ed5440 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec94168740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557963eeda40 con 0x557963edafa0 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec94168740 10 client.0 _ll_register_callbacks cb 0x7fec7c00fb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec8a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec8a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60742/0 (socket says 172.21.15.31:60742) 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec8a7fc640 1 -- 172.21.15.31:0/77047612 learned_addr learned my addr 172.21.15.31:0/77047612 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec8affd640 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ed9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec8a7fc640 1 -- 172.21.15.31:0/77047612 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963edafa0 msgr2=0x557963f50020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963edafa0 0x557963f50020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec8a7fc640 1 -- 172.21.15.31:0/77047612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 msgr2=0x557963ed9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ed9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec8a7fc640 1 -- 172.21.15.31:0/77047612 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557963f50710 con 0x557963ed5e30 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec8affd640 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ed9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:19.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec8a7fc640 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fec800c49a0 tx=0x7fec80005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6956d693e316579 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec6eff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec6eff5640 1 -- 172.21.15.31:0/77047612 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80085020 con 0x557963ed5e30 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec6eff5640 1 -- 172.21.15.31:0/77047612 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fec800c4c50 con 0x557963ed5e30 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.756+0000 7fec94168740 1 -- 172.21.15.31:0/77047612 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec800bc960 con 0x557963ed5e30 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec6eff5640 1 -- 172.21.15.31:0/77047612 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fec80049070 con 0x557963ed5e30 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec94168740 10 client.15273 fetch_fsmap learned FSMap version 26 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec94168740 1 -- 172.21.15.31:0/77047612 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557963f51a10 con 0x557963ed5e30 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec6eff5640 1 -- 172.21.15.31:0/77047612 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec800bc960 con 0x557963ed5e30 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec6eff5640 1 -- 172.21.15.31:0/77047612 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fec800c0170 con 0x557963ed5e30 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec94168740 10 client.15273 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec94168740 10 client.15273 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec94168740 20 client.15273 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec94168740 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x557963f565f0 0x557963f55300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec94168740 4 client.15273 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec94168740 1 -- 172.21.15.31:0/77047612 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "status"}) -- 0x557963e40790 con 0x557963f565f0 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.757+0000 7fec8affd640 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x557963f565f0 0x557963f55300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec8affd640 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x557963f565f0 0x557963f55300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b7af3a2dc36f24e7 server_cookie=e3de0fdc80618b in_seq=0 out_seq=0 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec6eff5640 10 client.15273 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec6eff5640 1 -- 172.21.15.31:0/77047612 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x557963e40790 con 0x557963f565f0 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec6eff5640 10 client.15273 handle_command_reply: tid=0 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 1 client.15273 shutdown 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 1 -- 172.21.15.31:0/77047612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 msgr2=0x557963ee6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fec800c49a0 tx=0x7fec80005be0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 1 -- 172.21.15.31:0/77047612 shutdown_connections 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x557963f565f0 0x557963f55300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557963edafa0 0x557963f50020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557963ed5e30 0x557963ee6820 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec6eff5640 0 client.15273 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 1 --2- 172.21.15.31:0/77047612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557963ed5440 0x557963ed9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 1 -- 172.21.15.31:0/77047612 >> 172.21.15.31:0/77047612 conn(0x557963ed0ab0 msgr2=0x557963ecb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 1 -- 172.21.15.31:0/77047612 shutdown_connections 2024-04-25T01:54:19.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 1 -- 172.21.15.31:0/77047612 wait complete. 2024-04-25T01:54:19.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.758+0000 7fec94168740 20 client.15273 trim_cache size 0 max 16384 2024-04-25T01:54:19.766 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:54:19.766 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:54:19.766 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 3, 2024-04-25T01:54:19.766 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24463, 2024-04-25T01:54:19.766 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T01:54:19.766 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T01:54:19.766 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T01:54:19.766 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 109.02564267, 2024-04-25T01:54:19.767 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T01:54:19.767 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2024-04-25T01:54:19.767 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:54:19.767 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 132.752369437 2024-04-25T01:54:19.767 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:54:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 -- 172.21.15.31:0/1115474646 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fec740a2b90 msgr2=0x7fec740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fec740a2b90 0x7fec740a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fec7c012a20 tx=0x7fec7c00ac70 comp rx=0 tx=0).stop 2024-04-25T01:54:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 -- 172.21.15.31:0/1115474646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec8c102980 msgr2=0x7fec8c197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec8c102980 0x7fec8c197f50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fec800314e0 tx=0x7fec80003c80 comp rx=0 tx=0).stop 2024-04-25T01:54:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 -- 172.21.15.31:0/1115474646 shutdown_connections 2024-04-25T01:54:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fec740a2b90 0x7fec740a5050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec8c103340 0x7fec8c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec8c102980 0x7fec8c197f50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 --2- 172.21.15.31:0/1115474646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec8c101790 0x7fec8c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 -- 172.21.15.31:0/1115474646 >> 172.21.15.31:0/1115474646 conn(0x7fec8c0fcd20 msgr2=0x7fec8c105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 -- 172.21.15.31:0/1115474646 shutdown_connections 2024-04-25T01:54:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.759+0000 7fec91461640 1 -- 172.21.15.31:0/1115474646 wait complete. 2024-04-25T01:54:19.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T01:54:19.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.872+0000 7fdc6b876640 1 -- 172.21.15.31:0/3854104195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc64100680 msgr2=0x7fdc64100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.872+0000 7fdc6b876640 1 --2- 172.21.15.31:0/3854104195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc64100680 0x7fdc64100aa0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdc54009ca0 tx=0x7fdc5402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.872+0000 7fdc6b876640 1 -- 172.21.15.31:0/3854104195 shutdown_connections 2024-04-25T01:54:19.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.872+0000 7fdc6b876640 1 --2- 172.21.15.31:0/3854104195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc64102230 0x7fdc64108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.872+0000 7fdc6b876640 1 --2- 172.21.15.31:0/3854104195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc64101870 0x7fdc64101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.872+0000 7fdc6b876640 1 --2- 172.21.15.31:0/3854104195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc64100680 0x7fdc64100aa0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.872+0000 7fdc6b876640 1 -- 172.21.15.31:0/3854104195 >> 172.21.15.31:0/3854104195 conn(0x7fdc640fbbf0 msgr2=0x7fdc640fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc6b876640 1 -- 172.21.15.31:0/3854104195 shutdown_connections 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc6b876640 1 -- 172.21.15.31:0/3854104195 wait complete. 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc6b876640 1 Processor -- start 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc6b876640 1 -- start start 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc6b876640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc64100680 0x7fdc64196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc6b876640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc64101870 0x7fdc64196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc6b876640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc64102230 0x7fdc6419ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc6b876640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6410c1b0 con 0x7fdc64101870 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc6b876640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6410c030 con 0x7fdc64100680 2024-04-25T01:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc6b876640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6410c330 con 0x7fdc64102230 2024-04-25T01:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc68e11640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc64101870 0x7fdc64196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc68e11640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc64101870 0x7fdc64196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60766/0 (socket says 172.21.15.31:60766) 2024-04-25T01:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.873+0000 7fdc68e11640 1 -- 172.21.15.31:0/1604099247 learned_addr learned my addr 172.21.15.31:0/1604099247 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc69612640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc64100680 0x7fdc64196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc69e13640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc64102230 0x7fdc6419ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc68e11640 1 -- 172.21.15.31:0/1604099247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc64102230 msgr2=0x7fdc6419ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc68e11640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc64102230 0x7fdc6419ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc68e11640 1 -- 172.21.15.31:0/1604099247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc64100680 msgr2=0x7fdc64196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc68e11640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc64100680 0x7fdc64196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc68e11640 1 -- 172.21.15.31:0/1604099247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6419e540 con 0x7fdc64101870 2024-04-25T01:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc68e11640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc64101870 0x7fdc64196e60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800d770 tx=0x7fdc5800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc627fc640 1 -- 172.21.15.31:0/1604099247 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58017070 con 0x7fdc64101870 2024-04-25T01:54:19.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc627fc640 1 -- 172.21.15.31:0/1604099247 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc580054d0 con 0x7fdc64101870 2024-04-25T01:54:19.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc6b876640 1 -- 172.21.15.31:0/1604099247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc64068930 con 0x7fdc64101870 2024-04-25T01:54:19.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc627fc640 1 -- 172.21.15.31:0/1604099247 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58004cf0 con 0x7fdc64101870 2024-04-25T01:54:19.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.874+0000 7fdc6b876640 1 -- 172.21.15.31:0/1604099247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc64068e30 con 0x7fdc64101870 2024-04-25T01:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.876+0000 7fdc627fc640 1 -- 172.21.15.31:0/1604099247 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fdc58004e90 con 0x7fdc64101870 2024-04-25T01:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc6e589740 1 Processor -- start 2024-04-25T01:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc6e589740 1 -- start start 2024-04-25T01:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f2850 0x55ba4f8f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 0x55ba4f8f7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f83b0 0x55ba4f8fa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc6e589740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f851f00 con 0x55ba4f8f83b0 2024-04-25T01:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc6e589740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f7fa750 con 0x55ba4f8f2850 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc6e589740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f84dec0 con 0x55ba4f8f3240 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc627fc640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc4c0a2b90 0x7fdc4c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc68e11640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f83b0 0x55ba4f8fa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc69e13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f2850 0x55ba4f8f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc68e11640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f83b0 0x55ba4f8fa840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60768/0 (socket says 172.21.15.31:60768) 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc69e13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f2850 0x55ba4f8f2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55138/0 (socket says 172.21.15.31:55138) 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc68e11640 1 -- 172.21.15.31:0/3155400284 learned_addr learned my addr 172.21.15.31:0/3155400284 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.877+0000 7fdc69e13640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc4c0a2b90 0x7fdc4c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc69612640 1 --2- 172.21.15.31:0/3155400284 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 0x55ba4f8f7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc627fc640 1 -- 172.21.15.31:0/1604099247 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fdc580ce9d0 con 0x7fdc64101870 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc69e13640 1 -- 172.21.15.31:0/3155400284 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 msgr2=0x55ba4f8f7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc69e13640 1 --2- 172.21.15.31:0/3155400284 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 0x55ba4f8f7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc69e13640 1 -- 172.21.15.31:0/3155400284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f83b0 msgr2=0x55ba4f8fa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc69e13640 1 --2- 172.21.15.31:0/3155400284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f83b0 0x55ba4f8fa840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc69e13640 1 -- 172.21.15.31:0/3155400284 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba4f5662d0 con 0x55ba4f8f2850 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc69612640 1 --2- 172.21.15.31:0/3155400284 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 0x55ba4f8f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc68e11640 1 --2- 172.21.15.31:0/3155400284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f83b0 0x55ba4f8fa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc69e13640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc4c0a2b90 0x7fdc4c0a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0107a0 tx=0x7fdc5c017000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc69e13640 1 --2- 172.21.15.31:0/3155400284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f2850 0x55ba4f8f2c70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0129d0 tx=0x7fdc5c012e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=181b73d9843823c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc4b7fe640 1 -- 172.21.15.31:0/3155400284 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c024030 con 0x55ba4f8f2850 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc4b7fe640 1 -- 172.21.15.31:0/3155400284 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc5c009d70 con 0x55ba4f8f2850 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.878+0000 7fdc4b7fe640 1 -- 172.21.15.31:0/3155400284 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c00fb50 con 0x55ba4f8f2850 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.879+0000 7fdc6e589740 1 -- 172.21.15.31:0/3155400284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f2850 msgr2=0x55ba4f8f2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.879+0000 7fdc6e589740 1 --2- 172.21.15.31:0/3155400284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f2850 0x55ba4f8f2c70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0129d0 tx=0x7fdc5c012e90 comp rx=0 tx=0).stop 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.879+0000 7fdc6e589740 1 -- 172.21.15.31:0/3155400284 shutdown_connections 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.879+0000 7fdc6e589740 1 --2- 172.21.15.31:0/3155400284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f83b0 0x55ba4f8fa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.879+0000 7fdc6e589740 1 --2- 172.21.15.31:0/3155400284 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 0x55ba4f8f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.879+0000 7fdc6e589740 1 --2- 172.21.15.31:0/3155400284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f2850 0x55ba4f8f2c70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.879+0000 7fdc6e589740 1 -- 172.21.15.31:0/3155400284 >> 172.21.15.31:0/3155400284 conn(0x55ba4f8edec0 msgr2=0x55ba4f8ee2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.879+0000 7fdc6e589740 1 -- 172.21.15.31:0/3155400284 shutdown_connections 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.879+0000 7fdc6e589740 1 -- 172.21.15.31:0/3155400284 wait complete. 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.879+0000 7fdc6e589740 1 -- start start 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.879+0000 7fdc6e589740 1 Processor -- start 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f8fcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f3240 0x55ba4f8fd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f83b0 0x55ba4f904190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc6e589740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f8fb300 con 0x55ba4f8f2850 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc6e589740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f80dcf0 con 0x55ba4f8f3240 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc6e589740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f8fb480 con 0x55ba4f8f83b0 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc6e589740 10 client.0 _ll_register_callbacks cb 0x55ba4f80d6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc69612640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f8fcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc69612640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f8fcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60774/0 (socket says 172.21.15.31:60774) 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc69612640 1 -- 172.21.15.31:0/1578204434 learned_addr learned my addr 172.21.15.31:0/1578204434 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc68e11640 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f3240 0x55ba4f8fd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc69e13640 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f83b0 0x55ba4f904190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc69612640 1 -- 172.21.15.31:0/1578204434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f83b0 msgr2=0x55ba4f904190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc69612640 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f83b0 0x55ba4f904190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc69612640 1 -- 172.21.15.31:0/1578204434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f3240 msgr2=0x55ba4f8fd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc69612640 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f3240 0x55ba4f8fd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc69612640 1 -- 172.21.15.31:0/1578204434 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba4f6306b0 con 0x55ba4f8f2850 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc69612640 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f8fcc10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdc641016d0 tx=0x7fdc5402b030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36f0556cdd62775 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc48ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.880+0000 7fdc48ff9640 1 -- 172.21.15.31:0/1578204434 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc54003970 con 0x55ba4f8f2850 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc6e589740 1 -- 172.21.15.31:0/1578204434 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc5c009d70 con 0x55ba4f8f2850 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc48ff9640 1 -- 172.21.15.31:0/1578204434 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fdc54003b10 con 0x55ba4f8f2850 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc48ff9640 1 -- 172.21.15.31:0/1578204434 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdc54003d10 con 0x55ba4f8f2850 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc6e589740 10 client.15285 fetch_fsmap learned FSMap version 26 2024-04-25T01:54:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc6e589740 1 -- 172.21.15.31:0/1578204434 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ba4f9071e0 con 0x55ba4f8f2850 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc48ff9640 1 -- 172.21.15.31:0/1578204434 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc5c009d70 con 0x55ba4f8f2850 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc48ff9640 1 -- 172.21.15.31:0/1578204434 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fdc54030860 con 0x55ba4f8f2850 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc6e589740 10 client.15285 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc6e589740 10 client.15285 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc6e589740 20 client.15285 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc6e589740 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ba4f9097f0 0x55ba4f90ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc6e589740 4 client.15285 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc6e589740 1 -- 172.21.15.31:0/1578204434 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba4f85f440 con 0x55ba4f9097f0 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.881+0000 7fdc68e11640 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ba4f9097f0 0x55ba4f90ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.882+0000 7fdc48ff9640 10 client.15285 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.882+0000 7fdc68e11640 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ba4f9097f0 0x55ba4f90ac60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ddb3f2beab3f5c1d server_cookie=1236f13561506e4d in_seq=0 out_seq=0 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.882+0000 7fdc48ff9640 1 -- 172.21.15.31:0/1578204434 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fdc58080ae0 con 0x55ba4f9097f0 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.882+0000 7fdc48ff9640 10 client.15285 handle_command_reply: tid=0 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 1 client.15285 shutdown 2024-04-25T01:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 1 -- 172.21.15.31:0/1578204434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 msgr2=0x55ba4f8fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f8fcc10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdc641016d0 tx=0x7fdc5402b030 comp rx=0 tx=0).stop 2024-04-25T01:54:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 1 -- 172.21.15.31:0/1578204434 shutdown_connections 2024-04-25T01:54:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ba4f9097f0 0x55ba4f90ac60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f83b0 0x55ba4f904190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f3240 0x55ba4f8fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc48ff9640 0 client.15285 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:54:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 1 --2- 172.21.15.31:0/1578204434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f8fcc10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 1 -- 172.21.15.31:0/1578204434 >> 172.21.15.31:0/1578204434 conn(0x55ba4f8edec0 msgr2=0x55ba4f8e8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 1 -- 172.21.15.31:0/1578204434 shutdown_connections 2024-04-25T01:54:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 1 -- 172.21.15.31:0/1578204434 wait complete. 2024-04-25T01:54:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.883+0000 7fdc6e589740 20 client.15285 trim_cache size 0 max 16384 2024-04-25T01:54:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.901+0000 7fdc6e589740 1 Processor -- start 2024-04-25T01:54:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.901+0000 7fdc6e589740 1 -- start start 2024-04-25T01:54:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.901+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f2850 0x55ba4f8e4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.901+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f3240 0x55ba4f9040b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.901+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f83b0 0x55ba4f8e8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.901+0000 7fdc6e589740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f908fa0 con 0x55ba4f8f3240 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.901+0000 7fdc6e589740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f909670 con 0x55ba4f8f83b0 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.901+0000 7fdc6e589740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f909d40 con 0x55ba4f8f2850 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.901+0000 7fdc68e11640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f3240 0x55ba4f9040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.901+0000 7fdc68e11640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f3240 0x55ba4f9040b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60786/0 (socket says 172.21.15.31:60786) 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.901+0000 7fdc68e11640 1 -- 172.21.15.31:0/2761979295 learned_addr learned my addr 172.21.15.31:0/2761979295 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc69e13640 1 --2- 172.21.15.31:0/2761979295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f83b0 0x55ba4f8e8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc68e11640 1 -- 172.21.15.31:0/2761979295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f2850 msgr2=0x55ba4f8e4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc68e11640 1 --2- 172.21.15.31:0/2761979295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f2850 0x55ba4f8e4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc68e11640 1 -- 172.21.15.31:0/2761979295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f83b0 msgr2=0x55ba4f8e8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc68e11640 1 --2- 172.21.15.31:0/2761979295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f83b0 0x55ba4f8e8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc68e11640 1 -- 172.21.15.31:0/2761979295 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba4f8edcf0 con 0x55ba4f8f3240 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc69e13640 1 --2- 172.21.15.31:0/2761979295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f83b0 0x55ba4f8e8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc68e11640 1 --2- 172.21.15.31:0/2761979295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f3240 0x55ba4f9040b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdc58096ff0 tx=0x7fdc580bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e95ed2ccfbc87b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc4affd640 1 -- 172.21.15.31:0/2761979295 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58020040 con 0x55ba4f8f3240 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc4affd640 1 -- 172.21.15.31:0/2761979295 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ba4f85f440 con 0x55ba4f8f3240 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc4affd640 1 -- 172.21.15.31:0/2761979295 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58099610 con 0x55ba4f8f3240 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc6e589740 1 -- 172.21.15.31:0/2761979295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f3240 msgr2=0x55ba4f9040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.902+0000 7fdc6e589740 1 --2- 172.21.15.31:0/2761979295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f3240 0x55ba4f9040b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdc58096ff0 tx=0x7fdc580bb040 comp rx=0 tx=0).stop 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 -- 172.21.15.31:0/2761979295 shutdown_connections 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 --2- 172.21.15.31:0/2761979295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f83b0 0x55ba4f8e8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 --2- 172.21.15.31:0/2761979295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f3240 0x55ba4f9040b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 --2- 172.21.15.31:0/2761979295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f2850 0x55ba4f8e4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 -- 172.21.15.31:0/2761979295 >> 172.21.15.31:0/2761979295 conn(0x55ba4f8edec0 msgr2=0x55ba4f8ee2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 -- 172.21.15.31:0/2761979295 shutdown_connections 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 -- 172.21.15.31:0/2761979295 wait complete. 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 -- start start 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 Processor -- start 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f9022f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 0x55ba4f902830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f83b0 0x55ba4f902e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f90aae0 con 0x55ba4f8f2850 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f90a410 con 0x55ba4f8f83b0 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ba4f90b1b0 con 0x55ba4f8f3240 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc6e589740 10 client.0 _ll_register_callbacks cb 0x7fdc540317f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc69612640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f9022f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc69612640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f9022f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60788/0 (socket says 172.21.15.31:60788) 2024-04-25T01:54:19.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.903+0000 7fdc69612640 1 -- 172.21.15.31:0/3303758650 learned_addr learned my addr 172.21.15.31:0/3303758650 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc68e11640 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 0x55ba4f902830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc69e13640 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f83b0 0x55ba4f902e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc69612640 1 -- 172.21.15.31:0/3303758650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 msgr2=0x55ba4f902830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc69612640 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 0x55ba4f902830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc69612640 1 -- 172.21.15.31:0/3303758650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f83b0 msgr2=0x55ba4f902e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc69612640 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f83b0 0x55ba4f902e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc69612640 1 -- 172.21.15.31:0/3303758650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba4f903690 con 0x55ba4f8f2850 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc68e11640 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 0x55ba4f902830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc69612640 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f9022f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdc54030df0 tx=0x7fdc54030850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463bdae46e518231 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc48ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc48ff9640 1 -- 172.21.15.31:0/3303758650 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc540518a0 con 0x55ba4f8f2850 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc6e589740 1 -- 172.21.15.31:0/3303758650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ba4f85f440 con 0x55ba4f8f2850 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc48ff9640 1 -- 172.21.15.31:0/3303758650 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fdc54051a40 con 0x55ba4f8f2850 2024-04-25T01:54:19.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc48ff9640 1 -- 172.21.15.31:0/3303758650 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdc54061050 con 0x55ba4f8f2850 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc6e589740 10 client.15291 fetch_fsmap learned FSMap version 26 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc6e589740 1 -- 172.21.15.31:0/3303758650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ba4f96f960 con 0x55ba4f8f2850 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.904+0000 7fdc48ff9640 1 -- 172.21.15.31:0/3303758650 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ba4f85f440 con 0x55ba4f8f2850 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.905+0000 7fdc48ff9640 1 -- 172.21.15.31:0/3303758650 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fdc54051c40 con 0x55ba4f8f2850 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.905+0000 7fdc6e589740 10 client.15291 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.905+0000 7fdc6e589740 10 client.15291 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.905+0000 7fdc6e589740 20 client.15291 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.905+0000 7fdc6e589740 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ba4f974580 0x55ba4f9709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.905+0000 7fdc6e589740 4 client.15291 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.905+0000 7fdc6e589740 1 -- 172.21.15.31:0/3303758650 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "status"}) -- 0x55ba4f859790 con 0x55ba4f974580 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.905+0000 7fdc68e11640 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ba4f974580 0x55ba4f9709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.905+0000 7fdc68e11640 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ba4f974580 0x55ba4f9709c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f0da8a82f410dc0f server_cookie=960016cf67b52be3 in_seq=0 out_seq=0 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.905+0000 7fdc48ff9640 10 client.15291 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc48ff9640 1 -- 172.21.15.31:0/3303758650 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55ba4f859790 con 0x55ba4f974580 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc48ff9640 10 client.15291 handle_command_reply: tid=0 2024-04-25T01:54:19.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 1 client.15291 shutdown 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 1 -- 172.21.15.31:0/3303758650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 msgr2=0x55ba4f9022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f9022f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdc54030df0 tx=0x7fdc54030850 comp rx=0 tx=0).stop 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 1 -- 172.21.15.31:0/3303758650 shutdown_connections 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ba4f974580 0x55ba4f9709c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4f8f83b0 0x55ba4f902e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc48ff9640 0 client.15291 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ba4f8f3240 0x55ba4f902830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 1 --2- 172.21.15.31:0/3303758650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba4f8f2850 0x55ba4f9022f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 1 -- 172.21.15.31:0/3303758650 >> 172.21.15.31:0/3303758650 conn(0x55ba4f8edec0 msgr2=0x55ba4f8e4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 1 -- 172.21.15.31:0/3303758650 shutdown_connections 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 1 -- 172.21.15.31:0/3303758650 wait complete. 2024-04-25T01:54:19.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6e589740 20 client.15291 trim_cache size 0 max 16384 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24473, 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 110.180838307, 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:54:19.915 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 129.901500736 2024-04-25T01:54:19.916 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:54:19.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6b876640 1 -- 172.21.15.31:0/1604099247 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc4c0a2b90 msgr2=0x7fdc4c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6b876640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc4c0a2b90 0x7fdc4c0a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0107a0 tx=0x7fdc5c017000 comp rx=0 tx=0).stop 2024-04-25T01:54:19.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6b876640 1 -- 172.21.15.31:0/1604099247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc64101870 msgr2=0x7fdc64196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:19.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6b876640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc64101870 0x7fdc64196e60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800d770 tx=0x7fdc5800dc30 comp rx=0 tx=0).stop 2024-04-25T01:54:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6b876640 1 -- 172.21.15.31:0/1604099247 shutdown_connections 2024-04-25T01:54:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6b876640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fdc4c0a2b90 0x7fdc4c0a5050 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6b876640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc64102230 0x7fdc6419ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6b876640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc64101870 0x7fdc64196e60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6b876640 1 --2- 172.21.15.31:0/1604099247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc64100680 0x7fdc64196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.906+0000 7fdc6b876640 1 -- 172.21.15.31:0/1604099247 >> 172.21.15.31:0/1604099247 conn(0x7fdc640fbbf0 msgr2=0x7fdc64104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.907+0000 7fdc6b876640 1 -- 172.21.15.31:0/1604099247 shutdown_connections 2024-04-25T01:54:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:19.907+0000 7fdc6b876640 1 -- 172.21.15.31:0/1604099247 wait complete. 2024-04-25T01:54:19.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T01:54:20.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 -- 172.21.15.31:0/799068355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e0102990 msgr2=0x7fb9e0102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 --2- 172.21.15.31:0/799068355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e0102990 0x7fb9e0102e10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009900 tx=0x7fb9c802a410 comp rx=0 tx=0).stop 2024-04-25T01:54:20.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 -- 172.21.15.31:0/799068355 shutdown_connections 2024-04-25T01:54:20.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 --2- 172.21.15.31:0/799068355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9e0103350 0x7fb9e0109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 --2- 172.21.15.31:0/799068355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e0102990 0x7fb9e0102e10 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 --2- 172.21.15.31:0/799068355 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9e01017a0 0x7fb9e0101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 -- 172.21.15.31:0/799068355 >> 172.21.15.31:0/799068355 conn(0x7fb9e00fcd10 msgr2=0x7fb9e00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 -- 172.21.15.31:0/799068355 shutdown_connections 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 -- 172.21.15.31:0/799068355 wait complete. 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 Processor -- start 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 -- start start 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9e01017a0 0x7fb9e01979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9e0102990 0x7fb9e0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e0103350 0x7fb9e019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e010d2b0 con 0x7fb9e0103350 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e010d130 con 0x7fb9e0102990 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.021+0000 7fb9e5789640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e010d430 con 0x7fb9e01017a0 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9df7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e0103350 0x7fb9e019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9df7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e0103350 0x7fb9e019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60804/0 (socket says 172.21.15.31:60804) 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9df7fe640 1 -- 172.21.15.31:0/2319335748 learned_addr learned my addr 172.21.15.31:0/2319335748 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9e0102990 0x7fb9e0197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9deffd640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9e01017a0 0x7fb9e01979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9df7fe640 1 -- 172.21.15.31:0/2319335748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9e01017a0 msgr2=0x7fb9e01979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9df7fe640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9e01017a0 0x7fb9e01979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9df7fe640 1 -- 172.21.15.31:0/2319335748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9e0102990 msgr2=0x7fb9e0197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9df7fe640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9e0102990 0x7fb9e0197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9df7fe640 1 -- 172.21.15.31:0/2319335748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e019f5d0 con 0x7fb9e0103350 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9e0102990 0x7fb9e0197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9df7fe640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e0103350 0x7fb9e019ef60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800eb10 tx=0x7fb9d800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9dcff9640 1 -- 172.21.15.31:0/2319335748 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8004b20 con 0x7fb9e0103350 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9dcff9640 1 -- 172.21.15.31:0/2319335748 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9d8004cc0 con 0x7fb9e0103350 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9e5789640 1 -- 172.21.15.31:0/2319335748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e019f8c0 con 0x7fb9e0103350 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9dcff9640 1 -- 172.21.15.31:0/2319335748 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8010690 con 0x7fb9e0103350 2024-04-25T01:54:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.022+0000 7fb9e5789640 1 -- 172.21.15.31:0/2319335748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e019fe10 con 0x7fb9e0103350 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9dcff9640 1 -- 172.21.15.31:0/2319335748 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb9d8005020 con 0x7fb9e0103350 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9e8575740 1 Processor -- start 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9e8575740 1 -- start start 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fcc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df20fd4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9e8575740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55df20f2ff00 con 0x55df20fcce30 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9e8575740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55df20ed8750 con 0x55df20fcc440 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9e8575740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55df20f2bec0 con 0x55df20fd1fa0 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9deffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9deffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60816/0 (socket says 172.21.15.31:60816) 2024-04-25T01:54:20.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9deffd640 1 -- 172.21.15.31:0/2986951075 learned_addr learned my addr 172.21.15.31:0/2986951075 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9dcff9640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb9cc0a2c10 0x7fb9cc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.025+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/2986951075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fcc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9df7fe640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb9cc0a2c10 0x7fb9cc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/2986951075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df20fd4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9deffd640 1 -- 172.21.15.31:0/2986951075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 msgr2=0x55df20fd4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9deffd640 1 --2- 172.21.15.31:0/2986951075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df20fd4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9deffd640 1 -- 172.21.15.31:0/2986951075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 msgr2=0x55df20fcc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9deffd640 1 --2- 172.21.15.31:0/2986951075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fcc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9dcff9640 1 -- 172.21.15.31:0/2319335748 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fb9d80cf610 con 0x7fb9e0103350 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9deffd640 1 -- 172.21.15.31:0/2986951075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df20c442d0 con 0x55df20fcce30 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/2986951075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fcc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9deffd640 1 --2- 172.21.15.31:0/2986951075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd1930 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000bd10 tx=0x7fb9d000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5207efccc6029300 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9df7fe640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb9cc0a2c10 0x7fb9cc0a50d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb9d809be30 tx=0x7fb9d80c6a30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9d57fa640 1 -- 172.21.15.31:0/2986951075 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0015070 con 0x55df20fcce30 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9d57fa640 1 -- 172.21.15.31:0/2986951075 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9d001a910 con 0x55df20fcce30 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.026+0000 7fb9d57fa640 1 -- 172.21.15.31:0/2986951075 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d00118e0 con 0x55df20fcce30 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 -- 172.21.15.31:0/2986951075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 msgr2=0x55df20fd1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 --2- 172.21.15.31:0/2986951075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd1930 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000bd10 tx=0x7fb9d000f730 comp rx=0 tx=0).stop 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 -- 172.21.15.31:0/2986951075 shutdown_connections 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 --2- 172.21.15.31:0/2986951075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df20fd4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 --2- 172.21.15.31:0/2986951075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd1930 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 --2- 172.21.15.31:0/2986951075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fcc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 -- 172.21.15.31:0/2986951075 >> 172.21.15.31:0/2986951075 conn(0x55df20fc7ab0 msgr2=0x55df20fc7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 -- 172.21.15.31:0/2986951075 shutdown_connections 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 -- 172.21.15.31:0/2986951075 wait complete. 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 -- start start 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 Processor -- start 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fd6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.027+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df20fddde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9e8575740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55df20fd4f50 con 0x55df20fcce30 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9e8575740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55df20eebcf0 con 0x55df20fcc440 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9e8575740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55df20fd50d0 con 0x55df20fd1fa0 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9e8575740 10 client.0 _ll_register_callbacks cb 0x55df20eeb6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9deffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9deffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60824/0 (socket says 172.21.15.31:60824) 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9deffd640 1 -- 172.21.15.31:0/251123737 learned_addr learned my addr 172.21.15.31:0/251123737 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fd6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df20fddde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9deffd640 1 -- 172.21.15.31:0/251123737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 msgr2=0x55df20fddde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9deffd640 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df20fddde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9deffd640 1 -- 172.21.15.31:0/251123737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 msgr2=0x55df20fd6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9deffd640 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fd6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9deffd640 1 -- 172.21.15.31:0/251123737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df20d0e6b0 con 0x55df20fcce30 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9deffd640 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd6da0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0000c00 tx=0x7fb9d001acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12954145bfc94ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9c67fc640 1 -- 172.21.15.31:0/251123737 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0028060 con 0x55df20fcce30 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9c67fc640 1 -- 172.21.15.31:0/251123737 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fb9d001a670 con 0x55df20fcce30 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9e8575740 1 -- 172.21.15.31:0/251123737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9d001a910 con 0x55df20fcce30 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9c67fc640 1 -- 172.21.15.31:0/251123737 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb9d0015070 con 0x55df20fcce30 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9e8575740 10 client.15303 fetch_fsmap learned FSMap version 26 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.028+0000 7fb9e8575740 1 -- 172.21.15.31:0/251123737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55df20fe0e40 con 0x55df20fcce30 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.029+0000 7fb9c67fc640 1 -- 172.21.15.31:0/251123737 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9d001a910 con 0x55df20fcce30 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.029+0000 7fb9c67fc640 1 -- 172.21.15.31:0/251123737 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb9d0004b10 con 0x55df20fcce30 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.029+0000 7fb9e8575740 10 client.15303 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.029+0000 7fb9e8575740 10 client.15303 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.029+0000 7fb9e8575740 20 client.15303 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.029+0000 7fb9e8575740 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55df20fe3170 0x55df20fe45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.029+0000 7fb9e8575740 4 client.15303 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:54:20.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.029+0000 7fb9e8575740 1 -- 172.21.15.31:0/251123737 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df20f3d440 con 0x55df20fe3170 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.029+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55df20fe3170 0x55df20fe45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9c67fc640 10 client.15303 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55df20fe3170 0x55df20fe45e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2fb9ec406fa40066 server_cookie=d553b70f80e3444a in_seq=0 out_seq=0 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9c67fc640 1 -- 172.21.15.31:0/251123737 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df20f3d440 con 0x55df20fe3170 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9c67fc640 10 client.15303 handle_command_reply: tid=0 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9e8575740 1 client.15303 shutdown 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9e8575740 1 -- 172.21.15.31:0/251123737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 msgr2=0x55df20fd6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9e8575740 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd6da0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0000c00 tx=0x7fb9d001acf0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9e8575740 1 -- 172.21.15.31:0/251123737 shutdown_connections 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9e8575740 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55df20fe3170 0x55df20fe45e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9e8575740 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df20fddde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9e8575740 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcce30 0x55df20fd6da0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9c67fc640 0 client.15303 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9e8575740 1 --2- 172.21.15.31:0/251123737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fd6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.030+0000 7fb9e8575740 1 -- 172.21.15.31:0/251123737 >> 172.21.15.31:0/251123737 conn(0x55df20fc7ab0 msgr2=0x55df20fc6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.031+0000 7fb9e8575740 1 -- 172.21.15.31:0/251123737 shutdown_connections 2024-04-25T01:54:20.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.031+0000 7fb9e8575740 1 -- 172.21.15.31:0/251123737 wait complete. 2024-04-25T01:54:20.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.031+0000 7fb9e8575740 20 client.15303 trim_cache size 0 max 16384 2024-04-25T01:54:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.049+0000 7fb9e8575740 1 Processor -- start 2024-04-25T01:54:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.049+0000 7fb9e8575740 1 -- start start 2024-04-25T01:54:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.049+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fd4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.049+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fcce30 0x55df20fddda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.049+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fd1fa0 0x55df20fc2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.049+0000 7fb9e8575740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55df20fe2920 con 0x55df20fd1fa0 2024-04-25T01:54:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.049+0000 7fb9e8575740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55df20fe2ff0 con 0x55df20fcc440 2024-04-25T01:54:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.049+0000 7fb9e8575740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55df20fe36c0 con 0x55df20fcce30 2024-04-25T01:54:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fd1fa0 0x55df20fc2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fd1fa0 0x55df20fc2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60834/0 (socket says 172.21.15.31:60834) 2024-04-25T01:54:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d7fff640 1 -- 172.21.15.31:0/968730872 learned_addr learned my addr 172.21.15.31:0/968730872 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/968730872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fd4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9deffd640 1 --2- 172.21.15.31:0/968730872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fcce30 0x55df20fddda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d7fff640 1 -- 172.21.15.31:0/968730872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fcce30 msgr2=0x55df20fddda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/968730872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fcce30 0x55df20fddda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d7fff640 1 -- 172.21.15.31:0/968730872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 msgr2=0x55df20fd4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/968730872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fd4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d7fff640 1 -- 172.21.15.31:0/968730872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df20fe4910 con 0x55df20fd1fa0 2024-04-25T01:54:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/968730872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fd1fa0 0x55df20fc2790 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8005c40 tx=0x7fb9c8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a0f2f268da6283 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d4ff9640 1 -- 172.21.15.31:0/968730872 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8051070 con 0x55df20fd1fa0 2024-04-25T01:54:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d4ff9640 1 -- 172.21.15.31:0/968730872 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9c80055a0 con 0x55df20fd1fa0 2024-04-25T01:54:20.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.050+0000 7fb9d4ff9640 1 -- 172.21.15.31:0/968730872 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8035040 con 0x55df20fd1fa0 2024-04-25T01:54:20.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.051+0000 7fb9e8575740 1 -- 172.21.15.31:0/968730872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fd1fa0 msgr2=0x55df20fc2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.051+0000 7fb9e8575740 1 --2- 172.21.15.31:0/968730872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fd1fa0 0x55df20fc2790 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8005c40 tx=0x7fb9c8009260 comp rx=0 tx=0).stop 2024-04-25T01:54:20.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.051+0000 7fb9e8575740 1 -- 172.21.15.31:0/968730872 shutdown_connections 2024-04-25T01:54:20.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.051+0000 7fb9e8575740 1 --2- 172.21.15.31:0/968730872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fd1fa0 0x55df20fc2790 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.051+0000 7fb9e8575740 1 --2- 172.21.15.31:0/968730872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fcce30 0x55df20fddda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.051+0000 7fb9e8575740 1 --2- 172.21.15.31:0/968730872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcc440 0x55df20fd4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.051+0000 7fb9e8575740 1 -- 172.21.15.31:0/968730872 >> 172.21.15.31:0/968730872 conn(0x55df20fc7ab0 msgr2=0x55df20fc7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.051+0000 7fb9e8575740 1 -- 172.21.15.31:0/968730872 shutdown_connections 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.051+0000 7fb9e8575740 1 -- 172.21.15.31:0/968730872 wait complete. 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9e8575740 1 -- start start 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9e8575740 1 Processor -- start 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcc440 0x55df20fdbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcce30 0x55df21045100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9e8575740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df21047650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9e8575740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55df20fe4460 con 0x55df20fcc440 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9e8575740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55df20fe3d90 con 0x55df20fcce30 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9e8575740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55df20fe4b30 con 0x55df20fd1fa0 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9e8575740 10 client.0 _ll_register_callbacks cb 0x7fb9d000b840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9d7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcc440 0x55df20fdbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9d7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcc440 0x55df20fdbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60838/0 (socket says 172.21.15.31:60838) 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9d7fff640 1 -- 172.21.15.31:0/4085008190 learned_addr learned my addr 172.21.15.31:0/4085008190 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9deffd640 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcce30 0x55df21045100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.052+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df21047650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9d7fff640 1 -- 172.21.15.31:0/4085008190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 msgr2=0x55df21047650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df21047650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9d7fff640 1 -- 172.21.15.31:0/4085008190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcce30 msgr2=0x55df21045100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcce30 0x55df21045100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9d7fff640 1 -- 172.21.15.31:0/4085008190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df21047b90 con 0x55df20fcc440 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9d7fff640 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcc440 0x55df20fdbd90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80319d0 tx=0x7fb9c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12d601def50ec32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9c67fc640 1 -- 172.21.15.31:0/4085008190 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8035030 con 0x55df20fcc440 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9c67fc640 1 -- 172.21.15.31:0/4085008190 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fb9c804a770 con 0x55df20fcc440 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9e8575740 1 -- 172.21.15.31:0/4085008190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55df20f37790 con 0x55df20fcc440 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9c67fc640 1 -- 172.21.15.31:0/4085008190 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb9c8051070 con 0x55df20fcc440 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9e8575740 10 client.15309 fetch_fsmap learned FSMap version 26 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9e8575740 1 -- 172.21.15.31:0/4085008190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55df21048d60 con 0x55df20fcc440 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9c67fc640 1 -- 172.21.15.31:0/4085008190 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55df20f37790 con 0x55df20fcc440 2024-04-25T01:54:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9c67fc640 1 -- 172.21.15.31:0/4085008190 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb9c8031ce0 con 0x55df20fcc440 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.053+0000 7fb9e8575740 10 client.15309 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.054+0000 7fb9e8575740 10 client.15309 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.054+0000 7fb9e8575740 20 client.15309 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.054+0000 7fb9e8575740 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55df2104d460 0x55df2104c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.054+0000 7fb9e8575740 4 client.15309 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.054+0000 7fb9e8575740 1 -- 172.21.15.31:0/4085008190 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "status"}) -- 0x55df20eb5b80 con 0x55df2104d460 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.054+0000 7fb9deffd640 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55df2104d460 0x55df2104c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.054+0000 7fb9deffd640 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55df2104d460 0x55df2104c170 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1f51ccd8697861dc server_cookie=9af6750a69249f08 in_seq=0 out_seq=0 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.054+0000 7fb9c67fc640 10 client.15309 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.054+0000 7fb9c67fc640 1 -- 172.21.15.31:0/4085008190 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55df20eb5b80 con 0x55df2104d460 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.054+0000 7fb9c67fc640 10 client.15309 handle_command_reply: tid=0 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.054+0000 7fb9e8575740 1 client.15309 shutdown 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e8575740 1 -- 172.21.15.31:0/4085008190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcc440 msgr2=0x55df20fdbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e8575740 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcc440 0x55df20fdbd90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80319d0 tx=0x7fb9c80057f0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e8575740 1 -- 172.21.15.31:0/4085008190 shutdown_connections 2024-04-25T01:54:20.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e8575740 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55df2104d460 0x55df2104c170 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e8575740 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55df20fd1fa0 0x55df21047650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e8575740 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55df20fcce30 0x55df21045100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9c67fc640 0 client.15309 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:54:20.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e8575740 1 --2- 172.21.15.31:0/4085008190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df20fcc440 0x55df20fdbd90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e8575740 1 -- 172.21.15.31:0/4085008190 >> 172.21.15.31:0/4085008190 conn(0x55df20fc7ab0 msgr2=0x55df20fce1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e8575740 1 -- 172.21.15.31:0/4085008190 shutdown_connections 2024-04-25T01:54:20.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e8575740 1 -- 172.21.15.31:0/4085008190 wait complete. 2024-04-25T01:54:20.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e8575740 20 client.15309 trim_cache size 0 max 16384 2024-04-25T01:54:20.062 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24487, 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 111.342502997, 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 127.377132176 2024-04-25T01:54:20.063 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:54:20.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e5789640 1 -- 172.21.15.31:0/2319335748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb9cc0a2c10 msgr2=0x7fb9cc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e5789640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb9cc0a2c10 0x7fb9cc0a50d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb9d809be30 tx=0x7fb9d80c6a30 comp rx=0 tx=0).stop 2024-04-25T01:54:20.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e5789640 1 -- 172.21.15.31:0/2319335748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e0103350 msgr2=0x7fb9e019ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e5789640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e0103350 0x7fb9e019ef60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800eb10 tx=0x7fb9d800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e5789640 1 -- 172.21.15.31:0/2319335748 shutdown_connections 2024-04-25T01:54:20.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e5789640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fb9cc0a2c10 0x7fb9cc0a50d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e5789640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e0103350 0x7fb9e019ef60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e5789640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9e0102990 0x7fb9e0197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e5789640 1 --2- 172.21.15.31:0/2319335748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9e01017a0 0x7fb9e01979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.055+0000 7fb9e5789640 1 -- 172.21.15.31:0/2319335748 >> 172.21.15.31:0/2319335748 conn(0x7fb9e00fcd10 msgr2=0x7fb9e0105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.056+0000 7fb9e5789640 1 -- 172.21.15.31:0/2319335748 shutdown_connections 2024-04-25T01:54:20.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.056+0000 7fb9e5789640 1 -- 172.21.15.31:0/2319335748 wait complete. 2024-04-25T01:54:20.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:54:20.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 -- 172.21.15.31:0/3805433821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef50103350 msgr2=0x7fef50109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 --2- 172.21.15.31:0/3805433821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef50103350 0x7fef50109be0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fef48009ca0 tx=0x7fef4802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 -- 172.21.15.31:0/3805433821 shutdown_connections 2024-04-25T01:54:20.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 --2- 172.21.15.31:0/3805433821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef50103350 0x7fef50109be0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 --2- 172.21.15.31:0/3805433821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef50102990 0x7fef50102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 --2- 172.21.15.31:0/3805433821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef501017a0 0x7fef50101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 -- 172.21.15.31:0/3805433821 >> 172.21.15.31:0/3805433821 conn(0x7fef500fcd10 msgr2=0x7fef500ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 -- 172.21.15.31:0/3805433821 shutdown_connections 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 -- 172.21.15.31:0/3805433821 wait complete. 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 Processor -- start 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 -- start start 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef501017a0 0x7fef50197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.172+0000 7fef58161640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef50102990 0x7fef50197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef58161640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef50103350 0x7fef5019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef58161640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fef5010d2d0 con 0x7fef50102990 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef58161640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef5010d150 con 0x7fef501017a0 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef58161640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fef5010d450 con 0x7fef50103350 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef556fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef50102990 0x7fef50197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef556fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef50102990 0x7fef50197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60868/0 (socket says 172.21.15.31:60868) 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef556fc640 1 -- 172.21.15.31:0/1878939683 learned_addr learned my addr 172.21.15.31:0/1878939683 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef55efd640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef501017a0 0x7fef50197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef566fe640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef50103350 0x7fef5019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef556fc640 1 -- 172.21.15.31:0/1878939683 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef50103350 msgr2=0x7fef5019ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef556fc640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef50103350 0x7fef5019ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef556fc640 1 -- 172.21.15.31:0/1878939683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef501017a0 msgr2=0x7fef50197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef556fc640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef501017a0 0x7fef50197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef556fc640 1 -- 172.21.15.31:0/1878939683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5019f590 con 0x7fef50102990 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef566fe640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef50103350 0x7fef5019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef556fc640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef50102990 0x7fef50197eb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fef4000d950 tx=0x7fef4000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef46ffd640 1 -- 172.21.15.31:0/1878939683 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef40017070 con 0x7fef50102990 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef46ffd640 1 -- 172.21.15.31:0/1878939683 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef40009c30 con 0x7fef50102990 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef58161640 1 -- 172.21.15.31:0/1878939683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5019f880 con 0x7fef50102990 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef46ffd640 1 -- 172.21.15.31:0/1878939683 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4000f620 con 0x7fef50102990 2024-04-25T01:54:20.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.173+0000 7fef58161640 1 -- 172.21.15.31:0/1878939683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef5019fdd0 con 0x7fef50102990 2024-04-25T01:54:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.175+0000 7fef46ffd640 1 -- 172.21.15.31:0/1878939683 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fef400109a0 con 0x7fef50102990 2024-04-25T01:54:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef5ae6d740 1 Processor -- start 2024-04-25T01:54:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef5ae6d740 1 -- start start 2024-04-25T01:54:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62440 0x55a9f5d62860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d62e30 0x55a9f5d67930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d6a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5cc5f00 con 0x55a9f5d62440 2024-04-25T01:54:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5c6e750 con 0x55a9f5d67fa0 2024-04-25T01:54:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5cc1ec0 con 0x55a9f5d62e30 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef566fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62440 0x55a9f5d62860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef566fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62440 0x55a9f5d62860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60874/0 (socket says 172.21.15.31:60874) 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef566fe640 1 -- 172.21.15.31:0/3245828608 learned_addr learned my addr 172.21.15.31:0/3245828608 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef55efd640 1 --2- 172.21.15.31:0/3245828608 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d62e30 0x55a9f5d67930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef46ffd640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef3c0a2b90 0x7fef3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef556fc640 1 --2- 172.21.15.31:0/3245828608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d6a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef566fe640 1 -- 172.21.15.31:0/3245828608 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d62e30 msgr2=0x55a9f5d67930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef566fe640 1 --2- 172.21.15.31:0/3245828608 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d62e30 0x55a9f5d67930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef566fe640 1 -- 172.21.15.31:0/3245828608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d67fa0 msgr2=0x55a9f5d6a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef566fe640 1 --2- 172.21.15.31:0/3245828608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d6a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef566fe640 1 -- 172.21.15.31:0/3245828608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9f59da2d0 con 0x55a9f5d62440 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.176+0000 7fef556fc640 1 --2- 172.21.15.31:0/3245828608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d6a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.177+0000 7fef46ffd640 1 -- 172.21.15.31:0/1878939683 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fef400cecc0 con 0x7fef50102990 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.177+0000 7fef566fe640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef3c0a2b90 0x7fef3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.177+0000 7fef566fe640 1 --2- 172.21.15.31:0/3245828608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62440 0x55a9f5d62860 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fef48009430 tx=0x7fef48005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c713c400abb8de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.177+0000 7fef3bfff640 1 -- 172.21.15.31:0/3245828608 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48031da0 con 0x55a9f5d62440 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.177+0000 7fef3bfff640 1 -- 172.21.15.31:0/3245828608 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef48036040 con 0x55a9f5d62440 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.177+0000 7fef3bfff640 1 -- 172.21.15.31:0/3245828608 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48030830 con 0x55a9f5d62440 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.177+0000 7fef566fe640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef3c0a2b90 0x7fef3c0a5050 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fef4804c000 tx=0x7fef48049000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.178+0000 7fef5ae6d740 1 -- 172.21.15.31:0/3245828608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62440 msgr2=0x55a9f5d62860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.178+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/3245828608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62440 0x55a9f5d62860 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fef48009430 tx=0x7fef48005f70 comp rx=0 tx=0).stop 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.178+0000 7fef5ae6d740 1 -- 172.21.15.31:0/3245828608 shutdown_connections 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.178+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/3245828608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d6a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.178+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/3245828608 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d62e30 0x55a9f5d67930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.178+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/3245828608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62440 0x55a9f5d62860 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.178+0000 7fef5ae6d740 1 -- 172.21.15.31:0/3245828608 >> 172.21.15.31:0/3245828608 conn(0x55a9f5d5dab0 msgr2=0x55a9f5d5deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.178+0000 7fef5ae6d740 1 -- 172.21.15.31:0/3245828608 shutdown_connections 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.178+0000 7fef5ae6d740 1 -- 172.21.15.31:0/3245828608 wait complete. 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.178+0000 7fef5ae6d740 1 -- start start 2024-04-25T01:54:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.178+0000 7fef5ae6d740 1 Processor -- start 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5d6c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5d6cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d73ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5d6add0 con 0x55a9f5d62e30 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5c81cf0 con 0x55a9f5d62440 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5d6af50 con 0x55a9f5d67fa0 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef5ae6d740 10 client.0 _ll_register_callbacks cb 0x55a9f5c816e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef556fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5d6cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef556fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5d6cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60882/0 (socket says 172.21.15.31:60882) 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef556fc640 1 -- 172.21.15.31:0/4293346019 learned_addr learned my addr 172.21.15.31:0/4293346019 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef55efd640 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5d6c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef566fe640 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d73ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef556fc640 1 -- 172.21.15.31:0/4293346019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 msgr2=0x55a9f5d73ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef556fc640 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d73ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef556fc640 1 -- 172.21.15.31:0/4293346019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 msgr2=0x55a9f5d6c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef556fc640 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5d6c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef556fc640 1 -- 172.21.15.31:0/4293346019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9f5aa46b0 con 0x55a9f5d62e30 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef566fe640 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d73ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef556fc640 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5d6cc60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fef40097ec0 tx=0x7fef4009ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2cb4a8315e84da3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef397fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef397fa640 1 -- 172.21.15.31:0/4293346019 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef40020040 con 0x55a9f5d62e30 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef397fa640 1 -- 172.21.15.31:0/4293346019 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fef40097b60 con 0x55a9f5d62e30 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef5ae6d740 1 -- 172.21.15.31:0/4293346019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef48036040 con 0x55a9f5d62e30 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef397fa640 1 -- 172.21.15.31:0/4293346019 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fef4009b4f0 con 0x55a9f5d62e30 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef5ae6d740 10 client.15321 fetch_fsmap learned FSMap version 26 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.179+0000 7fef5ae6d740 1 -- 172.21.15.31:0/4293346019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a9f5d76d10 con 0x55a9f5d62e30 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.180+0000 7fef397fa640 1 -- 172.21.15.31:0/4293346019 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef48036040 con 0x55a9f5d62e30 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.180+0000 7fef397fa640 1 -- 172.21.15.31:0/4293346019 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fef40010460 con 0x55a9f5d62e30 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.180+0000 7fef5ae6d740 10 client.15321 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.180+0000 7fef5ae6d740 10 client.15321 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:54:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.180+0000 7fef5ae6d740 20 client.15321 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.180+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a9f5d79340 0x55a9f5d7a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.180+0000 7fef5ae6d740 4 client.15321 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.180+0000 7fef5ae6d740 1 -- 172.21.15.31:0/4293346019 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9f5cd3440 con 0x55a9f5d79340 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.180+0000 7fef55efd640 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a9f5d79340 0x55a9f5d7a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.181+0000 7fef397fa640 10 client.15321 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.181+0000 7fef55efd640 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a9f5d79340 0x55a9f5d7a7b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8327512d87ea11ba server_cookie=ca2df08773603d8b in_seq=0 out_seq=0 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.181+0000 7fef397fa640 1 -- 172.21.15.31:0/4293346019 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a9f5cd3440 con 0x55a9f5d79340 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.181+0000 7fef397fa640 10 client.15321 handle_command_reply: tid=0 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.181+0000 7fef5ae6d740 1 client.15321 shutdown 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef5ae6d740 1 -- 172.21.15.31:0/4293346019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 msgr2=0x55a9f5d6cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5d6cc60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fef40097ec0 tx=0x7fef4009ae50 comp rx=0 tx=0).stop 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef5ae6d740 1 -- 172.21.15.31:0/4293346019 shutdown_connections 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a9f5d79340 0x55a9f5d7a7b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d73ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5d6cc60 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/4293346019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5d6c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef397fa640 0 client.15321 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef5ae6d740 1 -- 172.21.15.31:0/4293346019 >> 172.21.15.31:0/4293346019 conn(0x55a9f5d5dab0 msgr2=0x55a9f5d5cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef5ae6d740 1 -- 172.21.15.31:0/4293346019 shutdown_connections 2024-04-25T01:54:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef5ae6d740 1 -- 172.21.15.31:0/4293346019 wait complete. 2024-04-25T01:54:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.182+0000 7fef5ae6d740 20 client.15321 trim_cache size 0 max 16384 2024-04-25T01:54:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef5ae6d740 1 Processor -- start 2024-04-25T01:54:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef5ae6d740 1 -- start start 2024-04-25T01:54:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5d6a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d62e30 0x55a9f5d58690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d5c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5d78af0 con 0x55a9f5d67fa0 2024-04-25T01:54:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5d791c0 con 0x55a9f5d62440 2024-04-25T01:54:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5d79890 con 0x55a9f5d62e30 2024-04-25T01:54:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef566fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d5c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef566fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d5c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60886/0 (socket says 172.21.15.31:60886) 2024-04-25T01:54:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef566fe640 1 -- 172.21.15.31:0/2010937352 learned_addr learned my addr 172.21.15.31:0/2010937352 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef566fe640 1 -- 172.21.15.31:0/2010937352 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d62e30 msgr2=0x55a9f5d58690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef566fe640 1 --2- 172.21.15.31:0/2010937352 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d62e30 0x55a9f5d58690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef566fe640 1 -- 172.21.15.31:0/2010937352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 msgr2=0x55a9f5d6a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef566fe640 1 --2- 172.21.15.31:0/2010937352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5d6a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.200+0000 7fef566fe640 1 -- 172.21.15.31:0/2010937352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9f5d76b30 con 0x55a9f5d67fa0 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef566fe640 1 --2- 172.21.15.31:0/2010937352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d5c250 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fef48009a60 tx=0x7fef48031810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efc42beeb145fd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef3b7fe640 1 -- 172.21.15.31:0/2010937352 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48031da0 con 0x55a9f5d67fa0 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef3b7fe640 1 -- 172.21.15.31:0/2010937352 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef480305c0 con 0x55a9f5d67fa0 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef3b7fe640 1 -- 172.21.15.31:0/2010937352 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48004420 con 0x55a9f5d67fa0 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2010937352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d67fa0 msgr2=0x55a9f5d5c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/2010937352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d5c250 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fef48009a60 tx=0x7fef48031810 comp rx=0 tx=0).stop 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2010937352 shutdown_connections 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/2010937352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d67fa0 0x55a9f5d5c250 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/2010937352 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d62e30 0x55a9f5d58690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/2010937352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5d6a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2010937352 >> 172.21.15.31:0/2010937352 conn(0x55a9f5d5dab0 msgr2=0x55a9f5d5de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2010937352 shutdown_connections 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.201+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2010937352 wait complete. 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef5ae6d740 1 -- start start 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef5ae6d740 1 Processor -- start 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5ddd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5ddf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef5ae6d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 0x55a9f5de1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5d7a630 con 0x55a9f5d62e30 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5d79f60 con 0x55a9f5d62440 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef5ae6d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a9f5d7ad00 con 0x55a9f5d67fa0 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef5ae6d740 10 client.0 _ll_register_callbacks cb 0x7fef4009b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef556fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5ddf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef556fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5ddf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60892/0 (socket says 172.21.15.31:60892) 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef556fc640 1 -- 172.21.15.31:0/2563048439 learned_addr learned my addr 172.21.15.31:0/2563048439 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef566fe640 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 0x55a9f5de1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef556fc640 1 -- 172.21.15.31:0/2563048439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 msgr2=0x55a9f5de1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef556fc640 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 0x55a9f5de1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef556fc640 1 -- 172.21.15.31:0/2563048439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 msgr2=0x55a9f5ddd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef55efd640 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5ddd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef556fc640 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5ddd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef556fc640 1 -- 172.21.15.31:0/2563048439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9f5de22c0 con 0x55a9f5d62e30 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef566fe640 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 0x55a9f5de1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.202+0000 7fef55efd640 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5ddd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef556fc640 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5ddf830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fef40010950 tx=0x7fef400cef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c63a769137a744e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef397fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef397fa640 1 -- 172.21.15.31:0/2563048439 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef40020070 con 0x55a9f5d62e30 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef397fa640 1 -- 172.21.15.31:0/2563048439 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7fef40010460 con 0x55a9f5d62e30 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2563048439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef480305c0 con 0x55a9f5d62e30 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef397fa640 1 -- 172.21.15.31:0/2563048439 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fef40080050 con 0x55a9f5d62e30 2024-04-25T01:54:20.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef5ae6d740 10 client.15327 fetch_fsmap learned FSMap version 26 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2563048439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a9f5de22c0 con 0x55a9f5d62e30 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef397fa640 1 -- 172.21.15.31:0/2563048439 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef480305c0 con 0x55a9f5d62e30 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef397fa640 1 -- 172.21.15.31:0/2563048439 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fef40083020 con 0x55a9f5d62e30 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef5ae6d740 10 client.15327 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef5ae6d740 10 client.15327 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef5ae6d740 20 client.15327 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a9f5de7db0 0x55a9f5de6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef5ae6d740 4 client.15327 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2563048439 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "status"}) -- 0x55a9f5ccd790 con 0x55a9f5de7db0 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.203+0000 7fef55efd640 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a9f5de7db0 0x55a9f5de6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef55efd640 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a9f5de7db0 0x55a9f5de6ac0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4016ebaa41626d4 server_cookie=4ae40612d506cde9 in_seq=0 out_seq=0 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef397fa640 10 client.15327 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef397fa640 1 -- 172.21.15.31:0/2563048439 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a9f5ccd790 con 0x55a9f5de7db0 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef397fa640 10 client.15327 handle_command_reply: tid=0 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 1 client.15327 shutdown 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2563048439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 msgr2=0x55a9f5ddf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5ddf830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fef40010950 tx=0x7fef400cef60 comp rx=0 tx=0).stop 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2563048439 shutdown_connections 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a9f5de7db0 0x55a9f5de6ac0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a9f5d67fa0 0x55a9f5de1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9f5d62e30 0x55a9f5ddf830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef397fa640 0 client.15327 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 1 --2- 172.21.15.31:0/2563048439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9f5d62440 0x55a9f5ddd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2563048439 >> 172.21.15.31:0/2563048439 conn(0x55a9f5d5dab0 msgr2=0x55a9f5d58690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2563048439 shutdown_connections 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 1 -- 172.21.15.31:0/2563048439 wait complete. 2024-04-25T01:54:20.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.204+0000 7fef5ae6d740 20 client.15327 trim_cache size 0 max 16384 2024-04-25T01:54:20.217 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:54:20.217 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "36afbea4-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:54:20.217 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-04-25T01:54:20.218 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24503, 2024-04-25T01:54:20.218 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T01:54:20.218 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T01:54:20.218 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T01:54:20.218 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 114.48003132700001, 2024-04-25T01:54:20.218 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T01:54:20.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2024-04-25T01:54:20.218 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:54:20.218 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 118.919436394 2024-04-25T01:54:20.218 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 -- 172.21.15.31:0/1878939683 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef3c0a2b90 msgr2=0x7fef3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef3c0a2b90 0x7fef3c0a5050 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fef4804c000 tx=0x7fef48049000 comp rx=0 tx=0).stop 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 -- 172.21.15.31:0/1878939683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef50102990 msgr2=0x7fef50197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef50102990 0x7fef50197eb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fef4000d950 tx=0x7fef4000de10 comp rx=0 tx=0).stop 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 -- 172.21.15.31:0/1878939683 shutdown_connections 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7fef3c0a2b90 0x7fef3c0a5050 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef50103350 0x7fef5019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef50102990 0x7fef50197eb0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 --2- 172.21.15.31:0/1878939683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef501017a0 0x7fef50197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 -- 172.21.15.31:0/1878939683 >> 172.21.15.31:0/1878939683 conn(0x7fef500fcd10 msgr2=0x7fef50105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 -- 172.21.15.31:0/1878939683 shutdown_connections 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.205+0000 7fef58161640 1 -- 172.21.15.31:0/1878939683 wait complete. 2024-04-25T01:54:20.221 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T01:54:20.230 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T01:54:20.230 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T01:54:20.230 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T01:54:20.230 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-25T01:54:20.231 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:54:20.231 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T01:54:20.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-25T01:54:20.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.332+0000 7f5b66761640 1 -- 172.21.15.31:0/1733459866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b60103350 msgr2=0x7f5b60109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.332+0000 7f5b66761640 1 --2- 172.21.15.31:0/1733459866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b60103350 0x7f5b60109be0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009ca0 tx=0x7f5b5802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.332+0000 7f5b66761640 1 -- 172.21.15.31:0/1733459866 shutdown_connections 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.332+0000 7f5b66761640 1 --2- 172.21.15.31:0/1733459866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b60103350 0x7f5b60109be0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.332+0000 7f5b66761640 1 --2- 172.21.15.31:0/1733459866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b60102990 0x7f5b60102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.332+0000 7f5b66761640 1 --2- 172.21.15.31:0/1733459866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b601017a0 0x7f5b60101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.332+0000 7f5b66761640 1 -- 172.21.15.31:0/1733459866 >> 172.21.15.31:0/1733459866 conn(0x7f5b600fcd30 msgr2=0x7f5b600ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.332+0000 7f5b66761640 1 -- 172.21.15.31:0/1733459866 shutdown_connections 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.332+0000 7f5b66761640 1 -- 172.21.15.31:0/1733459866 wait complete. 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b66761640 1 Processor -- start 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b66761640 1 -- start start 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b66761640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b601017a0 0x7f5b600754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b66761640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b60102990 0x7f5b60073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b66761640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b60103350 0x7f5b60074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b66761640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6010d2d0 con 0x7f5b60102990 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b66761640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6010d150 con 0x7f5b601017a0 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b66761640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6010d450 con 0x7f5b60103350 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b5f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b60102990 0x7f5b60073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b5f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b60102990 0x7f5b60073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60914/0 (socket says 172.21.15.31:60914) 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b5f7fe640 1 -- 172.21.15.31:0/487498750 learned_addr learned my addr 172.21.15.31:0/487498750 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b5ffff640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b601017a0 0x7f5b600754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.333+0000 7f5b64cfe640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b60103350 0x7f5b60074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b5f7fe640 1 -- 172.21.15.31:0/487498750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b60103350 msgr2=0x7f5b60074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b5f7fe640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b60103350 0x7f5b60074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b5f7fe640 1 -- 172.21.15.31:0/487498750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b601017a0 msgr2=0x7f5b600754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b5f7fe640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b601017a0 0x7f5b600754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b5f7fe640 1 -- 172.21.15.31:0/487498750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b60074900 con 0x7f5b60102990 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b64cfe640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b60103350 0x7f5b60074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b5f7fe640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b60102990 0x7f5b60073b30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400d950 tx=0x7f5b5400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b5d7fa640 1 -- 172.21.15.31:0/487498750 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b54017070 con 0x7f5b60102990 2024-04-25T01:54:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b5d7fa640 1 -- 172.21.15.31:0/487498750 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b54009c30 con 0x7f5b60102990 2024-04-25T01:54:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b66761640 1 -- 172.21.15.31:0/487498750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b601a19b0 con 0x7f5b60102990 2024-04-25T01:54:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b5d7fa640 1 -- 172.21.15.31:0/487498750 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5400f620 con 0x7f5b60102990 2024-04-25T01:54:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.334+0000 7f5b66761640 1 -- 172.21.15.31:0/487498750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b601a1f00 con 0x7f5b60102990 2024-04-25T01:54:20.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.335+0000 7f5b66761640 1 -- 172.21.15.31:0/487498750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b60108a60 con 0x7f5b60102990 2024-04-25T01:54:20.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.336+0000 7f5b5d7fa640 1 -- 172.21.15.31:0/487498750 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5b540109a0 con 0x7f5b60102990 2024-04-25T01:54:20.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.337+0000 7f5b5d7fa640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b480a2c30 0x7f5b480a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.337+0000 7f5b5ffff640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b480a2c30 0x7f5b480a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.337+0000 7f5b5d7fa640 1 -- 172.21.15.31:0/487498750 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f5b540cecb0 con 0x7f5b60102990 2024-04-25T01:54:20.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.337+0000 7f5b5ffff640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b480a2c30 0x7f5b480a50f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5b601027f0 tx=0x7f5b50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.342+0000 7f5b5d7fa640 1 -- 172.21.15.31:0/487498750 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5b54009de0 con 0x7f5b60102990 2024-04-25T01:54:20.409 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:20 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2115868831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:20.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:20 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2115868831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:20.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:20 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2115868831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:20.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:20.581+0000 7f5b66761640 1 -- 172.21.15.31:0/487498750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f5b60062340 con 0x7f5b60102990 2024-04-25T01:54:21.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.165+0000 7f5b5d7fa640 1 -- 172.21.15.31:0/487498750 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f5b5409bbe0 con 0x7f5b60102990 2024-04-25T01:54:21.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 -- 172.21.15.31:0/487498750 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b480a2c30 msgr2=0x7f5b480a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:21.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b480a2c30 0x7f5b480a50f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5b601027f0 tx=0x7f5b50005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:21.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 -- 172.21.15.31:0/487498750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b60102990 msgr2=0x7f5b60073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:21.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b60102990 0x7f5b60073b30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400d950 tx=0x7f5b5400de10 comp rx=0 tx=0).stop 2024-04-25T01:54:21.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 -- 172.21.15.31:0/487498750 shutdown_connections 2024-04-25T01:54:21.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5b480a2c30 0x7f5b480a50f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b60103350 0x7f5b60074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b60102990 0x7f5b60073b30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 --2- 172.21.15.31:0/487498750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b601017a0 0x7f5b600754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 -- 172.21.15.31:0/487498750 >> 172.21.15.31:0/487498750 conn(0x7f5b600fcd30 msgr2=0x7f5b60105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:21.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 -- 172.21.15.31:0/487498750 shutdown_connections 2024-04-25T01:54:21.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.168+0000 7f5b66761640 1 -- 172.21.15.31:0/487498750 wait complete. 2024-04-25T01:54:21.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-25T01:54:21.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.308+0000 7f466d11d640 1 -- 172.21.15.31:0/2476703608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4668102250 msgr2=0x7f4668108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:21.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.308+0000 7f466d11d640 1 --2- 172.21.15.31:0/2476703608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4668102250 0x7f4668108ae0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4658009ca0 tx=0x7f465802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.308+0000 7f466d11d640 1 -- 172.21.15.31:0/2476703608 shutdown_connections 2024-04-25T01:54:21.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.308+0000 7f466d11d640 1 --2- 172.21.15.31:0/2476703608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4668102250 0x7f4668108ae0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.308+0000 7f466d11d640 1 --2- 172.21.15.31:0/2476703608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4668101890 0x7f4668101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.308+0000 7f466d11d640 1 --2- 172.21.15.31:0/2476703608 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46681006a0 0x7f4668100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.308+0000 7f466d11d640 1 -- 172.21.15.31:0/2476703608 >> 172.21.15.31:0/2476703608 conn(0x7f46680fbc10 msgr2=0x7f46680fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:21.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.308+0000 7f466d11d640 1 -- 172.21.15.31:0/2476703608 shutdown_connections 2024-04-25T01:54:21.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.308+0000 7f466d11d640 1 -- 172.21.15.31:0/2476703608 wait complete. 2024-04-25T01:54:21.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.309+0000 7f466d11d640 1 Processor -- start 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.309+0000 7f466d11d640 1 -- start start 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.309+0000 7f466d11d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46681006a0 0x7f46681978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.309+0000 7f466d11d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4668101890 0x7f4668197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.309+0000 7f466d11d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4668102250 0x7f466819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.309+0000 7f466d11d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f466810c200 con 0x7f4668102250 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.309+0000 7f466d11d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f466810c080 con 0x7f46681006a0 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.309+0000 7f466d11d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f466810c380 con 0x7f4668101890 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f4666d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46681006a0 0x7f46681978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f4666d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46681006a0 0x7f46681978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55344/0 (socket says 172.21.15.31:55344) 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f466759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4668102250 0x7f466819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f4666d9d640 1 -- 172.21.15.31:0/2990978147 learned_addr learned my addr 172.21.15.31:0/2990978147 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f466659c640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4668101890 0x7f4668197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:21.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f4666d9d640 1 -- 172.21.15.31:0/2990978147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4668101890 msgr2=0x7f4668197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f4666d9d640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4668101890 0x7f4668197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f4666d9d640 1 -- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4668102250 msgr2=0x7f466819ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f4666d9d640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4668102250 0x7f466819ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f4666d9d640 1 -- 172.21.15.31:0/2990978147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466819f4d0 con 0x7f46681006a0 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f466759e640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4668102250 0x7f466819ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f4666d9d640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46681006a0 0x7f46681978b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f466000d950 tx=0x7f466000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f464ffff640 1 -- 172.21.15.31:0/2990978147 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4660017020 con 0x7f46681006a0 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f466d11d640 1 -- 172.21.15.31:0/2990978147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f466819f7c0 con 0x7f46681006a0 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f464ffff640 1 -- 172.21.15.31:0/2990978147 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4660009c30 con 0x7f46681006a0 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f464ffff640 1 -- 172.21.15.31:0/2990978147 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466000f620 con 0x7f46681006a0 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.310+0000 7f466d11d640 1 -- 172.21.15.31:0/2990978147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f466819fd60 con 0x7f46681006a0 2024-04-25T01:54:21.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.311+0000 7f466d11d640 1 -- 172.21.15.31:0/2990978147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f466819ffc0 con 0x7f46681006a0 2024-04-25T01:54:21.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.313+0000 7f464ffff640 1 -- 172.21.15.31:0/2990978147 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f466001a020 con 0x7f46681006a0 2024-04-25T01:54:21.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.313+0000 7f464ffff640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f46500a2c80 0x7f46500a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:21.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.313+0000 7f466659c640 1 -- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f46500a2c80 msgr2=0x7f46500a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2306032998 2024-04-25T01:54:21.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.313+0000 7f464ffff640 1 -- 172.21.15.31:0/2990978147 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f46600cf330 con 0x7f46681006a0 2024-04-25T01:54:21.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.313+0000 7f466659c640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f46500a2c80 0x7f46500a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:54:21.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.317+0000 7f464ffff640 1 -- 172.21.15.31:0/2990978147 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4660021020 con 0x7f46681006a0 2024-04-25T01:54:21.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:21 smithi134 ceph-mon[33098]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:54:21.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:21 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/487498750' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:54:21.440 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: ignoring --setuser ceph since I am not root 2024-04-25T01:54:21.440 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: ignoring --setgroup ceph since I am not root 2024-04-25T01:54:21.440 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:21 smithi134 ceph-mgr[34959]: -- 172.21.15.134:0/2035562503 <== mon.1 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5559360d85a0 con 0x5559360ac800 2024-04-25T01:54:21.441 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:21.348+0000 7f983b001200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:54:21.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:21 smithi031 ceph-mon[29204]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:54:21.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:21 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/487498750' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:54:21.488 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:21 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: ignoring --setuser ceph since I am not root 2024-04-25T01:54:21.489 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:21 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: ignoring --setgroup ceph since I am not root 2024-04-25T01:54:21.489 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:21 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:21.368+0000 7f7df9d72200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:54:21.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.507+0000 7f466d11d640 1 -- 172.21.15.31:0/2990978147 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f46681a0190 con 0x7f46681006a0 2024-04-25T01:54:21.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.514+0000 7f466659c640 1 -- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f46500a2c80 msgr2=0x7f46500a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2306032998 2024-04-25T01:54:21.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.514+0000 7f466659c640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f46500a2c80 0x7f46500a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:54:21.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.514+0000 7f464ffff640 1 -- 172.21.15.31:0/2990978147 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7f466009c2c0 con 0x7f46681006a0 2024-04-25T01:54:21.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 -- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f46500a2c80 msgr2=0x7f46500a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:54:21.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f46500a2c80 0x7f46500a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 -- 172.21.15.31:0/2990978147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46681006a0 msgr2=0x7f46681978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:21.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46681006a0 0x7f46681978b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f466000d950 tx=0x7f466000de10 comp rx=0 tx=0).stop 2024-04-25T01:54:21.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 -- 172.21.15.31:0/2990978147 shutdown_connections 2024-04-25T01:54:21.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f46500a2c80 0x7f46500a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4668102250 0x7f466819ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4668101890 0x7f4668197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 --2- 172.21.15.31:0/2990978147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46681006a0 0x7f46681978b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 -- 172.21.15.31:0/2990978147 >> 172.21.15.31:0/2990978147 conn(0x7f46680fbc10 msgr2=0x7f46680fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 -- 172.21.15.31:0/2990978147 shutdown_connections 2024-04-25T01:54:21.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.517+0000 7f466d11d640 1 -- 172.21.15.31:0/2990978147 wait complete. 2024-04-25T01:54:21.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-25T01:54:21.541 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:21 smithi040 ceph-mon[32418]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:54:21.541 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:21 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/487498750' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:54:21.541 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: ignoring --setuser ceph since I am not root 2024-04-25T01:54:21.541 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: ignoring --setgroup ceph since I am not root 2024-04-25T01:54:21.541 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:21 smithi040 ceph-mgr[33800]: -- 172.21.15.40:0/30495947 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557a69c6e5a0 con 0x557a69c43000 2024-04-25T01:54:21.541 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:21.348+0000 7fe95aabb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:54:21.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.645+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/3086066990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a181029b0 msgr2=0x7f4a18102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.645+0000 7f4a1dcfc640 1 --2- 172.21.15.31:0/3086066990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a181029b0 0x7f4a18102e30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009900 tx=0x7f4a0c02a410 comp rx=0 tx=0).stop 2024-04-25T01:54:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.645+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/3086066990 shutdown_connections 2024-04-25T01:54:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.645+0000 7f4a1dcfc640 1 --2- 172.21.15.31:0/3086066990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a18103370 0x7f4a18109c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.645+0000 7f4a1dcfc640 1 --2- 172.21.15.31:0/3086066990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a181029b0 0x7f4a18102e30 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.645+0000 7f4a1dcfc640 1 --2- 172.21.15.31:0/3086066990 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a181017c0 0x7f4a18101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.645+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/3086066990 >> 172.21.15.31:0/3086066990 conn(0x7f4a180fcd30 msgr2=0x7f4a180ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.645+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/3086066990 shutdown_connections 2024-04-25T01:54:21.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.645+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/3086066990 wait complete. 2024-04-25T01:54:21.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a1dcfc640 1 Processor -- start 2024-04-25T01:54:21.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a1dcfc640 1 -- start start 2024-04-25T01:54:21.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a1dcfc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a181017c0 0x7f4a180754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:21.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a1dcfc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a181029b0 0x7f4a18073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:21.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a1dcfc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a18103370 0x7f4a18074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:21.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a1dcfc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1810d340 con 0x7f4a181029b0 2024-04-25T01:54:21.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a1dcfc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1810d1c0 con 0x7f4a181017c0 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a1dcfc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1810d4c0 con 0x7f4a18103370 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a16ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a181029b0 0x7f4a18073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a16ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a181029b0 0x7f4a18073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60960/0 (socket says 172.21.15.31:60960) 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a16ffd640 1 -- 172.21.15.31:0/2158584269 learned_addr learned my addr 172.21.15.31:0/2158584269 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a177fe640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a181017c0 0x7f4a180754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a17fff640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a18103370 0x7f4a18074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a16ffd640 1 -- 172.21.15.31:0/2158584269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a18103370 msgr2=0x7f4a18074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a16ffd640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a18103370 0x7f4a18074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a16ffd640 1 -- 172.21.15.31:0/2158584269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a181017c0 msgr2=0x7f4a180754a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a16ffd640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a181017c0 0x7f4a180754a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.646+0000 7f4a16ffd640 1 -- 172.21.15.31:0/2158584269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a18074930 con 0x7f4a181029b0 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.647+0000 7f4a16ffd640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a181029b0 0x7f4a18073b30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c0314e0 tx=0x7f4a0c003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.647+0000 7f4a14ff9640 1 -- 172.21.15.31:0/2158584269 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c03f070 con 0x7f4a181029b0 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.647+0000 7f4a14ff9640 1 -- 172.21.15.31:0/2158584269 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a0c031e40 con 0x7f4a181029b0 2024-04-25T01:54:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.647+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/2158584269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a181a1ac0 con 0x7f4a181029b0 2024-04-25T01:54:21.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.647+0000 7f4a14ff9640 1 -- 172.21.15.31:0/2158584269 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c035030 con 0x7f4a181029b0 2024-04-25T01:54:21.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.647+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/2158584269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a181a1f80 con 0x7f4a181029b0 2024-04-25T01:54:21.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.648+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/2158584269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a18108bb0 con 0x7f4a181029b0 2024-04-25T01:54:21.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.649+0000 7f4a14ff9640 1 -- 172.21.15.31:0/2158584269 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f4a0c042070 con 0x7f4a181029b0 2024-04-25T01:54:21.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.650+0000 7f4a14ff9640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4a000a2c80 0x7f4a000a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:21.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.650+0000 7f4a177fe640 1 -- 172.21.15.31:0/2158584269 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4a000a2c80 msgr2=0x7f4a000a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2306032998 2024-04-25T01:54:21.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.650+0000 7f4a14ff9640 1 -- 172.21.15.31:0/2158584269 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f4a0c0ef910 con 0x7f4a181029b0 2024-04-25T01:54:21.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.650+0000 7f4a177fe640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4a000a2c80 0x7f4a000a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:54:21.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.655+0000 7f4a14ff9640 1 -- 172.21.15.31:0/2158584269 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a0c0bc250 con 0x7f4a181029b0 2024-04-25T01:54:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.840+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/2158584269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f4a180623b0 con 0x7f4a181029b0 2024-04-25T01:54:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.844+0000 7f4a14ff9640 1 -- 172.21.15.31:0/2158584269 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) ==== 133+0+0 (secure 0 0 0) 0x7f4a0c0c0550 con 0x7f4a181029b0 2024-04-25T01:54:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.846+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/2158584269 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4a000a2c80 msgr2=0x7f4a000a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:54:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.846+0000 7f4a1dcfc640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4a000a2c80 0x7f4a000a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.846+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/2158584269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a181029b0 msgr2=0x7f4a18073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.846+0000 7f4a1dcfc640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a181029b0 0x7f4a18073b30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c0314e0 tx=0x7f4a0c003c80 comp rx=0 tx=0).stop 2024-04-25T01:54:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.846+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/2158584269 shutdown_connections 2024-04-25T01:54:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.846+0000 7f4a1dcfc640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f4a000a2c80 0x7f4a000a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.846+0000 7f4a1dcfc640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a18103370 0x7f4a18074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.846+0000 7f4a1dcfc640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a181029b0 0x7f4a18073b30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.846+0000 7f4a1dcfc640 1 --2- 172.21.15.31:0/2158584269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a181017c0 0x7f4a180754a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.846+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/2158584269 >> 172.21.15.31:0/2158584269 conn(0x7f4a180fcd30 msgr2=0x7f4a18105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.847+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/2158584269 shutdown_connections 2024-04-25T01:54:21.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:21.847+0000 7f4a1dcfc640 1 -- 172.21.15.31:0/2158584269 wait complete. 2024-04-25T01:54:21.858 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:21 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:21.563+0000 7f7df9d72200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:54:21.858 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:21 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:21.674+0000 7f7df9d72200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:54:21.859 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-25T01:54:21.895 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-25T01:54:21.895 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-04-25T01:54:21.895 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-04-25T01:54:21.896 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:54:21.896 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:54:21.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:21.896 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-04-25T01:54:21.909 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:21.539+0000 7fe95aabb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:54:21.910 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:21.645+0000 7fe95aabb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:54:21.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:21.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-04-25T01:54:21.939 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:21.534+0000 7f983b001200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:54:21.939 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:21.636+0000 7f983b001200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:54:21.981 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2024-04-25T01:54:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:54:21.983 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-25T01:54:21.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T01:54:22.167 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:22 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:22.003+0000 7f7df9d72200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:54:22.167 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:22 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:22.093+0000 7f7df9d72200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:54:22.282 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:22.409 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:21.965+0000 7fe95aabb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:54:22.410 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:22 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:22.053+0000 7fe95aabb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:54:22.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:22 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/487498750' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:54:22.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:22 smithi040 ceph-mon[32418]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-25T01:54:22.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:22 smithi040 ceph-mon[32418]: from='client.? ' entity='client.admin' 2024-04-25T01:54:22.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:22 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2158584269' entity='client.admin' 2024-04-25T01:54:22.439 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:21.945+0000 7f983b001200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:54:22.439 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:22 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:22.031+0000 7f983b001200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:54:22.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:22 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/487498750' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:54:22.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:22 smithi134 ceph-mon[33098]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-25T01:54:22.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:22 smithi134 ceph-mon[33098]: from='client.? ' entity='client.admin' 2024-04-25T01:54:22.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:22 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2158584269' entity='client.admin' 2024-04-25T01:54:22.442 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:22 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/487498750' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:54:22.442 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:22 smithi031 ceph-mon[29204]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-25T01:54:22.442 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:22 smithi031 ceph-mon[29204]: from='client.? ' entity='client.admin' 2024-04-25T01:54:22.442 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:22 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2158584269' entity='client.admin' 2024-04-25T01:54:23.553 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:23 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:23.211+0000 7f983b001200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:54:23.553 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:23 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:23.380+0000 7f983b001200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:54:23.553 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:23 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:23.551+0000 7f983b001200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:54:23.639 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:23 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:23.287+0000 7fe95aabb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:54:23.639 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:23 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:23.463+0000 7fe95aabb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:54:23.639 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:23 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:23.638+0000 7fe95aabb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:54:23.712 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:23 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:23.375+0000 7f7df9d72200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:54:23.713 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:23 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:23.565+0000 7f7df9d72200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:54:23.964 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:23 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:23.757+0000 7f7df9d72200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:54:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.064+0000 7f5bc39e3700 1 -- 172.21.15.31:0/413713002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bbc1024e0 msgr2=0x7f5bbc1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.064+0000 7f5bc39e3700 1 --2- 172.21.15.31:0/413713002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bbc1024e0 0x7f5bbc1028c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009990 tx=0x7f5bb0009ca0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.064+0000 7f5bc39e3700 1 -- 172.21.15.31:0/413713002 shutdown_connections 2024-04-25T01:54:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.064+0000 7f5bc39e3700 1 --2- 172.21.15.31:0/413713002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc1075a0 0x7f5bbc10f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.064+0000 7f5bc39e3700 1 --2- 172.21.15.31:0/413713002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bbc1024e0 0x7f5bbc1028c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.064+0000 7f5bc39e3700 1 --2- 172.21.15.31:0/413713002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc102e90 0x7f5bbc106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.064+0000 7f5bc39e3700 1 -- 172.21.15.31:0/413713002 >> 172.21.15.31:0/413713002 conn(0x7f5bbc0793e0 msgr2=0x7f5bbc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.064+0000 7f5bc39e3700 1 -- 172.21.15.31:0/413713002 shutdown_connections 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.065+0000 7f5bc39e3700 1 -- 172.21.15.31:0/413713002 wait complete. 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.065+0000 7f5bc39e3700 1 Processor -- start 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.065+0000 7f5bc39e3700 1 -- start start 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.065+0000 7f5bc39e3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bbc1024e0 0x7f5bbc19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.065+0000 7f5bc39e3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc102e90 0x7f5bbc19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.065+0000 7f5bc39e3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc1075a0 0x7f5bbc198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.065+0000 7f5bc39e3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5bbc112430 con 0x7f5bbc1024e0 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.065+0000 7f5bc39e3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5bbc1122b0 con 0x7f5bbc102e90 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.065+0000 7f5bc39e3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5bbc1125b0 con 0x7f5bbc1075a0 2024-04-25T01:54:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc177f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bbc1024e0 0x7f5bbc19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc177f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bbc1024e0 0x7f5bbc19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34980/0 (socket says 172.21.15.31:34980) 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc177f700 1 -- 172.21.15.31:0/1059136902 learned_addr learned my addr 172.21.15.31:0/1059136902 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc0f7e700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc102e90 0x7f5bbc19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc1f80700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc1075a0 0x7f5bbc198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc177f700 1 -- 172.21.15.31:0/1059136902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc1075a0 msgr2=0x7f5bbc198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc177f700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc1075a0 0x7f5bbc198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc177f700 1 -- 172.21.15.31:0/1059136902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc102e90 msgr2=0x7f5bbc19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc177f700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc102e90 0x7f5bbc19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc177f700 1 -- 172.21.15.31:0/1059136902 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bbc199010 con 0x7f5bbc1024e0 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc0f7e700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc102e90 0x7f5bbc19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc177f700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bbc1024e0 0x7f5bbc19e660 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0000c00 tx=0x7f5bb0004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bae7fc700 1 -- 172.21.15.31:0/1059136902 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0027070 con 0x7f5bbc1024e0 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bae7fc700 1 -- 172.21.15.31:0/1059136902 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bb0022470 con 0x7f5bbc1024e0 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bae7fc700 1 -- 172.21.15.31:0/1059136902 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb000b610 con 0x7f5bbc1024e0 2024-04-25T01:54:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc39e3700 1 -- 172.21.15.31:0/1059136902 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bbc1992a0 con 0x7f5bbc1024e0 2024-04-25T01:54:24.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.066+0000 7f5bc39e3700 1 -- 172.21.15.31:0/1059136902 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bbc1a5410 con 0x7f5bbc1024e0 2024-04-25T01:54:24.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.067+0000 7f5bc39e3700 1 -- 172.21.15.31:0/1059136902 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bbc10a7c0 con 0x7f5bbc1024e0 2024-04-25T01:54:24.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.069+0000 7f5bae7fc700 1 -- 172.21.15.31:0/1059136902 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f5bb002a070 con 0x7f5bbc1024e0 2024-04-25T01:54:24.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.069+0000 7f5bae7fc700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5ba80a2c70 0x7f5ba80a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:24.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.069+0000 7f5bc0f7e700 1 -- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5ba80a2c70 msgr2=0x7f5ba80a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2306032998 2024-04-25T01:54:24.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.069+0000 7f5bae7fc700 1 -- 172.21.15.31:0/1059136902 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f5bb00cff40 con 0x7f5bbc1024e0 2024-04-25T01:54:24.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.069+0000 7f5bc0f7e700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5ba80a2c70 0x7f5ba80a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:54:24.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.077+0000 7f5bae7fc700 1 -- 172.21.15.31:0/1059136902 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5bb0013070 con 0x7f5bbc1024e0 2024-04-25T01:54:24.140 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:23 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:23.841+0000 7f983b001200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:54:24.140 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:23 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:23.935+0000 7f983b001200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:54:24.140 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:24 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:24.138+0000 7f983b001200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:54:24.238 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:24 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:24.074+0000 7f7df9d72200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:54:24.238 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:24 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:24.174+0000 7f7df9d72200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:54:24.242 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:23 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:23.936+0000 7fe95aabb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:54:24.242 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:24.032+0000 7fe95aabb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:54:24.242 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:24.241+0000 7fe95aabb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:54:24.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.270+0000 7f5bc0f7e700 1 -- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5ba80a2c70 msgr2=0x7f5ba80a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2306032998 2024-04-25T01:54:24.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.270+0000 7f5bc0f7e700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5ba80a2c70 0x7f5ba80a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:54:24.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.331+0000 7f5bc39e3700 1 -- 172.21.15.31:0/1059136902 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5bbc1039a0 con 0x7f5bbc1024e0 2024-04-25T01:54:24.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.332+0000 7f5bae7fc700 1 -- 172.21.15.31:0/1059136902 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+73 (secure 0 0 0) 0x7f5bb009cf60 con 0x7f5bbc1024e0 2024-04-25T01:54:24.333 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T01:54:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 -- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5ba80a2c70 msgr2=0x7f5ba80a5130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:54:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5ba80a2c70 0x7f5ba80a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 -- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bbc1024e0 msgr2=0x7f5bbc19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bbc1024e0 0x7f5bbc19e660 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0000c00 tx=0x7f5bb0004790 comp rx=0 tx=0).stop 2024-04-25T01:54:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 -- 172.21.15.31:0/1059136902 shutdown_connections 2024-04-25T01:54:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f5ba80a2c70 0x7f5ba80a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bbc1075a0 0x7f5bbc198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bbc1024e0 0x7f5bbc19e660 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 --2- 172.21.15.31:0/1059136902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc102e90 0x7f5bbc19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 -- 172.21.15.31:0/1059136902 >> 172.21.15.31:0/1059136902 conn(0x7f5bbc0793e0 msgr2=0x7f5bbc108720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:24.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 -- 172.21.15.31:0/1059136902 shutdown_connections 2024-04-25T01:54:24.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:24.334+0000 7f5bc39e3700 1 -- 172.21.15.31:0/1059136902 wait complete. 2024-04-25T01:54:24.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:24 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1059136902' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:54:24.439 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:24 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:24.331+0000 7f983b001200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:54:24.606 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:24 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1059136902' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:54:24.606 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:24 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:24.396+0000 7f7df9d72200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:54:24.659 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:24 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1059136902' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:54:24.660 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:24.439+0000 7fe95aabb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:54:24.988 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:24 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:24.605+0000 7f7df9d72200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:54:25.160 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:54:25.160 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:54:25.160 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-04-25T01:54:25.160 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:54:25.160 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:54:25.160 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:54:25.160 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:54:25.160 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:54:25.160 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:54:25.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:25.161 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-04-25T01:54:25.178 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-04-25T01:54:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T01:54:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42106sec preferred_lft 42106sec 2024-04-25T01:54:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link noprefixroute 2024-04-25T01:54:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:54:25.180 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:54:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-04-25T01:54:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-04-25T01:54:25.180 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:54:25.180 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:54:25.181 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-25T01:54:25.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:54:25.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr flush dev ceph-brx 2024-04-25T01:54:25.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set ceph-brx up 2024-04-25T01:54:25.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:54:25.181 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-25T01:54:25.189 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:24 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:24.936+0000 7f983b001200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:54:25.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:54:25.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:54:25.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:25.376 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:54:25.387 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:25 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:25.256+0000 7f7df9d72200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:54:25.402 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-04-25T01:54:25.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:25.404 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-04-25T01:54:25.409 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:25 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:25.063+0000 7fe95aabb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:54:25.469 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-04-25T01:54:25.469 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:54:25.469 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T01:54:25.469 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T01:54:25.469 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:54:25.470 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:54:25.470 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-25T01:54:25.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T01:54:25.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T01:54:25.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T01:54:25.470 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-25T01:54:25.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:54:25.658 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:25 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:25.357+0000 7f983b001200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:54:25.658 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:25 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:25.453+0000 7f983b001200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:54:25.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:54:25.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:25.795 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-04-25T01:54:25.807 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:25 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:25.497+0000 7fe95aabb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:54:25.808 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:25 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:25.599+0000 7fe95aabb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:54:25.808 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:25 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:25.807+0000 7fe95aabb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:54:25.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:25.811 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list-id 2024-04-25T01:54:25.866 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:54:25.866 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-25T01:54:25.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:54:25.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T01:54:25.867 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-25T01:54:25.939 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:25 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:25.656+0000 7f983b001200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:54:25.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:54:25.966 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:25 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:25.733+0000 7f7df9d72200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:54:25.966 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:25 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:25.842+0000 7f7df9d72200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:54:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:54:25.987 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T01:54:25.988 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:54:25.988 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-25T01:54:25.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T01:54:25.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:54:25.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T01:54:25.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T01:54:25.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T01:54:25.988 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-25T01:54:26.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:54:26.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:54:26.116 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:54:26.116 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-25T01:54:26.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-04-25T01:54:26.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:54:26.117 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-25T01:54:26.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:54:26.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:54:26.195 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-04-25T01:54:26.195 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:54:26.195 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T01:54:26.210 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:54:26.210 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:54:26.210 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:54:26.238 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:26 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:26.064+0000 7f7df9d72200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:54:26.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo modprobe fuse 2024-04-25T01:54:26.320 DEBUG:teuthology.orchestra.run.smithi031:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:54:26.376 INFO:teuthology.orchestra.run.smithi031.stdout:/proc 2024-04-25T01:54:26.376 INFO:teuthology.orchestra.run.smithi031.stdout:/sys 2024-04-25T01:54:26.376 INFO:teuthology.orchestra.run.smithi031.stdout:/dev 2024-04-25T01:54:26.376 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/security 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/shm 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/pts 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/run 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/cgroup 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/pstore 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/bpf 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/ 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/selinux 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/hugepages 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/mqueue 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/debug 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/tracing 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/config 2024-04-25T01:54:26.377 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/fuse/connections 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/1000 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/283c45baf276424fe5163bac3ea5b6634b4b32b5f9ece690cfad8997d6239bc2/merged 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/61ed8f5f7e2719c0e477a7cbade6739005c798d6d6a136fd352bdac861319c70/merged 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/0 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/4ba0220ee3e500ef4fed751d69344d302c8844ea2928f17c01ee693e87a64810/merged 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/58283f14dde71fb4d4be39631de34f5c83f15a18935426a4f7c6e86e85fbd8c7/merged 2024-04-25T01:54:26.378 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/fb73eca7d453910274ff53cdec645bcbf781ee8e4b0dcb2734aaa79c05335f7e/merged 2024-04-25T01:54:26.379 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/a5926bb2b6af6d1104582470214d80e052fdbc033749f86130e57226edf3c748/merged 2024-04-25T01:54:26.379 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/00969b13d07f743f4e9abfb4c3b6fd665efb09d75e2cbf15813663d0e4d91f40/merged 2024-04-25T01:54:26.379 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/0cef8fcae0bb48009c367cf891b54235700798d677a330dc44d1ffc41fc35e11/merged 2024-04-25T01:54:26.379 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/0d8ee7e545acb606ddf5c7310bcda33153daafd0fdcd87f17a4899996c61b0dd/merged 2024-04-25T01:54:26.379 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/969b7a3790c9066d903ab0b8870bdaac10b34fc55e36d61fbd0bc0b93528a113/merged 2024-04-25T01:54:26.379 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns 2024-04-25T01:54:26.379 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:54:26.379 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:54:26.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:26.379 DEBUG:teuthology.orchestra.run.smithi031:> ls /sys/fs/fuse/connections 2024-04-25T01:54:26.432 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T01:54:26.432 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d --client_fs=cephfs) 2024-04-25T01:54:26.439 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:26.185+0000 7f983b001200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:54:26.439 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:26.277+0000 7f983b001200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:54:26.475 DEBUG:teuthology.orchestra.run.smithi031:> sudo modprobe fuse 2024-04-25T01:54:26.500 DEBUG:teuthology.orchestra.run.smithi031:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/proc 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/sys 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/dev 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/security 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/shm 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/pts 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/run 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/cgroup 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/pstore 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/bpf 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/ 2024-04-25T01:54:26.558 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/selinux 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/hugepages 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/mqueue 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/debug 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/tracing 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/config 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/fuse/connections 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/1000 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph 2024-04-25T01:54:26.559 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/283c45baf276424fe5163bac3ea5b6634b4b32b5f9ece690cfad8997d6239bc2/merged 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/61ed8f5f7e2719c0e477a7cbade6739005c798d6d6a136fd352bdac861319c70/merged 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/0 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/4ba0220ee3e500ef4fed751d69344d302c8844ea2928f17c01ee693e87a64810/merged 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/58283f14dde71fb4d4be39631de34f5c83f15a18935426a4f7c6e86e85fbd8c7/merged 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/fb73eca7d453910274ff53cdec645bcbf781ee8e4b0dcb2734aaa79c05335f7e/merged 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/a5926bb2b6af6d1104582470214d80e052fdbc033749f86130e57226edf3c748/merged 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/00969b13d07f743f4e9abfb4c3b6fd665efb09d75e2cbf15813663d0e4d91f40/merged 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/0cef8fcae0bb48009c367cf891b54235700798d677a330dc44d1ffc41fc35e11/merged 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/0d8ee7e545acb606ddf5c7310bcda33153daafd0fdcd87f17a4899996c61b0dd/merged 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/969b7a3790c9066d903ab0b8870bdaac10b34fc55e36d61fbd0bc0b93528a113/merged 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:54:26.560 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:54:26.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:26.561 DEBUG:teuthology.orchestra.run.smithi031:> ls /sys/fs/fuse/connections 2024-04-25T01:54:26.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi031.stderr:2024-04-25T01:54:26.576+0000 7f6eb371d480 -1 init, newargv = 0x55ba03179060 newargc=15 2024-04-25T01:54:26.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi031.stderr:ceph-fuse[81038]: starting ceph client 2024-04-25T01:54:26.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi031.stderr:ceph-fuse[81038]: starting fuse 2024-04-25T01:54:26.659 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:26.349+0000 7fe95aabb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:54:26.660 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:26.443+0000 7fe95aabb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:54:26.872 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:26.521+0000 7f983b001200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:54:26.872 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:26.656+0000 7f983b001200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:54:26.872 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:26.752+0000 7f983b001200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:54:26.872 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:26.870+0000 7f983b001200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:54:26.988 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:26 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:26.639+0000 7f7df9d72200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:54:26.988 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:26 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:26.737+0000 7f7df9d72200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:54:27.054 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:26.696+0000 7fe95aabb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:54:27.054 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:26.836+0000 7fe95aabb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:54:27.054 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:26.934+0000 7fe95aabb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:54:27.054 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:27 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:27.053+0000 7fe95aabb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:54:27.189 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:26.963+0000 7f983b001200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:54:27.359 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:26 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:26.994+0000 7f7df9d72200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:54:27.359 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:27 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:27.134+0000 7f7df9d72200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:54:27.359 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:27 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:27.235+0000 7f7df9d72200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:54:27.359 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:27 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:27.357+0000 7f7df9d72200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:54:27.410 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:27 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:27.146+0000 7fe95aabb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:54:27.694 DEBUG:teuthology.orchestra.run.smithi031:> sudo modprobe fuse 2024-04-25T01:54:27.717 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:27 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:27.455+0000 7f7df9d72200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:54:27.722 DEBUG:teuthology.orchestra.run.smithi031:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:54:27.778 INFO:teuthology.orchestra.run.smithi031.stdout:/proc 2024-04-25T01:54:27.778 INFO:teuthology.orchestra.run.smithi031.stdout:/sys 2024-04-25T01:54:27.778 INFO:teuthology.orchestra.run.smithi031.stdout:/dev 2024-04-25T01:54:27.778 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/security 2024-04-25T01:54:27.778 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/shm 2024-04-25T01:54:27.778 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/pts 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/run 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/cgroup 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/pstore 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/bpf 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/ 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/selinux 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/hugepages 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/mqueue 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/debug 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/tracing 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/config 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/fuse/connections 2024-04-25T01:54:27.779 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/1000 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/283c45baf276424fe5163bac3ea5b6634b4b32b5f9ece690cfad8997d6239bc2/merged 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/61ed8f5f7e2719c0e477a7cbade6739005c798d6d6a136fd352bdac861319c70/merged 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/0 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/4ba0220ee3e500ef4fed751d69344d302c8844ea2928f17c01ee693e87a64810/merged 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/58283f14dde71fb4d4be39631de34f5c83f15a18935426a4f7c6e86e85fbd8c7/merged 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/fb73eca7d453910274ff53cdec645bcbf781ee8e4b0dcb2734aaa79c05335f7e/merged 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/a5926bb2b6af6d1104582470214d80e052fdbc033749f86130e57226edf3c748/merged 2024-04-25T01:54:27.780 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/00969b13d07f743f4e9abfb4c3b6fd665efb09d75e2cbf15813663d0e4d91f40/merged 2024-04-25T01:54:27.781 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/0cef8fcae0bb48009c367cf891b54235700798d677a330dc44d1ffc41fc35e11/merged 2024-04-25T01:54:27.781 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/0d8ee7e545acb606ddf5c7310bcda33153daafd0fdcd87f17a4899996c61b0dd/merged 2024-04-25T01:54:27.781 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/containers/storage/overlay/969b7a3790c9066d903ab0b8870bdaac10b34fc55e36d61fbd0bc0b93528a113/merged 2024-04-25T01:54:27.781 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns 2024-04-25T01:54:27.781 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:54:27.781 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:54:27.781 INFO:teuthology.orchestra.run.smithi031.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T01:54:27.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:27.781 DEBUG:teuthology.orchestra.run.smithi031:> ls /sys/fs/fuse/connections 2024-04-25T01:54:27.833 INFO:teuthology.orchestra.run.smithi031.stdout:115 2024-04-25T01:54:27.834 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-25T01:54:27.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo stdin-killer -- python3 -c ' 2024-04-25T01:54:27.834 DEBUG:teuthology.orchestra.run.smithi031:> import glob 2024-04-25T01:54:27.834 DEBUG:teuthology.orchestra.run.smithi031:> import re 2024-04-25T01:54:27.834 DEBUG:teuthology.orchestra.run.smithi031:> import os 2024-04-25T01:54:27.834 DEBUG:teuthology.orchestra.run.smithi031:> import subprocess 2024-04-25T01:54:27.834 DEBUG:teuthology.orchestra.run.smithi031:> 2024-04-25T01:54:27.834 DEBUG:teuthology.orchestra.run.smithi031:> def _find_admin_socket(client_name): 2024-04-25T01:54:27.834 DEBUG:teuthology.orchestra.run.smithi031:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-25T01:54:27.834 DEBUG:teuthology.orchestra.run.smithi031:> files = glob.glob(asok_path) 2024-04-25T01:54:27.834 DEBUG:teuthology.orchestra.run.smithi031:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-25T01:54:27.834 DEBUG:teuthology.orchestra.run.smithi031:> 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> # Given a non-glob path, it better be there 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> if "*" not in asok_path: 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> assert(len(files) == 1) 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> return files[0] 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> for f in files: 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> contents = proc_f.read() 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> if mountpoint in contents: 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> return f 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> print(_find_admin_socket("client.0")) 2024-04-25T01:54:27.835 DEBUG:teuthology.orchestra.run.smithi031:> ' 2024-04-25T01:54:27.956 INFO:teuthology.orchestra.run.smithi031.stdout:/var/run/ceph/ceph-client.0.81038.asok 2024-04-25T01:54:27.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:54:27.967 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.81038.asok 2024-04-25T01:54:27.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:27.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.81038.asok status 2024-04-25T01:54:28.061 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:27 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:27.775+0000 7f983b001200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:54:28.062 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 01:54:27 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z[34955]: 2024-04-25T01:54:27.992+0000 7f983b001200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:54:28.109 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:54:28.109 INFO:teuthology.orchestra.run.smithi031.stdout: "metadata": { 2024-04-25T01:54:28.109 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-25T01:54:28.109 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:54:28.109 INFO:teuthology.orchestra.run.smithi031.stdout: "entity_id": "0", 2024-04-25T01:54:28.109 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": "smithi031", 2024-04-25T01:54:28.109 INFO:teuthology.orchestra.run.smithi031.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-25T01:54:28.109 INFO:teuthology.orchestra.run.smithi031.stdout: "pid": "81038", 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "root": "/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d" 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "dentry_count": 0, 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "dentry_pinned_count": 0, 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 15375, 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "inst": { 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "name": { 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client", 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "num": 15375 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": { 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2487181974 2024-04-25T01:54:28.110 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": { 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2487181974 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "inst_str": "client.15375 192.168.0.1:0/2487181974", 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "addr_str": "192.168.0.1:0/2487181974", 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "inode_count": 5, 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_epoch": 26, 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_epoch": 89, 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_epoch_barrier": 0, 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklisted": false, 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs" 2024-04-25T01:54:28.111 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:54:28.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:28.118 DEBUG:teuthology.orchestra.run.smithi031:> stat --file-system '--printf=%T 2024-04-25T01:54:28.118 DEBUG:teuthology.orchestra.run.smithi031:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-25T01:54:28.138 INFO:teuthology.orchestra.run.smithi031.stdout:fuseblk 2024-04-25T01:54:28.138 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-25T01:54:28.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:28.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:54:28.202 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T01:54:28.220 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:54:28.220 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T01:54:28.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-25T01:54:28.344 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-mon[29204]: Standby manager daemon z restarted 2024-04-25T01:54:28.344 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-mon[29204]: Standby manager daemon z started 2024-04-25T01:54:28.344 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:54:28.344 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:54:28.344 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:54:28.344 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:54:28.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.347+0000 7f47d87c1640 1 -- 172.21.15.31:0/2572665598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d0101890 msgr2=0x7f47d0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.347+0000 7f47d87c1640 1 --2- 172.21.15.31:0/2572665598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d0101890 0x7f47d0101d10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009900 tx=0x7f47c002a410 comp rx=0 tx=0).stop 2024-04-25T01:54:28.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.347+0000 7f47d87c1640 1 -- 172.21.15.31:0/2572665598 shutdown_connections 2024-04-25T01:54:28.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.347+0000 7f47d87c1640 1 --2- 172.21.15.31:0/2572665598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47d0102250 0x7f47d0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.347+0000 7f47d87c1640 1 --2- 172.21.15.31:0/2572665598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d0101890 0x7f47d0101d10 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.347+0000 7f47d87c1640 1 --2- 172.21.15.31:0/2572665598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d01006a0 0x7f47d0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.347+0000 7f47d87c1640 1 -- 172.21.15.31:0/2572665598 >> 172.21.15.31:0/2572665598 conn(0x7f47d00fbc10 msgr2=0x7f47d00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:28.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.347+0000 7f47d87c1640 1 -- 172.21.15.31:0/2572665598 shutdown_connections 2024-04-25T01:54:28.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.347+0000 7f47d87c1640 1 -- 172.21.15.31:0/2572665598 wait complete. 2024-04-25T01:54:28.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.347+0000 7f47d87c1640 1 Processor -- start 2024-04-25T01:54:28.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.347+0000 7f47d87c1640 1 -- start start 2024-04-25T01:54:28.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d87c1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d01006a0 0x7f47d0104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d87c1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d0101890 0x7f47d0102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d87c1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47d0102250 0x7f47d01031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d87c1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f47d010c1d0 con 0x7f47d01006a0 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d87c1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f47d010c050 con 0x7f47d0101890 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d87c1640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f47d010c350 con 0x7f47d0102250 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d655d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d01006a0 0x7f47d0104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d655d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d01006a0 0x7f47d0104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34990/0 (socket says 172.21.15.31:34990) 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d655d640 1 -- 172.21.15.31:0/364317173 learned_addr learned my addr 172.21.15.31:0/364317173 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d6d5e640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47d0102250 0x7f47d01031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d5d5c640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d0101890 0x7f47d0102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d655d640 1 -- 172.21.15.31:0/364317173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47d0102250 msgr2=0x7f47d01031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d655d640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47d0102250 0x7f47d01031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d655d640 1 -- 172.21.15.31:0/364317173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d0101890 msgr2=0x7f47d0102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d655d640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d0101890 0x7f47d0102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d655d640 1 -- 172.21.15.31:0/364317173 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d0103a40 con 0x7f47d01006a0 2024-04-25T01:54:28.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47d655d640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d01006a0 0x7f47d0104630 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00e9c0 tx=0x7f47cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:28.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.348+0000 7f47c77fe640 1 -- 172.21.15.31:0/364317173 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc00cd80 con 0x7f47d01006a0 2024-04-25T01:54:28.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.349+0000 7f47c77fe640 1 -- 172.21.15.31:0/364317173 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47cc0035f0 con 0x7f47d01006a0 2024-04-25T01:54:28.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.349+0000 7f47d87c1640 1 -- 172.21.15.31:0/364317173 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d01a1a50 con 0x7f47d01006a0 2024-04-25T01:54:28.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.349+0000 7f47c77fe640 1 -- 172.21.15.31:0/364317173 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc010690 con 0x7f47d01006a0 2024-04-25T01:54:28.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.349+0000 7f47d87c1640 1 -- 172.21.15.31:0/364317173 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d01a1fc0 con 0x7f47d01006a0 2024-04-25T01:54:28.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.350+0000 7f47d87c1640 1 -- 172.21.15.31:0/364317173 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d01a2250 con 0x7f47d01006a0 2024-04-25T01:54:28.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.351+0000 7f47c77fe640 1 -- 172.21.15.31:0/364317173 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 20) ==== 137850+0+0 (secure 0 0 0) 0x7f47cc0037a0 con 0x7f47d01006a0 2024-04-25T01:54:28.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.352+0000 7f47c77fe640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f47bc0a2d00 0x7f47bc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.352+0000 7f47d5d5c640 1 -- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f47bc0a2d00 msgr2=0x7f47bc0a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2306032998 2024-04-25T01:54:28.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.352+0000 7f47c77fe640 1 -- 172.21.15.31:0/364317173 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8182+0+0 (secure 0 0 0) 0x7f47cc0cf450 con 0x7f47d01006a0 2024-04-25T01:54:28.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.352+0000 7f47d5d5c640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f47bc0a2d00 0x7f47bc0a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:54:28.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.357+0000 7f47c77fe640 1 -- 172.21.15.31:0/364317173 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f47cc09c380 con 0x7f47d01006a0 2024-04-25T01:54:28.410 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:27 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:27.988+0000 7fe95aabb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:54:28.410 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 01:54:28 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y[33795]: 2024-04-25T01:54:28.211+0000 7fe95aabb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:54:28.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:28 smithi040 ceph-mon[32418]: Standby manager daemon z restarted 2024-04-25T01:54:28.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:28 smithi040 ceph-mon[32418]: Standby manager daemon z started 2024-04-25T01:54:28.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:28 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:54:28.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:28 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:54:28.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:28 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:54:28.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:28 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:54:28.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:28 smithi134 ceph-mon[33098]: Standby manager daemon z restarted 2024-04-25T01:54:28.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:28 smithi134 ceph-mon[33098]: Standby manager daemon z started 2024-04-25T01:54:28.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:28 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:54:28.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:28 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:54:28.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:28 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:54:28.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:28 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.134:0/893852809' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:54:28.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.553+0000 7f47d5d5c640 1 -- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f47bc0a2d00 msgr2=0x7f47bc0a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/2306032998 2024-04-25T01:54:28.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.553+0000 7f47d5d5c640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f47bc0a2d00 0x7f47bc0a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:54:28.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.583+0000 7f47d87c1640 1 -- 172.21.15.31:0/364317173 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f47d01a2420 con 0x7f47d01006a0 2024-04-25T01:54:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.590+0000 7f47c77fe640 1 -- 172.21.15.31:0/364317173 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 21) ==== 137518+0+0 (secure 0 0 0) 0x7f47cc0982f0 con 0x7f47d01006a0 2024-04-25T01:54:28.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.591+0000 7f47c77fe640 1 -- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f47bc0a2d00 msgr2=0x7f47bc0a51c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:54:28.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:28.591+0000 7f47c77fe640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f47bc0a2d00 0x7f47bc0a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.738 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.343+0000 7f7df9d72200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:54:28.738 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.576+0000 7f7df9d72200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:54:29.063 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-25T01:54:29.064 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: Standby manager daemon y restarted 2024-04-25T01:54:29.064 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: Standby manager daemon y started 2024-04-25T01:54:29.064 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:54:29.064 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.828+0000 7f7d5bebc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:54:29.064 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.828+0000 7f7d5bebc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:54:29.064 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.828+0000 7f7d5bebc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:54:29.064 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.828+0000 7f7d5bebc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:54:29.065 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.828+0000 7f7d5bebc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:54:29.065 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.829+0000 7f7d596b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:54:29.065 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.829+0000 7f7d596b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:54:29.065 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.829+0000 7f7d596b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:54:29.065 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.829+0000 7f7d596b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:54:29.065 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:54:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:54:28.829+0000 7f7d596b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: Standby manager daemon y restarted 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: Standby manager daemon y started 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: Active manager daemon x restarted 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: Activating manager daemon x 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: mgrmap e21: x(active, starting, since 0.0130214s), standbys: z, y 2024-04-25T01:54:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/364317173' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:54:29.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:54:29.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:54:29.069 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:54:29.069 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:54:29.069 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:54:29.410 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-25T01:54:29.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: Standby manager daemon y restarted 2024-04-25T01:54:29.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: Standby manager daemon y started 2024-04-25T01:54:29.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:54:29.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:54:29.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:54:29.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:54:29.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: Active manager daemon x restarted 2024-04-25T01:54:29.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: Activating manager daemon x 2024-04-25T01:54:29.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:54:29.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: mgrmap e21: x(active, starting, since 0.0130214s), standbys: z, y 2024-04-25T01:54:29.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/364317173' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:54:29.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:54:29.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: Manager daemon x is now available 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:54:29.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:54:29.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:29 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:54:29.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:54:29.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:54:29.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.? 172.21.15.40:0/2224167002' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:54:29.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: Active manager daemon x restarted 2024-04-25T01:54:29.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: Activating manager daemon x 2024-04-25T01:54:29.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:54:29.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: mgrmap e21: x(active, starting, since 0.0130214s), standbys: z, y 2024-04-25T01:54:29.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/364317173' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:54:29.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:54:29.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:54:29.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:54:29.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:54:29.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:54:29.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:54:29.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:54:29.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:54:29.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:54:29.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:54:29.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: Manager daemon x is now available 2024-04-25T01:54:29.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:54:29.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:54:29.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:29 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:54:29.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: Manager daemon x is now available 2024-04-25T01:54:29.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:54:29.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:54:29.488 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:29 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:54:29.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.583+0000 7f47c77fe640 1 -- 172.21.15.31:0/364317173 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v27) ==== 117+0+0 (secure 0 0 0) 0x7f47cc0c7f30 con 0x7f47d01006a0 2024-04-25T01:54:29.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.585+0000 7f47d87c1640 1 -- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d01006a0 msgr2=0x7f47d0104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.585+0000 7f47d87c1640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d01006a0 0x7f47d0104630 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00e9c0 tx=0x7f47cc00ee80 comp rx=0 tx=0).stop 2024-04-25T01:54:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.585+0000 7f47d87c1640 1 -- 172.21.15.31:0/364317173 shutdown_connections 2024-04-25T01:54:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.585+0000 7f47d87c1640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:6800/2306032998,v1:172.21.15.31:6801/2306032998] conn(0x7f47bc0a2d00 0x7f47bc0a51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.585+0000 7f47d87c1640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47d0102250 0x7f47d01031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.585+0000 7f47d87c1640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47d0101890 0x7f47d0102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.585+0000 7f47d87c1640 1 --2- 172.21.15.31:0/364317173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d01006a0 0x7f47d0104630 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.585+0000 7f47d87c1640 1 -- 172.21.15.31:0/364317173 >> 172.21.15.31:0/364317173 conn(0x7f47d00fbc10 msgr2=0x7f47d00fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.585+0000 7f47d87c1640 1 -- 172.21.15.31:0/364317173 shutdown_connections 2024-04-25T01:54:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.585+0000 7f47d87c1640 1 -- 172.21.15.31:0/364317173 wait complete. 2024-04-25T01:54:29.607 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T01:54:29.619 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-25T01:54:29.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:54:29.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.768+0000 7f89ab1d5640 1 -- 172.21.15.31:0/1505533384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a41006a0 msgr2=0x7f89a4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.768+0000 7f89ab1d5640 1 --2- 172.21.15.31:0/1505533384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a41006a0 0x7f89a4100ac0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009900 tx=0x7f89a002a410 comp rx=0 tx=0).stop 2024-04-25T01:54:29.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.768+0000 7f89ab1d5640 1 -- 172.21.15.31:0/1505533384 shutdown_connections 2024-04-25T01:54:29.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.768+0000 7f89ab1d5640 1 --2- 172.21.15.31:0/1505533384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89a4102250 0x7f89a4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.768+0000 7f89ab1d5640 1 --2- 172.21.15.31:0/1505533384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89a4101890 0x7f89a4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.768+0000 7f89ab1d5640 1 --2- 172.21.15.31:0/1505533384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a41006a0 0x7f89a4100ac0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.768+0000 7f89ab1d5640 1 -- 172.21.15.31:0/1505533384 >> 172.21.15.31:0/1505533384 conn(0x7f89a40fbc30 msgr2=0x7f89a40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:29.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.768+0000 7f89ab1d5640 1 -- 172.21.15.31:0/1505533384 shutdown_connections 2024-04-25T01:54:29.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.768+0000 7f89ab1d5640 1 -- 172.21.15.31:0/1505533384 wait complete. 2024-04-25T01:54:29.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89ab1d5640 1 Processor -- start 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89ab1d5640 1 -- start start 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89ab1d5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89a41006a0 0x7f89a4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89ab1d5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89a4101890 0x7f89a4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89ab1d5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a4102250 0x7f89a419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89ab1d5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f89a410c1d0 con 0x7f89a4102250 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89ab1d5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f89a410c050 con 0x7f89a41006a0 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89ab1d5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f89a410c350 con 0x7f89a4101890 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89a9772640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a4102250 0x7f89a419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89a8f71640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89a41006a0 0x7f89a4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89a9772640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a4102250 0x7f89a419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35136/0 (socket says 172.21.15.31:35136) 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f89a9772640 1 -- 172.21.15.31:0/914576174 learned_addr learned my addr 172.21.15.31:0/914576174 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.769+0000 7f899bfff640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89a4101890 0x7f89a4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f89a8f71640 1 -- 172.21.15.31:0/914576174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89a4101890 msgr2=0x7f89a4197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f89a8f71640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89a4101890 0x7f89a4197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f89a8f71640 1 -- 172.21.15.31:0/914576174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a4102250 msgr2=0x7f89a419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f89a8f71640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a4102250 0x7f89a419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f89a8f71640 1 -- 172.21.15.31:0/914576174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a419f620 con 0x7f89a41006a0 2024-04-25T01:54:29.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f89a9772640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a4102250 0x7f89a419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:29.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f89a8f71640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89a41006a0 0x7f89a4197a50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009230 tx=0x7f89a0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:29.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f8999ffb640 1 -- 172.21.15.31:0/914576174 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a003f030 con 0x7f89a41006a0 2024-04-25T01:54:29.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f8999ffb640 1 -- 172.21.15.31:0/914576174 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89a0003aa0 con 0x7f89a41006a0 2024-04-25T01:54:29.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f89ab1d5640 1 -- 172.21.15.31:0/914576174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a419f8b0 con 0x7f89a41006a0 2024-04-25T01:54:29.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f8999ffb640 1 -- 172.21.15.31:0/914576174 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a0035070 con 0x7f89a41006a0 2024-04-25T01:54:29.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.770+0000 7f89ab1d5640 1 -- 172.21.15.31:0/914576174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a419fd70 con 0x7f89a41006a0 2024-04-25T01:54:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.771+0000 7f89ab1d5640 1 -- 172.21.15.31:0/914576174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a419ffd0 con 0x7f89a41006a0 2024-04-25T01:54:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.773+0000 7f8999ffb640 1 -- 172.21.15.31:0/914576174 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f89a0042030 con 0x7f89a41006a0 2024-04-25T01:54:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.773+0000 7f8999ffb640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f898c0a2990 0x7f898c0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.773+0000 7f899bfff640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f898c0a2990 0x7f898c0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.773+0000 7f8999ffb640 1 -- 172.21.15.31:0/914576174 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f89a00ef6f0 con 0x7f89a41006a0 2024-04-25T01:54:29.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.774+0000 7f899bfff640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f898c0a2990 0x7f898c0a4e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f89a4198f10 tx=0x7f8994005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:29.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.777+0000 7f8999ffb640 1 -- 172.21.15.31:0/914576174 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f89a00bc220 con 0x7f89a41006a0 2024-04-25T01:54:29.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.965+0000 7f89ab1d5640 1 -- 172.21.15.31:0/914576174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89a41a01a0 con 0x7f89a41006a0 2024-04-25T01:54:29.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.966+0000 7f8999ffb640 1 -- 172.21.15.31:0/914576174 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22855 (secure 0 0 0) 0x7f89a00c0520 con 0x7f89a41006a0 2024-04-25T01:54:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:29.968 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:54:28.579231+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3045817976":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6800/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1358173279":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/686937010":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/3913527371":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:54:29.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 -- 172.21.15.31:0/914576174 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f898c0a2990 msgr2=0x7f898c0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f898c0a2990 0x7f898c0a4e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f89a4198f10 tx=0x7f8994005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:29.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 -- 172.21.15.31:0/914576174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89a41006a0 msgr2=0x7f89a4197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89a41006a0 0x7f89a4197a50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009230 tx=0x7f89a0004270 comp rx=0 tx=0).stop 2024-04-25T01:54:29.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 -- 172.21.15.31:0/914576174 shutdown_connections 2024-04-25T01:54:29.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f898c0a2990 0x7f898c0a4e50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a4102250 0x7f89a419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89a4101890 0x7f89a4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 --2- 172.21.15.31:0/914576174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89a41006a0 0x7f89a4197a50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 -- 172.21.15.31:0/914576174 >> 172.21.15.31:0/914576174 conn(0x7f89a40fbc30 msgr2=0x7f89a4069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:29.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 -- 172.21.15.31:0/914576174 shutdown_connections 2024-04-25T01:54:29.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:29.970+0000 7f89ab1d5640 1 -- 172.21.15.31:0/914576174 wait complete. 2024-04-25T01:54:29.983 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:23.432566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:24.255373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:54:29.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:54:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.092+0000 7f1e64ddb640 1 -- 172.21.15.31:0/1125684863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e601006a0 msgr2=0x7f1e60100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.092+0000 7f1e64ddb640 1 --2- 172.21.15.31:0/1125684863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e601006a0 0x7f1e60100ac0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1e58009900 tx=0x7f1e5802a410 comp rx=0 tx=0).stop 2024-04-25T01:54:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.092+0000 7f1e64ddb640 1 -- 172.21.15.31:0/1125684863 shutdown_connections 2024-04-25T01:54:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.092+0000 7f1e64ddb640 1 --2- 172.21.15.31:0/1125684863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e60102250 0x7f1e60108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.092+0000 7f1e64ddb640 1 --2- 172.21.15.31:0/1125684863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e60101890 0x7f1e60101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.092+0000 7f1e64ddb640 1 --2- 172.21.15.31:0/1125684863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e601006a0 0x7f1e60100ac0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.092+0000 7f1e64ddb640 1 -- 172.21.15.31:0/1125684863 >> 172.21.15.31:0/1125684863 conn(0x7f1e600fbc30 msgr2=0x7f1e600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.092+0000 7f1e64ddb640 1 -- 172.21.15.31:0/1125684863 shutdown_connections 2024-04-25T01:54:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.092+0000 7f1e64ddb640 1 -- 172.21.15.31:0/1125684863 wait complete. 2024-04-25T01:54:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.093+0000 7f1e64ddb640 1 Processor -- start 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.093+0000 7f1e64ddb640 1 -- start start 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.093+0000 7f1e64ddb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e601006a0 0x7f1e60197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.093+0000 7f1e64ddb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e60101890 0x7f1e60197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.093+0000 7f1e64ddb640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e60102250 0x7f1e6019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.093+0000 7f1e64ddb640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6010d320 con 0x7f1e60101890 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.093+0000 7f1e64ddb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6010d1a0 con 0x7f1e601006a0 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.093+0000 7f1e64ddb640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6010d4a0 con 0x7f1e60102250 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.093+0000 7f1e5dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e60101890 0x7f1e60197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.093+0000 7f1e5dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e60101890 0x7f1e60197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35158/0 (socket says 172.21.15.31:35158) 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e5dd9b640 1 -- 172.21.15.31:0/88333772 learned_addr learned my addr 172.21.15.31:0/88333772 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e5ed9d640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e60102250 0x7f1e6019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e5e59c640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e601006a0 0x7f1e60197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e5dd9b640 1 -- 172.21.15.31:0/88333772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e60102250 msgr2=0x7f1e6019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e5dd9b640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e60102250 0x7f1e6019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e5dd9b640 1 -- 172.21.15.31:0/88333772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e601006a0 msgr2=0x7f1e60197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e5dd9b640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e601006a0 0x7f1e60197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e5dd9b640 1 -- 172.21.15.31:0/88333772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6019f670 con 0x7f1e60101890 2024-04-25T01:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e5dd9b640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e60101890 0x7f1e60197f90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800b4a0 tx=0x7f1e4800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e577fe640 1 -- 172.21.15.31:0/88333772 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48014070 con 0x7f1e60101890 2024-04-25T01:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e577fe640 1 -- 172.21.15.31:0/88333772 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e48010ac0 con 0x7f1e60101890 2024-04-25T01:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e577fe640 1 -- 172.21.15.31:0/88333772 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48017c30 con 0x7f1e60101890 2024-04-25T01:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e64ddb640 1 -- 172.21.15.31:0/88333772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e6019f960 con 0x7f1e60101890 2024-04-25T01:54:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.094+0000 7f1e64ddb640 1 -- 172.21.15.31:0/88333772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e6019ff30 con 0x7f1e60101890 2024-04-25T01:54:30.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.095+0000 7f1e64ddb640 1 -- 172.21.15.31:0/88333772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e60107a90 con 0x7f1e60101890 2024-04-25T01:54:30.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.096+0000 7f1e577fe640 1 -- 172.21.15.31:0/88333772 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f1e48017070 con 0x7f1e60101890 2024-04-25T01:54:30.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.097+0000 7f1e577fe640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e4c0a2990 0x7f1e4c0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.097+0000 7f1e577fe640 1 -- 172.21.15.31:0/88333772 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f1e480ce3e0 con 0x7f1e60101890 2024-04-25T01:54:30.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.097+0000 7f1e5e59c640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e4c0a2990 0x7f1e4c0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.098+0000 7f1e5e59c640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e4c0a2990 0x7f1e4c0a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1e58004770 tx=0x7f1e58000bc0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:30.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.102+0000 7f1e577fe640 1 -- 172.21.15.31:0/88333772 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1e4809b560 con 0x7f1e60101890 2024-04-25T01:54:30.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.289+0000 7f1e64ddb640 1 -- 172.21.15.31:0/88333772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1e60062340 con 0x7f1e60101890 2024-04-25T01:54:30.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.290+0000 7f1e577fe640 1 -- 172.21.15.31:0/88333772 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f1e4809ae60 con 0x7f1e60101890 2024-04-25T01:54:30.291 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T01:54:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 -- 172.21.15.31:0/88333772 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e4c0a2990 msgr2=0x7f1e4c0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e4c0a2990 0x7f1e4c0a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1e58004770 tx=0x7f1e58000bc0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 -- 172.21.15.31:0/88333772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e60101890 msgr2=0x7f1e60197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e60101890 0x7f1e60197f90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800b4a0 tx=0x7f1e4800b960 comp rx=0 tx=0).stop 2024-04-25T01:54:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 -- 172.21.15.31:0/88333772 shutdown_connections 2024-04-25T01:54:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e4c0a2990 0x7f1e4c0a4e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e60102250 0x7f1e6019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e60101890 0x7f1e60197f90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 --2- 172.21.15.31:0/88333772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e601006a0 0x7f1e60197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 -- 172.21.15.31:0/88333772 >> 172.21.15.31:0/88333772 conn(0x7f1e600fbc30 msgr2=0x7f1e60104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:30.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 -- 172.21.15.31:0/88333772 shutdown_connections 2024-04-25T01:54:30.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.292+0000 7f1e64ddb640 1 -- 172.21.15.31:0/88333772 wait complete. 2024-04-25T01:54:30.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:54:30.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.412+0000 7f354dd48640 1 -- 172.21.15.31:0/3770630657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3548074120 msgr2=0x7f3548074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.412+0000 7f354dd48640 1 --2- 172.21.15.31:0/3770630657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3548074120 0x7f3548074580 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3538009900 tx=0x7f353802a480 comp rx=0 tx=0).stop 2024-04-25T01:54:30.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.412+0000 7f354dd48640 1 -- 172.21.15.31:0/3770630657 shutdown_connections 2024-04-25T01:54:30.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.412+0000 7f354dd48640 1 --2- 172.21.15.31:0/3770630657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3548109650 0x7f354810bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.412+0000 7f354dd48640 1 --2- 172.21.15.31:0/3770630657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3548074120 0x7f3548074580 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.412+0000 7f354dd48640 1 --2- 172.21.15.31:0/3770630657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3548075700 0x7f3548073b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.412+0000 7f354dd48640 1 -- 172.21.15.31:0/3770630657 >> 172.21.15.31:0/3770630657 conn(0x7f35480fcd10 msgr2=0x7f35480ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:30.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.412+0000 7f354dd48640 1 -- 172.21.15.31:0/3770630657 shutdown_connections 2024-04-25T01:54:30.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.412+0000 7f354dd48640 1 -- 172.21.15.31:0/3770630657 wait complete. 2024-04-25T01:54:30.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.413+0000 7f354dd48640 1 Processor -- start 2024-04-25T01:54:30.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.413+0000 7f354dd48640 1 -- start start 2024-04-25T01:54:30.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.413+0000 7f354dd48640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3548074120 0x7f3548199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f354dd48640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3548075700 0x7f354819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f354dd48640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3548109650 0x7f35481a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f354dd48640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f354810f4d0 con 0x7f3548075700 2024-04-25T01:54:30.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f354dd48640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f354810f350 con 0x7f3548074120 2024-04-25T01:54:30.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f354dd48640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f354810f650 con 0x7f3548109650 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3546ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3548075700 0x7f354819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3546ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3548075700 0x7f354819a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35186/0 (socket says 172.21.15.31:35186) 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3546ffd640 1 -- 172.21.15.31:0/2992928543 learned_addr learned my addr 172.21.15.31:0/2992928543 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f35477fe640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3548074120 0x7f3548199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3547fff640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3548109650 0x7f35481a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3546ffd640 1 -- 172.21.15.31:0/2992928543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3548109650 msgr2=0x7f35481a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3546ffd640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3548109650 0x7f35481a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3546ffd640 1 -- 172.21.15.31:0/2992928543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3548074120 msgr2=0x7f3548199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3546ffd640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3548074120 0x7f3548199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3546ffd640 1 -- 172.21.15.31:0/2992928543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35481a17f0 con 0x7f3548075700 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f35477fe640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3548074120 0x7f3548199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3546ffd640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3548075700 0x7f354819a110 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3538005940 tx=0x7f35380042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3544ff9640 1 -- 172.21.15.31:0/2992928543 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353803f030 con 0x7f3548075700 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f3544ff9640 1 -- 172.21.15.31:0/2992928543 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35380318f0 con 0x7f3548075700 2024-04-25T01:54:30.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.414+0000 7f354dd48640 1 -- 172.21.15.31:0/2992928543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35481a1a80 con 0x7f3548075700 2024-04-25T01:54:30.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.415+0000 7f3544ff9640 1 -- 172.21.15.31:0/2992928543 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3538035070 con 0x7f3548075700 2024-04-25T01:54:30.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.415+0000 7f354dd48640 1 -- 172.21.15.31:0/2992928543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35481a1fc0 con 0x7f3548075700 2024-04-25T01:54:30.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.416+0000 7f354dd48640 1 -- 172.21.15.31:0/2992928543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3548108410 con 0x7f3548075700 2024-04-25T01:54:30.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.417+0000 7f3544ff9640 1 -- 172.21.15.31:0/2992928543 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f3538042030 con 0x7f3548075700 2024-04-25T01:54:30.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.418+0000 7f3544ff9640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f35340a2a10 0x7f35340a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.418+0000 7f35477fe640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f35340a2a10 0x7f35340a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.418+0000 7f3544ff9640 1 -- 172.21.15.31:0/2992928543 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f35380ef710 con 0x7f3548075700 2024-04-25T01:54:30.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.419+0000 7f35477fe640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f35340a2a10 0x7f35340a4ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3530009fc0 tx=0x7f3530005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:30.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.422+0000 7f3544ff9640 1 -- 172.21.15.31:0/2992928543 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f35380bc240 con 0x7f3548075700 2024-04-25T01:54:30.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.610+0000 7f3544ff9640 1 -- 172.21.15.31:0/2992928543 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 23) ==== 137953+0+0 (secure 0 0 0) 0x7f35380b88b0 con 0x7f3548075700 2024-04-25T01:54:30.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.639+0000 7f354dd48640 1 -- 172.21.15.31:0/2992928543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f354802d3c0 con 0x7f3548075700 2024-04-25T01:54:30.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.639+0000 7f3544ff9640 1 -- 172.21.15.31:0/2992928543 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f35380b3b10 con 0x7f3548075700 2024-04-25T01:54:30.640 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T01:54:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.641+0000 7f354dd48640 1 -- 172.21.15.31:0/2992928543 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f35340a2a10 msgr2=0x7f35340a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.641+0000 7f354dd48640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f35340a2a10 0x7f35340a4ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3530009fc0 tx=0x7f3530005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.641+0000 7f354dd48640 1 -- 172.21.15.31:0/2992928543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3548075700 msgr2=0x7f354819a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.641+0000 7f354dd48640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3548075700 0x7f354819a110 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3538005940 tx=0x7f35380042f0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.641+0000 7f354dd48640 1 -- 172.21.15.31:0/2992928543 shutdown_connections 2024-04-25T01:54:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.641+0000 7f354dd48640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f35340a2a10 0x7f35340a4ed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.641+0000 7f354dd48640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3548109650 0x7f35481a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.641+0000 7f354dd48640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3548075700 0x7f354819a110 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.641+0000 7f354dd48640 1 --2- 172.21.15.31:0/2992928543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3548074120 0x7f3548199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.641+0000 7f354dd48640 1 -- 172.21.15.31:0/2992928543 >> 172.21.15.31:0/2992928543 conn(0x7f35480fcd10 msgr2=0x7f35481055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:30.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.642+0000 7f354dd48640 1 -- 172.21.15.31:0/2992928543 shutdown_connections 2024-04-25T01:54:30.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.642+0000 7f354dd48640 1 -- 172.21.15.31:0/2992928543 wait complete. 2024-04-25T01:54:30.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:54:30.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.780+0000 7f0590d90640 1 -- 172.21.15.31:0/4092012728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f058c102230 msgr2=0x7f058c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.780+0000 7f0590d90640 1 --2- 172.21.15.31:0/4092012728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f058c102230 0x7f058c108ac0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0584009ca0 tx=0x7f058402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.780+0000 7f0590d90640 1 -- 172.21.15.31:0/4092012728 shutdown_connections 2024-04-25T01:54:30.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.780+0000 7f0590d90640 1 --2- 172.21.15.31:0/4092012728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f058c102230 0x7f058c108ac0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.780+0000 7f0590d90640 1 --2- 172.21.15.31:0/4092012728 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c101870 0x7f058c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.780+0000 7f0590d90640 1 --2- 172.21.15.31:0/4092012728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f058c100680 0x7f058c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.780+0000 7f0590d90640 1 -- 172.21.15.31:0/4092012728 >> 172.21.15.31:0/4092012728 conn(0x7f058c0fbbf0 msgr2=0x7f058c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:30.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.780+0000 7f0590d90640 1 -- 172.21.15.31:0/4092012728 shutdown_connections 2024-04-25T01:54:30.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.780+0000 7f0590d90640 1 -- 172.21.15.31:0/4092012728 wait complete. 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f0590d90640 1 Processor -- start 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f0590d90640 1 -- start start 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f0590d90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f058c100680 0x7f058c199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f0590d90640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c101870 0x7f058c19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f0590d90640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f058c102230 0x7f058c1a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f0590d90640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f058c10d330 con 0x7f058c102230 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f0590d90640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f058c10d1b0 con 0x7f058c100680 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f0590d90640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f058c10d4b0 con 0x7f058c101870 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f058ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f058c102230 0x7f058c1a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f058a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f058c100680 0x7f058c199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f058ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f058c102230 0x7f058c1a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35220/0 (socket says 172.21.15.31:35220) 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f058ad9d640 1 -- 172.21.15.31:0/3172648553 learned_addr learned my addr 172.21.15.31:0/3172648553 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.781+0000 7f0589d9b640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c101870 0x7f058c19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.782+0000 7f058a59c640 1 -- 172.21.15.31:0/3172648553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c101870 msgr2=0x7f058c19a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.782+0000 7f058a59c640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c101870 0x7f058c19a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.782+0000 7f058a59c640 1 -- 172.21.15.31:0/3172648553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f058c102230 msgr2=0x7f058c1a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.782+0000 7f058a59c640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f058c102230 0x7f058c1a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.782+0000 7f058a59c640 1 -- 172.21.15.31:0/3172648553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c1a17c0 con 0x7f058c100680 2024-04-25T01:54:30.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.782+0000 7f058a59c640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f058c100680 0x7f058c199ba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f057c00e9c0 tx=0x7f057c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:30.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.782+0000 7f057b7fe640 1 -- 172.21.15.31:0/3172648553 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c00ccc0 con 0x7f058c100680 2024-04-25T01:54:30.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.782+0000 7f0590d90640 1 -- 172.21.15.31:0/3172648553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058c1a1ab0 con 0x7f058c100680 2024-04-25T01:54:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.782+0000 7f057b7fe640 1 -- 172.21.15.31:0/3172648553 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f057c0035f0 con 0x7f058c100680 2024-04-25T01:54:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.782+0000 7f057b7fe640 1 -- 172.21.15.31:0/3172648553 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c00ce60 con 0x7f058c100680 2024-04-25T01:54:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.782+0000 7f0590d90640 1 -- 172.21.15.31:0/3172648553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058c1a2050 con 0x7f058c100680 2024-04-25T01:54:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.784+0000 7f0590d90640 1 -- 172.21.15.31:0/3172648553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f058c1a2280 con 0x7f058c100680 2024-04-25T01:54:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.785+0000 7f057b7fe640 1 -- 172.21.15.31:0/3172648553 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 23) ==== 137953+0+0 (secure 0 0 0) 0x7f057c0037a0 con 0x7f058c100680 2024-04-25T01:54:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.785+0000 7f057b7fe640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05740a2d00 0x7f05740a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.785+0000 7f057b7fe640 1 -- 172.21.15.31:0/3172648553 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f057c0cf8a0 con 0x7f058c100680 2024-04-25T01:54:30.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.785+0000 7f0589d9b640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05740a2d00 0x7f05740a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.786+0000 7f0589d9b640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05740a2d00 0x7f05740a51c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f058c19b0b0 tx=0x7f0580005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:30.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.791+0000 7f057b7fe640 1 -- 172.21.15.31:0/3172648553 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f057c09c4b0 con 0x7f058c100680 2024-04-25T01:54:30.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:30 smithi040 ceph-mon[32418]: [25/Apr/2024:01:54:29] ENGINE Bus STARTING 2024-04-25T01:54:30.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:30 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/364317173' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T01:54:30.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:30 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:54:30.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:30 smithi040 ceph-mon[32418]: mgrmap e22: x(active, since 1.02505s), standbys: z, y 2024-04-25T01:54:30.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:30 smithi040 ceph-mon[32418]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:30.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:30 smithi040 ceph-mon[32418]: [25/Apr/2024:01:54:29] ENGINE Serving on https://172.21.15.31:7150 2024-04-25T01:54:30.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:30 smithi040 ceph-mon[32418]: [25/Apr/2024:01:54:29] ENGINE Serving on http://172.21.15.31:8765 2024-04-25T01:54:30.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:30 smithi040 ceph-mon[32418]: [25/Apr/2024:01:54:29] ENGINE Bus STARTED 2024-04-25T01:54:30.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:30 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/914576174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:30.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:30 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/88333772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:30.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:30 smithi134 ceph-mon[33098]: [25/Apr/2024:01:54:29] ENGINE Bus STARTING 2024-04-25T01:54:30.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:30 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/364317173' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T01:54:30.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:30 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:54:30.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:30 smithi134 ceph-mon[33098]: mgrmap e22: x(active, since 1.02505s), standbys: z, y 2024-04-25T01:54:30.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:30 smithi134 ceph-mon[33098]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:30.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:30 smithi134 ceph-mon[33098]: [25/Apr/2024:01:54:29] ENGINE Serving on https://172.21.15.31:7150 2024-04-25T01:54:30.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:30 smithi134 ceph-mon[33098]: [25/Apr/2024:01:54:29] ENGINE Serving on http://172.21.15.31:8765 2024-04-25T01:54:30.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:30 smithi134 ceph-mon[33098]: [25/Apr/2024:01:54:29] ENGINE Bus STARTED 2024-04-25T01:54:30.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:30 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/914576174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:30.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:30 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/88333772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:30.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.980+0000 7f0590d90640 1 -- 172.21.15.31:0/3172648553 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f058c1a2450 con 0x7f058c100680 2024-04-25T01:54:30.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.980+0000 7f057b7fe640 1 -- 172.21.15.31:0/3172648553 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f057c09bdb0 con 0x7f058c100680 2024-04-25T01:54:30.982 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T01:54:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.982+0000 7f0590d90640 1 -- 172.21.15.31:0/3172648553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05740a2d00 msgr2=0x7f05740a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.982+0000 7f0590d90640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05740a2d00 0x7f05740a51c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f058c19b0b0 tx=0x7f0580005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.982+0000 7f0590d90640 1 -- 172.21.15.31:0/3172648553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f058c100680 msgr2=0x7f058c199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.982+0000 7f0590d90640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f058c100680 0x7f058c199ba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f057c00e9c0 tx=0x7f057c00ee80 comp rx=0 tx=0).stop 2024-04-25T01:54:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.983+0000 7f0590d90640 1 -- 172.21.15.31:0/3172648553 shutdown_connections 2024-04-25T01:54:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.983+0000 7f0590d90640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05740a2d00 0x7f05740a51c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.983+0000 7f0590d90640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f058c102230 0x7f058c1a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.983+0000 7f0590d90640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c101870 0x7f058c19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.983+0000 7f0590d90640 1 --2- 172.21.15.31:0/3172648553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f058c100680 0x7f058c199ba0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.983+0000 7f0590d90640 1 -- 172.21.15.31:0/3172648553 >> 172.21.15.31:0/3172648553 conn(0x7f058c0fbbf0 msgr2=0x7f058c109480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:30.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.983+0000 7f0590d90640 1 -- 172.21.15.31:0/3172648553 shutdown_connections 2024-04-25T01:54:30.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:30.983+0000 7f0590d90640 1 -- 172.21.15.31:0/3172648553 wait complete. 2024-04-25T01:54:30.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:30 smithi031 ceph-mon[29204]: [25/Apr/2024:01:54:29] ENGINE Bus STARTING 2024-04-25T01:54:30.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:30 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/364317173' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T01:54:30.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:30 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:54:30.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:30 smithi031 ceph-mon[29204]: mgrmap e22: x(active, since 1.02505s), standbys: z, y 2024-04-25T01:54:30.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:30 smithi031 ceph-mon[29204]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:30.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:30 smithi031 ceph-mon[29204]: [25/Apr/2024:01:54:29] ENGINE Serving on https://172.21.15.31:7150 2024-04-25T01:54:30.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:30 smithi031 ceph-mon[29204]: [25/Apr/2024:01:54:29] ENGINE Serving on http://172.21.15.31:8765 2024-04-25T01:54:30.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:30 smithi031 ceph-mon[29204]: [25/Apr/2024:01:54:29] ENGINE Bus STARTED 2024-04-25T01:54:30.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:30 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/914576174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:30.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:30 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/88333772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:30.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:54:31.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.097+0000 7f48c9122640 1 -- 172.21.15.31:0/1025535405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48c4101870 msgr2=0x7f48c4101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.097+0000 7f48c9122640 1 --2- 172.21.15.31:0/1025535405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48c4101870 0x7f48c4101cf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f48b8009900 tx=0x7f48b802a410 comp rx=0 tx=0).stop 2024-04-25T01:54:31.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.097+0000 7f48c9122640 1 -- 172.21.15.31:0/1025535405 shutdown_connections 2024-04-25T01:54:31.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.097+0000 7f48c9122640 1 --2- 172.21.15.31:0/1025535405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48c4102230 0x7f48c4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.097+0000 7f48c9122640 1 --2- 172.21.15.31:0/1025535405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48c4101870 0x7f48c4101cf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.097+0000 7f48c9122640 1 --2- 172.21.15.31:0/1025535405 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4100680 0x7f48c4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.097+0000 7f48c9122640 1 -- 172.21.15.31:0/1025535405 >> 172.21.15.31:0/1025535405 conn(0x7f48c40fbbf0 msgr2=0x7f48c40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.097+0000 7f48c9122640 1 -- 172.21.15.31:0/1025535405 shutdown_connections 2024-04-25T01:54:31.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.097+0000 7f48c9122640 1 -- 172.21.15.31:0/1025535405 wait complete. 2024-04-25T01:54:31.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.098+0000 7f48c9122640 1 Processor -- start 2024-04-25T01:54:31.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.098+0000 7f48c9122640 1 -- start start 2024-04-25T01:54:31.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.098+0000 7f48c9122640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48c4100680 0x7f48c4104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.098+0000 7f48c9122640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4101870 0x7f48c4102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.098+0000 7f48c9122640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48c4102230 0x7f48c4103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.098+0000 7f48c9122640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f48c410c1b0 con 0x7f48c4102230 2024-04-25T01:54:31.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.098+0000 7f48c9122640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f48c410c030 con 0x7f48c4100680 2024-04-25T01:54:31.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.098+0000 7f48c9122640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f48c410c330 con 0x7f48c4101870 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48c4102230 0x7f48c4103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c2d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48c4100680 0x7f48c4104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48c4102230 0x7f48c4103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35240/0 (socket says 172.21.15.31:35240) 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c359e640 1 -- 172.21.15.31:0/1778549472 learned_addr learned my addr 172.21.15.31:0/1778549472 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c259c640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4101870 0x7f48c4102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c2d9d640 1 -- 172.21.15.31:0/1778549472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4101870 msgr2=0x7f48c4102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c2d9d640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4101870 0x7f48c4102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c2d9d640 1 -- 172.21.15.31:0/1778549472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48c4102230 msgr2=0x7f48c4103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c2d9d640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48c4102230 0x7f48c4103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c2d9d640 1 -- 172.21.15.31:0/1778549472 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c4103a20 con 0x7f48c4100680 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c359e640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48c4102230 0x7f48c4103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c2d9d640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48c4100680 0x7f48c4104610 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f48b400e9c0 tx=0x7f48b400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48abfff640 1 -- 172.21.15.31:0/1778549472 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b400ccc0 con 0x7f48c4100680 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c9122640 1 -- 172.21.15.31:0/1778549472 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c41a1b20 con 0x7f48c4100680 2024-04-25T01:54:31.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48abfff640 1 -- 172.21.15.31:0/1778549472 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48b40035f0 con 0x7f48c4100680 2024-04-25T01:54:31.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48abfff640 1 -- 172.21.15.31:0/1778549472 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b400ce60 con 0x7f48c4100680 2024-04-25T01:54:31.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.099+0000 7f48c9122640 1 -- 172.21.15.31:0/1778549472 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c41a1ff0 con 0x7f48c4100680 2024-04-25T01:54:31.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.101+0000 7f48c9122640 1 -- 172.21.15.31:0/1778549472 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48c4068560 con 0x7f48c4100680 2024-04-25T01:54:31.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.101+0000 7f48abfff640 1 -- 172.21.15.31:0/1778549472 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 23) ==== 137953+0+0 (secure 0 0 0) 0x7f48b4004030 con 0x7f48c4100680 2024-04-25T01:54:31.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.102+0000 7f48abfff640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48ac0a2d00 0x7f48ac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.102+0000 7f48c259c640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48ac0a2d00 0x7f48ac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.102+0000 7f48abfff640 1 -- 172.21.15.31:0/1778549472 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f48b40cf800 con 0x7f48c4100680 2024-04-25T01:54:31.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.103+0000 7f48c259c640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48ac0a2d00 0x7f48ac0a51c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f48b8006010 tx=0x7f48b8000bc0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.107+0000 7f48abfff640 1 -- 172.21.15.31:0/1778549472 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f48b4004320 con 0x7f48c4100680 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.295+0000 7f48c9122640 1 -- 172.21.15.31:0/1778549472 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48c41a23d0 con 0x7f48c4100680 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.297+0000 7f48abfff640 1 -- 172.21.15.31:0/1778549472 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22855 (secure 0 0 0) 0x7f48b409c410 con 0x7f48c4100680 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:31.299 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:54:28.579231+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3045817976":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6800/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1358173279":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/686937010":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/3913527371":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:54:31.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 -- 172.21.15.31:0/1778549472 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48ac0a2d00 msgr2=0x7f48ac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48ac0a2d00 0x7f48ac0a51c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f48b8006010 tx=0x7f48b8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 -- 172.21.15.31:0/1778549472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48c4100680 msgr2=0x7f48c4104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48c4100680 0x7f48c4104610 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f48b400e9c0 tx=0x7f48b400ee80 comp rx=0 tx=0).stop 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 -- 172.21.15.31:0/1778549472 shutdown_connections 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48ac0a2d00 0x7f48ac0a51c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48c4102230 0x7f48c4103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4101870 0x7f48c4102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 --2- 172.21.15.31:0/1778549472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48c4100680 0x7f48c4104610 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 -- 172.21.15.31:0/1778549472 >> 172.21.15.31:0/1778549472 conn(0x7f48c40fbbf0 msgr2=0x7f48c40fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 -- 172.21.15.31:0/1778549472 shutdown_connections 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.299+0000 7f48c9122640 1 -- 172.21.15.31:0/1778549472 wait complete. 2024-04-25T01:54:31.318 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:23.432566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:24.255373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:54:31.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:54:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.425+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3696777611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e4101870 msgr2=0x7f23e4101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.425+0000 7f23ea7b7640 1 --2- 172.21.15.31:0/3696777611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e4101870 0x7f23e4101cf0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009900 tx=0x7f23d802a410 comp rx=0 tx=0).stop 2024-04-25T01:54:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.425+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3696777611 shutdown_connections 2024-04-25T01:54:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.425+0000 7f23ea7b7640 1 --2- 172.21.15.31:0/3696777611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23e4102230 0x7f23e4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.425+0000 7f23ea7b7640 1 --2- 172.21.15.31:0/3696777611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e4101870 0x7f23e4101cf0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.425+0000 7f23ea7b7640 1 --2- 172.21.15.31:0/3696777611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4100680 0x7f23e4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.425+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3696777611 >> 172.21.15.31:0/3696777611 conn(0x7f23e40fbbf0 msgr2=0x7f23e40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.425+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3696777611 shutdown_connections 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.425+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3696777611 wait complete. 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.425+0000 7f23ea7b7640 1 Processor -- start 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.425+0000 7f23ea7b7640 1 -- start start 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23ea7b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e4100680 0x7f23e4197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23ea7b7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4101870 0x7f23e4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23ea7b7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23e4102230 0x7f23e419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23ea7b7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f23e410c190 con 0x7f23e4100680 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23ea7b7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f23e410c010 con 0x7f23e4101870 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23ea7b7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f23e410c310 con 0x7f23e4102230 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23e3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e4100680 0x7f23e4197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23e37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4101870 0x7f23e4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23e3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e4100680 0x7f23e4197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35258/0 (socket says 172.21.15.31:35258) 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23e3fff640 1 -- 172.21.15.31:0/3861600017 learned_addr learned my addr 172.21.15.31:0/3861600017 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23e8d54640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23e4102230 0x7f23e419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23e3fff640 1 -- 172.21.15.31:0/3861600017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23e4102230 msgr2=0x7f23e419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23e3fff640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23e4102230 0x7f23e419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23e3fff640 1 -- 172.21.15.31:0/3861600017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4101870 msgr2=0x7f23e4197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23e3fff640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4101870 0x7f23e4197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23e3fff640 1 -- 172.21.15.31:0/3861600017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e419f640 con 0x7f23e4100680 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.426+0000 7f23e37fe640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4101870 0x7f23e4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.427+0000 7f23e3fff640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e4100680 0x7f23e4197a70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f23d400e9c0 tx=0x7f23d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.427+0000 7f23e17fa640 1 -- 172.21.15.31:0/3861600017 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d400cd80 con 0x7f23e4100680 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.427+0000 7f23e17fa640 1 -- 172.21.15.31:0/3861600017 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23d40035f0 con 0x7f23e4100680 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.427+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3861600017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e419f930 con 0x7f23e4100680 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.427+0000 7f23e17fa640 1 -- 172.21.15.31:0/3861600017 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4010690 con 0x7f23e4100680 2024-04-25T01:54:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.427+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3861600017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e419fe50 con 0x7f23e4100680 2024-04-25T01:54:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.428+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3861600017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23e41a00e0 con 0x7f23e4100680 2024-04-25T01:54:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.430+0000 7f23e17fa640 1 -- 172.21.15.31:0/3861600017 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 23) ==== 137953+0+0 (secure 0 0 0) 0x7f23d40037a0 con 0x7f23e4100680 2024-04-25T01:54:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.430+0000 7f23e17fa640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f23cc0a2d00 0x7f23cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.430+0000 7f23e37fe640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f23cc0a2d00 0x7f23cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.430+0000 7f23e17fa640 1 -- 172.21.15.31:0/3861600017 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f23d40cf7a0 con 0x7f23e4100680 2024-04-25T01:54:31.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.431+0000 7f23e37fe640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f23cc0a2d00 0x7f23cc0a51c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f23d8006010 tx=0x7f23d8000bc0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.436+0000 7f23e17fa640 1 -- 172.21.15.31:0/3861600017 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f23d409c410 con 0x7f23e4100680 2024-04-25T01:54:31.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.625+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3861600017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f23e41a0350 con 0x7f23e4100680 2024-04-25T01:54:31.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.625+0000 7f23e17fa640 1 -- 172.21.15.31:0/3861600017 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f23d409bd10 con 0x7f23e4100680 2024-04-25T01:54:31.626 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T01:54:31.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.627+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3861600017 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f23cc0a2d00 msgr2=0x7f23cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.627+0000 7f23ea7b7640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f23cc0a2d00 0x7f23cc0a51c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f23d8006010 tx=0x7f23d8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.628+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3861600017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e4100680 msgr2=0x7f23e4197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.628+0000 7f23ea7b7640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e4100680 0x7f23e4197a70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f23d400e9c0 tx=0x7f23d400ee80 comp rx=0 tx=0).stop 2024-04-25T01:54:31.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.628+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3861600017 shutdown_connections 2024-04-25T01:54:31.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.628+0000 7f23ea7b7640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f23cc0a2d00 0x7f23cc0a51c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.628+0000 7f23ea7b7640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23e4102230 0x7f23e419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.628+0000 7f23ea7b7640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23e4101870 0x7f23e4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.628+0000 7f23ea7b7640 1 --2- 172.21.15.31:0/3861600017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23e4100680 0x7f23e4197a70 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.628+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3861600017 >> 172.21.15.31:0/3861600017 conn(0x7f23e40fbbf0 msgr2=0x7f23e40688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.628+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3861600017 shutdown_connections 2024-04-25T01:54:31.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.628+0000 7f23ea7b7640 1 -- 172.21.15.31:0/3861600017 wait complete. 2024-04-25T01:54:31.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:54:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.746+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/3625148297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e64100680 msgr2=0x7f4e64100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.746+0000 7f4e6ac4d640 1 --2- 172.21.15.31:0/3625148297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e64100680 0x7f4e64100aa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009900 tx=0x7f4e5802a410 comp rx=0 tx=0).stop 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.746+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/3625148297 shutdown_connections 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.746+0000 7f4e6ac4d640 1 --2- 172.21.15.31:0/3625148297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e64102230 0x7f4e64108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.746+0000 7f4e6ac4d640 1 --2- 172.21.15.31:0/3625148297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e64101870 0x7f4e64101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.746+0000 7f4e6ac4d640 1 --2- 172.21.15.31:0/3625148297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e64100680 0x7f4e64100aa0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.746+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/3625148297 >> 172.21.15.31:0/3625148297 conn(0x7f4e640fbbf0 msgr2=0x7f4e640fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.746+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/3625148297 shutdown_connections 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.746+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/3625148297 wait complete. 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e6ac4d640 1 Processor -- start 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e6ac4d640 1 -- start start 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e6ac4d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e64100680 0x7f4e64104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e6ac4d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e64101870 0x7f4e64102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e6ac4d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e64102230 0x7f4e64103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e6ac4d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6410c180 con 0x7f4e64101870 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e6ac4d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6410c000 con 0x7f4e64100680 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e6ac4d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6410c300 con 0x7f4e64102230 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e63fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e64101870 0x7f4e64102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e689e9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e64100680 0x7f4e64104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e63fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e64101870 0x7f4e64102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35280/0 (socket says 172.21.15.31:35280) 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e63fff640 1 -- 172.21.15.31:0/4159761118 learned_addr learned my addr 172.21.15.31:0/4159761118 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e691ea640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e64102230 0x7f4e64103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e689e9640 1 -- 172.21.15.31:0/4159761118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e64102230 msgr2=0x7f4e64103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e689e9640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e64102230 0x7f4e64103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e689e9640 1 -- 172.21.15.31:0/4159761118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e64101870 msgr2=0x7f4e64102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e689e9640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e64101870 0x7f4e64102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e689e9640 1 -- 172.21.15.31:0/4159761118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e64103a20 con 0x7f4e64100680 2024-04-25T01:54:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.747+0000 7f4e63fff640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e64101870 0x7f4e64102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.748+0000 7f4e689e9640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e64100680 0x7f4e64104610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009230 tx=0x7f4e58004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.748+0000 7f4e61ffb640 1 -- 172.21.15.31:0/4159761118 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5803f030 con 0x7f4e64100680 2024-04-25T01:54:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.748+0000 7f4e61ffb640 1 -- 172.21.15.31:0/4159761118 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e58003aa0 con 0x7f4e64100680 2024-04-25T01:54:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.748+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/4159761118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e641a1af0 con 0x7f4e64100680 2024-04-25T01:54:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.748+0000 7f4e61ffb640 1 -- 172.21.15.31:0/4159761118 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e58035070 con 0x7f4e64100680 2024-04-25T01:54:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.748+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/4159761118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e641a2010 con 0x7f4e64100680 2024-04-25T01:54:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.749+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/4159761118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e641a21e0 con 0x7f4e64100680 2024-04-25T01:54:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.750+0000 7f4e61ffb640 1 -- 172.21.15.31:0/4159761118 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 23) ==== 137953+0+0 (secure 0 0 0) 0x7f4e58042030 con 0x7f4e64100680 2024-04-25T01:54:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.751+0000 7f4e61ffb640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4e540a2d00 0x7f4e540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.751+0000 7f4e61ffb640 1 -- 172.21.15.31:0/4159761118 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4e580efa00 con 0x7f4e64100680 2024-04-25T01:54:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.751+0000 7f4e63fff640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4e540a2d00 0x7f4e540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.751+0000 7f4e63fff640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4e540a2d00 0x7f4e540a51c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4e500095f0 tx=0x7f4e50005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.754+0000 7f4e61ffb640 1 -- 172.21.15.31:0/4159761118 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4e580bbfc0 con 0x7f4e64100680 2024-04-25T01:54:31.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:31 smithi040 ceph-mon[32418]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:31.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:31 smithi040 ceph-mon[32418]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T01:54:31.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:31 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2992928543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:31.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:31 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:31 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:31 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:31 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:31 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:31 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:31 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3172648553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:31.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:31 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1778549472' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:31.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:31 smithi134 ceph-mon[33098]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:31.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:31 smithi134 ceph-mon[33098]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T01:54:31.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:31 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2992928543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:31.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:31 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:31 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:31 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:31 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:31 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:31 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:31 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3172648553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:31.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:31 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1778549472' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:31.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.945+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/4159761118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4e641a23b0 con 0x7f4e64100680 2024-04-25T01:54:31.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.945+0000 7f4e61ffb640 1 -- 172.21.15.31:0/4159761118 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f4e580c02c0 con 0x7f4e64100680 2024-04-25T01:54:31.947 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T01:54:31.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/4159761118 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4e540a2d00 msgr2=0x7f4e540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4e540a2d00 0x7f4e540a51c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4e500095f0 tx=0x7f4e50005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:31.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/4159761118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e64100680 msgr2=0x7f4e64104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e64100680 0x7f4e64104610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009230 tx=0x7f4e58004270 comp rx=0 tx=0).stop 2024-04-25T01:54:31.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/4159761118 shutdown_connections 2024-04-25T01:54:31.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4e540a2d00 0x7f4e540a51c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e64102230 0x7f4e64103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e64101870 0x7f4e64102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 --2- 172.21.15.31:0/4159761118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e64100680 0x7f4e64104610 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/4159761118 >> 172.21.15.31:0/4159761118 conn(0x7f4e640fbbf0 msgr2=0x7f4e64068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/4159761118 shutdown_connections 2024-04-25T01:54:31.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:31.947+0000 7f4e6ac4d640 1 -- 172.21.15.31:0/4159761118 wait complete. 2024-04-25T01:54:31.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:31 smithi031 ceph-mon[29204]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:31.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:31 smithi031 ceph-mon[29204]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T01:54:31.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:31 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2992928543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:31.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:31 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:31 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:31 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:31 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:31 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:31 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:31.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:31 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3172648553' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:31.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:31 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1778549472' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:31.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:54:32.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.078+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/480052054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1a4109620 msgr2=0x7fb1a410bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.078+0000 7fb1a9ae1640 1 --2- 172.21.15.31:0/480052054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1a4109620 0x7fb1a410bac0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb19c009ca0 tx=0x7fb19c02a950 comp rx=0 tx=0).stop 2024-04-25T01:54:32.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.078+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/480052054 shutdown_connections 2024-04-25T01:54:32.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.078+0000 7fb1a9ae1640 1 --2- 172.21.15.31:0/480052054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1a4109620 0x7fb1a410bac0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.078+0000 7fb1a9ae1640 1 --2- 172.21.15.31:0/480052054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1a4074140 0x7fb1a40745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.078+0000 7fb1a9ae1640 1 --2- 172.21.15.31:0/480052054 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4075720 0x7fb1a4073b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.078+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/480052054 >> 172.21.15.31:0/480052054 conn(0x7fb1a40fcd20 msgr2=0x7fb1a40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:32.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.078+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/480052054 shutdown_connections 2024-04-25T01:54:32.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.078+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/480052054 wait complete. 2024-04-25T01:54:32.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a9ae1640 1 Processor -- start 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a9ae1640 1 -- start start 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a9ae1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4074140 0x7fb1a4199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a9ae1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1a4075720 0x7fb1a419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a9ae1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1a4109620 0x7fb1a41a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a9ae1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a410f4c0 con 0x7fb1a4075720 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a9ae1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a410f340 con 0x7fb1a4109620 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a9ae1640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a410f640 con 0x7fb1a4074140 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1a4075720 0x7fb1a419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1a4075720 0x7fb1a419a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35296/0 (socket says 172.21.15.31:35296) 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a2ffd640 1 -- 172.21.15.31:0/1382593427 learned_addr learned my addr 172.21.15.31:0/1382593427 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a37fe640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4074140 0x7fb1a4199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a3fff640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1a4109620 0x7fb1a41a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a2ffd640 1 -- 172.21.15.31:0/1382593427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4074140 msgr2=0x7fb1a4199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a2ffd640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4074140 0x7fb1a4199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a2ffd640 1 -- 172.21.15.31:0/1382593427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1a4109620 msgr2=0x7fb1a41a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.079+0000 7fb1a2ffd640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1a4109620 0x7fb1a41a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.080+0000 7fb1a2ffd640 1 -- 172.21.15.31:0/1382593427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a41a17a0 con 0x7fb1a4075720 2024-04-25T01:54:32.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.080+0000 7fb1a37fe640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4074140 0x7fb1a4199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:32.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.080+0000 7fb1a2ffd640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1a4075720 0x7fb1a419a0c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb19800d950 tx=0x7fb19800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:32.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.080+0000 7fb1a0ff9640 1 -- 172.21.15.31:0/1382593427 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb198017070 con 0x7fb1a4075720 2024-04-25T01:54:32.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.080+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/1382593427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a41a1a90 con 0x7fb1a4075720 2024-04-25T01:54:32.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.080+0000 7fb1a0ff9640 1 -- 172.21.15.31:0/1382593427 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb198009c30 con 0x7fb1a4075720 2024-04-25T01:54:32.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.080+0000 7fb1a0ff9640 1 -- 172.21.15.31:0/1382593427 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19800f620 con 0x7fb1a4075720 2024-04-25T01:54:32.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.080+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/1382593427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a41a1fe0 con 0x7fb1a4075720 2024-04-25T01:54:32.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.081+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/1382593427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1a41083e0 con 0x7fb1a4075720 2024-04-25T01:54:32.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.082+0000 7fb1a0ff9640 1 -- 172.21.15.31:0/1382593427 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 23) ==== 137953+0+0 (secure 0 0 0) 0x7fb1980109a0 con 0x7fb1a4075720 2024-04-25T01:54:32.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.083+0000 7fb1a0ff9640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb18c0a2d00 0x7fb18c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.083+0000 7fb1a37fe640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb18c0a2d00 0x7fb18c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.083+0000 7fb1a0ff9640 1 -- 172.21.15.31:0/1382593427 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fb1980cf050 con 0x7fb1a4075720 2024-04-25T01:54:32.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.083+0000 7fb1a37fe640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb18c0a2d00 0x7fb18c0a51c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb194009fc0 tx=0x7fb194005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:32.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.089+0000 7fb1a0ff9640 1 -- 172.21.15.31:0/1382593427 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb198009de0 con 0x7fb1a4075720 2024-04-25T01:54:32.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.287+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/1382593427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb1a402d390 con 0x7fb1a4075720 2024-04-25T01:54:32.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.288+0000 7fb1a0ff9640 1 -- 172.21.15.31:0/1382593427 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fb19809bcc0 con 0x7fb1a4075720 2024-04-25T01:54:32.289 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T01:54:32.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/1382593427 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb18c0a2d00 msgr2=0x7fb18c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb18c0a2d00 0x7fb18c0a51c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb194009fc0 tx=0x7fb194005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:32.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/1382593427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1a4075720 msgr2=0x7fb1a419a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1a4075720 0x7fb1a419a0c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb19800d950 tx=0x7fb19800de10 comp rx=0 tx=0).stop 2024-04-25T01:54:32.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/1382593427 shutdown_connections 2024-04-25T01:54:32.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb18c0a2d00 0x7fb18c0a51c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1a4109620 0x7fb1a41a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1a4075720 0x7fb1a419a0c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 --2- 172.21.15.31:0/1382593427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a4074140 0x7fb1a4199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/1382593427 >> 172.21.15.31:0/1382593427 conn(0x7fb1a40fcd20 msgr2=0x7fb1a41055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:32.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/1382593427 shutdown_connections 2024-04-25T01:54:32.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.290+0000 7fb1a9ae1640 1 -- 172.21.15.31:0/1382593427 wait complete. 2024-04-25T01:54:32.303 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-25T01:54:32.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:54:32.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.428+0000 7f202d600640 1 -- 172.21.15.31:0/3858811324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20281015f0 msgr2=0x7f2028101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.428+0000 7f202d600640 1 --2- 172.21.15.31:0/3858811324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20281015f0 0x7f2028101a10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2018009ca0 tx=0x7f201802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.429+0000 7f202d600640 1 -- 172.21.15.31:0/3858811324 shutdown_connections 2024-04-25T01:54:32.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.429+0000 7f202d600640 1 --2- 172.21.15.31:0/3858811324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20281031a0 0x7f2028109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.429+0000 7f202d600640 1 --2- 172.21.15.31:0/3858811324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20281027e0 0x7f2028102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.429+0000 7f202d600640 1 --2- 172.21.15.31:0/3858811324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20281015f0 0x7f2028101a10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.429+0000 7f202d600640 1 -- 172.21.15.31:0/3858811324 >> 172.21.15.31:0/3858811324 conn(0x7f20280fcb80 msgr2=0x7f20280fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:32.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.429+0000 7f202d600640 1 -- 172.21.15.31:0/3858811324 shutdown_connections 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.429+0000 7f202d600640 1 -- 172.21.15.31:0/3858811324 wait complete. 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.429+0000 7f202d600640 1 Processor -- start 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f202d600640 1 -- start start 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f202d600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20281015f0 0x7f20281978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f202d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20281027e0 0x7f2028197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f202d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20281031a0 0x7f202819ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f202d600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f202810d170 con 0x7f20281031a0 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f202d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f202810cff0 con 0x7f20281027e0 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f202d600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f202810d2f0 con 0x7f20281015f0 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f20277fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20281031a0 0x7f202819ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f20267fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20281027e0 0x7f2028197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f20267fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20281027e0 0x7f2028197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:57806/0 (socket says 172.21.15.31:57806) 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f20267fc640 1 -- 172.21.15.31:0/1398454445 learned_addr learned my addr 172.21.15.31:0/1398454445 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:32.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f20277fe640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20281031a0 0x7f202819ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35320/0 (socket says 172.21.15.31:35320) 2024-04-25T01:54:32.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f2026ffd640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20281015f0 0x7f20281978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f20267fc640 1 -- 172.21.15.31:0/1398454445 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20281015f0 msgr2=0x7f20281978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f20267fc640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20281015f0 0x7f20281978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f20267fc640 1 -- 172.21.15.31:0/1398454445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20281031a0 msgr2=0x7f202819ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f20267fc640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20281031a0 0x7f202819ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.430+0000 7f20267fc640 1 -- 172.21.15.31:0/1398454445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202819f4e0 con 0x7f20281027e0 2024-04-25T01:54:32.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.431+0000 7f20267fc640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20281027e0 0x7f2028197e00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e9f0 tx=0x7f201c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:32.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.431+0000 7f200ffff640 1 -- 172.21.15.31:0/1398454445 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c00ccc0 con 0x7f20281027e0 2024-04-25T01:54:32.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.431+0000 7f202d600640 1 -- 172.21.15.31:0/1398454445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202819f7d0 con 0x7f20281027e0 2024-04-25T01:54:32.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.431+0000 7f200ffff640 1 -- 172.21.15.31:0/1398454445 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f201c00ce60 con 0x7f20281027e0 2024-04-25T01:54:32.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.431+0000 7f200ffff640 1 -- 172.21.15.31:0/1398454445 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c010430 con 0x7f20281027e0 2024-04-25T01:54:32.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.431+0000 7f202d600640 1 -- 172.21.15.31:0/1398454445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f202819fcd0 con 0x7f20281027e0 2024-04-25T01:54:32.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.432+0000 7f202d600640 1 -- 172.21.15.31:0/1398454445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2028108a00 con 0x7f20281027e0 2024-04-25T01:54:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.433+0000 7f200ffff640 1 -- 172.21.15.31:0/1398454445 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 23) ==== 137953+0+0 (secure 0 0 0) 0x7f201c005020 con 0x7f20281027e0 2024-04-25T01:54:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.434+0000 7f200ffff640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f20100a2d00 0x7f20100a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.434+0000 7f2026ffd640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f20100a2d00 0x7f20100a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.434+0000 7f200ffff640 1 -- 172.21.15.31:0/1398454445 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f201c0cf840 con 0x7f20281027e0 2024-04-25T01:54:32.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.434+0000 7f2026ffd640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f20100a2d00 0x7f20100a51c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2018000f80 tx=0x7f2018031000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:32.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.438+0000 7f200ffff640 1 -- 172.21.15.31:0/1398454445 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f201c09c450 con 0x7f20281027e0 2024-04-25T01:54:32.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.673+0000 7f202d600640 1 -- 172.21.15.31:0/1398454445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f202802d170 con 0x7f20281027e0 2024-04-25T01:54:32.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.674+0000 7f200ffff640 1 -- 172.21.15.31:0/1398454445 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11923 (secure 0 0 0) 0x7f201c09bd50 con 0x7f20281027e0 2024-04-25T01:54:32.675 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:32.675 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:54:28.592746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24451,24463,24503,24487,24473]},"id":1}]} 2024-04-25T01:54:32.676 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-04-25T01:54:32.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 -- 172.21.15.31:0/1398454445 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f20100a2d00 msgr2=0x7f20100a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f20100a2d00 0x7f20100a51c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2018000f80 tx=0x7f2018031000 comp rx=0 tx=0).stop 2024-04-25T01:54:32.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 -- 172.21.15.31:0/1398454445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20281027e0 msgr2=0x7f2028197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20281027e0 0x7f2028197e00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e9f0 tx=0x7f201c00edf0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 -- 172.21.15.31:0/1398454445 shutdown_connections 2024-04-25T01:54:32.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f20100a2d00 0x7f20100a51c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20281031a0 0x7f202819ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20281027e0 0x7f2028197e00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 --2- 172.21.15.31:0/1398454445 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20281015f0 0x7f20281978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 -- 172.21.15.31:0/1398454445 >> 172.21.15.31:0/1398454445 conn(0x7f20280fcb80 msgr2=0x7f2028105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:32.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 -- 172.21.15.31:0/1398454445 shutdown_connections 2024-04-25T01:54:32.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.676+0000 7f202d600640 1 -- 172.21.15.31:0/1398454445 wait complete. 2024-04-25T01:54:32.689 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-25T01:54:32.689 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-25T01:54:32.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:54:32.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.794+0000 7f2687d09640 1 -- 172.21.15.31:0/2900008002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680101870 msgr2=0x7f2680101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.794+0000 7f2687d09640 1 --2- 172.21.15.31:0/2900008002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680101870 0x7f2680101cf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2670009900 tx=0x7f267002a410 comp rx=0 tx=0).stop 2024-04-25T01:54:32.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.794+0000 7f2687d09640 1 -- 172.21.15.31:0/2900008002 shutdown_connections 2024-04-25T01:54:32.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.794+0000 7f2687d09640 1 --2- 172.21.15.31:0/2900008002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2680102230 0x7f2680108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.794+0000 7f2687d09640 1 --2- 172.21.15.31:0/2900008002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680101870 0x7f2680101cf0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.794+0000 7f2687d09640 1 --2- 172.21.15.31:0/2900008002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2680100680 0x7f2680100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.794+0000 7f2687d09640 1 -- 172.21.15.31:0/2900008002 >> 172.21.15.31:0/2900008002 conn(0x7f26800fbbf0 msgr2=0x7f26800fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:32.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.794+0000 7f2687d09640 1 -- 172.21.15.31:0/2900008002 shutdown_connections 2024-04-25T01:54:32.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.794+0000 7f2687d09640 1 -- 172.21.15.31:0/2900008002 wait complete. 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.794+0000 7f2687d09640 1 Processor -- start 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.794+0000 7f2687d09640 1 -- start start 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f2687d09640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2680100680 0x7f26801968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f2687d09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680101870 0x7f2680196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f2687d09640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2680102230 0x7f268019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f2687d09640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f268010c1b0 con 0x7f2680101870 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f2687d09640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f268010c030 con 0x7f2680100680 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f2687d09640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f268010c330 con 0x7f2680102230 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f26852a4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680101870 0x7f2680196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f26852a4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680101870 0x7f2680196e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38662/0 (socket says 172.21.15.31:38662) 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f26852a4640 1 -- 172.21.15.31:0/3085046261 learned_addr learned my addr 172.21.15.31:0/3085046261 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:32.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f26862a6640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2680102230 0x7f268019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f2685aa5640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2680100680 0x7f26801968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f26852a4640 1 -- 172.21.15.31:0/3085046261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2680102230 msgr2=0x7f268019de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f26852a4640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2680102230 0x7f268019de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f26852a4640 1 -- 172.21.15.31:0/3085046261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2680100680 msgr2=0x7f26801968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f26852a4640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2680100680 0x7f26801968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f26852a4640 1 -- 172.21.15.31:0/3085046261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268019e500 con 0x7f2680101870 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.795+0000 7f26852a4640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680101870 0x7f2680196e20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2670005940 tx=0x7f26700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.796+0000 7f2676ffd640 1 -- 172.21.15.31:0/3085046261 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267003f030 con 0x7f2680101870 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.796+0000 7f2676ffd640 1 -- 172.21.15.31:0/3085046261 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26700318f0 con 0x7f2680101870 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.796+0000 7f2687d09640 1 -- 172.21.15.31:0/3085046261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268019e790 con 0x7f2680101870 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.796+0000 7f2676ffd640 1 -- 172.21.15.31:0/3085046261 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670035070 con 0x7f2680101870 2024-04-25T01:54:32.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.796+0000 7f2687d09640 1 -- 172.21.15.31:0/3085046261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268019ec80 con 0x7f2680101870 2024-04-25T01:54:32.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.797+0000 7f2687d09640 1 -- 172.21.15.31:0/3085046261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2680107a20 con 0x7f2680101870 2024-04-25T01:54:32.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.798+0000 7f2676ffd640 1 -- 172.21.15.31:0/3085046261 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 23) ==== 137953+0+0 (secure 0 0 0) 0x7f2670042030 con 0x7f2680101870 2024-04-25T01:54:32.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.799+0000 7f2676ffd640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f266c0a2d00 0x7f266c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:32.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.799+0000 7f2685aa5640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f266c0a2d00 0x7f266c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:32.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.799+0000 7f2676ffd640 1 -- 172.21.15.31:0/3085046261 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f26700ef950 con 0x7f2680101870 2024-04-25T01:54:32.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.800+0000 7f2685aa5640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f266c0a2d00 0x7f266c0a51c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f26801016d0 tx=0x7f267c005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:32.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:32.804+0000 7f2676ffd640 1 -- 172.21.15.31:0/3085046261 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f26700bc5c0 con 0x7f2680101870 2024-04-25T01:54:32.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3861600017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:32.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi134 to 176.2M 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: Unable to set osd_memory_target on smithi134 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: Unable to set osd_memory_target on smithi040 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/4159761118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T01:54:32.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:54:32.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:54:32.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:54:32.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-25T01:54:32.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-25T01:54:32.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-25T01:54:32.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:32 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1382593427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:32.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3861600017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:32.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi134 to 176.2M 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: Unable to set osd_memory_target on smithi134 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: Unable to set osd_memory_target on smithi040 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/4159761118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-25T01:54:32.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-25T01:54:32.942 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:32 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1382593427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:32.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3861600017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:32.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi134 to 176.2M 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: Unable to set osd_memory_target on smithi134 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: Unable to set osd_memory_target on smithi040 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/4159761118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-25T01:54:32.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:32 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1382593427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:33.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.015+0000 7f2687d09640 1 -- 172.21.15.31:0/3085046261 --> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2680061b80 con 0x7f266c0a2d00 2024-04-25T01:54:33.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.023+0000 7f2676ffd640 1 -- 172.21.15.31:0/3085046261 <== mgr.15351 v2:172.21.15.31:6800/1834919992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258073 (secure 0 0 0) 0x7f2680061b80 con 0x7f266c0a2d00 2024-04-25T01:54:33.027 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:33.028 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T01:54:33.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 -- 172.21.15.31:0/3085046261 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f266c0a2d00 msgr2=0x7f266c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f266c0a2d00 0x7f266c0a51c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f26801016d0 tx=0x7f267c005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:33.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 -- 172.21.15.31:0/3085046261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680101870 msgr2=0x7f2680196e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680101870 0x7f2680196e20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2670005940 tx=0x7f26700042f0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 -- 172.21.15.31:0/3085046261 shutdown_connections 2024-04-25T01:54:33.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f266c0a2d00 0x7f266c0a51c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2680102230 0x7f268019de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680101870 0x7f2680196e20 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 --2- 172.21.15.31:0/3085046261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2680100680 0x7f26801968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 -- 172.21.15.31:0/3085046261 >> 172.21.15.31:0/3085046261 conn(0x7f26800fbbf0 msgr2=0x7f2680104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:33.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 -- 172.21.15.31:0/3085046261 shutdown_connections 2024-04-25T01:54:33.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.026+0000 7f2687d09640 1 -- 172.21.15.31:0/3085046261 wait complete. 2024-04-25T01:54:33.039 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T01:54:32.598067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193324,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1439,"num_read_kb":888,"num_write":560,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1241782,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":945,"ondisk_log_size":945,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373876,"kb_used_data":12216,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567948,"statfs":{"total":1151940427776,"available":1151557578752,"internally_reserved":0,"allocated":12509184,"data_stored":9261821,"data_compressed":225161,"data_compressed_allocated":294912,"data_compressed_original":3473408,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.984749"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845657+0000","last_change":"2024-04-25T01:52:25.523199+0000","last_active":"2024-04-25T01:54:28.845657+0000","last_peered":"2024-04-25T01:54:28.845657+0000","last_clean":"2024-04-25T01:54:28.845657+0000","last_became_active":"2024-04-25T01:52:25.522052+0000","last_became_peered":"2024-04-25T01:52:25.522052+0000","last_unstale":"2024-04-25T01:54:28.845657+0000","last_undegraded":"2024-04-25T01:54:28.845657+0000","last_fullsized":"2024-04-25T01:54:28.845657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:07:27.095370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846530+0000","last_change":"2024-04-25T01:52:25.528431+0000","last_active":"2024-04-25T01:54:28.846530+0000","last_peered":"2024-04-25T01:54:28.846530+0000","last_clean":"2024-04-25T01:54:28.846530+0000","last_became_active":"2024-04-25T01:52:25.528122+0000","last_became_peered":"2024-04-25T01:52:25.528122+0000","last_unstale":"2024-04-25T01:54:28.846530+0000","last_undegraded":"2024-04-25T01:54:28.846530+0000","last_fullsized":"2024-04-25T01:54:28.846530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:26:56.794582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846475+0000","last_change":"2024-04-25T01:52:25.532963+0000","last_active":"2024-04-25T01:54:28.846475+0000","last_peered":"2024-04-25T01:54:28.846475+0000","last_clean":"2024-04-25T01:54:28.846475+0000","last_became_active":"2024-04-25T01:52:25.532770+0000","last_became_peered":"2024-04-25T01:52:25.532770+0000","last_unstale":"2024-04-25T01:54:28.846475+0000","last_undegraded":"2024-04-25T01:54:28.846475+0000","last_fullsized":"2024-04-25T01:54:28.846475+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:13.398868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585908+0000","last_change":"2024-04-25T01:53:24.968017+0000","last_active":"2024-04-25T01:54:28.585908+0000","last_peered":"2024-04-25T01:54:28.585908+0000","last_clean":"2024-04-25T01:54:28.585908+0000","last_became_active":"2024-04-25T01:53:24.967740+0000","last_became_peered":"2024-04-25T01:53:24.967740+0000","last_unstale":"2024-04-25T01:54:28.585908+0000","last_undegraded":"2024-04-25T01:54:28.585908+0000","last_fullsized":"2024-04-25T01:54:28.585908+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:45.219797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676410+0000","last_change":"2024-04-25T01:52:25.523624+0000","last_active":"2024-04-25T01:54:28.676410+0000","last_peered":"2024-04-25T01:54:28.676410+0000","last_clean":"2024-04-25T01:54:28.676410+0000","last_became_active":"2024-04-25T01:52:25.523289+0000","last_became_peered":"2024-04-25T01:52:25.523289+0000","last_unstale":"2024-04-25T01:54:28.676410+0000","last_undegraded":"2024-04-25T01:54:28.676410+0000","last_fullsized":"2024-04-25T01:54:28.676410+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:09.688472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685245+0000","last_change":"2024-04-25T01:52:25.527123+0000","last_active":"2024-04-25T01:54:28.685245+0000","last_peered":"2024-04-25T01:54:28.685245+0000","last_clean":"2024-04-25T01:54:28.685245+0000","last_became_active":"2024-04-25T01:52:25.526808+0000","last_became_peered":"2024-04-25T01:52:25.526808+0000","last_unstale":"2024-04-25T01:54:28.685245+0000","last_undegraded":"2024-04-25T01:54:28.685245+0000","last_fullsized":"2024-04-25T01:54:28.685245+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:31.654857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837586+0000","last_change":"2024-04-25T01:52:25.517011+0000","last_active":"2024-04-25T01:54:28.837586+0000","last_peered":"2024-04-25T01:54:28.837586+0000","last_clean":"2024-04-25T01:54:28.837586+0000","last_became_active":"2024-04-25T01:52:25.516680+0000","last_became_peered":"2024-04-25T01:52:25.516680+0000","last_unstale":"2024-04-25T01:54:28.837586+0000","last_undegraded":"2024-04-25T01:54:28.837586+0000","last_fullsized":"2024-04-25T01:54:28.837586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:03.388576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676583+0000","last_change":"2024-04-25T01:52:25.531079+0000","last_active":"2024-04-25T01:54:28.676583+0000","last_peered":"2024-04-25T01:54:28.676583+0000","last_clean":"2024-04-25T01:54:28.676583+0000","last_became_active":"2024-04-25T01:52:25.529200+0000","last_became_peered":"2024-04-25T01:52:25.529200+0000","last_unstale":"2024-04-25T01:54:28.676583+0000","last_undegraded":"2024-04-25T01:54:28.676583+0000","last_fullsized":"2024-04-25T01:54:28.676583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:30.484010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.698038+0000","last_change":"2024-04-25T01:52:25.533421+0000","last_active":"2024-04-25T01:54:28.698038+0000","last_peered":"2024-04-25T01:54:28.698038+0000","last_clean":"2024-04-25T01:54:28.698038+0000","last_became_active":"2024-04-25T01:52:25.531781+0000","last_became_peered":"2024-04-25T01:52:25.531781+0000","last_unstale":"2024-04-25T01:54:28.698038+0000","last_undegraded":"2024-04-25T01:54:28.698038+0000","last_fullsized":"2024-04-25T01:54:28.698038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:32.505793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685423+0000","last_change":"2024-04-25T01:52:25.530223+0000","last_active":"2024-04-25T01:54:28.685423+0000","last_peered":"2024-04-25T01:54:28.685423+0000","last_clean":"2024-04-25T01:54:28.685423+0000","last_became_active":"2024-04-25T01:52:25.529909+0000","last_became_peered":"2024-04-25T01:52:25.529909+0000","last_unstale":"2024-04-25T01:54:28.685423+0000","last_undegraded":"2024-04-25T01:54:28.685423+0000","last_fullsized":"2024-04-25T01:54:28.685423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:07.429026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838182+0000","last_change":"2024-04-25T01:52:25.524502+0000","last_active":"2024-04-25T01:54:28.838182+0000","last_peered":"2024-04-25T01:54:28.838182+0000","last_clean":"2024-04-25T01:54:28.838182+0000","last_became_active":"2024-04-25T01:52:25.524018+0000","last_became_peered":"2024-04-25T01:52:25.524018+0000","last_unstale":"2024-04-25T01:54:28.838182+0000","last_undegraded":"2024-04-25T01:54:28.838182+0000","last_fullsized":"2024-04-25T01:54:28.838182+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:23.760389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.587839+0000","last_change":"2024-04-25T01:52:25.533593+0000","last_active":"2024-04-25T01:54:28.587839+0000","last_peered":"2024-04-25T01:54:28.587839+0000","last_clean":"2024-04-25T01:54:28.587839+0000","last_became_active":"2024-04-25T01:52:25.532745+0000","last_became_peered":"2024-04-25T01:52:25.532745+0000","last_unstale":"2024-04-25T01:54:28.587839+0000","last_undegraded":"2024-04-25T01:54:28.587839+0000","last_fullsized":"2024-04-25T01:54:28.587839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:45.083267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908390+0000","last_change":"2024-04-25T01:52:25.528563+0000","last_active":"2024-04-25T01:54:28.908390+0000","last_peered":"2024-04-25T01:54:28.908390+0000","last_clean":"2024-04-25T01:54:28.908390+0000","last_became_active":"2024-04-25T01:52:25.528236+0000","last_became_peered":"2024-04-25T01:52:25.528236+0000","last_unstale":"2024-04-25T01:54:28.908390+0000","last_undegraded":"2024-04-25T01:54:28.908390+0000","last_fullsized":"2024-04-25T01:54:28.908390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:28:27.222799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845935+0000","last_change":"2024-04-25T01:52:25.524951+0000","last_active":"2024-04-25T01:54:28.845935+0000","last_peered":"2024-04-25T01:54:28.845935+0000","last_clean":"2024-04-25T01:54:28.845935+0000","last_became_active":"2024-04-25T01:52:25.524617+0000","last_became_peered":"2024-04-25T01:52:25.524617+0000","last_unstale":"2024-04-25T01:54:28.845935+0000","last_undegraded":"2024-04-25T01:54:28.845935+0000","last_fullsized":"2024-04-25T01:54:28.845935+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:03.561883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845850+0000","last_change":"2024-04-25T01:52:25.532471+0000","last_active":"2024-04-25T01:54:28.845850+0000","last_peered":"2024-04-25T01:54:28.845850+0000","last_clean":"2024-04-25T01:54:28.845850+0000","last_became_active":"2024-04-25T01:52:25.532131+0000","last_became_peered":"2024-04-25T01:52:25.532131+0000","last_unstale":"2024-04-25T01:54:28.845850+0000","last_undegraded":"2024-04-25T01:54:28.845850+0000","last_fullsized":"2024-04-25T01:54:28.845850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:03.478816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697542+0000","last_change":"2024-04-25T01:52:25.529160+0000","last_active":"2024-04-25T01:54:28.697542+0000","last_peered":"2024-04-25T01:54:28.697542+0000","last_clean":"2024-04-25T01:54:28.697542+0000","last_became_active":"2024-04-25T01:52:25.528854+0000","last_became_peered":"2024-04-25T01:52:25.528854+0000","last_unstale":"2024-04-25T01:54:28.697542+0000","last_undegraded":"2024-04-25T01:54:28.697542+0000","last_fullsized":"2024-04-25T01:54:28.697542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:54.713715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845791+0000","last_change":"2024-04-25T01:52:25.523372+0000","last_active":"2024-04-25T01:54:28.845791+0000","last_peered":"2024-04-25T01:54:28.845791+0000","last_clean":"2024-04-25T01:54:28.845791+0000","last_became_active":"2024-04-25T01:52:25.522538+0000","last_became_peered":"2024-04-25T01:52:25.522538+0000","last_unstale":"2024-04-25T01:54:28.845791+0000","last_undegraded":"2024-04-25T01:54:28.845791+0000","last_fullsized":"2024-04-25T01:54:28.845791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:47.213815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585865+0000","last_change":"2024-04-25T01:52:25.522920+0000","last_active":"2024-04-25T01:54:28.585865+0000","last_peered":"2024-04-25T01:54:28.585865+0000","last_clean":"2024-04-25T01:54:28.585865+0000","last_became_active":"2024-04-25T01:52:25.522594+0000","last_became_peered":"2024-04-25T01:52:25.522594+0000","last_unstale":"2024-04-25T01:54:28.585865+0000","last_undegraded":"2024-04-25T01:54:28.585865+0000","last_fullsized":"2024-04-25T01:54:28.585865+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:45.043667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837469+0000","last_change":"2024-04-25T01:52:25.532738+0000","last_active":"2024-04-25T01:54:28.837469+0000","last_peered":"2024-04-25T01:54:28.837469+0000","last_clean":"2024-04-25T01:54:28.837469+0000","last_became_active":"2024-04-25T01:52:25.532237+0000","last_became_peered":"2024-04-25T01:52:25.532237+0000","last_unstale":"2024-04-25T01:54:28.837469+0000","last_undegraded":"2024-04-25T01:54:28.837469+0000","last_fullsized":"2024-04-25T01:54:28.837469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:54.456704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'14","reported_seq":154,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.935594+0000","last_change":"2024-04-25T01:52:24.515686+0000","last_active":"2024-04-25T01:54:28.935594+0000","last_peered":"2024-04-25T01:54:28.935594+0000","last_clean":"2024-04-25T01:54:28.935594+0000","last_became_active":"2024-04-25T01:52:24.515246+0000","last_became_peered":"2024-04-25T01:52:24.515246+0000","last_unstale":"2024-04-25T01:54:28.935594+0000","last_undegraded":"2024-04-25T01:54:28.935594+0000","last_fullsized":"2024-04-25T01:54:28.935594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:14.325747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677009+0000","last_change":"2024-04-25T01:52:25.528796+0000","last_active":"2024-04-25T01:54:28.677009+0000","last_peered":"2024-04-25T01:54:28.677009+0000","last_clean":"2024-04-25T01:54:28.677009+0000","last_became_active":"2024-04-25T01:52:25.528453+0000","last_became_peered":"2024-04-25T01:52:25.528453+0000","last_unstale":"2024-04-25T01:54:28.677009+0000","last_undegraded":"2024-04-25T01:54:28.677009+0000","last_fullsized":"2024-04-25T01:54:28.677009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:29.432519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677315+0000","last_change":"2024-04-25T01:52:24.520229+0000","last_active":"2024-04-25T01:54:28.677315+0000","last_peered":"2024-04-25T01:54:28.677315+0000","last_clean":"2024-04-25T01:54:28.677315+0000","last_became_active":"2024-04-25T01:52:24.519932+0000","last_became_peered":"2024-04-25T01:52:24.519932+0000","last_unstale":"2024-04-25T01:54:28.677315+0000","last_undegraded":"2024-04-25T01:54:28.677315+0000","last_fullsized":"2024-04-25T01:54:28.677315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.786949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846741+0000","last_change":"2024-04-25T01:52:25.532822+0000","last_active":"2024-04-25T01:54:28.846741+0000","last_peered":"2024-04-25T01:54:28.846741+0000","last_clean":"2024-04-25T01:54:28.846741+0000","last_became_active":"2024-04-25T01:52:25.532462+0000","last_became_peered":"2024-04-25T01:52:25.532462+0000","last_unstale":"2024-04-25T01:54:28.846741+0000","last_undegraded":"2024-04-25T01:54:28.846741+0000","last_fullsized":"2024-04-25T01:54:28.846741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:55.569883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'12","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676809+0000","last_change":"2024-04-25T01:52:24.512923+0000","last_active":"2024-04-25T01:54:28.676809+0000","last_peered":"2024-04-25T01:54:28.676809+0000","last_clean":"2024-04-25T01:54:28.676809+0000","last_became_active":"2024-04-25T01:52:24.512604+0000","last_became_peered":"2024-04-25T01:52:24.512604+0000","last_unstale":"2024-04-25T01:54:28.676809+0000","last_undegraded":"2024-04-25T01:54:28.676809+0000","last_fullsized":"2024-04-25T01:54:28.676809+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:41.640234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697690+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:54:28.697690+0000","last_peered":"2024-04-25T01:54:28.697690+0000","last_clean":"2024-04-25T01:54:28.697690+0000","last_became_active":"2024-04-25T01:52:25.526447+0000","last_became_peered":"2024-04-25T01:52:25.526447+0000","last_unstale":"2024-04-25T01:54:28.697690+0000","last_undegraded":"2024-04-25T01:54:28.697690+0000","last_fullsized":"2024-04-25T01:54:28.697690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:10:18.313830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676651+0000","last_change":"2024-04-25T01:52:24.511565+0000","last_active":"2024-04-25T01:54:28.676651+0000","last_peered":"2024-04-25T01:54:28.676651+0000","last_clean":"2024-04-25T01:54:28.676651+0000","last_became_active":"2024-04-25T01:52:24.511258+0000","last_became_peered":"2024-04-25T01:52:24.511258+0000","last_unstale":"2024-04-25T01:54:28.676651+0000","last_undegraded":"2024-04-25T01:54:28.676651+0000","last_fullsized":"2024-04-25T01:54:28.676651+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:38.733338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585663+0000","last_change":"2024-04-25T01:53:25.360965+0000","last_active":"2024-04-25T01:54:28.585663+0000","last_peered":"2024-04-25T01:54:28.585663+0000","last_clean":"2024-04-25T01:54:28.585663+0000","last_became_active":"2024-04-25T01:53:25.360482+0000","last_became_peered":"2024-04-25T01:53:25.360482+0000","last_unstale":"2024-04-25T01:54:28.585663+0000","last_undegraded":"2024-04-25T01:54:28.585663+0000","last_fullsized":"2024-04-25T01:54:28.585663+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:05:01.122687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.947950+0000","last_change":"2024-04-25T01:52:24.514854+0000","last_active":"2024-04-25T01:54:28.947950+0000","last_peered":"2024-04-25T01:54:28.947950+0000","last_clean":"2024-04-25T01:54:28.947950+0000","last_became_active":"2024-04-25T01:52:24.514548+0000","last_became_peered":"2024-04-25T01:52:24.514548+0000","last_unstale":"2024-04-25T01:54:28.947950+0000","last_undegraded":"2024-04-25T01:54:28.947950+0000","last_fullsized":"2024-04-25T01:54:28.947950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:38.539202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1506,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676465+0000","last_change":"2024-04-25T01:52:25.523888+0000","last_active":"2024-04-25T01:54:28.676465+0000","last_peered":"2024-04-25T01:54:28.676465+0000","last_clean":"2024-04-25T01:54:28.676465+0000","last_became_active":"2024-04-25T01:52:25.523581+0000","last_became_peered":"2024-04-25T01:52:25.523581+0000","last_unstale":"2024-04-25T01:54:28.676465+0000","last_undegraded":"2024-04-25T01:54:28.676465+0000","last_fullsized":"2024-04-25T01:54:28.676465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:59.521757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685081+0000","last_change":"2024-04-25T01:53:24.972169+0000","last_active":"2024-04-25T01:54:28.685081+0000","last_peered":"2024-04-25T01:54:28.685081+0000","last_clean":"2024-04-25T01:54:28.685081+0000","last_became_active":"2024-04-25T01:53:24.966425+0000","last_became_peered":"2024-04-25T01:53:24.966425+0000","last_unstale":"2024-04-25T01:54:28.685081+0000","last_undegraded":"2024-04-25T01:54:28.685081+0000","last_fullsized":"2024-04-25T01:54:28.685081+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:31.052476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.586003+0000","last_change":"2024-04-25T01:52:25.523381+0000","last_active":"2024-04-25T01:54:28.586003+0000","last_peered":"2024-04-25T01:54:28.586003+0000","last_clean":"2024-04-25T01:54:28.586003+0000","last_became_active":"2024-04-25T01:52:25.522978+0000","last_became_peered":"2024-04-25T01:52:25.522978+0000","last_unstale":"2024-04-25T01:54:28.586003+0000","last_undegraded":"2024-04-25T01:54:28.586003+0000","last_fullsized":"2024-04-25T01:54:28.586003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:45.702788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908283+0000","last_change":"2024-04-25T01:52:24.521280+0000","last_active":"2024-04-25T01:54:28.908283+0000","last_peered":"2024-04-25T01:54:28.908283+0000","last_clean":"2024-04-25T01:54:28.908283+0000","last_became_active":"2024-04-25T01:52:24.520959+0000","last_became_peered":"2024-04-25T01:52:24.520959+0000","last_unstale":"2024-04-25T01:54:28.908283+0000","last_undegraded":"2024-04-25T01:54:28.908283+0000","last_fullsized":"2024-04-25T01:54:28.908283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:55.037678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676418+0000","last_change":"2024-04-25T01:52:25.524331+0000","last_active":"2024-04-25T01:54:28.676418+0000","last_peered":"2024-04-25T01:54:28.676418+0000","last_clean":"2024-04-25T01:54:28.676418+0000","last_became_active":"2024-04-25T01:52:25.523431+0000","last_became_peered":"2024-04-25T01:52:25.523431+0000","last_unstale":"2024-04-25T01:54:28.676418+0000","last_undegraded":"2024-04-25T01:54:28.676418+0000","last_fullsized":"2024-04-25T01:54:28.676418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:55.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585532+0000","last_change":"2024-04-25T01:52:24.515697+0000","last_active":"2024-04-25T01:54:28.585532+0000","last_peered":"2024-04-25T01:54:28.585532+0000","last_clean":"2024-04-25T01:54:28.585532+0000","last_became_active":"2024-04-25T01:52:24.515285+0000","last_became_peered":"2024-04-25T01:52:24.515285+0000","last_unstale":"2024-04-25T01:54:28.585532+0000","last_undegraded":"2024-04-25T01:54:28.585532+0000","last_fullsized":"2024-04-25T01:54:28.585532+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:24.249203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585591+0000","last_change":"2024-04-25T01:52:25.522415+0000","last_active":"2024-04-25T01:54:28.585591+0000","last_peered":"2024-04-25T01:54:28.585591+0000","last_clean":"2024-04-25T01:54:28.585591+0000","last_became_active":"2024-04-25T01:52:25.521270+0000","last_became_peered":"2024-04-25T01:52:25.521270+0000","last_unstale":"2024-04-25T01:54:28.585591+0000","last_undegraded":"2024-04-25T01:54:28.585591+0000","last_fullsized":"2024-04-25T01:54:28.585591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:53.514169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'7","reported_seq":262,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.947238+0000","last_change":"2024-04-25T01:52:24.509525+0000","last_active":"2024-04-25T01:54:28.947238+0000","last_peered":"2024-04-25T01:54:28.947238+0000","last_clean":"2024-04-25T01:54:28.947238+0000","last_became_active":"2024-04-25T01:52:24.509184+0000","last_became_peered":"2024-04-25T01:52:24.509184+0000","last_unstale":"2024-04-25T01:54:28.947238+0000","last_undegraded":"2024-04-25T01:54:28.947238+0000","last_fullsized":"2024-04-25T01:54:28.947238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:16:43.301401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697979+0000","last_change":"2024-04-25T01:52:25.525293+0000","last_active":"2024-04-25T01:54:28.697979+0000","last_peered":"2024-04-25T01:54:28.697979+0000","last_clean":"2024-04-25T01:54:28.697979+0000","last_became_active":"2024-04-25T01:52:25.524982+0000","last_became_peered":"2024-04-25T01:52:25.524982+0000","last_unstale":"2024-04-25T01:54:28.697979+0000","last_undegraded":"2024-04-25T01:54:28.697979+0000","last_fullsized":"2024-04-25T01:54:28.697979+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:22.222898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838052+0000","last_change":"2024-04-25T01:52:24.519834+0000","last_active":"2024-04-25T01:54:28.838052+0000","last_peered":"2024-04-25T01:54:28.838052+0000","last_clean":"2024-04-25T01:54:28.838052+0000","last_became_active":"2024-04-25T01:52:24.519521+0000","last_became_peered":"2024-04-25T01:52:24.519521+0000","last_unstale":"2024-04-25T01:54:28.838052+0000","last_undegraded":"2024-04-25T01:54:28.838052+0000","last_fullsized":"2024-04-25T01:54:28.838052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:02.414527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846665+0000","last_change":"2024-04-25T01:52:25.523809+0000","last_active":"2024-04-25T01:54:28.846665+0000","last_peered":"2024-04-25T01:54:28.846665+0000","last_clean":"2024-04-25T01:54:28.846665+0000","last_became_active":"2024-04-25T01:52:25.523406+0000","last_became_peered":"2024-04-25T01:52:25.523406+0000","last_unstale":"2024-04-25T01:54:28.846665+0000","last_undegraded":"2024-04-25T01:54:28.846665+0000","last_fullsized":"2024-04-25T01:54:28.846665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:22.169633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585864+0000","last_change":"2024-04-25T01:52:24.510226+0000","last_active":"2024-04-25T01:54:28.585864+0000","last_peered":"2024-04-25T01:54:28.585864+0000","last_clean":"2024-04-25T01:54:28.585864+0000","last_became_active":"2024-04-25T01:52:24.509667+0000","last_became_peered":"2024-04-25T01:52:24.509667+0000","last_unstale":"2024-04-25T01:54:28.585864+0000","last_undegraded":"2024-04-25T01:54:28.585864+0000","last_fullsized":"2024-04-25T01:54:28.585864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:49.180675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697870+0000","last_change":"2024-04-25T01:52:25.531567+0000","last_active":"2024-04-25T01:54:28.697870+0000","last_peered":"2024-04-25T01:54:28.697870+0000","last_clean":"2024-04-25T01:54:28.697870+0000","last_became_active":"2024-04-25T01:52:25.531200+0000","last_became_peered":"2024-04-25T01:52:25.531200+0000","last_unstale":"2024-04-25T01:54:28.697870+0000","last_undegraded":"2024-04-25T01:54:28.697870+0000","last_fullsized":"2024-04-25T01:54:28.697870+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:51:14.336413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'38","reported_seq":323,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:29.457625+0000","last_change":"2024-04-25T01:53:25.369844+0000","last_active":"2024-04-25T01:54:29.457625+0000","last_peered":"2024-04-25T01:54:29.457625+0000","last_clean":"2024-04-25T01:54:29.457625+0000","last_became_active":"2024-04-25T01:53:25.362162+0000","last_became_peered":"2024-04-25T01:53:25.362162+0000","last_unstale":"2024-04-25T01:54:29.457625+0000","last_undegraded":"2024-04-25T01:54:29.457625+0000","last_fullsized":"2024-04-25T01:54:29.457625+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:27:01.730381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66884,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":236,"num_read_kb":183,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676640+0000","last_change":"2024-04-25T01:52:25.525931+0000","last_active":"2024-04-25T01:54:28.676640+0000","last_peered":"2024-04-25T01:54:28.676640+0000","last_clean":"2024-04-25T01:54:28.676640+0000","last_became_active":"2024-04-25T01:52:25.525460+0000","last_became_peered":"2024-04-25T01:52:25.525460+0000","last_unstale":"2024-04-25T01:54:28.676640+0000","last_undegraded":"2024-04-25T01:54:28.676640+0000","last_fullsized":"2024-04-25T01:54:28.676640+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:46.569587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837433+0000","last_change":"2024-04-25T01:52:24.505470+0000","last_active":"2024-04-25T01:54:28.837433+0000","last_peered":"2024-04-25T01:54:28.837433+0000","last_clean":"2024-04-25T01:54:28.837433+0000","last_became_active":"2024-04-25T01:52:24.504677+0000","last_became_peered":"2024-04-25T01:52:24.504677+0000","last_unstale":"2024-04-25T01:54:28.837433+0000","last_undegraded":"2024-04-25T01:54:28.837433+0000","last_fullsized":"2024-04-25T01:54:28.837433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:01.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908209+0000","last_change":"2024-04-25T01:52:25.526364+0000","last_active":"2024-04-25T01:54:28.908209+0000","last_peered":"2024-04-25T01:54:28.908209+0000","last_clean":"2024-04-25T01:54:28.908209+0000","last_became_active":"2024-04-25T01:52:25.525376+0000","last_became_peered":"2024-04-25T01:52:25.525376+0000","last_unstale":"2024-04-25T01:54:28.908209+0000","last_undegraded":"2024-04-25T01:54:28.908209+0000","last_fullsized":"2024-04-25T01:54:28.908209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:26.770994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585776+0000","last_change":"2024-04-25T01:52:24.520287+0000","last_active":"2024-04-25T01:54:28.585776+0000","last_peered":"2024-04-25T01:54:28.585776+0000","last_clean":"2024-04-25T01:54:28.585776+0000","last_became_active":"2024-04-25T01:52:24.519967+0000","last_became_peered":"2024-04-25T01:52:24.519967+0000","last_unstale":"2024-04-25T01:54:28.585776+0000","last_undegraded":"2024-04-25T01:54:28.585776+0000","last_fullsized":"2024-04-25T01:54:28.585776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:45.078410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837933+0000","last_change":"2024-04-25T01:52:25.525766+0000","last_active":"2024-04-25T01:54:28.837933+0000","last_peered":"2024-04-25T01:54:28.837933+0000","last_clean":"2024-04-25T01:54:28.837933+0000","last_became_active":"2024-04-25T01:52:25.525350+0000","last_became_peered":"2024-04-25T01:52:25.525350+0000","last_unstale":"2024-04-25T01:54:28.837933+0000","last_undegraded":"2024-04-25T01:54:28.837933+0000","last_fullsized":"2024-04-25T01:54:28.837933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:05.387502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'10","reported_seq":153,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.936284+0000","last_change":"2024-04-25T01:52:24.513674+0000","last_active":"2024-04-25T01:54:28.936284+0000","last_peered":"2024-04-25T01:54:28.936284+0000","last_clean":"2024-04-25T01:54:28.936284+0000","last_became_active":"2024-04-25T01:52:24.513270+0000","last_became_peered":"2024-04-25T01:52:24.513270+0000","last_unstale":"2024-04-25T01:54:28.936284+0000","last_undegraded":"2024-04-25T01:54:28.936284+0000","last_fullsized":"2024-04-25T01:54:28.936284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:23.385928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":6,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837493+0000","last_change":"2024-04-25T01:52:25.517166+0000","last_active":"2024-04-25T01:54:28.837493+0000","last_peered":"2024-04-25T01:54:28.837493+0000","last_clean":"2024-04-25T01:54:28.837493+0000","last_became_active":"2024-04-25T01:52:25.516853+0000","last_became_peered":"2024-04-25T01:52:25.516853+0000","last_unstale":"2024-04-25T01:54:28.837493+0000","last_undegraded":"2024-04-25T01:54:28.837493+0000","last_fullsized":"2024-04-25T01:54:28.837493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:18.502558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'17","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676369+0000","last_change":"2024-04-25T01:52:24.514695+0000","last_active":"2024-04-25T01:54:28.676369+0000","last_peered":"2024-04-25T01:54:28.676369+0000","last_clean":"2024-04-25T01:54:28.676369+0000","last_became_active":"2024-04-25T01:52:24.514370+0000","last_became_peered":"2024-04-25T01:52:24.514370+0000","last_unstale":"2024-04-25T01:54:28.676369+0000","last_undegraded":"2024-04-25T01:54:28.676369+0000","last_fullsized":"2024-04-25T01:54:28.676369+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:28:28.537307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585698+0000","last_change":"2024-04-25T01:52:25.527062+0000","last_active":"2024-04-25T01:54:28.585698+0000","last_peered":"2024-04-25T01:54:28.585698+0000","last_clean":"2024-04-25T01:54:28.585698+0000","last_became_active":"2024-04-25T01:52:25.526757+0000","last_became_peered":"2024-04-25T01:52:25.526757+0000","last_unstale":"2024-04-25T01:54:28.585698+0000","last_undegraded":"2024-04-25T01:54:28.585698+0000","last_fullsized":"2024-04-25T01:54:28.585698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:36.360030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676786+0000","last_change":"2024-04-25T01:52:24.512235+0000","last_active":"2024-04-25T01:54:28.676786+0000","last_peered":"2024-04-25T01:54:28.676786+0000","last_clean":"2024-04-25T01:54:28.676786+0000","last_became_active":"2024-04-25T01:52:24.511910+0000","last_became_peered":"2024-04-25T01:52:24.511910+0000","last_unstale":"2024-04-25T01:54:28.676786+0000","last_undegraded":"2024-04-25T01:54:28.676786+0000","last_fullsized":"2024-04-25T01:54:28.676786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:38.435734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845541+0000","last_change":"2024-04-25T01:52:25.527854+0000","last_active":"2024-04-25T01:54:28.845541+0000","last_peered":"2024-04-25T01:54:28.845541+0000","last_clean":"2024-04-25T01:54:28.845541+0000","last_became_active":"2024-04-25T01:52:25.527481+0000","last_became_peered":"2024-04-25T01:52:25.527481+0000","last_unstale":"2024-04-25T01:54:28.845541+0000","last_undegraded":"2024-04-25T01:54:28.845541+0000","last_fullsized":"2024-04-25T01:54:28.845541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:09:11.281982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585833+0000","last_change":"2024-04-25T01:52:24.513463+0000","last_active":"2024-04-25T01:54:28.585833+0000","last_peered":"2024-04-25T01:54:28.585833+0000","last_clean":"2024-04-25T01:54:28.585833+0000","last_became_active":"2024-04-25T01:52:24.513176+0000","last_became_peered":"2024-04-25T01:52:24.513176+0000","last_unstale":"2024-04-25T01:54:28.585833+0000","last_undegraded":"2024-04-25T01:54:28.585833+0000","last_fullsized":"2024-04-25T01:54:28.585833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:13:33.507559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685167+0000","last_change":"2024-04-25T01:52:25.528316+0000","last_active":"2024-04-25T01:54:28.685167+0000","last_peered":"2024-04-25T01:54:28.685167+0000","last_clean":"2024-04-25T01:54:28.685167+0000","last_became_active":"2024-04-25T01:52:25.527991+0000","last_became_peered":"2024-04-25T01:52:25.527991+0000","last_unstale":"2024-04-25T01:54:28.685167+0000","last_undegraded":"2024-04-25T01:54:28.685167+0000","last_fullsized":"2024-04-25T01:54:28.685167+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:55.871663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'6","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.936148+0000","last_change":"2024-04-25T01:52:24.519357+0000","last_active":"2024-04-25T01:54:28.936148+0000","last_peered":"2024-04-25T01:54:28.936148+0000","last_clean":"2024-04-25T01:54:28.936148+0000","last_became_active":"2024-04-25T01:52:24.519025+0000","last_became_peered":"2024-04-25T01:52:24.519025+0000","last_unstale":"2024-04-25T01:54:28.936148+0000","last_undegraded":"2024-04-25T01:54:28.936148+0000","last_fullsized":"2024-04-25T01:54:28.936148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:03.476554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685163+0000","last_change":"2024-04-25T01:52:25.527922+0000","last_active":"2024-04-25T01:54:28.685163+0000","last_peered":"2024-04-25T01:54:28.685163+0000","last_clean":"2024-04-25T01:54:28.685163+0000","last_became_active":"2024-04-25T01:52:25.527596+0000","last_became_peered":"2024-04-25T01:52:25.527596+0000","last_unstale":"2024-04-25T01:54:28.685163+0000","last_undegraded":"2024-04-25T01:54:28.685163+0000","last_fullsized":"2024-04-25T01:54:28.685163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:56.002208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585732+0000","last_change":"2024-04-25T01:52:25.533800+0000","last_active":"2024-04-25T01:54:28.585732+0000","last_peered":"2024-04-25T01:54:28.585732+0000","last_clean":"2024-04-25T01:54:28.585732+0000","last_became_active":"2024-04-25T01:52:25.533484+0000","last_became_peered":"2024-04-25T01:52:25.533484+0000","last_unstale":"2024-04-25T01:54:28.585732+0000","last_undegraded":"2024-04-25T01:54:28.585732+0000","last_fullsized":"2024-04-25T01:54:28.585732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:13.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'7","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.935719+0000","last_change":"2024-04-25T01:52:24.506068+0000","last_active":"2024-04-25T01:54:28.935719+0000","last_peered":"2024-04-25T01:54:28.935719+0000","last_clean":"2024-04-25T01:54:28.935719+0000","last_became_active":"2024-04-25T01:52:24.505667+0000","last_became_peered":"2024-04-25T01:52:24.505667+0000","last_unstale":"2024-04-25T01:54:28.935719+0000","last_undegraded":"2024-04-25T01:54:28.935719+0000","last_fullsized":"2024-04-25T01:54:28.935719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:30.494376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676897+0000","last_change":"2024-04-25T01:52:25.530681+0000","last_active":"2024-04-25T01:54:28.676897+0000","last_peered":"2024-04-25T01:54:28.676897+0000","last_clean":"2024-04-25T01:54:28.676897+0000","last_became_active":"2024-04-25T01:52:25.529035+0000","last_became_peered":"2024-04-25T01:52:25.529035+0000","last_unstale":"2024-04-25T01:54:28.676897+0000","last_undegraded":"2024-04-25T01:54:28.676897+0000","last_fullsized":"2024-04-25T01:54:28.676897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:28.787180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'6","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.587716+0000","last_change":"2024-04-25T01:52:24.508948+0000","last_active":"2024-04-25T01:54:28.587716+0000","last_peered":"2024-04-25T01:54:28.587716+0000","last_clean":"2024-04-25T01:54:28.587716+0000","last_became_active":"2024-04-25T01:52:24.508563+0000","last_became_peered":"2024-04-25T01:52:24.508563+0000","last_unstale":"2024-04-25T01:54:28.587716+0000","last_undegraded":"2024-04-25T01:54:28.587716+0000","last_fullsized":"2024-04-25T01:54:28.587716+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:37.591783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":355,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837553+0000","last_change":"2024-04-25T01:51:15.621251+0000","last_active":"2024-04-25T01:54:28.837553+0000","last_peered":"2024-04-25T01:54:28.837553+0000","last_clean":"2024-04-25T01:54:28.837553+0000","last_became_active":"2024-04-25T01:51:15.620771+0000","last_became_peered":"2024-04-25T01:51:15.620771+0000","last_unstale":"2024-04-25T01:54:28.837553+0000","last_undegraded":"2024-04-25T01:54:28.837553+0000","last_fullsized":"2024-04-25T01:54:28.837553+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_clean_scrub_stamp":"2024-04-25T01:47:50.544829+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:44.048065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908179+0000","last_change":"2024-04-25T01:52:25.530216+0000","last_active":"2024-04-25T01:54:28.908179+0000","last_peered":"2024-04-25T01:54:28.908179+0000","last_clean":"2024-04-25T01:54:28.908179+0000","last_became_active":"2024-04-25T01:52:25.529813+0000","last_became_peered":"2024-04-25T01:52:25.529813+0000","last_unstale":"2024-04-25T01:54:28.908179+0000","last_undegraded":"2024-04-25T01:54:28.908179+0000","last_fullsized":"2024-04-25T01:54:28.908179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:57.656375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845310+0000","last_change":"2024-04-25T01:53:24.979114+0000","last_active":"2024-04-25T01:54:28.845310+0000","last_peered":"2024-04-25T01:54:28.845310+0000","last_clean":"2024-04-25T01:54:28.845310+0000","last_became_active":"2024-04-25T01:53:24.972591+0000","last_became_peered":"2024-04-25T01:53:24.972591+0000","last_unstale":"2024-04-25T01:54:28.845310+0000","last_undegraded":"2024-04-25T01:54:28.845310+0000","last_fullsized":"2024-04-25T01:54:28.845310+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:22.792173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676695+0000","last_change":"2024-04-25T01:52:24.514786+0000","last_active":"2024-04-25T01:54:28.676695+0000","last_peered":"2024-04-25T01:54:28.676695+0000","last_clean":"2024-04-25T01:54:28.676695+0000","last_became_active":"2024-04-25T01:52:24.514486+0000","last_became_peered":"2024-04-25T01:52:24.514486+0000","last_unstale":"2024-04-25T01:54:28.676695+0000","last_undegraded":"2024-04-25T01:54:28.676695+0000","last_fullsized":"2024-04-25T01:54:28.676695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:42.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908392+0000","last_change":"2024-04-25T01:52:25.526518+0000","last_active":"2024-04-25T01:54:28.908392+0000","last_peered":"2024-04-25T01:54:28.908392+0000","last_clean":"2024-04-25T01:54:28.908392+0000","last_became_active":"2024-04-25T01:52:25.525917+0000","last_became_peered":"2024-04-25T01:52:25.525917+0000","last_unstale":"2024-04-25T01:54:28.908392+0000","last_undegraded":"2024-04-25T01:54:28.908392+0000","last_fullsized":"2024-04-25T01:54:28.908392+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:20.776385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'48","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.791943+0000","last_change":"2024-04-25T01:52:24.511561+0000","last_active":"2024-04-25T01:54:28.791943+0000","last_peered":"2024-04-25T01:54:28.791943+0000","last_clean":"2024-04-25T01:54:28.791943+0000","last_became_active":"2024-04-25T01:52:24.511165+0000","last_became_peered":"2024-04-25T01:52:24.511165+0000","last_unstale":"2024-04-25T01:54:28.791943+0000","last_undegraded":"2024-04-25T01:54:28.791943+0000","last_fullsized":"2024-04-25T01:54:28.791943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:27:41.006137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677414+0000","last_change":"2024-04-25T01:52:25.527991+0000","last_active":"2024-04-25T01:54:28.677414+0000","last_peered":"2024-04-25T01:54:28.677414+0000","last_clean":"2024-04-25T01:54:28.677414+0000","last_became_active":"2024-04-25T01:52:25.527677+0000","last_became_peered":"2024-04-25T01:52:25.527677+0000","last_unstale":"2024-04-25T01:54:28.677414+0000","last_undegraded":"2024-04-25T01:54:28.677414+0000","last_fullsized":"2024-04-25T01:54:28.677414+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:07:24.698774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.587918+0000","last_change":"2024-04-25T01:52:24.523536+0000","last_active":"2024-04-25T01:54:28.587918+0000","last_peered":"2024-04-25T01:54:28.587918+0000","last_clean":"2024-04-25T01:54:28.587918+0000","last_became_active":"2024-04-25T01:52:24.523233+0000","last_became_peered":"2024-04-25T01:52:24.523233+0000","last_unstale":"2024-04-25T01:54:28.587918+0000","last_undegraded":"2024-04-25T01:54:28.587918+0000","last_fullsized":"2024-04-25T01:54:28.587918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:49:44.444968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846557+0000","last_change":"2024-04-25T01:52:25.527090+0000","last_active":"2024-04-25T01:54:28.846557+0000","last_peered":"2024-04-25T01:54:28.846557+0000","last_clean":"2024-04-25T01:54:28.846557+0000","last_became_active":"2024-04-25T01:52:25.526771+0000","last_became_peered":"2024-04-25T01:52:25.526771+0000","last_unstale":"2024-04-25T01:54:28.846557+0000","last_undegraded":"2024-04-25T01:54:28.846557+0000","last_fullsized":"2024-04-25T01:54:28.846557+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:58.758748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697610+0000","last_change":"2024-04-25T01:52:24.516469+0000","last_active":"2024-04-25T01:54:28.697610+0000","last_peered":"2024-04-25T01:54:28.697610+0000","last_clean":"2024-04-25T01:54:28.697610+0000","last_became_active":"2024-04-25T01:52:24.515948+0000","last_became_peered":"2024-04-25T01:52:24.515948+0000","last_unstale":"2024-04-25T01:54:28.697610+0000","last_undegraded":"2024-04-25T01:54:28.697610+0000","last_fullsized":"2024-04-25T01:54:28.697610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:25.122153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676369+0000","last_change":"2024-04-25T01:52:25.524967+0000","last_active":"2024-04-25T01:54:28.676369+0000","last_peered":"2024-04-25T01:54:28.676369+0000","last_clean":"2024-04-25T01:54:28.676369+0000","last_became_active":"2024-04-25T01:52:25.524670+0000","last_became_peered":"2024-04-25T01:52:25.524670+0000","last_unstale":"2024-04-25T01:54:28.676369+0000","last_undegraded":"2024-04-25T01:54:28.676369+0000","last_fullsized":"2024-04-25T01:54:28.676369+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:08.829052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676711+0000","last_change":"2024-04-25T01:52:24.512187+0000","last_active":"2024-04-25T01:54:28.676711+0000","last_peered":"2024-04-25T01:54:28.676711+0000","last_clean":"2024-04-25T01:54:28.676711+0000","last_became_active":"2024-04-25T01:52:24.510303+0000","last_became_peered":"2024-04-25T01:52:24.510303+0000","last_unstale":"2024-04-25T01:54:28.676711+0000","last_undegraded":"2024-04-25T01:54:28.676711+0000","last_fullsized":"2024-04-25T01:54:28.676711+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:42:31.596355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845405+0000","last_change":"2024-04-25T01:52:25.530386+0000","last_active":"2024-04-25T01:54:28.845405+0000","last_peered":"2024-04-25T01:54:28.845405+0000","last_clean":"2024-04-25T01:54:28.845405+0000","last_became_active":"2024-04-25T01:52:25.530034+0000","last_became_peered":"2024-04-25T01:52:25.530034+0000","last_unstale":"2024-04-25T01:54:28.845405+0000","last_undegraded":"2024-04-25T01:54:28.845405+0000","last_fullsized":"2024-04-25T01:54:28.845405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:11.470123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'4","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845367+0000","last_change":"2024-04-25T01:53:24.976373+0000","last_active":"2024-04-25T01:54:28.845367+0000","last_peered":"2024-04-25T01:54:28.845367+0000","last_clean":"2024-04-25T01:54:28.845367+0000","last_became_active":"2024-04-25T01:53:24.971146+0000","last_became_peered":"2024-04-25T01:53:24.971146+0000","last_unstale":"2024-04-25T01:54:28.845367+0000","last_undegraded":"2024-04-25T01:54:28.845367+0000","last_fullsized":"2024-04-25T01:54:28.845367+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:33.079071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677039+0000","last_change":"2024-04-25T01:52:25.525695+0000","last_active":"2024-04-25T01:54:28.677039+0000","last_peered":"2024-04-25T01:54:28.677039+0000","last_clean":"2024-04-25T01:54:28.677039+0000","last_became_active":"2024-04-25T01:52:25.524830+0000","last_became_peered":"2024-04-25T01:52:25.524830+0000","last_unstale":"2024-04-25T01:54:28.677039+0000","last_undegraded":"2024-04-25T01:54:28.677039+0000","last_fullsized":"2024-04-25T01:54:28.677039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:04:58.449350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676468+0000","last_change":"2024-04-25T01:52:24.519232+0000","last_active":"2024-04-25T01:54:28.676468+0000","last_peered":"2024-04-25T01:54:28.676468+0000","last_clean":"2024-04-25T01:54:28.676468+0000","last_became_active":"2024-04-25T01:52:24.518902+0000","last_became_peered":"2024-04-25T01:52:24.518902+0000","last_unstale":"2024-04-25T01:54:28.676468+0000","last_undegraded":"2024-04-25T01:54:28.676468+0000","last_fullsized":"2024-04-25T01:54:28.676468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.173730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837719+0000","last_change":"2024-04-25T01:52:25.522955+0000","last_active":"2024-04-25T01:54:28.837719+0000","last_peered":"2024-04-25T01:54:28.837719+0000","last_clean":"2024-04-25T01:54:28.837719+0000","last_became_active":"2024-04-25T01:52:25.522626+0000","last_became_peered":"2024-04-25T01:52:25.522626+0000","last_unstale":"2024-04-25T01:54:28.837719+0000","last_undegraded":"2024-04-25T01:54:28.837719+0000","last_fullsized":"2024-04-25T01:54:28.837719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:47.218461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'9","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.947801+0000","last_change":"2024-04-25T01:52:24.516383+0000","last_active":"2024-04-25T01:54:28.947801+0000","last_peered":"2024-04-25T01:54:28.947801+0000","last_clean":"2024-04-25T01:54:28.947801+0000","last_became_active":"2024-04-25T01:52:24.515096+0000","last_became_peered":"2024-04-25T01:52:24.515096+0000","last_unstale":"2024-04-25T01:54:28.947801+0000","last_undegraded":"2024-04-25T01:54:28.947801+0000","last_fullsized":"2024-04-25T01:54:28.947801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:52.658161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2586,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838178+0000","last_change":"2024-04-25T01:52:25.524874+0000","last_active":"2024-04-25T01:54:28.838178+0000","last_peered":"2024-04-25T01:54:28.838178+0000","last_clean":"2024-04-25T01:54:28.838178+0000","last_became_active":"2024-04-25T01:52:25.523532+0000","last_became_peered":"2024-04-25T01:52:25.523532+0000","last_unstale":"2024-04-25T01:54:28.838178+0000","last_undegraded":"2024-04-25T01:54:28.838178+0000","last_fullsized":"2024-04-25T01:54:28.838178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:10.746446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'41","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.789815+0000","last_change":"2024-04-25T01:52:24.526212+0000","last_active":"2024-04-25T01:54:28.789815+0000","last_peered":"2024-04-25T01:54:28.789815+0000","last_clean":"2024-04-25T01:54:28.789815+0000","last_became_active":"2024-04-25T01:52:24.525882+0000","last_became_peered":"2024-04-25T01:52:24.525882+0000","last_unstale":"2024-04-25T01:54:28.789815+0000","last_undegraded":"2024-04-25T01:54:28.789815+0000","last_fullsized":"2024-04-25T01:54:28.789815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:04.792560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837993+0000","last_change":"2024-04-25T01:52:25.517387+0000","last_active":"2024-04-25T01:54:28.837993+0000","last_peered":"2024-04-25T01:54:28.837993+0000","last_clean":"2024-04-25T01:54:28.837993+0000","last_became_active":"2024-04-25T01:52:25.517047+0000","last_became_peered":"2024-04-25T01:52:25.517047+0000","last_unstale":"2024-04-25T01:54:28.837993+0000","last_undegraded":"2024-04-25T01:54:28.837993+0000","last_fullsized":"2024-04-25T01:54:28.837993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:39.462474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908253+0000","last_change":"2024-04-25T01:52:24.506510+0000","last_active":"2024-04-25T01:54:28.908253+0000","last_peered":"2024-04-25T01:54:28.908253+0000","last_clean":"2024-04-25T01:54:28.908253+0000","last_became_active":"2024-04-25T01:52:24.506184+0000","last_became_peered":"2024-04-25T01:52:24.506184+0000","last_unstale":"2024-04-25T01:54:28.908253+0000","last_undegraded":"2024-04-25T01:54:28.908253+0000","last_fullsized":"2024-04-25T01:54:28.908253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:56.754386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838180+0000","last_change":"2024-04-25T01:52:25.525197+0000","last_active":"2024-04-25T01:54:28.838180+0000","last_peered":"2024-04-25T01:54:28.838180+0000","last_clean":"2024-04-25T01:54:28.838180+0000","last_became_active":"2024-04-25T01:52:25.524692+0000","last_became_peered":"2024-04-25T01:52:25.524692+0000","last_unstale":"2024-04-25T01:54:28.838180+0000","last_undegraded":"2024-04-25T01:54:28.838180+0000","last_fullsized":"2024-04-25T01:54:28.838180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.264227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845432+0000","last_change":"2024-04-25T01:52:25.531060+0000","last_active":"2024-04-25T01:54:28.845432+0000","last_peered":"2024-04-25T01:54:28.845432+0000","last_clean":"2024-04-25T01:54:28.845432+0000","last_became_active":"2024-04-25T01:52:25.530774+0000","last_became_peered":"2024-04-25T01:52:25.530774+0000","last_unstale":"2024-04-25T01:54:28.845432+0000","last_undegraded":"2024-04-25T01:54:28.845432+0000","last_fullsized":"2024-04-25T01:54:28.845432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:32.168699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837364+0000","last_change":"2024-04-25T01:52:25.535381+0000","last_active":"2024-04-25T01:54:28.837364+0000","last_peered":"2024-04-25T01:54:28.837364+0000","last_clean":"2024-04-25T01:54:28.837364+0000","last_became_active":"2024-04-25T01:52:25.533623+0000","last_became_peered":"2024-04-25T01:52:25.533623+0000","last_unstale":"2024-04-25T01:54:28.837364+0000","last_undegraded":"2024-04-25T01:54:28.837364+0000","last_fullsized":"2024-04-25T01:54:28.837364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:14.032508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837321+0000","last_change":"2024-04-25T01:52:25.528803+0000","last_active":"2024-04-25T01:54:28.837321+0000","last_peered":"2024-04-25T01:54:28.837321+0000","last_clean":"2024-04-25T01:54:28.837321+0000","last_became_active":"2024-04-25T01:52:25.528505+0000","last_became_peered":"2024-04-25T01:52:25.528505+0000","last_unstale":"2024-04-25T01:54:28.837321+0000","last_undegraded":"2024-04-25T01:54:28.837321+0000","last_fullsized":"2024-04-25T01:54:28.837321+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:35.811181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846209+0000","last_change":"2024-04-25T01:52:25.523620+0000","last_active":"2024-04-25T01:54:28.846209+0000","last_peered":"2024-04-25T01:54:28.846209+0000","last_clean":"2024-04-25T01:54:28.846209+0000","last_became_active":"2024-04-25T01:52:25.523007+0000","last_became_peered":"2024-04-25T01:52:25.523007+0000","last_unstale":"2024-04-25T01:54:28.846209+0000","last_undegraded":"2024-04-25T01:54:28.846209+0000","last_fullsized":"2024-04-25T01:54:28.846209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:23.587838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908298+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:54:28.908298+0000","last_peered":"2024-04-25T01:54:28.908298+0000","last_clean":"2024-04-25T01:54:28.908298+0000","last_became_active":"2024-04-25T01:52:25.526398+0000","last_became_peered":"2024-04-25T01:52:25.526398+0000","last_unstale":"2024-04-25T01:54:28.908298+0000","last_undegraded":"2024-04-25T01:54:28.908298+0000","last_fullsized":"2024-04-25T01:54:28.908298+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:01:37.286676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845498+0000","last_change":"2024-04-25T01:52:25.524999+0000","last_active":"2024-04-25T01:54:28.845498+0000","last_peered":"2024-04-25T01:54:28.845498+0000","last_clean":"2024-04-25T01:54:28.845498+0000","last_became_active":"2024-04-25T01:52:25.524652+0000","last_became_peered":"2024-04-25T01:52:25.524652+0000","last_unstale":"2024-04-25T01:54:28.845498+0000","last_undegraded":"2024-04-25T01:54:28.845498+0000","last_fullsized":"2024-04-25T01:54:28.845498+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:47:37.463701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837706+0000","last_change":"2024-04-25T01:52:25.532996+0000","last_active":"2024-04-25T01:54:28.837706+0000","last_peered":"2024-04-25T01:54:28.837706+0000","last_clean":"2024-04-25T01:54:28.837706+0000","last_became_active":"2024-04-25T01:52:25.530230+0000","last_became_peered":"2024-04-25T01:52:25.530230+0000","last_unstale":"2024-04-25T01:54:28.837706+0000","last_undegraded":"2024-04-25T01:54:28.837706+0000","last_fullsized":"2024-04-25T01:54:28.837706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:08.996434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676996+0000","last_change":"2024-04-25T01:52:25.523838+0000","last_active":"2024-04-25T01:54:28.676996+0000","last_peered":"2024-04-25T01:54:28.676996+0000","last_clean":"2024-04-25T01:54:28.676996+0000","last_became_active":"2024-04-25T01:52:25.523297+0000","last_became_peered":"2024-04-25T01:52:25.523297+0000","last_unstale":"2024-04-25T01:54:28.676996+0000","last_undegraded":"2024-04-25T01:54:28.676996+0000","last_fullsized":"2024-04-25T01:54:28.676996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:52.938763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585617+0000","last_change":"2024-04-25T01:52:25.519028+0000","last_active":"2024-04-25T01:54:28.585617+0000","last_peered":"2024-04-25T01:54:28.585617+0000","last_clean":"2024-04-25T01:54:28.585617+0000","last_became_active":"2024-04-25T01:52:25.518716+0000","last_became_peered":"2024-04-25T01:52:25.518716+0000","last_unstale":"2024-04-25T01:54:28.585617+0000","last_undegraded":"2024-04-25T01:54:28.585617+0000","last_fullsized":"2024-04-25T01:54:28.585617+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:37:31.573403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676618+0000","last_change":"2024-04-25T01:52:25.533894+0000","last_active":"2024-04-25T01:54:28.676618+0000","last_peered":"2024-04-25T01:54:28.676618+0000","last_clean":"2024-04-25T01:54:28.676618+0000","last_became_active":"2024-04-25T01:52:25.533510+0000","last_became_peered":"2024-04-25T01:52:25.533510+0000","last_unstale":"2024-04-25T01:54:28.676618+0000","last_undegraded":"2024-04-25T01:54:28.676618+0000","last_fullsized":"2024-04-25T01:54:28.676618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:07.128296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697784+0000","last_change":"2024-04-25T01:52:25.524693+0000","last_active":"2024-04-25T01:54:28.697784+0000","last_peered":"2024-04-25T01:54:28.697784+0000","last_clean":"2024-04-25T01:54:28.697784+0000","last_became_active":"2024-04-25T01:52:25.524366+0000","last_became_peered":"2024-04-25T01:52:25.524366+0000","last_unstale":"2024-04-25T01:54:28.697784+0000","last_undegraded":"2024-04-25T01:54:28.697784+0000","last_fullsized":"2024-04-25T01:54:28.697784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:23.133722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908579+0000","last_change":"2024-04-25T01:52:25.525936+0000","last_active":"2024-04-25T01:54:28.908579+0000","last_peered":"2024-04-25T01:54:28.908579+0000","last_clean":"2024-04-25T01:54:28.908579+0000","last_became_active":"2024-04-25T01:52:25.525586+0000","last_became_peered":"2024-04-25T01:52:25.525586+0000","last_unstale":"2024-04-25T01:54:28.908579+0000","last_undegraded":"2024-04-25T01:54:28.908579+0000","last_fullsized":"2024-04-25T01:54:28.908579+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:34.756898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845420+0000","last_change":"2024-04-25T01:53:24.970621+0000","last_active":"2024-04-25T01:54:28.845420+0000","last_peered":"2024-04-25T01:54:28.845420+0000","last_clean":"2024-04-25T01:54:28.845420+0000","last_became_active":"2024-04-25T01:53:24.970184+0000","last_became_peered":"2024-04-25T01:53:24.970184+0000","last_unstale":"2024-04-25T01:54:28.845420+0000","last_undegraded":"2024-04-25T01:54:28.845420+0000","last_fullsized":"2024-04-25T01:54:28.845420+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:52:43.072630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106956,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":602722,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1303,"num_read_kb":769,"num_write":377,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":671744,"data_stored":1808166,"data_compressed":89205,"data_compressed_allocated":118784,"data_compressed_original":1622016,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318308,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":478166,"data_compressed":2900,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579946,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32316,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712836,"statfs":{"total":95995035648,"available":95961944064,"internally_reserved":0,"allocated":1167360,"data_stored":1268070,"data_compressed":46559,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743470,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":869896,"data_compressed":22046,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005109,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":870274,"data_compressed":22046,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":1072369,"data_compressed":38552,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":476890,"data_compressed":2900,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":544687,"data_compressed":4751,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":469863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":470150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1001038,"data_compressed":28202,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":539071,"data_compressed":12497,"data_compressed_allocated":16384,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1201347,"data_compressed":44708,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":69129,"data_compressed":1851,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":69391,"data_compressed":9597,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135304,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":202027,"data_compressed":8007,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:54:33.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:54:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.186+0000 7f93aea2c640 1 -- 172.21.15.31:0/1149402068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93a81029d0 msgr2=0x7f93a8102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.186+0000 7f93aea2c640 1 --2- 172.21.15.31:0/1149402068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93a81029d0 0x7f93a8102db0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9398009900 tx=0x7f939802a510 comp rx=0 tx=0).stop 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.186+0000 7f93aea2c640 1 -- 172.21.15.31:0/1149402068 shutdown_connections 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.186+0000 7f93aea2c640 1 --2- 172.21.15.31:0/1149402068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93a810bd10 0x7f93a810e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.186+0000 7f93aea2c640 1 --2- 172.21.15.31:0/1149402068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93a81032f0 0x7f93a810b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.186+0000 7f93aea2c640 1 --2- 172.21.15.31:0/1149402068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93a81029d0 0x7f93a8102db0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.186+0000 7f93aea2c640 1 -- 172.21.15.31:0/1149402068 >> 172.21.15.31:0/1149402068 conn(0x7f93a8077740 msgr2=0x7f93a8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.187+0000 7f93aea2c640 1 -- 172.21.15.31:0/1149402068 shutdown_connections 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.187+0000 7f93aea2c640 1 -- 172.21.15.31:0/1149402068 wait complete. 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.187+0000 7f93aea2c640 1 Processor -- start 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.187+0000 7f93aea2c640 1 -- start start 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93aea2c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93a81029d0 0x7f93a8100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93aea2c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93a81032f0 0x7f93a8100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93aea2c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93a810bd10 0x7f93a81010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93aea2c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f93a81119c0 con 0x7f93a810bd10 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93aea2c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f93a8111840 con 0x7f93a81032f0 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93aea2c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f93a8111b40 con 0x7f93a81029d0 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93a7fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93a81029d0 0x7f93a8100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93acfc9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93a810bd10 0x7f93a81010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93a77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93a81032f0 0x7f93a8100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93a7fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93a81029d0 0x7f93a8100620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:50936/0 (socket says 172.21.15.31:50936) 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93a77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93a81032f0 0x7f93a8100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40172/0 (socket says 172.21.15.31:40172) 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93acfc9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93a810bd10 0x7f93a81010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38674/0 (socket says 172.21.15.31:38674) 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93a7fff640 1 -- 172.21.15.31:0/2886868678 learned_addr learned my addr 172.21.15.31:0/2886868678 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93acfc9640 1 -- 172.21.15.31:0/2886868678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93a81029d0 msgr2=0x7f93a8100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93acfc9640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93a81029d0 0x7f93a8100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93acfc9640 1 -- 172.21.15.31:0/2886868678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93a81032f0 msgr2=0x7f93a8100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93acfc9640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93a81032f0 0x7f93a8100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93acfc9640 1 -- 172.21.15.31:0/2886868678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a81a6360 con 0x7f93a810bd10 2024-04-25T01:54:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93a7fff640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93a81029d0 0x7f93a8100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.188+0000 7f93acfc9640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93a810bd10 0x7f93a81010a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f93a000cab0 tx=0x7f93a000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.191+0000 7f93a57fa640 1 -- 172.21.15.31:0/2886868678 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a000ae20 con 0x7f93a810bd10 2024-04-25T01:54:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.191+0000 7f93aea2c640 1 -- 172.21.15.31:0/2886868678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93a81a6650 con 0x7f93a810bd10 2024-04-25T01:54:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.191+0000 7f93a57fa640 1 -- 172.21.15.31:0/2886868678 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93a000f900 con 0x7f93a810bd10 2024-04-25T01:54:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.191+0000 7f93aea2c640 1 -- 172.21.15.31:0/2886868678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93a81a6ba0 con 0x7f93a810bd10 2024-04-25T01:54:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.191+0000 7f93a57fa640 1 -- 172.21.15.31:0/2886868678 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a0018a00 con 0x7f93a810bd10 2024-04-25T01:54:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.192+0000 7f93aea2c640 1 -- 172.21.15.31:0/2886868678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9394005290 con 0x7f93a810bd10 2024-04-25T01:54:33.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.193+0000 7f93a57fa640 1 -- 172.21.15.31:0/2886868678 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7f93a000f450 con 0x7f93a810bd10 2024-04-25T01:54:33.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.193+0000 7f93a57fa640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93900a2dc0 0x7f93900a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.194+0000 7f93a57fa640 1 -- 172.21.15.31:0/2886868678 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f93a00cfad0 con 0x7f93a810bd10 2024-04-25T01:54:33.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.194+0000 7f93a7fff640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93900a2dc0 0x7f93900a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.194+0000 7f93a7fff640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93900a2dc0 0x7f93900a5280 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9398031440 tx=0x7f93980058d0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:33.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.199+0000 7f93a57fa640 1 -- 172.21.15.31:0/2886868678 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f93a0014030 con 0x7f93a810bd10 2024-04-25T01:54:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.390+0000 7f93aea2c640 1 -- 172.21.15.31:0/2886868678 --> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9394000bc0 con 0x7f93900a2dc0 2024-04-25T01:54:33.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.398+0000 7f93a57fa640 1 -- 172.21.15.31:0/2886868678 <== mgr.15351 v2:172.21.15.31:6800/1834919992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258073 (secure 0 0 0) 0x7f9394000bc0 con 0x7f93900a2dc0 2024-04-25T01:54:33.401 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:33.403 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T01:54:33.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.400+0000 7f93aea2c640 1 -- 172.21.15.31:0/2886868678 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93900a2dc0 msgr2=0x7f93900a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.400+0000 7f93aea2c640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93900a2dc0 0x7f93900a5280 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9398031440 tx=0x7f93980058d0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.401+0000 7f93aea2c640 1 -- 172.21.15.31:0/2886868678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93a810bd10 msgr2=0x7f93a81010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.401+0000 7f93aea2c640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93a810bd10 0x7f93a81010a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f93a000cab0 tx=0x7f93a000cf70 comp rx=0 tx=0).stop 2024-04-25T01:54:33.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.401+0000 7f93aea2c640 1 -- 172.21.15.31:0/2886868678 shutdown_connections 2024-04-25T01:54:33.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.401+0000 7f93aea2c640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93900a2dc0 0x7f93900a5280 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.401+0000 7f93aea2c640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93a810bd10 0x7f93a81010a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.401+0000 7f93aea2c640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93a81032f0 0x7f93a8100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.401+0000 7f93aea2c640 1 --2- 172.21.15.31:0/2886868678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93a81029d0 0x7f93a8100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.401+0000 7f93aea2c640 1 -- 172.21.15.31:0/2886868678 >> 172.21.15.31:0/2886868678 conn(0x7f93a8077740 msgr2=0x7f93a80fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:33.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.401+0000 7f93aea2c640 1 -- 172.21.15.31:0/2886868678 shutdown_connections 2024-04-25T01:54:33.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.401+0000 7f93aea2c640 1 -- 172.21.15.31:0/2886868678 wait complete. 2024-04-25T01:54:33.413 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T01:54:32.598067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193324,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1439,"num_read_kb":888,"num_write":560,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1241782,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":945,"ondisk_log_size":945,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373876,"kb_used_data":12216,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567948,"statfs":{"total":1151940427776,"available":1151557578752,"internally_reserved":0,"allocated":12509184,"data_stored":9261821,"data_compressed":225161,"data_compressed_allocated":294912,"data_compressed_original":3473408,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.984749"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845657+0000","last_change":"2024-04-25T01:52:25.523199+0000","last_active":"2024-04-25T01:54:28.845657+0000","last_peered":"2024-04-25T01:54:28.845657+0000","last_clean":"2024-04-25T01:54:28.845657+0000","last_became_active":"2024-04-25T01:52:25.522052+0000","last_became_peered":"2024-04-25T01:52:25.522052+0000","last_unstale":"2024-04-25T01:54:28.845657+0000","last_undegraded":"2024-04-25T01:54:28.845657+0000","last_fullsized":"2024-04-25T01:54:28.845657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:07:27.095370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846530+0000","last_change":"2024-04-25T01:52:25.528431+0000","last_active":"2024-04-25T01:54:28.846530+0000","last_peered":"2024-04-25T01:54:28.846530+0000","last_clean":"2024-04-25T01:54:28.846530+0000","last_became_active":"2024-04-25T01:52:25.528122+0000","last_became_peered":"2024-04-25T01:52:25.528122+0000","last_unstale":"2024-04-25T01:54:28.846530+0000","last_undegraded":"2024-04-25T01:54:28.846530+0000","last_fullsized":"2024-04-25T01:54:28.846530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:26:56.794582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846475+0000","last_change":"2024-04-25T01:52:25.532963+0000","last_active":"2024-04-25T01:54:28.846475+0000","last_peered":"2024-04-25T01:54:28.846475+0000","last_clean":"2024-04-25T01:54:28.846475+0000","last_became_active":"2024-04-25T01:52:25.532770+0000","last_became_peered":"2024-04-25T01:52:25.532770+0000","last_unstale":"2024-04-25T01:54:28.846475+0000","last_undegraded":"2024-04-25T01:54:28.846475+0000","last_fullsized":"2024-04-25T01:54:28.846475+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:13.398868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585908+0000","last_change":"2024-04-25T01:53:24.968017+0000","last_active":"2024-04-25T01:54:28.585908+0000","last_peered":"2024-04-25T01:54:28.585908+0000","last_clean":"2024-04-25T01:54:28.585908+0000","last_became_active":"2024-04-25T01:53:24.967740+0000","last_became_peered":"2024-04-25T01:53:24.967740+0000","last_unstale":"2024-04-25T01:54:28.585908+0000","last_undegraded":"2024-04-25T01:54:28.585908+0000","last_fullsized":"2024-04-25T01:54:28.585908+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:45.219797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676410+0000","last_change":"2024-04-25T01:52:25.523624+0000","last_active":"2024-04-25T01:54:28.676410+0000","last_peered":"2024-04-25T01:54:28.676410+0000","last_clean":"2024-04-25T01:54:28.676410+0000","last_became_active":"2024-04-25T01:52:25.523289+0000","last_became_peered":"2024-04-25T01:52:25.523289+0000","last_unstale":"2024-04-25T01:54:28.676410+0000","last_undegraded":"2024-04-25T01:54:28.676410+0000","last_fullsized":"2024-04-25T01:54:28.676410+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:09.688472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685245+0000","last_change":"2024-04-25T01:52:25.527123+0000","last_active":"2024-04-25T01:54:28.685245+0000","last_peered":"2024-04-25T01:54:28.685245+0000","last_clean":"2024-04-25T01:54:28.685245+0000","last_became_active":"2024-04-25T01:52:25.526808+0000","last_became_peered":"2024-04-25T01:52:25.526808+0000","last_unstale":"2024-04-25T01:54:28.685245+0000","last_undegraded":"2024-04-25T01:54:28.685245+0000","last_fullsized":"2024-04-25T01:54:28.685245+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:31.654857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837586+0000","last_change":"2024-04-25T01:52:25.517011+0000","last_active":"2024-04-25T01:54:28.837586+0000","last_peered":"2024-04-25T01:54:28.837586+0000","last_clean":"2024-04-25T01:54:28.837586+0000","last_became_active":"2024-04-25T01:52:25.516680+0000","last_became_peered":"2024-04-25T01:52:25.516680+0000","last_unstale":"2024-04-25T01:54:28.837586+0000","last_undegraded":"2024-04-25T01:54:28.837586+0000","last_fullsized":"2024-04-25T01:54:28.837586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:03.388576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676583+0000","last_change":"2024-04-25T01:52:25.531079+0000","last_active":"2024-04-25T01:54:28.676583+0000","last_peered":"2024-04-25T01:54:28.676583+0000","last_clean":"2024-04-25T01:54:28.676583+0000","last_became_active":"2024-04-25T01:52:25.529200+0000","last_became_peered":"2024-04-25T01:52:25.529200+0000","last_unstale":"2024-04-25T01:54:28.676583+0000","last_undegraded":"2024-04-25T01:54:28.676583+0000","last_fullsized":"2024-04-25T01:54:28.676583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:30.484010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.698038+0000","last_change":"2024-04-25T01:52:25.533421+0000","last_active":"2024-04-25T01:54:28.698038+0000","last_peered":"2024-04-25T01:54:28.698038+0000","last_clean":"2024-04-25T01:54:28.698038+0000","last_became_active":"2024-04-25T01:52:25.531781+0000","last_became_peered":"2024-04-25T01:52:25.531781+0000","last_unstale":"2024-04-25T01:54:28.698038+0000","last_undegraded":"2024-04-25T01:54:28.698038+0000","last_fullsized":"2024-04-25T01:54:28.698038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:32.505793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685423+0000","last_change":"2024-04-25T01:52:25.530223+0000","last_active":"2024-04-25T01:54:28.685423+0000","last_peered":"2024-04-25T01:54:28.685423+0000","last_clean":"2024-04-25T01:54:28.685423+0000","last_became_active":"2024-04-25T01:52:25.529909+0000","last_became_peered":"2024-04-25T01:52:25.529909+0000","last_unstale":"2024-04-25T01:54:28.685423+0000","last_undegraded":"2024-04-25T01:54:28.685423+0000","last_fullsized":"2024-04-25T01:54:28.685423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:07.429026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838182+0000","last_change":"2024-04-25T01:52:25.524502+0000","last_active":"2024-04-25T01:54:28.838182+0000","last_peered":"2024-04-25T01:54:28.838182+0000","last_clean":"2024-04-25T01:54:28.838182+0000","last_became_active":"2024-04-25T01:52:25.524018+0000","last_became_peered":"2024-04-25T01:52:25.524018+0000","last_unstale":"2024-04-25T01:54:28.838182+0000","last_undegraded":"2024-04-25T01:54:28.838182+0000","last_fullsized":"2024-04-25T01:54:28.838182+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:23.760389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.587839+0000","last_change":"2024-04-25T01:52:25.533593+0000","last_active":"2024-04-25T01:54:28.587839+0000","last_peered":"2024-04-25T01:54:28.587839+0000","last_clean":"2024-04-25T01:54:28.587839+0000","last_became_active":"2024-04-25T01:52:25.532745+0000","last_became_peered":"2024-04-25T01:52:25.532745+0000","last_unstale":"2024-04-25T01:54:28.587839+0000","last_undegraded":"2024-04-25T01:54:28.587839+0000","last_fullsized":"2024-04-25T01:54:28.587839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:45.083267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908390+0000","last_change":"2024-04-25T01:52:25.528563+0000","last_active":"2024-04-25T01:54:28.908390+0000","last_peered":"2024-04-25T01:54:28.908390+0000","last_clean":"2024-04-25T01:54:28.908390+0000","last_became_active":"2024-04-25T01:52:25.528236+0000","last_became_peered":"2024-04-25T01:52:25.528236+0000","last_unstale":"2024-04-25T01:54:28.908390+0000","last_undegraded":"2024-04-25T01:54:28.908390+0000","last_fullsized":"2024-04-25T01:54:28.908390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:28:27.222799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845935+0000","last_change":"2024-04-25T01:52:25.524951+0000","last_active":"2024-04-25T01:54:28.845935+0000","last_peered":"2024-04-25T01:54:28.845935+0000","last_clean":"2024-04-25T01:54:28.845935+0000","last_became_active":"2024-04-25T01:52:25.524617+0000","last_became_peered":"2024-04-25T01:52:25.524617+0000","last_unstale":"2024-04-25T01:54:28.845935+0000","last_undegraded":"2024-04-25T01:54:28.845935+0000","last_fullsized":"2024-04-25T01:54:28.845935+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:03.561883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845850+0000","last_change":"2024-04-25T01:52:25.532471+0000","last_active":"2024-04-25T01:54:28.845850+0000","last_peered":"2024-04-25T01:54:28.845850+0000","last_clean":"2024-04-25T01:54:28.845850+0000","last_became_active":"2024-04-25T01:52:25.532131+0000","last_became_peered":"2024-04-25T01:52:25.532131+0000","last_unstale":"2024-04-25T01:54:28.845850+0000","last_undegraded":"2024-04-25T01:54:28.845850+0000","last_fullsized":"2024-04-25T01:54:28.845850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:03.478816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697542+0000","last_change":"2024-04-25T01:52:25.529160+0000","last_active":"2024-04-25T01:54:28.697542+0000","last_peered":"2024-04-25T01:54:28.697542+0000","last_clean":"2024-04-25T01:54:28.697542+0000","last_became_active":"2024-04-25T01:52:25.528854+0000","last_became_peered":"2024-04-25T01:52:25.528854+0000","last_unstale":"2024-04-25T01:54:28.697542+0000","last_undegraded":"2024-04-25T01:54:28.697542+0000","last_fullsized":"2024-04-25T01:54:28.697542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:54.713715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845791+0000","last_change":"2024-04-25T01:52:25.523372+0000","last_active":"2024-04-25T01:54:28.845791+0000","last_peered":"2024-04-25T01:54:28.845791+0000","last_clean":"2024-04-25T01:54:28.845791+0000","last_became_active":"2024-04-25T01:52:25.522538+0000","last_became_peered":"2024-04-25T01:52:25.522538+0000","last_unstale":"2024-04-25T01:54:28.845791+0000","last_undegraded":"2024-04-25T01:54:28.845791+0000","last_fullsized":"2024-04-25T01:54:28.845791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:47.213815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585865+0000","last_change":"2024-04-25T01:52:25.522920+0000","last_active":"2024-04-25T01:54:28.585865+0000","last_peered":"2024-04-25T01:54:28.585865+0000","last_clean":"2024-04-25T01:54:28.585865+0000","last_became_active":"2024-04-25T01:52:25.522594+0000","last_became_peered":"2024-04-25T01:52:25.522594+0000","last_unstale":"2024-04-25T01:54:28.585865+0000","last_undegraded":"2024-04-25T01:54:28.585865+0000","last_fullsized":"2024-04-25T01:54:28.585865+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:45.043667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837469+0000","last_change":"2024-04-25T01:52:25.532738+0000","last_active":"2024-04-25T01:54:28.837469+0000","last_peered":"2024-04-25T01:54:28.837469+0000","last_clean":"2024-04-25T01:54:28.837469+0000","last_became_active":"2024-04-25T01:52:25.532237+0000","last_became_peered":"2024-04-25T01:52:25.532237+0000","last_unstale":"2024-04-25T01:54:28.837469+0000","last_undegraded":"2024-04-25T01:54:28.837469+0000","last_fullsized":"2024-04-25T01:54:28.837469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:54.456704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'14","reported_seq":154,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.935594+0000","last_change":"2024-04-25T01:52:24.515686+0000","last_active":"2024-04-25T01:54:28.935594+0000","last_peered":"2024-04-25T01:54:28.935594+0000","last_clean":"2024-04-25T01:54:28.935594+0000","last_became_active":"2024-04-25T01:52:24.515246+0000","last_became_peered":"2024-04-25T01:52:24.515246+0000","last_unstale":"2024-04-25T01:54:28.935594+0000","last_undegraded":"2024-04-25T01:54:28.935594+0000","last_fullsized":"2024-04-25T01:54:28.935594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:14.325747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677009+0000","last_change":"2024-04-25T01:52:25.528796+0000","last_active":"2024-04-25T01:54:28.677009+0000","last_peered":"2024-04-25T01:54:28.677009+0000","last_clean":"2024-04-25T01:54:28.677009+0000","last_became_active":"2024-04-25T01:52:25.528453+0000","last_became_peered":"2024-04-25T01:52:25.528453+0000","last_unstale":"2024-04-25T01:54:28.677009+0000","last_undegraded":"2024-04-25T01:54:28.677009+0000","last_fullsized":"2024-04-25T01:54:28.677009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:29.432519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677315+0000","last_change":"2024-04-25T01:52:24.520229+0000","last_active":"2024-04-25T01:54:28.677315+0000","last_peered":"2024-04-25T01:54:28.677315+0000","last_clean":"2024-04-25T01:54:28.677315+0000","last_became_active":"2024-04-25T01:52:24.519932+0000","last_became_peered":"2024-04-25T01:52:24.519932+0000","last_unstale":"2024-04-25T01:54:28.677315+0000","last_undegraded":"2024-04-25T01:54:28.677315+0000","last_fullsized":"2024-04-25T01:54:28.677315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.786949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846741+0000","last_change":"2024-04-25T01:52:25.532822+0000","last_active":"2024-04-25T01:54:28.846741+0000","last_peered":"2024-04-25T01:54:28.846741+0000","last_clean":"2024-04-25T01:54:28.846741+0000","last_became_active":"2024-04-25T01:52:25.532462+0000","last_became_peered":"2024-04-25T01:52:25.532462+0000","last_unstale":"2024-04-25T01:54:28.846741+0000","last_undegraded":"2024-04-25T01:54:28.846741+0000","last_fullsized":"2024-04-25T01:54:28.846741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:55.569883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'12","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676809+0000","last_change":"2024-04-25T01:52:24.512923+0000","last_active":"2024-04-25T01:54:28.676809+0000","last_peered":"2024-04-25T01:54:28.676809+0000","last_clean":"2024-04-25T01:54:28.676809+0000","last_became_active":"2024-04-25T01:52:24.512604+0000","last_became_peered":"2024-04-25T01:52:24.512604+0000","last_unstale":"2024-04-25T01:54:28.676809+0000","last_undegraded":"2024-04-25T01:54:28.676809+0000","last_fullsized":"2024-04-25T01:54:28.676809+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:41.640234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697690+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:54:28.697690+0000","last_peered":"2024-04-25T01:54:28.697690+0000","last_clean":"2024-04-25T01:54:28.697690+0000","last_became_active":"2024-04-25T01:52:25.526447+0000","last_became_peered":"2024-04-25T01:52:25.526447+0000","last_unstale":"2024-04-25T01:54:28.697690+0000","last_undegraded":"2024-04-25T01:54:28.697690+0000","last_fullsized":"2024-04-25T01:54:28.697690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:10:18.313830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676651+0000","last_change":"2024-04-25T01:52:24.511565+0000","last_active":"2024-04-25T01:54:28.676651+0000","last_peered":"2024-04-25T01:54:28.676651+0000","last_clean":"2024-04-25T01:54:28.676651+0000","last_became_active":"2024-04-25T01:52:24.511258+0000","last_became_peered":"2024-04-25T01:52:24.511258+0000","last_unstale":"2024-04-25T01:54:28.676651+0000","last_undegraded":"2024-04-25T01:54:28.676651+0000","last_fullsized":"2024-04-25T01:54:28.676651+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:38.733338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585663+0000","last_change":"2024-04-25T01:53:25.360965+0000","last_active":"2024-04-25T01:54:28.585663+0000","last_peered":"2024-04-25T01:54:28.585663+0000","last_clean":"2024-04-25T01:54:28.585663+0000","last_became_active":"2024-04-25T01:53:25.360482+0000","last_became_peered":"2024-04-25T01:53:25.360482+0000","last_unstale":"2024-04-25T01:54:28.585663+0000","last_undegraded":"2024-04-25T01:54:28.585663+0000","last_fullsized":"2024-04-25T01:54:28.585663+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:05:01.122687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.947950+0000","last_change":"2024-04-25T01:52:24.514854+0000","last_active":"2024-04-25T01:54:28.947950+0000","last_peered":"2024-04-25T01:54:28.947950+0000","last_clean":"2024-04-25T01:54:28.947950+0000","last_became_active":"2024-04-25T01:52:24.514548+0000","last_became_peered":"2024-04-25T01:52:24.514548+0000","last_unstale":"2024-04-25T01:54:28.947950+0000","last_undegraded":"2024-04-25T01:54:28.947950+0000","last_fullsized":"2024-04-25T01:54:28.947950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:38.539202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1506,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676465+0000","last_change":"2024-04-25T01:52:25.523888+0000","last_active":"2024-04-25T01:54:28.676465+0000","last_peered":"2024-04-25T01:54:28.676465+0000","last_clean":"2024-04-25T01:54:28.676465+0000","last_became_active":"2024-04-25T01:52:25.523581+0000","last_became_peered":"2024-04-25T01:52:25.523581+0000","last_unstale":"2024-04-25T01:54:28.676465+0000","last_undegraded":"2024-04-25T01:54:28.676465+0000","last_fullsized":"2024-04-25T01:54:28.676465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:59.521757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685081+0000","last_change":"2024-04-25T01:53:24.972169+0000","last_active":"2024-04-25T01:54:28.685081+0000","last_peered":"2024-04-25T01:54:28.685081+0000","last_clean":"2024-04-25T01:54:28.685081+0000","last_became_active":"2024-04-25T01:53:24.966425+0000","last_became_peered":"2024-04-25T01:53:24.966425+0000","last_unstale":"2024-04-25T01:54:28.685081+0000","last_undegraded":"2024-04-25T01:54:28.685081+0000","last_fullsized":"2024-04-25T01:54:28.685081+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:31.052476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.586003+0000","last_change":"2024-04-25T01:52:25.523381+0000","last_active":"2024-04-25T01:54:28.586003+0000","last_peered":"2024-04-25T01:54:28.586003+0000","last_clean":"2024-04-25T01:54:28.586003+0000","last_became_active":"2024-04-25T01:52:25.522978+0000","last_became_peered":"2024-04-25T01:52:25.522978+0000","last_unstale":"2024-04-25T01:54:28.586003+0000","last_undegraded":"2024-04-25T01:54:28.586003+0000","last_fullsized":"2024-04-25T01:54:28.586003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:45.702788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908283+0000","last_change":"2024-04-25T01:52:24.521280+0000","last_active":"2024-04-25T01:54:28.908283+0000","last_peered":"2024-04-25T01:54:28.908283+0000","last_clean":"2024-04-25T01:54:28.908283+0000","last_became_active":"2024-04-25T01:52:24.520959+0000","last_became_peered":"2024-04-25T01:52:24.520959+0000","last_unstale":"2024-04-25T01:54:28.908283+0000","last_undegraded":"2024-04-25T01:54:28.908283+0000","last_fullsized":"2024-04-25T01:54:28.908283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:55.037678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676418+0000","last_change":"2024-04-25T01:52:25.524331+0000","last_active":"2024-04-25T01:54:28.676418+0000","last_peered":"2024-04-25T01:54:28.676418+0000","last_clean":"2024-04-25T01:54:28.676418+0000","last_became_active":"2024-04-25T01:52:25.523431+0000","last_became_peered":"2024-04-25T01:52:25.523431+0000","last_unstale":"2024-04-25T01:54:28.676418+0000","last_undegraded":"2024-04-25T01:54:28.676418+0000","last_fullsized":"2024-04-25T01:54:28.676418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:55.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585532+0000","last_change":"2024-04-25T01:52:24.515697+0000","last_active":"2024-04-25T01:54:28.585532+0000","last_peered":"2024-04-25T01:54:28.585532+0000","last_clean":"2024-04-25T01:54:28.585532+0000","last_became_active":"2024-04-25T01:52:24.515285+0000","last_became_peered":"2024-04-25T01:52:24.515285+0000","last_unstale":"2024-04-25T01:54:28.585532+0000","last_undegraded":"2024-04-25T01:54:28.585532+0000","last_fullsized":"2024-04-25T01:54:28.585532+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:24.249203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585591+0000","last_change":"2024-04-25T01:52:25.522415+0000","last_active":"2024-04-25T01:54:28.585591+0000","last_peered":"2024-04-25T01:54:28.585591+0000","last_clean":"2024-04-25T01:54:28.585591+0000","last_became_active":"2024-04-25T01:52:25.521270+0000","last_became_peered":"2024-04-25T01:52:25.521270+0000","last_unstale":"2024-04-25T01:54:28.585591+0000","last_undegraded":"2024-04-25T01:54:28.585591+0000","last_fullsized":"2024-04-25T01:54:28.585591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:53.514169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'7","reported_seq":262,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.947238+0000","last_change":"2024-04-25T01:52:24.509525+0000","last_active":"2024-04-25T01:54:28.947238+0000","last_peered":"2024-04-25T01:54:28.947238+0000","last_clean":"2024-04-25T01:54:28.947238+0000","last_became_active":"2024-04-25T01:52:24.509184+0000","last_became_peered":"2024-04-25T01:52:24.509184+0000","last_unstale":"2024-04-25T01:54:28.947238+0000","last_undegraded":"2024-04-25T01:54:28.947238+0000","last_fullsized":"2024-04-25T01:54:28.947238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:16:43.301401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697979+0000","last_change":"2024-04-25T01:52:25.525293+0000","last_active":"2024-04-25T01:54:28.697979+0000","last_peered":"2024-04-25T01:54:28.697979+0000","last_clean":"2024-04-25T01:54:28.697979+0000","last_became_active":"2024-04-25T01:52:25.524982+0000","last_became_peered":"2024-04-25T01:52:25.524982+0000","last_unstale":"2024-04-25T01:54:28.697979+0000","last_undegraded":"2024-04-25T01:54:28.697979+0000","last_fullsized":"2024-04-25T01:54:28.697979+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:22.222898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838052+0000","last_change":"2024-04-25T01:52:24.519834+0000","last_active":"2024-04-25T01:54:28.838052+0000","last_peered":"2024-04-25T01:54:28.838052+0000","last_clean":"2024-04-25T01:54:28.838052+0000","last_became_active":"2024-04-25T01:52:24.519521+0000","last_became_peered":"2024-04-25T01:52:24.519521+0000","last_unstale":"2024-04-25T01:54:28.838052+0000","last_undegraded":"2024-04-25T01:54:28.838052+0000","last_fullsized":"2024-04-25T01:54:28.838052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:02.414527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846665+0000","last_change":"2024-04-25T01:52:25.523809+0000","last_active":"2024-04-25T01:54:28.846665+0000","last_peered":"2024-04-25T01:54:28.846665+0000","last_clean":"2024-04-25T01:54:28.846665+0000","last_became_active":"2024-04-25T01:52:25.523406+0000","last_became_peered":"2024-04-25T01:52:25.523406+0000","last_unstale":"2024-04-25T01:54:28.846665+0000","last_undegraded":"2024-04-25T01:54:28.846665+0000","last_fullsized":"2024-04-25T01:54:28.846665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:22.169633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585864+0000","last_change":"2024-04-25T01:52:24.510226+0000","last_active":"2024-04-25T01:54:28.585864+0000","last_peered":"2024-04-25T01:54:28.585864+0000","last_clean":"2024-04-25T01:54:28.585864+0000","last_became_active":"2024-04-25T01:52:24.509667+0000","last_became_peered":"2024-04-25T01:52:24.509667+0000","last_unstale":"2024-04-25T01:54:28.585864+0000","last_undegraded":"2024-04-25T01:54:28.585864+0000","last_fullsized":"2024-04-25T01:54:28.585864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:49.180675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697870+0000","last_change":"2024-04-25T01:52:25.531567+0000","last_active":"2024-04-25T01:54:28.697870+0000","last_peered":"2024-04-25T01:54:28.697870+0000","last_clean":"2024-04-25T01:54:28.697870+0000","last_became_active":"2024-04-25T01:52:25.531200+0000","last_became_peered":"2024-04-25T01:52:25.531200+0000","last_unstale":"2024-04-25T01:54:28.697870+0000","last_undegraded":"2024-04-25T01:54:28.697870+0000","last_fullsized":"2024-04-25T01:54:28.697870+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:51:14.336413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'38","reported_seq":323,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:29.457625+0000","last_change":"2024-04-25T01:53:25.369844+0000","last_active":"2024-04-25T01:54:29.457625+0000","last_peered":"2024-04-25T01:54:29.457625+0000","last_clean":"2024-04-25T01:54:29.457625+0000","last_became_active":"2024-04-25T01:53:25.362162+0000","last_became_peered":"2024-04-25T01:53:25.362162+0000","last_unstale":"2024-04-25T01:54:29.457625+0000","last_undegraded":"2024-04-25T01:54:29.457625+0000","last_fullsized":"2024-04-25T01:54:29.457625+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:27:01.730381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66884,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":236,"num_read_kb":183,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676640+0000","last_change":"2024-04-25T01:52:25.525931+0000","last_active":"2024-04-25T01:54:28.676640+0000","last_peered":"2024-04-25T01:54:28.676640+0000","last_clean":"2024-04-25T01:54:28.676640+0000","last_became_active":"2024-04-25T01:52:25.525460+0000","last_became_peered":"2024-04-25T01:52:25.525460+0000","last_unstale":"2024-04-25T01:54:28.676640+0000","last_undegraded":"2024-04-25T01:54:28.676640+0000","last_fullsized":"2024-04-25T01:54:28.676640+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:46.569587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837433+0000","last_change":"2024-04-25T01:52:24.505470+0000","last_active":"2024-04-25T01:54:28.837433+0000","last_peered":"2024-04-25T01:54:28.837433+0000","last_clean":"2024-04-25T01:54:28.837433+0000","last_became_active":"2024-04-25T01:52:24.504677+0000","last_became_peered":"2024-04-25T01:52:24.504677+0000","last_unstale":"2024-04-25T01:54:28.837433+0000","last_undegraded":"2024-04-25T01:54:28.837433+0000","last_fullsized":"2024-04-25T01:54:28.837433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:01.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908209+0000","last_change":"2024-04-25T01:52:25.526364+0000","last_active":"2024-04-25T01:54:28.908209+0000","last_peered":"2024-04-25T01:54:28.908209+0000","last_clean":"2024-04-25T01:54:28.908209+0000","last_became_active":"2024-04-25T01:52:25.525376+0000","last_became_peered":"2024-04-25T01:52:25.525376+0000","last_unstale":"2024-04-25T01:54:28.908209+0000","last_undegraded":"2024-04-25T01:54:28.908209+0000","last_fullsized":"2024-04-25T01:54:28.908209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:26.770994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585776+0000","last_change":"2024-04-25T01:52:24.520287+0000","last_active":"2024-04-25T01:54:28.585776+0000","last_peered":"2024-04-25T01:54:28.585776+0000","last_clean":"2024-04-25T01:54:28.585776+0000","last_became_active":"2024-04-25T01:52:24.519967+0000","last_became_peered":"2024-04-25T01:52:24.519967+0000","last_unstale":"2024-04-25T01:54:28.585776+0000","last_undegraded":"2024-04-25T01:54:28.585776+0000","last_fullsized":"2024-04-25T01:54:28.585776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:45.078410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837933+0000","last_change":"2024-04-25T01:52:25.525766+0000","last_active":"2024-04-25T01:54:28.837933+0000","last_peered":"2024-04-25T01:54:28.837933+0000","last_clean":"2024-04-25T01:54:28.837933+0000","last_became_active":"2024-04-25T01:52:25.525350+0000","last_became_peered":"2024-04-25T01:52:25.525350+0000","last_unstale":"2024-04-25T01:54:28.837933+0000","last_undegraded":"2024-04-25T01:54:28.837933+0000","last_fullsized":"2024-04-25T01:54:28.837933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:05.387502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'10","reported_seq":153,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.936284+0000","last_change":"2024-04-25T01:52:24.513674+0000","last_active":"2024-04-25T01:54:28.936284+0000","last_peered":"2024-04-25T01:54:28.936284+0000","last_clean":"2024-04-25T01:54:28.936284+0000","last_became_active":"2024-04-25T01:52:24.513270+0000","last_became_peered":"2024-04-25T01:52:24.513270+0000","last_unstale":"2024-04-25T01:54:28.936284+0000","last_undegraded":"2024-04-25T01:54:28.936284+0000","last_fullsized":"2024-04-25T01:54:28.936284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:23.385928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":6,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837493+0000","last_change":"2024-04-25T01:52:25.517166+0000","last_active":"2024-04-25T01:54:28.837493+0000","last_peered":"2024-04-25T01:54:28.837493+0000","last_clean":"2024-04-25T01:54:28.837493+0000","last_became_active":"2024-04-25T01:52:25.516853+0000","last_became_peered":"2024-04-25T01:52:25.516853+0000","last_unstale":"2024-04-25T01:54:28.837493+0000","last_undegraded":"2024-04-25T01:54:28.837493+0000","last_fullsized":"2024-04-25T01:54:28.837493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:18.502558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'17","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676369+0000","last_change":"2024-04-25T01:52:24.514695+0000","last_active":"2024-04-25T01:54:28.676369+0000","last_peered":"2024-04-25T01:54:28.676369+0000","last_clean":"2024-04-25T01:54:28.676369+0000","last_became_active":"2024-04-25T01:52:24.514370+0000","last_became_peered":"2024-04-25T01:52:24.514370+0000","last_unstale":"2024-04-25T01:54:28.676369+0000","last_undegraded":"2024-04-25T01:54:28.676369+0000","last_fullsized":"2024-04-25T01:54:28.676369+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:28:28.537307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585698+0000","last_change":"2024-04-25T01:52:25.527062+0000","last_active":"2024-04-25T01:54:28.585698+0000","last_peered":"2024-04-25T01:54:28.585698+0000","last_clean":"2024-04-25T01:54:28.585698+0000","last_became_active":"2024-04-25T01:52:25.526757+0000","last_became_peered":"2024-04-25T01:52:25.526757+0000","last_unstale":"2024-04-25T01:54:28.585698+0000","last_undegraded":"2024-04-25T01:54:28.585698+0000","last_fullsized":"2024-04-25T01:54:28.585698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:36.360030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676786+0000","last_change":"2024-04-25T01:52:24.512235+0000","last_active":"2024-04-25T01:54:28.676786+0000","last_peered":"2024-04-25T01:54:28.676786+0000","last_clean":"2024-04-25T01:54:28.676786+0000","last_became_active":"2024-04-25T01:52:24.511910+0000","last_became_peered":"2024-04-25T01:52:24.511910+0000","last_unstale":"2024-04-25T01:54:28.676786+0000","last_undegraded":"2024-04-25T01:54:28.676786+0000","last_fullsized":"2024-04-25T01:54:28.676786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:38.435734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845541+0000","last_change":"2024-04-25T01:52:25.527854+0000","last_active":"2024-04-25T01:54:28.845541+0000","last_peered":"2024-04-25T01:54:28.845541+0000","last_clean":"2024-04-25T01:54:28.845541+0000","last_became_active":"2024-04-25T01:52:25.527481+0000","last_became_peered":"2024-04-25T01:52:25.527481+0000","last_unstale":"2024-04-25T01:54:28.845541+0000","last_undegraded":"2024-04-25T01:54:28.845541+0000","last_fullsized":"2024-04-25T01:54:28.845541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:09:11.281982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585833+0000","last_change":"2024-04-25T01:52:24.513463+0000","last_active":"2024-04-25T01:54:28.585833+0000","last_peered":"2024-04-25T01:54:28.585833+0000","last_clean":"2024-04-25T01:54:28.585833+0000","last_became_active":"2024-04-25T01:52:24.513176+0000","last_became_peered":"2024-04-25T01:52:24.513176+0000","last_unstale":"2024-04-25T01:54:28.585833+0000","last_undegraded":"2024-04-25T01:54:28.585833+0000","last_fullsized":"2024-04-25T01:54:28.585833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:13:33.507559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685167+0000","last_change":"2024-04-25T01:52:25.528316+0000","last_active":"2024-04-25T01:54:28.685167+0000","last_peered":"2024-04-25T01:54:28.685167+0000","last_clean":"2024-04-25T01:54:28.685167+0000","last_became_active":"2024-04-25T01:52:25.527991+0000","last_became_peered":"2024-04-25T01:52:25.527991+0000","last_unstale":"2024-04-25T01:54:28.685167+0000","last_undegraded":"2024-04-25T01:54:28.685167+0000","last_fullsized":"2024-04-25T01:54:28.685167+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:55.871663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'6","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.936148+0000","last_change":"2024-04-25T01:52:24.519357+0000","last_active":"2024-04-25T01:54:28.936148+0000","last_peered":"2024-04-25T01:54:28.936148+0000","last_clean":"2024-04-25T01:54:28.936148+0000","last_became_active":"2024-04-25T01:52:24.519025+0000","last_became_peered":"2024-04-25T01:52:24.519025+0000","last_unstale":"2024-04-25T01:54:28.936148+0000","last_undegraded":"2024-04-25T01:54:28.936148+0000","last_fullsized":"2024-04-25T01:54:28.936148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:03.476554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685163+0000","last_change":"2024-04-25T01:52:25.527922+0000","last_active":"2024-04-25T01:54:28.685163+0000","last_peered":"2024-04-25T01:54:28.685163+0000","last_clean":"2024-04-25T01:54:28.685163+0000","last_became_active":"2024-04-25T01:52:25.527596+0000","last_became_peered":"2024-04-25T01:52:25.527596+0000","last_unstale":"2024-04-25T01:54:28.685163+0000","last_undegraded":"2024-04-25T01:54:28.685163+0000","last_fullsized":"2024-04-25T01:54:28.685163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:56.002208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585732+0000","last_change":"2024-04-25T01:52:25.533800+0000","last_active":"2024-04-25T01:54:28.585732+0000","last_peered":"2024-04-25T01:54:28.585732+0000","last_clean":"2024-04-25T01:54:28.585732+0000","last_became_active":"2024-04-25T01:52:25.533484+0000","last_became_peered":"2024-04-25T01:52:25.533484+0000","last_unstale":"2024-04-25T01:54:28.585732+0000","last_undegraded":"2024-04-25T01:54:28.585732+0000","last_fullsized":"2024-04-25T01:54:28.585732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:13.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'7","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.935719+0000","last_change":"2024-04-25T01:52:24.506068+0000","last_active":"2024-04-25T01:54:28.935719+0000","last_peered":"2024-04-25T01:54:28.935719+0000","last_clean":"2024-04-25T01:54:28.935719+0000","last_became_active":"2024-04-25T01:52:24.505667+0000","last_became_peered":"2024-04-25T01:52:24.505667+0000","last_unstale":"2024-04-25T01:54:28.935719+0000","last_undegraded":"2024-04-25T01:54:28.935719+0000","last_fullsized":"2024-04-25T01:54:28.935719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:30.494376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676897+0000","last_change":"2024-04-25T01:52:25.530681+0000","last_active":"2024-04-25T01:54:28.676897+0000","last_peered":"2024-04-25T01:54:28.676897+0000","last_clean":"2024-04-25T01:54:28.676897+0000","last_became_active":"2024-04-25T01:52:25.529035+0000","last_became_peered":"2024-04-25T01:52:25.529035+0000","last_unstale":"2024-04-25T01:54:28.676897+0000","last_undegraded":"2024-04-25T01:54:28.676897+0000","last_fullsized":"2024-04-25T01:54:28.676897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:28.787180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'6","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.587716+0000","last_change":"2024-04-25T01:52:24.508948+0000","last_active":"2024-04-25T01:54:28.587716+0000","last_peered":"2024-04-25T01:54:28.587716+0000","last_clean":"2024-04-25T01:54:28.587716+0000","last_became_active":"2024-04-25T01:52:24.508563+0000","last_became_peered":"2024-04-25T01:52:24.508563+0000","last_unstale":"2024-04-25T01:54:28.587716+0000","last_undegraded":"2024-04-25T01:54:28.587716+0000","last_fullsized":"2024-04-25T01:54:28.587716+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:37.591783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":355,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837553+0000","last_change":"2024-04-25T01:51:15.621251+0000","last_active":"2024-04-25T01:54:28.837553+0000","last_peered":"2024-04-25T01:54:28.837553+0000","last_clean":"2024-04-25T01:54:28.837553+0000","last_became_active":"2024-04-25T01:51:15.620771+0000","last_became_peered":"2024-04-25T01:51:15.620771+0000","last_unstale":"2024-04-25T01:54:28.837553+0000","last_undegraded":"2024-04-25T01:54:28.837553+0000","last_fullsized":"2024-04-25T01:54:28.837553+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_clean_scrub_stamp":"2024-04-25T01:47:50.544829+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:44.048065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908179+0000","last_change":"2024-04-25T01:52:25.530216+0000","last_active":"2024-04-25T01:54:28.908179+0000","last_peered":"2024-04-25T01:54:28.908179+0000","last_clean":"2024-04-25T01:54:28.908179+0000","last_became_active":"2024-04-25T01:52:25.529813+0000","last_became_peered":"2024-04-25T01:52:25.529813+0000","last_unstale":"2024-04-25T01:54:28.908179+0000","last_undegraded":"2024-04-25T01:54:28.908179+0000","last_fullsized":"2024-04-25T01:54:28.908179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:57.656375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845310+0000","last_change":"2024-04-25T01:53:24.979114+0000","last_active":"2024-04-25T01:54:28.845310+0000","last_peered":"2024-04-25T01:54:28.845310+0000","last_clean":"2024-04-25T01:54:28.845310+0000","last_became_active":"2024-04-25T01:53:24.972591+0000","last_became_peered":"2024-04-25T01:53:24.972591+0000","last_unstale":"2024-04-25T01:54:28.845310+0000","last_undegraded":"2024-04-25T01:54:28.845310+0000","last_fullsized":"2024-04-25T01:54:28.845310+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:22.792173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676695+0000","last_change":"2024-04-25T01:52:24.514786+0000","last_active":"2024-04-25T01:54:28.676695+0000","last_peered":"2024-04-25T01:54:28.676695+0000","last_clean":"2024-04-25T01:54:28.676695+0000","last_became_active":"2024-04-25T01:52:24.514486+0000","last_became_peered":"2024-04-25T01:52:24.514486+0000","last_unstale":"2024-04-25T01:54:28.676695+0000","last_undegraded":"2024-04-25T01:54:28.676695+0000","last_fullsized":"2024-04-25T01:54:28.676695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:42.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908392+0000","last_change":"2024-04-25T01:52:25.526518+0000","last_active":"2024-04-25T01:54:28.908392+0000","last_peered":"2024-04-25T01:54:28.908392+0000","last_clean":"2024-04-25T01:54:28.908392+0000","last_became_active":"2024-04-25T01:52:25.525917+0000","last_became_peered":"2024-04-25T01:52:25.525917+0000","last_unstale":"2024-04-25T01:54:28.908392+0000","last_undegraded":"2024-04-25T01:54:28.908392+0000","last_fullsized":"2024-04-25T01:54:28.908392+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:20.776385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'48","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.791943+0000","last_change":"2024-04-25T01:52:24.511561+0000","last_active":"2024-04-25T01:54:28.791943+0000","last_peered":"2024-04-25T01:54:28.791943+0000","last_clean":"2024-04-25T01:54:28.791943+0000","last_became_active":"2024-04-25T01:52:24.511165+0000","last_became_peered":"2024-04-25T01:52:24.511165+0000","last_unstale":"2024-04-25T01:54:28.791943+0000","last_undegraded":"2024-04-25T01:54:28.791943+0000","last_fullsized":"2024-04-25T01:54:28.791943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:27:41.006137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677414+0000","last_change":"2024-04-25T01:52:25.527991+0000","last_active":"2024-04-25T01:54:28.677414+0000","last_peered":"2024-04-25T01:54:28.677414+0000","last_clean":"2024-04-25T01:54:28.677414+0000","last_became_active":"2024-04-25T01:52:25.527677+0000","last_became_peered":"2024-04-25T01:52:25.527677+0000","last_unstale":"2024-04-25T01:54:28.677414+0000","last_undegraded":"2024-04-25T01:54:28.677414+0000","last_fullsized":"2024-04-25T01:54:28.677414+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:07:24.698774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.587918+0000","last_change":"2024-04-25T01:52:24.523536+0000","last_active":"2024-04-25T01:54:28.587918+0000","last_peered":"2024-04-25T01:54:28.587918+0000","last_clean":"2024-04-25T01:54:28.587918+0000","last_became_active":"2024-04-25T01:52:24.523233+0000","last_became_peered":"2024-04-25T01:52:24.523233+0000","last_unstale":"2024-04-25T01:54:28.587918+0000","last_undegraded":"2024-04-25T01:54:28.587918+0000","last_fullsized":"2024-04-25T01:54:28.587918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:49:44.444968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846557+0000","last_change":"2024-04-25T01:52:25.527090+0000","last_active":"2024-04-25T01:54:28.846557+0000","last_peered":"2024-04-25T01:54:28.846557+0000","last_clean":"2024-04-25T01:54:28.846557+0000","last_became_active":"2024-04-25T01:52:25.526771+0000","last_became_peered":"2024-04-25T01:52:25.526771+0000","last_unstale":"2024-04-25T01:54:28.846557+0000","last_undegraded":"2024-04-25T01:54:28.846557+0000","last_fullsized":"2024-04-25T01:54:28.846557+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:58.758748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697610+0000","last_change":"2024-04-25T01:52:24.516469+0000","last_active":"2024-04-25T01:54:28.697610+0000","last_peered":"2024-04-25T01:54:28.697610+0000","last_clean":"2024-04-25T01:54:28.697610+0000","last_became_active":"2024-04-25T01:52:24.515948+0000","last_became_peered":"2024-04-25T01:52:24.515948+0000","last_unstale":"2024-04-25T01:54:28.697610+0000","last_undegraded":"2024-04-25T01:54:28.697610+0000","last_fullsized":"2024-04-25T01:54:28.697610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:25.122153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676369+0000","last_change":"2024-04-25T01:52:25.524967+0000","last_active":"2024-04-25T01:54:28.676369+0000","last_peered":"2024-04-25T01:54:28.676369+0000","last_clean":"2024-04-25T01:54:28.676369+0000","last_became_active":"2024-04-25T01:52:25.524670+0000","last_became_peered":"2024-04-25T01:52:25.524670+0000","last_unstale":"2024-04-25T01:54:28.676369+0000","last_undegraded":"2024-04-25T01:54:28.676369+0000","last_fullsized":"2024-04-25T01:54:28.676369+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:08.829052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676711+0000","last_change":"2024-04-25T01:52:24.512187+0000","last_active":"2024-04-25T01:54:28.676711+0000","last_peered":"2024-04-25T01:54:28.676711+0000","last_clean":"2024-04-25T01:54:28.676711+0000","last_became_active":"2024-04-25T01:52:24.510303+0000","last_became_peered":"2024-04-25T01:52:24.510303+0000","last_unstale":"2024-04-25T01:54:28.676711+0000","last_undegraded":"2024-04-25T01:54:28.676711+0000","last_fullsized":"2024-04-25T01:54:28.676711+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:42:31.596355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845405+0000","last_change":"2024-04-25T01:52:25.530386+0000","last_active":"2024-04-25T01:54:28.845405+0000","last_peered":"2024-04-25T01:54:28.845405+0000","last_clean":"2024-04-25T01:54:28.845405+0000","last_became_active":"2024-04-25T01:52:25.530034+0000","last_became_peered":"2024-04-25T01:52:25.530034+0000","last_unstale":"2024-04-25T01:54:28.845405+0000","last_undegraded":"2024-04-25T01:54:28.845405+0000","last_fullsized":"2024-04-25T01:54:28.845405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:11.470123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'4","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845367+0000","last_change":"2024-04-25T01:53:24.976373+0000","last_active":"2024-04-25T01:54:28.845367+0000","last_peered":"2024-04-25T01:54:28.845367+0000","last_clean":"2024-04-25T01:54:28.845367+0000","last_became_active":"2024-04-25T01:53:24.971146+0000","last_became_peered":"2024-04-25T01:53:24.971146+0000","last_unstale":"2024-04-25T01:54:28.845367+0000","last_undegraded":"2024-04-25T01:54:28.845367+0000","last_fullsized":"2024-04-25T01:54:28.845367+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:33.079071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677039+0000","last_change":"2024-04-25T01:52:25.525695+0000","last_active":"2024-04-25T01:54:28.677039+0000","last_peered":"2024-04-25T01:54:28.677039+0000","last_clean":"2024-04-25T01:54:28.677039+0000","last_became_active":"2024-04-25T01:52:25.524830+0000","last_became_peered":"2024-04-25T01:52:25.524830+0000","last_unstale":"2024-04-25T01:54:28.677039+0000","last_undegraded":"2024-04-25T01:54:28.677039+0000","last_fullsized":"2024-04-25T01:54:28.677039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:04:58.449350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676468+0000","last_change":"2024-04-25T01:52:24.519232+0000","last_active":"2024-04-25T01:54:28.676468+0000","last_peered":"2024-04-25T01:54:28.676468+0000","last_clean":"2024-04-25T01:54:28.676468+0000","last_became_active":"2024-04-25T01:52:24.518902+0000","last_became_peered":"2024-04-25T01:52:24.518902+0000","last_unstale":"2024-04-25T01:54:28.676468+0000","last_undegraded":"2024-04-25T01:54:28.676468+0000","last_fullsized":"2024-04-25T01:54:28.676468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.173730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837719+0000","last_change":"2024-04-25T01:52:25.522955+0000","last_active":"2024-04-25T01:54:28.837719+0000","last_peered":"2024-04-25T01:54:28.837719+0000","last_clean":"2024-04-25T01:54:28.837719+0000","last_became_active":"2024-04-25T01:52:25.522626+0000","last_became_peered":"2024-04-25T01:52:25.522626+0000","last_unstale":"2024-04-25T01:54:28.837719+0000","last_undegraded":"2024-04-25T01:54:28.837719+0000","last_fullsized":"2024-04-25T01:54:28.837719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:47.218461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'9","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.947801+0000","last_change":"2024-04-25T01:52:24.516383+0000","last_active":"2024-04-25T01:54:28.947801+0000","last_peered":"2024-04-25T01:54:28.947801+0000","last_clean":"2024-04-25T01:54:28.947801+0000","last_became_active":"2024-04-25T01:52:24.515096+0000","last_became_peered":"2024-04-25T01:52:24.515096+0000","last_unstale":"2024-04-25T01:54:28.947801+0000","last_undegraded":"2024-04-25T01:54:28.947801+0000","last_fullsized":"2024-04-25T01:54:28.947801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:52.658161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2586,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838178+0000","last_change":"2024-04-25T01:52:25.524874+0000","last_active":"2024-04-25T01:54:28.838178+0000","last_peered":"2024-04-25T01:54:28.838178+0000","last_clean":"2024-04-25T01:54:28.838178+0000","last_became_active":"2024-04-25T01:52:25.523532+0000","last_became_peered":"2024-04-25T01:52:25.523532+0000","last_unstale":"2024-04-25T01:54:28.838178+0000","last_undegraded":"2024-04-25T01:54:28.838178+0000","last_fullsized":"2024-04-25T01:54:28.838178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:10.746446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'41","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.789815+0000","last_change":"2024-04-25T01:52:24.526212+0000","last_active":"2024-04-25T01:54:28.789815+0000","last_peered":"2024-04-25T01:54:28.789815+0000","last_clean":"2024-04-25T01:54:28.789815+0000","last_became_active":"2024-04-25T01:52:24.525882+0000","last_became_peered":"2024-04-25T01:52:24.525882+0000","last_unstale":"2024-04-25T01:54:28.789815+0000","last_undegraded":"2024-04-25T01:54:28.789815+0000","last_fullsized":"2024-04-25T01:54:28.789815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:04.792560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837993+0000","last_change":"2024-04-25T01:52:25.517387+0000","last_active":"2024-04-25T01:54:28.837993+0000","last_peered":"2024-04-25T01:54:28.837993+0000","last_clean":"2024-04-25T01:54:28.837993+0000","last_became_active":"2024-04-25T01:52:25.517047+0000","last_became_peered":"2024-04-25T01:52:25.517047+0000","last_unstale":"2024-04-25T01:54:28.837993+0000","last_undegraded":"2024-04-25T01:54:28.837993+0000","last_fullsized":"2024-04-25T01:54:28.837993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:39.462474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908253+0000","last_change":"2024-04-25T01:52:24.506510+0000","last_active":"2024-04-25T01:54:28.908253+0000","last_peered":"2024-04-25T01:54:28.908253+0000","last_clean":"2024-04-25T01:54:28.908253+0000","last_became_active":"2024-04-25T01:52:24.506184+0000","last_became_peered":"2024-04-25T01:52:24.506184+0000","last_unstale":"2024-04-25T01:54:28.908253+0000","last_undegraded":"2024-04-25T01:54:28.908253+0000","last_fullsized":"2024-04-25T01:54:28.908253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:56.754386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838180+0000","last_change":"2024-04-25T01:52:25.525197+0000","last_active":"2024-04-25T01:54:28.838180+0000","last_peered":"2024-04-25T01:54:28.838180+0000","last_clean":"2024-04-25T01:54:28.838180+0000","last_became_active":"2024-04-25T01:52:25.524692+0000","last_became_peered":"2024-04-25T01:52:25.524692+0000","last_unstale":"2024-04-25T01:54:28.838180+0000","last_undegraded":"2024-04-25T01:54:28.838180+0000","last_fullsized":"2024-04-25T01:54:28.838180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.264227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845432+0000","last_change":"2024-04-25T01:52:25.531060+0000","last_active":"2024-04-25T01:54:28.845432+0000","last_peered":"2024-04-25T01:54:28.845432+0000","last_clean":"2024-04-25T01:54:28.845432+0000","last_became_active":"2024-04-25T01:52:25.530774+0000","last_became_peered":"2024-04-25T01:52:25.530774+0000","last_unstale":"2024-04-25T01:54:28.845432+0000","last_undegraded":"2024-04-25T01:54:28.845432+0000","last_fullsized":"2024-04-25T01:54:28.845432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:32.168699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837364+0000","last_change":"2024-04-25T01:52:25.535381+0000","last_active":"2024-04-25T01:54:28.837364+0000","last_peered":"2024-04-25T01:54:28.837364+0000","last_clean":"2024-04-25T01:54:28.837364+0000","last_became_active":"2024-04-25T01:52:25.533623+0000","last_became_peered":"2024-04-25T01:52:25.533623+0000","last_unstale":"2024-04-25T01:54:28.837364+0000","last_undegraded":"2024-04-25T01:54:28.837364+0000","last_fullsized":"2024-04-25T01:54:28.837364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:14.032508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837321+0000","last_change":"2024-04-25T01:52:25.528803+0000","last_active":"2024-04-25T01:54:28.837321+0000","last_peered":"2024-04-25T01:54:28.837321+0000","last_clean":"2024-04-25T01:54:28.837321+0000","last_became_active":"2024-04-25T01:52:25.528505+0000","last_became_peered":"2024-04-25T01:52:25.528505+0000","last_unstale":"2024-04-25T01:54:28.837321+0000","last_undegraded":"2024-04-25T01:54:28.837321+0000","last_fullsized":"2024-04-25T01:54:28.837321+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:35.811181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846209+0000","last_change":"2024-04-25T01:52:25.523620+0000","last_active":"2024-04-25T01:54:28.846209+0000","last_peered":"2024-04-25T01:54:28.846209+0000","last_clean":"2024-04-25T01:54:28.846209+0000","last_became_active":"2024-04-25T01:52:25.523007+0000","last_became_peered":"2024-04-25T01:52:25.523007+0000","last_unstale":"2024-04-25T01:54:28.846209+0000","last_undegraded":"2024-04-25T01:54:28.846209+0000","last_fullsized":"2024-04-25T01:54:28.846209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:23.587838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908298+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:54:28.908298+0000","last_peered":"2024-04-25T01:54:28.908298+0000","last_clean":"2024-04-25T01:54:28.908298+0000","last_became_active":"2024-04-25T01:52:25.526398+0000","last_became_peered":"2024-04-25T01:52:25.526398+0000","last_unstale":"2024-04-25T01:54:28.908298+0000","last_undegraded":"2024-04-25T01:54:28.908298+0000","last_fullsized":"2024-04-25T01:54:28.908298+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:01:37.286676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845498+0000","last_change":"2024-04-25T01:52:25.524999+0000","last_active":"2024-04-25T01:54:28.845498+0000","last_peered":"2024-04-25T01:54:28.845498+0000","last_clean":"2024-04-25T01:54:28.845498+0000","last_became_active":"2024-04-25T01:52:25.524652+0000","last_became_peered":"2024-04-25T01:52:25.524652+0000","last_unstale":"2024-04-25T01:54:28.845498+0000","last_undegraded":"2024-04-25T01:54:28.845498+0000","last_fullsized":"2024-04-25T01:54:28.845498+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:47:37.463701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837706+0000","last_change":"2024-04-25T01:52:25.532996+0000","last_active":"2024-04-25T01:54:28.837706+0000","last_peered":"2024-04-25T01:54:28.837706+0000","last_clean":"2024-04-25T01:54:28.837706+0000","last_became_active":"2024-04-25T01:52:25.530230+0000","last_became_peered":"2024-04-25T01:52:25.530230+0000","last_unstale":"2024-04-25T01:54:28.837706+0000","last_undegraded":"2024-04-25T01:54:28.837706+0000","last_fullsized":"2024-04-25T01:54:28.837706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:08.996434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676996+0000","last_change":"2024-04-25T01:52:25.523838+0000","last_active":"2024-04-25T01:54:28.676996+0000","last_peered":"2024-04-25T01:54:28.676996+0000","last_clean":"2024-04-25T01:54:28.676996+0000","last_became_active":"2024-04-25T01:52:25.523297+0000","last_became_peered":"2024-04-25T01:52:25.523297+0000","last_unstale":"2024-04-25T01:54:28.676996+0000","last_undegraded":"2024-04-25T01:54:28.676996+0000","last_fullsized":"2024-04-25T01:54:28.676996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:52.938763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585617+0000","last_change":"2024-04-25T01:52:25.519028+0000","last_active":"2024-04-25T01:54:28.585617+0000","last_peered":"2024-04-25T01:54:28.585617+0000","last_clean":"2024-04-25T01:54:28.585617+0000","last_became_active":"2024-04-25T01:52:25.518716+0000","last_became_peered":"2024-04-25T01:52:25.518716+0000","last_unstale":"2024-04-25T01:54:28.585617+0000","last_undegraded":"2024-04-25T01:54:28.585617+0000","last_fullsized":"2024-04-25T01:54:28.585617+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:37:31.573403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676618+0000","last_change":"2024-04-25T01:52:25.533894+0000","last_active":"2024-04-25T01:54:28.676618+0000","last_peered":"2024-04-25T01:54:28.676618+0000","last_clean":"2024-04-25T01:54:28.676618+0000","last_became_active":"2024-04-25T01:52:25.533510+0000","last_became_peered":"2024-04-25T01:52:25.533510+0000","last_unstale":"2024-04-25T01:54:28.676618+0000","last_undegraded":"2024-04-25T01:54:28.676618+0000","last_fullsized":"2024-04-25T01:54:28.676618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:07.128296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697784+0000","last_change":"2024-04-25T01:52:25.524693+0000","last_active":"2024-04-25T01:54:28.697784+0000","last_peered":"2024-04-25T01:54:28.697784+0000","last_clean":"2024-04-25T01:54:28.697784+0000","last_became_active":"2024-04-25T01:52:25.524366+0000","last_became_peered":"2024-04-25T01:52:25.524366+0000","last_unstale":"2024-04-25T01:54:28.697784+0000","last_undegraded":"2024-04-25T01:54:28.697784+0000","last_fullsized":"2024-04-25T01:54:28.697784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:23.133722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908579+0000","last_change":"2024-04-25T01:52:25.525936+0000","last_active":"2024-04-25T01:54:28.908579+0000","last_peered":"2024-04-25T01:54:28.908579+0000","last_clean":"2024-04-25T01:54:28.908579+0000","last_became_active":"2024-04-25T01:52:25.525586+0000","last_became_peered":"2024-04-25T01:52:25.525586+0000","last_unstale":"2024-04-25T01:54:28.908579+0000","last_undegraded":"2024-04-25T01:54:28.908579+0000","last_fullsized":"2024-04-25T01:54:28.908579+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:34.756898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845420+0000","last_change":"2024-04-25T01:53:24.970621+0000","last_active":"2024-04-25T01:54:28.845420+0000","last_peered":"2024-04-25T01:54:28.845420+0000","last_clean":"2024-04-25T01:54:28.845420+0000","last_became_active":"2024-04-25T01:53:24.970184+0000","last_became_peered":"2024-04-25T01:53:24.970184+0000","last_unstale":"2024-04-25T01:54:28.845420+0000","last_undegraded":"2024-04-25T01:54:28.845420+0000","last_fullsized":"2024-04-25T01:54:28.845420+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:52:43.072630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106956,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":602722,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1303,"num_read_kb":769,"num_write":377,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":671744,"data_stored":1808166,"data_compressed":89205,"data_compressed_allocated":118784,"data_compressed_original":1622016,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318308,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":478166,"data_compressed":2900,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579946,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32316,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712836,"statfs":{"total":95995035648,"available":95961944064,"internally_reserved":0,"allocated":1167360,"data_stored":1268070,"data_compressed":46559,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743470,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":869896,"data_compressed":22046,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005109,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":870274,"data_compressed":22046,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":1072369,"data_compressed":38552,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":476890,"data_compressed":2900,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":544687,"data_compressed":4751,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":469863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":470150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1001038,"data_compressed":28202,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":539071,"data_compressed":12497,"data_compressed_allocated":16384,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1201347,"data_compressed":44708,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":69129,"data_compressed":1851,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":69391,"data_compressed":9597,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135304,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":202027,"data_compressed":8007,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:54:33.417 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-25T01:54:33.417 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:54:33.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.557+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/373272172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 msgr2=0x7f0b48106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.557+0000 7f0b4ed3b640 1 --2- 172.21.15.31:0/373272172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b48106a10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009cd0 tx=0x7f0b4002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.557+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/373272172 shutdown_connections 2024-04-25T01:54:33.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.557+0000 7f0b4ed3b640 1 --2- 172.21.15.31:0/373272172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100fe0 0x7f0b4810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.557+0000 7f0b4ed3b640 1 --2- 172.21.15.31:0/373272172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100620 0x7f0b48100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.557+0000 7f0b4ed3b640 1 --2- 172.21.15.31:0/373272172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b48106a10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.557+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/373272172 >> 172.21.15.31:0/373272172 conn(0x7f0b48077740 msgr2=0x7f0b48077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:33.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.557+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/373272172 shutdown_connections 2024-04-25T01:54:33.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.557+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/373272172 wait complete. 2024-04-25T01:54:33.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b4ed3b640 1 Processor -- start 2024-04-25T01:54:33.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b4ed3b640 1 -- start start 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b4ed3b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100620 0x7f0b48197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b4ed3b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100fe0 0x7f0b48198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b4ed3b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b4ed3b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0b481108f0 con 0x7f0b48106630 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b4ed3b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b48110770 con 0x7f0b48100fe0 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b4ed3b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0b48110a70 con 0x7f0b48100620 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b4d2d8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b4d2d8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38694/0 (socket says 172.21.15.31:38694) 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b4d2d8640 1 -- 172.21.15.31:0/4273604054 learned_addr learned my addr 172.21.15.31:0/4273604054 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.558+0000 7f0b47fff640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100fe0 0x7f0b48198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b4cad7640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100620 0x7f0b48197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b4d2d8640 1 -- 172.21.15.31:0/4273604054 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100620 msgr2=0x7f0b48197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b4d2d8640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100620 0x7f0b48197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b4d2d8640 1 -- 172.21.15.31:0/4273604054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100fe0 msgr2=0x7f0b48198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b4d2d8640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100fe0 0x7f0b48198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b4d2d8640 1 -- 172.21.15.31:0/4273604054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4819cfb0 con 0x7f0b48106630 2024-04-25T01:54:33.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b4cad7640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100620 0x7f0b48197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:33.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b4d2d8640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c880 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0b3800d750 tx=0x7f0b3800db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:33.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b45ffb640 1 -- 172.21.15.31:0/4273604054 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b38017070 con 0x7f0b48106630 2024-04-25T01:54:33.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b45ffb640 1 -- 172.21.15.31:0/4273604054 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b380097c0 con 0x7f0b48106630 2024-04-25T01:54:33.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/4273604054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4819d2a0 con 0x7f0b48106630 2024-04-25T01:54:33.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b45ffb640 1 -- 172.21.15.31:0/4273604054 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b38004b80 con 0x7f0b48106630 2024-04-25T01:54:33.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.559+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/4273604054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b481a4b30 con 0x7f0b48106630 2024-04-25T01:54:33.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.560+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/4273604054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b48068560 con 0x7f0b48106630 2024-04-25T01:54:33.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.561+0000 7f0b45ffb640 1 -- 172.21.15.31:0/4273604054 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7f0b38004d20 con 0x7f0b48106630 2024-04-25T01:54:33.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.562+0000 7f0b45ffb640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e10 0x7f0b340a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.562+0000 7f0b4cad7640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e10 0x7f0b340a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.562+0000 7f0b45ffb640 1 -- 172.21.15.31:0/4273604054 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0b380cef80 con 0x7f0b48106630 2024-04-25T01:54:33.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.563+0000 7f0b4cad7640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e10 0x7f0b340a52d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b40006010 tx=0x7f0b400093f0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:33.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.567+0000 7f0b45ffb640 1 -- 172.21.15.31:0/4273604054 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0b3809ba20 con 0x7f0b48106630 2024-04-25T01:54:33.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:54:33.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:54:33.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: Updating smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:54:33.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:54:33.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1398454445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: Updating smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:54:33.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:54:33.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:54:33.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.752+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/4273604054 --> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0b48061b80 con 0x7f0b340a2e10 2024-04-25T01:54:33.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.759+0000 7f0b45ffb640 1 -- 172.21.15.31:0/4273604054 <== mgr.15351 v2:172.21.15.31:6800/1834919992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258073 (secure 0 0 0) 0x7f0b48061b80 con 0x7f0b340a2e10 2024-04-25T01:54:33.763 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:33.764 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T01:54:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/4273604054 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e10 msgr2=0x7f0b340a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e10 0x7f0b340a52d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b40006010 tx=0x7f0b400093f0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/4273604054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 msgr2=0x7f0b4819c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c880 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0b3800d750 tx=0x7f0b3800db50 comp rx=0 tx=0).stop 2024-04-25T01:54:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/4273604054 shutdown_connections 2024-04-25T01:54:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e10 0x7f0b340a52d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c880 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100fe0 0x7f0b48198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 --2- 172.21.15.31:0/4273604054 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100620 0x7f0b48197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/4273604054 >> 172.21.15.31:0/4273604054 conn(0x7f0b48077740 msgr2=0x7f0b48068c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:33.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/4273604054 shutdown_connections 2024-04-25T01:54:33.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.763+0000 7f0b4ed3b640 1 -- 172.21.15.31:0/4273604054 wait complete. 2024-04-25T01:54:33.775 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T01:54:32.598067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193324,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1439,"num_read_kb":888,"num_write":560,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1241782,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":945,"ondisk_log_size":945,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373876,"kb_used_data":12216,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567948,"statfs":{"total":1151940427776,"available":1151557578752,"internally_reserved":0,"allocated":12509184,"data_stored":9261821,"data_compressed":225161,"data_compressed_allocated":294912,"data_compressed_original":3473408,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.984749"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845657+0000","last_change":"2024-04-25T01:52:25.523199+0000","last_active":"2024-04-25T01:54:28.845657+0000","last_peered":"2024-04-25T01:54:28.845657+0000","last_clean":"2024-04-25T01:54:28.845657+0000","last_became_active":"2024-04-25T01:52:25.522052+0000","last_became_peered":"2024-04-25T01:52:25.522052+0000","last_unstale":"2024-04-25T01:54:28.845657+0000","last_undegraded":"2024-04-25T01:54:28.845657+0000","last_fullsized":"2024-04-25T01:54:28.845657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:07:27.095370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846530+0000","last_change":"2024-04-25T01:52:25.528431+0000","last_active":"2024-04-25T01:54:28.846530+0000","last_peered":"2024-04-25T01:54:28.846530+0000","last_clean":"2024-04-25T01:54:28.846530+0000","last_became_active":"2024-04-25T01:52:25.528122+0000","last_became_peered":"2024-04-25T01:52:25.528122+0000","last_unstale":"2024-04-25T01:54:28.846530+0000","last_undegraded":"2024-04-25T01:54:28.846530+0000","last_fullsized":"2024-04-25T01:54:28.846530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:26:56.794582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846475+0000","last_change":"2024-04-25T01:52:25.532963+0000","last_active":"2024-04-25T01:54:28.846475+0000","last_peered":"2024-04-25T01:54:28.846475+0000","last_clean":"2024-04-25T01:54:28.846475+0000","last_became_active":"2024-04-25T01:52:25.532770+0000","last_became_peered":"2024-04-25T01:52:25.532770+0000","last_unstale":"2024-04-25T01:54:28.846475+0000","last_undegraded":"2024-04-25T01:54:28.846475+0000","last_fullsized":"2024-04-25T01:54:28.846475+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:13.398868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585908+0000","last_change":"2024-04-25T01:53:24.968017+0000","last_active":"2024-04-25T01:54:28.585908+0000","last_peered":"2024-04-25T01:54:28.585908+0000","last_clean":"2024-04-25T01:54:28.585908+0000","last_became_active":"2024-04-25T01:53:24.967740+0000","last_became_peered":"2024-04-25T01:53:24.967740+0000","last_unstale":"2024-04-25T01:54:28.585908+0000","last_undegraded":"2024-04-25T01:54:28.585908+0000","last_fullsized":"2024-04-25T01:54:28.585908+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:45.219797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676410+0000","last_change":"2024-04-25T01:52:25.523624+0000","last_active":"2024-04-25T01:54:28.676410+0000","last_peered":"2024-04-25T01:54:28.676410+0000","last_clean":"2024-04-25T01:54:28.676410+0000","last_became_active":"2024-04-25T01:52:25.523289+0000","last_became_peered":"2024-04-25T01:52:25.523289+0000","last_unstale":"2024-04-25T01:54:28.676410+0000","last_undegraded":"2024-04-25T01:54:28.676410+0000","last_fullsized":"2024-04-25T01:54:28.676410+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:09.688472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685245+0000","last_change":"2024-04-25T01:52:25.527123+0000","last_active":"2024-04-25T01:54:28.685245+0000","last_peered":"2024-04-25T01:54:28.685245+0000","last_clean":"2024-04-25T01:54:28.685245+0000","last_became_active":"2024-04-25T01:52:25.526808+0000","last_became_peered":"2024-04-25T01:52:25.526808+0000","last_unstale":"2024-04-25T01:54:28.685245+0000","last_undegraded":"2024-04-25T01:54:28.685245+0000","last_fullsized":"2024-04-25T01:54:28.685245+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:31.654857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837586+0000","last_change":"2024-04-25T01:52:25.517011+0000","last_active":"2024-04-25T01:54:28.837586+0000","last_peered":"2024-04-25T01:54:28.837586+0000","last_clean":"2024-04-25T01:54:28.837586+0000","last_became_active":"2024-04-25T01:52:25.516680+0000","last_became_peered":"2024-04-25T01:52:25.516680+0000","last_unstale":"2024-04-25T01:54:28.837586+0000","last_undegraded":"2024-04-25T01:54:28.837586+0000","last_fullsized":"2024-04-25T01:54:28.837586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:03.388576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676583+0000","last_change":"2024-04-25T01:52:25.531079+0000","last_active":"2024-04-25T01:54:28.676583+0000","last_peered":"2024-04-25T01:54:28.676583+0000","last_clean":"2024-04-25T01:54:28.676583+0000","last_became_active":"2024-04-25T01:52:25.529200+0000","last_became_peered":"2024-04-25T01:52:25.529200+0000","last_unstale":"2024-04-25T01:54:28.676583+0000","last_undegraded":"2024-04-25T01:54:28.676583+0000","last_fullsized":"2024-04-25T01:54:28.676583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:30.484010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.698038+0000","last_change":"2024-04-25T01:52:25.533421+0000","last_active":"2024-04-25T01:54:28.698038+0000","last_peered":"2024-04-25T01:54:28.698038+0000","last_clean":"2024-04-25T01:54:28.698038+0000","last_became_active":"2024-04-25T01:52:25.531781+0000","last_became_peered":"2024-04-25T01:52:25.531781+0000","last_unstale":"2024-04-25T01:54:28.698038+0000","last_undegraded":"2024-04-25T01:54:28.698038+0000","last_fullsized":"2024-04-25T01:54:28.698038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:32.505793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685423+0000","last_change":"2024-04-25T01:52:25.530223+0000","last_active":"2024-04-25T01:54:28.685423+0000","last_peered":"2024-04-25T01:54:28.685423+0000","last_clean":"2024-04-25T01:54:28.685423+0000","last_became_active":"2024-04-25T01:52:25.529909+0000","last_became_peered":"2024-04-25T01:52:25.529909+0000","last_unstale":"2024-04-25T01:54:28.685423+0000","last_undegraded":"2024-04-25T01:54:28.685423+0000","last_fullsized":"2024-04-25T01:54:28.685423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:07.429026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838182+0000","last_change":"2024-04-25T01:52:25.524502+0000","last_active":"2024-04-25T01:54:28.838182+0000","last_peered":"2024-04-25T01:54:28.838182+0000","last_clean":"2024-04-25T01:54:28.838182+0000","last_became_active":"2024-04-25T01:52:25.524018+0000","last_became_peered":"2024-04-25T01:52:25.524018+0000","last_unstale":"2024-04-25T01:54:28.838182+0000","last_undegraded":"2024-04-25T01:54:28.838182+0000","last_fullsized":"2024-04-25T01:54:28.838182+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:23.760389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.587839+0000","last_change":"2024-04-25T01:52:25.533593+0000","last_active":"2024-04-25T01:54:28.587839+0000","last_peered":"2024-04-25T01:54:28.587839+0000","last_clean":"2024-04-25T01:54:28.587839+0000","last_became_active":"2024-04-25T01:52:25.532745+0000","last_became_peered":"2024-04-25T01:52:25.532745+0000","last_unstale":"2024-04-25T01:54:28.587839+0000","last_undegraded":"2024-04-25T01:54:28.587839+0000","last_fullsized":"2024-04-25T01:54:28.587839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:45.083267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908390+0000","last_change":"2024-04-25T01:52:25.528563+0000","last_active":"2024-04-25T01:54:28.908390+0000","last_peered":"2024-04-25T01:54:28.908390+0000","last_clean":"2024-04-25T01:54:28.908390+0000","last_became_active":"2024-04-25T01:52:25.528236+0000","last_became_peered":"2024-04-25T01:52:25.528236+0000","last_unstale":"2024-04-25T01:54:28.908390+0000","last_undegraded":"2024-04-25T01:54:28.908390+0000","last_fullsized":"2024-04-25T01:54:28.908390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:28:27.222799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845935+0000","last_change":"2024-04-25T01:52:25.524951+0000","last_active":"2024-04-25T01:54:28.845935+0000","last_peered":"2024-04-25T01:54:28.845935+0000","last_clean":"2024-04-25T01:54:28.845935+0000","last_became_active":"2024-04-25T01:52:25.524617+0000","last_became_peered":"2024-04-25T01:52:25.524617+0000","last_unstale":"2024-04-25T01:54:28.845935+0000","last_undegraded":"2024-04-25T01:54:28.845935+0000","last_fullsized":"2024-04-25T01:54:28.845935+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:03.561883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845850+0000","last_change":"2024-04-25T01:52:25.532471+0000","last_active":"2024-04-25T01:54:28.845850+0000","last_peered":"2024-04-25T01:54:28.845850+0000","last_clean":"2024-04-25T01:54:28.845850+0000","last_became_active":"2024-04-25T01:52:25.532131+0000","last_became_peered":"2024-04-25T01:52:25.532131+0000","last_unstale":"2024-04-25T01:54:28.845850+0000","last_undegraded":"2024-04-25T01:54:28.845850+0000","last_fullsized":"2024-04-25T01:54:28.845850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:03.478816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697542+0000","last_change":"2024-04-25T01:52:25.529160+0000","last_active":"2024-04-25T01:54:28.697542+0000","last_peered":"2024-04-25T01:54:28.697542+0000","last_clean":"2024-04-25T01:54:28.697542+0000","last_became_active":"2024-04-25T01:52:25.528854+0000","last_became_peered":"2024-04-25T01:52:25.528854+0000","last_unstale":"2024-04-25T01:54:28.697542+0000","last_undegraded":"2024-04-25T01:54:28.697542+0000","last_fullsized":"2024-04-25T01:54:28.697542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:54.713715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845791+0000","last_change":"2024-04-25T01:52:25.523372+0000","last_active":"2024-04-25T01:54:28.845791+0000","last_peered":"2024-04-25T01:54:28.845791+0000","last_clean":"2024-04-25T01:54:28.845791+0000","last_became_active":"2024-04-25T01:52:25.522538+0000","last_became_peered":"2024-04-25T01:52:25.522538+0000","last_unstale":"2024-04-25T01:54:28.845791+0000","last_undegraded":"2024-04-25T01:54:28.845791+0000","last_fullsized":"2024-04-25T01:54:28.845791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:47.213815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585865+0000","last_change":"2024-04-25T01:52:25.522920+0000","last_active":"2024-04-25T01:54:28.585865+0000","last_peered":"2024-04-25T01:54:28.585865+0000","last_clean":"2024-04-25T01:54:28.585865+0000","last_became_active":"2024-04-25T01:52:25.522594+0000","last_became_peered":"2024-04-25T01:52:25.522594+0000","last_unstale":"2024-04-25T01:54:28.585865+0000","last_undegraded":"2024-04-25T01:54:28.585865+0000","last_fullsized":"2024-04-25T01:54:28.585865+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:45.043667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837469+0000","last_change":"2024-04-25T01:52:25.532738+0000","last_active":"2024-04-25T01:54:28.837469+0000","last_peered":"2024-04-25T01:54:28.837469+0000","last_clean":"2024-04-25T01:54:28.837469+0000","last_became_active":"2024-04-25T01:52:25.532237+0000","last_became_peered":"2024-04-25T01:52:25.532237+0000","last_unstale":"2024-04-25T01:54:28.837469+0000","last_undegraded":"2024-04-25T01:54:28.837469+0000","last_fullsized":"2024-04-25T01:54:28.837469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:54.456704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'14","reported_seq":154,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.935594+0000","last_change":"2024-04-25T01:52:24.515686+0000","last_active":"2024-04-25T01:54:28.935594+0000","last_peered":"2024-04-25T01:54:28.935594+0000","last_clean":"2024-04-25T01:54:28.935594+0000","last_became_active":"2024-04-25T01:52:24.515246+0000","last_became_peered":"2024-04-25T01:52:24.515246+0000","last_unstale":"2024-04-25T01:54:28.935594+0000","last_undegraded":"2024-04-25T01:54:28.935594+0000","last_fullsized":"2024-04-25T01:54:28.935594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:14.325747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677009+0000","last_change":"2024-04-25T01:52:25.528796+0000","last_active":"2024-04-25T01:54:28.677009+0000","last_peered":"2024-04-25T01:54:28.677009+0000","last_clean":"2024-04-25T01:54:28.677009+0000","last_became_active":"2024-04-25T01:52:25.528453+0000","last_became_peered":"2024-04-25T01:52:25.528453+0000","last_unstale":"2024-04-25T01:54:28.677009+0000","last_undegraded":"2024-04-25T01:54:28.677009+0000","last_fullsized":"2024-04-25T01:54:28.677009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:29.432519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677315+0000","last_change":"2024-04-25T01:52:24.520229+0000","last_active":"2024-04-25T01:54:28.677315+0000","last_peered":"2024-04-25T01:54:28.677315+0000","last_clean":"2024-04-25T01:54:28.677315+0000","last_became_active":"2024-04-25T01:52:24.519932+0000","last_became_peered":"2024-04-25T01:52:24.519932+0000","last_unstale":"2024-04-25T01:54:28.677315+0000","last_undegraded":"2024-04-25T01:54:28.677315+0000","last_fullsized":"2024-04-25T01:54:28.677315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:29.786949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846741+0000","last_change":"2024-04-25T01:52:25.532822+0000","last_active":"2024-04-25T01:54:28.846741+0000","last_peered":"2024-04-25T01:54:28.846741+0000","last_clean":"2024-04-25T01:54:28.846741+0000","last_became_active":"2024-04-25T01:52:25.532462+0000","last_became_peered":"2024-04-25T01:52:25.532462+0000","last_unstale":"2024-04-25T01:54:28.846741+0000","last_undegraded":"2024-04-25T01:54:28.846741+0000","last_fullsized":"2024-04-25T01:54:28.846741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:55.569883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'12","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676809+0000","last_change":"2024-04-25T01:52:24.512923+0000","last_active":"2024-04-25T01:54:28.676809+0000","last_peered":"2024-04-25T01:54:28.676809+0000","last_clean":"2024-04-25T01:54:28.676809+0000","last_became_active":"2024-04-25T01:52:24.512604+0000","last_became_peered":"2024-04-25T01:52:24.512604+0000","last_unstale":"2024-04-25T01:54:28.676809+0000","last_undegraded":"2024-04-25T01:54:28.676809+0000","last_fullsized":"2024-04-25T01:54:28.676809+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:55:41.640234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697690+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:54:28.697690+0000","last_peered":"2024-04-25T01:54:28.697690+0000","last_clean":"2024-04-25T01:54:28.697690+0000","last_became_active":"2024-04-25T01:52:25.526447+0000","last_became_peered":"2024-04-25T01:52:25.526447+0000","last_unstale":"2024-04-25T01:54:28.697690+0000","last_undegraded":"2024-04-25T01:54:28.697690+0000","last_fullsized":"2024-04-25T01:54:28.697690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:10:18.313830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676651+0000","last_change":"2024-04-25T01:52:24.511565+0000","last_active":"2024-04-25T01:54:28.676651+0000","last_peered":"2024-04-25T01:54:28.676651+0000","last_clean":"2024-04-25T01:54:28.676651+0000","last_became_active":"2024-04-25T01:52:24.511258+0000","last_became_peered":"2024-04-25T01:52:24.511258+0000","last_unstale":"2024-04-25T01:54:28.676651+0000","last_undegraded":"2024-04-25T01:54:28.676651+0000","last_fullsized":"2024-04-25T01:54:28.676651+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:38.733338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585663+0000","last_change":"2024-04-25T01:53:25.360965+0000","last_active":"2024-04-25T01:54:28.585663+0000","last_peered":"2024-04-25T01:54:28.585663+0000","last_clean":"2024-04-25T01:54:28.585663+0000","last_became_active":"2024-04-25T01:53:25.360482+0000","last_became_peered":"2024-04-25T01:53:25.360482+0000","last_unstale":"2024-04-25T01:54:28.585663+0000","last_undegraded":"2024-04-25T01:54:28.585663+0000","last_fullsized":"2024-04-25T01:54:28.585663+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:05:01.122687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.947950+0000","last_change":"2024-04-25T01:52:24.514854+0000","last_active":"2024-04-25T01:54:28.947950+0000","last_peered":"2024-04-25T01:54:28.947950+0000","last_clean":"2024-04-25T01:54:28.947950+0000","last_became_active":"2024-04-25T01:52:24.514548+0000","last_became_peered":"2024-04-25T01:52:24.514548+0000","last_unstale":"2024-04-25T01:54:28.947950+0000","last_undegraded":"2024-04-25T01:54:28.947950+0000","last_fullsized":"2024-04-25T01:54:28.947950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:39:38.539202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1506,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676465+0000","last_change":"2024-04-25T01:52:25.523888+0000","last_active":"2024-04-25T01:54:28.676465+0000","last_peered":"2024-04-25T01:54:28.676465+0000","last_clean":"2024-04-25T01:54:28.676465+0000","last_became_active":"2024-04-25T01:52:25.523581+0000","last_became_peered":"2024-04-25T01:52:25.523581+0000","last_unstale":"2024-04-25T01:54:28.676465+0000","last_undegraded":"2024-04-25T01:54:28.676465+0000","last_fullsized":"2024-04-25T01:54:28.676465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:59.521757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'7","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685081+0000","last_change":"2024-04-25T01:53:24.972169+0000","last_active":"2024-04-25T01:54:28.685081+0000","last_peered":"2024-04-25T01:54:28.685081+0000","last_clean":"2024-04-25T01:54:28.685081+0000","last_became_active":"2024-04-25T01:53:24.966425+0000","last_became_peered":"2024-04-25T01:53:24.966425+0000","last_unstale":"2024-04-25T01:54:28.685081+0000","last_undegraded":"2024-04-25T01:54:28.685081+0000","last_fullsized":"2024-04-25T01:54:28.685081+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:48:31.052476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.586003+0000","last_change":"2024-04-25T01:52:25.523381+0000","last_active":"2024-04-25T01:54:28.586003+0000","last_peered":"2024-04-25T01:54:28.586003+0000","last_clean":"2024-04-25T01:54:28.586003+0000","last_became_active":"2024-04-25T01:52:25.522978+0000","last_became_peered":"2024-04-25T01:52:25.522978+0000","last_unstale":"2024-04-25T01:54:28.586003+0000","last_undegraded":"2024-04-25T01:54:28.586003+0000","last_fullsized":"2024-04-25T01:54:28.586003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:45.702788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908283+0000","last_change":"2024-04-25T01:52:24.521280+0000","last_active":"2024-04-25T01:54:28.908283+0000","last_peered":"2024-04-25T01:54:28.908283+0000","last_clean":"2024-04-25T01:54:28.908283+0000","last_became_active":"2024-04-25T01:52:24.520959+0000","last_became_peered":"2024-04-25T01:52:24.520959+0000","last_unstale":"2024-04-25T01:54:28.908283+0000","last_undegraded":"2024-04-25T01:54:28.908283+0000","last_fullsized":"2024-04-25T01:54:28.908283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:55.037678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676418+0000","last_change":"2024-04-25T01:52:25.524331+0000","last_active":"2024-04-25T01:54:28.676418+0000","last_peered":"2024-04-25T01:54:28.676418+0000","last_clean":"2024-04-25T01:54:28.676418+0000","last_became_active":"2024-04-25T01:52:25.523431+0000","last_became_peered":"2024-04-25T01:52:25.523431+0000","last_unstale":"2024-04-25T01:54:28.676418+0000","last_undegraded":"2024-04-25T01:54:28.676418+0000","last_fullsized":"2024-04-25T01:54:28.676418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:20:55.005784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585532+0000","last_change":"2024-04-25T01:52:24.515697+0000","last_active":"2024-04-25T01:54:28.585532+0000","last_peered":"2024-04-25T01:54:28.585532+0000","last_clean":"2024-04-25T01:54:28.585532+0000","last_became_active":"2024-04-25T01:52:24.515285+0000","last_became_peered":"2024-04-25T01:52:24.515285+0000","last_unstale":"2024-04-25T01:54:28.585532+0000","last_undegraded":"2024-04-25T01:54:28.585532+0000","last_fullsized":"2024-04-25T01:54:28.585532+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:52:24.249203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585591+0000","last_change":"2024-04-25T01:52:25.522415+0000","last_active":"2024-04-25T01:54:28.585591+0000","last_peered":"2024-04-25T01:54:28.585591+0000","last_clean":"2024-04-25T01:54:28.585591+0000","last_became_active":"2024-04-25T01:52:25.521270+0000","last_became_peered":"2024-04-25T01:52:25.521270+0000","last_unstale":"2024-04-25T01:54:28.585591+0000","last_undegraded":"2024-04-25T01:54:28.585591+0000","last_fullsized":"2024-04-25T01:54:28.585591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:53.514169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'7","reported_seq":262,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.947238+0000","last_change":"2024-04-25T01:52:24.509525+0000","last_active":"2024-04-25T01:54:28.947238+0000","last_peered":"2024-04-25T01:54:28.947238+0000","last_clean":"2024-04-25T01:54:28.947238+0000","last_became_active":"2024-04-25T01:52:24.509184+0000","last_became_peered":"2024-04-25T01:52:24.509184+0000","last_unstale":"2024-04-25T01:54:28.947238+0000","last_undegraded":"2024-04-25T01:54:28.947238+0000","last_fullsized":"2024-04-25T01:54:28.947238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:16:43.301401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697979+0000","last_change":"2024-04-25T01:52:25.525293+0000","last_active":"2024-04-25T01:54:28.697979+0000","last_peered":"2024-04-25T01:54:28.697979+0000","last_clean":"2024-04-25T01:54:28.697979+0000","last_became_active":"2024-04-25T01:52:25.524982+0000","last_became_peered":"2024-04-25T01:52:25.524982+0000","last_unstale":"2024-04-25T01:54:28.697979+0000","last_undegraded":"2024-04-25T01:54:28.697979+0000","last_fullsized":"2024-04-25T01:54:28.697979+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:22.222898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838052+0000","last_change":"2024-04-25T01:52:24.519834+0000","last_active":"2024-04-25T01:54:28.838052+0000","last_peered":"2024-04-25T01:54:28.838052+0000","last_clean":"2024-04-25T01:54:28.838052+0000","last_became_active":"2024-04-25T01:52:24.519521+0000","last_became_peered":"2024-04-25T01:52:24.519521+0000","last_unstale":"2024-04-25T01:54:28.838052+0000","last_undegraded":"2024-04-25T01:54:28.838052+0000","last_fullsized":"2024-04-25T01:54:28.838052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:02.414527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846665+0000","last_change":"2024-04-25T01:52:25.523809+0000","last_active":"2024-04-25T01:54:28.846665+0000","last_peered":"2024-04-25T01:54:28.846665+0000","last_clean":"2024-04-25T01:54:28.846665+0000","last_became_active":"2024-04-25T01:52:25.523406+0000","last_became_peered":"2024-04-25T01:52:25.523406+0000","last_unstale":"2024-04-25T01:54:28.846665+0000","last_undegraded":"2024-04-25T01:54:28.846665+0000","last_fullsized":"2024-04-25T01:54:28.846665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:22.169633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585864+0000","last_change":"2024-04-25T01:52:24.510226+0000","last_active":"2024-04-25T01:54:28.585864+0000","last_peered":"2024-04-25T01:54:28.585864+0000","last_clean":"2024-04-25T01:54:28.585864+0000","last_became_active":"2024-04-25T01:52:24.509667+0000","last_became_peered":"2024-04-25T01:52:24.509667+0000","last_unstale":"2024-04-25T01:54:28.585864+0000","last_undegraded":"2024-04-25T01:54:28.585864+0000","last_fullsized":"2024-04-25T01:54:28.585864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:49.180675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697870+0000","last_change":"2024-04-25T01:52:25.531567+0000","last_active":"2024-04-25T01:54:28.697870+0000","last_peered":"2024-04-25T01:54:28.697870+0000","last_clean":"2024-04-25T01:54:28.697870+0000","last_became_active":"2024-04-25T01:52:25.531200+0000","last_became_peered":"2024-04-25T01:52:25.531200+0000","last_unstale":"2024-04-25T01:54:28.697870+0000","last_undegraded":"2024-04-25T01:54:28.697870+0000","last_fullsized":"2024-04-25T01:54:28.697870+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:51:14.336413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'38","reported_seq":323,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:29.457625+0000","last_change":"2024-04-25T01:53:25.369844+0000","last_active":"2024-04-25T01:54:29.457625+0000","last_peered":"2024-04-25T01:54:29.457625+0000","last_clean":"2024-04-25T01:54:29.457625+0000","last_became_active":"2024-04-25T01:53:25.362162+0000","last_became_peered":"2024-04-25T01:53:25.362162+0000","last_unstale":"2024-04-25T01:54:29.457625+0000","last_undegraded":"2024-04-25T01:54:29.457625+0000","last_fullsized":"2024-04-25T01:54:29.457625+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:27:01.730381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66884,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":236,"num_read_kb":183,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676640+0000","last_change":"2024-04-25T01:52:25.525931+0000","last_active":"2024-04-25T01:54:28.676640+0000","last_peered":"2024-04-25T01:54:28.676640+0000","last_clean":"2024-04-25T01:54:28.676640+0000","last_became_active":"2024-04-25T01:52:25.525460+0000","last_became_peered":"2024-04-25T01:52:25.525460+0000","last_unstale":"2024-04-25T01:54:28.676640+0000","last_undegraded":"2024-04-25T01:54:28.676640+0000","last_fullsized":"2024-04-25T01:54:28.676640+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:09:46.569587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837433+0000","last_change":"2024-04-25T01:52:24.505470+0000","last_active":"2024-04-25T01:54:28.837433+0000","last_peered":"2024-04-25T01:54:28.837433+0000","last_clean":"2024-04-25T01:54:28.837433+0000","last_became_active":"2024-04-25T01:52:24.504677+0000","last_became_peered":"2024-04-25T01:52:24.504677+0000","last_unstale":"2024-04-25T01:54:28.837433+0000","last_undegraded":"2024-04-25T01:54:28.837433+0000","last_fullsized":"2024-04-25T01:54:28.837433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:01.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908209+0000","last_change":"2024-04-25T01:52:25.526364+0000","last_active":"2024-04-25T01:54:28.908209+0000","last_peered":"2024-04-25T01:54:28.908209+0000","last_clean":"2024-04-25T01:54:28.908209+0000","last_became_active":"2024-04-25T01:52:25.525376+0000","last_became_peered":"2024-04-25T01:52:25.525376+0000","last_unstale":"2024-04-25T01:54:28.908209+0000","last_undegraded":"2024-04-25T01:54:28.908209+0000","last_fullsized":"2024-04-25T01:54:28.908209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:26.770994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585776+0000","last_change":"2024-04-25T01:52:24.520287+0000","last_active":"2024-04-25T01:54:28.585776+0000","last_peered":"2024-04-25T01:54:28.585776+0000","last_clean":"2024-04-25T01:54:28.585776+0000","last_became_active":"2024-04-25T01:52:24.519967+0000","last_became_peered":"2024-04-25T01:52:24.519967+0000","last_unstale":"2024-04-25T01:54:28.585776+0000","last_undegraded":"2024-04-25T01:54:28.585776+0000","last_fullsized":"2024-04-25T01:54:28.585776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:45.078410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837933+0000","last_change":"2024-04-25T01:52:25.525766+0000","last_active":"2024-04-25T01:54:28.837933+0000","last_peered":"2024-04-25T01:54:28.837933+0000","last_clean":"2024-04-25T01:54:28.837933+0000","last_became_active":"2024-04-25T01:52:25.525350+0000","last_became_peered":"2024-04-25T01:52:25.525350+0000","last_unstale":"2024-04-25T01:54:28.837933+0000","last_undegraded":"2024-04-25T01:54:28.837933+0000","last_fullsized":"2024-04-25T01:54:28.837933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:05.387502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'10","reported_seq":153,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.936284+0000","last_change":"2024-04-25T01:52:24.513674+0000","last_active":"2024-04-25T01:54:28.936284+0000","last_peered":"2024-04-25T01:54:28.936284+0000","last_clean":"2024-04-25T01:54:28.936284+0000","last_became_active":"2024-04-25T01:52:24.513270+0000","last_became_peered":"2024-04-25T01:52:24.513270+0000","last_unstale":"2024-04-25T01:54:28.936284+0000","last_undegraded":"2024-04-25T01:54:28.936284+0000","last_fullsized":"2024-04-25T01:54:28.936284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:23.385928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":6,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837493+0000","last_change":"2024-04-25T01:52:25.517166+0000","last_active":"2024-04-25T01:54:28.837493+0000","last_peered":"2024-04-25T01:54:28.837493+0000","last_clean":"2024-04-25T01:54:28.837493+0000","last_became_active":"2024-04-25T01:52:25.516853+0000","last_became_peered":"2024-04-25T01:52:25.516853+0000","last_unstale":"2024-04-25T01:54:28.837493+0000","last_undegraded":"2024-04-25T01:54:28.837493+0000","last_fullsized":"2024-04-25T01:54:28.837493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:18.502558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'17","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676369+0000","last_change":"2024-04-25T01:52:24.514695+0000","last_active":"2024-04-25T01:54:28.676369+0000","last_peered":"2024-04-25T01:54:28.676369+0000","last_clean":"2024-04-25T01:54:28.676369+0000","last_became_active":"2024-04-25T01:52:24.514370+0000","last_became_peered":"2024-04-25T01:52:24.514370+0000","last_unstale":"2024-04-25T01:54:28.676369+0000","last_undegraded":"2024-04-25T01:54:28.676369+0000","last_fullsized":"2024-04-25T01:54:28.676369+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:28:28.537307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585698+0000","last_change":"2024-04-25T01:52:25.527062+0000","last_active":"2024-04-25T01:54:28.585698+0000","last_peered":"2024-04-25T01:54:28.585698+0000","last_clean":"2024-04-25T01:54:28.585698+0000","last_became_active":"2024-04-25T01:52:25.526757+0000","last_became_peered":"2024-04-25T01:52:25.526757+0000","last_unstale":"2024-04-25T01:54:28.585698+0000","last_undegraded":"2024-04-25T01:54:28.585698+0000","last_fullsized":"2024-04-25T01:54:28.585698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:36.360030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676786+0000","last_change":"2024-04-25T01:52:24.512235+0000","last_active":"2024-04-25T01:54:28.676786+0000","last_peered":"2024-04-25T01:54:28.676786+0000","last_clean":"2024-04-25T01:54:28.676786+0000","last_became_active":"2024-04-25T01:52:24.511910+0000","last_became_peered":"2024-04-25T01:52:24.511910+0000","last_unstale":"2024-04-25T01:54:28.676786+0000","last_undegraded":"2024-04-25T01:54:28.676786+0000","last_fullsized":"2024-04-25T01:54:28.676786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:38.435734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845541+0000","last_change":"2024-04-25T01:52:25.527854+0000","last_active":"2024-04-25T01:54:28.845541+0000","last_peered":"2024-04-25T01:54:28.845541+0000","last_clean":"2024-04-25T01:54:28.845541+0000","last_became_active":"2024-04-25T01:52:25.527481+0000","last_became_peered":"2024-04-25T01:52:25.527481+0000","last_unstale":"2024-04-25T01:54:28.845541+0000","last_undegraded":"2024-04-25T01:54:28.845541+0000","last_fullsized":"2024-04-25T01:54:28.845541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:09:11.281982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585833+0000","last_change":"2024-04-25T01:52:24.513463+0000","last_active":"2024-04-25T01:54:28.585833+0000","last_peered":"2024-04-25T01:54:28.585833+0000","last_clean":"2024-04-25T01:54:28.585833+0000","last_became_active":"2024-04-25T01:52:24.513176+0000","last_became_peered":"2024-04-25T01:52:24.513176+0000","last_unstale":"2024-04-25T01:54:28.585833+0000","last_undegraded":"2024-04-25T01:54:28.585833+0000","last_fullsized":"2024-04-25T01:54:28.585833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:13:33.507559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685167+0000","last_change":"2024-04-25T01:52:25.528316+0000","last_active":"2024-04-25T01:54:28.685167+0000","last_peered":"2024-04-25T01:54:28.685167+0000","last_clean":"2024-04-25T01:54:28.685167+0000","last_became_active":"2024-04-25T01:52:25.527991+0000","last_became_peered":"2024-04-25T01:52:25.527991+0000","last_unstale":"2024-04-25T01:54:28.685167+0000","last_undegraded":"2024-04-25T01:54:28.685167+0000","last_fullsized":"2024-04-25T01:54:28.685167+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:55.871663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'6","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.936148+0000","last_change":"2024-04-25T01:52:24.519357+0000","last_active":"2024-04-25T01:54:28.936148+0000","last_peered":"2024-04-25T01:54:28.936148+0000","last_clean":"2024-04-25T01:54:28.936148+0000","last_became_active":"2024-04-25T01:52:24.519025+0000","last_became_peered":"2024-04-25T01:52:24.519025+0000","last_unstale":"2024-04-25T01:54:28.936148+0000","last_undegraded":"2024-04-25T01:54:28.936148+0000","last_fullsized":"2024-04-25T01:54:28.936148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:03.476554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.685163+0000","last_change":"2024-04-25T01:52:25.527922+0000","last_active":"2024-04-25T01:54:28.685163+0000","last_peered":"2024-04-25T01:54:28.685163+0000","last_clean":"2024-04-25T01:54:28.685163+0000","last_became_active":"2024-04-25T01:52:25.527596+0000","last_became_peered":"2024-04-25T01:52:25.527596+0000","last_unstale":"2024-04-25T01:54:28.685163+0000","last_undegraded":"2024-04-25T01:54:28.685163+0000","last_fullsized":"2024-04-25T01:54:28.685163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:11:56.002208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585732+0000","last_change":"2024-04-25T01:52:25.533800+0000","last_active":"2024-04-25T01:54:28.585732+0000","last_peered":"2024-04-25T01:54:28.585732+0000","last_clean":"2024-04-25T01:54:28.585732+0000","last_became_active":"2024-04-25T01:52:25.533484+0000","last_became_peered":"2024-04-25T01:52:25.533484+0000","last_unstale":"2024-04-25T01:54:28.585732+0000","last_undegraded":"2024-04-25T01:54:28.585732+0000","last_fullsized":"2024-04-25T01:54:28.585732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:13.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'7","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.935719+0000","last_change":"2024-04-25T01:52:24.506068+0000","last_active":"2024-04-25T01:54:28.935719+0000","last_peered":"2024-04-25T01:54:28.935719+0000","last_clean":"2024-04-25T01:54:28.935719+0000","last_became_active":"2024-04-25T01:52:24.505667+0000","last_became_peered":"2024-04-25T01:52:24.505667+0000","last_unstale":"2024-04-25T01:54:28.935719+0000","last_undegraded":"2024-04-25T01:54:28.935719+0000","last_fullsized":"2024-04-25T01:54:28.935719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:30.494376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676897+0000","last_change":"2024-04-25T01:52:25.530681+0000","last_active":"2024-04-25T01:54:28.676897+0000","last_peered":"2024-04-25T01:54:28.676897+0000","last_clean":"2024-04-25T01:54:28.676897+0000","last_became_active":"2024-04-25T01:52:25.529035+0000","last_became_peered":"2024-04-25T01:52:25.529035+0000","last_unstale":"2024-04-25T01:54:28.676897+0000","last_undegraded":"2024-04-25T01:54:28.676897+0000","last_fullsized":"2024-04-25T01:54:28.676897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:28.787180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'6","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.587716+0000","last_change":"2024-04-25T01:52:24.508948+0000","last_active":"2024-04-25T01:54:28.587716+0000","last_peered":"2024-04-25T01:54:28.587716+0000","last_clean":"2024-04-25T01:54:28.587716+0000","last_became_active":"2024-04-25T01:52:24.508563+0000","last_became_peered":"2024-04-25T01:52:24.508563+0000","last_unstale":"2024-04-25T01:54:28.587716+0000","last_undegraded":"2024-04-25T01:54:28.587716+0000","last_fullsized":"2024-04-25T01:54:28.587716+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:33:37.591783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":355,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837553+0000","last_change":"2024-04-25T01:51:15.621251+0000","last_active":"2024-04-25T01:54:28.837553+0000","last_peered":"2024-04-25T01:54:28.837553+0000","last_clean":"2024-04-25T01:54:28.837553+0000","last_became_active":"2024-04-25T01:51:15.620771+0000","last_became_peered":"2024-04-25T01:51:15.620771+0000","last_unstale":"2024-04-25T01:54:28.837553+0000","last_undegraded":"2024-04-25T01:54:28.837553+0000","last_fullsized":"2024-04-25T01:54:28.837553+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:50.544829+0000","last_clean_scrub_stamp":"2024-04-25T01:47:50.544829+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:44.048065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908179+0000","last_change":"2024-04-25T01:52:25.530216+0000","last_active":"2024-04-25T01:54:28.908179+0000","last_peered":"2024-04-25T01:54:28.908179+0000","last_clean":"2024-04-25T01:54:28.908179+0000","last_became_active":"2024-04-25T01:52:25.529813+0000","last_became_peered":"2024-04-25T01:52:25.529813+0000","last_unstale":"2024-04-25T01:54:28.908179+0000","last_undegraded":"2024-04-25T01:54:28.908179+0000","last_fullsized":"2024-04-25T01:54:28.908179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:57.656375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845310+0000","last_change":"2024-04-25T01:53:24.979114+0000","last_active":"2024-04-25T01:54:28.845310+0000","last_peered":"2024-04-25T01:54:28.845310+0000","last_clean":"2024-04-25T01:54:28.845310+0000","last_became_active":"2024-04-25T01:53:24.972591+0000","last_became_peered":"2024-04-25T01:53:24.972591+0000","last_unstale":"2024-04-25T01:54:28.845310+0000","last_undegraded":"2024-04-25T01:54:28.845310+0000","last_fullsized":"2024-04-25T01:54:28.845310+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:27:22.792173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676695+0000","last_change":"2024-04-25T01:52:24.514786+0000","last_active":"2024-04-25T01:54:28.676695+0000","last_peered":"2024-04-25T01:54:28.676695+0000","last_clean":"2024-04-25T01:54:28.676695+0000","last_became_active":"2024-04-25T01:52:24.514486+0000","last_became_peered":"2024-04-25T01:52:24.514486+0000","last_unstale":"2024-04-25T01:54:28.676695+0000","last_undegraded":"2024-04-25T01:54:28.676695+0000","last_fullsized":"2024-04-25T01:54:28.676695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:42.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908392+0000","last_change":"2024-04-25T01:52:25.526518+0000","last_active":"2024-04-25T01:54:28.908392+0000","last_peered":"2024-04-25T01:54:28.908392+0000","last_clean":"2024-04-25T01:54:28.908392+0000","last_became_active":"2024-04-25T01:52:25.525917+0000","last_became_peered":"2024-04-25T01:52:25.525917+0000","last_unstale":"2024-04-25T01:54:28.908392+0000","last_undegraded":"2024-04-25T01:54:28.908392+0000","last_fullsized":"2024-04-25T01:54:28.908392+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:20.776385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'48","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.791943+0000","last_change":"2024-04-25T01:52:24.511561+0000","last_active":"2024-04-25T01:54:28.791943+0000","last_peered":"2024-04-25T01:54:28.791943+0000","last_clean":"2024-04-25T01:54:28.791943+0000","last_became_active":"2024-04-25T01:52:24.511165+0000","last_became_peered":"2024-04-25T01:52:24.511165+0000","last_unstale":"2024-04-25T01:54:28.791943+0000","last_undegraded":"2024-04-25T01:54:28.791943+0000","last_fullsized":"2024-04-25T01:54:28.791943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:27:41.006137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677414+0000","last_change":"2024-04-25T01:52:25.527991+0000","last_active":"2024-04-25T01:54:28.677414+0000","last_peered":"2024-04-25T01:54:28.677414+0000","last_clean":"2024-04-25T01:54:28.677414+0000","last_became_active":"2024-04-25T01:52:25.527677+0000","last_became_peered":"2024-04-25T01:52:25.527677+0000","last_unstale":"2024-04-25T01:54:28.677414+0000","last_undegraded":"2024-04-25T01:54:28.677414+0000","last_fullsized":"2024-04-25T01:54:28.677414+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:07:24.698774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.587918+0000","last_change":"2024-04-25T01:52:24.523536+0000","last_active":"2024-04-25T01:54:28.587918+0000","last_peered":"2024-04-25T01:54:28.587918+0000","last_clean":"2024-04-25T01:54:28.587918+0000","last_became_active":"2024-04-25T01:52:24.523233+0000","last_became_peered":"2024-04-25T01:52:24.523233+0000","last_unstale":"2024-04-25T01:54:28.587918+0000","last_undegraded":"2024-04-25T01:54:28.587918+0000","last_fullsized":"2024-04-25T01:54:28.587918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:49:44.444968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846557+0000","last_change":"2024-04-25T01:52:25.527090+0000","last_active":"2024-04-25T01:54:28.846557+0000","last_peered":"2024-04-25T01:54:28.846557+0000","last_clean":"2024-04-25T01:54:28.846557+0000","last_became_active":"2024-04-25T01:52:25.526771+0000","last_became_peered":"2024-04-25T01:52:25.526771+0000","last_unstale":"2024-04-25T01:54:28.846557+0000","last_undegraded":"2024-04-25T01:54:28.846557+0000","last_fullsized":"2024-04-25T01:54:28.846557+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:58.758748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697610+0000","last_change":"2024-04-25T01:52:24.516469+0000","last_active":"2024-04-25T01:54:28.697610+0000","last_peered":"2024-04-25T01:54:28.697610+0000","last_clean":"2024-04-25T01:54:28.697610+0000","last_became_active":"2024-04-25T01:52:24.515948+0000","last_became_peered":"2024-04-25T01:52:24.515948+0000","last_unstale":"2024-04-25T01:54:28.697610+0000","last_undegraded":"2024-04-25T01:54:28.697610+0000","last_fullsized":"2024-04-25T01:54:28.697610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:25.122153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676369+0000","last_change":"2024-04-25T01:52:25.524967+0000","last_active":"2024-04-25T01:54:28.676369+0000","last_peered":"2024-04-25T01:54:28.676369+0000","last_clean":"2024-04-25T01:54:28.676369+0000","last_became_active":"2024-04-25T01:52:25.524670+0000","last_became_peered":"2024-04-25T01:52:25.524670+0000","last_unstale":"2024-04-25T01:54:28.676369+0000","last_undegraded":"2024-04-25T01:54:28.676369+0000","last_fullsized":"2024-04-25T01:54:28.676369+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:08.829052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676711+0000","last_change":"2024-04-25T01:52:24.512187+0000","last_active":"2024-04-25T01:54:28.676711+0000","last_peered":"2024-04-25T01:54:28.676711+0000","last_clean":"2024-04-25T01:54:28.676711+0000","last_became_active":"2024-04-25T01:52:24.510303+0000","last_became_peered":"2024-04-25T01:52:24.510303+0000","last_unstale":"2024-04-25T01:54:28.676711+0000","last_undegraded":"2024-04-25T01:54:28.676711+0000","last_fullsized":"2024-04-25T01:54:28.676711+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:42:31.596355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845405+0000","last_change":"2024-04-25T01:52:25.530386+0000","last_active":"2024-04-25T01:54:28.845405+0000","last_peered":"2024-04-25T01:54:28.845405+0000","last_clean":"2024-04-25T01:54:28.845405+0000","last_became_active":"2024-04-25T01:52:25.530034+0000","last_became_peered":"2024-04-25T01:52:25.530034+0000","last_unstale":"2024-04-25T01:54:28.845405+0000","last_undegraded":"2024-04-25T01:54:28.845405+0000","last_fullsized":"2024-04-25T01:54:28.845405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:34:11.470123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'4","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845367+0000","last_change":"2024-04-25T01:53:24.976373+0000","last_active":"2024-04-25T01:54:28.845367+0000","last_peered":"2024-04-25T01:54:28.845367+0000","last_clean":"2024-04-25T01:54:28.845367+0000","last_became_active":"2024-04-25T01:53:24.971146+0000","last_became_peered":"2024-04-25T01:53:24.971146+0000","last_unstale":"2024-04-25T01:54:28.845367+0000","last_undegraded":"2024-04-25T01:54:28.845367+0000","last_fullsized":"2024-04-25T01:54:28.845367+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:33.079071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.677039+0000","last_change":"2024-04-25T01:52:25.525695+0000","last_active":"2024-04-25T01:54:28.677039+0000","last_peered":"2024-04-25T01:54:28.677039+0000","last_clean":"2024-04-25T01:54:28.677039+0000","last_became_active":"2024-04-25T01:52:25.524830+0000","last_became_peered":"2024-04-25T01:52:25.524830+0000","last_unstale":"2024-04-25T01:54:28.677039+0000","last_undegraded":"2024-04-25T01:54:28.677039+0000","last_fullsized":"2024-04-25T01:54:28.677039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:04:58.449350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676468+0000","last_change":"2024-04-25T01:52:24.519232+0000","last_active":"2024-04-25T01:54:28.676468+0000","last_peered":"2024-04-25T01:54:28.676468+0000","last_clean":"2024-04-25T01:54:28.676468+0000","last_became_active":"2024-04-25T01:52:24.518902+0000","last_became_peered":"2024-04-25T01:52:24.518902+0000","last_unstale":"2024-04-25T01:54:28.676468+0000","last_undegraded":"2024-04-25T01:54:28.676468+0000","last_fullsized":"2024-04-25T01:54:28.676468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.173730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837719+0000","last_change":"2024-04-25T01:52:25.522955+0000","last_active":"2024-04-25T01:54:28.837719+0000","last_peered":"2024-04-25T01:54:28.837719+0000","last_clean":"2024-04-25T01:54:28.837719+0000","last_became_active":"2024-04-25T01:52:25.522626+0000","last_became_peered":"2024-04-25T01:52:25.522626+0000","last_unstale":"2024-04-25T01:54:28.837719+0000","last_undegraded":"2024-04-25T01:54:28.837719+0000","last_fullsized":"2024-04-25T01:54:28.837719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:12:47.218461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'9","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.947801+0000","last_change":"2024-04-25T01:52:24.516383+0000","last_active":"2024-04-25T01:54:28.947801+0000","last_peered":"2024-04-25T01:54:28.947801+0000","last_clean":"2024-04-25T01:54:28.947801+0000","last_became_active":"2024-04-25T01:52:24.515096+0000","last_became_peered":"2024-04-25T01:52:24.515096+0000","last_unstale":"2024-04-25T01:54:28.947801+0000","last_undegraded":"2024-04-25T01:54:28.947801+0000","last_fullsized":"2024-04-25T01:54:28.947801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:52.658161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2586,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838178+0000","last_change":"2024-04-25T01:52:25.524874+0000","last_active":"2024-04-25T01:54:28.838178+0000","last_peered":"2024-04-25T01:54:28.838178+0000","last_clean":"2024-04-25T01:54:28.838178+0000","last_became_active":"2024-04-25T01:52:25.523532+0000","last_became_peered":"2024-04-25T01:52:25.523532+0000","last_unstale":"2024-04-25T01:54:28.838178+0000","last_undegraded":"2024-04-25T01:54:28.838178+0000","last_fullsized":"2024-04-25T01:54:28.838178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:00:10.746446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'41","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.789815+0000","last_change":"2024-04-25T01:52:24.526212+0000","last_active":"2024-04-25T01:54:28.789815+0000","last_peered":"2024-04-25T01:54:28.789815+0000","last_clean":"2024-04-25T01:54:28.789815+0000","last_became_active":"2024-04-25T01:52:24.525882+0000","last_became_peered":"2024-04-25T01:52:24.525882+0000","last_unstale":"2024-04-25T01:54:28.789815+0000","last_undegraded":"2024-04-25T01:54:28.789815+0000","last_fullsized":"2024-04-25T01:54:28.789815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:04.792560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837993+0000","last_change":"2024-04-25T01:52:25.517387+0000","last_active":"2024-04-25T01:54:28.837993+0000","last_peered":"2024-04-25T01:54:28.837993+0000","last_clean":"2024-04-25T01:54:28.837993+0000","last_became_active":"2024-04-25T01:52:25.517047+0000","last_became_peered":"2024-04-25T01:52:25.517047+0000","last_unstale":"2024-04-25T01:54:28.837993+0000","last_undegraded":"2024-04-25T01:54:28.837993+0000","last_fullsized":"2024-04-25T01:54:28.837993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:39.462474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908253+0000","last_change":"2024-04-25T01:52:24.506510+0000","last_active":"2024-04-25T01:54:28.908253+0000","last_peered":"2024-04-25T01:54:28.908253+0000","last_clean":"2024-04-25T01:54:28.908253+0000","last_became_active":"2024-04-25T01:52:24.506184+0000","last_became_peered":"2024-04-25T01:52:24.506184+0000","last_unstale":"2024-04-25T01:54:28.908253+0000","last_undegraded":"2024-04-25T01:54:28.908253+0000","last_fullsized":"2024-04-25T01:54:28.908253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:23.489219+0000","last_clean_scrub_stamp":"2024-04-25T01:52:23.489219+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:17:56.754386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.838180+0000","last_change":"2024-04-25T01:52:25.525197+0000","last_active":"2024-04-25T01:54:28.838180+0000","last_peered":"2024-04-25T01:54:28.838180+0000","last_clean":"2024-04-25T01:54:28.838180+0000","last_became_active":"2024-04-25T01:52:25.524692+0000","last_became_peered":"2024-04-25T01:52:25.524692+0000","last_unstale":"2024-04-25T01:54:28.838180+0000","last_undegraded":"2024-04-25T01:54:28.838180+0000","last_fullsized":"2024-04-25T01:54:28.838180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:50.264227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845432+0000","last_change":"2024-04-25T01:52:25.531060+0000","last_active":"2024-04-25T01:54:28.845432+0000","last_peered":"2024-04-25T01:54:28.845432+0000","last_clean":"2024-04-25T01:54:28.845432+0000","last_became_active":"2024-04-25T01:52:25.530774+0000","last_became_peered":"2024-04-25T01:52:25.530774+0000","last_unstale":"2024-04-25T01:54:28.845432+0000","last_undegraded":"2024-04-25T01:54:28.845432+0000","last_fullsized":"2024-04-25T01:54:28.845432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:32.168699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837364+0000","last_change":"2024-04-25T01:52:25.535381+0000","last_active":"2024-04-25T01:54:28.837364+0000","last_peered":"2024-04-25T01:54:28.837364+0000","last_clean":"2024-04-25T01:54:28.837364+0000","last_became_active":"2024-04-25T01:52:25.533623+0000","last_became_peered":"2024-04-25T01:52:25.533623+0000","last_unstale":"2024-04-25T01:54:28.837364+0000","last_undegraded":"2024-04-25T01:54:28.837364+0000","last_fullsized":"2024-04-25T01:54:28.837364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:14.032508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837321+0000","last_change":"2024-04-25T01:52:25.528803+0000","last_active":"2024-04-25T01:54:28.837321+0000","last_peered":"2024-04-25T01:54:28.837321+0000","last_clean":"2024-04-25T01:54:28.837321+0000","last_became_active":"2024-04-25T01:52:25.528505+0000","last_became_peered":"2024-04-25T01:52:25.528505+0000","last_unstale":"2024-04-25T01:54:28.837321+0000","last_undegraded":"2024-04-25T01:54:28.837321+0000","last_fullsized":"2024-04-25T01:54:28.837321+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:35.811181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.846209+0000","last_change":"2024-04-25T01:52:25.523620+0000","last_active":"2024-04-25T01:54:28.846209+0000","last_peered":"2024-04-25T01:54:28.846209+0000","last_clean":"2024-04-25T01:54:28.846209+0000","last_became_active":"2024-04-25T01:52:25.523007+0000","last_became_peered":"2024-04-25T01:52:25.523007+0000","last_unstale":"2024-04-25T01:54:28.846209+0000","last_undegraded":"2024-04-25T01:54:28.846209+0000","last_fullsized":"2024-04-25T01:54:28.846209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:23.587838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908298+0000","last_change":"2024-04-25T01:52:25.526752+0000","last_active":"2024-04-25T01:54:28.908298+0000","last_peered":"2024-04-25T01:54:28.908298+0000","last_clean":"2024-04-25T01:54:28.908298+0000","last_became_active":"2024-04-25T01:52:25.526398+0000","last_became_peered":"2024-04-25T01:52:25.526398+0000","last_unstale":"2024-04-25T01:54:28.908298+0000","last_undegraded":"2024-04-25T01:54:28.908298+0000","last_fullsized":"2024-04-25T01:54:28.908298+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:01:37.286676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845498+0000","last_change":"2024-04-25T01:52:25.524999+0000","last_active":"2024-04-25T01:54:28.845498+0000","last_peered":"2024-04-25T01:54:28.845498+0000","last_clean":"2024-04-25T01:54:28.845498+0000","last_became_active":"2024-04-25T01:52:25.524652+0000","last_became_peered":"2024-04-25T01:52:25.524652+0000","last_unstale":"2024-04-25T01:54:28.845498+0000","last_undegraded":"2024-04-25T01:54:28.845498+0000","last_fullsized":"2024-04-25T01:54:28.845498+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:47:37.463701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.837706+0000","last_change":"2024-04-25T01:52:25.532996+0000","last_active":"2024-04-25T01:54:28.837706+0000","last_peered":"2024-04-25T01:54:28.837706+0000","last_clean":"2024-04-25T01:54:28.837706+0000","last_became_active":"2024-04-25T01:52:25.530230+0000","last_became_peered":"2024-04-25T01:52:25.530230+0000","last_unstale":"2024-04-25T01:54:28.837706+0000","last_undegraded":"2024-04-25T01:54:28.837706+0000","last_fullsized":"2024-04-25T01:54:28.837706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:24:08.996434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676996+0000","last_change":"2024-04-25T01:52:25.523838+0000","last_active":"2024-04-25T01:54:28.676996+0000","last_peered":"2024-04-25T01:54:28.676996+0000","last_clean":"2024-04-25T01:54:28.676996+0000","last_became_active":"2024-04-25T01:52:25.523297+0000","last_became_peered":"2024-04-25T01:52:25.523297+0000","last_unstale":"2024-04-25T01:54:28.676996+0000","last_undegraded":"2024-04-25T01:54:28.676996+0000","last_fullsized":"2024-04-25T01:54:28.676996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:52.938763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.585617+0000","last_change":"2024-04-25T01:52:25.519028+0000","last_active":"2024-04-25T01:54:28.585617+0000","last_peered":"2024-04-25T01:54:28.585617+0000","last_clean":"2024-04-25T01:54:28.585617+0000","last_became_active":"2024-04-25T01:52:25.518716+0000","last_became_peered":"2024-04-25T01:52:25.518716+0000","last_unstale":"2024-04-25T01:54:28.585617+0000","last_undegraded":"2024-04-25T01:54:28.585617+0000","last_fullsized":"2024-04-25T01:54:28.585617+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:37:31.573403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.676618+0000","last_change":"2024-04-25T01:52:25.533894+0000","last_active":"2024-04-25T01:54:28.676618+0000","last_peered":"2024-04-25T01:54:28.676618+0000","last_clean":"2024-04-25T01:54:28.676618+0000","last_became_active":"2024-04-25T01:52:25.533510+0000","last_became_peered":"2024-04-25T01:52:25.533510+0000","last_unstale":"2024-04-25T01:54:28.676618+0000","last_undegraded":"2024-04-25T01:54:28.676618+0000","last_fullsized":"2024-04-25T01:54:28.676618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:40:07.128296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.697784+0000","last_change":"2024-04-25T01:52:25.524693+0000","last_active":"2024-04-25T01:54:28.697784+0000","last_peered":"2024-04-25T01:54:28.697784+0000","last_clean":"2024-04-25T01:54:28.697784+0000","last_became_active":"2024-04-25T01:52:25.524366+0000","last_became_peered":"2024-04-25T01:52:25.524366+0000","last_unstale":"2024-04-25T01:54:28.697784+0000","last_undegraded":"2024-04-25T01:54:28.697784+0000","last_fullsized":"2024-04-25T01:54:28.697784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:23.133722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.908579+0000","last_change":"2024-04-25T01:52:25.525936+0000","last_active":"2024-04-25T01:54:28.908579+0000","last_peered":"2024-04-25T01:54:28.908579+0000","last_clean":"2024-04-25T01:54:28.908579+0000","last_became_active":"2024-04-25T01:52:25.525586+0000","last_became_peered":"2024-04-25T01:52:25.525586+0000","last_unstale":"2024-04-25T01:54:28.908579+0000","last_undegraded":"2024-04-25T01:54:28.908579+0000","last_fullsized":"2024-04-25T01:54:28.908579+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:34.756898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:54:28.845420+0000","last_change":"2024-04-25T01:53:24.970621+0000","last_active":"2024-04-25T01:54:28.845420+0000","last_peered":"2024-04-25T01:54:28.845420+0000","last_clean":"2024-04-25T01:54:28.845420+0000","last_became_active":"2024-04-25T01:53:24.970184+0000","last_became_peered":"2024-04-25T01:53:24.970184+0000","last_unstale":"2024-04-25T01:54:28.845420+0000","last_undegraded":"2024-04-25T01:54:28.845420+0000","last_fullsized":"2024-04-25T01:54:28.845420+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:24.493828+0000","last_clean_scrub_stamp":"2024-04-25T01:52:24.493828+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:52:43.072630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106956,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":602722,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1303,"num_read_kb":769,"num_write":377,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":671744,"data_stored":1808166,"data_compressed":89205,"data_compressed_allocated":118784,"data_compressed_original":1622016,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318308,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":478166,"data_compressed":2900,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579946,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32316,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712836,"statfs":{"total":95995035648,"available":95961944064,"internally_reserved":0,"allocated":1167360,"data_stored":1268070,"data_compressed":46559,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743470,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":869896,"data_compressed":22046,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005109,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":870274,"data_compressed":22046,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":1072369,"data_compressed":38552,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":476890,"data_compressed":2900,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":544687,"data_compressed":4751,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":469863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":470150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1001038,"data_compressed":28202,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":539071,"data_compressed":12497,"data_compressed_allocated":16384,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1201347,"data_compressed":44708,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35652,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":69129,"data_compressed":1851,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":69391,"data_compressed":9597,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135304,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":202027,"data_compressed":8007,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:54:33.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:54:33.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:54:33.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:54:33.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: Updating smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1398454445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: Updating smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.911 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:54:33.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:54:33.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:54:33.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.916+0000 7f8659a4c640 1 -- 172.21.15.31:0/3647242789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f865410bd60 msgr2=0x7f865410e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.916+0000 7f8659a4c640 1 --2- 172.21.15.31:0/3647242789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f865410bd60 0x7f865410e1d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f864c009ca0 tx=0x7f864c02a950 comp rx=0 tx=0).stop 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.916+0000 7f8659a4c640 1 -- 172.21.15.31:0/3647242789 shutdown_connections 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.916+0000 7f8659a4c640 1 --2- 172.21.15.31:0/3647242789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f865410bd60 0x7f865410e1d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.916+0000 7f8659a4c640 1 --2- 172.21.15.31:0/3647242789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8654103340 0x7f865410b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.916+0000 7f8659a4c640 1 --2- 172.21.15.31:0/3647242789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8654102a20 0x7f8654102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.916+0000 7f8659a4c640 1 -- 172.21.15.31:0/3647242789 >> 172.21.15.31:0/3647242789 conn(0x7f86540777a0 msgr2=0x7f8654077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.916+0000 7f8659a4c640 1 -- 172.21.15.31:0/3647242789 shutdown_connections 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.916+0000 7f8659a4c640 1 -- 172.21.15.31:0/3647242789 wait complete. 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f8659a4c640 1 Processor -- start 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f8659a4c640 1 -- start start 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f8659a4c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8654102a20 0x7f8654197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f8659a4c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8654103340 0x7f8654198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f8659a4c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f865410bd60 0x7f865419c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f8659a4c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86541119e0 con 0x7f8654103340 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f8659a4c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8654111860 con 0x7f8654102a20 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f8659a4c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8654111b60 con 0x7f865410bd60 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f864bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8654103340 0x7f8654198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f864bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8654103340 0x7f8654198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38720/0 (socket says 172.21.15.31:38720) 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f864bfff640 1 -- 172.21.15.31:0/3518851084 learned_addr learned my addr 172.21.15.31:0/3518851084 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f8652ffd640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8654102a20 0x7f8654197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.917+0000 7f86537fe640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f865410bd60 0x7f865419c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f864bfff640 1 -- 172.21.15.31:0/3518851084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f865410bd60 msgr2=0x7f865419c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f864bfff640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f865410bd60 0x7f865419c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f864bfff640 1 -- 172.21.15.31:0/3518851084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8654102a20 msgr2=0x7f8654197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f864bfff640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8654102a20 0x7f8654197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f864bfff640 1 -- 172.21.15.31:0/3518851084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865419cee0 con 0x7f8654103340 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f8652ffd640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8654102a20 0x7f8654197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:33.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f864bfff640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8654103340 0x7f8654198480 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f863c00d770 tx=0x7f863c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:33.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f8658a4a640 1 -- 172.21.15.31:0/3518851084 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c017070 con 0x7f8654103340 2024-04-25T01:54:33.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f8658a4a640 1 -- 172.21.15.31:0/3518851084 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f863c0054d0 con 0x7f8654103340 2024-04-25T01:54:33.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f8659a4c640 1 -- 172.21.15.31:0/3518851084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f865419d1d0 con 0x7f8654103340 2024-04-25T01:54:33.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f8658a4a640 1 -- 172.21.15.31:0/3518851084 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c004cf0 con 0x7f8654103340 2024-04-25T01:54:33.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.918+0000 7f8659a4c640 1 -- 172.21.15.31:0/3518851084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86541a4a60 con 0x7f8654103340 2024-04-25T01:54:33.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.919+0000 7f8659a4c640 1 -- 172.21.15.31:0/3518851084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86540689b0 con 0x7f8654103340 2024-04-25T01:54:33.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.920+0000 7f8658a4a640 1 -- 172.21.15.31:0/3518851084 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7f863c004e90 con 0x7f8654103340 2024-04-25T01:54:33.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.921+0000 7f8658a4a640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f86400a2d90 0x7f86400a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:33.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.921+0000 7f8652ffd640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f86400a2d90 0x7f86400a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:33.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.921+0000 7f8658a4a640 1 -- 172.21.15.31:0/3518851084 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f863c0cef20 con 0x7f8654103340 2024-04-25T01:54:33.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.921+0000 7f8652ffd640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f86400a2d90 0x7f86400a5250 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8644009fc0 tx=0x7f8644005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:33.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:33.925+0000 7f8658a4a640 1 -- 172.21.15.31:0/3518851084 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f863c09b370 con 0x7f8654103340 2024-04-25T01:54:33.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:54:33.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:54:33.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: Updating smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1398454445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: Updating smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: Updating smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: Updating smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:33.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:54:33.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:54:33.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:54:33.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:54:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.111+0000 7f8659a4c640 1 -- 172.21.15.31:0/3518851084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8654106f40 con 0x7f8654103340 2024-04-25T01:54:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.113+0000 7f8658a4a640 1 -- 172.21.15.31:0/3518851084 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22855 (secure 0 0 0) 0x7f863c09f670 con 0x7f8654103340 2024-04-25T01:54:34.115 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:34.115 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:54:28.579231+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3045817976":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6800/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1358173279":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/686937010":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/3913527371":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 -- 172.21.15.31:0/3518851084 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f86400a2d90 msgr2=0x7f86400a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f86400a2d90 0x7f86400a5250 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8644009fc0 tx=0x7f8644005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 -- 172.21.15.31:0/3518851084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8654103340 msgr2=0x7f8654198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8654103340 0x7f8654198480 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f863c00d770 tx=0x7f863c00dc30 comp rx=0 tx=0).stop 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 -- 172.21.15.31:0/3518851084 shutdown_connections 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f86400a2d90 0x7f86400a5250 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f865410bd60 0x7f865419c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8654103340 0x7f8654198480 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 --2- 172.21.15.31:0/3518851084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8654102a20 0x7f8654197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 -- 172.21.15.31:0/3518851084 >> 172.21.15.31:0/3518851084 conn(0x7f86540777a0 msgr2=0x7f86540fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 -- 172.21.15.31:0/3518851084 shutdown_connections 2024-04-25T01:54:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.115+0000 7f8659a4c640 1 -- 172.21.15.31:0/3518851084 wait complete. 2024-04-25T01:54:34.128 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:23.432566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:24.255373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:54:34.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:54:34.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.232+0000 7f953a472640 1 -- 172.21.15.31:0/2548221941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9534101710 msgr2=0x7f9534101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.232+0000 7f953a472640 1 --2- 172.21.15.31:0/2548221941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9534101710 0x7f9534101b90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9528009930 tx=0x7f952802a420 comp rx=0 tx=0).stop 2024-04-25T01:54:34.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.232+0000 7f953a472640 1 -- 172.21.15.31:0/2548221941 shutdown_connections 2024-04-25T01:54:34.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.232+0000 7f953a472640 1 --2- 172.21.15.31:0/2548221941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95341020d0 0x7f953410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.232+0000 7f953a472640 1 --2- 172.21.15.31:0/2548221941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9534101710 0x7f9534101b90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.232+0000 7f953a472640 1 --2- 172.21.15.31:0/2548221941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9534107720 0x7f9534107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.232+0000 7f953a472640 1 -- 172.21.15.31:0/2548221941 >> 172.21.15.31:0/2548221941 conn(0x7f95340fd180 msgr2=0x7f95340ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:34.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.232+0000 7f953a472640 1 -- 172.21.15.31:0/2548221941 shutdown_connections 2024-04-25T01:54:34.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.233+0000 7f953a472640 1 -- 172.21.15.31:0/2548221941 wait complete. 2024-04-25T01:54:34.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.233+0000 7f953a472640 1 Processor -- start 2024-04-25T01:54:34.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.233+0000 7f953a472640 1 -- start start 2024-04-25T01:54:34.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.233+0000 7f953a472640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9534101710 0x7f9534197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.233+0000 7f953a472640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95341020d0 0x7f95341984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.233+0000 7f953a472640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9534107720 0x7f953419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.233+0000 7f953a472640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95341119a0 con 0x7f9534101710 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.233+0000 7f953a472640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9534111820 con 0x7f9534107720 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.233+0000 7f953a472640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9534111b20 con 0x7f95341020d0 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f9533fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9534101710 0x7f9534197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f9533fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9534101710 0x7f9534197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38728/0 (socket says 172.21.15.31:38728) 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f9533fff640 1 -- 172.21.15.31:0/424567685 learned_addr learned my addr 172.21.15.31:0/424567685 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f9538a0f640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9534107720 0x7f953419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f95337fe640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95341020d0 0x7f95341984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f9533fff640 1 -- 172.21.15.31:0/424567685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95341020d0 msgr2=0x7f95341984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f9533fff640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95341020d0 0x7f95341984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f9533fff640 1 -- 172.21.15.31:0/424567685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9534107720 msgr2=0x7f953419c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f9533fff640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9534107720 0x7f953419c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f9533fff640 1 -- 172.21.15.31:0/424567685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f953419cf80 con 0x7f9534101710 2024-04-25T01:54:34.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f9533fff640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9534101710 0x7f9534197f90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f952400d950 tx=0x7f952400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:34.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f95317fa640 1 -- 172.21.15.31:0/424567685 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9524017030 con 0x7f9534101710 2024-04-25T01:54:34.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f95317fa640 1 -- 172.21.15.31:0/424567685 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9524004020 con 0x7f9534101710 2024-04-25T01:54:34.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f953a472640 1 -- 172.21.15.31:0/424567685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f953419d270 con 0x7f9534101710 2024-04-25T01:54:34.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f95317fa640 1 -- 172.21.15.31:0/424567685 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95240047a0 con 0x7f9534101710 2024-04-25T01:54:34.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.234+0000 7f953a472640 1 -- 172.21.15.31:0/424567685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95341a4b00 con 0x7f9534101710 2024-04-25T01:54:34.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.235+0000 7f953a472640 1 -- 172.21.15.31:0/424567685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9534102e50 con 0x7f9534101710 2024-04-25T01:54:34.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.236+0000 7f95317fa640 1 -- 172.21.15.31:0/424567685 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7f95240041d0 con 0x7f9534101710 2024-04-25T01:54:34.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.237+0000 7f95317fa640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f951c0a2d90 0x7f951c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.237+0000 7f95337fe640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f951c0a2d90 0x7f951c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.237+0000 7f95317fa640 1 -- 172.21.15.31:0/424567685 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f95240cf090 con 0x7f9534101710 2024-04-25T01:54:34.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.238+0000 7f95337fe640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f951c0a2d90 0x7f951c0a5250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9528000c00 tx=0x7f9528032ba0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:34.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.241+0000 7f95317fa640 1 -- 172.21.15.31:0/424567685 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f952409bb30 con 0x7f9534101710 2024-04-25T01:54:34.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.425+0000 7f953a472640 1 -- 172.21.15.31:0/424567685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9534109cd0 con 0x7f9534101710 2024-04-25T01:54:34.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.425+0000 7f95317fa640 1 -- 172.21.15.31:0/424567685 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f952409b430 con 0x7f9534101710 2024-04-25T01:54:34.426 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 -- 172.21.15.31:0/424567685 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f951c0a2d90 msgr2=0x7f951c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f951c0a2d90 0x7f951c0a5250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9528000c00 tx=0x7f9528032ba0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 -- 172.21.15.31:0/424567685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9534101710 msgr2=0x7f9534197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9534101710 0x7f9534197f90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f952400d950 tx=0x7f952400de10 comp rx=0 tx=0).stop 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 -- 172.21.15.31:0/424567685 shutdown_connections 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f951c0a2d90 0x7f951c0a5250 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9534107720 0x7f953419c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95341020d0 0x7f95341984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 --2- 172.21.15.31:0/424567685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9534101710 0x7f9534197f90 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 -- 172.21.15.31:0/424567685 >> 172.21.15.31:0/424567685 conn(0x7f95340fd180 msgr2=0x7f95340fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 -- 172.21.15.31:0/424567685 shutdown_connections 2024-04-25T01:54:34.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.427+0000 7f953a472640 1 -- 172.21.15.31:0/424567685 wait complete. 2024-04-25T01:54:34.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:54:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.543+0000 7f49b459e640 1 -- 172.21.15.31:0/759131818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ac107720 msgr2=0x7f49ac107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.543+0000 7f49b459e640 1 --2- 172.21.15.31:0/759131818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ac107720 0x7f49ac107b00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009930 tx=0x7f49a802a420 comp rx=0 tx=0).stop 2024-04-25T01:54:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.543+0000 7f49b459e640 1 -- 172.21.15.31:0/759131818 shutdown_connections 2024-04-25T01:54:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.543+0000 7f49b459e640 1 --2- 172.21.15.31:0/759131818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49ac1020d0 0x7f49ac10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.543+0000 7f49b459e640 1 --2- 172.21.15.31:0/759131818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49ac101710 0x7f49ac101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.543+0000 7f49b459e640 1 --2- 172.21.15.31:0/759131818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ac107720 0x7f49ac107b00 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.543+0000 7f49b459e640 1 -- 172.21.15.31:0/759131818 >> 172.21.15.31:0/759131818 conn(0x7f49ac0fd180 msgr2=0x7f49ac0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.543+0000 7f49b459e640 1 -- 172.21.15.31:0/759131818 shutdown_connections 2024-04-25T01:54:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.543+0000 7f49b459e640 1 -- 172.21.15.31:0/759131818 wait complete. 2024-04-25T01:54:34.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b459e640 1 Processor -- start 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b459e640 1 -- start start 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b459e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49ac101710 0x7f49ac197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b459e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49ac1020d0 0x7f49ac1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b459e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ac107720 0x7f49ac19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b459e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac111a30 con 0x7f49ac107720 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b459e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac1118b0 con 0x7f49ac101710 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b459e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac111bb0 con 0x7f49ac1020d0 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b2b3b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ac107720 0x7f49ac19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b2b3b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ac107720 0x7f49ac19c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38760/0 (socket says 172.21.15.31:38760) 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b2b3b640 1 -- 172.21.15.31:0/1276274216 learned_addr learned my addr 172.21.15.31:0/1276274216 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.544+0000 7f49b233a640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49ac101710 0x7f49ac197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.545+0000 7f49b2b3b640 1 -- 172.21.15.31:0/1276274216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49ac1020d0 msgr2=0x7f49ac1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.545+0000 7f49b2b3b640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49ac1020d0 0x7f49ac1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.545+0000 7f49b2b3b640 1 -- 172.21.15.31:0/1276274216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49ac101710 msgr2=0x7f49ac197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.545+0000 7f49b2b3b640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49ac101710 0x7f49ac197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.545+0000 7f49b2b3b640 1 -- 172.21.15.31:0/1276274216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ac19cf60 con 0x7f49ac107720 2024-04-25T01:54:34.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.545+0000 7f49b2b3b640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ac107720 0x7f49ac19c830 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f49a400eaa0 tx=0x7f49a400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:34.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.545+0000 7f49a37fe640 1 -- 172.21.15.31:0/1276274216 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a400cdb0 con 0x7f49ac107720 2024-04-25T01:54:34.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.545+0000 7f49a37fe640 1 -- 172.21.15.31:0/1276274216 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49a4005020 con 0x7f49ac107720 2024-04-25T01:54:34.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.545+0000 7f49b459e640 1 -- 172.21.15.31:0/1276274216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ac19d250 con 0x7f49ac107720 2024-04-25T01:54:34.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.545+0000 7f49a37fe640 1 -- 172.21.15.31:0/1276274216 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a40116d0 con 0x7f49ac107720 2024-04-25T01:54:34.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.545+0000 7f49b459e640 1 -- 172.21.15.31:0/1276274216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ac1a4ae0 con 0x7f49ac107720 2024-04-25T01:54:34.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.546+0000 7f49b459e640 1 -- 172.21.15.31:0/1276274216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ac0689b0 con 0x7f49ac107720 2024-04-25T01:54:34.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.547+0000 7f49a37fe640 1 -- 172.21.15.31:0/1276274216 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7f49a40051d0 con 0x7f49ac107720 2024-04-25T01:54:34.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.548+0000 7f49a37fe640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f49980a2e10 0x7f49980a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.548+0000 7f49a37fe640 1 -- 172.21.15.31:0/1276274216 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f49a40cf660 con 0x7f49ac107720 2024-04-25T01:54:34.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.548+0000 7f49b233a640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f49980a2e10 0x7f49980a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.548+0000 7f49b233a640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f49980a2e10 0x7f49980a52d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f49a8031440 tx=0x7f49a8005a20 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:34.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.552+0000 7f49a37fe640 1 -- 172.21.15.31:0/1276274216 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f49a4016030 con 0x7f49ac107720 2024-04-25T01:54:34.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:34 smithi031 ceph-mon[29204]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:34.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:34 smithi031 ceph-mon[29204]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:34.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:34 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3518851084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:34.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:34 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/424567685' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:34.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.738+0000 7f49b459e640 1 -- 172.21.15.31:0/1276274216 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f49ac02d170 con 0x7f49ac107720 2024-04-25T01:54:34.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.738+0000 7f49a37fe640 1 -- 172.21.15.31:0/1276274216 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f49a409c160 con 0x7f49ac107720 2024-04-25T01:54:34.739 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T01:54:34.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 -- 172.21.15.31:0/1276274216 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f49980a2e10 msgr2=0x7f49980a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f49980a2e10 0x7f49980a52d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f49a8031440 tx=0x7f49a8005a20 comp rx=0 tx=0).stop 2024-04-25T01:54:34.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 -- 172.21.15.31:0/1276274216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ac107720 msgr2=0x7f49ac19c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ac107720 0x7f49ac19c830 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f49a400eaa0 tx=0x7f49a400ef60 comp rx=0 tx=0).stop 2024-04-25T01:54:34.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 -- 172.21.15.31:0/1276274216 shutdown_connections 2024-04-25T01:54:34.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f49980a2e10 0x7f49980a52d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ac107720 0x7f49ac19c830 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49ac1020d0 0x7f49ac1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 --2- 172.21.15.31:0/1276274216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49ac101710 0x7f49ac197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 -- 172.21.15.31:0/1276274216 >> 172.21.15.31:0/1276274216 conn(0x7f49ac0fd180 msgr2=0x7f49ac0fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:34.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 -- 172.21.15.31:0/1276274216 shutdown_connections 2024-04-25T01:54:34.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.740+0000 7f49b459e640 1 -- 172.21.15.31:0/1276274216 wait complete. 2024-04-25T01:54:34.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:54:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.856+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3798170461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e41020d0 msgr2=0x7ff8e410e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.856+0000 7ff8e9cee640 1 --2- 172.21.15.31:0/3798170461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e41020d0 0x7ff8e410e5e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009cd0 tx=0x7ff8dc02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.856+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3798170461 shutdown_connections 2024-04-25T01:54:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.856+0000 7ff8e9cee640 1 --2- 172.21.15.31:0/3798170461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e41020d0 0x7ff8e410e5e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.856+0000 7ff8e9cee640 1 --2- 172.21.15.31:0/3798170461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8e4101710 0x7ff8e4101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.856+0000 7ff8e9cee640 1 --2- 172.21.15.31:0/3798170461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8e4107720 0x7ff8e4107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.856+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3798170461 >> 172.21.15.31:0/3798170461 conn(0x7ff8e40fd180 msgr2=0x7ff8e40ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.856+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3798170461 shutdown_connections 2024-04-25T01:54:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.856+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3798170461 wait complete. 2024-04-25T01:54:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e9cee640 1 Processor -- start 2024-04-25T01:54:34.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e9cee640 1 -- start start 2024-04-25T01:54:34.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e9cee640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8e4101710 0x7ff8e4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e9cee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e41020d0 0x7ff8e41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e9cee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8e4107720 0x7ff8e419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e9cee640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e41119e0 con 0x7ff8e41020d0 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e9cee640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4111860 con 0x7ff8e4107720 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e9cee640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4111b60 con 0x7ff8e4101710 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e41020d0 0x7ff8e41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e41020d0 0x7ff8e41984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38784/0 (socket says 172.21.15.31:38784) 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e2ffd640 1 -- 172.21.15.31:0/3963992245 learned_addr learned my addr 172.21.15.31:0/3963992245 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e37fe640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8e4101710 0x7ff8e4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.857+0000 7ff8e3fff640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8e4107720 0x7ff8e419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e2ffd640 1 -- 172.21.15.31:0/3963992245 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8e4101710 msgr2=0x7ff8e4197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e2ffd640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8e4101710 0x7ff8e4197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e2ffd640 1 -- 172.21.15.31:0/3963992245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8e4107720 msgr2=0x7ff8e419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e2ffd640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8e4107720 0x7ff8e419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e2ffd640 1 -- 172.21.15.31:0/3963992245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e419cf90 con 0x7ff8e41020d0 2024-04-25T01:54:34.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e37fe640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8e4101710 0x7ff8e4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:34.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e2ffd640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e41020d0 0x7ff8e41984e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400c890 tx=0x7ff8d400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:34.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e0ff9640 1 -- 172.21.15.31:0/3963992245 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4004220 con 0x7ff8e41020d0 2024-04-25T01:54:34.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3963992245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e419d280 con 0x7ff8e41020d0 2024-04-25T01:54:34.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e0ff9640 1 -- 172.21.15.31:0/3963992245 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8d40043c0 con 0x7ff8e41020d0 2024-04-25T01:54:34.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e0ff9640 1 -- 172.21.15.31:0/3963992245 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4004a00 con 0x7ff8e41020d0 2024-04-25T01:54:34.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.858+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3963992245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e41a4b10 con 0x7ff8e41020d0 2024-04-25T01:54:34.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.859+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3963992245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e40689b0 con 0x7ff8e41020d0 2024-04-25T01:54:34.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.860+0000 7ff8e0ff9640 1 -- 172.21.15.31:0/3963992245 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7ff8d4004ba0 con 0x7ff8e41020d0 2024-04-25T01:54:34.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.861+0000 7ff8e0ff9640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff8d00a2d90 0x7ff8d00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.861+0000 7ff8e37fe640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff8d00a2d90 0x7ff8d00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.861+0000 7ff8e0ff9640 1 -- 172.21.15.31:0/3963992245 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7ff8d40cf310 con 0x7ff8e41020d0 2024-04-25T01:54:34.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.861+0000 7ff8e37fe640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff8d00a2d90 0x7ff8d00a5250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009930 tx=0x7ff8cc008040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:34.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:34.865+0000 7ff8e0ff9640 1 -- 172.21.15.31:0/3963992245 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff8d4014030 con 0x7ff8e41020d0 2024-04-25T01:54:34.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:34 smithi040 ceph-mon[32418]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:34.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:34 smithi040 ceph-mon[32418]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:34.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:34 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3518851084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:34.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:34 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/424567685' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:34.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:34 smithi134 ceph-mon[33098]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:34.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:34 smithi134 ceph-mon[33098]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:34.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:34 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3518851084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:34.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:34 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/424567685' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:35.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.051+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3963992245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff8e4109710 con 0x7ff8e41020d0 2024-04-25T01:54:35.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.051+0000 7ff8e0ff9640 1 -- 172.21.15.31:0/3963992245 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7ff8d409be10 con 0x7ff8e41020d0 2024-04-25T01:54:35.052 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T01:54:35.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3963992245 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff8d00a2d90 msgr2=0x7ff8d00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff8d00a2d90 0x7ff8d00a5250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009930 tx=0x7ff8cc008040 comp rx=0 tx=0).stop 2024-04-25T01:54:35.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3963992245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e41020d0 msgr2=0x7ff8e41984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e41020d0 0x7ff8e41984e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400c890 tx=0x7ff8d400cd50 comp rx=0 tx=0).stop 2024-04-25T01:54:35.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3963992245 shutdown_connections 2024-04-25T01:54:35.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff8d00a2d90 0x7ff8d00a5250 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8e4107720 0x7ff8e419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8e41020d0 0x7ff8e41984e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 --2- 172.21.15.31:0/3963992245 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8e4101710 0x7ff8e4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3963992245 >> 172.21.15.31:0/3963992245 conn(0x7ff8e40fd180 msgr2=0x7ff8e40fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3963992245 shutdown_connections 2024-04-25T01:54:35.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.053+0000 7ff8e9cee640 1 -- 172.21.15.31:0/3963992245 wait complete. 2024-04-25T01:54:35.065 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:54:35.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.168+0000 7f9eede2a640 1 -- 172.21.15.31:0/1800664022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ee810e0d0 msgr2=0x7f9ee81104c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.168+0000 7f9eede2a640 1 --2- 172.21.15.31:0/1800664022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ee810e0d0 0x7f9ee81104c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009ca0 tx=0x7f9ee002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.168+0000 7f9eede2a640 1 -- 172.21.15.31:0/1800664022 shutdown_connections 2024-04-25T01:54:35.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.168+0000 7f9eede2a640 1 --2- 172.21.15.31:0/1800664022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ee810e0d0 0x7f9ee81104c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.168+0000 7f9eede2a640 1 --2- 172.21.15.31:0/1800664022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ee8100e40 0x7f9ee810db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.168+0000 7f9eede2a640 1 --2- 172.21.15.31:0/1800664022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ee8100520 0x7f9ee8100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.168+0000 7f9eede2a640 1 -- 172.21.15.31:0/1800664022 >> 172.21.15.31:0/1800664022 conn(0x7f9ee80769c0 msgr2=0x7f9ee8076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.168+0000 7f9eede2a640 1 -- 172.21.15.31:0/1800664022 shutdown_connections 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.168+0000 7f9eede2a640 1 -- 172.21.15.31:0/1800664022 wait complete. 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.168+0000 7f9eede2a640 1 Processor -- start 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9eede2a640 1 -- start start 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9eede2a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ee8100520 0x7f9ee819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9eede2a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ee8100e40 0x7f9ee819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9eede2a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ee810e0d0 0x7f9ee819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9eede2a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee8113c00 con 0x7f9ee8100e40 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9eede2a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee8113a80 con 0x7f9ee810e0d0 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9eede2a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee8113d80 con 0x7f9ee8100520 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9ee6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ee8100e40 0x7f9ee819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9ee6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ee8100e40 0x7f9ee819a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38800/0 (socket says 172.21.15.31:38800) 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9ee6ffd640 1 -- 172.21.15.31:0/2449320549 learned_addr learned my addr 172.21.15.31:0/2449320549 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9ee77fe640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ee8100520 0x7f9ee819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9ee7fff640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ee810e0d0 0x7f9ee819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9ee6ffd640 1 -- 172.21.15.31:0/2449320549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ee8100520 msgr2=0x7f9ee819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9ee6ffd640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ee8100520 0x7f9ee819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9ee6ffd640 1 -- 172.21.15.31:0/2449320549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ee810e0d0 msgr2=0x7f9ee819ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9ee6ffd640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ee810e0d0 0x7f9ee819ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9ee6ffd640 1 -- 172.21.15.31:0/2449320549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee819f160 con 0x7f9ee8100e40 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.169+0000 7f9ee6ffd640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ee8100e40 0x7f9ee819a700 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c890 tx=0x7f9edc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.170+0000 7f9ee4ff9640 1 -- 172.21.15.31:0/2449320549 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc004220 con 0x7f9ee8100e40 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.170+0000 7f9ee4ff9640 1 -- 172.21.15.31:0/2449320549 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9edc0043c0 con 0x7f9ee8100e40 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.170+0000 7f9eede2a640 1 -- 172.21.15.31:0/2449320549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee819f450 con 0x7f9ee8100e40 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.170+0000 7f9ee4ff9640 1 -- 172.21.15.31:0/2449320549 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc004a00 con 0x7f9ee8100e40 2024-04-25T01:54:35.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.170+0000 7f9eede2a640 1 -- 172.21.15.31:0/2449320549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee81a6ce0 con 0x7f9ee8100e40 2024-04-25T01:54:35.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.171+0000 7f9eede2a640 1 -- 172.21.15.31:0/2449320549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee806a5c0 con 0x7f9ee8100e40 2024-04-25T01:54:35.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.172+0000 7f9ee4ff9640 1 -- 172.21.15.31:0/2449320549 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7f9edc004ba0 con 0x7f9ee8100e40 2024-04-25T01:54:35.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.173+0000 7f9ee4ff9640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9ed00a2d90 0x7f9ed00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.173+0000 7f9ee77fe640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9ed00a2d90 0x7f9ed00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.173+0000 7f9ee4ff9640 1 -- 172.21.15.31:0/2449320549 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9edc0cf370 con 0x7f9ee8100e40 2024-04-25T01:54:35.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.173+0000 7f9ee77fe640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9ed00a2d90 0x7f9ed00a5250 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009890 tx=0x7f9ed8008040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.178+0000 7f9ee4ff9640 1 -- 172.21.15.31:0/2449320549 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9edc014030 con 0x7f9ee8100e40 2024-04-25T01:54:35.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.403+0000 7f9eede2a640 1 -- 172.21.15.31:0/2449320549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ee819b4c0 con 0x7f9ee8100e40 2024-04-25T01:54:35.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.403+0000 7f9ee4ff9640 1 -- 172.21.15.31:0/2449320549 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11923 (secure 0 0 0) 0x7f9edc09be10 con 0x7f9ee8100e40 2024-04-25T01:54:35.405 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:35.405 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:54:28.592746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24451,24463,24503,24487,24473]},"id":1}]} 2024-04-25T01:54:35.405 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-04-25T01:54:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.405+0000 7f9eede2a640 1 -- 172.21.15.31:0/2449320549 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9ed00a2d90 msgr2=0x7f9ed00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.405+0000 7f9eede2a640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9ed00a2d90 0x7f9ed00a5250 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009890 tx=0x7f9ed8008040 comp rx=0 tx=0).stop 2024-04-25T01:54:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.406+0000 7f9eede2a640 1 -- 172.21.15.31:0/2449320549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ee8100e40 msgr2=0x7f9ee819a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.406+0000 7f9eede2a640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ee8100e40 0x7f9ee819a700 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c890 tx=0x7f9edc00cd50 comp rx=0 tx=0).stop 2024-04-25T01:54:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.406+0000 7f9eede2a640 1 -- 172.21.15.31:0/2449320549 shutdown_connections 2024-04-25T01:54:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.406+0000 7f9eede2a640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9ed00a2d90 0x7f9ed00a5250 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.406+0000 7f9eede2a640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ee810e0d0 0x7f9ee819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.406+0000 7f9eede2a640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ee8100e40 0x7f9ee819a700 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.406+0000 7f9eede2a640 1 --2- 172.21.15.31:0/2449320549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ee8100520 0x7f9ee819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.406+0000 7f9eede2a640 1 -- 172.21.15.31:0/2449320549 >> 172.21.15.31:0/2449320549 conn(0x7f9ee80769c0 msgr2=0x7f9ee810f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.406+0000 7f9eede2a640 1 -- 172.21.15.31:0/2449320549 shutdown_connections 2024-04-25T01:54:35.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.406+0000 7f9eede2a640 1 -- 172.21.15.31:0/2449320549 wait complete. 2024-04-25T01:54:35.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:54:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.520+0000 7fb7e855c640 1 -- 172.21.15.31:0/1258544718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e0102160 msgr2=0x7fb7e010e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.520+0000 7fb7e855c640 1 --2- 172.21.15.31:0/1258544718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e0102160 0x7fb7e010e5a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009ca0 tx=0x7fb7d802a950 comp rx=0 tx=0).stop 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.520+0000 7fb7e855c640 1 -- 172.21.15.31:0/1258544718 shutdown_connections 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.520+0000 7fb7e855c640 1 --2- 172.21.15.31:0/1258544718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e0102160 0x7fb7e010e5a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.520+0000 7fb7e855c640 1 --2- 172.21.15.31:0/1258544718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e01017c0 0x7fb7e0101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.520+0000 7fb7e855c640 1 --2- 172.21.15.31:0/1258544718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7e0107750 0x7fb7e0107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.520+0000 7fb7e855c640 1 -- 172.21.15.31:0/1258544718 >> 172.21.15.31:0/1258544718 conn(0x7fb7e00788d0 msgr2=0x7fb7e0078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.520+0000 7fb7e855c640 1 -- 172.21.15.31:0/1258544718 shutdown_connections 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.520+0000 7fb7e855c640 1 -- 172.21.15.31:0/1258544718 wait complete. 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.520+0000 7fb7e855c640 1 Processor -- start 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.520+0000 7fb7e855c640 1 -- start start 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e855c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7e01017c0 0x7fb7e0197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e855c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e0102160 0x7fb7e0198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e855c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e0107750 0x7fb7e019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e855c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e01119c0 con 0x7fb7e0107750 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e855c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e0111840 con 0x7fb7e0102160 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e855c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e0111b40 con 0x7fb7e01017c0 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e6af9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e0107750 0x7fb7e019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e6af9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e0107750 0x7fb7e019c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38820/0 (socket says 172.21.15.31:38820) 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e6af9640 1 -- 172.21.15.31:0/3500476928 learned_addr learned my addr 172.21.15.31:0/3500476928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e5af7640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e0102160 0x7fb7e0198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e62f8640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7e01017c0 0x7fb7e0197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e6af9640 1 -- 172.21.15.31:0/3500476928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7e01017c0 msgr2=0x7fb7e0197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e6af9640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7e01017c0 0x7fb7e0197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e6af9640 1 -- 172.21.15.31:0/3500476928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e0102160 msgr2=0x7fb7e0198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e6af9640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e0102160 0x7fb7e0198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e6af9640 1 -- 172.21.15.31:0/3500476928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e019ced0 con 0x7fb7e0107750 2024-04-25T01:54:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e5af7640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e0102160 0x7fb7e0198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e6af9640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e0107750 0x7fb7e019c7a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8006010 tx=0x7fb7d802f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7d37fe640 1 -- 172.21.15.31:0/3500476928 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8034040 con 0x7fb7e0107750 2024-04-25T01:54:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e855c640 1 -- 172.21.15.31:0/3500476928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e019d160 con 0x7fb7e0107750 2024-04-25T01:54:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7d37fe640 1 -- 172.21.15.31:0/3500476928 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb7d8003e00 con 0x7fb7e0107750 2024-04-25T01:54:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7d37fe640 1 -- 172.21.15.31:0/3500476928 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8045480 con 0x7fb7e0107750 2024-04-25T01:54:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.521+0000 7fb7e855c640 1 -- 172.21.15.31:0/3500476928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e01a4a50 con 0x7fb7e0107750 2024-04-25T01:54:35.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.522+0000 7fb7e855c640 1 -- 172.21.15.31:0/3500476928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7e00689b0 con 0x7fb7e0107750 2024-04-25T01:54:35.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.524+0000 7fb7d37fe640 1 -- 172.21.15.31:0/3500476928 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7fb7d8037020 con 0x7fb7e0107750 2024-04-25T01:54:35.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.524+0000 7fb7d37fe640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb7cc0a2d90 0x7fb7cc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.524+0000 7fb7e62f8640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb7cc0a2d90 0x7fb7cc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.524+0000 7fb7d37fe640 1 -- 172.21.15.31:0/3500476928 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fb7d80f06a0 con 0x7fb7e0107750 2024-04-25T01:54:35.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.525+0000 7fb7e62f8640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb7cc0a2d90 0x7fb7cc0a5250 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc005d70 tx=0x7fb7dc005d00 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.528+0000 7fb7d37fe640 1 -- 172.21.15.31:0/3500476928 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb7d80bd340 con 0x7fb7e0107750 2024-04-25T01:54:35.713 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:35 smithi031 ceph-mon[29204]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:35.713 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:35 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1276274216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:35.713 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:35 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3963992245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:35.713 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:35 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2449320549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:35.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.716+0000 7fb7e855c640 1 -- 172.21.15.31:0/3500476928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7e0199230 con 0x7fb7e0107750 2024-04-25T01:54:35.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.718+0000 7fb7d37fe640 1 -- 172.21.15.31:0/3500476928 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22855 (secure 0 0 0) 0x7fb7e0199230 con 0x7fb7e0107750 2024-04-25T01:54:35.720 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:54:35.720 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T01:54:28.579231+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3045817976":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6800/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1358173279":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/686937010":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/3913527371":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:54:35.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 -- 172.21.15.31:0/3500476928 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb7cc0a2d90 msgr2=0x7fb7cc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb7cc0a2d90 0x7fb7cc0a5250 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc005d70 tx=0x7fb7dc005d00 comp rx=0 tx=0).stop 2024-04-25T01:54:35.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 -- 172.21.15.31:0/3500476928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e0107750 msgr2=0x7fb7e019c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e0107750 0x7fb7e019c7a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8006010 tx=0x7fb7d802f640 comp rx=0 tx=0).stop 2024-04-25T01:54:35.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 -- 172.21.15.31:0/3500476928 shutdown_connections 2024-04-25T01:54:35.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb7cc0a2d90 0x7fb7cc0a5250 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7e0107750 0x7fb7e019c7a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e0102160 0x7fb7e0198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 --2- 172.21.15.31:0/3500476928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7e01017c0 0x7fb7e0197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 -- 172.21.15.31:0/3500476928 >> 172.21.15.31:0/3500476928 conn(0x7fb7e00788d0 msgr2=0x7fb7e0100600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 -- 172.21.15.31:0/3500476928 shutdown_connections 2024-04-25T01:54:35.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.720+0000 7fb7e855c640 1 -- 172.21.15.31:0/3500476928 wait complete. 2024-04-25T01:54:35.733 DEBUG:tasks.fwd_scrub:Yielding 2024-04-25T01:54:35.733 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T01:54:35.742 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:54:35.742 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T01:54:35.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-25T01:54:35.743 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-25T01:54:35.743 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:54:35.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:54:35.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:54:35.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7ff0e0934640 1 -- 172.21.15.31:0/1793963089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0dc100610 msgr2=0x7ff0dc100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7ff0e0934640 1 --2- 172.21.15.31:0/1793963089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0dc100610 0x7ff0dc100a90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009930 tx=0x7ff0d002a420 comp rx=0 tx=0).stop 2024-04-25T01:54:35.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7fbfb8615640 1 -- 172.21.15.31:0/1640933137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb0101020 msgr2=0x7fbfb010d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7fbfb8615640 1 --2- 172.21.15.31:0/1640933137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb0101020 0x7fbfb010d530 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009cd0 tx=0x7fbfa802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7ff0e0934640 1 -- 172.21.15.31:0/1793963089 shutdown_connections 2024-04-25T01:54:35.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7ff0e0934640 1 --2- 172.21.15.31:0/1793963089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0dc100fd0 0x7ff0dc10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7ff0e0934640 1 --2- 172.21.15.31:0/1793963089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0dc100610 0x7ff0dc100a90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7ff0e0934640 1 --2- 172.21.15.31:0/1793963089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0dc1066b0 0x7ff0dc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7ff0e0934640 1 -- 172.21.15.31:0/1793963089 >> 172.21.15.31:0/1793963089 conn(0x7ff0dc077740 msgr2=0x7ff0dc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7ff0e0934640 1 -- 172.21.15.31:0/1793963089 shutdown_connections 2024-04-25T01:54:35.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7fbfb8615640 1 -- 172.21.15.31:0/1640933137 shutdown_connections 2024-04-25T01:54:35.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7ff0e0934640 1 -- 172.21.15.31:0/1793963089 wait complete. 2024-04-25T01:54:35.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7fbfb8615640 1 --2- 172.21.15.31:0/1640933137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb0101020 0x7fbfb010d530 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7fbfb8615640 1 --2- 172.21.15.31:0/1640933137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb0100660 0x7fbfb0100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7fbfb8615640 1 --2- 172.21.15.31:0/1640933137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfb0106670 0x7fbfb0106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7fbfb8615640 1 -- 172.21.15.31:0/1640933137 >> 172.21.15.31:0/1640933137 conn(0x7fbfb00769c0 msgr2=0x7fbfb0076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7fbfb8615640 1 -- 172.21.15.31:0/1640933137 shutdown_connections 2024-04-25T01:54:35.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7fbfb8615640 1 -- 172.21.15.31:0/1640933137 wait complete. 2024-04-25T01:54:35.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.848+0000 7fbfb8615640 1 Processor -- start 2024-04-25T01:54:35.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb8615640 1 -- start start 2024-04-25T01:54:35.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0e0934640 1 Processor -- start 2024-04-25T01:54:35.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0e0934640 1 -- start start 2024-04-25T01:54:35.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb8615640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfb0100660 0x7fbfb019d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb8615640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb0101020 0x7fbfb019ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb8615640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb0106670 0x7fbfb0197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0e0934640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0dc100610 0x7ff0dc19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0e0934640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0dc100fd0 0x7ff0dc19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0e0934640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0dc1066b0 0x7ff0dc19e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb8615640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb01108f0 con 0x7fbfb0101020 2024-04-25T01:54:35.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0e0934640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc1119e0 con 0x7ff0dc1066b0 2024-04-25T01:54:35.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0e0934640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc111860 con 0x7ff0dc100fd0 2024-04-25T01:54:35.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb5bb0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb0101020 0x7fbfb019ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0e0934640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc111b60 con 0x7ff0dc100610 2024-04-25T01:54:35.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb8615640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb0110770 con 0x7fbfb0100660 2024-04-25T01:54:35.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0da59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0dc100610 0x7ff0dc19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0dad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0dc1066b0 0x7ff0dc19e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0d9d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0dc100fd0 0x7ff0dc19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0da59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0dc100610 0x7ff0dc19a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:51098/0 (socket says 172.21.15.31:51098) 2024-04-25T01:54:35.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0da59c640 1 -- 172.21.15.31:0/1198690005 learned_addr learned my addr 172.21.15.31:0/1198690005 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:35.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb63b1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfb0100660 0x7fbfb019d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb5bb0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb0101020 0x7fbfb019ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38850/0 (socket says 172.21.15.31:38850) 2024-04-25T01:54:35.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb5bb0640 1 -- 172.21.15.31:0/1847957808 learned_addr learned my addr 172.21.15.31:0/1847957808 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:35.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb6bb2640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb0106670 0x7fbfb0197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7fbfb8615640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb0110a70 con 0x7fbfb0106670 2024-04-25T01:54:35.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0d9d9b640 1 -- 172.21.15.31:0/1198690005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0dc100610 msgr2=0x7ff0dc19a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0d9d9b640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0dc100610 0x7ff0dc19a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0d9d9b640 1 -- 172.21.15.31:0/1198690005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0dc1066b0 msgr2=0x7ff0dc19e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0d9d9b640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0dc1066b0 0x7ff0dc19e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0d9d9b640 1 -- 172.21.15.31:0/1198690005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0dc19f0d0 con 0x7ff0dc100fd0 2024-04-25T01:54:35.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0da59c640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0dc100610 0x7ff0dc19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:35.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.849+0000 7ff0dad9d640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0dc1066b0 0x7ff0dc19e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:35.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfb63b1640 1 -- 172.21.15.31:0/1847957808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb0106670 msgr2=0x7fbfb0197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfb63b1640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb0106670 0x7fbfb0197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfb63b1640 1 -- 172.21.15.31:0/1847957808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb0101020 msgr2=0x7fbfb019ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfb63b1640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb0101020 0x7fbfb019ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfb63b1640 1 -- 172.21.15.31:0/1847957808 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb01981a0 con 0x7fbfb0100660 2024-04-25T01:54:35.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfb6bb2640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb0106670 0x7fbfb0197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:35.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfb5bb0640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb0101020 0x7fbfb019ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:35.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7ff0d9d9b640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0dc100fd0 0x7ff0dc19a620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0031440 tx=0x7ff0d0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfb63b1640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfb0100660 0x7fbfb019d870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00e9f0 tx=0x7fbfac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7ff0cb7fe640 1 -- 172.21.15.31:0/1198690005 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d003f070 con 0x7ff0dc100fd0 2024-04-25T01:54:35.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7ff0e0934640 1 -- 172.21.15.31:0/1198690005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0dc19f360 con 0x7ff0dc100fd0 2024-04-25T01:54:35.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7ff0cb7fe640 1 -- 172.21.15.31:0/1198690005 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0d0003aa0 con 0x7ff0dc100fd0 2024-04-25T01:54:35.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7ff0cb7fe640 1 -- 172.21.15.31:0/1198690005 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0035070 con 0x7ff0dc100fd0 2024-04-25T01:54:35.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7ff0e0934640 1 -- 172.21.15.31:0/1198690005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0dc1a6c50 con 0x7ff0dc100fd0 2024-04-25T01:54:35.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfa77fe640 1 -- 172.21.15.31:0/1847957808 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac00ccc0 con 0x7fbfb0100660 2024-04-25T01:54:35.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfb8615640 1 -- 172.21.15.31:0/1847957808 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb0198490 con 0x7fbfb0100660 2024-04-25T01:54:35.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfa77fe640 1 -- 172.21.15.31:0/1847957808 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfac0108e0 con 0x7fbfb0100660 2024-04-25T01:54:35.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfa77fe640 1 -- 172.21.15.31:0/1847957808 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac018880 con 0x7fbfb0100660 2024-04-25T01:54:35.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.850+0000 7fbfb8615640 1 -- 172.21.15.31:0/1847957808 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb01a4730 con 0x7fbfb0100660 2024-04-25T01:54:35.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.851+0000 7fbfb8615640 1 -- 172.21.15.31:0/1847957808 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfb004f500 con 0x7fbfb0100660 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.852+0000 7ff0cb7fe640 1 -- 172.21.15.31:0/1198690005 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7ff0d0043400 con 0x7ff0dc100fd0 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.852+0000 7ff0e3640740 1 Processor -- start 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0e3640740 1 -- start start 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe69980 0x55c93fe67cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe68290 0x55c93fe68710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe73010 0x55c93fe75400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0e3640740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c93fdcc2c0 con 0x55c93fe68290 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0e3640740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c93fd74910 con 0x55c93fe73010 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0e3640740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c93fdc8280 con 0x55c93fe69980 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0da59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe68290 0x55c93fe68710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0da59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe68290 0x55c93fe68710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38860/0 (socket says 172.21.15.31:38860) 2024-04-25T01:54:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0da59c640 1 -- 172.21.15.31:0/2062698084 learned_addr learned my addr 172.21.15.31:0/2062698084 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:35.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0dad9d640 1 --2- 172.21.15.31:0/2062698084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe69980 0x55c93fe67cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0d9d9b640 1 --2- 172.21.15.31:0/2062698084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe73010 0x55c93fe75400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7fbfa77fe640 1 -- 172.21.15.31:0/1847957808 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7fbfac010a90 con 0x7fbfb0100660 2024-04-25T01:54:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0da59c640 1 -- 172.21.15.31:0/2062698084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe69980 msgr2=0x55c93fe67cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0da59c640 1 --2- 172.21.15.31:0/2062698084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe69980 0x55c93fe67cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0da59c640 1 -- 172.21.15.31:0/2062698084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe73010 msgr2=0x55c93fe75400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0da59c640 1 --2- 172.21.15.31:0/2062698084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe73010 0x55c93fe75400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0da59c640 1 -- 172.21.15.31:0/2062698084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c93fd85fb0 con 0x55c93fe68290 2024-04-25T01:54:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0dad9d640 1 --2- 172.21.15.31:0/2062698084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe69980 0x55c93fe67cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.853+0000 7ff0cb7fe640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff0c40a2d90 0x7ff0c40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.854+0000 7ff0cb7fe640 1 -- 172.21.15.31:0/1198690005 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7ff0d00ef720 con 0x7ff0dc100fd0 2024-04-25T01:54:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.854+0000 7ff0dad9d640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff0c40a2d90 0x7ff0c40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.854+0000 7ff0da59c640 1 --2- 172.21.15.31:0/2062698084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe68290 0x55c93fe68710 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0115c0 tx=0x7ff0cc011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8faeae31eba44f61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.854+0000 7ff0c8ff9640 1 -- 172.21.15.31:0/2062698084 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc005020 con 0x55c93fe68290 2024-04-25T01:54:35.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.854+0000 7ff0c8ff9640 1 -- 172.21.15.31:0/2062698084 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0cc0047a0 con 0x55c93fe68290 2024-04-25T01:54:35.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.854+0000 7ff0c8ff9640 1 -- 172.21.15.31:0/2062698084 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc00f8a0 con 0x55c93fe68290 2024-04-25T01:54:35.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.854+0000 7fbfa77fe640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbf9c0a2e40 0x7fbf9c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.854+0000 7fbfb5bb0640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbf9c0a2e40 0x7fbf9c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.854+0000 7fbfa77fe640 1 -- 172.21.15.31:0/1847957808 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fbfac0cfd00 con 0x7fbfb0100660 2024-04-25T01:54:35.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7fbfb5bb0640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbf9c0a2e40 0x7fbf9c0a5300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0076de0 tx=0x7fbfa0005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0dad9d640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff0c40a2d90 0x7ff0c40a5250 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4005fd0 tx=0x7ff0d4009500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0e3640740 1 -- 172.21.15.31:0/2062698084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe68290 msgr2=0x55c93fe68710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0e3640740 1 --2- 172.21.15.31:0/2062698084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe68290 0x55c93fe68710 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0115c0 tx=0x7ff0cc011a80 comp rx=0 tx=0).stop 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0e3640740 1 -- 172.21.15.31:0/2062698084 shutdown_connections 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0e3640740 1 --2- 172.21.15.31:0/2062698084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe73010 0x55c93fe75400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0e3640740 1 --2- 172.21.15.31:0/2062698084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe68290 0x55c93fe68710 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0e3640740 1 --2- 172.21.15.31:0/2062698084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe69980 0x55c93fe67cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0e3640740 1 -- 172.21.15.31:0/2062698084 >> 172.21.15.31:0/2062698084 conn(0x55c93fe30950 msgr2=0x55c93fe647f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0e3640740 1 -- 172.21.15.31:0/2062698084 shutdown_connections 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0e3640740 1 -- 172.21.15.31:0/2062698084 wait complete. 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0e3640740 1 -- start start 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.855+0000 7ff0e3640740 1 Processor -- start 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe68290 0x55c93fe6ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe69980 0x55c93fe70450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe73010 0x55c93fe70990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0e3640740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c93fe75ea0 con 0x55c93fe73010 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0e3640740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c93fd87e90 con 0x55c93fe69980 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0e3640740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c93fe76020 con 0x55c93fe68290 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0e3640740 10 client.0 _ll_register_callbacks cb 0x55c93fdc3970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0dad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe73010 0x55c93fe70990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0dad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe73010 0x55c93fe70990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38868/0 (socket says 172.21.15.31:38868) 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0dad9d640 1 -- 172.21.15.31:0/3952845380 learned_addr learned my addr 172.21.15.31:0/3952845380 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0d9d9b640 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe69980 0x55c93fe70450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0da59c640 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe68290 0x55c93fe6ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0dad9d640 1 -- 172.21.15.31:0/3952845380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe68290 msgr2=0x55c93fe6ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0dad9d640 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe68290 0x55c93fe6ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0dad9d640 1 -- 172.21.15.31:0/3952845380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe69980 msgr2=0x55c93fe70450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0dad9d640 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe69980 0x55c93fe70450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0dad9d640 1 -- 172.21.15.31:0/3952845380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c93fba87e0 con 0x55c93fe73010 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0dad9d640 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe73010 0x55c93fe70990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40120f0 tx=0x7ff0d4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35fe12119eb0e83d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3952845380 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4011bd0 con 0x55c93fe73010 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3952845380 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7ff0d4011d70 con 0x55c93fe73010 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.856+0000 7ff0e3640740 1 -- 172.21.15.31:0/3952845380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0cc0047a0 con 0x55c93fe73010 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3952845380 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff0d402d020 con 0x55c93fe73010 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0e3640740 10 client.15549 fetch_fsmap learned FSMap version 27 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0e3640740 1 -- 172.21.15.31:0/3952845380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c93fc08f70 con 0x55c93fe73010 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3952845380 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0cc0047a0 con 0x55c93fe73010 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3952845380 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff0d4032430 con 0x55c93fe73010 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0e3640740 10 client.15549 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0e3640740 10 client.15549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0e3640740 20 client.15549 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c93fe830e0 0x55c93fe85530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0e3640740 4 client.15549 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0e3640740 1 -- 172.21.15.31:0/3952845380 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c93fda05a0 con 0x55c93fe830e0 2024-04-25T01:54:35.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.857+0000 7ff0da59c640 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c93fe830e0 0x55c93fe85530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.858+0000 7fbfa77fe640 1 -- 172.21.15.31:0/1847957808 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbfac09c1b0 con 0x7fbfb0100660 2024-04-25T01:54:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.858+0000 7ff0da59c640 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c93fe830e0 0x55c93fe85530 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf8890a75f82ec49 server_cookie=b5913563ca4e6257 in_seq=0 out_seq=0 2024-04-25T01:54:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.858+0000 7ff0c1ffb640 10 client.15549 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.858+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3952845380 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c93fda05a0 con 0x55c93fe830e0 2024-04-25T01:54:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.858+0000 7ff0c1ffb640 10 client.15549 handle_command_reply: tid=0 2024-04-25T01:54:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 1 client.15549 shutdown 2024-04-25T01:54:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 1 -- 172.21.15.31:0/3952845380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe73010 msgr2=0x55c93fe70990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe73010 0x55c93fe70990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40120f0 tx=0x7ff0d4004ab0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 1 -- 172.21.15.31:0/3952845380 shutdown_connections 2024-04-25T01:54:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c93fe830e0 0x55c93fe85530 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe73010 0x55c93fe70990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe69980 0x55c93fe70450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0c1ffb640 0 client.15549 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3952845380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe68290 0x55c93fe6ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 1 -- 172.21.15.31:0/3952845380 >> 172.21.15.31:0/3952845380 conn(0x55c93fe30950 msgr2=0x55c93fe6a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 1 -- 172.21.15.31:0/3952845380 shutdown_connections 2024-04-25T01:54:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 1 -- 172.21.15.31:0/3952845380 wait complete. 2024-04-25T01:54:35.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.859+0000 7ff0e3640740 20 client.15549 trim_cache size 0 max 16384 2024-04-25T01:54:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.876+0000 7ff0e3640740 1 Processor -- start 2024-04-25T01:54:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.876+0000 7ff0e3640740 1 -- start start 2024-04-25T01:54:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe68290 0x55c93fe7eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe637d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe73010 0x55c93fe8ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0e3640740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc1af4a0 con 0x55c93fe69980 2024-04-25T01:54:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0e3640740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc1b0fe0 con 0x55c93fe73010 2024-04-25T01:54:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0e3640740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc1afb70 con 0x55c93fe68290 2024-04-25T01:54:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0d9d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe637d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0d9d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe637d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38880/0 (socket says 172.21.15.31:38880) 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0d9d9b640 1 -- 172.21.15.31:0/1450050944 learned_addr learned my addr 172.21.15.31:0/1450050944 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0dad9d640 1 --2- 172.21.15.31:0/1450050944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe73010 0x55c93fe8ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0da59c640 1 --2- 172.21.15.31:0/1450050944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe68290 0x55c93fe7eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0d9d9b640 1 -- 172.21.15.31:0/1450050944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe68290 msgr2=0x55c93fe7eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0d9d9b640 1 --2- 172.21.15.31:0/1450050944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe68290 0x55c93fe7eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0d9d9b640 1 -- 172.21.15.31:0/1450050944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe73010 msgr2=0x55c93fe8ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0d9d9b640 1 --2- 172.21.15.31:0/1450050944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe73010 0x55c93fe8ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0d9d9b640 1 -- 172.21.15.31:0/1450050944 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c93fe64a00 con 0x55c93fe69980 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0da59c640 1 --2- 172.21.15.31:0/1450050944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe68290 0x55c93fe7eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0dad9d640 1 --2- 172.21.15.31:0/1450050944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe73010 0x55c93fe8ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.877+0000 7ff0d9d9b640 1 --2- 172.21.15.31:0/1450050944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe637d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00b8490 tx=0x7ff0d00058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317614d838cd4478 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0c3fff640 1 -- 172.21.15.31:0/1450050944 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d00f1e60 con 0x55c93fe69980 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0c3fff640 1 -- 172.21.15.31:0/1450050944 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0d00bc1c0 con 0x55c93fe69980 2024-04-25T01:54:35.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0c3fff640 1 -- 172.21.15.31:0/1450050944 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0048040 con 0x55c93fe69980 2024-04-25T01:54:35.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0e3640740 1 -- 172.21.15.31:0/1450050944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 msgr2=0x55c93fe637d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0e3640740 1 --2- 172.21.15.31:0/1450050944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe637d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00b8490 tx=0x7ff0d00058d0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0e3640740 1 -- 172.21.15.31:0/1450050944 shutdown_connections 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0e3640740 1 --2- 172.21.15.31:0/1450050944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe73010 0x55c93fe8ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0e3640740 1 --2- 172.21.15.31:0/1450050944 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe637d0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0e3640740 1 --2- 172.21.15.31:0/1450050944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe68290 0x55c93fe7eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0e3640740 1 -- 172.21.15.31:0/1450050944 >> 172.21.15.31:0/1450050944 conn(0x55c93fe30950 msgr2=0x55c93fe630c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0e3640740 1 -- 172.21.15.31:0/1450050944 shutdown_connections 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.878+0000 7ff0e3640740 1 -- 172.21.15.31:0/1450050944 wait complete. 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0e3640740 1 -- start start 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0e3640740 1 Processor -- start 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe68290 0x55c93fe7b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe78280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0e3640740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe73010 0x55c93fe787c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0e3640740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc1b0910 con 0x55c93fe69980 2024-04-25T01:54:35.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0e3640740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc1b0240 con 0x55c93fe68290 2024-04-25T01:54:35.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0e3640740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc1ad480 con 0x55c93fe73010 2024-04-25T01:54:35.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0e3640740 10 client.0 _ll_register_callbacks cb 0x7ff0d4012880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:35.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0d9d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe78280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0d9d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe78280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38884/0 (socket says 172.21.15.31:38884) 2024-04-25T01:54:35.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0d9d9b640 1 -- 172.21.15.31:0/3068053068 learned_addr learned my addr 172.21.15.31:0/3068053068 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:35.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0da59c640 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe68290 0x55c93fe7b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0d9d9b640 1 -- 172.21.15.31:0/3068053068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe73010 msgr2=0x55c93fe787c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:35.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0dad9d640 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe73010 0x55c93fe787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0d9d9b640 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe73010 0x55c93fe787c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0d9d9b640 1 -- 172.21.15.31:0/3068053068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe68290 msgr2=0x55c93fe7b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:35.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0d9d9b640 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe68290 0x55c93fe7b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0d9d9b640 1 -- 172.21.15.31:0/3068053068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c93faa0170 con 0x55c93fe69980 2024-04-25T01:54:35.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.879+0000 7ff0da59c640 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe68290 0x55c93fe7b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:35.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0d9d9b640 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe78280 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009900 tx=0x7ff0d00095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35eb8e299fd70f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3068053068 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d00bfe70 con 0x55c93fe69980 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3068053068 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7ff0d00b8130 con 0x55c93fe69980 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0e3640740 1 -- 172.21.15.31:0/3068053068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0d00bc1c0 con 0x55c93fe69980 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3068053068 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff0d00c4730 con 0x55c93fe69980 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0e3640740 10 client.15555 fetch_fsmap learned FSMap version 27 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0e3640740 1 -- 172.21.15.31:0/3068053068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c93feea190 con 0x55c93fe69980 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3068053068 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0d00bc1c0 con 0x55c93fe69980 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3068053068 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff0d004b060 con 0x55c93fe69980 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0e3640740 10 client.15555 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0e3640740 10 client.15555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0e3640740 20 client.15555 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c93feea800 0x55c93feecc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:35.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0e3640740 4 client.15555 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.880+0000 7ff0e3640740 1 -- 172.21.15.31:0/3068053068 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c93fdd9800 con 0x55c93feea800 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.881+0000 7ff0da59c640 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c93feea800 0x55c93feecc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.881+0000 7ff0da59c640 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c93feea800 0x55c93feecc20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a416423452527488 server_cookie=6e8197fd423e1fba in_seq=0 out_seq=0 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.881+0000 7ff0c1ffb640 10 client.15555 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0c1ffb640 1 -- 172.21.15.31:0/3068053068 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c93fdd9800 con 0x55c93feea800 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0c1ffb640 10 client.15555 handle_command_reply: tid=0 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 1 client.15555 shutdown 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 1 -- 172.21.15.31:0/3068053068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 msgr2=0x55c93fe78280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe78280 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009900 tx=0x7ff0d00095e0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 1 -- 172.21.15.31:0/3068053068 shutdown_connections 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c93feea800 0x55c93feecc20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c93fe73010 0x55c93fe787c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c93fe69980 0x55c93fe78280 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0c1ffb640 0 client.15555 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:35.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 1 --2- 172.21.15.31:0/3068053068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c93fe68290 0x55c93fe7b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 1 -- 172.21.15.31:0/3068053068 >> 172.21.15.31:0/3068053068 conn(0x55c93fe30950 msgr2=0x55c93fe63950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 1 -- 172.21.15.31:0/3068053068 shutdown_connections 2024-04-25T01:54:35.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 1 -- 172.21.15.31:0/3068053068 wait complete. 2024-04-25T01:54:35.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.882+0000 7ff0e3640740 20 client.15555 trim_cache size 0 max 16384 2024-04-25T01:54:35.893 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:54:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T01:54:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "00b417e0-3b27-44ad-a554-9d45f2a7ec3a", 2024-04-25T01:54:35.893 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T01:54:35.893 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:54:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 -- 172.21.15.31:0/1198690005 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff0c40a2d90 msgr2=0x7ff0c40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff0c40a2d90 0x7ff0c40a5250 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4005fd0 tx=0x7ff0d4009500 comp rx=0 tx=0).stop 2024-04-25T01:54:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 -- 172.21.15.31:0/1198690005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0dc100fd0 msgr2=0x7ff0dc19a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0dc100fd0 0x7ff0dc19a620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0031440 tx=0x7ff0d0004270 comp rx=0 tx=0).stop 2024-04-25T01:54:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 -- 172.21.15.31:0/1198690005 shutdown_connections 2024-04-25T01:54:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff0c40a2d90 0x7ff0c40a5250 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0dc1066b0 0x7ff0dc19e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0dc100fd0 0x7ff0dc19a620 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 --2- 172.21.15.31:0/1198690005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0dc100610 0x7ff0dc19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 -- 172.21.15.31:0/1198690005 >> 172.21.15.31:0/1198690005 conn(0x7ff0dc077740 msgr2=0x7ff0dc0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 -- 172.21.15.31:0/1198690005 shutdown_connections 2024-04-25T01:54:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.883+0000 7ff0e0934640 1 -- 172.21.15.31:0/1198690005 wait complete. 2024-04-25T01:54:35.896 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:54:35.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:54:35.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:35 smithi040 ceph-mon[32418]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:35.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:35 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1276274216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:35.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:35 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3963992245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:35.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:35 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2449320549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:35.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:35 smithi134 ceph-mon[33098]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:54:35.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:35 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1276274216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:35.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:35 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3963992245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:35.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:35 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2449320549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:36.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.999+0000 7f5d043d5640 1 -- 172.21.15.31:0/2032134033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cfc100fd0 msgr2=0x7f5cfc10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.999+0000 7f5d043d5640 1 --2- 172.21.15.31:0/2032134033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cfc100fd0 0x7f5cfc10d4e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4009cd0 tx=0x7f5cf402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.999+0000 7f5d043d5640 1 -- 172.21.15.31:0/2032134033 shutdown_connections 2024-04-25T01:54:36.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.999+0000 7f5d043d5640 1 --2- 172.21.15.31:0/2032134033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cfc100fd0 0x7f5cfc10d4e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.999+0000 7f5d043d5640 1 --2- 172.21.15.31:0/2032134033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cfc100610 0x7f5cfc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.999+0000 7f5d043d5640 1 --2- 172.21.15.31:0/2032134033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cfc1066b0 0x7f5cfc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.999+0000 7f5d043d5640 1 -- 172.21.15.31:0/2032134033 >> 172.21.15.31:0/2032134033 conn(0x7f5cfc077740 msgr2=0x7f5cfc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:36.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.999+0000 7f5d043d5640 1 -- 172.21.15.31:0/2032134033 shutdown_connections 2024-04-25T01:54:36.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:35.999+0000 7f5d043d5640 1 -- 172.21.15.31:0/2032134033 wait complete. 2024-04-25T01:54:36.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d043d5640 1 Processor -- start 2024-04-25T01:54:36.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d043d5640 1 -- start start 2024-04-25T01:54:36.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d043d5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cfc100610 0x7f5cfc197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d043d5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cfc100fd0 0x7f5cfc1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d043d5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cfc1066b0 0x7f5cfc19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d043d5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1108f0 con 0x7f5cfc100610 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d043d5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc110770 con 0x7f5cfc100fd0 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d043d5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc110a70 con 0x7f5cfc1066b0 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d02171640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cfc100610 0x7f5cfc197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d02171640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cfc100610 0x7f5cfc197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38898/0 (socket says 172.21.15.31:38898) 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d02171640 1 -- 172.21.15.31:0/2656966941 learned_addr learned my addr 172.21.15.31:0/2656966941 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.000+0000 7f5d01970640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cfc100fd0 0x7f5cfc1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5d02171640 1 -- 172.21.15.31:0/2656966941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cfc1066b0 msgr2=0x7f5cfc19c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5d02972640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cfc1066b0 0x7f5cfc19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5d02171640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cfc1066b0 0x7f5cfc19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5d02171640 1 -- 172.21.15.31:0/2656966941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cfc100fd0 msgr2=0x7f5cfc1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5d02171640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cfc100fd0 0x7f5cfc1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5d02171640 1 -- 172.21.15.31:0/2656966941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cfc19ce50 con 0x7f5cfc100610 2024-04-25T01:54:36.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5d02171640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cfc100610 0x7f5cfc197e60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c970 tx=0x7f5cf800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:36.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5cf37fe640 1 -- 172.21.15.31:0/2656966941 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8004220 con 0x7f5cfc100610 2024-04-25T01:54:36.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5d043d5640 1 -- 172.21.15.31:0/2656966941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cfc19d140 con 0x7f5cfc100610 2024-04-25T01:54:36.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5cf37fe640 1 -- 172.21.15.31:0/2656966941 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cf80043c0 con 0x7f5cfc100610 2024-04-25T01:54:36.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5cf37fe640 1 -- 172.21.15.31:0/2656966941 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8004a00 con 0x7f5cfc100610 2024-04-25T01:54:36.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.001+0000 7f5d043d5640 1 -- 172.21.15.31:0/2656966941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cfc1a49d0 con 0x7f5cfc100610 2024-04-25T01:54:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.004+0000 7f5d071be740 1 Processor -- start 2024-04-25T01:54:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.004+0000 7f5d071be740 1 -- start start 2024-04-25T01:54:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.004+0000 7f5d071be740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb0da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.004+0000 7f5d071be740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0df50 0x557a8fb180e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.004+0000 7f5d071be740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb18620 0x557a8fb1aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.004+0000 7f5d071be740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557a8fa726b0 con 0x557a8fb0d630 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.004+0000 7f5d071be740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557a8fa1a880 con 0x557a8fb0df50 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.004+0000 7f5d071be740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557a8fa6e670 con 0x557a8fb18620 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.004+0000 7f5d02972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb0da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.004+0000 7f5d02972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb0da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38912/0 (socket says 172.21.15.31:38912) 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5d02972640 1 -- 172.21.15.31:0/4280852202 learned_addr learned my addr 172.21.15.31:0/4280852202 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5cf37fe640 1 -- 172.21.15.31:0/2656966941 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7f5cf80079b0 con 0x7f5cfc100610 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5d01970640 1 --2- 172.21.15.31:0/4280852202 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb18620 0x557a8fb1aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5d02972640 1 -- 172.21.15.31:0/4280852202 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb18620 msgr2=0x557a8fb1aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5d02972640 1 --2- 172.21.15.31:0/4280852202 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb18620 0x557a8fb1aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5d02972640 1 -- 172.21.15.31:0/4280852202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0df50 msgr2=0x557a8fb180e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5d02972640 1 --2- 172.21.15.31:0/4280852202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0df50 0x557a8fb180e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5d02972640 1 -- 172.21.15.31:0/4280852202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a8f8507d0 con 0x557a8fb0d630 2024-04-25T01:54:36.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5d02972640 1 --2- 172.21.15.31:0/4280852202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb0da10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4009df0 tx=0x7f5cf40314c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5edefa3586ab791 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5cf0bfe640 1 -- 172.21.15.31:0/4280852202 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4031a50 con 0x557a8fb0d630 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5cf0bfe640 1 -- 172.21.15.31:0/4280852202 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cf4036040 con 0x557a8fb0d630 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5cf0bfe640 1 -- 172.21.15.31:0/4280852202 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf40045f0 con 0x557a8fb0d630 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.005+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ce80a2d90 0x7f5ce80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d02171640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ce80a2d90 0x7f5ce80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5cf37fe640 1 -- 172.21.15.31:0/2656966941 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5cf80cf320 con 0x7f5cfc100610 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d071be740 1 -- 172.21.15.31:0/4280852202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 msgr2=0x557a8fb0da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d071be740 1 --2- 172.21.15.31:0/4280852202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb0da10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4009df0 tx=0x7f5cf40314c0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d02171640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ce80a2d90 0x7f5ce80a5250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80cca60 tx=0x7f5cf801cb40 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d071be740 1 -- 172.21.15.31:0/4280852202 shutdown_connections 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d071be740 1 --2- 172.21.15.31:0/4280852202 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb18620 0x557a8fb1aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d071be740 1 --2- 172.21.15.31:0/4280852202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0df50 0x557a8fb180e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d071be740 1 --2- 172.21.15.31:0/4280852202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb0da10 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d071be740 1 -- 172.21.15.31:0/4280852202 >> 172.21.15.31:0/4280852202 conn(0x557a8fb09950 msgr2=0x557a8fb09d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d071be740 1 -- 172.21.15.31:0/4280852202 shutdown_connections 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d071be740 1 -- 172.21.15.31:0/4280852202 wait complete. 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.006+0000 7f5d071be740 1 -- start start 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d071be740 1 Processor -- start 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d071be740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb155c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d071be740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0df50 0x557a8fb15b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d071be740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb18620 0x557a8fb16040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d071be740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557a8fb1b3b0 con 0x557a8fb0d630 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d071be740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557a8fa2df50 con 0x557a8fb18620 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d071be740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557a8fb1b530 con 0x557a8fb0df50 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d071be740 10 client.0 _ll_register_callbacks cb 0x557a8fa69000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d02972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb155c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d02972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb155c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38914/0 (socket says 172.21.15.31:38914) 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d02972640 1 -- 172.21.15.31:0/2254576340 learned_addr learned my addr 172.21.15.31:0/2254576340 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d01970640 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0df50 0x557a8fb15b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d02972640 1 -- 172.21.15.31:0/2254576340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0df50 msgr2=0x557a8fb15b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d02972640 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0df50 0x557a8fb15b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d02972640 1 -- 172.21.15.31:0/2254576340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb18620 msgr2=0x557a8fb16040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d02972640 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb18620 0x557a8fb16040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d02972640 1 -- 172.21.15.31:0/2254576340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a8f7863f0 con 0x557a8fb0d630 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5d02972640 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb155c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5cf402a920 tx=0x7f5cf4004330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d792948e54ac21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.007+0000 7f5ce5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2254576340 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4054060 con 0x557a8fb0d630 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2254576340 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5cf4004520 con 0x557a8fb0d630 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5d071be740 1 -- 172.21.15.31:0/2254576340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cf4036040 con 0x557a8fb0d630 2024-04-25T01:54:36.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2254576340 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5cf4004720 con 0x557a8fb0d630 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5d071be740 10 client.15567 fetch_fsmap learned FSMap version 27 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5d071be740 1 -- 172.21.15.31:0/2254576340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557a8f7460f0 con 0x557a8fb0d630 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2254576340 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cf4036040 con 0x557a8fb0d630 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2254576340 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5cf4030070 con 0x557a8fb0d630 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5d071be740 10 client.15567 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5d071be740 10 client.15567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5d071be740 20 client.15567 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5d071be740 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a8fb28730 0x557a8fb2ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5d071be740 4 client.15567 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.008+0000 7f5d071be740 1 -- 172.21.15.31:0/2254576340 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a8fa7fbf0 con 0x557a8fb28730 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.009+0000 7f5d01970640 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a8fb28730 0x557a8fb2ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.009+0000 7f5ce5ffb640 10 client.15567 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.009+0000 7f5d01970640 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a8fb28730 0x557a8fb2ab20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53222b1e03f22c8b server_cookie=323e19074cc2c981 in_seq=0 out_seq=0 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2254576340 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a8fa7fbf0 con 0x557a8fb28730 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5ce5ffb640 10 client.15567 handle_command_reply: tid=0 2024-04-25T01:54:36.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 1 client.15567 shutdown 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 1 -- 172.21.15.31:0/2254576340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 msgr2=0x557a8fb155c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb155c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5cf402a920 tx=0x7f5cf4004330 comp rx=0 tx=0).stop 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 1 -- 172.21.15.31:0/2254576340 shutdown_connections 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a8fb28730 0x557a8fb2ab20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb18620 0x557a8fb16040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0df50 0x557a8fb15b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 1 --2- 172.21.15.31:0/2254576340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb0d630 0x557a8fb155c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5ce5ffb640 0 client.15567 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 1 -- 172.21.15.31:0/2254576340 >> 172.21.15.31:0/2254576340 conn(0x557a8fb09950 msgr2=0x557a8fb0f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 1 -- 172.21.15.31:0/2254576340 shutdown_connections 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 1 -- 172.21.15.31:0/2254576340 wait complete. 2024-04-25T01:54:36.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.010+0000 7f5d071be740 20 client.15567 trim_cache size 0 max 16384 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d071be740 1 Processor -- start 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d071be740 1 -- start start 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d071be740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0d630 0x557a8fb17cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d071be740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0df50 0x557a8fb14720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d071be740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb08970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d071be740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1ad1e0 con 0x557a8fb18620 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d071be740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1aed20 con 0x557a8fb0d630 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d071be740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1ad8b0 con 0x557a8fb0df50 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d02972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb08970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d02972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb08970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38924/0 (socket says 172.21.15.31:38924) 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d02972640 1 -- 172.21.15.31:0/1142853901 learned_addr learned my addr 172.21.15.31:0/1142853901 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d02972640 1 --2- 172.21.15.31:0/1142853901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0d630 0x557a8fb17cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d01970640 1 --2- 172.21.15.31:0/1142853901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0df50 0x557a8fb14720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d02972640 1 -- 172.21.15.31:0/1142853901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0df50 msgr2=0x557a8fb14720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d02972640 1 --2- 172.21.15.31:0/1142853901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0df50 0x557a8fb14720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d02972640 1 -- 172.21.15.31:0/1142853901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0d630 msgr2=0x557a8fb17cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d02972640 1 --2- 172.21.15.31:0/1142853901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0d630 0x557a8fb17cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d02972640 1 -- 172.21.15.31:0/1142853901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a8fafb090 con 0x557a8fb18620 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.028+0000 7f5d02972640 1 --2- 172.21.15.31:0/1142853901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb08970 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4030650 tx=0x7f5cf4047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8eefa851f4933eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5ce7fff640 1 -- 172.21.15.31:0/1142853901 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4054070 con 0x557a8fb18620 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5ce7fff640 1 -- 172.21.15.31:0/1142853901 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cf4036040 con 0x557a8fb18620 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5ce7fff640 1 -- 172.21.15.31:0/1142853901 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4030cd0 con 0x557a8fb18620 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5d071be740 1 -- 172.21.15.31:0/1142853901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 msgr2=0x557a8fb08970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5d071be740 1 --2- 172.21.15.31:0/1142853901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb08970 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4030650 tx=0x7f5cf4047040 comp rx=0 tx=0).stop 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5d071be740 1 -- 172.21.15.31:0/1142853901 shutdown_connections 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5d071be740 1 --2- 172.21.15.31:0/1142853901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb08970 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5d071be740 1 --2- 172.21.15.31:0/1142853901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0df50 0x557a8fb14720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5d071be740 1 --2- 172.21.15.31:0/1142853901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0d630 0x557a8fb17cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5d071be740 1 -- 172.21.15.31:0/1142853901 >> 172.21.15.31:0/1142853901 conn(0x557a8fb09950 msgr2=0x557a8fb08260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:36.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5d071be740 1 -- 172.21.15.31:0/1142853901 shutdown_connections 2024-04-25T01:54:36.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.029+0000 7f5d071be740 1 -- 172.21.15.31:0/1142853901 wait complete. 2024-04-25T01:54:36.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d071be740 1 -- start start 2024-04-25T01:54:36.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d071be740 1 Processor -- start 2024-04-25T01:54:36.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d071be740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0d630 0x557a8fb2e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d071be740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0df50 0x557a8fb2b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d071be740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb2ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d071be740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1ae650 con 0x557a8fb18620 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d071be740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1adf80 con 0x557a8fb0df50 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d071be740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1ab1c0 con 0x557a8fb0d630 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d071be740 10 client.0 _ll_register_callbacks cb 0x7f5cf403f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d02972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb2ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d02972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb2ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38938/0 (socket says 172.21.15.31:38938) 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d02972640 1 -- 172.21.15.31:0/2714604925 learned_addr learned my addr 172.21.15.31:0/2714604925 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d02972640 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0d630 0x557a8fb2e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d02972640 1 -- 172.21.15.31:0/2714604925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0d630 msgr2=0x557a8fb2e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d02972640 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0d630 0x557a8fb2e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d01970640 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0df50 0x557a8fb2b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d02972640 1 -- 172.21.15.31:0/2714604925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0df50 msgr2=0x557a8fb2b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d02972640 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0df50 0x557a8fb2b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d02972640 1 -- 172.21.15.31:0/2714604925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a8fb2c260 con 0x557a8fb18620 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.030+0000 7f5d01970640 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0df50 0x557a8fb2b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5d02972640 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb2ba60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4031040 tx=0x7f5cf405b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3fb7c58576f748 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5ce5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2714604925 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4054040 con 0x557a8fb18620 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2714604925 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5cf4003970 con 0x557a8fb18620 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5d071be740 1 -- 172.21.15.31:0/2714604925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cf4036040 con 0x557a8fb18620 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2714604925 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5cf40308d0 con 0x557a8fb18620 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5d071be740 10 client.15573 fetch_fsmap learned FSMap version 27 2024-04-25T01:54:36.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5d071be740 1 -- 172.21.15.31:0/2714604925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557a8fb8ea20 con 0x557a8fb18620 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2714604925 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cf4036040 con 0x557a8fb18620 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2714604925 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5cf4030040 con 0x557a8fb18620 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5d071be740 10 client.15573 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5d071be740 10 client.15573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5d071be740 20 client.15573 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5d071be740 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a8fb8f080 0x557a8fb91470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5d071be740 4 client.15573 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5d071be740 1 -- 172.21.15.31:0/2714604925 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557a8fa79f40 con 0x557a8fb8f080 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.031+0000 7f5d01970640 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a8fb8f080 0x557a8fb91470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5ce5ffb640 10 client.15573 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d01970640 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a8fb8f080 0x557a8fb91470 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fa0f991443f63b8 server_cookie=8a7f199712d80441 in_seq=0 out_seq=0 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5ce5ffb640 1 -- 172.21.15.31:0/2714604925 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557a8fa79f40 con 0x557a8fb8f080 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5ce5ffb640 10 client.15573 handle_command_reply: tid=0 2024-04-25T01:54:36.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 1 client.15573 shutdown 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 1 -- 172.21.15.31:0/2714604925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 msgr2=0x557a8fb2ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb2ba60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4031040 tx=0x7f5cf405b040 comp rx=0 tx=0).stop 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 1 -- 172.21.15.31:0/2714604925 shutdown_connections 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a8fb8f080 0x557a8fb91470 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8fb18620 0x557a8fb2ba60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5ce5ffb640 0 client.15573 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a8fb0df50 0x557a8fb2b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 1 --2- 172.21.15.31:0/2714604925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a8fb0d630 0x557a8fb2e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 1 -- 172.21.15.31:0/2714604925 >> 172.21.15.31:0/2714604925 conn(0x557a8fb09950 msgr2=0x557a8fb24010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 1 -- 172.21.15.31:0/2714604925 shutdown_connections 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 1 -- 172.21.15.31:0/2714604925 wait complete. 2024-04-25T01:54:36.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.032+0000 7f5d071be740 20 client.15573 trim_cache size 0 max 16384 2024-04-25T01:54:36.041 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:54:36.041 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:54:36.041 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T01:54:36.041 INFO:teuthology.orchestra.run.smithi031.stdout: "00b417e0-3b27-44ad-a554-9d45f2a7ec3a": { 2024-04-25T01:54:36.041 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T01:54:36.041 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "00b417e0-3b27-44ad-a554-9d45f2a7ec3a", 2024-04-25T01:54:36.041 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T01:54:36.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T01:54:36.041 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T01:54:36.041 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:54:36.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 -- 172.21.15.31:0/2656966941 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ce80a2d90 msgr2=0x7f5ce80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ce80a2d90 0x7f5ce80a5250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80cca60 tx=0x7f5cf801cb40 comp rx=0 tx=0).stop 2024-04-25T01:54:36.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 -- 172.21.15.31:0/2656966941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cfc100610 msgr2=0x7f5cfc197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cfc100610 0x7f5cfc197e60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c970 tx=0x7f5cf800ce30 comp rx=0 tx=0).stop 2024-04-25T01:54:36.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 -- 172.21.15.31:0/2656966941 shutdown_connections 2024-04-25T01:54:36.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ce80a2d90 0x7f5ce80a5250 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cfc1066b0 0x7f5cfc19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cfc100fd0 0x7f5cfc1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 --2- 172.21.15.31:0/2656966941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cfc100610 0x7f5cfc197e60 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 -- 172.21.15.31:0/2656966941 >> 172.21.15.31:0/2656966941 conn(0x7f5cfc077740 msgr2=0x7f5cfc0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 -- 172.21.15.31:0/2656966941 shutdown_connections 2024-04-25T01:54:36.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.033+0000 7f5d043d5640 1 -- 172.21.15.31:0/2656966941 wait complete. 2024-04-25T01:54:36.045 INFO:tasks.cephfs.filesystem:scrub status for tag:00b417e0-3b27-44ad-a554-9d45f2a7ec3a - {'path': '/', 'tag': '00b417e0-3b27-44ad-a554-9d45f2a7ec3a', 'options': 'recursive,force'} 2024-04-25T01:54:36.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.073+0000 7fbfb8615640 1 -- 172.21.15.31:0/1847957808 --> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fbfb006a6e0 con 0x7fbf9c0a2e40 2024-04-25T01:54:36.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.141+0000 7fbfa77fe640 1 -- 172.21.15.31:0/1847957808 <== mgr.15351 v2:172.21.15.31:6800/1834919992 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fbfb006a6e0 con 0x7fbf9c0a2e40 2024-04-25T01:54:36.142 INFO:teuthology.orchestra.run.smithi031.stdout:Schedule set for path / 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 -- 172.21.15.31:0/1847957808 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbf9c0a2e40 msgr2=0x7fbf9c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbf9c0a2e40 0x7fbf9c0a5300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0076de0 tx=0x7fbfa0005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 -- 172.21.15.31:0/1847957808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfb0100660 msgr2=0x7fbfb019d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfb0100660 0x7fbfb019d870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00e9f0 tx=0x7fbfac00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 -- 172.21.15.31:0/1847957808 shutdown_connections 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbf9c0a2e40 0x7fbf9c0a5300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb0106670 0x7fbfb0197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb0101020 0x7fbfb019ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 --2- 172.21.15.31:0/1847957808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfb0100660 0x7fbfb019d870 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 -- 172.21.15.31:0/1847957808 >> 172.21.15.31:0/1847957808 conn(0x7fbfb00769c0 msgr2=0x7fbfb0069fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 -- 172.21.15.31:0/1847957808 shutdown_connections 2024-04-25T01:54:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.143+0000 7fbfb8615640 1 -- 172.21.15.31:0/1847957808 wait complete. 2024-04-25T01:54:36.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-25T01:54:36.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.299+0000 7f1dfdf09640 1 -- 172.21.15.31:0/1759183360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df8100e70 msgr2=0x7f1df810dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.299+0000 7f1dfdf09640 1 --2- 172.21.15.31:0/1759183360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df8100e70 0x7f1df810dbc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009900 tx=0x7f1dec02a410 comp rx=0 tx=0).stop 2024-04-25T01:54:36.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.299+0000 7f1dfdf09640 1 -- 172.21.15.31:0/1759183360 shutdown_connections 2024-04-25T01:54:36.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.299+0000 7f1dfdf09640 1 --2- 172.21.15.31:0/1759183360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1df810e100 0x7f1df81104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.299+0000 7f1dfdf09640 1 --2- 172.21.15.31:0/1759183360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df8100e70 0x7f1df810dbc0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.299+0000 7f1dfdf09640 1 --2- 172.21.15.31:0/1759183360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8100550 0x7f1df8100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.299+0000 7f1dfdf09640 1 -- 172.21.15.31:0/1759183360 >> 172.21.15.31:0/1759183360 conn(0x7f1df80777c0 msgr2=0x7f1df8077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:36.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.299+0000 7f1dfdf09640 1 -- 172.21.15.31:0/1759183360 shutdown_connections 2024-04-25T01:54:36.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.299+0000 7f1dfdf09640 1 -- 172.21.15.31:0/1759183360 wait complete. 2024-04-25T01:54:36.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.300+0000 7f1dfdf09640 1 Processor -- start 2024-04-25T01:54:36.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.300+0000 7f1dfdf09640 1 -- start start 2024-04-25T01:54:36.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.300+0000 7f1dfdf09640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8100550 0x7f1df819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.300+0000 7f1dfdf09640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1df8100e70 0x7f1df819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.300+0000 7f1dfdf09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df810e100 0x7f1df819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.300+0000 7f1dfdf09640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8113c40 con 0x7f1df810e100 2024-04-25T01:54:36.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.300+0000 7f1dfdf09640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8113ac0 con 0x7f1df8100550 2024-04-25T01:54:36.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.300+0000 7f1dfdf09640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8113dc0 con 0x7f1df8100e70 2024-04-25T01:54:36.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df810e100 0x7f1df819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df810e100 0x7f1df819ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38952/0 (socket says 172.21.15.31:38952) 2024-04-25T01:54:36.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df7fff640 1 -- 172.21.15.31:0/4197185107 learned_addr learned my addr 172.21.15.31:0/4197185107 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df77fe640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8100550 0x7f1df819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df6ffd640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1df8100e70 0x7f1df819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df7fff640 1 -- 172.21.15.31:0/4197185107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1df8100e70 msgr2=0x7f1df819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df7fff640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1df8100e70 0x7f1df819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df7fff640 1 -- 172.21.15.31:0/4197185107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8100550 msgr2=0x7f1df819a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df7fff640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8100550 0x7f1df819a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df7fff640 1 -- 172.21.15.31:0/4197185107 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df819f180 con 0x7f1df810e100 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df6ffd640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1df8100e70 0x7f1df819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df7fff640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df810e100 0x7f1df819ea50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1df000b9e0 tx=0x7f1df000bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df4ff9640 1 -- 172.21.15.31:0/4197185107 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0019070 con 0x7f1df810e100 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df4ff9640 1 -- 172.21.15.31:0/4197185107 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1df0014440 con 0x7f1df810e100 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1dfdf09640 1 -- 172.21.15.31:0/4197185107 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df819f470 con 0x7f1df810e100 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1df4ff9640 1 -- 172.21.15.31:0/4197185107 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df00125a0 con 0x7f1df810e100 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.301+0000 7f1dfdf09640 1 -- 172.21.15.31:0/4197185107 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df81a6d00 con 0x7f1df810e100 2024-04-25T01:54:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.303+0000 7f1dfdf09640 1 -- 172.21.15.31:0/4197185107 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df804f500 con 0x7f1df810e100 2024-04-25T01:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.304+0000 7f1df4ff9640 1 -- 172.21.15.31:0/4197185107 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7f1df001c070 con 0x7f1df810e100 2024-04-25T01:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.304+0000 7f1df4ff9640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1de00a2ec0 0x7f1de00a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.305+0000 7f1df77fe640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1de00a2ec0 0x7f1de00a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.305+0000 7f1df4ff9640 1 -- 172.21.15.31:0/4197185107 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f1df00ce960 con 0x7f1df810e100 2024-04-25T01:54:36.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.305+0000 7f1df77fe640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1de00a2ec0 0x7f1de00a5380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1de800abe0 tx=0x7f1de800a2d0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:36.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.309+0000 7f1df4ff9640 1 -- 172.21.15.31:0/4197185107 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1df009b570 con 0x7f1df810e100 2024-04-25T01:54:36.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.518+0000 7f1dfdf09640 1 -- 172.21.15.31:0/4197185107 --> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f1df8061b80 con 0x7f1de00a2ec0 2024-04-25T01:54:36.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.579+0000 7f1df4ff9640 1 -- 172.21.15.31:0/4197185107 <== mgr.15351 v2:172.21.15.31:6800/1834919992 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f1df8061b80 con 0x7f1de00a2ec0 2024-04-25T01:54:36.580 INFO:teuthology.orchestra.run.smithi031.stdout:Retention added to path / 2024-04-25T01:54:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 -- 172.21.15.31:0/4197185107 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1de00a2ec0 msgr2=0x7f1de00a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1de00a2ec0 0x7f1de00a5380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1de800abe0 tx=0x7f1de800a2d0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 -- 172.21.15.31:0/4197185107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df810e100 msgr2=0x7f1df819ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df810e100 0x7f1df819ea50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1df000b9e0 tx=0x7f1df000bde0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 -- 172.21.15.31:0/4197185107 shutdown_connections 2024-04-25T01:54:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1de00a2ec0 0x7f1de00a5380 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1df810e100 0x7f1df819ea50 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1df8100e70 0x7f1df819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 --2- 172.21.15.31:0/4197185107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8100550 0x7f1df819a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 -- 172.21.15.31:0/4197185107 >> 172.21.15.31:0/4197185107 conn(0x7f1df80777c0 msgr2=0x7f1df810f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 -- 172.21.15.31:0/4197185107 shutdown_connections 2024-04-25T01:54:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.581+0000 7f1dfdf09640 1 -- 172.21.15.31:0/4197185107 wait complete. 2024-04-25T01:54:36.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T01:54:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.695+0000 7f70f727b640 1 -- 172.21.15.31:0/2570405771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70f0102100 msgr2=0x7f70f010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.695+0000 7f70f727b640 1 --2- 172.21.15.31:0/2570405771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70f0102100 0x7f70f010e610 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f70e8009cd0 tx=0x7f70e802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.695+0000 7f70f727b640 1 -- 172.21.15.31:0/2570405771 shutdown_connections 2024-04-25T01:54:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.695+0000 7f70f727b640 1 --2- 172.21.15.31:0/2570405771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70f0102100 0x7f70f010e610 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.695+0000 7f70f727b640 1 --2- 172.21.15.31:0/2570405771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70f0101740 0x7f70f0101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.695+0000 7f70f727b640 1 --2- 172.21.15.31:0/2570405771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70f0107750 0x7f70f0107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.695+0000 7f70f727b640 1 -- 172.21.15.31:0/2570405771 >> 172.21.15.31:0/2570405771 conn(0x7f70f00fd210 msgr2=0x7f70f00ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.695+0000 7f70f727b640 1 -- 172.21.15.31:0/2570405771 shutdown_connections 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.695+0000 7f70f727b640 1 -- 172.21.15.31:0/2570405771 wait complete. 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f727b640 1 Processor -- start 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f727b640 1 -- start start 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f727b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70f0101740 0x7f70f0073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f727b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70f0102100 0x7f70f0073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f727b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70f0107750 0x7f70f00744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f727b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f70f0111a10 con 0x7f70f0107750 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f727b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f70f0111890 con 0x7f70f0102100 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f727b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f70f0111b90 con 0x7f70f0101740 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f5818640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70f0107750 0x7f70f00744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f5818640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70f0107750 0x7f70f00744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38974/0 (socket says 172.21.15.31:38974) 2024-04-25T01:54:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f5818640 1 -- 172.21.15.31:0/2603554575 learned_addr learned my addr 172.21.15.31:0/2603554575 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f5017640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70f0101740 0x7f70f0073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f4816640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70f0102100 0x7f70f0073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f5818640 1 -- 172.21.15.31:0/2603554575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70f0101740 msgr2=0x7f70f0073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f5818640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70f0101740 0x7f70f0073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f5818640 1 -- 172.21.15.31:0/2603554575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70f0102100 msgr2=0x7f70f0073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f5818640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70f0102100 0x7f70f0073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f5818640 1 -- 172.21.15.31:0/2603554575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f01a6330 con 0x7f70f0107750 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.696+0000 7f70f5818640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70f0107750 0x7f70f00744c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f70e8005cc0 tx=0x7f70e8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.697+0000 7f70e67fc640 1 -- 172.21.15.31:0/2603554575 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e8004450 con 0x7f70f0107750 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.697+0000 7f70f727b640 1 -- 172.21.15.31:0/2603554575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70f01a65c0 con 0x7f70f0107750 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.697+0000 7f70e67fc640 1 -- 172.21.15.31:0/2603554575 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70e8031c90 con 0x7f70f0107750 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.697+0000 7f70e67fc640 1 -- 172.21.15.31:0/2603554575 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e8030e70 con 0x7f70f0107750 2024-04-25T01:54:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.697+0000 7f70f727b640 1 -- 172.21.15.31:0/2603554575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70f01a6b60 con 0x7f70f0107750 2024-04-25T01:54:36.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.698+0000 7f70f727b640 1 -- 172.21.15.31:0/2603554575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70f004f500 con 0x7f70f0107750 2024-04-25T01:54:36.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.699+0000 7f70e67fc640 1 -- 172.21.15.31:0/2603554575 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7f70e80045f0 con 0x7f70f0107750 2024-04-25T01:54:36.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.700+0000 7f70e67fc640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70dc0a2e40 0x7f70dc0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:36.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.700+0000 7f70f5017640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70dc0a2e40 0x7f70dc0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:36.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.700+0000 7f70e67fc640 1 -- 172.21.15.31:0/2603554575 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f70e802e030 con 0x7f70f0107750 2024-04-25T01:54:36.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.700+0000 7f70f5017640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70dc0a2e40 0x7f70dc0a5300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f70ec00ab60 tx=0x7f70ec00a2d0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.703+0000 7f70e67fc640 1 -- 172.21.15.31:0/2603554575 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f70e804b060 con 0x7f70f0107750 2024-04-25T01:54:36.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:36 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3500476928' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:36.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:36 smithi040 ceph-mon[32418]: scrub queued for path: / 2024-04-25T01:54:36.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:36 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:54:36.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:36 smithi040 ceph-mon[32418]: scrub summary: active paths [/] 2024-04-25T01:54:36.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:36 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:54:36.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:36 smithi040 ceph-mon[32418]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:36.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:36 smithi040 ceph-mon[32418]: scrub summary: idle 2024-04-25T01:54:36.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.910+0000 7f70f727b640 1 -- 172.21.15.31:0/2603554575 --> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f70f0109d40 con 0x7f70dc0a2e40 2024-04-25T01:54:36.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.918+0000 7f70e67fc640 1 -- 172.21.15.31:0/2603554575 <== mgr.15351 v2:172.21.15.31:6800/1834919992 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f70ec00fbc0 con 0x7f70dc0a2e40 2024-04-25T01:54:36.919 INFO:teuthology.orchestra.run.smithi031.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T01:54:36", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-25T01:54:36.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.920+0000 7f70f727b640 1 -- 172.21.15.31:0/2603554575 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70dc0a2e40 msgr2=0x7f70dc0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.920+0000 7f70f727b640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70dc0a2e40 0x7f70dc0a5300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f70ec00ab60 tx=0x7f70ec00a2d0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.921+0000 7f70f727b640 1 -- 172.21.15.31:0/2603554575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70f0107750 msgr2=0x7f70f00744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:36.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.921+0000 7f70f727b640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70f0107750 0x7f70f00744c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f70e8005cc0 tx=0x7f70e8003c20 comp rx=0 tx=0).stop 2024-04-25T01:54:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.921+0000 7f70f727b640 1 -- 172.21.15.31:0/2603554575 shutdown_connections 2024-04-25T01:54:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.921+0000 7f70f727b640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70dc0a2e40 0x7f70dc0a5300 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.921+0000 7f70f727b640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70f0107750 0x7f70f00744c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.921+0000 7f70f727b640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70f0102100 0x7f70f0073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.921+0000 7f70f727b640 1 --2- 172.21.15.31:0/2603554575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70f0101740 0x7f70f0073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.921+0000 7f70f727b640 1 -- 172.21.15.31:0/2603554575 >> 172.21.15.31:0/2603554575 conn(0x7f70f00fd210 msgr2=0x7f70f00fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.921+0000 7f70f727b640 1 -- 172.21.15.31:0/2603554575 shutdown_connections 2024-04-25T01:54:36.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:36.921+0000 7f70f727b640 1 -- 172.21.15.31:0/2603554575 wait complete. 2024-04-25T01:54:36.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:36 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3500476928' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:36.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:36 smithi031 ceph-mon[29204]: scrub queued for path: / 2024-04-25T01:54:36.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:36 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:54:36.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:36 smithi031 ceph-mon[29204]: scrub summary: active paths [/] 2024-04-25T01:54:36.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:36 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:54:36.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:36 smithi031 ceph-mon[29204]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:36.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:36 smithi031 ceph-mon[29204]: scrub summary: idle 2024-04-25T01:54:36.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T01:54:36.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:36 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3500476928' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:36.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:36 smithi134 ceph-mon[33098]: scrub queued for path: / 2024-04-25T01:54:36.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:36 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:54:36.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:36 smithi134 ceph-mon[33098]: scrub summary: active paths [/] 2024-04-25T01:54:36.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:36 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:54:36.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:36 smithi134 ceph-mon[33098]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:36.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:36 smithi134 ceph-mon[33098]: scrub summary: idle 2024-04-25T01:54:37.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.036+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2517817637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f78101030 msgr2=0x7f9f7810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:37.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.036+0000 7f9f7dea5640 1 --2- 172.21.15.31:0/2517817637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f78101030 0x7f9f7810d4f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009cd0 tx=0x7f9f7002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:54:37.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.036+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2517817637 shutdown_connections 2024-04-25T01:54:37.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.036+0000 7f9f7dea5640 1 --2- 172.21.15.31:0/2517817637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f78101030 0x7f9f7810d4f0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:37.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.036+0000 7f9f7dea5640 1 --2- 172.21.15.31:0/2517817637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f78100670 0x7f9f78100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:37.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.036+0000 7f9f7dea5640 1 --2- 172.21.15.31:0/2517817637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f78106630 0x7f9f78106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:37.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.036+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2517817637 >> 172.21.15.31:0/2517817637 conn(0x7f9f780777c0 msgr2=0x7f9f78077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:37.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.036+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2517817637 shutdown_connections 2024-04-25T01:54:37.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.036+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2517817637 wait complete. 2024-04-25T01:54:37.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.036+0000 7f9f7dea5640 1 Processor -- start 2024-04-25T01:54:37.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.036+0000 7f9f7dea5640 1 -- start start 2024-04-25T01:54:37.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f7dea5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f78100670 0x7f9f78197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:37.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f7dea5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f78101030 0x7f9f78198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:37.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f7dea5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f78106630 0x7f9f7819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:37.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f7dea5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9f781108f0 con 0x7f9f78106630 2024-04-25T01:54:37.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f7dea5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78110770 con 0x7f9f78101030 2024-04-25T01:54:37.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f7dea5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78110a70 con 0x7f9f78100670 2024-04-25T01:54:37.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f77fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f78106630 0x7f9f7819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:37.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f77fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f78106630 0x7f9f7819c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38994/0 (socket says 172.21.15.31:38994) 2024-04-25T01:54:37.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f77fff640 1 -- 172.21.15.31:0/2695623916 learned_addr learned my addr 172.21.15.31:0/2695623916 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:37.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f777fe640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f78100670 0x7f9f78197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f77fff640 1 -- 172.21.15.31:0/2695623916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f78100670 msgr2=0x7f9f78197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f77fff640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f78100670 0x7f9f78197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f77fff640 1 -- 172.21.15.31:0/2695623916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f78101030 msgr2=0x7f9f78198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f77fff640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f78101030 0x7f9f78198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f77fff640 1 -- 172.21.15.31:0/2695623916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7819cec0 con 0x7f9f78106630 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f77fff640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f78106630 0x7f9f7819c790 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9f70005cc0 tx=0x7f9f70003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f74ff9640 1 -- 172.21.15.31:0/2695623916 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f70004450 con 0x7f9f78106630 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2695623916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7819d150 con 0x7f9f78106630 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f74ff9640 1 -- 172.21.15.31:0/2695623916 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f70031c90 con 0x7f9f78106630 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.037+0000 7f9f74ff9640 1 -- 172.21.15.31:0/2695623916 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7003b400 con 0x7f9f78106630 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.038+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2695623916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f781a4a40 con 0x7f9f78106630 2024-04-25T01:54:37.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.039+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2695623916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f7804f500 con 0x7f9f78106630 2024-04-25T01:54:37.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.040+0000 7f9f74ff9640 1 -- 172.21.15.31:0/2695623916 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7f9f70004680 con 0x7f9f78106630 2024-04-25T01:54:37.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.040+0000 7f9f74ff9640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f600a2e40 0x7f9f600a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:37.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.040+0000 7f9f777fe640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f600a2e40 0x7f9f600a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:37.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.041+0000 7f9f74ff9640 1 -- 172.21.15.31:0/2695623916 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9f7002e030 con 0x7f9f78106630 2024-04-25T01:54:37.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.041+0000 7f9f777fe640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f600a2e40 0x7f9f600a5300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9f68005fd0 tx=0x7f9f6800b510 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:37.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.045+0000 7f9f74ff9640 1 -- 172.21.15.31:0/2695623916 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9f700bd330 con 0x7f9f78106630 2024-04-25T01:54:37.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.254+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2695623916 --> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f9f78061b80 con 0x7f9f600a2e40 2024-04-25T01:54:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.261+0000 7f9f74ff9640 1 -- 172.21.15.31:0/2695623916 <== mgr.15351 v2:172.21.15.31:6800/1834919992 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f9f78061b80 con 0x7f9f600a2e40 2024-04-25T01:54:37.262 INFO:teuthology.orchestra.run.smithi031.stdout:/ 1m 6m3h 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2695623916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f600a2e40 msgr2=0x7f9f600a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f600a2e40 0x7f9f600a5300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9f68005fd0 tx=0x7f9f6800b510 comp rx=0 tx=0).stop 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2695623916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f78106630 msgr2=0x7f9f7819c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f78106630 0x7f9f7819c790 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9f70005cc0 tx=0x7f9f70003c20 comp rx=0 tx=0).stop 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2695623916 shutdown_connections 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f600a2e40 0x7f9f600a5300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f78106630 0x7f9f7819c790 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f78101030 0x7f9f78198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 --2- 172.21.15.31:0/2695623916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f78100670 0x7f9f78197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2695623916 >> 172.21.15.31:0/2695623916 conn(0x7f9f780777c0 msgr2=0x7f9f78069130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2695623916 shutdown_connections 2024-04-25T01:54:37.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:54:37.265+0000 7f9f7dea5640 1 -- 172.21.15.31:0/2695623916 wait complete. 2024-04-25T01:54:37.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-25T01:54:37.307 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-25T01:54:37.318 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-25T01:54:37.340 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T01:54:37.340 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-25T01:54:37.350 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T01:54:37.385 INFO:tasks.workunit:Pulling workunits from ref d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:54:37.385 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T01:54:37.385 INFO:tasks.workunit:timeout=3h 2024-04-25T01:54:37.385 INFO:tasks.workunit:cleanup=True 2024-04-25T01:54:37.386 DEBUG:teuthology.orchestra.run.smithi031:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T01:54:37.401 INFO:teuthology.orchestra.run.smithi031.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T01:54:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-25T01:54:37.402 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 73h/115d Inode: 1 Links: 2 2024-04-25T01:54:37.402 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:54:37.402 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-25T01:54:37.402 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-25 01:52:32.118849590 +0000 2024-04-25T01:54:37.402 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-25 01:52:32.118849590 +0000 2024-04-25T01:54:37.402 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-25 01:54:28.200035444 +0000 2024-04-25T01:54:37.402 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2024-04-25T01:54:37.402 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T01:54:37.402 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T01:54:37.474 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:54:37.531 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T01:54:37.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:37 smithi040 ceph-mon[32418]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:37.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:37 smithi040 ceph-mon[32418]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-25T01:54:37.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:37 smithi040 ceph-mon[32418]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:37.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:37 smithi134 ceph-mon[33098]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:37.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:37 smithi134 ceph-mon[33098]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-25T01:54:37.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:37 smithi134 ceph-mon[33098]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:37.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:37 smithi031 ceph-mon[29204]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:37.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:37 smithi031 ceph-mon[29204]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-25T01:54:37.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:37 smithi031 ceph-mon[29204]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:38.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:38 smithi040 ceph-mon[32418]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:38.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:38 smithi134 ceph-mon[33098]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:38.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:38 smithi031 ceph-mon[29204]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:39.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:39 smithi040 ceph-mon[32418]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-25T01:54:39.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:39 smithi134 ceph-mon[33098]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-25T01:54:39.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:39 smithi031 ceph-mon[29204]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-25T01:54:41.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:41 smithi040 ceph-mon[32418]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-04-25T01:54:41.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:41 smithi134 ceph-mon[33098]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-04-25T01:54:41.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:41 smithi031 ceph-mon[29204]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-04-25T01:54:43.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:43 smithi040 ceph-mon[32418]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-25T01:54:43.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:43 smithi134 ceph-mon[33098]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-25T01:54:43.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:43 smithi031 ceph-mon[29204]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-25T01:54:45.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:45 smithi040 ceph-mon[32418]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-25T01:54:45.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:45 smithi134 ceph-mon[33098]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-25T01:54:45.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:45 smithi031 ceph-mon[29204]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-25T01:54:47.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:47 smithi040 ceph-mon[32418]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-04-25T01:54:47.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:47 smithi134 ceph-mon[33098]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-04-25T01:54:47.977 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:47 smithi031 ceph-mon[29204]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-04-25T01:54:49.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:49 smithi040 ceph-mon[32418]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-25T01:54:49.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:49 smithi134 ceph-mon[33098]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-25T01:54:49.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:49 smithi031 ceph-mon[29204]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-25T01:54:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.491+0000 7f78b685d640 1 -- 172.21.15.31:0/886166565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78b0107760 msgr2=0x7f78b0107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.491+0000 7f78b685d640 1 --2- 172.21.15.31:0/886166565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78b0107760 0x7f78b0107b40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009930 tx=0x7f78a002a420 comp rx=0 tx=0).stop 2024-04-25T01:54:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.491+0000 7f78b685d640 1 -- 172.21.15.31:0/886166565 shutdown_connections 2024-04-25T01:54:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.491+0000 7f78b685d640 1 --2- 172.21.15.31:0/886166565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78b0102110 0x7f78b010e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.491+0000 7f78b685d640 1 --2- 172.21.15.31:0/886166565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78b0101750 0x7f78b0101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.491+0000 7f78b685d640 1 --2- 172.21.15.31:0/886166565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78b0107760 0x7f78b0107b40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.491+0000 7f78b685d640 1 -- 172.21.15.31:0/886166565 >> 172.21.15.31:0/886166565 conn(0x7f78b00fd200 msgr2=0x7f78b00ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.491+0000 7f78b685d640 1 -- 172.21.15.31:0/886166565 shutdown_connections 2024-04-25T01:54:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.491+0000 7f78b685d640 1 -- 172.21.15.31:0/886166565 wait complete. 2024-04-25T01:54:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78b685d640 1 Processor -- start 2024-04-25T01:54:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78b685d640 1 -- start start 2024-04-25T01:54:50.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78b685d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78b0101750 0x7f78b0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78b685d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78b0102110 0x7f78b0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78b685d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78b0107760 0x7f78b019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78b685d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f78b0111a70 con 0x7f78b0107760 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78b685d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01118f0 con 0x7f78b0101750 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78b685d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f78b0111bf0 con 0x7f78b0102110 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78b4dfa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78b0107760 0x7f78b019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78b4dfa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78b0107760 0x7f78b019c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60636/0 (socket says 172.21.15.31:60636) 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78b4dfa640 1 -- 172.21.15.31:0/712232120 learned_addr learned my addr 172.21.15.31:0/712232120 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78af7fe640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78b0102110 0x7f78b0198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.492+0000 7f78affff640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78b0101750 0x7f78b0197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78b4dfa640 1 -- 172.21.15.31:0/712232120 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78b0102110 msgr2=0x7f78b0198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78b0102110 0x7f78b0198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78b4dfa640 1 -- 172.21.15.31:0/712232120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78b0101750 msgr2=0x7f78b0197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78b0101750 0x7f78b0197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78b4dfa640 1 -- 172.21.15.31:0/712232120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b019cee0 con 0x7f78b0107760 2024-04-25T01:54:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78affff640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78b0101750 0x7f78b0197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78b0107760 0x7f78b019c7b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f78a800eb10 tx=0x7f78a800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78ad7fa640 1 -- 172.21.15.31:0/712232120 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8004b20 con 0x7f78b0107760 2024-04-25T01:54:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78ad7fa640 1 -- 172.21.15.31:0/712232120 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a8004cc0 con 0x7f78b0107760 2024-04-25T01:54:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78b685d640 1 -- 172.21.15.31:0/712232120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b019d1d0 con 0x7f78b0107760 2024-04-25T01:54:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78ad7fa640 1 -- 172.21.15.31:0/712232120 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8010690 con 0x7f78b0107760 2024-04-25T01:54:50.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.493+0000 7f78b685d640 1 -- 172.21.15.31:0/712232120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b01a4a60 con 0x7f78b0107760 2024-04-25T01:54:50.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.495+0000 7f78ad7fa640 1 -- 172.21.15.31:0/712232120 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 24) ==== 138009+0+0 (secure 0 0 0) 0x7f78a8005020 con 0x7f78b0107760 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78b956f740 1 Processor -- start 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78b956f740 1 -- start start 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78b956f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5df240 0x55bcba5df620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78ad7fa640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f78980a2e40 0x7f78980a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78b956f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5dfb60 0x55bcba5e9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78b956f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5ec620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78b956f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bcba543cb0 con 0x55bcba5ea230 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78b956f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bcba4ec5b0 con 0x55bcba5dfb60 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78b956f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bcba53fc70 con 0x55bcba5df240 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78affff640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f78980a2e40 0x7f78980a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78b4dfa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78b4dfa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5ec620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60642/0 (socket says 172.21.15.31:60642) 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78b4dfa640 1 -- 172.21.15.31:0/495051480 learned_addr learned my addr 172.21.15.31:0/495051480 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:50.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78af7fe640 1 --2- 172.21.15.31:0/495051480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5df240 0x55bcba5df620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78ad7fa640 1 -- 172.21.15.31:0/712232120 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f78a80cfbc0 con 0x7f78b0107760 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.496+0000 7f78af7fe640 1 --2- 172.21.15.31:0/495051480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5dfb60 0x55bcba5e9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.497+0000 7f78b4dfa640 1 -- 172.21.15.31:0/495051480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5df240 msgr2=0x55bcba5df620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.497+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/495051480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5df240 0x55bcba5df620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.497+0000 7f78b4dfa640 1 -- 172.21.15.31:0/495051480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5dfb60 msgr2=0x55bcba5e9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.497+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/495051480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5dfb60 0x55bcba5e9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.497+0000 7f78b4dfa640 1 -- 172.21.15.31:0/495051480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcba258210 con 0x55bcba5ea230 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.497+0000 7f78affff640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f78980a2e40 0x7f78980a5300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78a0006010 tx=0x7f78a0000bc0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.497+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/495051480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5ec620 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f78a809bef0 tx=0x7f78a8007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e8209433c77aa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.497+0000 7f78967fc640 1 -- 172.21.15.31:0/495051480 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a809baa0 con 0x55bcba5ea230 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.497+0000 7f78967fc640 1 -- 172.21.15.31:0/495051480 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a8012eb0 con 0x55bcba5ea230 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.497+0000 7f78967fc640 1 -- 172.21.15.31:0/495051480 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a80b5160 con 0x55bcba5ea230 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.498+0000 7f78b956f740 1 -- 172.21.15.31:0/495051480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 msgr2=0x55bcba5ec620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.498+0000 7f78b956f740 1 --2- 172.21.15.31:0/495051480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5ec620 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f78a809bef0 tx=0x7f78a8007850 comp rx=0 tx=0).stop 2024-04-25T01:54:50.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.498+0000 7f78b956f740 1 -- 172.21.15.31:0/495051480 shutdown_connections 2024-04-25T01:54:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.498+0000 7f78b956f740 1 --2- 172.21.15.31:0/495051480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5ec620 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.498+0000 7f78b956f740 1 --2- 172.21.15.31:0/495051480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5dfb60 0x55bcba5e9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.498+0000 7f78b956f740 1 --2- 172.21.15.31:0/495051480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5df240 0x55bcba5df620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.498+0000 7f78b956f740 1 -- 172.21.15.31:0/495051480 >> 172.21.15.31:0/495051480 conn(0x55bcba5db560 msgr2=0x55bcba5db940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.498+0000 7f78b956f740 1 -- 172.21.15.31:0/495051480 shutdown_connections 2024-04-25T01:54:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.498+0000 7f78b956f740 1 -- 172.21.15.31:0/495051480 wait complete. 2024-04-25T01:54:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.498+0000 7f78b956f740 1 -- start start 2024-04-25T01:54:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.498+0000 7f78b956f740 1 Processor -- start 2024-04-25T01:54:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b956f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5df240 0x55bcba5e71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b956f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5dfb60 0x55bcba5e7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b956f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5e7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b956f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bcba5ed020 con 0x55bcba5ea230 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b956f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bcba4ffb50 con 0x55bcba5df240 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b956f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bcba5ed1a0 con 0x55bcba5dfb60 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b956f740 10 client.0 _ll_register_callbacks cb 0x55bcba53aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b4dfa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5e7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78af7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5dfb60 0x55bcba5e7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b4dfa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5e7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60658/0 (socket says 172.21.15.31:60658) 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b4dfa640 1 -- 172.21.15.31:0/580897413 learned_addr learned my addr 172.21.15.31:0/580897413 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78affff640 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5df240 0x55bcba5e71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b4dfa640 1 -- 172.21.15.31:0/580897413 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5dfb60 msgr2=0x55bcba5e7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5dfb60 0x55bcba5e7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b4dfa640 1 -- 172.21.15.31:0/580897413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5df240 msgr2=0x55bcba5e71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5df240 0x55bcba5e71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b4dfa640 1 -- 172.21.15.31:0/580897413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcba324620 con 0x55bcba5ea230 2024-04-25T01:54:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5e7c40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f78a8005850 tx=0x7f78a8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae34f15c72d2da1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.499+0000 7f788f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f788f7fe640 1 -- 172.21.15.31:0/580897413 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a80ba060 con 0x55bcba5ea230 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f788f7fe640 1 -- 172.21.15.31:0/580897413 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f78a8012ab0 con 0x55bcba5ea230 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f78b956f740 1 -- 172.21.15.31:0/580897413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bcba5511f0 con 0x55bcba5ea230 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f788f7fe640 1 -- 172.21.15.31:0/580897413 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f78a80b5340 con 0x55bcba5ea230 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f78b956f740 10 client.15603 fetch_fsmap learned FSMap version 27 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f78b956f740 1 -- 172.21.15.31:0/580897413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bcba5fd630 con 0x55bcba5ea230 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f788f7fe640 1 -- 172.21.15.31:0/580897413 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55bcba5511f0 con 0x55bcba5ea230 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f788f7fe640 1 -- 172.21.15.31:0/580897413 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f78a809f070 con 0x55bcba5ea230 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f78b956f740 10 client.15603 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f78b956f740 10 client.15603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f78b956f740 20 client.15603 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f78b956f740 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bcba5fe980 0x55bcba600d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f78b956f740 4 client.15603 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.500+0000 7f78b956f740 1 -- 172.21.15.31:0/580897413 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcba54b540 con 0x55bcba5fe980 2024-04-25T01:54:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.501+0000 7f78affff640 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bcba5fe980 0x55bcba600d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.501+0000 7f78affff640 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bcba5fe980 0x55bcba600d70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7d6d391f25f978e server_cookie=710811e3cd7e9418 in_seq=0 out_seq=0 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.501+0000 7f788f7fe640 10 client.15603 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f788f7fe640 1 -- 172.21.15.31:0/580897413 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bcba54b540 con 0x55bcba5fe980 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f788f7fe640 10 client.15603 handle_command_reply: tid=0 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 1 client.15603 shutdown 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 1 -- 172.21.15.31:0/580897413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 msgr2=0x55bcba5e7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5e7c40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f78a8005850 tx=0x7f78a8004ab0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 1 -- 172.21.15.31:0/580897413 shutdown_connections 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bcba5fe980 0x55bcba600d70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5ea230 0x55bcba5e7c40 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5dfb60 0x55bcba5e7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 1 --2- 172.21.15.31:0/580897413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5df240 0x55bcba5e71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 1 -- 172.21.15.31:0/580897413 >> 172.21.15.31:0/580897413 conn(0x55bcba5db560 msgr2=0x55bcba5e1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f788f7fe640 0 client.15603 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 1 -- 172.21.15.31:0/580897413 shutdown_connections 2024-04-25T01:54:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 1 -- 172.21.15.31:0/580897413 wait complete. 2024-04-25T01:54:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.502+0000 7f78b956f740 20 client.15603 trim_cache size 0 max 16384 2024-04-25T01:54:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.517+0000 7f78b956f740 1 Processor -- start 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.517+0000 7f78b956f740 1 -- start start 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.517+0000 7f78b956f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba5e98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.517+0000 7f78b956f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5dfb60 0x55bcba5e1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.517+0000 7f78b956f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5ea230 0x55bcba603d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.517+0000 7f78b956f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01ab0e0 con 0x55bcba5df240 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.517+0000 7f78b956f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01acc20 con 0x55bcba5ea230 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.517+0000 7f78b956f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01ab7b0 con 0x55bcba5dfb60 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba5e98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba5e98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60664/0 (socket says 172.21.15.31:60664) 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78affff640 1 -- 172.21.15.31:0/2483609858 learned_addr learned my addr 172.21.15.31:0/2483609858 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78affff640 1 -- 172.21.15.31:0/2483609858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5dfb60 msgr2=0x55bcba5e1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/2483609858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5ea230 0x55bcba603d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78af7fe640 1 --2- 172.21.15.31:0/2483609858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5dfb60 0x55bcba5e1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78affff640 1 --2- 172.21.15.31:0/2483609858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5dfb60 0x55bcba5e1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78affff640 1 -- 172.21.15.31:0/2483609858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5ea230 msgr2=0x55bcba603d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78affff640 1 --2- 172.21.15.31:0/2483609858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5ea230 0x55bcba603d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78affff640 1 -- 172.21.15.31:0/2483609858 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcba5e1ff0 con 0x55bcba5df240 2024-04-25T01:54:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/2483609858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5ea230 0x55bcba603d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78af7fe640 1 --2- 172.21.15.31:0/2483609858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5dfb60 0x55bcba5e1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f78affff640 1 --2- 172.21.15.31:0/2483609858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba5e98f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f78a0031440 tx=0x7f78a003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e4c5b5cd4e074b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f7895ffb640 1 -- 172.21.15.31:0/2483609858 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0056070 con 0x55bcba5df240 2024-04-25T01:54:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f7895ffb640 1 -- 172.21.15.31:0/2483609858 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a0050d60 con 0x55bcba5df240 2024-04-25T01:54:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.518+0000 7f7895ffb640 1 -- 172.21.15.31:0/2483609858 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a004d030 con 0x55bcba5df240 2024-04-25T01:54:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.519+0000 7f78b956f740 1 -- 172.21.15.31:0/2483609858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 msgr2=0x55bcba5e98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.519+0000 7f78b956f740 1 --2- 172.21.15.31:0/2483609858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba5e98f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f78a0031440 tx=0x7f78a003f000 comp rx=0 tx=0).stop 2024-04-25T01:54:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.519+0000 7f78b956f740 1 -- 172.21.15.31:0/2483609858 shutdown_connections 2024-04-25T01:54:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.519+0000 7f78b956f740 1 --2- 172.21.15.31:0/2483609858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5ea230 0x55bcba603d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.519+0000 7f78b956f740 1 --2- 172.21.15.31:0/2483609858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5dfb60 0x55bcba5e1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.519+0000 7f78b956f740 1 --2- 172.21.15.31:0/2483609858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba5e98f0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.519+0000 7f78b956f740 1 -- 172.21.15.31:0/2483609858 >> 172.21.15.31:0/2483609858 conn(0x55bcba5db560 msgr2=0x55bcba5da5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.519+0000 7f78b956f740 1 -- 172.21.15.31:0/2483609858 shutdown_connections 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.519+0000 7f78b956f740 1 -- 172.21.15.31:0/2483609858 wait complete. 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.519+0000 7f78b956f740 1 -- start start 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78b956f740 1 Processor -- start 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78b956f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba6042f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78b956f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5dfb60 0x55bcba601680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78b956f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5ea230 0x55bcba601dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78b956f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01ac550 con 0x55bcba5df240 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78b956f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01abe80 con 0x55bcba5dfb60 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78b956f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01a90c0 con 0x55bcba5ea230 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78b956f740 10 client.0 _ll_register_callbacks cb 0x7f78a4009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba6042f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba6042f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60672/0 (socket says 172.21.15.31:60672) 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78affff640 1 -- 172.21.15.31:0/272556963 learned_addr learned my addr 172.21.15.31:0/272556963 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78b4dfa640 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5ea230 0x55bcba601dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78af7fe640 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5dfb60 0x55bcba601680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78affff640 1 -- 172.21.15.31:0/272556963 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5ea230 msgr2=0x55bcba601dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78affff640 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5ea230 0x55bcba601dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78affff640 1 -- 172.21.15.31:0/272556963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5dfb60 msgr2=0x55bcba601680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78affff640 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5dfb60 0x55bcba601680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78affff640 1 -- 172.21.15.31:0/272556963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcba663b60 con 0x55bcba5df240 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.520+0000 7f78affff640 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba6042f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f78a002a940 tx=0x7f78a00043d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0f71e3d7bf5add server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f788f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f788f7fe640 1 -- 172.21.15.31:0/272556963 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0056040 con 0x55bcba5df240 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f78b956f740 1 -- 172.21.15.31:0/272556963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78a0050d60 con 0x55bcba5df240 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f788f7fe640 1 -- 172.21.15.31:0/272556963 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f78a0004b90 con 0x55bcba5df240 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f788f7fe640 1 -- 172.21.15.31:0/272556963 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f78a004d040 con 0x55bcba5df240 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f78b956f740 10 client.15609 fetch_fsmap learned FSMap version 27 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f78b956f740 1 -- 172.21.15.31:0/272556963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bcba664c80 con 0x55bcba5df240 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f788f7fe640 1 -- 172.21.15.31:0/272556963 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a0050d60 con 0x55bcba5df240 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f788f7fe640 1 -- 172.21.15.31:0/272556963 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f78a00325d0 con 0x55bcba5df240 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f78b956f740 10 client.15609 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:54:50.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f78b956f740 10 client.15609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:54:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f78b956f740 20 client.15609 populate_metadata read hostname 'smithi031' 2024-04-25T01:54:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f78b956f740 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bcba665400 0x55bcba667820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f78b956f740 4 client.15609 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:54:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.521+0000 7f78b956f740 1 -- 172.21.15.31:0/272556963 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bcba4c98e0 con 0x55bcba665400 2024-04-25T01:54:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.522+0000 7f78af7fe640 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bcba665400 0x55bcba667820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.522+0000 7f788f7fe640 10 client.15609 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:50.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.522+0000 7f78af7fe640 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bcba665400 0x55bcba667820 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e889f731d184dfd server_cookie=cd0bc9f54dc46448 in_seq=0 out_seq=0 2024-04-25T01:54:50.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f788f7fe640 1 -- 172.21.15.31:0/272556963 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bcba4c98e0 con 0x55bcba665400 2024-04-25T01:54:50.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f788f7fe640 10 client.15609 handle_command_reply: tid=0 2024-04-25T01:54:50.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 1 client.15609 shutdown 2024-04-25T01:54:50.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 1 -- 172.21.15.31:0/272556963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 msgr2=0x55bcba6042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba6042f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f78a002a940 tx=0x7f78a00043d0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 1 -- 172.21.15.31:0/272556963 shutdown_connections 2024-04-25T01:54:50.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bcba665400 0x55bcba667820 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bcba5ea230 0x55bcba601dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcba5dfb60 0x55bcba601680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 1 --2- 172.21.15.31:0/272556963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bcba5df240 0x55bcba6042f0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f788f7fe640 0 client.15609 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:54:50.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 1 -- 172.21.15.31:0/272556963 >> 172.21.15.31:0/272556963 conn(0x55bcba5db560 msgr2=0x55bcba5f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:50.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 1 -- 172.21.15.31:0/272556963 shutdown_connections 2024-04-25T01:54:50.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 1 -- 172.21.15.31:0/272556963 wait complete. 2024-04-25T01:54:50.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.534+0000 7f78b956f740 20 client.15609 trim_cache size 0 max 16384 2024-04-25T01:54:50.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:54:50.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:54:50.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:54:50.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:54:50.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 -- 172.21.15.31:0/712232120 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f78980a2e40 msgr2=0x7f78980a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f78980a2e40 0x7f78980a5300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78a0006010 tx=0x7f78a0000bc0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 -- 172.21.15.31:0/712232120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78b0107760 msgr2=0x7f78b019c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78b0107760 0x7f78b019c7b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f78a800eb10 tx=0x7f78a800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 -- 172.21.15.31:0/712232120 shutdown_connections 2024-04-25T01:54:50.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f78980a2e40 0x7f78980a5300 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78b0107760 0x7f78b019c7b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78b0102110 0x7f78b0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 --2- 172.21.15.31:0/712232120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78b0101750 0x7f78b0197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 -- 172.21.15.31:0/712232120 >> 172.21.15.31:0/712232120 conn(0x7f78b00fd200 msgr2=0x7f78b00fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:50.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 -- 172.21.15.31:0/712232120 shutdown_connections 2024-04-25T01:54:50.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:54:50.535+0000 7f78b685d640 1 -- 172.21.15.31:0/712232120 wait complete. 2024-04-25T01:54:51.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:51 smithi040 ceph-mon[32418]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-25T01:54:51.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:51 smithi134 ceph-mon[33098]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-25T01:54:51.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:51 smithi031 ceph-mon[29204]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-25T01:54:53.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:53 smithi040 ceph-mon[32418]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T01:54:53.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:53 smithi134 ceph-mon[33098]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T01:54:53.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:53 smithi031 ceph-mon[29204]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T01:54:55.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:55 smithi040 ceph-mon[32418]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-25T01:54:55.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:55 smithi134 ceph-mon[33098]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-25T01:54:55.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:55 smithi031 ceph-mon[29204]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-25T01:54:57.910 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:57 smithi040 ceph-mon[32418]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 17 op/s 2024-04-25T01:54:57.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:57 smithi134 ceph-mon[33098]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 17 op/s 2024-04-25T01:54:57.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:57 smithi031 ceph-mon[29204]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 17 op/s 2024-04-25T01:54:59.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:54:59 smithi134 ceph-mon[33098]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-04-25T01:54:59.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:54:59 smithi031 ceph-mon[29204]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-04-25T01:55:00.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:54:59 smithi040 ceph-mon[32418]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-04-25T01:55:00.988 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:55:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:55:00.590+0000 7f7d361b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:00.988 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:55:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:55:00.590+0000 7f7d361b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:00.989 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:55:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:55:00.590+0000 7f7d361b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:00.989 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:55:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:55:00.590+0000 7f7d361b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:00.989 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 01:55:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T01:55:00.590+0000 7f7d361b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:01.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:01 smithi134 ceph-mon[33098]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.1 KiB/s wr, 17 op/s 2024-04-25T01:55:01.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:01 smithi031 ceph-mon[29204]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.1 KiB/s wr, 17 op/s 2024-04-25T01:55:02.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:01 smithi040 ceph-mon[32418]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.1 KiB/s wr, 17 op/s 2024-04-25T01:55:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.635+0000 7fa4652f5640 1 -- 172.21.15.31:0/33643974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa460103310 msgr2=0x7fa46010b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.635+0000 7fa4652f5640 1 --2- 172.21.15.31:0/33643974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa460103310 0x7fa46010b7f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa454009900 tx=0x7fa45402a480 comp rx=0 tx=0).stop 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.636+0000 7fa4652f5640 1 -- 172.21.15.31:0/33643974 shutdown_connections 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.636+0000 7fa4652f5640 1 --2- 172.21.15.31:0/33643974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa46010bdc0 0x7fa46010e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.636+0000 7fa4652f5640 1 --2- 172.21.15.31:0/33643974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa460103310 0x7fa46010b7f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.636+0000 7fa4652f5640 1 --2- 172.21.15.31:0/33643974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4601029f0 0x7fa460102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.636+0000 7fa4652f5640 1 -- 172.21.15.31:0/33643974 >> 172.21.15.31:0/33643974 conn(0x7fa4600769c0 msgr2=0x7fa460076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.636+0000 7fa4652f5640 1 -- 172.21.15.31:0/33643974 shutdown_connections 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.636+0000 7fa4652f5640 1 -- 172.21.15.31:0/33643974 wait complete. 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.636+0000 7fa4652f5640 1 Processor -- start 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa4652f5640 1 -- start start 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa4652f5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4601029f0 0x7fa460197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa4652f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa460103310 0x7fa4601983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa4652f5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa46010bdc0 0x7fa46019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa4652f5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa460111a00 con 0x7fa46010bdc0 2024-04-25T01:55:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa4652f5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa460111880 con 0x7fa460103310 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa4652f5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa460111b80 con 0x7fa4601029f0 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa46010bdc0 0x7fa46019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4601029f0 0x7fa460197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4601029f0 0x7fa460197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:57500/0 (socket says 172.21.15.31:57500) 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45effd640 1 -- 172.21.15.31:0/602496711 learned_addr learned my addr 172.21.15.31:0/602496711 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa46010bdc0 0x7fa46019c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57582/0 (socket says 172.21.15.31:57582) 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa460103310 0x7fa4601983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45f7fe640 1 -- 172.21.15.31:0/602496711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4601029f0 msgr2=0x7fa460197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45f7fe640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4601029f0 0x7fa460197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45f7fe640 1 -- 172.21.15.31:0/602496711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa460103310 msgr2=0x7fa4601983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45f7fe640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa460103310 0x7fa4601983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45f7fe640 1 -- 172.21.15.31:0/602496711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46019ce90 con 0x7fa46010bdc0 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45effd640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4601029f0 0x7fa460197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.637+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa460103310 0x7fa4601983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.638+0000 7fa45f7fe640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa46010bdc0 0x7fa46019c760 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa45800d650 tx=0x7fa45800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.638+0000 7fa447fff640 1 -- 172.21.15.31:0/602496711 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458017070 con 0x7fa46010bdc0 2024-04-25T01:55:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.638+0000 7fa447fff640 1 -- 172.21.15.31:0/602496711 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa458009820 con 0x7fa46010bdc0 2024-04-25T01:55:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.638+0000 7fa4652f5640 1 -- 172.21.15.31:0/602496711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46019d180 con 0x7fa46010bdc0 2024-04-25T01:55:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.638+0000 7fa447fff640 1 -- 172.21.15.31:0/602496711 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458004c60 con 0x7fa46010bdc0 2024-04-25T01:55:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.638+0000 7fa4652f5640 1 -- 172.21.15.31:0/602496711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4601a4a10 con 0x7fa46010bdc0 2024-04-25T01:55:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.640+0000 7fa447fff640 1 -- 172.21.15.31:0/602496711 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7fa4580099d0 con 0x7fa46010bdc0 2024-04-25T01:55:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.640+0000 7fa468000740 1 Processor -- start 2024-04-25T01:55:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.640+0000 7fa468000740 1 -- start start 2024-04-25T01:55:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa468000740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd11140 0x55559bd11520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa468000740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd20100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa468000740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd20640 0x55559bd22a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa468000740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55559bc75cb0 con 0x55559bd20640 2024-04-25T01:55:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa468000740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55559bc1e5b0 con 0x55559bd11140 2024-04-25T01:55:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa468000740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55559bc71c70 con 0x55559bd11a60 2024-04-25T01:55:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45e7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd20640 0x55559bd22a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45e7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd20640 0x55559bd22a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57592/0 (socket says 172.21.15.31:57592) 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45e7fc640 1 -- 172.21.15.31:0/3363390820 learned_addr learned my addr 172.21.15.31:0/3363390820 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45effd640 1 --2- 172.21.15.31:0/3363390820 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd20100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/3363390820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd11140 0x55559bd11520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa447fff640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4480a2ec0 0x7fa4480a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45e7fc640 1 -- 172.21.15.31:0/3363390820 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 msgr2=0x55559bd20100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/3363390820 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd20100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45e7fc640 1 -- 172.21.15.31:0/3363390820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd11140 msgr2=0x55559bd11520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/3363390820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd11140 0x55559bd11520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45e7fc640 1 -- 172.21.15.31:0/3363390820 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55559b98a210 con 0x55559bd20640 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45effd640 1 --2- 172.21.15.31:0/3363390820 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd20100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.641+0000 7fa45f7fe640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4480a2ec0 0x7fa4480a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.642+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/3363390820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd20640 0x55559bd22a30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa454005c40 tx=0x7fa454005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245614300d370e72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.642+0000 7fa447fff640 1 -- 172.21.15.31:0/602496711 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa4580cef70 con 0x7fa46010bdc0 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.642+0000 7fa4457fa640 1 -- 172.21.15.31:0/3363390820 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454035040 con 0x55559bd20640 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.642+0000 7fa4457fa640 1 -- 172.21.15.31:0/3363390820 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa454004030 con 0x55559bd20640 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.642+0000 7fa4457fa640 1 -- 172.21.15.31:0/3363390820 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454003ab0 con 0x55559bd20640 2024-04-25T01:55:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.642+0000 7fa468000740 1 -- 172.21.15.31:0/3363390820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd20640 msgr2=0x55559bd22a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.642+0000 7fa45f7fe640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4480a2ec0 0x7fa4480a5380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa4580cc590 tx=0x7fa4580c5dc0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.642+0000 7fa468000740 1 --2- 172.21.15.31:0/3363390820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd20640 0x55559bd22a30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa454005c40 tx=0x7fa454005bd0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 -- 172.21.15.31:0/3363390820 shutdown_connections 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 --2- 172.21.15.31:0/3363390820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd20640 0x55559bd22a30 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 --2- 172.21.15.31:0/3363390820 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd20100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 --2- 172.21.15.31:0/3363390820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd11140 0x55559bd11520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 -- 172.21.15.31:0/3363390820 >> 172.21.15.31:0/3363390820 conn(0x55559bd0d560 msgr2=0x55559bd0d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 -- 172.21.15.31:0/3363390820 shutdown_connections 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 -- 172.21.15.31:0/3363390820 wait complete. 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 -- start start 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 Processor -- start 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11140 0x55559bd1d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11a60 0x55559bd1dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 0x55559bd1e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55559bd233a0 con 0x55559bd11a60 2024-04-25T01:55:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55559bc31b50 con 0x55559bd20640 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.643+0000 7fa468000740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55559bd23520 con 0x55559bd11140 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa45effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11a60 0x55559bd1dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa468000740 10 client.0 _ll_register_callbacks cb 0x55559bc6cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa45effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11a60 0x55559bd1dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57604/0 (socket says 172.21.15.31:57604) 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa45effd640 1 -- 172.21.15.31:0/1364602381 learned_addr learned my addr 172.21.15.31:0/1364602381 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11140 0x55559bd1d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 0x55559bd1e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa45effd640 1 -- 172.21.15.31:0/1364602381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11140 msgr2=0x55559bd1d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa45effd640 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11140 0x55559bd1d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa45effd640 1 -- 172.21.15.31:0/1364602381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 msgr2=0x55559bd1e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa45effd640 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 0x55559bd1e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa45effd640 1 -- 172.21.15.31:0/1364602381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55559ba56620 con 0x55559bd11a60 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa45effd640 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11a60 0x55559bd1dae0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa45000ff40 tx=0x7fa4500128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17be3ce936297201 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa442ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa442ff5640 1 -- 172.21.15.31:0/1364602381 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450010430 con 0x55559bd11a60 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa442ff5640 1 -- 172.21.15.31:0/1364602381 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa4500105d0 con 0x55559bd11a60 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa468000740 1 -- 172.21.15.31:0/1364602381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa454004030 con 0x55559bd11a60 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa442ff5640 1 -- 172.21.15.31:0/1364602381 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa4500107d0 con 0x55559bd11a60 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa468000740 10 client.15624 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.644+0000 7fa468000740 1 -- 172.21.15.31:0/1364602381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55559bd2f640 con 0x55559bd11a60 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.645+0000 7fa442ff5640 1 -- 172.21.15.31:0/1364602381 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa454004030 con 0x55559bd11a60 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.645+0000 7fa442ff5640 1 -- 172.21.15.31:0/1364602381 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa45000ea10 con 0x55559bd11a60 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.645+0000 7fa468000740 10 client.15624 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.645+0000 7fa468000740 10 client.15624 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.645+0000 7fa468000740 20 client.15624 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.645+0000 7fa468000740 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55559bd30ba0 0x55559bd32fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.645+0000 7fa468000740 4 client.15624 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.645+0000 7fa468000740 1 -- 172.21.15.31:0/1364602381 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55559bc831f0 con 0x55559bd30ba0 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.645+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55559bd30ba0 0x55559bd32fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.646+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55559bd30ba0 0x55559bd32fb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd8c46d0b2cc8d19 server_cookie=3d99ef1e13dc7cce in_seq=0 out_seq=0 2024-04-25T01:55:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.646+0000 7fa442ff5640 10 client.15624 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.646+0000 7fa442ff5640 1 -- 172.21.15.31:0/1364602381 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55559bc831f0 con 0x55559bd30ba0 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.646+0000 7fa442ff5640 10 client.15624 handle_command_reply: tid=0 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.646+0000 7fa468000740 1 client.15624 shutdown 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa468000740 1 -- 172.21.15.31:0/1364602381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11a60 msgr2=0x55559bd1dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa468000740 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11a60 0x55559bd1dae0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa45000ff40 tx=0x7fa4500128e0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa468000740 1 -- 172.21.15.31:0/1364602381 shutdown_connections 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa468000740 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55559bd30ba0 0x55559bd32fb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa468000740 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 0x55559bd1e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa468000740 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11a60 0x55559bd1dae0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa468000740 1 --2- 172.21.15.31:0/1364602381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11140 0x55559bd1d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa442ff5640 0 client.15624 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa468000740 1 -- 172.21.15.31:0/1364602381 >> 172.21.15.31:0/1364602381 conn(0x55559bd0d560 msgr2=0x55559bd13390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa468000740 1 -- 172.21.15.31:0/1364602381 shutdown_connections 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa468000740 1 -- 172.21.15.31:0/1364602381 wait complete. 2024-04-25T01:55:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.647+0000 7fa468000740 20 client.15624 trim_cache size 0 max 16384 2024-04-25T01:55:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa468000740 1 Processor -- start 2024-04-25T01:55:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa468000740 1 -- start start 2024-04-25T01:55:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa468000740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd1fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa468000740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd32f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa468000740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 0x55559bd2c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa468000740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601aaf70 con 0x55559bd11140 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa468000740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601acab0 con 0x55559bd20640 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa468000740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601ab640 con 0x55559bd11a60 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa45e7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd1fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa45e7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd1fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57612/0 (socket says 172.21.15.31:57612) 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa45e7fc640 1 -- 172.21.15.31:0/2545068733 learned_addr learned my addr 172.21.15.31:0/2545068733 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa45e7fc640 1 -- 172.21.15.31:0/2545068733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 msgr2=0x55559bd32f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/2545068733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd32f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa45e7fc640 1 -- 172.21.15.31:0/2545068733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 msgr2=0x55559bd2c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/2545068733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 0x55559bd2c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.662+0000 7fa45e7fc640 1 -- 172.21.15.31:0/2545068733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55559bd1baf0 con 0x55559bd11140 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/2545068733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd1fcd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa454033000 tx=0x7fa454005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ebd72ced4b05e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa444ff9640 1 -- 172.21.15.31:0/2545068733 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454035040 con 0x55559bd11140 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa444ff9640 1 -- 172.21.15.31:0/2545068733 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4540568d0 con 0x55559bd11140 2024-04-25T01:55:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa444ff9640 1 -- 172.21.15.31:0/2545068733 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454032360 con 0x55559bd11140 2024-04-25T01:55:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa468000740 1 -- 172.21.15.31:0/2545068733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 msgr2=0x55559bd1fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa468000740 1 --2- 172.21.15.31:0/2545068733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd1fcd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa454033000 tx=0x7fa454005bd0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa468000740 1 -- 172.21.15.31:0/2545068733 shutdown_connections 2024-04-25T01:55:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa468000740 1 --2- 172.21.15.31:0/2545068733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 0x55559bd2c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa468000740 1 --2- 172.21.15.31:0/2545068733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd32f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa468000740 1 --2- 172.21.15.31:0/2545068733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd1fcd0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa468000740 1 -- 172.21.15.31:0/2545068733 >> 172.21.15.31:0/2545068733 conn(0x55559bd0d560 msgr2=0x55559bd0bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa468000740 1 -- 172.21.15.31:0/2545068733 shutdown_connections 2024-04-25T01:55:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.663+0000 7fa468000740 1 -- 172.21.15.31:0/2545068733 wait complete. 2024-04-25T01:55:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa468000740 1 -- start start 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa468000740 1 Processor -- start 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa468000740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd24190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa468000740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd246d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa468000740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 0x55559bd24c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa468000740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601ac3e0 con 0x55559bd11140 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa468000740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601abd10 con 0x55559bd20640 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa468000740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa4601a8f70 con 0x55559bd11a60 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa468000740 10 client.0 _ll_register_callbacks cb 0x7fa45000f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa45e7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd24190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa45e7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd24190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57618/0 (socket says 172.21.15.31:57618) 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa45e7fc640 1 -- 172.21.15.31:0/3245476724 learned_addr learned my addr 172.21.15.31:0/3245476724 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa45effd640 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd246d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa45e7fc640 1 -- 172.21.15.31:0/3245476724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 msgr2=0x55559bd246d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd246d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa45e7fc640 1 -- 172.21.15.31:0/3245476724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 msgr2=0x55559bd24c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 0x55559bd24c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa45e7fc640 1 -- 172.21.15.31:0/3245476724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55559bd96130 con 0x55559bd11140 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.664+0000 7fa45effd640 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd246d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa45e7fc640 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd24190 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa454032f00 tx=0x7fa454005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655b642e10b02f82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa442ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa442ff5640 1 -- 172.21.15.31:0/3245476724 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454063060 con 0x55559bd11140 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa442ff5640 1 -- 172.21.15.31:0/3245476724 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa454056630 con 0x55559bd11140 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa468000740 1 -- 172.21.15.31:0/3245476724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4540568d0 con 0x55559bd11140 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa442ff5640 1 -- 172.21.15.31:0/3245476724 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa454035040 con 0x55559bd11140 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa468000740 10 client.15630 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa468000740 1 -- 172.21.15.31:0/3245476724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55559bd973f0 con 0x55559bd11140 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa442ff5640 1 -- 172.21.15.31:0/3245476724 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4540568d0 con 0x55559bd11140 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa442ff5640 1 -- 172.21.15.31:0/3245476724 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa454056ce0 con 0x55559bd11140 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa468000740 10 client.15630 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa468000740 10 client.15630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:03.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa468000740 20 client.15630 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:03.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa468000740 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55559bd97670 0x55559bd99a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa468000740 4 client.15630 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:55:03.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.665+0000 7fa468000740 1 -- 172.21.15.31:0/3245476724 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55559bc7d540 con 0x55559bd97670 2024-04-25T01:55:03.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.666+0000 7fa45effd640 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55559bd97670 0x55559bd99a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.666+0000 7fa442ff5640 10 client.15630 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:03.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.666+0000 7fa45effd640 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55559bd97670 0x55559bd99a60 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e659a1427371633 server_cookie=1235344dfd255ce0 in_seq=0 out_seq=0 2024-04-25T01:55:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa442ff5640 1 -- 172.21.15.31:0/3245476724 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55559bc7d540 con 0x55559bd97670 2024-04-25T01:55:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa442ff5640 10 client.15630 handle_command_reply: tid=0 2024-04-25T01:55:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 1 client.15630 shutdown 2024-04-25T01:55:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 1 -- 172.21.15.31:0/3245476724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 msgr2=0x55559bd24190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd24190 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa454032f00 tx=0x7fa454005970 comp rx=0 tx=0).stop 2024-04-25T01:55:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 1 -- 172.21.15.31:0/3245476724 shutdown_connections 2024-04-25T01:55:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55559bd97670 0x55559bd99a60 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55559bd20640 0x55559bd24c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55559bd11a60 0x55559bd246d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 1 --2- 172.21.15.31:0/3245476724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55559bd11140 0x55559bd24190 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 1 -- 172.21.15.31:0/3245476724 >> 172.21.15.31:0/3245476724 conn(0x55559bd0d560 msgr2=0x55559bd0c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa442ff5640 0 client.15630 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 1 -- 172.21.15.31:0/3245476724 shutdown_connections 2024-04-25T01:55:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 1 -- 172.21.15.31:0/3245476724 wait complete. 2024-04-25T01:55:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.673+0000 7fa468000740 20 client.15630 trim_cache size 0 max 16384 2024-04-25T01:55:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:55:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:55:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:55:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:55:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 -- 172.21.15.31:0/602496711 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4480a2ec0 msgr2=0x7fa4480a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4480a2ec0 0x7fa4480a5380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa4580cc590 tx=0x7fa4580c5dc0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 -- 172.21.15.31:0/602496711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa46010bdc0 msgr2=0x7fa46019c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa46010bdc0 0x7fa46019c760 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa45800d650 tx=0x7fa45800db10 comp rx=0 tx=0).stop 2024-04-25T01:55:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 -- 172.21.15.31:0/602496711 shutdown_connections 2024-04-25T01:55:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4480a2ec0 0x7fa4480a5380 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa46010bdc0 0x7fa46019c760 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa460103310 0x7fa4601983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 --2- 172.21.15.31:0/602496711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4601029f0 0x7fa460197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 -- 172.21.15.31:0/602496711 >> 172.21.15.31:0/602496711 conn(0x7fa4600769c0 msgr2=0x7fa4600fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 -- 172.21.15.31:0/602496711 shutdown_connections 2024-04-25T01:55:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:03.674+0000 7fa4652f5640 1 -- 172.21.15.31:0/602496711 wait complete. 2024-04-25T01:55:03.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:03 smithi134 ceph-mon[33098]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-25T01:55:03.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:03 smithi134 ceph-mon[33098]: mgrmap e25: x(active, since 34s), standbys: z, y 2024-04-25T01:55:03.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:03 smithi031 ceph-mon[29204]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-25T01:55:03.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:03 smithi031 ceph-mon[29204]: mgrmap e25: x(active, since 34s), standbys: z, y 2024-04-25T01:55:04.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:03 smithi040 ceph-mon[32418]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-25T01:55:04.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:03 smithi040 ceph-mon[32418]: mgrmap e25: x(active, since 34s), standbys: z, y 2024-04-25T01:55:05.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:05 smithi134 ceph-mon[33098]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:55:05.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:05 smithi031 ceph-mon[29204]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:55:06.046 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:55:06.046 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:55:06.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:05 smithi040 ceph-mon[32418]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-25T01:55:06.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.164+0000 7f0d03f29640 1 -- 172.21.15.31:0/1647090662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cfc107730 msgr2=0x7f0cfc107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.164+0000 7f0d03f29640 1 --2- 172.21.15.31:0/1647090662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cfc107730 0x7f0cfc107b10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8009930 tx=0x7f0cf802a420 comp rx=0 tx=0).stop 2024-04-25T01:55:06.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.164+0000 7f0d03f29640 1 -- 172.21.15.31:0/1647090662 shutdown_connections 2024-04-25T01:55:06.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.164+0000 7f0d03f29640 1 --2- 172.21.15.31:0/1647090662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cfc06a780 0x7f0cfc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.164+0000 7f0d03f29640 1 --2- 172.21.15.31:0/1647090662 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cfc069dc0 0x7f0cfc06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.164+0000 7f0d03f29640 1 --2- 172.21.15.31:0/1647090662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cfc107730 0x7f0cfc107b10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.164+0000 7f0d03f29640 1 -- 172.21.15.31:0/1647090662 >> 172.21.15.31:0/1647090662 conn(0x7f0cfc0776a0 msgr2=0x7f0cfc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.164+0000 7f0d03f29640 1 -- 172.21.15.31:0/1647090662 shutdown_connections 2024-04-25T01:55:06.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.164+0000 7f0d03f29640 1 -- 172.21.15.31:0/1647090662 wait complete. 2024-04-25T01:55:06.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.164+0000 7f0d03f29640 1 Processor -- start 2024-04-25T01:55:06.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d03f29640 1 -- start start 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d03f29640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cfc069dc0 0x7f0cfc100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d03f29640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cfc06a780 0x7f0cfc100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d03f29640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cfc107730 0x7f0cfc1010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d03f29640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc1119f0 con 0x7f0cfc069dc0 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d03f29640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc111870 con 0x7f0cfc107730 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d03f29640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc111b70 con 0x7f0cfc06a780 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d01cc5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cfc069dc0 0x7f0cfc100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d01cc5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cfc069dc0 0x7f0cfc100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57634/0 (socket says 172.21.15.31:57634) 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d01cc5640 1 -- 172.21.15.31:0/1645973342 learned_addr learned my addr 172.21.15.31:0/1645973342 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d014c4640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cfc06a780 0x7f0cfc100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d024c6640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cfc107730 0x7f0cfc1010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d01cc5640 1 -- 172.21.15.31:0/1645973342 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cfc06a780 msgr2=0x7f0cfc100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cfc06a780 0x7f0cfc100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d01cc5640 1 -- 172.21.15.31:0/1645973342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cfc107730 msgr2=0x7f0cfc1010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cfc107730 0x7f0cfc1010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d01cc5640 1 -- 172.21.15.31:0/1645973342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cfc1a64b0 con 0x7f0cfc069dc0 2024-04-25T01:55:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.165+0000 7f0d024c6640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cfc107730 0x7f0cfc1010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.166+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cfc069dc0 0x7f0cfc100670 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8004770 tx=0x7f0cf80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.166+0000 7f0cf2ffd640 1 -- 172.21.15.31:0/1645973342 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf803f070 con 0x7f0cfc069dc0 2024-04-25T01:55:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.166+0000 7f0cf2ffd640 1 -- 172.21.15.31:0/1645973342 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0cf803b460 con 0x7f0cfc069dc0 2024-04-25T01:55:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.166+0000 7f0cf2ffd640 1 -- 172.21.15.31:0/1645973342 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8035070 con 0x7f0cfc069dc0 2024-04-25T01:55:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.166+0000 7f0d03f29640 1 -- 172.21.15.31:0/1645973342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cfc1a6740 con 0x7f0cfc069dc0 2024-04-25T01:55:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.166+0000 7f0d03f29640 1 -- 172.21.15.31:0/1645973342 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cfc1a6c30 con 0x7f0cfc069dc0 2024-04-25T01:55:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.168+0000 7f0cf2ffd640 1 -- 172.21.15.31:0/1645973342 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f0cf803b610 con 0x7f0cfc069dc0 2024-04-25T01:55:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d06c3a740 1 Processor -- start 2024-04-25T01:55:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d06c3a740 1 -- start start 2024-04-25T01:55:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471a92a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92f50 0x55c471a9d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a9d620 0x55c471a9fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c4719f76b0 con 0x55c471a92630 2024-04-25T01:55:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c47199f880 con 0x55c471a92f50 2024-04-25T01:55:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c4719f3670 con 0x55c471a9d620 2024-04-25T01:55:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0cf2ffd640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0ce80a2e20 0x7f0ce80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d024c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471a92a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d024c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471a92a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57644/0 (socket says 172.21.15.31:57644) 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d01cc5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92f50 0x55c471a9d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d024c6640 1 -- 172.21.15.31:0/1235326109 learned_addr learned my addr 172.21.15.31:0/1235326109 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d024c6640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0ce80a2e20 0x7f0ce80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.169+0000 7f0d014c4640 1 --2- 172.21.15.31:0/1235326109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a9d620 0x55c471a9fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.170+0000 7f0cf2ffd640 1 -- 172.21.15.31:0/1645973342 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0cf80efac0 con 0x7f0cfc069dc0 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.170+0000 7f0d024c6640 1 -- 172.21.15.31:0/1235326109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a9d620 msgr2=0x55c471a9fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.170+0000 7f0d024c6640 1 --2- 172.21.15.31:0/1235326109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a9d620 0x55c471a9fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.170+0000 7f0d024c6640 1 -- 172.21.15.31:0/1235326109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92f50 msgr2=0x55c471a9d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.170+0000 7f0d024c6640 1 --2- 172.21.15.31:0/1235326109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92f50 0x55c471a9d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.170+0000 7f0d024c6640 1 -- 172.21.15.31:0/1235326109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4717d57d0 con 0x55c471a92630 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.170+0000 7f0d024c6640 1 --2- 172.21.15.31:0/1235326109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471a92a10 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4013de0 tx=0x7f0cf4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1bffa3a9ea3576 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.170+0000 7f0d024c6640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0ce80a2e20 0x7f0ce80a52e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4010c40 tx=0x7f0cf4015000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.170+0000 7f0ce7fff640 1 -- 172.21.15.31:0/1235326109 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4010e50 con 0x55c471a92630 2024-04-25T01:55:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.170+0000 7f0ce7fff640 1 -- 172.21.15.31:0/1235326109 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0cf4013740 con 0x55c471a92630 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.170+0000 7f0ce7fff640 1 -- 172.21.15.31:0/1235326109 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf40053a0 con 0x55c471a92630 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.171+0000 7f0d06c3a740 1 -- 172.21.15.31:0/1235326109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 msgr2=0x55c471a92a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.171+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/1235326109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471a92a10 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4013de0 tx=0x7f0cf4010740 comp rx=0 tx=0).stop 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.171+0000 7f0d06c3a740 1 -- 172.21.15.31:0/1235326109 shutdown_connections 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.171+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/1235326109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a9d620 0x55c471a9fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.171+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/1235326109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92f50 0x55c471a9d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.171+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/1235326109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471a92a10 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.171+0000 7f0d06c3a740 1 -- 172.21.15.31:0/1235326109 >> 172.21.15.31:0/1235326109 conn(0x55c471a8e950 msgr2=0x55c471a8ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.171+0000 7f0d06c3a740 1 -- 172.21.15.31:0/1235326109 shutdown_connections 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.171+0000 7f0d06c3a740 1 -- 172.21.15.31:0/1235326109 wait complete. 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.171+0000 7f0d06c3a740 1 -- start start 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.171+0000 7f0d06c3a740 1 Processor -- start 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92630 0x55c471a9a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a92f50 0x55c471a9ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a9d620 0x55c471a9b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c471aa03b0 con 0x55c471a9d620 2024-04-25T01:55:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c4719b2f50 con 0x55c471a92630 2024-04-25T01:55:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c471aa0530 con 0x55c471a92f50 2024-04-25T01:55:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d06c3a740 10 client.0 _ll_register_callbacks cb 0x55c4719ee000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d024c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a9d620 0x55c471a9b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d024c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a9d620 0x55c471a9b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57650/0 (socket says 172.21.15.31:57650) 2024-04-25T01:55:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d024c6640 1 -- 172.21.15.31:0/157302762 learned_addr learned my addr 172.21.15.31:0/157302762 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d014c4640 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a92f50 0x55c471a9ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92630 0x55c471a9a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d024c6640 1 -- 172.21.15.31:0/157302762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a92f50 msgr2=0x55c471a9ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d024c6640 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a92f50 0x55c471a9ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d024c6640 1 -- 172.21.15.31:0/157302762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92630 msgr2=0x55c471a9a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d024c6640 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92630 0x55c471a9a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.172+0000 7f0d024c6640 1 -- 172.21.15.31:0/157302762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c47170b3f0 con 0x55c471a9d620 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.173+0000 7f0d024c6640 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a9d620 0x55c471a9b0c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4013b20 tx=0x7f0cf400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531c53a985014758 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.173+0000 7f0ce57fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.173+0000 7f0ce57fa640 1 -- 172.21.15.31:0/157302762 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf402d020 con 0x55c471a9d620 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.173+0000 7f0ce57fa640 1 -- 172.21.15.31:0/157302762 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0cf4013260 con 0x55c471a9d620 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.173+0000 7f0d06c3a740 1 -- 172.21.15.31:0/157302762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf4013740 con 0x55c471a9d620 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.173+0000 7f0ce57fa640 1 -- 172.21.15.31:0/157302762 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cf4038050 con 0x55c471a9d620 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.173+0000 7f0d06c3a740 10 client.15642 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.173+0000 7f0d06c3a740 1 -- 172.21.15.31:0/157302762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c4716cb0f0 con 0x55c471a9d620 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.173+0000 7f0ce57fa640 1 -- 172.21.15.31:0/157302762 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0cf4013740 con 0x55c471a9d620 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.173+0000 7f0ce57fa640 1 -- 172.21.15.31:0/157302762 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0cf401c070 con 0x55c471a9d620 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.174+0000 7f0d06c3a740 10 client.15642 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.174+0000 7f0d06c3a740 10 client.15642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.174+0000 7f0d06c3a740 20 client.15642 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.174+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c471aad920 0x55c471aafd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.174+0000 7f0d06c3a740 4 client.15642 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.174+0000 7f0d06c3a740 1 -- 172.21.15.31:0/157302762 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c471a04bf0 con 0x55c471aad920 2024-04-25T01:55:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.174+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c471aad920 0x55c471aafd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.174+0000 7f0ce57fa640 10 client.15642 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.174+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c471aad920 0x55c471aafd10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22a5f56df090a0fd server_cookie=501d9c69cf9acd6b in_seq=0 out_seq=0 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0ce57fa640 1 -- 172.21.15.31:0/157302762 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c471a04bf0 con 0x55c471aad920 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0ce57fa640 10 client.15642 handle_command_reply: tid=0 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 1 client.15642 shutdown 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 1 -- 172.21.15.31:0/157302762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a9d620 msgr2=0x55c471a9b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a9d620 0x55c471a9b0c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4013b20 tx=0x7f0cf400be50 comp rx=0 tx=0).stop 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 1 -- 172.21.15.31:0/157302762 shutdown_connections 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c471aad920 0x55c471aafd10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a9d620 0x55c471a9b0c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a92f50 0x55c471a9ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/157302762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92630 0x55c471a9a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0ce57fa640 0 client.15642 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:06.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 1 -- 172.21.15.31:0/157302762 >> 172.21.15.31:0/157302762 conn(0x55c471a8e950 msgr2=0x55c471a94750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 1 -- 172.21.15.31:0/157302762 shutdown_connections 2024-04-25T01:55:06.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 1 -- 172.21.15.31:0/157302762 wait complete. 2024-04-25T01:55:06.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.175+0000 7f0d06c3a740 20 client.15642 trim_cache size 0 max 16384 2024-04-25T01:55:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.193+0000 7f0d06c3a740 1 Processor -- start 2024-04-25T01:55:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.193+0000 7f0d06c3a740 1 -- start start 2024-04-25T01:55:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.193+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.193+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92f50 0x55c471ab5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.193+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a9d620 0x55c471ab5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.193+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc1af440 con 0x55c471a92630 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.193+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc1b0f80 con 0x55c471a92f50 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.193+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc1afb10 con 0x55c471a9d620 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.193+0000 7f0d01cc5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.193+0000 7f0d01cc5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57662/0 (socket says 172.21.15.31:57662) 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.193+0000 7f0d01cc5640 1 -- 172.21.15.31:0/778958324 learned_addr learned my addr 172.21.15.31:0/778958324 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0d01cc5640 1 -- 172.21.15.31:0/778958324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a9d620 msgr2=0x55c471ab5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/778958324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a9d620 0x55c471ab5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0d01cc5640 1 -- 172.21.15.31:0/778958324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92f50 msgr2=0x55c471ab5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/778958324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92f50 0x55c471ab5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0d01cc5640 1 -- 172.21.15.31:0/778958324 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c471a9fa10 con 0x55c471a92630 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/778958324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa9270 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8005940 tx=0x7f0cf808eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d50000509451be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0ce77fe640 1 -- 172.21.15.31:0/778958324 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8088070 con 0x55c471a92630 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0ce77fe640 1 -- 172.21.15.31:0/778958324 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0cf80be050 con 0x55c471a92630 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0ce77fe640 1 -- 172.21.15.31:0/778958324 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8048040 con 0x55c471a92630 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0d06c3a740 1 -- 172.21.15.31:0/778958324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 msgr2=0x55c471aa9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/778958324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa9270 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8005940 tx=0x7f0cf808eec0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0d06c3a740 1 -- 172.21.15.31:0/778958324 shutdown_connections 2024-04-25T01:55:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.194+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/778958324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a9d620 0x55c471ab5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/778958324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a92f50 0x55c471ab5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/778958324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa9270 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 -- 172.21.15.31:0/778958324 >> 172.21.15.31:0/778958324 conn(0x55c471a8e950 msgr2=0x55c471a99670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 -- 172.21.15.31:0/778958324 shutdown_connections 2024-04-25T01:55:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 -- 172.21.15.31:0/778958324 wait complete. 2024-04-25T01:55:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 -- start start 2024-04-25T01:55:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 Processor -- start 2024-04-25T01:55:06.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a92f50 0x55c471aa5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a9d620 0x55c471a96de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc1b08b0 con 0x55c471a92630 2024-04-25T01:55:06.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc1b01e0 con 0x55c471a9d620 2024-04-25T01:55:06.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc1ad420 con 0x55c471a92f50 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d06c3a740 10 client.0 _ll_register_callbacks cb 0x7f0cf80bbca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d01cc5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d01cc5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57668/0 (socket says 172.21.15.31:57668) 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.195+0000 7f0d01cc5640 1 -- 172.21.15.31:0/3874164812 learned_addr learned my addr 172.21.15.31:0/3874164812 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0d024c6640 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a9d620 0x55c471a96de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0d014c4640 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a92f50 0x55c471aa5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0d01cc5640 1 -- 172.21.15.31:0/3874164812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a92f50 msgr2=0x55c471aa5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a92f50 0x55c471aa5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0d01cc5640 1 -- 172.21.15.31:0/3874164812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a9d620 msgr2=0x55c471a96de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a9d620 0x55c471a96de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0d01cc5640 1 -- 172.21.15.31:0/3874164812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c471a97610 con 0x55c471a92630 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0d01cc5640 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa5780 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80c4590 tx=0x7f0cf8009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b18ba5a5c8487e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0ce57fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:06.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0ce57fa640 1 -- 172.21.15.31:0/3874164812 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8048070 con 0x55c471a92630 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0ce57fa640 1 -- 172.21.15.31:0/3874164812 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0cf80c4630 con 0x55c471a92630 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0d06c3a740 1 -- 172.21.15.31:0/3874164812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf80be050 con 0x55c471a92630 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0ce57fa640 1 -- 172.21.15.31:0/3874164812 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cf8088040 con 0x55c471a92630 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0d06c3a740 10 client.15648 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0d06c3a740 1 -- 172.21.15.31:0/3874164812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c471b13880 con 0x55c471a92630 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.196+0000 7f0ce57fa640 1 -- 172.21.15.31:0/3874164812 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0cf80be050 con 0x55c471a92630 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.197+0000 7f0ce57fa640 1 -- 172.21.15.31:0/3874164812 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0cf804b060 con 0x55c471a92630 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.197+0000 7f0d06c3a740 10 client.15648 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.197+0000 7f0d06c3a740 10 client.15648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.197+0000 7f0d06c3a740 20 client.15648 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.197+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c471b13f70 0x55c471b16390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.197+0000 7f0d06c3a740 4 client.15648 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.197+0000 7f0d06c3a740 1 -- 172.21.15.31:0/3874164812 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c4719fef40 con 0x55c471b13f70 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.197+0000 7f0d014c4640 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c471b13f70 0x55c471b16390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.197+0000 7f0ce57fa640 10 client.15648 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:06.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.197+0000 7f0d014c4640 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c471b13f70 0x55c471b16390 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52c73d97621dafcd server_cookie=dbef535c0ad99eae in_seq=0 out_seq=0 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0ce57fa640 1 -- 172.21.15.31:0/3874164812 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c4719fef40 con 0x55c471b13f70 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0ce57fa640 10 client.15648 handle_command_reply: tid=0 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 1 client.15648 shutdown 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 1 -- 172.21.15.31:0/3874164812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 msgr2=0x55c471aa5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa5780 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80c4590 tx=0x7f0cf8009680 comp rx=0 tx=0).stop 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 1 -- 172.21.15.31:0/3874164812 shutdown_connections 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c471b13f70 0x55c471b16390 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c471a9d620 0x55c471a96de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c471a92f50 0x55c471aa5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 1 --2- 172.21.15.31:0/3874164812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c471a92630 0x55c471aa5780 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0ce57fa640 0 client.15648 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 1 -- 172.21.15.31:0/3874164812 >> 172.21.15.31:0/3874164812 conn(0x55c471a8e950 msgr2=0x55c471a9a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 1 -- 172.21.15.31:0/3874164812 shutdown_connections 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 1 -- 172.21.15.31:0/3874164812 wait complete. 2024-04-25T01:55:06.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d06c3a740 20 client.15648 trim_cache size 0 max 16384 2024-04-25T01:55:06.205 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:55:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T01:55:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T01:55:06.205 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:55:06.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d03f29640 1 -- 172.21.15.31:0/1645973342 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0ce80a2e20 msgr2=0x7f0ce80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d03f29640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0ce80a2e20 0x7f0ce80a52e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4010c40 tx=0x7f0cf4015000 comp rx=0 tx=0).stop 2024-04-25T01:55:06.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d03f29640 1 -- 172.21.15.31:0/1645973342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cfc069dc0 msgr2=0x7f0cfc100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d03f29640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cfc069dc0 0x7f0cfc100670 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8004770 tx=0x7f0cf80042f0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d03f29640 1 -- 172.21.15.31:0/1645973342 shutdown_connections 2024-04-25T01:55:06.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d03f29640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0ce80a2e20 0x7f0ce80a52e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d03f29640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cfc107730 0x7f0cfc1010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d03f29640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cfc06a780 0x7f0cfc100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.198+0000 7f0d03f29640 1 --2- 172.21.15.31:0/1645973342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cfc069dc0 0x7f0cfc100670 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.199+0000 7f0d03f29640 1 -- 172.21.15.31:0/1645973342 >> 172.21.15.31:0/1645973342 conn(0x7f0cfc0776a0 msgr2=0x7f0cfc0fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.199+0000 7f0d03f29640 1 -- 172.21.15.31:0/1645973342 shutdown_connections 2024-04-25T01:55:06.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.199+0000 7f0d03f29640 1 -- 172.21.15.31:0/1645973342 wait complete. 2024-04-25T01:55:06.211 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:55:06.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:55:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.312+0000 7f9720fb6640 1 -- 172.21.15.31:0/3075898852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f971c10bd60 msgr2=0x7f971c10e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.312+0000 7f9720fb6640 1 --2- 172.21.15.31:0/3075898852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f971c10bd60 0x7f971c10e1d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9714009ca0 tx=0x7f971402a970 comp rx=0 tx=0).stop 2024-04-25T01:55:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.312+0000 7f9720fb6640 1 -- 172.21.15.31:0/3075898852 shutdown_connections 2024-04-25T01:55:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.312+0000 7f9720fb6640 1 --2- 172.21.15.31:0/3075898852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f971c10bd60 0x7f971c10e1d0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.312+0000 7f9720fb6640 1 --2- 172.21.15.31:0/3075898852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f971c103340 0x7f971c10b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.312+0000 7f9720fb6640 1 --2- 172.21.15.31:0/3075898852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f971c102a20 0x7f971c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.312+0000 7f9720fb6640 1 -- 172.21.15.31:0/3075898852 >> 172.21.15.31:0/3075898852 conn(0x7f971c0777a0 msgr2=0x7f971c077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.312+0000 7f9720fb6640 1 -- 172.21.15.31:0/3075898852 shutdown_connections 2024-04-25T01:55:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.312+0000 7f9720fb6640 1 -- 172.21.15.31:0/3075898852 wait complete. 2024-04-25T01:55:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f9720fb6640 1 Processor -- start 2024-04-25T01:55:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f9720fb6640 1 -- start start 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f9720fb6640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f971c102a20 0x7f971c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f9720fb6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f971c103340 0x7f971c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f9720fb6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f971c10bd60 0x7f971c19c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f9720fb6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f971c111a90 con 0x7f971c10bd60 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f9720fb6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f971c111910 con 0x7f971c103340 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f9720fb6640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f971c111c10 con 0x7f971c102a20 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f971ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f971c10bd60 0x7f971c19c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f971ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f971c10bd60 0x7f971c19c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57682/0 (socket says 172.21.15.31:57682) 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f971ad9d640 1 -- 172.21.15.31:0/3850726545 learned_addr learned my addr 172.21.15.31:0/3850726545 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f9719d9b640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f971c103340 0x7f971c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f971a59c640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f971c102a20 0x7f971c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f971ad9d640 1 -- 172.21.15.31:0/3850726545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f971c102a20 msgr2=0x7f971c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f971ad9d640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f971c102a20 0x7f971c197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f971ad9d640 1 -- 172.21.15.31:0/3850726545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f971c103340 msgr2=0x7f971c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f971ad9d640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f971c103340 0x7f971c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f971ad9d640 1 -- 172.21.15.31:0/3850726545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971c19ced0 con 0x7f971c10bd60 2024-04-25T01:55:06.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f9719d9b640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f971c103340 0x7f971c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:06.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.313+0000 7f971ad9d640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f971c10bd60 0x7f971c19c7a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f97140098d0 tx=0x7f9714003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.314+0000 7f970b7fe640 1 -- 172.21.15.31:0/3850726545 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714034030 con 0x7f971c10bd60 2024-04-25T01:55:06.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.314+0000 7f970b7fe640 1 -- 172.21.15.31:0/3850726545 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f971402f870 con 0x7f971c10bd60 2024-04-25T01:55:06.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.314+0000 7f970b7fe640 1 -- 172.21.15.31:0/3850726545 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714039680 con 0x7f971c10bd60 2024-04-25T01:55:06.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.314+0000 7f9720fb6640 1 -- 172.21.15.31:0/3850726545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f971c19d160 con 0x7f971c10bd60 2024-04-25T01:55:06.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.314+0000 7f9720fb6640 1 -- 172.21.15.31:0/3850726545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f971c1a4a50 con 0x7f971c10bd60 2024-04-25T01:55:06.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.315+0000 7f9720fb6640 1 -- 172.21.15.31:0/3850726545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f971c0689b0 con 0x7f971c10bd60 2024-04-25T01:55:06.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.316+0000 7f970b7fe640 1 -- 172.21.15.31:0/3850726545 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f9714037020 con 0x7f971c10bd60 2024-04-25T01:55:06.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.317+0000 7f970b7fe640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f97040a2e20 0x7f97040a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.317+0000 7f971a59c640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f97040a2e20 0x7f97040a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.317+0000 7f970b7fe640 1 -- 172.21.15.31:0/3850726545 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f97140f0880 con 0x7f971c10bd60 2024-04-25T01:55:06.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.317+0000 7f971a59c640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f97040a2e20 0x7f97040a52e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f970c00ba40 tx=0x7f970c005cc0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.321+0000 7f970b7fe640 1 -- 172.21.15.31:0/3850726545 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f97140bd240 con 0x7f971c10bd60 2024-04-25T01:55:06.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.545+0000 7f9720fb6640 1 -- 172.21.15.31:0/3850726545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f971c1a4c20 con 0x7f971c10bd60 2024-04-25T01:55:06.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.545+0000 7f970b7fe640 1 -- 172.21.15.31:0/3850726545 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11923 (secure 0 0 0) 0x7f97140bcb40 con 0x7f971c10bd60 2024-04-25T01:55:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:55:06.550 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:54:28.592746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24451,24463,24503,24487,24473]},"id":1}]} 2024-04-25T01:55:06.551 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-04-25T01:55:06.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 -- 172.21.15.31:0/3850726545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f97040a2e20 msgr2=0x7f97040a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f97040a2e20 0x7f97040a52e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f970c00ba40 tx=0x7f970c005cc0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 -- 172.21.15.31:0/3850726545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f971c10bd60 msgr2=0x7f971c19c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f971c10bd60 0x7f971c19c7a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f97140098d0 tx=0x7f9714003f80 comp rx=0 tx=0).stop 2024-04-25T01:55:06.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 -- 172.21.15.31:0/3850726545 shutdown_connections 2024-04-25T01:55:06.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f97040a2e20 0x7f97040a52e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f971c10bd60 0x7f971c19c7a0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f971c103340 0x7f971c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 --2- 172.21.15.31:0/3850726545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f971c102a20 0x7f971c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 -- 172.21.15.31:0/3850726545 >> 172.21.15.31:0/3850726545 conn(0x7f971c0777a0 msgr2=0x7f971c0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 -- 172.21.15.31:0/3850726545 shutdown_connections 2024-04-25T01:55:06.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.547+0000 7f9720fb6640 1 -- 172.21.15.31:0/3850726545 wait complete. 2024-04-25T01:55:06.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:55:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.663+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/2129121591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 msgr2=0x7f0b48106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.663+0000 7f0b4cc1b640 1 --2- 172.21.15.31:0/2129121591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b48106a10 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009930 tx=0x7f0b3802a420 comp rx=0 tx=0).stop 2024-04-25T01:55:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.663+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/2129121591 shutdown_connections 2024-04-25T01:55:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.663+0000 7f0b4cc1b640 1 --2- 172.21.15.31:0/2129121591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100fe0 0x7f0b4810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.663+0000 7f0b4cc1b640 1 --2- 172.21.15.31:0/2129121591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100620 0x7f0b48100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.663+0000 7f0b4cc1b640 1 --2- 172.21.15.31:0/2129121591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b48106a10 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.663+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/2129121591 >> 172.21.15.31:0/2129121591 conn(0x7f0b480776a0 msgr2=0x7f0b48077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.663+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/2129121591 shutdown_connections 2024-04-25T01:55:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.663+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/2129121591 wait complete. 2024-04-25T01:55:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b4cc1b640 1 Processor -- start 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b4cc1b640 1 -- start start 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b4cc1b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100620 0x7f0b48197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b4cc1b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100fe0 0x7f0b48198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b4cc1b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b4cc1b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0b481108f0 con 0x7f0b48106630 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b4cc1b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b48110770 con 0x7f0b48100620 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b4cc1b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0b48110a70 con 0x7f0b48100fe0 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b46d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b46d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57698/0 (socket says 172.21.15.31:57698) 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b46d9d640 1 -- 172.21.15.31:0/437879512 learned_addr learned my addr 172.21.15.31:0/437879512 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100fe0 0x7f0b48198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b4659c640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100620 0x7f0b48197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b46d9d640 1 -- 172.21.15.31:0/437879512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100fe0 msgr2=0x7f0b48198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b46d9d640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100fe0 0x7f0b48198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b46d9d640 1 -- 172.21.15.31:0/437879512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100620 msgr2=0x7f0b48197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b46d9d640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100620 0x7f0b48197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b46d9d640 1 -- 172.21.15.31:0/437879512 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4819cf30 con 0x7f0b48106630 2024-04-25T01:55:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.664+0000 7f0b4659c640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100620 0x7f0b48197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.665+0000 7f0b46d9d640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0b4000eaa0 tx=0x7f0b4000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.665+0000 7f0b3f7fe640 1 -- 172.21.15.31:0/437879512 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4000cdb0 con 0x7f0b48106630 2024-04-25T01:55:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.665+0000 7f0b3f7fe640 1 -- 172.21.15.31:0/437879512 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b40005020 con 0x7f0b48106630 2024-04-25T01:55:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.665+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/437879512 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4819d220 con 0x7f0b48106630 2024-04-25T01:55:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.665+0000 7f0b3f7fe640 1 -- 172.21.15.31:0/437879512 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b400116d0 con 0x7f0b48106630 2024-04-25T01:55:06.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.665+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/437879512 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b481a4ab0 con 0x7f0b48106630 2024-04-25T01:55:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.667+0000 7f0b3f7fe640 1 -- 172.21.15.31:0/437879512 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f0b400051d0 con 0x7f0b48106630 2024-04-25T01:55:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.667+0000 7f0b4f9fd740 1 Processor -- start 2024-04-25T01:55:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b4f9fd740 1 -- start start 2024-04-25T01:55:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644bdb580 0x55b644bdb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdbea0 0x55b644be6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644be8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b644b40670 con 0x55b644be6570 2024-04-25T01:55:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b644ae8970 con 0x55b644bdb580 2024-04-25T01:55:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b644b3c630 con 0x55b644bdbea0 2024-04-25T01:55:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b3f7fe640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e20 0x7f0b340a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b3ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644be8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b3ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644be8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57710/0 (socket says 172.21.15.31:57710) 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b3ffff640 1 -- 172.21.15.31:0/789760564 learned_addr learned my addr 172.21.15.31:0/789760564 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b4659c640 1 --2- 172.21.15.31:0/789760564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdbea0 0x55b644be6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b46d9d640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e20 0x7f0b340a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/789760564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644bdb580 0x55b644bdb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b3f7fe640 1 -- 172.21.15.31:0/437879512 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0b400cf9d0 con 0x7f0b48106630 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b3ffff640 1 -- 172.21.15.31:0/789760564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdbea0 msgr2=0x55b644be6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/789760564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdbea0 0x55b644be6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b3ffff640 1 -- 172.21.15.31:0/789760564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644bdb580 msgr2=0x55b644bdb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/789760564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644bdb580 0x55b644bdb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.668+0000 7f0b3ffff640 1 -- 172.21.15.31:0/789760564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b644b0c400 con 0x55b644be6570 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.669+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/789760564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644be8960 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000ae00 tx=0x7f0b3000a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82c0f8108aeec84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.669+0000 7f0b46d9d640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e20 0x7f0b340a52e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0b400ccff0 tx=0x7f0b4006ed00 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.669+0000 7f0b3cff9640 1 -- 172.21.15.31:0/789760564 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3001b070 con 0x55b644be6570 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.669+0000 7f0b3cff9640 1 -- 172.21.15.31:0/789760564 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b3000fd70 con 0x55b644be6570 2024-04-25T01:55:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.669+0000 7f0b3cff9640 1 -- 172.21.15.31:0/789760564 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30014650 con 0x55b644be6570 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/789760564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 msgr2=0x55b644be8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/789760564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644be8960 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000ae00 tx=0x7f0b3000a490 comp rx=0 tx=0).stop 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/789760564 shutdown_connections 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/789760564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644be8960 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/789760564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdbea0 0x55b644be6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/789760564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644bdb580 0x55b644bdb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/789760564 >> 172.21.15.31:0/789760564 conn(0x55b644bd78a0 msgr2=0x55b644bd7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/789760564 shutdown_connections 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/789760564 wait complete. 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 -- start start 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 Processor -- start 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdb580 0x55b644be5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdbea0 0x55b644be31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644be6570 0x55b644be3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b644be9360 con 0x55b644bdbea0 2024-04-25T01:55:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b644afc050 con 0x55b644be6570 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b644be94e0 con 0x55b644bdb580 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4f9fd740 10 client.0 _ll_register_callbacks cb 0x55b644853be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.670+0000 7f0b4659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdbea0 0x55b644be31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b4659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdbea0 0x55b644be31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57724/0 (socket says 172.21.15.31:57724) 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b4659c640 1 -- 172.21.15.31:0/505776375 learned_addr learned my addr 172.21.15.31:0/505776375 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644be6570 0x55b644be3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdb580 0x55b644be5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b4659c640 1 -- 172.21.15.31:0/505776375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdb580 msgr2=0x55b644be5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b4659c640 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdb580 0x55b644be5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b4659c640 1 -- 172.21.15.31:0/505776375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644be6570 msgr2=0x55b644be3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b4659c640 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644be6570 0x55b644be3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b4659c640 1 -- 172.21.15.31:0/505776375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b644acddc0 con 0x55b644bdbea0 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b4659c640 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdbea0 0x55b644be31f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009930 tx=0x7f0b380316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b76b112ececa8c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b2dffb640 1 -- 172.21.15.31:0/505776375 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b38040070 con 0x55b644bdbea0 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b2dffb640 1 -- 172.21.15.31:0/505776375 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0b38004450 con 0x55b644bdbea0 2024-04-25T01:55:06.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/505776375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b3000fd70 con 0x55b644bdbea0 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b2dffb640 1 -- 172.21.15.31:0/505776375 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0b38035040 con 0x55b644bdbea0 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b4f9fd740 10 client.15666 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/505776375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b6448543f0 con 0x55b644bdbea0 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.671+0000 7f0b2dffb640 1 -- 172.21.15.31:0/505776375 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b3000fd70 con 0x55b644bdbea0 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.672+0000 7f0b2dffb640 1 -- 172.21.15.31:0/505776375 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0b380049a0 con 0x55b644bdbea0 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.672+0000 7f0b4f9fd740 10 client.15666 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.672+0000 7f0b4f9fd740 10 client.15666 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.672+0000 7f0b4f9fd740 20 client.15666 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.672+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55b644bf6530 0x55b644bf8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.672+0000 7f0b4f9fd740 4 client.15666 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.672+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/505776375 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b644b4dbb0 con 0x55b644bf6530 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.672+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55b644bf6530 0x55b644bf8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.672+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55b644bf6530 0x55b644bf8920 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=41e18714f351d399 server_cookie=421f22098b6bfdab in_seq=0 out_seq=0 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.672+0000 7f0b2dffb640 10 client.15666 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b2dffb640 1 -- 172.21.15.31:0/505776375 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b644b4dbb0 con 0x55b644bf6530 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b2dffb640 10 client.15666 handle_command_reply: tid=0 2024-04-25T01:55:06.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b4f9fd740 1 client.15666 shutdown 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/505776375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdbea0 msgr2=0x55b644be31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdbea0 0x55b644be31f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009930 tx=0x7f0b380316d0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/505776375 shutdown_connections 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55b644bf6530 0x55b644bf8920 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644be6570 0x55b644be3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdbea0 0x55b644be31f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b2dffb640 0 client.15666 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/505776375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdb580 0x55b644be5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.673+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/505776375 >> 172.21.15.31:0/505776375 conn(0x55b644bd78a0 msgr2=0x55b644bdd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.674+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/505776375 shutdown_connections 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.674+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/505776375 wait complete. 2024-04-25T01:55:06.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.674+0000 7f0b4f9fd740 20 client.15666 trim_cache size 0 max 16384 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.687+0000 7f0b4f9fd740 1 Processor -- start 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.687+0000 7f0b4f9fd740 1 -- start start 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdb580 0x55b644bf2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644bdbea0 0x55b644be5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644bfe1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0b481ad320 con 0x55b644be6570 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b481aee60 con 0x55b644bdbea0 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0b481ad9f0 con 0x55b644bdb580 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b3ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644bfe1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b3ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644bfe1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57732/0 (socket says 172.21.15.31:57732) 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b3ffff640 1 -- 172.21.15.31:0/2747276498 learned_addr learned my addr 172.21.15.31:0/2747276498 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/2747276498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdb580 0x55b644bf2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b4659c640 1 --2- 172.21.15.31:0/2747276498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644bdbea0 0x55b644be5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b3ffff640 1 -- 172.21.15.31:0/2747276498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdb580 msgr2=0x55b644bf2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/2747276498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdb580 0x55b644bf2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b3ffff640 1 -- 172.21.15.31:0/2747276498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644bdbea0 msgr2=0x55b644be5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/2747276498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644bdbea0 0x55b644be5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b3ffff640 1 -- 172.21.15.31:0/2747276498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b644bdbcd0 con 0x55b644be6570 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/2747276498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644bfe1d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0b30006aa0 tx=0x7f0b30016db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d0885d8c1095d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b2ffff640 1 -- 172.21.15.31:0/2747276498 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3001b070 con 0x55b644be6570 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b2ffff640 1 -- 172.21.15.31:0/2747276498 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b3001c040 con 0x55b644be6570 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.688+0000 7f0b2ffff640 1 -- 172.21.15.31:0/2747276498 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30029700 con 0x55b644be6570 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.689+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2747276498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 msgr2=0x55b644bfe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.689+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/2747276498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644bfe1d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0b30006aa0 tx=0x7f0b30016db0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.689+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2747276498 shutdown_connections 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.689+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/2747276498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644be6570 0x55b644bfe1d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.689+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/2747276498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644bdbea0 0x55b644be5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.689+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/2747276498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdb580 0x55b644bf2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.689+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2747276498 >> 172.21.15.31:0/2747276498 conn(0x55b644bd78a0 msgr2=0x55b644bde860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.689+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2747276498 shutdown_connections 2024-04-25T01:55:06.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.689+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2747276498 wait complete. 2024-04-25T01:55:06.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.689+0000 7f0b4f9fd740 1 -- start start 2024-04-25T01:55:06.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b4f9fd740 1 Processor -- start 2024-04-25T01:55:06.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdb580 0x55b644bee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdbea0 0x55b644be9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b4f9fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644be6570 0x55b644bea290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0b481ae790 con 0x55b644bdb580 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b481ae0c0 con 0x55b644be6570 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b4f9fd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0b481ab300 con 0x55b644bdbea0 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b4f9fd740 10 client.0 _ll_register_callbacks cb 0x7f0b38032d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b3ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdb580 0x55b644bee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b3ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdb580 0x55b644bee9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57746/0 (socket says 172.21.15.31:57746) 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b3ffff640 1 -- 172.21.15.31:0/2353755316 learned_addr learned my addr 172.21.15.31:0/2353755316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b4659c640 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdbea0 0x55b644be9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b3ffff640 1 -- 172.21.15.31:0/2353755316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdbea0 msgr2=0x55b644be9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdbea0 0x55b644be9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b3ffff640 1 -- 172.21.15.31:0/2353755316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644be6570 msgr2=0x55b644bea290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644be6570 0x55b644bea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b3ffff640 1 -- 172.21.15.31:0/2353755316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b64491e7d0 con 0x55b644bdb580 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b4659c640 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdbea0 0x55b644be9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:06.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b3ffff640 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdb580 0x55b644bee9e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000a2c0 tx=0x7f0b3000a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963c5ab3cfdeb9ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.690+0000 7f0b2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b2dffb640 1 -- 172.21.15.31:0/2353755316 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3001b050 con 0x55b644bdb580 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b2dffb640 1 -- 172.21.15.31:0/2353755316 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0b300059c0 con 0x55b644bdb580 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2353755316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b3001c040 con 0x55b644bdb580 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b2dffb640 1 -- 172.21.15.31:0/2353755316 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0b30045050 con 0x55b644bdb580 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b4f9fd740 10 client.15672 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2353755316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b644c5c770 con 0x55b644bdb580 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b2dffb640 1 -- 172.21.15.31:0/2353755316 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b3001c040 con 0x55b644bdb580 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b2dffb640 1 -- 172.21.15.31:0/2353755316 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0b30015040 con 0x55b644bdb580 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b4f9fd740 10 client.15672 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b4f9fd740 10 client.15672 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b4f9fd740 20 client.15672 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55b644c5cbf0 0x55b644c5efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b4f9fd740 4 client.15672 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2353755316 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b644b47f00 con 0x55b644c5cbf0 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.691+0000 7f0b4659c640 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55b644c5cbf0 0x55b644c5efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4659c640 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55b644c5cbf0 0x55b644c5efe0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ab02858ab9133623 server_cookie=f72511476893eec in_seq=0 out_seq=0 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b2dffb640 10 client.15672 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b2dffb640 1 -- 172.21.15.31:0/2353755316 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b644b47f00 con 0x55b644c5cbf0 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b2dffb640 10 client.15672 handle_command_reply: tid=0 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 1 client.15672 shutdown 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2353755316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdb580 msgr2=0x55b644bee9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdb580 0x55b644bee9e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000a2c0 tx=0x7f0b3000a410 comp rx=0 tx=0).stop 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2353755316 shutdown_connections 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x55b644c5cbf0 0x55b644c5efe0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b644be6570 0x55b644bea290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b644bdbea0 0x55b644be9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b2dffb640 0 client.15672 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 1 --2- 172.21.15.31:0/2353755316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b644bdb580 0x55b644bee9e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2353755316 >> 172.21.15.31:0/2353755316 conn(0x55b644bd78a0 msgr2=0x55b644bf1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2353755316 shutdown_connections 2024-04-25T01:55:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 1 -- 172.21.15.31:0/2353755316 wait complete. 2024-04-25T01:55:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.692+0000 7f0b4f9fd740 20 client.15672 trim_cache size 0 max 16384 2024-04-25T01:55:06.700 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:55:06.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/437879512 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e20 msgr2=0x7f0b340a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e20 0x7f0b340a52e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0b400ccff0 tx=0x7f0b4006ed00 comp rx=0 tx=0).stop 2024-04-25T01:55:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/437879512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 msgr2=0x7f0b4819c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0b4000eaa0 tx=0x7f0b4000ef60 comp rx=0 tx=0).stop 2024-04-25T01:55:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/437879512 shutdown_connections 2024-04-25T01:55:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0b340a2e20 0x7f0b340a52e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b48106630 0x7f0b4819c800 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b48100fe0 0x7f0b48198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 --2- 172.21.15.31:0/437879512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b48100620 0x7f0b48197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/437879512 >> 172.21.15.31:0/437879512 conn(0x7f0b480776a0 msgr2=0x7f0b480ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/437879512 shutdown_connections 2024-04-25T01:55:06.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.693+0000 7f0b4cc1b640 1 -- 172.21.15.31:0/437879512 wait complete. 2024-04-25T01:55:06.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:55:06.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.808+0000 7f3d77d63640 1 -- 172.21.15.31:0/3736084423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70106840 msgr2=0x7f3d70106c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.808+0000 7f3d77d63640 1 --2- 172.21.15.31:0/3736084423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70106840 0x7f3d70106c20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009930 tx=0x7f3d6c02a420 comp rx=0 tx=0).stop 2024-04-25T01:55:06.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.808+0000 7f3d77d63640 1 -- 172.21.15.31:0/3736084423 shutdown_connections 2024-04-25T01:55:06.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.808+0000 7f3d77d63640 1 --2- 172.21.15.31:0/3736084423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d70101710 0x7f3d7010d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.808+0000 7f3d77d63640 1 --2- 172.21.15.31:0/3736084423 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d70100d50 0x7f3d701011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.808+0000 7f3d77d63640 1 --2- 172.21.15.31:0/3736084423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70106840 0x7f3d70106c20 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.808+0000 7f3d77d63640 1 -- 172.21.15.31:0/3736084423 >> 172.21.15.31:0/3736084423 conn(0x7f3d70076c00 msgr2=0x7f3d700fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.808+0000 7f3d77d63640 1 -- 172.21.15.31:0/3736084423 shutdown_connections 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.808+0000 7f3d77d63640 1 -- 172.21.15.31:0/3736084423 wait complete. 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d77d63640 1 Processor -- start 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d77d63640 1 -- start start 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d77d63640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d70100d50 0x7f3d70198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d77d63640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d70101710 0x7f3d70198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d77d63640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70106840 0x7f3d7019caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d77d63640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3d70110b20 con 0x7f3d70106840 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d77d63640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701109a0 con 0x7f3d70100d50 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d77d63640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3d70110ca0 con 0x7f3d70101710 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d76300640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70106840 0x7f3d7019caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d76300640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70106840 0x7f3d7019caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57760/0 (socket says 172.21.15.31:57760) 2024-04-25T01:55:06.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d76300640 1 -- 172.21.15.31:0/3982894920 learned_addr learned my addr 172.21.15.31:0/3982894920 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d752fe640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d70101710 0x7f3d70198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.809+0000 7f3d75aff640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d70100d50 0x7f3d70198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.810+0000 7f3d76300640 1 -- 172.21.15.31:0/3982894920 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d70101710 msgr2=0x7f3d70198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.810+0000 7f3d76300640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d70101710 0x7f3d70198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.810+0000 7f3d76300640 1 -- 172.21.15.31:0/3982894920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d70100d50 msgr2=0x7f3d70198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.810+0000 7f3d76300640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d70100d50 0x7f3d70198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.810+0000 7f3d76300640 1 -- 172.21.15.31:0/3982894920 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7019d220 con 0x7f3d70106840 2024-04-25T01:55:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.810+0000 7f3d76300640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70106840 0x7f3d7019caf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800b970 tx=0x7f3d6800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.810+0000 7f3d66ffd640 1 -- 172.21.15.31:0/3982894920 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d68019070 con 0x7f3d70106840 2024-04-25T01:55:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.810+0000 7f3d66ffd640 1 -- 172.21.15.31:0/3982894920 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d68014440 con 0x7f3d70106840 2024-04-25T01:55:06.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.810+0000 7f3d77d63640 1 -- 172.21.15.31:0/3982894920 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d7019d510 con 0x7f3d70106840 2024-04-25T01:55:06.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.810+0000 7f3d66ffd640 1 -- 172.21.15.31:0/3982894920 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d680075d0 con 0x7f3d70106840 2024-04-25T01:55:06.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.810+0000 7f3d77d63640 1 -- 172.21.15.31:0/3982894920 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d701a4da0 con 0x7f3d70106840 2024-04-25T01:55:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.812+0000 7f3d66ffd640 1 -- 172.21.15.31:0/3982894920 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f3d6801c070 con 0x7f3d70106840 2024-04-25T01:55:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d7aa75740 1 Processor -- start 2024-04-25T01:55:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d7aa75740 1 -- start start 2024-04-25T01:55:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d66ffd640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3d5c0a2e20 0x7f3d5c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 0x560561dae690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561daec60 0x560561dbd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dbd980 0x560561dbfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560561d13670 con 0x560561daec60 2024-04-25T01:55:06.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560561cbb970 con 0x560561dbd980 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560561d0f630 con 0x560561dae2b0 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d66ffd640 1 -- 172.21.15.31:0/3982894920 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f3d680cebe0 con 0x7f3d70106840 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d752fe640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3d5c0a2e20 0x7f3d5c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d752fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561daec60 0x560561dbd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d752fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561daec60 0x560561dbd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57776/0 (socket says 172.21.15.31:57776) 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d752fe640 1 -- 172.21.15.31:0/2748642866 learned_addr learned my addr 172.21.15.31:0/2748642866 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d75aff640 1 --2- 172.21.15.31:0/2748642866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 0x560561dae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.813+0000 7f3d76300640 1 --2- 172.21.15.31:0/2748642866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dbd980 0x560561dbfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d752fe640 1 -- 172.21.15.31:0/2748642866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 msgr2=0x560561dae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d752fe640 1 --2- 172.21.15.31:0/2748642866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 0x560561dae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d752fe640 1 -- 172.21.15.31:0/2748642866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dbd980 msgr2=0x560561dbfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d752fe640 1 --2- 172.21.15.31:0/2748642866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dbd980 0x560561dbfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d752fe640 1 -- 172.21.15.31:0/2748642866 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560561cdf400 con 0x560561daec60 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d75aff640 1 --2- 172.21.15.31:0/2748642866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 0x560561dae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d76300640 1 --2- 172.21.15.31:0/2748642866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dbd980 0x560561dbfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d752fe640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3d5c0a2e20 0x7f3d5c0a52e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3d60005d00 tx=0x7f3d6000bb90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d752fe640 1 --2- 172.21.15.31:0/2748642866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561daec60 0x560561dbd440 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3d600123f0 tx=0x7f3d6001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65363ff9846cdaaf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d5bfff640 1 -- 172.21.15.31:0/2748642866 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d60024070 con 0x560561daec60 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d5bfff640 1 -- 172.21.15.31:0/2748642866 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d60010e00 con 0x560561daec60 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.814+0000 7f3d5bfff640 1 -- 172.21.15.31:0/2748642866 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d60004cc0 con 0x560561daec60 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.815+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2748642866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561daec60 msgr2=0x560561dbd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.815+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2748642866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561daec60 0x560561dbd440 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3d600123f0 tx=0x7f3d6001c000 comp rx=0 tx=0).stop 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.815+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2748642866 shutdown_connections 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.815+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2748642866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dbd980 0x560561dbfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.815+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2748642866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561daec60 0x560561dbd440 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.815+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2748642866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 0x560561dae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.815+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2748642866 >> 172.21.15.31:0/2748642866 conn(0x560561daa8a0 msgr2=0x560561daac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.815+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2748642866 shutdown_connections 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.815+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2748642866 wait complete. 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.815+0000 7f3d7aa75740 1 -- start start 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.815+0000 7f3d7aa75740 1 Processor -- start 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dae2b0 0x560561dbd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561daec60 0x560561dba610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dbabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560561dc06e0 con 0x560561dbd980 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560561ccf050 con 0x560561dae2b0 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560561dc0860 con 0x560561daec60 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d7aa75740 10 client.0 _ll_register_callbacks cb 0x560561a26be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d76300640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dbabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d76300640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dbabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57780/0 (socket says 172.21.15.31:57780) 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d76300640 1 -- 172.21.15.31:0/2830741999 learned_addr learned my addr 172.21.15.31:0/2830741999 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d75aff640 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dae2b0 0x560561dbd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d76300640 1 -- 172.21.15.31:0/2830741999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561daec60 msgr2=0x560561dba610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d76300640 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561daec60 0x560561dba610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d76300640 1 -- 172.21.15.31:0/2830741999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dae2b0 msgr2=0x560561dbd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d76300640 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dae2b0 0x560561dbd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d76300640 1 -- 172.21.15.31:0/2830741999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560561ca0dc0 con 0x560561dbd980 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d75aff640 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dae2b0 0x560561dbd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d76300640 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dbabb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3d68097a30 tx=0x7f3d680bb0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6180f1292ae2e88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d597fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d597fa640 1 -- 172.21.15.31:0/2830741999 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6801b690 con 0x560561dbd980 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d597fa640 1 -- 172.21.15.31:0/2830741999 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f3d6805c6c0 con 0x560561dbd980 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.816+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2830741999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d60010e00 con 0x560561dbd980 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d597fa640 1 -- 172.21.15.31:0/2830741999 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3d680b3020 con 0x560561dbd980 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d7aa75740 10 client.15684 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2830741999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560561a273f0 con 0x560561dbd980 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d597fa640 1 -- 172.21.15.31:0/2830741999 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d60010e00 con 0x560561dbd980 2024-04-25T01:55:06.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d597fa640 1 -- 172.21.15.31:0/2830741999 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3d6801b2c0 con 0x560561dbd980 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d7aa75740 10 client.15684 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d7aa75740 10 client.15684 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d7aa75740 20 client.15684 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x560561dcd9c0 0x560561dcfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d7aa75740 4 client.15684 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2830741999 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560561d20bb0 con 0x560561dcd9c0 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.817+0000 7f3d75aff640 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x560561dcd9c0 0x560561dcfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.818+0000 7f3d597fa640 10 client.15684 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.818+0000 7f3d75aff640 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x560561dcd9c0 0x560561dcfe50 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9755fad43e37a5b4 server_cookie=228951b34080dd77 in_seq=0 out_seq=0 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d597fa640 1 -- 172.21.15.31:0/2830741999 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560561d20bb0 con 0x560561dcd9c0 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d597fa640 10 client.15684 handle_command_reply: tid=0 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 1 client.15684 shutdown 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2830741999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 msgr2=0x560561dbabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dbabb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3d68097a30 tx=0x7f3d680bb0b0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2830741999 shutdown_connections 2024-04-25T01:55:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x560561dcd9c0 0x560561dcfe50 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dbabb0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561daec60 0x560561dba610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2830741999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dae2b0 0x560561dbd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d597fa640 0 client.15684 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:55:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2830741999 >> 172.21.15.31:0/2830741999 conn(0x560561daa8a0 msgr2=0x560561db0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2830741999 shutdown_connections 2024-04-25T01:55:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2830741999 wait complete. 2024-04-25T01:55:06.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.819+0000 7f3d7aa75740 20 client.15684 trim_cache size 0 max 16384 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.832+0000 7f3d7aa75740 1 Processor -- start 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.832+0000 7f3d7aa75740 1 -- start start 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 0x560561dc9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561daec60 0x560561dbce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dd5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701ab190 con 0x560561dbd980 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701accd0 con 0x560561daec60 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701ab860 con 0x560561dae2b0 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d76300640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dd5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d76300640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dd5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57782/0 (socket says 172.21.15.31:57782) 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d76300640 1 -- 172.21.15.31:0/3638259884 learned_addr learned my addr 172.21.15.31:0/3638259884 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d752fe640 1 --2- 172.21.15.31:0/3638259884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561daec60 0x560561dbce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d76300640 1 -- 172.21.15.31:0/3638259884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 msgr2=0x560561dc9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d75aff640 1 --2- 172.21.15.31:0/3638259884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 0x560561dc9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d76300640 1 --2- 172.21.15.31:0/3638259884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 0x560561dc9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d76300640 1 -- 172.21.15.31:0/3638259884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561daec60 msgr2=0x560561dbce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d76300640 1 --2- 172.21.15.31:0/3638259884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561daec60 0x560561dbce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d76300640 1 -- 172.21.15.31:0/3638259884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560561af17d0 con 0x560561dbd980 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d752fe640 1 --2- 172.21.15.31:0/3638259884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561daec60 0x560561dbce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d75aff640 1 --2- 172.21.15.31:0/3638259884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 0x560561dc9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d76300640 1 --2- 172.21.15.31:0/3638259884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dd5720 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f3d6809ad60 tx=0x7f3d68099570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5792ac5dde9b4d99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d5b7fe640 1 -- 172.21.15.31:0/3638259884 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d68014960 con 0x560561dbd980 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d5b7fe640 1 -- 172.21.15.31:0/3638259884 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d6807e070 con 0x560561dbd980 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.833+0000 7f3d5b7fe640 1 -- 172.21.15.31:0/3638259884 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d68095e30 con 0x560561dbd980 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.834+0000 7f3d7aa75740 1 -- 172.21.15.31:0/3638259884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 msgr2=0x560561dd5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.834+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/3638259884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dd5720 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f3d6809ad60 tx=0x7f3d68099570 comp rx=0 tx=0).stop 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.834+0000 7f3d7aa75740 1 -- 172.21.15.31:0/3638259884 shutdown_connections 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.834+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/3638259884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dd5720 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.834+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/3638259884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561daec60 0x560561dbce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.834+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/3638259884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561dae2b0 0x560561dc9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.834+0000 7f3d7aa75740 1 -- 172.21.15.31:0/3638259884 >> 172.21.15.31:0/3638259884 conn(0x560561daa8a0 msgr2=0x560561db5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.834+0000 7f3d7aa75740 1 -- 172.21.15.31:0/3638259884 shutdown_connections 2024-04-25T01:55:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.834+0000 7f3d7aa75740 1 -- 172.21.15.31:0/3638259884 wait complete. 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.834+0000 7f3d7aa75740 1 -- start start 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.834+0000 7f3d7aa75740 1 Processor -- start 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dae2b0 0x560561dc5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561daec60 0x560561dc10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d7aa75740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dc1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701ac600 con 0x560561dbd980 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701abf30 con 0x560561dae2b0 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d7aa75740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3d701a9190 con 0x560561daec60 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d7aa75740 10 client.0 _ll_register_callbacks cb 0x7f3d6809aec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d76300640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dc1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d76300640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dc1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57786/0 (socket says 172.21.15.31:57786) 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d76300640 1 -- 172.21.15.31:0/2937486129 learned_addr learned my addr 172.21.15.31:0/2937486129 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d752fe640 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561daec60 0x560561dc10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d75aff640 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dae2b0 0x560561dc5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d76300640 1 -- 172.21.15.31:0/2937486129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561daec60 msgr2=0x560561dc10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d76300640 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561daec60 0x560561dc10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d76300640 1 -- 172.21.15.31:0/2937486129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dae2b0 msgr2=0x560561dc5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d76300640 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dae2b0 0x560561dc5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d76300640 1 -- 172.21.15.31:0/2937486129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560561dc1c40 con 0x560561dbd980 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d76300640 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dc1600 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f3d68099310 tx=0x7f3d68004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e9a64a03a4ba41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.835+0000 7f3d597fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d597fa640 1 -- 172.21.15.31:0/2937486129 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6809b060 con 0x560561dbd980 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d597fa640 1 -- 172.21.15.31:0/2937486129 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f3d68095bd0 con 0x560561dbd980 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2937486129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d6807e070 con 0x560561dbd980 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d597fa640 1 -- 172.21.15.31:0/2937486129 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3d680aa050 con 0x560561dbd980 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d7aa75740 10 client.15690 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2937486129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560561e33d50 con 0x560561dbd980 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d597fa640 1 -- 172.21.15.31:0/2937486129 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d6807e070 con 0x560561dbd980 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d597fa640 1 -- 172.21.15.31:0/2937486129 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3d6809a4b0 con 0x560561dbd980 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d7aa75740 10 client.15690 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d7aa75740 10 client.15690 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d7aa75740 20 client.15690 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x560561e34130 0x560561e36520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d7aa75740 4 client.15690 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2937486129 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560561d1af00 con 0x560561e34130 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.836+0000 7f3d75aff640 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x560561e34130 0x560561e36520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d597fa640 10 client.15690 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d75aff640 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x560561e34130 0x560561e36520 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c3f23ecd9a2ca962 server_cookie=cb0fbdcac0aedcd0 in_seq=0 out_seq=0 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d597fa640 1 -- 172.21.15.31:0/2937486129 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560561d1af00 con 0x560561e34130 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d597fa640 10 client.15690 handle_command_reply: tid=0 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 1 client.15690 shutdown 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2937486129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 msgr2=0x560561dc1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dc1600 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f3d68099310 tx=0x7f3d68004b50 comp rx=0 tx=0).stop 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2937486129 shutdown_connections 2024-04-25T01:55:06.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x560561e34130 0x560561e36520 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560561dbd980 0x560561dc1600 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560561daec60 0x560561dc10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d597fa640 0 client.15690 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:55:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 1 --2- 172.21.15.31:0/2937486129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560561dae2b0 0x560561dc5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2937486129 >> 172.21.15.31:0/2937486129 conn(0x560561daa8a0 msgr2=0x560561da92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2937486129 shutdown_connections 2024-04-25T01:55:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 1 -- 172.21.15.31:0/2937486129 wait complete. 2024-04-25T01:55:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.837+0000 7f3d7aa75740 20 client.15690 trim_cache size 0 max 16384 2024-04-25T01:55:06.845 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 -- 172.21.15.31:0/3982894920 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3d5c0a2e20 msgr2=0x7f3d5c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3d5c0a2e20 0x7f3d5c0a52e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3d60005d00 tx=0x7f3d6000bb90 comp rx=0 tx=0).stop 2024-04-25T01:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 -- 172.21.15.31:0/3982894920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70106840 msgr2=0x7f3d7019caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70106840 0x7f3d7019caf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800b970 tx=0x7f3d6800be30 comp rx=0 tx=0).stop 2024-04-25T01:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 -- 172.21.15.31:0/3982894920 shutdown_connections 2024-04-25T01:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3d5c0a2e20 0x7f3d5c0a52e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d70106840 0x7f3d7019caf0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d70101710 0x7f3d70198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 --2- 172.21.15.31:0/3982894920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d70100d50 0x7f3d70198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 -- 172.21.15.31:0/3982894920 >> 172.21.15.31:0/3982894920 conn(0x7f3d70076c00 msgr2=0x7f3d700ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 -- 172.21.15.31:0/3982894920 shutdown_connections 2024-04-25T01:55:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.838+0000 7f3d77d63640 1 -- 172.21.15.31:0/3982894920 wait complete. 2024-04-25T01:55:06.849 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:06 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3850726545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:06.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:55:06.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:06 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3850726545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:06.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.953+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/4052098827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f44100990 msgr2=0x7f5f44100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.953+0000 7f5f4b9a2640 1 --2- 172.21.15.31:0/4052098827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f44100990 0x7f5f44100df0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5f34009900 tx=0x7f5f3402a410 comp rx=0 tx=0).stop 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.953+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/4052098827 shutdown_connections 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.953+0000 7f5f4b9a2640 1 --2- 172.21.15.31:0/4052098827 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f44101330 0x7f5f4410d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.953+0000 7f5f4b9a2640 1 --2- 172.21.15.31:0/4052098827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f44100990 0x7f5f44100df0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.953+0000 7f5f4b9a2640 1 --2- 172.21.15.31:0/4052098827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f44106690 0x7f5f44106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.953+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/4052098827 >> 172.21.15.31:0/4052098827 conn(0x7f5f44076440 msgr2=0x7f5f44076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.953+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/4052098827 shutdown_connections 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.953+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/4052098827 wait complete. 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f4b9a2640 1 Processor -- start 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f4b9a2640 1 -- start start 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f4b9a2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f44100990 0x7f5f44198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f4b9a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f44101330 0x7f5f44199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f4b9a2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f44106690 0x7f5f441a3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f4b9a2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441108e0 con 0x7f5f44101330 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f4b9a2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f44110760 con 0x7f5f44106690 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f4b9a2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5f44110a60 con 0x7f5f44100990 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f48f3d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f44101330 0x7f5f44199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f48f3d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f44101330 0x7f5f44199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57802/0 (socket says 172.21.15.31:57802) 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f48f3d640 1 -- 172.21.15.31:0/3022307021 learned_addr learned my addr 172.21.15.31:0/3022307021 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f4973e640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f44100990 0x7f5f44198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f44106690 0x7f5f441a3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f48f3d640 1 -- 172.21.15.31:0/3022307021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f44100990 msgr2=0x7f5f44198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f48f3d640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f44100990 0x7f5f44198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f48f3d640 1 -- 172.21.15.31:0/3022307021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f44106690 msgr2=0x7f5f441a3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f48f3d640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f44106690 0x7f5f441a3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f48f3d640 1 -- 172.21.15.31:0/3022307021 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f441a42c0 con 0x7f5f44101330 2024-04-25T01:55:06.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f44106690 0x7f5f441a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.954+0000 7f5f48f3d640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f44101330 0x7f5f44199240 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5f34005ab0 tx=0x7f5f34003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.955+0000 7f5f3a7fc640 1 -- 172.21.15.31:0/3022307021 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3403f070 con 0x7f5f44101330 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.955+0000 7f5f3a7fc640 1 -- 172.21.15.31:0/3022307021 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f34031d60 con 0x7f5f44101330 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.955+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/3022307021 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f441a4490 con 0x7f5f44101330 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.955+0000 7f5f3a7fc640 1 -- 172.21.15.31:0/3022307021 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f34035030 con 0x7f5f44101330 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.955+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/3022307021 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f441a48f0 con 0x7f5f44101330 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.957+0000 7f5f3a7fc640 1 -- 172.21.15.31:0/3022307021 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f5f34042070 con 0x7f5f44101330 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.957+0000 7f5f4e794740 1 Processor -- start 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.957+0000 7f5f4e794740 1 -- start start 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b32960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32ea0 0x560255b3d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f3a7fc640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5f300a2e20 0x7f5f300a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 0x560255b3f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f4e794740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560255a97670 con 0x560255b32580 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f4e794740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560255a3f970 con 0x560255b32ea0 2024-04-25T01:55:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f4e794740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560255a93630 con 0x560255b3d570 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f49f3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b32960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f49f3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b32960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57804/0 (socket says 172.21.15.31:57804) 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f49f3f640 1 -- 172.21.15.31:0/3837326288 learned_addr learned my addr 172.21.15.31:0/3837326288 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f48f3d640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5f300a2e20 0x7f5f300a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f3a7fc640 1 -- 172.21.15.31:0/3022307021 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5f340efef0 con 0x7f5f44101330 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f4973e640 1 --2- 172.21.15.31:0/3837326288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32ea0 0x560255b3d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/3837326288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 0x560255b3f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f49f3f640 1 -- 172.21.15.31:0/3837326288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 msgr2=0x560255b3f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/3837326288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 0x560255b3f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f49f3f640 1 -- 172.21.15.31:0/3837326288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32ea0 msgr2=0x560255b3d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/3837326288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32ea0 0x560255b3d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f49f3f640 1 -- 172.21.15.31:0/3837326288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560255a63400 con 0x560255b32580 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f48f3d640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5f300a2e20 0x7f5f300a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f340ed570 tx=0x7f5f34049dd0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/3837326288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b32960 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c013770 tx=0x7f5f3c009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be440988d08eacf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.958+0000 7f5f2f7fe640 1 -- 172.21.15.31:0/3837326288 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c019030 con 0x560255b32580 2024-04-25T01:55:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.959+0000 7f5f2f7fe640 1 -- 172.21.15.31:0/3837326288 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f3c013db0 con 0x560255b32580 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.959+0000 7f5f2f7fe640 1 -- 172.21.15.31:0/3837326288 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c005260 con 0x560255b32580 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.959+0000 7f5f4e794740 1 -- 172.21.15.31:0/3837326288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 msgr2=0x560255b32960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.959+0000 7f5f4e794740 1 --2- 172.21.15.31:0/3837326288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b32960 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c013770 tx=0x7f5f3c009890 comp rx=0 tx=0).stop 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.959+0000 7f5f4e794740 1 -- 172.21.15.31:0/3837326288 shutdown_connections 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.959+0000 7f5f4e794740 1 --2- 172.21.15.31:0/3837326288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 0x560255b3f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.959+0000 7f5f4e794740 1 --2- 172.21.15.31:0/3837326288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32ea0 0x560255b3d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.959+0000 7f5f4e794740 1 --2- 172.21.15.31:0/3837326288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b32960 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.959+0000 7f5f4e794740 1 -- 172.21.15.31:0/3837326288 >> 172.21.15.31:0/3837326288 conn(0x560255b2e8a0 msgr2=0x560255b2ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.959+0000 7f5f4e794740 1 -- 172.21.15.31:0/3837326288 shutdown_connections 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.959+0000 7f5f4e794740 1 -- 172.21.15.31:0/3837326288 wait complete. 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4e794740 1 -- start start 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4e794740 1 Processor -- start 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32580 0x560255b3cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32ea0 0x560255b3a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 0x560255b3a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4e794740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560255b40360 con 0x560255b32ea0 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4e794740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560255a53050 con 0x560255b32580 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4e794740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560255b404e0 con 0x560255b3d570 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4e794740 10 client.0 _ll_register_callbacks cb 0x5602557aabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4973e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32ea0 0x560255b3a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4973e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32ea0 0x560255b3a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57816/0 (socket says 172.21.15.31:57816) 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4973e640 1 -- 172.21.15.31:0/2673007571 learned_addr learned my addr 172.21.15.31:0/2673007571 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32580 0x560255b3cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 0x560255b3a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4973e640 1 -- 172.21.15.31:0/2673007571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 msgr2=0x560255b3a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4973e640 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 0x560255b3a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4973e640 1 -- 172.21.15.31:0/2673007571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32580 msgr2=0x560255b3cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4973e640 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32580 0x560255b3cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.960+0000 7f5f4973e640 1 -- 172.21.15.31:0/2673007571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560255a24dc0 con 0x560255b32ea0 2024-04-25T01:55:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32580 0x560255b3cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f4973e640 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32ea0 0x560255b3a200 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000bf40 tx=0x7f5f400118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb76b991f17749d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f2cff9640 1 -- 172.21.15.31:0/2673007571 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40004020 con 0x560255b32ea0 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f2cff9640 1 -- 172.21.15.31:0/2673007571 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5f400041c0 con 0x560255b32ea0 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f4e794740 1 -- 172.21.15.31:0/2673007571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f3c013db0 con 0x560255b32ea0 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f2cff9640 1 -- 172.21.15.31:0/2673007571 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5f40004400 con 0x560255b32ea0 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f4e794740 10 client.15702 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f4e794740 1 -- 172.21.15.31:0/2673007571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5602557ab3f0 con 0x560255b32ea0 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f2cff9640 1 -- 172.21.15.31:0/2673007571 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f3c013db0 con 0x560255b32ea0 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f2cff9640 1 -- 172.21.15.31:0/2673007571 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5f40010700 con 0x560255b32ea0 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f4e794740 10 client.15702 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f4e794740 10 client.15702 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f4e794740 20 client.15702 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f4e794740 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x560255b4d6d0 0x560255b4fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f4e794740 4 client.15702 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.961+0000 7f5f4e794740 1 -- 172.21.15.31:0/2673007571 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560255aa4bb0 con 0x560255b4d6d0 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.962+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x560255b4d6d0 0x560255b4fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.962+0000 7f5f2cff9640 10 client.15702 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.962+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x560255b4d6d0 0x560255b4fb10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1a5e1dd5c8ab801 server_cookie=f1ce39b56d285922 in_seq=0 out_seq=0 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f2cff9640 1 -- 172.21.15.31:0/2673007571 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560255aa4bb0 con 0x560255b4d6d0 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f2cff9640 10 client.15702 handle_command_reply: tid=0 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 1 client.15702 shutdown 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 1 -- 172.21.15.31:0/2673007571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32ea0 msgr2=0x560255b3a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32ea0 0x560255b3a200 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000bf40 tx=0x7f5f400118d0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 1 -- 172.21.15.31:0/2673007571 shutdown_connections 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x560255b4d6d0 0x560255b4fb10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 0x560255b3a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32ea0 0x560255b3a200 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 1 --2- 172.21.15.31:0/2673007571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32580 0x560255b3cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f2cff9640 0 client.15702 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:55:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 1 -- 172.21.15.31:0/2673007571 >> 172.21.15.31:0/2673007571 conn(0x560255b2e8a0 msgr2=0x560255b2db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 1 -- 172.21.15.31:0/2673007571 shutdown_connections 2024-04-25T01:55:06.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 1 -- 172.21.15.31:0/2673007571 wait complete. 2024-04-25T01:55:06.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.963+0000 7f5f4e794740 20 client.15702 trim_cache size 0 max 16384 2024-04-25T01:55:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f4e794740 1 Processor -- start 2024-04-25T01:55:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f4e794740 1 -- start start 2024-04-25T01:55:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32580 0x560255b49250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b32ea0 0x560255b3ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b3d570 0x560255b553e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f4e794740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441ad2d0 con 0x560255b3d570 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f4e794740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441aee10 con 0x560255b32580 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f4e794740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441ad9a0 con 0x560255b32ea0 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f49f3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b3d570 0x560255b553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f49f3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b3d570 0x560255b553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57832/0 (socket says 172.21.15.31:57832) 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f49f3f640 1 -- 172.21.15.31:0/1296134400 learned_addr learned my addr 172.21.15.31:0/1296134400 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f49f3f640 1 -- 172.21.15.31:0/1296134400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b32ea0 msgr2=0x560255b3ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/1296134400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b32ea0 0x560255b3ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f49f3f640 1 -- 172.21.15.31:0/1296134400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32580 msgr2=0x560255b49250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/1296134400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32580 0x560255b49250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f49f3f640 1 -- 172.21.15.31:0/1296134400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602558757d0 con 0x560255b3d570 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/1296134400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b3d570 0x560255b553e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c013fd0 tx=0x7f5f3c013590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a594b9e4f9f0ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f2effd640 1 -- 172.21.15.31:0/1296134400 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c019040 con 0x560255b3d570 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f2effd640 1 -- 172.21.15.31:0/1296134400 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f3c013db0 con 0x560255b3d570 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.977+0000 7f5f2effd640 1 -- 172.21.15.31:0/1296134400 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c02b3c0 con 0x560255b3d570 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.978+0000 7f5f4e794740 1 -- 172.21.15.31:0/1296134400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b3d570 msgr2=0x560255b553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.978+0000 7f5f4e794740 1 --2- 172.21.15.31:0/1296134400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b3d570 0x560255b553e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c013fd0 tx=0x7f5f3c013590 comp rx=0 tx=0).stop 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.978+0000 7f5f4e794740 1 -- 172.21.15.31:0/1296134400 shutdown_connections 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.978+0000 7f5f4e794740 1 --2- 172.21.15.31:0/1296134400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b3d570 0x560255b553e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.978+0000 7f5f4e794740 1 --2- 172.21.15.31:0/1296134400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b32ea0 0x560255b3ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.978+0000 7f5f4e794740 1 --2- 172.21.15.31:0/1296134400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32580 0x560255b49250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.978+0000 7f5f4e794740 1 -- 172.21.15.31:0/1296134400 >> 172.21.15.31:0/1296134400 conn(0x560255b2e8a0 msgr2=0x560255b34560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.978+0000 7f5f4e794740 1 -- 172.21.15.31:0/1296134400 shutdown_connections 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.978+0000 7f5f4e794740 1 -- 172.21.15.31:0/1296134400 wait complete. 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.978+0000 7f5f4e794740 1 -- start start 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.978+0000 7f5f4e794740 1 Processor -- start 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b37290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32ea0 0x560255b377d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f4e794740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 0x560255bb2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f4e794740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441ae740 con 0x560255b32580 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f4e794740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441ae070 con 0x560255b32ea0 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f4e794740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5f441ab2b0 con 0x560255b3d570 2024-04-25T01:55:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f4e794740 10 client.0 _ll_register_callbacks cb 0x7f5f4000bb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f49f3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b37290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f49f3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b37290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57834/0 (socket says 172.21.15.31:57834) 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f49f3f640 1 -- 172.21.15.31:0/3019954898 learned_addr learned my addr 172.21.15.31:0/3019954898 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f4973e640 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32ea0 0x560255b377d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f49f3f640 1 -- 172.21.15.31:0/3019954898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 msgr2=0x560255bb2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 0x560255bb2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f49f3f640 1 -- 172.21.15.31:0/3019954898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32ea0 msgr2=0x560255b377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32ea0 0x560255b377d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f49f3f640 1 -- 172.21.15.31:0/3019954898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560255bb3190 con 0x560255b32580 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f4973e640 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32ea0 0x560255b377d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f49f3f640 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b37290 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c013fa0 tx=0x7f5f3c009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b79f2dc78a4f51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.979+0000 7f5f2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f2cff9640 1 -- 172.21.15.31:0/3019954898 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c019070 con 0x560255b32580 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f2cff9640 1 -- 172.21.15.31:0/3019954898 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5f3c013040 con 0x560255b32580 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f4e794740 1 -- 172.21.15.31:0/3019954898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f3c013db0 con 0x560255b32580 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f2cff9640 1 -- 172.21.15.31:0/3019954898 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5f3c013240 con 0x560255b32580 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f4e794740 10 client.15708 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f4e794740 1 -- 172.21.15.31:0/3019954898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560255bb42f0 con 0x560255b32580 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f2cff9640 1 -- 172.21.15.31:0/3019954898 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f3c013db0 con 0x560255b32580 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f2cff9640 1 -- 172.21.15.31:0/3019954898 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5f3c01e740 con 0x560255b32580 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f4e794740 10 client.15708 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f4e794740 10 client.15708 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f4e794740 20 client.15708 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f4e794740 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x560255bb4710 0x560255bb6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f4e794740 4 client.15708 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f4e794740 1 -- 172.21.15.31:0/3019954898 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560255a9ef00 con 0x560255bb4710 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.980+0000 7f5f4973e640 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x560255bb4710 0x560255bb6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f2cff9640 10 client.15708 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:55:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4973e640 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x560255bb4710 0x560255bb6b00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e63578e9e8586fea server_cookie=e1fb3a2761647e35 in_seq=0 out_seq=0 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f2cff9640 1 -- 172.21.15.31:0/3019954898 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560255a9ef00 con 0x560255bb4710 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f2cff9640 10 client.15708 handle_command_reply: tid=0 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 1 client.15708 shutdown 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 1 -- 172.21.15.31:0/3019954898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 msgr2=0x560255b37290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b37290 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c013fa0 tx=0x7f5f3c009880 comp rx=0 tx=0).stop 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 1 -- 172.21.15.31:0/3019954898 shutdown_connections 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x560255bb4710 0x560255bb6b00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560255b3d570 0x560255bb2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f2cff9640 0 client.15708 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255b32ea0 0x560255b377d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 1 --2- 172.21.15.31:0/3019954898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560255b32580 0x560255b37290 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 1 -- 172.21.15.31:0/3019954898 >> 172.21.15.31:0/3019954898 conn(0x560255b2e8a0 msgr2=0x560255b34d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 1 -- 172.21.15.31:0/3019954898 shutdown_connections 2024-04-25T01:55:06.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 1 -- 172.21.15.31:0/3019954898 wait complete. 2024-04-25T01:55:06.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.981+0000 7f5f4e794740 20 client.15708 trim_cache size 0 max 16384 2024-04-25T01:55:06.989 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/3022307021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5f300a2e20 msgr2=0x7f5f300a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5f300a2e20 0x7f5f300a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f340ed570 tx=0x7f5f34049dd0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/3022307021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f44101330 msgr2=0x7f5f44199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f44101330 0x7f5f44199240 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5f34005ab0 tx=0x7f5f34003cd0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/3022307021 shutdown_connections 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5f300a2e20 0x7f5f300a52e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f44106690 0x7f5f441a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f44101330 0x7f5f44199240 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 --2- 172.21.15.31:0/3022307021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f44100990 0x7f5f44198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/3022307021 >> 172.21.15.31:0/3022307021 conn(0x7f5f44076440 msgr2=0x7f5f440ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/3022307021 shutdown_connections 2024-04-25T01:55:06.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:06.982+0000 7f5f4b9a2640 1 -- 172.21.15.31:0/3022307021 wait complete. 2024-04-25T01:55:06.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.096+0000 7f811e272640 1 -- 172.21.15.31:0/334638366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8118069dc0 msgr2=0x7f811806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.096+0000 7f811e272640 1 --2- 172.21.15.31:0/334638366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8118069dc0 0x7f811806a240 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8100009930 tx=0x7f810002a420 comp rx=0 tx=0).stop 2024-04-25T01:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.096+0000 7f811e272640 1 -- 172.21.15.31:0/334638366 shutdown_connections 2024-04-25T01:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.096+0000 7f811e272640 1 --2- 172.21.15.31:0/334638366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f811806a780 0x7f811810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.096+0000 7f811e272640 1 --2- 172.21.15.31:0/334638366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8118069dc0 0x7f811806a240 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.096+0000 7f811e272640 1 --2- 172.21.15.31:0/334638366 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8118107730 0x7f8118107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.096+0000 7f811e272640 1 -- 172.21.15.31:0/334638366 >> 172.21.15.31:0/334638366 conn(0x7f81180776a0 msgr2=0x7f8118077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.096+0000 7f811e272640 1 -- 172.21.15.31:0/334638366 shutdown_connections 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.096+0000 7f811e272640 1 -- 172.21.15.31:0/334638366 wait complete. 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.097+0000 7f811e272640 1 Processor -- start 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.097+0000 7f811e272640 1 -- start start 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.097+0000 7f811e272640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8118069dc0 0x7f8118197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.097+0000 7f811e272640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811806a780 0x7f8118198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.097+0000 7f811e272640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8118107730 0x7f811819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.097+0000 7f811e272640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f81181119f0 con 0x7f811806a780 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.097+0000 7f811e272640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8118111870 con 0x7f8118107730 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.097+0000 7f811e272640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8118111b70 con 0x7f8118069dc0 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.097+0000 7f81177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811806a780 0x7f8118198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.097+0000 7f81177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811806a780 0x7f8118198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57860/0 (socket says 172.21.15.31:57860) 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.097+0000 7f81177fe640 1 -- 172.21.15.31:0/3922631268 learned_addr learned my addr 172.21.15.31:0/3922631268 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f811c80f640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8118107730 0x7f811819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f81177fe640 1 -- 172.21.15.31:0/3922631268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8118069dc0 msgr2=0x7f8118197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f81177fe640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8118069dc0 0x7f8118197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f81177fe640 1 -- 172.21.15.31:0/3922631268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8118107730 msgr2=0x7f811819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f81177fe640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8118107730 0x7f811819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f81177fe640 1 -- 172.21.15.31:0/3922631268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811819cf00 con 0x7f811806a780 2024-04-25T01:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f81177fe640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811806a780 0x7f8118198450 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f8100005940 tx=0x7f81000042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f81157fa640 1 -- 172.21.15.31:0/3922631268 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810003f030 con 0x7f811806a780 2024-04-25T01:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f81157fa640 1 -- 172.21.15.31:0/3922631268 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8100003cb0 con 0x7f811806a780 2024-04-25T01:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f811e272640 1 -- 172.21.15.31:0/3922631268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811819d190 con 0x7f811806a780 2024-04-25T01:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f81157fa640 1 -- 172.21.15.31:0/3922631268 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8100035070 con 0x7f811806a780 2024-04-25T01:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.098+0000 7f811e272640 1 -- 172.21.15.31:0/3922631268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81181a4a80 con 0x7f811806a780 2024-04-25T01:55:07.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.100+0000 7f81157fa640 1 -- 172.21.15.31:0/3922631268 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f8100042030 con 0x7f811806a780 2024-04-25T01:55:07.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8120f7d740 1 Processor -- start 2024-04-25T01:55:07.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8120f7d740 1 -- start start 2024-04-25T01:55:07.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0580 0x55bbec0d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0ea0 0x55bbec0db030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 0x55bbec0dd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8120f7d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bbec035670 con 0x55bbec0d0ea0 2024-04-25T01:55:07.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8120f7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bbebfdd970 con 0x55bbec0db570 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8120f7d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bbec031630 con 0x55bbec0d0580 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8117fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0ea0 0x55bbec0db030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8117fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0ea0 0x55bbec0db030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57866/0 (socket says 172.21.15.31:57866) 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8117fff640 1 -- 172.21.15.31:0/2042162863 learned_addr learned my addr 172.21.15.31:0/2042162863 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f81157fa640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f81040a2ea0 0x7f81040a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f811c80f640 1 --2- 172.21.15.31:0/2042162863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0580 0x55bbec0d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f811c80f640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f81040a2ea0 0x7f81040a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f81177fe640 1 --2- 172.21.15.31:0/2042162863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 0x55bbec0dd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8117fff640 1 -- 172.21.15.31:0/2042162863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0580 msgr2=0x55bbec0d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f81157fa640 1 -- 172.21.15.31:0/3922631268 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f81000efbb0 con 0x7f811806a780 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8117fff640 1 --2- 172.21.15.31:0/2042162863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0580 0x55bbec0d0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8117fff640 1 -- 172.21.15.31:0/2042162863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 msgr2=0x55bbec0dd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8117fff640 1 --2- 172.21.15.31:0/2042162863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 0x55bbec0dd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f8117fff640 1 -- 172.21.15.31:0/2042162863 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbec001400 con 0x55bbec0d0ea0 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f811c80f640 1 --2- 172.21.15.31:0/2042162863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0580 0x55bbec0d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.101+0000 7f81177fe640 1 --2- 172.21.15.31:0/2042162863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 0x55bbec0dd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f8117fff640 1 --2- 172.21.15.31:0/2042162863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0ea0 0x55bbec0db030 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8108009820 tx=0x7f8108009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e040789bd42f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f811c80f640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f81040a2ea0 0x7f81040a5360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8110013600 tx=0x7f8110004a40 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f810e7fc640 1 -- 172.21.15.31:0/2042162863 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108015020 con 0x55bbec0d0ea0 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f810e7fc640 1 -- 172.21.15.31:0/2042162863 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8108011420 con 0x55bbec0d0ea0 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f810e7fc640 1 -- 172.21.15.31:0/2042162863 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108010580 con 0x55bbec0d0ea0 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f8120f7d740 1 -- 172.21.15.31:0/2042162863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0ea0 msgr2=0x55bbec0db030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f8120f7d740 1 --2- 172.21.15.31:0/2042162863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0ea0 0x55bbec0db030 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8108009820 tx=0x7f8108009ce0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f8120f7d740 1 -- 172.21.15.31:0/2042162863 shutdown_connections 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f8120f7d740 1 --2- 172.21.15.31:0/2042162863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 0x55bbec0dd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f8120f7d740 1 --2- 172.21.15.31:0/2042162863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0ea0 0x55bbec0db030 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f8120f7d740 1 --2- 172.21.15.31:0/2042162863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0580 0x55bbec0d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.102+0000 7f8120f7d740 1 -- 172.21.15.31:0/2042162863 >> 172.21.15.31:0/2042162863 conn(0x55bbec0cc8a0 msgr2=0x55bbec0ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8120f7d740 1 -- 172.21.15.31:0/2042162863 shutdown_connections 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8120f7d740 1 -- 172.21.15.31:0/2042162863 wait complete. 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8120f7d740 1 -- start start 2024-04-25T01:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8120f7d740 1 Processor -- start 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0d0580 0x55bbec0dae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0d8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0d8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8120f7d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bbec0de360 con 0x55bbec0db570 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8120f7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bbebff1050 con 0x55bbec0d0580 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8120f7d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bbec0de4e0 con 0x55bbec0d0ea0 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8120f7d740 10 client.0 _ll_register_callbacks cb 0x55bbebd48be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f811c80f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0d8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f811c80f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0d8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57876/0 (socket says 172.21.15.31:57876) 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f811c80f640 1 -- 172.21.15.31:0/3779974857 learned_addr learned my addr 172.21.15.31:0/3779974857 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.103+0000 7f8117fff640 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0d0580 0x55bbec0dae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f811c80f640 1 -- 172.21.15.31:0/3779974857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 msgr2=0x55bbec0d8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f811c80f640 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0d8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f811c80f640 1 -- 172.21.15.31:0/3779974857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0d0580 msgr2=0x55bbec0dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f811c80f640 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0d0580 0x55bbec0dae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f811c80f640 1 -- 172.21.15.31:0/3779974857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbebfc2dc0 con 0x55bbec0db570 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f811c80f640 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0d8680 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f811000ef00 tx=0x7f8110004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc1a4f0c7631af1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f80f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f80f77fe640 1 -- 172.21.15.31:0/3779974857 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110022070 con 0x55bbec0db570 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f80f77fe640 1 -- 172.21.15.31:0/3779974857 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f8110010c80 con 0x55bbec0db570 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f8120f7d740 1 -- 172.21.15.31:0/3779974857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8108011420 con 0x55bbec0db570 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f80f77fe640 1 -- 172.21.15.31:0/3779974857 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f811002e050 con 0x55bbec0db570 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f8120f7d740 10 client.15720 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f8120f7d740 1 -- 172.21.15.31:0/3779974857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bbebd493f0 con 0x55bbec0db570 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.104+0000 7f80f77fe640 1 -- 172.21.15.31:0/3779974857 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8108011420 con 0x55bbec0db570 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.105+0000 7f80f77fe640 1 -- 172.21.15.31:0/3779974857 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f811001d040 con 0x55bbec0db570 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.105+0000 7f8120f7d740 10 client.15720 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.105+0000 7f8120f7d740 10 client.15720 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.105+0000 7f8120f7d740 20 client.15720 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.105+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bbec0eb4c0 0x55bbec0ed980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.105+0000 7f8120f7d740 4 client.15720 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.105+0000 7f8120f7d740 1 -- 172.21.15.31:0/3779974857 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbec042bb0 con 0x55bbec0eb4c0 2024-04-25T01:55:07.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.105+0000 7f8117fff640 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bbec0eb4c0 0x55bbec0ed980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.105+0000 7f80f77fe640 10 client.15720 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:55:07.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.105+0000 7f8117fff640 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bbec0eb4c0 0x55bbec0ed980 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c3348f91d2ac4d47 server_cookie=82f9ce2630b55ae0 in_seq=0 out_seq=0 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.106+0000 7f80f77fe640 1 -- 172.21.15.31:0/3779974857 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bbec042bb0 con 0x55bbec0eb4c0 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.106+0000 7f80f77fe640 10 client.15720 handle_command_reply: tid=0 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.106+0000 7f8120f7d740 1 client.15720 shutdown 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.106+0000 7f8120f7d740 1 -- 172.21.15.31:0/3779974857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 msgr2=0x55bbec0d8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.106+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0d8680 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f811000ef00 tx=0x7f8110004ab0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.107+0000 7f8120f7d740 1 -- 172.21.15.31:0/3779974857 shutdown_connections 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.107+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bbec0eb4c0 0x55bbec0ed980 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.107+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0d8680 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.107+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0d8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.107+0000 7f80f77fe640 0 client.15720 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.107+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3779974857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0d0580 0x55bbec0dae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.107+0000 7f8120f7d740 1 -- 172.21.15.31:0/3779974857 >> 172.21.15.31:0/3779974857 conn(0x55bbec0cc8a0 msgr2=0x55bbec0cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.107+0000 7f8120f7d740 1 -- 172.21.15.31:0/3779974857 shutdown_connections 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.107+0000 7f8120f7d740 1 -- 172.21.15.31:0/3779974857 wait complete. 2024-04-25T01:55:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.107+0000 7f8120f7d740 20 client.15720 trim_cache size 0 max 16384 2024-04-25T01:55:07.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8120f7d740 1 Processor -- start 2024-04-25T01:55:07.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8120f7d740 1 -- start start 2024-04-25T01:55:07.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0580 0x55bbec0d3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0da980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 0x55bbec0f3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8120f7d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f81181ad120 con 0x55bbec0d0580 2024-04-25T01:55:07.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8120f7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f81181aec60 con 0x55bbec0db570 2024-04-25T01:55:07.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8120f7d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f81181ad7f0 con 0x55bbec0d0ea0 2024-04-25T01:55:07.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8117fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0580 0x55bbec0d3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8117fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0580 0x55bbec0d3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57888/0 (socket says 172.21.15.31:57888) 2024-04-25T01:55:07.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8117fff640 1 -- 172.21.15.31:0/909185293 learned_addr learned my addr 172.21.15.31:0/909185293 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f811c80f640 1 --2- 172.21.15.31:0/909185293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 0x55bbec0f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f81177fe640 1 --2- 172.21.15.31:0/909185293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0da980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8117fff640 1 -- 172.21.15.31:0/909185293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 msgr2=0x55bbec0da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8117fff640 1 --2- 172.21.15.31:0/909185293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0da980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8117fff640 1 -- 172.21.15.31:0/909185293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 msgr2=0x55bbec0f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8117fff640 1 --2- 172.21.15.31:0/909185293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 0x55bbec0f3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f8117fff640 1 -- 172.21.15.31:0/909185293 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbebe137d0 con 0x55bbec0d0580 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f81177fe640 1 --2- 172.21.15.31:0/909185293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0da980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.120+0000 7f811c80f640 1 --2- 172.21.15.31:0/909185293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 0x55bbec0f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f8117fff640 1 --2- 172.21.15.31:0/909185293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0580 0x55bbec0d3860 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8108009820 tx=0x7f810801c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16c89a553d23d86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f810dffb640 1 -- 172.21.15.31:0/909185293 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108015040 con 0x55bbec0d0580 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f810dffb640 1 -- 172.21.15.31:0/909185293 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f810802c070 con 0x55bbec0d0580 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f810dffb640 1 -- 172.21.15.31:0/909185293 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810800bea0 con 0x55bbec0d0580 2024-04-25T01:55:07.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f8120f7d740 1 -- 172.21.15.31:0/909185293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0580 msgr2=0x55bbec0d3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f8120f7d740 1 --2- 172.21.15.31:0/909185293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0580 0x55bbec0d3860 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8108009820 tx=0x7f810801c040 comp rx=0 tx=0).stop 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f8120f7d740 1 -- 172.21.15.31:0/909185293 shutdown_connections 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f8120f7d740 1 --2- 172.21.15.31:0/909185293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0db570 0x55bbec0f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f8120f7d740 1 --2- 172.21.15.31:0/909185293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0da980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f8120f7d740 1 --2- 172.21.15.31:0/909185293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0d0580 0x55bbec0d3860 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f8120f7d740 1 -- 172.21.15.31:0/909185293 >> 172.21.15.31:0/909185293 conn(0x55bbec0cc8a0 msgr2=0x55bbec0cb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f8120f7d740 1 -- 172.21.15.31:0/909185293 shutdown_connections 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.121+0000 7f8120f7d740 1 -- 172.21.15.31:0/909185293 wait complete. 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f8120f7d740 1 -- start start 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f8120f7d740 1 Processor -- start 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0d0580 0x55bbec0e3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0dee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f8120f7d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0df360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f8120f7d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f81181ae590 con 0x55bbec0db570 2024-04-25T01:55:07.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f8120f7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f81181adec0 con 0x55bbec0d0580 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f8120f7d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f81181ab100 con 0x55bbec0d0ea0 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f8120f7d740 10 client.0 _ll_register_callbacks cb 0x7f8110007a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f811c80f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0df360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f811c80f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0df360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57898/0 (socket says 172.21.15.31:57898) 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f811c80f640 1 -- 172.21.15.31:0/3214021463 learned_addr learned my addr 172.21.15.31:0/3214021463 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.122+0000 7f8117fff640 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0d0580 0x55bbec0e3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f81177fe640 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0dee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f811c80f640 1 -- 172.21.15.31:0/3214021463 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 msgr2=0x55bbec0dee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f811c80f640 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0dee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f811c80f640 1 -- 172.21.15.31:0/3214021463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0d0580 msgr2=0x55bbec0e3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f811c80f640 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0d0580 0x55bbec0e3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f811c80f640 1 -- 172.21.15.31:0/3214021463 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbec0df9a0 con 0x55bbec0db570 2024-04-25T01:55:07.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f81177fe640 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0dee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f811c80f640 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0df360 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8110015040 tx=0x7f8110010950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4a8c5c9f613b0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f80f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f80f77fe640 1 -- 172.21.15.31:0/3214021463 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110036020 con 0x55bbec0db570 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f80f77fe640 1 -- 172.21.15.31:0/3214021463 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f8110010c80 con 0x55bbec0db570 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f8120f7d740 1 -- 172.21.15.31:0/3214021463 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f810802c070 con 0x55bbec0db570 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f80f77fe640 1 -- 172.21.15.31:0/3214021463 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8110041050 con 0x55bbec0db570 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f8120f7d740 10 client.15726 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f8120f7d740 1 -- 172.21.15.31:0/3214021463 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bbec151860 con 0x55bbec0db570 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.123+0000 7f80f77fe640 1 -- 172.21.15.31:0/3214021463 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f810802c070 con 0x55bbec0db570 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f80f77fe640 1 -- 172.21.15.31:0/3214021463 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f811001d050 con 0x55bbec0db570 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f8120f7d740 10 client.15726 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f8120f7d740 10 client.15726 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f8120f7d740 20 client.15726 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bbec151fd0 0x55bbec1543f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f8120f7d740 4 client.15726 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f8120f7d740 1 -- 172.21.15.31:0/3214021463 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bbec03cf00 con 0x55bbec151fd0 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f8117fff640 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bbec151fd0 0x55bbec1543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f8117fff640 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bbec151fd0 0x55bbec1543f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f4d821da5fd065f2 server_cookie=aee56fca7689b0a0 in_seq=0 out_seq=0 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f80f77fe640 10 client.15726 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f80f77fe640 1 -- 172.21.15.31:0/3214021463 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bbec03cf00 con 0x55bbec151fd0 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f80f77fe640 10 client.15726 handle_command_reply: tid=0 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.124+0000 7f8120f7d740 1 client.15726 shutdown 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f8120f7d740 1 -- 172.21.15.31:0/3214021463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 msgr2=0x55bbec0df360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0df360 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8110015040 tx=0x7f8110010950 comp rx=0 tx=0).stop 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f8120f7d740 1 -- 172.21.15.31:0/3214021463 shutdown_connections 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bbec151fd0 0x55bbec1543f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bbec0db570 0x55bbec0df360 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bbec0d0ea0 0x55bbec0dee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f80f77fe640 0 client.15726 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:55:07.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f8120f7d740 1 --2- 172.21.15.31:0/3214021463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bbec0d0580 0x55bbec0e3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f8120f7d740 1 -- 172.21.15.31:0/3214021463 >> 172.21.15.31:0/3214021463 conn(0x55bbec0cc8a0 msgr2=0x55bbec0e6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f8120f7d740 1 -- 172.21.15.31:0/3214021463 shutdown_connections 2024-04-25T01:55:07.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f8120f7d740 1 -- 172.21.15.31:0/3214021463 wait complete. 2024-04-25T01:55:07.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f8120f7d740 20 client.15726 trim_cache size 0 max 16384 2024-04-25T01:55:07.133 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:55:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f811e272640 1 -- 172.21.15.31:0/3922631268 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f81040a2ea0 msgr2=0x7f81040a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f811e272640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f81040a2ea0 0x7f81040a5360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8110013600 tx=0x7f8110004a40 comp rx=0 tx=0).stop 2024-04-25T01:55:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f811e272640 1 -- 172.21.15.31:0/3922631268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811806a780 msgr2=0x7f8118198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f811e272640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811806a780 0x7f8118198450 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f8100005940 tx=0x7f81000042f0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f811e272640 1 -- 172.21.15.31:0/3922631268 shutdown_connections 2024-04-25T01:55:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f811e272640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f81040a2ea0 0x7f81040a5360 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f811e272640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8118107730 0x7f811819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f811e272640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f811806a780 0x7f8118198450 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f811e272640 1 --2- 172.21.15.31:0/3922631268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8118069dc0 0x7f8118197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.125+0000 7f811e272640 1 -- 172.21.15.31:0/3922631268 >> 172.21.15.31:0/3922631268 conn(0x7f81180776a0 msgr2=0x7f81180fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.126+0000 7f811e272640 1 -- 172.21.15.31:0/3922631268 shutdown_connections 2024-04-25T01:55:07.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.126+0000 7f811e272640 1 -- 172.21.15.31:0/3922631268 wait complete. 2024-04-25T01:55:07.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:55:07.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:06 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3850726545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:07.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.240+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1536517081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e0107730 msgr2=0x7fa7e0107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.240+0000 7fa7e5fa5640 1 --2- 172.21.15.31:0/1536517081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e0107730 0x7fa7e0107b10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009930 tx=0x7fa7d002a420 comp rx=0 tx=0).stop 2024-04-25T01:55:07.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.240+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1536517081 shutdown_connections 2024-04-25T01:55:07.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.240+0000 7fa7e5fa5640 1 --2- 172.21.15.31:0/1536517081 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7e006a780 0x7fa7e010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.240+0000 7fa7e5fa5640 1 --2- 172.21.15.31:0/1536517081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e0069dc0 0x7fa7e006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.240+0000 7fa7e5fa5640 1 --2- 172.21.15.31:0/1536517081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e0107730 0x7fa7e0107b10 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.240+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1536517081 >> 172.21.15.31:0/1536517081 conn(0x7fa7e00776a0 msgr2=0x7fa7e0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.240+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1536517081 shutdown_connections 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.240+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1536517081 wait complete. 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7e5fa5640 1 Processor -- start 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7e5fa5640 1 -- start start 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7e5fa5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7e0069dc0 0x7fa7e019a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7e5fa5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e006a780 0x7fa7e019a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7e5fa5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e0107730 0x7fa7e019ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7e5fa5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e01119f0 con 0x7fa7e0107730 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7e5fa5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0111870 con 0x7fa7e006a780 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7e5fa5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0111b70 con 0x7fa7e0069dc0 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e0107730 0x7fa7e019ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e0107730 0x7fa7e019ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57918/0 (socket says 172.21.15.31:57918) 2024-04-25T01:55:07.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7dffff640 1 -- 172.21.15.31:0/1579513786 learned_addr learned my addr 172.21.15.31:0/1579513786 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.241+0000 7fa7deffd640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e006a780 0x7fa7e019a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7e0069dc0 0x7fa7e019a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7deffd640 1 -- 172.21.15.31:0/1579513786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7e0069dc0 msgr2=0x7fa7e019a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7deffd640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7e0069dc0 0x7fa7e019a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7deffd640 1 -- 172.21.15.31:0/1579513786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e0107730 msgr2=0x7fa7e019ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7deffd640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e0107730 0x7fa7e019ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7deffd640 1 -- 172.21.15.31:0/1579513786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e019f140 con 0x7fa7e006a780 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7dffff640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e0107730 0x7fa7e019ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7e0069dc0 0x7fa7e019a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7deffd640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e006a780 0x7fa7e019a690 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4005fd0 tx=0x7fa7d400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7dcff9640 1 -- 172.21.15.31:0/1579513786 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4009c30 con 0x7fa7e006a780 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1579513786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e019f430 con 0x7fa7e006a780 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1579513786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e01a6cc0 con 0x7fa7e006a780 2024-04-25T01:55:07.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7dcff9640 1 -- 172.21.15.31:0/1579513786 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7d4009dd0 con 0x7fa7e006a780 2024-04-25T01:55:07.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.242+0000 7fa7dcff9640 1 -- 172.21.15.31:0/1579513786 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4020060 con 0x7fa7e006a780 2024-04-25T01:55:07.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7e8d8d740 1 Processor -- start 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7e8d8d740 1 -- start start 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7dcff9640 1 -- 172.21.15.31:0/1579513786 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7fa7d40035f0 con 0x7fa7e006a780 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94ad960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5628b9405670 con 0x5628b94ab570 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5628b93ad970 con 0x5628b94a0ea0 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5628b9401630 con 0x5628b94a0580 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7deffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7dffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7deffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94ad960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57922/0 (socket says 172.21.15.31:57922) 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7deffd640 1 -- 172.21.15.31:0/3131329799 learned_addr learned my addr 172.21.15.31:0/3131329799 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/3131329799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7dffff640 1 -- 172.21.15.31:0/3131329799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 msgr2=0x5628b94ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7dffff640 1 --2- 172.21.15.31:0/3131329799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94ab030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7dffff640 1 -- 172.21.15.31:0/3131329799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 msgr2=0x5628b94ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7dffff640 1 --2- 172.21.15.31:0/3131329799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7dffff640 1 -- 172.21.15.31:0/3131329799 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b93d1400 con 0x5628b94a0580 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7deffd640 1 --2- 172.21.15.31:0/3131329799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.245+0000 7fa7dcff9640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa7c80a2e20 0x7fa7c80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7dffff640 1 --2- 172.21.15.31:0/3131329799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94a0960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800e4d0 tx=0x7fa7d8009710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae7611e8d94e287f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa7c80a2e20 0x7fa7c80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7dcff9640 1 -- 172.21.15.31:0/1579513786 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa7d40cedf0 con 0x7fa7e006a780 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7c5ffb640 1 -- 172.21.15.31:0/3131329799 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8018030 con 0x5628b94a0580 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7c5ffb640 1 -- 172.21.15.31:0/3131329799 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7d8014b40 con 0x5628b94a0580 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7c5ffb640 1 -- 172.21.15.31:0/3131329799 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8013be0 con 0x5628b94a0580 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3131329799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 msgr2=0x5628b94a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa7c80a2e20 0x7fa7c80a52e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0000c00 tx=0x7fa7d00058b0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/3131329799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94a0960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800e4d0 tx=0x7fa7d8009710 comp rx=0 tx=0).stop 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3131329799 shutdown_connections 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/3131329799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/3131329799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/3131329799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94a0960 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.246+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3131329799 >> 172.21.15.31:0/3131329799 conn(0x5628b949c8a0 msgr2=0x5628b949cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3131329799 shutdown_connections 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3131329799 wait complete. 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7e8d8d740 1 -- start start 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7e8d8d740 1 Processor -- start 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94aae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94a8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94a8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5628b94ae360 con 0x5628b94ab570 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5628b93c1050 con 0x5628b94a0ea0 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5628b94ae4e0 con 0x5628b94a0580 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7e8d8d740 10 client.0 _ll_register_callbacks cb 0x5628b9118be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:07.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94a8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94a8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57928/0 (socket says 172.21.15.31:57928) 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.247+0000 7fa7dffff640 1 -- 172.21.15.31:0/3446796664 learned_addr learned my addr 172.21.15.31:0/3446796664 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7deffd640 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94a8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7dffff640 1 -- 172.21.15.31:0/3446796664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 msgr2=0x5628b94aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7dffff640 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7dffff640 1 -- 172.21.15.31:0/3446796664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 msgr2=0x5628b94a8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7dffff640 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94a8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7dffff640 1 -- 172.21.15.31:0/3446796664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b9392dc0 con 0x5628b94ab570 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7dffff640 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94a8670 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800f010 tx=0x7fa7d800e1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d379f07e8777d1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7beffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7beffd640 1 -- 172.21.15.31:0/3446796664 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8013d70 con 0x5628b94ab570 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7beffd640 1 -- 172.21.15.31:0/3446796664 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa7d8029480 con 0x5628b94ab570 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3446796664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7d8014b40 con 0x5628b94ab570 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7beffd640 1 -- 172.21.15.31:0/3446796664 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa7d8029680 con 0x5628b94ab570 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7e8d8d740 10 client.15738 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3446796664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5628b91193f0 con 0x5628b94ab570 2024-04-25T01:55:07.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.248+0000 7fa7beffd640 1 -- 172.21.15.31:0/3446796664 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7d8014b40 con 0x5628b94ab570 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.249+0000 7fa7beffd640 1 -- 172.21.15.31:0/3446796664 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa7d801e450 con 0x5628b94ab570 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.249+0000 7fa7e8d8d740 10 client.15738 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.249+0000 7fa7e8d8d740 10 client.15738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.249+0000 7fa7e8d8d740 20 client.15738 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.249+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5628b94bb480 0x5628b94bd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.249+0000 7fa7e8d8d740 4 client.15738 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.249+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3446796664 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628b9412bb0 con 0x5628b94bb480 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.249+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5628b94bb480 0x5628b94bd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.249+0000 7fa7beffd640 10 client.15738 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.249+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5628b94bb480 0x5628b94bd870 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34cf1eefc0b47ad8 server_cookie=9a68f0e5d59dbe6a in_seq=0 out_seq=0 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7beffd640 1 -- 172.21.15.31:0/3446796664 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5628b9412bb0 con 0x5628b94bb480 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7beffd640 10 client.15738 handle_command_reply: tid=0 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7e8d8d740 1 client.15738 shutdown 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3446796664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 msgr2=0x5628b94a8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94a8670 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800f010 tx=0x7fa7d800e1c0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3446796664 shutdown_connections 2024-04-25T01:55:07.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5628b94bb480 0x5628b94bd870 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94a8670 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94a8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7beffd640 0 client.15738 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/3446796664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94aae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3446796664 >> 172.21.15.31:0/3446796664 conn(0x5628b949c8a0 msgr2=0x5628b949bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.250+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3446796664 shutdown_connections 2024-04-25T01:55:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.251+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/3446796664 wait complete. 2024-04-25T01:55:07.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.251+0000 7fa7e8d8d740 20 client.15738 trim_cache size 0 max 16384 2024-04-25T01:55:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7e8d8d740 1 Processor -- start 2024-04-25T01:55:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7e8d8d740 1 -- start start 2024-04-25T01:55:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94a0580 0x5628b94b7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94ab570 0x5628b94aa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc00e600 con 0x5628b94a0580 2024-04-25T01:55:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc010140 con 0x5628b94a0ea0 2024-04-25T01:55:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc00ecd0 con 0x5628b94ab570 2024-04-25T01:55:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7df7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94a0580 0x5628b94b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7df7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94a0580 0x5628b94b7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57940/0 (socket says 172.21.15.31:57940) 2024-04-25T01:55:07.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7df7fe640 1 -- 172.21.15.31:0/615109039 learned_addr learned my addr 172.21.15.31:0/615109039 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7deffd640 1 --2- 172.21.15.31:0/615109039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7dffff640 1 --2- 172.21.15.31:0/615109039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94ab570 0x5628b94aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7df7fe640 1 -- 172.21.15.31:0/615109039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94ab570 msgr2=0x5628b94aa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/615109039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94ab570 0x5628b94aa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7df7fe640 1 -- 172.21.15.31:0/615109039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 msgr2=0x5628b94a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/615109039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94a35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7df7fe640 1 -- 172.21.15.31:0/615109039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b91e37d0 con 0x5628b94a0580 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7dffff640 1 --2- 172.21.15.31:0/615109039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94ab570 0x5628b94aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.264+0000 7fa7deffd640 1 --2- 172.21.15.31:0/615109039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/615109039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94a0580 0x5628b94b7280 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0031630 tx=0x7fa7d003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c35f88ff5cbb6f9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7c57fa640 1 -- 172.21.15.31:0/615109039 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0035040 con 0x5628b94a0580 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7c57fa640 1 -- 172.21.15.31:0/615109039 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7d0055070 con 0x5628b94a0580 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7c57fa640 1 -- 172.21.15.31:0/615109039 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d00055b0 con 0x5628b94a0580 2024-04-25T01:55:07.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/615109039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94a0580 msgr2=0x5628b94b7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/615109039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94a0580 0x5628b94b7280 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0031630 tx=0x7fa7d003e000 comp rx=0 tx=0).stop 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/615109039 shutdown_connections 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/615109039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94ab570 0x5628b94aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/615109039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/615109039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94a0580 0x5628b94b7280 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/615109039 >> 172.21.15.31:0/615109039 conn(0x5628b949c8a0 msgr2=0x5628b94a2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/615109039 shutdown_connections 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.265+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/615109039 wait complete. 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7e8d8d740 1 -- start start 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7e8d8d740 1 Processor -- start 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94aed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7e8d8d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc00fa70 con 0x5628b94ab570 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc00f3a0 con 0x5628b94a0ea0 2024-04-25T01:55:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7e8d8d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc00c5e0 con 0x5628b94a0580 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7e8d8d740 10 client.0 _ll_register_callbacks cb 0x7fa7d40d27d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57942/0 (socket says 172.21.15.31:57942) 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7dffff640 1 -- 172.21.15.31:0/1714173765 learned_addr learned my addr 172.21.15.31:0/1714173765 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7deffd640 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.266+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7dffff640 1 -- 172.21.15.31:0/1714173765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 msgr2=0x5628b94b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7dffff640 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7dffff640 1 -- 172.21.15.31:0/1714173765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 msgr2=0x5628b94aed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7dffff640 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94aed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7dffff640 1 -- 172.21.15.31:0/1714173765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b94af7d0 con 0x5628b94ab570 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7dffff640 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94af290 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8004b90 tx=0x7fa7d801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf95a21413347b33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7beffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:07.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7beffd640 1 -- 172.21.15.31:0/1714173765 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8018030 con 0x5628b94ab570 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7beffd640 1 -- 172.21.15.31:0/1714173765 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa7d8029480 con 0x5628b94ab570 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/1714173765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7d0055070 con 0x5628b94ab570 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7beffd640 1 -- 172.21.15.31:0/1714173765 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa7d8043050 con 0x5628b94ab570 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7e8d8d740 10 client.15744 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/1714173765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5628b95214b0 con 0x5628b94ab570 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7beffd640 1 -- 172.21.15.31:0/1714173765 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7d0055070 con 0x5628b94ab570 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7beffd640 1 -- 172.21.15.31:0/1714173765 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa7d8029b60 con 0x5628b94ab570 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7e8d8d740 10 client.15744 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7e8d8d740 10 client.15744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7e8d8d740 20 client.15744 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5628b9521bd0 0x5628b9523ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7e8d8d740 4 client.15744 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.267+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/1714173765 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628b94a8a00 con 0x5628b9521bd0 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.268+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5628b9521bd0 0x5628b9523ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:07.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.268+0000 7fa7beffd640 10 client.15744 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.268+0000 7fa7df7fe640 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5628b9521bd0 0x5628b9523ff0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1d0de2e28147678 server_cookie=e067df2185385c81 in_seq=0 out_seq=0 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.268+0000 7fa7beffd640 1 -- 172.21.15.31:0/1714173765 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628b94a8a00 con 0x5628b9521bd0 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.268+0000 7fa7beffd640 10 client.15744 handle_command_reply: tid=0 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.268+0000 7fa7e8d8d740 1 client.15744 shutdown 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/1714173765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 msgr2=0x5628b94af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94af290 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8004b90 tx=0x7fa7d801ec70 comp rx=0 tx=0).stop 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/1714173765 shutdown_connections 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5628b9521bd0 0x5628b9523ff0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628b94ab570 0x5628b94af290 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628b94a0ea0 0x5628b94aed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7beffd640 0 client.15744 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e8d8d740 1 --2- 172.21.15.31:0/1714173765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5628b94a0580 0x5628b94b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/1714173765 >> 172.21.15.31:0/1714173765 conn(0x5628b949c8a0 msgr2=0x5628b949bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/1714173765 shutdown_connections 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e8d8d740 1 -- 172.21.15.31:0/1714173765 wait complete. 2024-04-25T01:55:07.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e8d8d740 20 client.15744 trim_cache size 0 max 16384 2024-04-25T01:55:07.277 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:55:07.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1579513786 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa7c80a2e20 msgr2=0x7fa7c80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa7c80a2e20 0x7fa7c80a52e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0000c00 tx=0x7fa7d00058b0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1579513786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e006a780 msgr2=0x7fa7e019a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e006a780 0x7fa7e019a690 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4005fd0 tx=0x7fa7d400dde0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1579513786 shutdown_connections 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa7c80a2e20 0x7fa7c80a52e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e0107730 0x7fa7e019ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7e006a780 0x7fa7e019a690 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 --2- 172.21.15.31:0/1579513786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7e0069dc0 0x7fa7e019a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1579513786 >> 172.21.15.31:0/1579513786 conn(0x7fa7e00776a0 msgr2=0x7fa7e010cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1579513786 shutdown_connections 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:07.269+0000 7fa7e5fa5640 1 -- 172.21.15.31:0/1579513786 wait complete. 2024-04-25T01:55:07.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:07 smithi134 ceph-mon[33098]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-04-25T01:55:07.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:07 smithi031 ceph-mon[29204]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-04-25T01:55:08.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:07 smithi040 ceph-mon[32418]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-04-25T01:55:08.282 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:55:08.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:08.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:55:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.394+0000 7f49255ee640 1 -- 172.21.15.31:0/1769407983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49201066b0 msgr2=0x7f4920106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.394+0000 7f49255ee640 1 --2- 172.21.15.31:0/1769407983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49201066b0 0x7f4920106a90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f4918009930 tx=0x7f491802a420 comp rx=0 tx=0).stop 2024-04-25T01:55:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.394+0000 7f49255ee640 1 -- 172.21.15.31:0/1769407983 shutdown_connections 2024-04-25T01:55:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.394+0000 7f49255ee640 1 --2- 172.21.15.31:0/1769407983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4920100fd0 0x7f492010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.394+0000 7f49255ee640 1 --2- 172.21.15.31:0/1769407983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4920100610 0x7f4920100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.395+0000 7f49255ee640 1 --2- 172.21.15.31:0/1769407983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49201066b0 0x7f4920106a90 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.395+0000 7f49255ee640 1 -- 172.21.15.31:0/1769407983 >> 172.21.15.31:0/1769407983 conn(0x7f4920077740 msgr2=0x7f4920077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.395+0000 7f49255ee640 1 -- 172.21.15.31:0/1769407983 shutdown_connections 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.395+0000 7f49255ee640 1 -- 172.21.15.31:0/1769407983 wait complete. 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.395+0000 7f49255ee640 1 Processor -- start 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.395+0000 7f49255ee640 1 -- start start 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f49255ee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4920100610 0x7f4920197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f49255ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4920100fd0 0x7f49201983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f49255ee640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49201066b0 0x7f492019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f49255ee640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49201119e0 con 0x7f4920100610 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f49255ee640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4920111860 con 0x7f4920100fd0 2024-04-25T01:55:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f49255ee640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4920111b60 con 0x7f49201066b0 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f491effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4920100610 0x7f4920197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f491effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4920100610 0x7f4920197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57958/0 (socket says 172.21.15.31:57958) 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f491effd640 1 -- 172.21.15.31:0/4063928138 learned_addr learned my addr 172.21.15.31:0/4063928138 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f491f7fe640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49201066b0 0x7f492019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f491e7fc640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4920100fd0 0x7f49201983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f491effd640 1 -- 172.21.15.31:0/4063928138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49201066b0 msgr2=0x7f492019c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f491effd640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49201066b0 0x7f492019c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f491effd640 1 -- 172.21.15.31:0/4063928138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4920100fd0 msgr2=0x7f49201983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f491effd640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4920100fd0 0x7f49201983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f491effd640 1 -- 172.21.15.31:0/4063928138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492019ce70 con 0x7f4920100610 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.396+0000 7f491effd640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4920100610 0x7f4920197e80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4918031440 tx=0x7f4918004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.397+0000 7f4917fff640 1 -- 172.21.15.31:0/4063928138 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491803f030 con 0x7f4920100610 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.397+0000 7f49255ee640 1 -- 172.21.15.31:0/4063928138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f492019d100 con 0x7f4920100610 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.397+0000 7f4917fff640 1 -- 172.21.15.31:0/4063928138 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4918003aa0 con 0x7f4920100610 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.397+0000 7f49255ee640 1 -- 172.21.15.31:0/4063928138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49201a49f0 con 0x7f4920100610 2024-04-25T01:55:08.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.397+0000 7f4917fff640 1 -- 172.21.15.31:0/4063928138 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4918049060 con 0x7f4920100610 2024-04-25T01:55:08.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.399+0000 7f4917fff640 1 -- 172.21.15.31:0/4063928138 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f4918042030 con 0x7f4920100610 2024-04-25T01:55:08.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.399+0000 7f49283d8740 1 Processor -- start 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.399+0000 7f49283d8740 1 -- start start 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f4917fff640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f490c0a2e20 0x7f490c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f49283d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d7183fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f49283d8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d71840710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f49283d8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d7184f420 0x555d71851810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f49283d8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555d717a42c0 con 0x555d71840290 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f49283d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d7174c910 con 0x555d71841980 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f49283d8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555d717a0280 con 0x555d7184f420 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f4917fff640 1 -- 172.21.15.31:0/4063928138 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f49180eff30 con 0x7f4920100610 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491f7fe640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f490c0a2e20 0x7f490c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491e7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d71840710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491e7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d71840710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57966/0 (socket says 172.21.15.31:57966) 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491e7fc640 1 -- 172.21.15.31:0/2163453235 learned_addr learned my addr 172.21.15.31:0/2163453235 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491f7fe640 1 --2- 172.21.15.31:0/2163453235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d7184f420 0x555d71851810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491effd640 1 --2- 172.21.15.31:0/2163453235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d7183fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491e7fc640 1 -- 172.21.15.31:0/2163453235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d7184f420 msgr2=0x555d71851810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491e7fc640 1 --2- 172.21.15.31:0/2163453235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d7184f420 0x555d71851810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491e7fc640 1 -- 172.21.15.31:0/2163453235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 msgr2=0x555d7183fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491e7fc640 1 --2- 172.21.15.31:0/2163453235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d7183fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491e7fc640 1 -- 172.21.15.31:0/2163453235 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d7175dfb0 con 0x555d71840290 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491f7fe640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f490c0a2e20 0x7f490c0a52e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4910005fd0 tx=0x7f4910004410 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f491e7fc640 1 --2- 172.21.15.31:0/2163453235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d71840710 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f490800aca0 tx=0x7f490800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474a4d75211e928b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.400+0000 7f49157fa640 1 -- 172.21.15.31:0/2163453235 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490801d070 con 0x555d71840290 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.401+0000 7f49157fa640 1 -- 172.21.15.31:0/2163453235 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4908018af0 con 0x555d71840290 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.401+0000 7f49157fa640 1 -- 172.21.15.31:0/2163453235 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908010a60 con 0x555d71840290 2024-04-25T01:55:08.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.401+0000 7f49283d8740 1 -- 172.21.15.31:0/2163453235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 msgr2=0x555d71840710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.401+0000 7f49283d8740 1 --2- 172.21.15.31:0/2163453235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d71840710 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f490800aca0 tx=0x7f490800f790 comp rx=0 tx=0).stop 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.401+0000 7f49283d8740 1 -- 172.21.15.31:0/2163453235 shutdown_connections 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.401+0000 7f49283d8740 1 --2- 172.21.15.31:0/2163453235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d7184f420 0x555d71851810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.401+0000 7f49283d8740 1 --2- 172.21.15.31:0/2163453235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d71840710 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.401+0000 7f49283d8740 1 --2- 172.21.15.31:0/2163453235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d7183fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.401+0000 7f49283d8740 1 -- 172.21.15.31:0/2163453235 >> 172.21.15.31:0/2163453235 conn(0x555d71808950 msgr2=0x555d7183c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.401+0000 7f49283d8740 1 -- 172.21.15.31:0/2163453235 shutdown_connections 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.401+0000 7f49283d8740 1 -- 172.21.15.31:0/2163453235 wait complete. 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f49283d8740 1 -- start start 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f49283d8740 1 Processor -- start 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f49283d8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d71840290 0x555d7184c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f49283d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d7184c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f49283d8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d7184cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f49283d8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555d718522b0 con 0x555d7184f420 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f49283d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d7175fe90 con 0x555d71841980 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f49283d8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555d71852430 con 0x555d71840290 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f49283d8740 10 client.0 _ll_register_callbacks cb 0x555d7179b970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f491f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d7184cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f491f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d7184cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57982/0 (socket says 172.21.15.31:57982) 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.402+0000 7f491f7fe640 1 -- 172.21.15.31:0/2903359461 learned_addr learned my addr 172.21.15.31:0/2903359461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f491e7fc640 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d7184c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f491f7fe640 1 -- 172.21.15.31:0/2903359461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d71840290 msgr2=0x555d7184c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f491f7fe640 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d71840290 0x555d7184c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f491f7fe640 1 -- 172.21.15.31:0/2903359461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 msgr2=0x555d7184c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f491f7fe640 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d7184c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f491f7fe640 1 -- 172.21.15.31:0/2903359461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d715807e0 con 0x555d7184f420 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f491e7fc640 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d7184c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f491f7fe640 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d7184cd60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4910009e40 tx=0x7f4910004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff18001d0516f97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f49067fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f49067fc640 1 -- 172.21.15.31:0/2903359461 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491000fd50 con 0x555d7184f420 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f49067fc640 1 -- 172.21.15.31:0/2903359461 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4910027400 con 0x555d7184f420 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f49283d8740 1 -- 172.21.15.31:0/2903359461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4908018af0 con 0x555d7184f420 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f49067fc640 1 -- 172.21.15.31:0/2903359461 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4910012e10 con 0x555d7184f420 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f49283d8740 10 client.15756 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f49283d8740 1 -- 172.21.15.31:0/2903359461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555d715e0f70 con 0x555d7184f420 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f49067fc640 1 -- 172.21.15.31:0/2903359461 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4908018af0 con 0x555d7184f420 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.403+0000 7f49067fc640 1 -- 172.21.15.31:0/2903359461 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f49100305d0 con 0x555d7184f420 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.404+0000 7f49283d8740 10 client.15756 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.404+0000 7f49283d8740 10 client.15756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.404+0000 7f49283d8740 20 client.15756 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.404+0000 7f49283d8740 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d7185f470 0x555d71861930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.404+0000 7f49283d8740 4 client.15756 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:08.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.404+0000 7f49283d8740 1 -- 172.21.15.31:0/2903359461 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d717785a0 con 0x555d7185f470 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.404+0000 7f491effd640 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d7185f470 0x555d71861930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.404+0000 7f491effd640 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d7185f470 0x555d71861930 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e39a0c581edc5509 server_cookie=906b46f22452b7a2 in_seq=0 out_seq=0 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.404+0000 7f49067fc640 10 client.15756 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49067fc640 1 -- 172.21.15.31:0/2903359461 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d717785a0 con 0x555d7185f470 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49067fc640 10 client.15756 handle_command_reply: tid=0 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49283d8740 1 client.15756 shutdown 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49283d8740 1 -- 172.21.15.31:0/2903359461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 msgr2=0x555d7184cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49283d8740 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d7184cd60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4910009e40 tx=0x7f4910004ab0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49283d8740 1 -- 172.21.15.31:0/2903359461 shutdown_connections 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49283d8740 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d7185f470 0x555d71861930 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49283d8740 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d7184cd60 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49067fc640 0 client.15756 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49283d8740 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d7184c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49283d8740 1 --2- 172.21.15.31:0/2903359461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d71840290 0x555d7184c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49283d8740 1 -- 172.21.15.31:0/2903359461 >> 172.21.15.31:0/2903359461 conn(0x555d71808950 msgr2=0x555d71842050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49283d8740 1 -- 172.21.15.31:0/2903359461 shutdown_connections 2024-04-25T01:55:08.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.405+0000 7f49283d8740 1 -- 172.21.15.31:0/2903359461 wait complete. 2024-04-25T01:55:08.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.406+0000 7f49283d8740 20 client.15756 trim_cache size 0 max 16384 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.423+0000 7f49283d8740 1 Processor -- start 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.423+0000 7f49283d8740 1 -- start start 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.423+0000 7f49283d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71840290 0x555d7184b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f49283d8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d71841980 0x555d7185b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f49283d8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d71867200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f49283d8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49201aafa0 con 0x555d7184f420 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f49283d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f49201acae0 con 0x555d71840290 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f49283d8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f49201ab670 con 0x555d71841980 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d71867200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d71867200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57996/0 (socket says 172.21.15.31:57996) 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491f7fe640 1 -- 172.21.15.31:0/1524900322 learned_addr learned my addr 172.21.15.31:0/1524900322 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:08.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491e7fc640 1 --2- 172.21.15.31:0/1524900322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d71841980 0x555d7185b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491f7fe640 1 -- 172.21.15.31:0/1524900322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d71841980 msgr2=0x555d7185b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491f7fe640 1 --2- 172.21.15.31:0/1524900322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d71841980 0x555d7185b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491f7fe640 1 -- 172.21.15.31:0/1524900322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71840290 msgr2=0x555d7184b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491effd640 1 --2- 172.21.15.31:0/1524900322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71840290 0x555d7184b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491f7fe640 1 --2- 172.21.15.31:0/1524900322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71840290 0x555d7184b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491f7fe640 1 -- 172.21.15.31:0/1524900322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d71478170 con 0x555d7184f420 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491e7fc640 1 --2- 172.21.15.31:0/1524900322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d71841980 0x555d7185b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f491f7fe640 1 --2- 172.21.15.31:0/1524900322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d71867200 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4910003bb0 tx=0x7f4910004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c9626b0795e60c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f4914ff9640 1 -- 172.21.15.31:0/1524900322 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4910030dc0 con 0x555d7184f420 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f4914ff9640 1 -- 172.21.15.31:0/1524900322 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4910030430 con 0x555d7184f420 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.424+0000 7f4914ff9640 1 -- 172.21.15.31:0/1524900322 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4910039a40 con 0x555d7184f420 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.425+0000 7f49283d8740 1 -- 172.21.15.31:0/1524900322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 msgr2=0x555d71867200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.425+0000 7f49283d8740 1 --2- 172.21.15.31:0/1524900322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d71867200 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4910003bb0 tx=0x7f4910004990 comp rx=0 tx=0).stop 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.425+0000 7f49283d8740 1 -- 172.21.15.31:0/1524900322 shutdown_connections 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.425+0000 7f49283d8740 1 --2- 172.21.15.31:0/1524900322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7184f420 0x555d71867200 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.425+0000 7f49283d8740 1 --2- 172.21.15.31:0/1524900322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d71841980 0x555d7185b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.425+0000 7f49283d8740 1 --2- 172.21.15.31:0/1524900322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71840290 0x555d7184b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.425+0000 7f49283d8740 1 -- 172.21.15.31:0/1524900322 >> 172.21.15.31:0/1524900322 conn(0x555d71808950 msgr2=0x555d7183b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.425+0000 7f49283d8740 1 -- 172.21.15.31:0/1524900322 shutdown_connections 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.425+0000 7f49283d8740 1 -- 172.21.15.31:0/1524900322 wait complete. 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.425+0000 7f49283d8740 1 -- start start 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.425+0000 7f49283d8740 1 Processor -- start 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f49283d8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d718578e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f49283d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d71848570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f49283d8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d7184f420 0x555d71848ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f49283d8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49201ac410 con 0x555d71840290 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f49283d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f49201abd40 con 0x555d71841980 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f49283d8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f49201a8f80 con 0x555d7184f420 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f49283d8740 10 client.0 _ll_register_callbacks cb 0x7f4910003a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d718578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d718578e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58010/0 (socket says 172.21.15.31:58010) 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491effd640 1 -- 172.21.15.31:0/1163174651 learned_addr learned my addr 172.21.15.31:0/1163174651 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:08.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491f7fe640 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d7184f420 0x555d71848ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491e7fc640 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d71848570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491effd640 1 -- 172.21.15.31:0/1163174651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d7184f420 msgr2=0x555d71848ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491effd640 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d7184f420 0x555d71848ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491effd640 1 -- 172.21.15.31:0/1163174651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 msgr2=0x555d71848570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491effd640 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d71848570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491effd640 1 -- 172.21.15.31:0/1163174651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d71849020 con 0x555d71840290 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491f7fe640 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d7184f420 0x555d71848ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f491effd640 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d718578e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f49180b8780 tx=0x7f491804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94aa0aa0fc99d993 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f49067fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f49067fc640 1 -- 172.21.15.31:0/1163174651 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4918035070 con 0x555d71840290 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f49067fc640 1 -- 172.21.15.31:0/1163174651 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f49180b8990 con 0x555d71840290 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.426+0000 7f49283d8740 1 -- 172.21.15.31:0/1163174651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4910030430 con 0x555d71840290 2024-04-25T01:55:08.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f49067fc640 1 -- 172.21.15.31:0/1163174651 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49180bca20 con 0x555d71840290 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f49283d8740 10 client.15762 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f49283d8740 1 -- 172.21.15.31:0/1163174651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555d7184a310 con 0x555d71840290 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f49067fc640 1 -- 172.21.15.31:0/1163174651 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4910030430 con 0x555d71840290 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f49067fc640 1 -- 172.21.15.31:0/1163174651 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f491803bca0 con 0x555d71840290 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f49283d8740 10 client.15762 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f49283d8740 10 client.15762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f49283d8740 20 client.15762 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f49283d8740 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d718c6f20 0x555d718c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f49283d8740 4 client.15762 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f49283d8740 1 -- 172.21.15.31:0/1163174651 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555d717b1800 con 0x555d718c6f20 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.427+0000 7f491e7fc640 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d718c6f20 0x555d718c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.428+0000 7f491e7fc640 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d718c6f20 0x555d718c9310 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cff6a4715375031e server_cookie=1e221cfa2ab0f079 in_seq=0 out_seq=0 2024-04-25T01:55:08.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.428+0000 7f49067fc640 10 client.15762 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:08.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49067fc640 1 -- 172.21.15.31:0/1163174651 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555d717b1800 con 0x555d718c6f20 2024-04-25T01:55:08.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49067fc640 10 client.15762 handle_command_reply: tid=0 2024-04-25T01:55:08.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 1 client.15762 shutdown 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 1 -- 172.21.15.31:0/1163174651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 msgr2=0x555d718578e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d718578e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f49180b8780 tx=0x7f491804c000 comp rx=0 tx=0).stop 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 1 -- 172.21.15.31:0/1163174651 shutdown_connections 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d718c6f20 0x555d718c9310 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d7184f420 0x555d71848ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d71841980 0x555d71848570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 1 --2- 172.21.15.31:0/1163174651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d71840290 0x555d718578e0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49067fc640 0 client.15762 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 1 -- 172.21.15.31:0/1163174651 >> 172.21.15.31:0/1163174651 conn(0x555d71808950 msgr2=0x555d71841d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 1 -- 172.21.15.31:0/1163174651 shutdown_connections 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 1 -- 172.21.15.31:0/1163174651 wait complete. 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.429+0000 7f49283d8740 20 client.15762 trim_cache size 0 max 16384 2024-04-25T01:55:08.436 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:55:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T01:55:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "8905e97f-ebef-44fe-9a8f-6ecaed8e5000", 2024-04-25T01:55:08.437 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T01:55:08.437 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:55:08.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 -- 172.21.15.31:0/4063928138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f490c0a2e20 msgr2=0x7f490c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f490c0a2e20 0x7f490c0a52e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4910005fd0 tx=0x7f4910004410 comp rx=0 tx=0).stop 2024-04-25T01:55:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 -- 172.21.15.31:0/4063928138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4920100610 msgr2=0x7f4920197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4920100610 0x7f4920197e80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4918031440 tx=0x7f4918004270 comp rx=0 tx=0).stop 2024-04-25T01:55:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 -- 172.21.15.31:0/4063928138 shutdown_connections 2024-04-25T01:55:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f490c0a2e20 0x7f490c0a52e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49201066b0 0x7f492019c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4920100fd0 0x7f49201983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 --2- 172.21.15.31:0/4063928138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4920100610 0x7f4920197e80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 -- 172.21.15.31:0/4063928138 >> 172.21.15.31:0/4063928138 conn(0x7f4920077740 msgr2=0x7f49200ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 -- 172.21.15.31:0/4063928138 shutdown_connections 2024-04-25T01:55:08.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.430+0000 7f49255ee640 1 -- 172.21.15.31:0/4063928138 wait complete. 2024-04-25T01:55:08.442 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:55:08.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:55:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.545+0000 7f9bb97ed640 1 -- 172.21.15.31:0/359553114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb4069dc0 msgr2=0x7f9bb406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.545+0000 7f9bb97ed640 1 --2- 172.21.15.31:0/359553114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb4069dc0 0x7f9bb406a240 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009930 tx=0x7f9ba802a440 comp rx=0 tx=0).stop 2024-04-25T01:55:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.545+0000 7f9bb97ed640 1 -- 172.21.15.31:0/359553114 shutdown_connections 2024-04-25T01:55:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.545+0000 7f9bb97ed640 1 --2- 172.21.15.31:0/359553114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bb406a780 0x7f9bb410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.545+0000 7f9bb97ed640 1 --2- 172.21.15.31:0/359553114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb4069dc0 0x7f9bb406a240 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.545+0000 7f9bb97ed640 1 --2- 172.21.15.31:0/359553114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bb4107730 0x7f9bb4107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.545+0000 7f9bb97ed640 1 -- 172.21.15.31:0/359553114 >> 172.21.15.31:0/359553114 conn(0x7f9bb40776a0 msgr2=0x7f9bb4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.545+0000 7f9bb97ed640 1 -- 172.21.15.31:0/359553114 shutdown_connections 2024-04-25T01:55:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.545+0000 7f9bb97ed640 1 -- 172.21.15.31:0/359553114 wait complete. 2024-04-25T01:55:08.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.545+0000 7f9bb97ed640 1 Processor -- start 2024-04-25T01:55:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb97ed640 1 -- start start 2024-04-25T01:55:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb97ed640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb4069dc0 0x7f9bb4197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb97ed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bb406a780 0x7f9bb4198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb97ed640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bb4107730 0x7f9bb419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb97ed640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41119b0 con 0x7f9bb4069dc0 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb97ed640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb4111830 con 0x7f9bb406a780 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb97ed640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb4111b30 con 0x7f9bb4107730 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb4069dc0 0x7f9bb4197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb4069dc0 0x7f9bb4197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58028/0 (socket says 172.21.15.31:58028) 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb2ffd640 1 -- 172.21.15.31:0/1561925696 learned_addr learned my addr 172.21.15.31:0/1561925696 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb27fc640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bb406a780 0x7f9bb4198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb2ffd640 1 -- 172.21.15.31:0/1561925696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bb4107730 msgr2=0x7f9bb419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bb4107730 0x7f9bb419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb2ffd640 1 -- 172.21.15.31:0/1561925696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bb406a780 msgr2=0x7f9bb4198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bb406a780 0x7f9bb4198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.546+0000 7f9bb2ffd640 1 -- 172.21.15.31:0/1561925696 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb419ce20 con 0x7f9bb4069dc0 2024-04-25T01:55:08.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.547+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb4069dc0 0x7f9bb4197e30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400c920 tx=0x7f9ba400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.547+0000 7f9b9bfff640 1 -- 172.21.15.31:0/1561925696 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4007c70 con 0x7f9bb4069dc0 2024-04-25T01:55:08.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.547+0000 7f9b9bfff640 1 -- 172.21.15.31:0/1561925696 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ba4007e10 con 0x7f9bb4069dc0 2024-04-25T01:55:08.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.547+0000 7f9bb97ed640 1 -- 172.21.15.31:0/1561925696 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb419d110 con 0x7f9bb4069dc0 2024-04-25T01:55:08.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.547+0000 7f9b9bfff640 1 -- 172.21.15.31:0/1561925696 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4018820 con 0x7f9bb4069dc0 2024-04-25T01:55:08.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.547+0000 7f9bb97ed640 1 -- 172.21.15.31:0/1561925696 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb41a49a0 con 0x7f9bb4069dc0 2024-04-25T01:55:08.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9b9bfff640 1 -- 172.21.15.31:0/1561925696 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f9ba40189c0 con 0x7f9bb4069dc0 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bbc5cf740 1 Processor -- start 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bbc5cf740 1 -- start start 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa52402a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa5240d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5240d620 0x55fa5240fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fa523676b0 con 0x55fa52402f50 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fa5230f880 con 0x55fa5240d620 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fa52363670 con 0x55fa52402630 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bb2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa5240d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bb2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa5240d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58032/0 (socket says 172.21.15.31:58032) 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bb2ffd640 1 -- 172.21.15.31:0/237840978 learned_addr learned my addr 172.21.15.31:0/237840978 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bb37fe640 1 --2- 172.21.15.31:0/237840978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa52402a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.550+0000 7f9bb27fc640 1 --2- 172.21.15.31:0/237840978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5240d620 0x55fa5240fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9b9bfff640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9b9c0a2ea0 0x7f9b9c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9bb2ffd640 1 -- 172.21.15.31:0/237840978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 msgr2=0x55fa52402a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/237840978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa52402a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9bb2ffd640 1 -- 172.21.15.31:0/237840978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5240d620 msgr2=0x55fa5240fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/237840978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5240d620 0x55fa5240fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9bb2ffd640 1 -- 172.21.15.31:0/237840978 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa521457d0 con 0x55fa52402f50 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9bb37fe640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9b9c0a2ea0 0x7f9b9c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9b9bfff640 1 -- 172.21.15.31:0/1561925696 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9ba40cf700 con 0x7f9bb4069dc0 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/237840978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa5240d0e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40cce30 tx=0x7f9ba40148a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b404d4d4826651a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9b997fa640 1 -- 172.21.15.31:0/237840978 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba409b380 con 0x55fa52402f50 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9bb37fe640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9b9c0a2ea0 0x7f9b9c0a5360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9bac005fd0 tx=0x7f9bac004a40 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9b997fa640 1 -- 172.21.15.31:0/237840978 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ba400fb00 con 0x55fa52402f50 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.551+0000 7f9b997fa640 1 -- 172.21.15.31:0/237840978 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4018cb0 con 0x55fa52402f50 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.552+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/237840978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 msgr2=0x55fa5240d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.552+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/237840978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa5240d0e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40cce30 tx=0x7f9ba40148a0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.552+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/237840978 shutdown_connections 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.552+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/237840978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5240d620 0x55fa5240fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.552+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/237840978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa5240d0e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.552+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/237840978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa52402a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.552+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/237840978 >> 172.21.15.31:0/237840978 conn(0x55fa523fe950 msgr2=0x55fa523fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.552+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/237840978 shutdown_connections 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.552+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/237840978 wait complete. 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.552+0000 7f9bbc5cf740 1 -- start start 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.552+0000 7f9bbc5cf740 1 Processor -- start 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa5240a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa52402f50 0x55fa5240aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5240d620 0x55fa5240afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fa524103b0 con 0x55fa5240d620 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fa52322f50 con 0x55fa52402f50 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fa52410530 con 0x55fa52402630 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bbc5cf740 10 client.0 _ll_register_callbacks cb 0x55fa5235e000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5240d620 0x55fa5240afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb2ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa5240a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5240d620 0x55fa5240afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58034/0 (socket says 172.21.15.31:58034) 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb37fe640 1 -- 172.21.15.31:0/276235834 learned_addr learned my addr 172.21.15.31:0/276235834 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb27fc640 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa52402f50 0x55fa5240aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb2ffd640 1 -- 172.21.15.31:0/276235834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa52402f50 msgr2=0x55fa5240aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa52402f50 0x55fa5240aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb2ffd640 1 -- 172.21.15.31:0/276235834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5240d620 msgr2=0x55fa5240afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5240d620 0x55fa5240afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb2ffd640 1 -- 172.21.15.31:0/276235834 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa5207b3f0 con 0x55fa52402630 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb37fe640 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5240d620 0x55fa5240afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa5240a550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40983f0 tx=0x7f9ba4099f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1ea0cfa124f6822 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.553+0000 7f9b96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9b96ff5640 1 -- 172.21.15.31:0/276235834 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4018cb0 con 0x55fa52402630 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9b96ff5640 1 -- 172.21.15.31:0/276235834 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9ba40d5170 con 0x55fa52402630 2024-04-25T01:55:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/276235834 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fa52374bf0 con 0x55fa52402630 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9b96ff5640 1 -- 172.21.15.31:0/276235834 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9ba409b380 con 0x55fa52402630 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9bbc5cf740 10 client.25318 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/276235834 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fa5203b0f0 con 0x55fa52402630 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9b96ff5640 1 -- 172.21.15.31:0/276235834 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55fa52374bf0 con 0x55fa52402630 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9b96ff5640 1 -- 172.21.15.31:0/276235834 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9ba40a4200 con 0x55fa52402630 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9bbc5cf740 10 client.25318 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9bbc5cf740 10 client.25318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9bbc5cf740 20 client.25318 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa5241d6d0 0x55fa5241fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9bbc5cf740 4 client.25318 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/276235834 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa5236ef40 con 0x55fa5241d6d0 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.554+0000 7f9bb27fc640 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa5241d6d0 0x55fa5241fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.555+0000 7f9bb27fc640 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa5241d6d0 0x55fa5241fac0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42eeabc1fae33755 server_cookie=249507f4d62712ff in_seq=0 out_seq=0 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.555+0000 7f9b96ff5640 10 client.25318 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.555+0000 7f9b96ff5640 1 -- 172.21.15.31:0/276235834 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa5236ef40 con 0x55fa5241d6d0 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.555+0000 7f9b96ff5640 10 client.25318 handle_command_reply: tid=0 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 1 client.25318 shutdown 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/276235834 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 msgr2=0x55fa5240a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa5240a550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40983f0 tx=0x7f9ba4099f30 comp rx=0 tx=0).stop 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/276235834 shutdown_connections 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa5241d6d0 0x55fa5241fac0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5240d620 0x55fa5240afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9b96ff5640 0 client.25318 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa52402f50 0x55fa5240aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/276235834 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa5240a550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/276235834 >> 172.21.15.31:0/276235834 conn(0x55fa523fe950 msgr2=0x55fa52404750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/276235834 shutdown_connections 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/276235834 wait complete. 2024-04-25T01:55:08.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.556+0000 7f9bbc5cf740 20 client.25318 trim_cache size 0 max 16384 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bbc5cf740 1 Processor -- start 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bbc5cf740 1 -- start start 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa52419270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52404d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5240d620 0x55fa524250f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41ad1c0 con 0x55fa52402f50 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41aed00 con 0x55fa5240d620 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41ad890 con 0x55fa52402630 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bb27fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52404d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bb27fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52404d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58048/0 (socket says 172.21.15.31:58048) 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bb27fc640 1 -- 172.21.15.31:0/1775441115 learned_addr learned my addr 172.21.15.31:0/1775441115 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bb27fc640 1 -- 172.21.15.31:0/1775441115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 msgr2=0x55fa52419270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bb37fe640 1 --2- 172.21.15.31:0/1775441115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5240d620 0x55fa524250f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/1775441115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa52419270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bb27fc640 1 --2- 172.21.15.31:0/1775441115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa52419270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bb27fc640 1 -- 172.21.15.31:0/1775441115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5240d620 msgr2=0x55fa524250f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bb27fc640 1 --2- 172.21.15.31:0/1775441115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5240d620 0x55fa524250f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.575+0000 7f9bb27fc640 1 -- 172.21.15.31:0/1775441115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa52403450 con 0x55fa52402f50 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9bb37fe640 1 --2- 172.21.15.31:0/1775441115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5240d620 0x55fa524250f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9bb27fc640 1 --2- 172.21.15.31:0/1775441115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52404d90 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009930 tx=0x7f9ba8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=789c70156a952aad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9b98ff9640 1 -- 172.21.15.31:0/1775441115 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8035040 con 0x55fa52402f50 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9b98ff9640 1 -- 172.21.15.31:0/1775441115 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ba8049870 con 0x55fa52402f50 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9b98ff9640 1 -- 172.21.15.31:0/1775441115 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba80047a0 con 0x55fa52402f50 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/1775441115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 msgr2=0x55fa52404d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/1775441115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52404d90 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009930 tx=0x7f9ba8004040 comp rx=0 tx=0).stop 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/1775441115 shutdown_connections 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/1775441115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5240d620 0x55fa524250f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/1775441115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52404d90 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/1775441115 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa52402630 0x55fa52419270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/1775441115 >> 172.21.15.31:0/1775441115 conn(0x55fa523fe950 msgr2=0x55fa523fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/1775441115 shutdown_connections 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.576+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/1775441115 wait complete. 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bbc5cf740 1 -- start start 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bbc5cf740 1 Processor -- start 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa52402630 0x55fa524159a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52410d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bbc5cf740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5240d620 0x55fa52411280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41ae630 con 0x55fa52402f50 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41adf60 con 0x55fa52402630 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bbc5cf740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb41ab1a0 con 0x55fa5240d620 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bbc5cf740 10 client.0 _ll_register_callbacks cb 0x7f9ba8005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bb27fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52410d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bb27fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52410d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58060/0 (socket says 172.21.15.31:58060) 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bb27fc640 1 -- 172.21.15.31:0/3110945289 learned_addr learned my addr 172.21.15.31:0/3110945289 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bb37fe640 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5240d620 0x55fa52411280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa52402630 0x55fa524159a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bb27fc640 1 -- 172.21.15.31:0/3110945289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5240d620 msgr2=0x55fa52411280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bb27fc640 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5240d620 0x55fa52411280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bb27fc640 1 -- 172.21.15.31:0/3110945289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa52402630 msgr2=0x55fa524159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bb27fc640 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa52402630 0x55fa524159a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bb27fc640 1 -- 172.21.15.31:0/3110945289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa524118c0 con 0x55fa52402f50 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.577+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa52402630 0x55fa524159a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9bb27fc640 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52410d40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9ba802a960 tx=0x7f9ba8005360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca4b0baee0e2975 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9b96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9b96ff5640 1 -- 172.21.15.31:0/3110945289 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8057060 con 0x55fa52402f50 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9b96ff5640 1 -- 172.21.15.31:0/3110945289 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9ba8004270 con 0x55fa52402f50 2024-04-25T01:55:08.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/3110945289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ba8049870 con 0x55fa52402f50 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9b96ff5640 1 -- 172.21.15.31:0/3110945289 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9ba8035040 con 0x55fa52402f50 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9bbc5cf740 10 client.15780 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/3110945289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fa52487b60 con 0x55fa52402f50 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9b96ff5640 1 -- 172.21.15.31:0/3110945289 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ba8049870 con 0x55fa52402f50 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9b96ff5640 1 -- 172.21.15.31:0/3110945289 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9ba80035f0 con 0x55fa52402f50 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9bbc5cf740 10 client.15780 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9bbc5cf740 10 client.15780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9bbc5cf740 20 client.15780 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa52488290 0x55fa5248a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9bbc5cf740 4 client.15780 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.578+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/3110945289 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fa522ec6f0 con 0x55fa52488290 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa52488290 0x55fa5248a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9b96ff5640 10 client.15780 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bb2ffd640 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa52488290 0x55fa5248a6b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c578e5fdf2a6d078 server_cookie=6bd96ae16e8c008e in_seq=0 out_seq=0 2024-04-25T01:55:08.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9b96ff5640 1 -- 172.21.15.31:0/3110945289 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fa522ec6f0 con 0x55fa52488290 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9b96ff5640 10 client.15780 handle_command_reply: tid=0 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bbc5cf740 1 client.15780 shutdown 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/3110945289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 msgr2=0x55fa52410d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52410d40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9ba802a960 tx=0x7f9ba8005360 comp rx=0 tx=0).stop 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/3110945289 shutdown_connections 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa52488290 0x55fa5248a6b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5240d620 0x55fa52411280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa52402f50 0x55fa52410d40 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9b96ff5640 0 client.15780 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bbc5cf740 1 --2- 172.21.15.31:0/3110945289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa52402630 0x55fa524159a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/3110945289 >> 172.21.15.31:0/3110945289 conn(0x55fa523fe950 msgr2=0x55fa523fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/3110945289 shutdown_connections 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.579+0000 7f9bbc5cf740 1 -- 172.21.15.31:0/3110945289 wait complete. 2024-04-25T01:55:08.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bbc5cf740 20 client.15780 trim_cache size 0 max 16384 2024-04-25T01:55:08.588 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:55:08.588 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:55:08.588 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T01:55:08.588 INFO:teuthology.orchestra.run.smithi031.stdout: "8905e97f-ebef-44fe-9a8f-6ecaed8e5000": { 2024-04-25T01:55:08.588 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T01:55:08.588 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "8905e97f-ebef-44fe-9a8f-6ecaed8e5000", 2024-04-25T01:55:08.588 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T01:55:08.588 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T01:55:08.588 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T01:55:08.588 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:55:08.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 -- 172.21.15.31:0/1561925696 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9b9c0a2ea0 msgr2=0x7f9b9c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9b9c0a2ea0 0x7f9b9c0a5360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9bac005fd0 tx=0x7f9bac004a40 comp rx=0 tx=0).stop 2024-04-25T01:55:08.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 -- 172.21.15.31:0/1561925696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb4069dc0 msgr2=0x7f9bb4197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:08.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb4069dc0 0x7f9bb4197e30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400c920 tx=0x7f9ba400cde0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 -- 172.21.15.31:0/1561925696 shutdown_connections 2024-04-25T01:55:08.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9b9c0a2ea0 0x7f9b9c0a5360 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bb4107730 0x7f9bb419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bb406a780 0x7f9bb4198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 --2- 172.21.15.31:0/1561925696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bb4069dc0 0x7f9bb4197e30 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:08.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 -- 172.21.15.31:0/1561925696 >> 172.21.15.31:0/1561925696 conn(0x7f9bb40776a0 msgr2=0x7f9bb40fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:08.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 -- 172.21.15.31:0/1561925696 shutdown_connections 2024-04-25T01:55:08.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:08.580+0000 7f9bb97ed640 1 -- 172.21.15.31:0/1561925696 wait complete. 2024-04-25T01:55:08.593 INFO:tasks.cephfs.filesystem:scrub status for tag:8905e97f-ebef-44fe-9a8f-6ecaed8e5000 - {'path': '/', 'tag': '8905e97f-ebef-44fe-9a8f-6ecaed8e5000', 'options': 'recursive,force'} 2024-04-25T01:55:09.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:09 smithi134 ceph-mon[33098]: scrub queued for path: / 2024-04-25T01:55:09.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:09 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:09.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:09 smithi134 ceph-mon[33098]: scrub summary: active paths [/] 2024-04-25T01:55:09.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:09 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:09.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:09 smithi134 ceph-mon[33098]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-25T01:55:09.962 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:09 smithi040 ceph-mon[32418]: scrub queued for path: / 2024-04-25T01:55:09.963 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:09 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:09.963 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:09 smithi040 ceph-mon[32418]: scrub summary: active paths [/] 2024-04-25T01:55:09.963 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:09 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:09.963 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:09 smithi040 ceph-mon[32418]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-25T01:55:09.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:09 smithi031 ceph-mon[29204]: scrub queued for path: / 2024-04-25T01:55:09.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:09 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:09.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:09 smithi031 ceph-mon[29204]: scrub summary: active paths [/] 2024-04-25T01:55:09.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:09 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:09.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:09 smithi031 ceph-mon[29204]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-25T01:55:11.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:11 smithi134 ceph-mon[33098]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-25T01:55:11.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:11 smithi134 ceph-mon[33098]: scrub summary: idle 2024-04-25T01:55:11.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:11 smithi031 ceph-mon[29204]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-25T01:55:11.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:11 smithi031 ceph-mon[29204]: scrub summary: idle 2024-04-25T01:55:12.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:11 smithi040 ceph-mon[32418]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-25T01:55:12.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:11 smithi040 ceph-mon[32418]: scrub summary: idle 2024-04-25T01:55:13.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:13 smithi031 ceph-mon[29204]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-04-25T01:55:14.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:13 smithi040 ceph-mon[32418]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-04-25T01:55:14.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:13 smithi134 ceph-mon[33098]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-04-25T01:55:15.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:15 smithi031 ceph-mon[29204]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-04-25T01:55:16.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:15 smithi040 ceph-mon[32418]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-04-25T01:55:16.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:15 smithi134 ceph-mon[33098]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-04-25T01:55:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.766+0000 7ff7e341c640 1 -- 172.21.15.31:0/1924087428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7dc101030 msgr2=0x7ff7dc10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.766+0000 7ff7e341c640 1 --2- 172.21.15.31:0/1924087428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7dc101030 0x7ff7dc10d4f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4009cd0 tx=0x7ff7d402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.766+0000 7ff7e341c640 1 -- 172.21.15.31:0/1924087428 shutdown_connections 2024-04-25T01:55:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.766+0000 7ff7e341c640 1 --2- 172.21.15.31:0/1924087428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7dc101030 0x7ff7dc10d4f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.766+0000 7ff7e341c640 1 --2- 172.21.15.31:0/1924087428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff7dc100670 0x7ff7dc100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.766+0000 7ff7e341c640 1 --2- 172.21.15.31:0/1924087428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7dc106630 0x7ff7dc106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.766+0000 7ff7e341c640 1 -- 172.21.15.31:0/1924087428 >> 172.21.15.31:0/1924087428 conn(0x7ff7dc0777c0 msgr2=0x7ff7dc077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.767+0000 7ff7e341c640 1 -- 172.21.15.31:0/1924087428 shutdown_connections 2024-04-25T01:55:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.767+0000 7ff7e341c640 1 -- 172.21.15.31:0/1924087428 wait complete. 2024-04-25T01:55:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.767+0000 7ff7e341c640 1 Processor -- start 2024-04-25T01:55:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.767+0000 7ff7e341c640 1 -- start start 2024-04-25T01:55:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.767+0000 7ff7e341c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7dc100670 0x7ff7dc19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e341c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff7dc101030 0x7ff7dc19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e341c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7dc106630 0x7ff7dc19ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e341c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1108f0 con 0x7ff7dc106630 2024-04-25T01:55:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e341c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc110770 con 0x7ff7dc100670 2024-04-25T01:55:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e341c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc110a70 con 0x7ff7dc101030 2024-04-25T01:55:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e19b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7dc106630 0x7ff7dc19ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e19b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7dc106630 0x7ff7dc19ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35402/0 (socket says 172.21.15.31:35402) 2024-04-25T01:55:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e19b9640 1 -- 172.21.15.31:0/1500096426 learned_addr learned my addr 172.21.15.31:0/1500096426 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff7dc101030 0x7ff7dc19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e11b8640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7dc100670 0x7ff7dc19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e19b9640 1 -- 172.21.15.31:0/1500096426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff7dc101030 msgr2=0x7ff7dc19a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e19b9640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff7dc101030 0x7ff7dc19a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e19b9640 1 -- 172.21.15.31:0/1500096426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7dc100670 msgr2=0x7ff7dc19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e19b9640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7dc100670 0x7ff7dc19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e19b9640 1 -- 172.21.15.31:0/1500096426 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7dc19f150 con 0x7ff7dc106630 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e19b9640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7dc106630 0x7ff7dc19ea20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4005cc0 tx=0x7ff7d4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7d27fc640 1 -- 172.21.15.31:0/1500096426 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4004450 con 0x7ff7dc106630 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7d27fc640 1 -- 172.21.15.31:0/1500096426 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7d4031c90 con 0x7ff7dc106630 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7d27fc640 1 -- 172.21.15.31:0/1500096426 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d403b400 con 0x7ff7dc106630 2024-04-25T01:55:16.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.768+0000 7ff7e341c640 1 -- 172.21.15.31:0/1500096426 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7dc19f3e0 con 0x7ff7dc106630 2024-04-25T01:55:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.769+0000 7ff7e341c640 1 -- 172.21.15.31:0/1500096426 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7dc1a6cd0 con 0x7ff7dc106630 2024-04-25T01:55:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.771+0000 7ff7d27fc640 1 -- 172.21.15.31:0/1500096426 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7ff7d4004680 con 0x7ff7dc106630 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e6207740 1 Processor -- start 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e6207740 1 -- start start 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7d27fc640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff7c80a2f40 0x7ff7c80a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cf5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5b60 0x55a690cffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690d00230 0x55a690d02620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7d27fc640 1 -- 172.21.15.31:0/1500096426 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7ff7d402e030 con 0x7ff7dc106630 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e6207740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a690c59cb0 con 0x55a690cf5240 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e6207740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a690c025b0 con 0x55a690cf5b60 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e6207740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a690c55c70 con 0x55a690d00230 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff7c80a2f40 0x7ff7c80a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e11b8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cf5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e11b8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cf5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35418/0 (socket says 172.21.15.31:35418) 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e11b8640 1 -- 172.21.15.31:0/44553961 learned_addr learned my addr 172.21.15.31:0/44553961 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e11b8640 1 -- 172.21.15.31:0/44553961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690d00230 msgr2=0x55a690d02620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e19b9640 1 --2- 172.21.15.31:0/44553961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690d00230 0x55a690d02620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/44553961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5b60 0x55a690cffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e11b8640 1 --2- 172.21.15.31:0/44553961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690d00230 0x55a690d02620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e11b8640 1 -- 172.21.15.31:0/44553961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5b60 msgr2=0x55a690cffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e11b8640 1 --2- 172.21.15.31:0/44553961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5b60 0x55a690cffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e11b8640 1 -- 172.21.15.31:0/44553961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a69096e210 con 0x55a690cf5240 2024-04-25T01:55:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.772+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/44553961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5b60 0x55a690cffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.773+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff7c80a2f40 0x7ff7c80a5400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc10cc70 tx=0x7ff7cc0041e0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.773+0000 7ff7e11b8640 1 --2- 172.21.15.31:0/44553961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cf5620 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff7d800aba0 tx=0x7ff7d800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e9d3ea910aa8a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.773+0000 7ff7c77fe640 1 -- 172.21.15.31:0/44553961 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d801d070 con 0x55a690cf5240 2024-04-25T01:55:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.773+0000 7ff7c77fe640 1 -- 172.21.15.31:0/44553961 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7d8018af0 con 0x55a690cf5240 2024-04-25T01:55:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.773+0000 7ff7c77fe640 1 -- 172.21.15.31:0/44553961 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d8010a60 con 0x55a690cf5240 2024-04-25T01:55:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.773+0000 7ff7e6207740 1 -- 172.21.15.31:0/44553961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 msgr2=0x55a690cf5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.773+0000 7ff7e6207740 1 --2- 172.21.15.31:0/44553961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cf5620 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff7d800aba0 tx=0x7ff7d800f790 comp rx=0 tx=0).stop 2024-04-25T01:55:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 -- 172.21.15.31:0/44553961 shutdown_connections 2024-04-25T01:55:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 --2- 172.21.15.31:0/44553961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690d00230 0x55a690d02620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 --2- 172.21.15.31:0/44553961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5b60 0x55a690cffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 --2- 172.21.15.31:0/44553961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cf5620 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 -- 172.21.15.31:0/44553961 >> 172.21.15.31:0/44553961 conn(0x55a690cf1560 msgr2=0x55a690cf1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 -- 172.21.15.31:0/44553961 shutdown_connections 2024-04-25T01:55:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 -- 172.21.15.31:0/44553961 wait complete. 2024-04-25T01:55:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 -- start start 2024-04-25T01:55:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 Processor -- start 2024-04-25T01:55:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cfd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690cf5b60 0x55a690cfd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690d00230 0x55a690cfdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a690d03020 con 0x55a690cf5240 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a690c15b50 con 0x55a690d00230 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a690d031a0 con 0x55a690cf5b60 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e6207740 10 client.0 _ll_register_callbacks cb 0x55a690c50ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e11b8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cfd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e11b8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cfd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35426/0 (socket says 172.21.15.31:35426) 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.774+0000 7ff7e11b8640 1 -- 172.21.15.31:0/2745436157 learned_addr learned my addr 172.21.15.31:0/2745436157 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690cf5b60 0x55a690cfd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7e19b9640 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690d00230 0x55a690cfdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7e11b8640 1 -- 172.21.15.31:0/2745436157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690cf5b60 msgr2=0x55a690cfd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7e11b8640 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690cf5b60 0x55a690cfd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7e11b8640 1 -- 172.21.15.31:0/2745436157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690d00230 msgr2=0x55a690cfdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7e11b8640 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690d00230 0x55a690cfdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7e11b8640 1 -- 172.21.15.31:0/2745436157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a690a3a620 con 0x55a690cf5240 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7e11b8640 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cfd130 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8010740 tx=0x7ff7d80186f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfd86e48752ab36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/2745436157 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d8027020 con 0x55a690cf5240 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/2745436157 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7ff7d8005040 con 0x55a690cf5240 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7e6207740 1 -- 172.21.15.31:0/2745436157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7d8018af0 con 0x55a690cf5240 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/2745436157 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff7d8032050 con 0x55a690cf5240 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7e6207740 10 client.15792 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7e6207740 1 -- 172.21.15.31:0/2745436157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a690d0f150 con 0x55a690cf5240 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.775+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/2745436157 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7d8018af0 con 0x55a690cf5240 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.776+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/2745436157 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff7d80046c0 con 0x55a690cf5240 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.776+0000 7ff7e6207740 10 client.15792 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.776+0000 7ff7e6207740 10 client.15792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.776+0000 7ff7e6207740 20 client.15792 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.776+0000 7ff7e6207740 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a690d105b0 0x55a690d129a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.776+0000 7ff7e6207740 4 client.15792 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.776+0000 7ff7e6207740 1 -- 172.21.15.31:0/2745436157 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a690c671f0 con 0x55a690d105b0 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.776+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a690d105b0 0x55a690d129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.777+0000 7ff7c4ff9640 10 client.15792 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.777+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a690d105b0 0x55a690d129a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1af9842b7ac2d465 server_cookie=f9969de1b8581b7b in_seq=0 out_seq=0 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.777+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/2745436157 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a690c671f0 con 0x55a690d105b0 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.777+0000 7ff7c4ff9640 10 client.15792 handle_command_reply: tid=0 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.777+0000 7ff7e6207740 1 client.15792 shutdown 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.777+0000 7ff7e6207740 1 -- 172.21.15.31:0/2745436157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 msgr2=0x55a690cfd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.777+0000 7ff7e6207740 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cfd130 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8010740 tx=0x7ff7d80186f0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.778+0000 7ff7e6207740 1 -- 172.21.15.31:0/2745436157 shutdown_connections 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.778+0000 7ff7e6207740 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a690d105b0 0x55a690d129a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.778+0000 7ff7e6207740 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690d00230 0x55a690cfdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.778+0000 7ff7e6207740 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690cf5b60 0x55a690cfd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.778+0000 7ff7c4ff9640 0 client.15792 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.778+0000 7ff7e6207740 1 --2- 172.21.15.31:0/2745436157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5240 0x55a690cfd130 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.778+0000 7ff7e6207740 1 -- 172.21.15.31:0/2745436157 >> 172.21.15.31:0/2745436157 conn(0x55a690cf1560 msgr2=0x55a690cf7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.778+0000 7ff7e6207740 1 -- 172.21.15.31:0/2745436157 shutdown_connections 2024-04-25T01:55:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.778+0000 7ff7e6207740 1 -- 172.21.15.31:0/2745436157 wait complete. 2024-04-25T01:55:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.778+0000 7ff7e6207740 20 client.15792 trim_cache size 0 max 16384 2024-04-25T01:55:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e6207740 1 Processor -- start 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e6207740 1 -- start start 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5240 0x55a690d0bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5b60 0x55a690cfc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690d00230 0x55a690d18000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e6207740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1af330 con 0x55a690cf5b60 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e6207740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1b0e70 con 0x55a690cf5240 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e6207740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1afa00 con 0x55a690d00230 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e09b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5b60 0x55a690cfc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e09b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5b60 0x55a690cfc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35434/0 (socket says 172.21.15.31:35434) 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e09b7640 1 -- 172.21.15.31:0/1557577498 learned_addr learned my addr 172.21.15.31:0/1557577498 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e19b9640 1 --2- 172.21.15.31:0/1557577498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690d00230 0x55a690d18000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e11b8640 1 --2- 172.21.15.31:0/1557577498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5240 0x55a690d0bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e09b7640 1 -- 172.21.15.31:0/1557577498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690d00230 msgr2=0x55a690d18000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/1557577498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690d00230 0x55a690d18000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e09b7640 1 -- 172.21.15.31:0/1557577498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5240 msgr2=0x55a690d0bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/1557577498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5240 0x55a690d0bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.793+0000 7ff7e09b7640 1 -- 172.21.15.31:0/1557577498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a690cf7840 con 0x55a690cf5b60 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.794+0000 7ff7e19b9640 1 --2- 172.21.15.31:0/1557577498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690d00230 0x55a690d18000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.794+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/1557577498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5b60 0x55a690cfc290 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x55a690cfc990 tx=0x7ff7cc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6cc1111e3e9207 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.794+0000 7ff7c6ffd640 1 -- 172.21.15.31:0/1557577498 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc02d070 con 0x55a690cf5b60 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.794+0000 7ff7c6ffd640 1 -- 172.21.15.31:0/1557577498 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7cc0210d0 con 0x55a690cf5b60 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.794+0000 7ff7c6ffd640 1 -- 172.21.15.31:0/1557577498 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc01f420 con 0x55a690cf5b60 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.795+0000 7ff7e6207740 1 -- 172.21.15.31:0/1557577498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5b60 msgr2=0x55a690cfc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.795+0000 7ff7e6207740 1 --2- 172.21.15.31:0/1557577498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5b60 0x55a690cfc290 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x55a690cfc990 tx=0x7ff7cc015000 comp rx=0 tx=0).stop 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.795+0000 7ff7e6207740 1 -- 172.21.15.31:0/1557577498 shutdown_connections 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.795+0000 7ff7e6207740 1 --2- 172.21.15.31:0/1557577498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690d00230 0x55a690d18000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.795+0000 7ff7e6207740 1 --2- 172.21.15.31:0/1557577498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690cf5b60 0x55a690cfc290 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.795+0000 7ff7e6207740 1 --2- 172.21.15.31:0/1557577498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5240 0x55a690d0bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.795+0000 7ff7e6207740 1 -- 172.21.15.31:0/1557577498 >> 172.21.15.31:0/1557577498 conn(0x55a690cf1560 msgr2=0x55a690cf1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.795+0000 7ff7e6207740 1 -- 172.21.15.31:0/1557577498 shutdown_connections 2024-04-25T01:55:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.795+0000 7ff7e6207740 1 -- 172.21.15.31:0/1557577498 wait complete. 2024-04-25T01:55:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.795+0000 7ff7e6207740 1 -- start start 2024-04-25T01:55:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.795+0000 7ff7e6207740 1 Processor -- start 2024-04-25T01:55:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5240 0x55a690d083f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690cf5b60 0x55a690d08930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e6207740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690d00230 0x55a690cf9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e6207740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1b07a0 con 0x55a690d00230 2024-04-25T01:55:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e6207740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1b00d0 con 0x55a690cf5240 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e6207740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc1ad310 con 0x55a690cf5b60 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e6207740 10 client.0 _ll_register_callbacks cb 0x7ff7d801b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e19b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690d00230 0x55a690cf9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e19b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690d00230 0x55a690cf9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35438/0 (socket says 172.21.15.31:35438) 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e19b9640 1 -- 172.21.15.31:0/1474015144 learned_addr learned my addr 172.21.15.31:0/1474015144 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e11b8640 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5240 0x55a690d083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e09b7640 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690cf5b60 0x55a690d08930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e19b9640 1 -- 172.21.15.31:0/1474015144 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690cf5b60 msgr2=0x55a690d08930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e19b9640 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690cf5b60 0x55a690d08930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e19b9640 1 -- 172.21.15.31:0/1474015144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5240 msgr2=0x55a690d083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e19b9640 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5240 0x55a690d083f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e19b9640 1 -- 172.21.15.31:0/1474015144 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a690cf9e70 con 0x55a690d00230 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e19b9640 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690d00230 0x55a690cf9730 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40bc7c0 tx=0x7ff7d402a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b024b46fb318211 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/1474015144 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d40da060 con 0x55a690d00230 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/1474015144 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7ff7d40edf90 con 0x55a690d00230 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.796+0000 7ff7e6207740 1 -- 172.21.15.31:0/1474015144 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7cc0210d0 con 0x55a690d00230 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/1474015144 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff7d40d4030 con 0x55a690d00230 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7e6207740 10 client.15798 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7e6207740 1 -- 172.21.15.31:0/1474015144 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a690d76260 con 0x55a690d00230 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/1474015144 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7cc0210d0 con 0x55a690d00230 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/1474015144 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff7d404d020 con 0x55a690d00230 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7e6207740 10 client.15798 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7e6207740 10 client.15798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7e6207740 20 client.15798 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7e6207740 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a690d76a30 0x55a690d78e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7e6207740 4 client.15798 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7e6207740 1 -- 172.21.15.31:0/1474015144 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a690c61540 con 0x55a690d76a30 2024-04-25T01:55:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.797+0000 7ff7e11b8640 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a690d76a30 0x55a690d78e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.798+0000 7ff7e11b8640 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a690d76a30 0x55a690d78e20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=107f296c56f4d170 server_cookie=efd70002592b1060 in_seq=0 out_seq=0 2024-04-25T01:55:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.798+0000 7ff7c4ff9640 10 client.15798 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.801+0000 7ff7c4ff9640 1 -- 172.21.15.31:0/1474015144 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a690c61540 con 0x55a690d76a30 2024-04-25T01:55:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.801+0000 7ff7c4ff9640 10 client.15798 handle_command_reply: tid=0 2024-04-25T01:55:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.801+0000 7ff7e6207740 1 client.15798 shutdown 2024-04-25T01:55:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.801+0000 7ff7e6207740 1 -- 172.21.15.31:0/1474015144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690d00230 msgr2=0x55a690cf9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.801+0000 7ff7e6207740 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690d00230 0x55a690cf9730 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40bc7c0 tx=0x7ff7d402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e6207740 1 -- 172.21.15.31:0/1474015144 shutdown_connections 2024-04-25T01:55:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e6207740 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a690d76a30 0x55a690d78e20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e6207740 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a690d00230 0x55a690cf9730 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e6207740 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a690cf5b60 0x55a690d08930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e6207740 1 --2- 172.21.15.31:0/1474015144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a690cf5240 0x55a690d083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e6207740 1 -- 172.21.15.31:0/1474015144 >> 172.21.15.31:0/1474015144 conn(0x55a690cf1560 msgr2=0x55a690cefd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7c4ff9640 0 client.15798 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e6207740 1 -- 172.21.15.31:0/1474015144 shutdown_connections 2024-04-25T01:55:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e6207740 1 -- 172.21.15.31:0/1474015144 wait complete. 2024-04-25T01:55:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e6207740 20 client.15798 trim_cache size 0 max 16384 2024-04-25T01:55:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:55:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:55:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:55:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:55:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e341c640 1 -- 172.21.15.31:0/1500096426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff7c80a2f40 msgr2=0x7ff7c80a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e341c640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff7c80a2f40 0x7ff7c80a5400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc10cc70 tx=0x7ff7cc0041e0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e341c640 1 -- 172.21.15.31:0/1500096426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7dc106630 msgr2=0x7ff7dc19ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e341c640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7dc106630 0x7ff7dc19ea20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4005cc0 tx=0x7ff7d4003c20 comp rx=0 tx=0).stop 2024-04-25T01:55:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e341c640 1 -- 172.21.15.31:0/1500096426 shutdown_connections 2024-04-25T01:55:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e341c640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff7c80a2f40 0x7ff7c80a5400 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e341c640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7dc106630 0x7ff7dc19ea20 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e341c640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff7dc101030 0x7ff7dc19a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e341c640 1 --2- 172.21.15.31:0/1500096426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7dc100670 0x7ff7dc19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.802+0000 7ff7e341c640 1 -- 172.21.15.31:0/1500096426 >> 172.21.15.31:0/1500096426 conn(0x7ff7dc0777c0 msgr2=0x7ff7dc10bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.803+0000 7ff7e341c640 1 -- 172.21.15.31:0/1500096426 shutdown_connections 2024-04-25T01:55:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:16.803+0000 7ff7e341c640 1 -- 172.21.15.31:0/1500096426 wait complete. 2024-04-25T01:55:17.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:17 smithi031 ceph-mon[29204]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 27 op/s 2024-04-25T01:55:18.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:17 smithi040 ceph-mon[32418]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 27 op/s 2024-04-25T01:55:18.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:17 smithi134 ceph-mon[33098]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 27 op/s 2024-04-25T01:55:19.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:19 smithi031 ceph-mon[29204]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:55:20.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:19 smithi040 ceph-mon[32418]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:55:20.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:19 smithi134 ceph-mon[33098]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:55:21.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:21 smithi031 ceph-mon[29204]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T01:55:22.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:21 smithi040 ceph-mon[32418]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T01:55:22.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:21 smithi134 ceph-mon[33098]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T01:55:23.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:23 smithi031 ceph-mon[29204]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:24.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:23 smithi040 ceph-mon[32418]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:24.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:23 smithi134 ceph-mon[33098]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:25.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:25 smithi031 ceph-mon[29204]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:55:26.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:25 smithi040 ceph-mon[32418]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:55:26.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:25 smithi134 ceph-mon[33098]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:55:27.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:27 smithi031 ceph-mon[29204]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T01:55:28.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:27 smithi040 ceph-mon[32418]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T01:55:28.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:27 smithi134 ceph-mon[33098]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T01:55:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.930+0000 7f15b7c31640 1 -- 172.21.15.31:0/1088256982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b010bd80 msgr2=0x7f15b010e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.930+0000 7f15b7c31640 1 --2- 172.21.15.31:0/1088256982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b010bd80 0x7f15b010e1d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009ca0 tx=0x7f15a802a950 comp rx=0 tx=0).stop 2024-04-25T01:55:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.930+0000 7f15b7c31640 1 -- 172.21.15.31:0/1088256982 shutdown_connections 2024-04-25T01:55:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.930+0000 7f15b7c31640 1 --2- 172.21.15.31:0/1088256982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b010bd80 0x7f15b010e1d0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.930+0000 7f15b7c31640 1 --2- 172.21.15.31:0/1088256982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15b01032d0 0x7f15b010b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.930+0000 7f15b7c31640 1 --2- 172.21.15.31:0/1088256982 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b01029b0 0x7f15b0102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.930+0000 7f15b7c31640 1 -- 172.21.15.31:0/1088256982 >> 172.21.15.31:0/1088256982 conn(0x7f15b0077850 msgr2=0x7f15b0077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.930+0000 7f15b7c31640 1 -- 172.21.15.31:0/1088256982 shutdown_connections 2024-04-25T01:55:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.930+0000 7f15b7c31640 1 -- 172.21.15.31:0/1088256982 wait complete. 2024-04-25T01:55:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b7c31640 1 Processor -- start 2024-04-25T01:55:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b7c31640 1 -- start start 2024-04-25T01:55:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b7c31640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b01029b0 0x7f15b0100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b7c31640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b01032d0 0x7f15b0100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b7c31640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15b010bd80 0x7f15b0101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b7c31640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f15b0111a30 con 0x7f15b01032d0 2024-04-25T01:55:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b7c31640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01118b0 con 0x7f15b010bd80 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b7c31640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f15b0111bb0 con 0x7f15b01029b0 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b51cc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b01032d0 0x7f15b0100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b51cc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b01032d0 0x7f15b0100b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57268/0 (socket says 172.21.15.31:57268) 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b51cc640 1 -- 172.21.15.31:0/1939470241 learned_addr learned my addr 172.21.15.31:0/1939470241 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b59cd640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b01029b0 0x7f15b0100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.931+0000 7f15b61ce640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15b010bd80 0x7f15b0101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15b51cc640 1 -- 172.21.15.31:0/1939470241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b01029b0 msgr2=0x7f15b0100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15b51cc640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b01029b0 0x7f15b0100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15b51cc640 1 -- 172.21.15.31:0/1939470241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15b010bd80 msgr2=0x7f15b0101080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15b51cc640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15b010bd80 0x7f15b0101080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15b51cc640 1 -- 172.21.15.31:0/1939470241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b01a6270 con 0x7f15b01032d0 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15b61ce640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15b010bd80 0x7f15b0101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15b51cc640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b01032d0 0x7f15b0100b40 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f15a000d950 tx=0x7f15a000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15a6ffd640 1 -- 172.21.15.31:0/1939470241 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a0017030 con 0x7f15b01032d0 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15a6ffd640 1 -- 172.21.15.31:0/1939470241 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15a0004020 con 0x7f15b01032d0 2024-04-25T01:55:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15a6ffd640 1 -- 172.21.15.31:0/1939470241 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a0005570 con 0x7f15b01032d0 2024-04-25T01:55:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15b7c31640 1 -- 172.21.15.31:0/1939470241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b01a6560 con 0x7f15b01032d0 2024-04-25T01:55:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.932+0000 7f15b7c31640 1 -- 172.21.15.31:0/1939470241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b01a6b00 con 0x7f15b01032d0 2024-04-25T01:55:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.936+0000 7f15a6ffd640 1 -- 172.21.15.31:0/1939470241 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f15a00041d0 con 0x7f15b01032d0 2024-04-25T01:55:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.936+0000 7f15baa1a740 1 Processor -- start 2024-04-25T01:55:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.936+0000 7f15baa1a740 1 -- start start 2024-04-25T01:55:29.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.936+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 0x556051de9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.936+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9b60 0x556051df3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15b61ce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 0x556051de9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15b61ce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 0x556051de9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33280/0 (socket says 172.21.15.31:33280) 2024-04-25T01:55:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051df4230 0x556051df6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15b61ce640 1 -- 172.21.15.31:0/3383674124 learned_addr learned my addr 172.21.15.31:0/3383674124 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15baa1a740 1 -- 172.21.15.31:0/3383674124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556051d4dcb0 con 0x556051de9b60 2024-04-25T01:55:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15baa1a740 1 -- 172.21.15.31:0/3383674124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556051cf65b0 con 0x556051de9240 2024-04-25T01:55:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15baa1a740 1 -- 172.21.15.31:0/3383674124 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556051d49c70 con 0x556051df4230 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15b59cd640 1 --2- 172.21.15.31:0/3383674124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9b60 0x556051df3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15a6ffd640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f159c0a2f40 0x7f159c0a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15b61ce640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f159c0a2f40 0x7f159c0a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15a6ffd640 1 -- 172.21.15.31:0/1939470241 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f15a00ce9a0 con 0x7f15b01032d0 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15b61ce640 1 -- 172.21.15.31:0/3383674124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051df4230 msgr2=0x556051df6620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15b61ce640 1 --2- 172.21.15.31:0/3383674124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051df4230 0x556051df6620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15b61ce640 1 -- 172.21.15.31:0/3383674124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9b60 msgr2=0x556051df3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15b61ce640 1 --2- 172.21.15.31:0/3383674124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9b60 0x556051df3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15b61ce640 1 -- 172.21.15.31:0/3383674124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556051a62210 con 0x556051de9240 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.937+0000 7f15b59cd640 1 --2- 172.21.15.31:0/3383674124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9b60 0x556051df3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.938+0000 7f15b61ce640 1 --2- 172.21.15.31:0/3383674124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 0x556051de9620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15a8003da0 tx=0x7f15a8003dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30da81fdefafc63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.938+0000 7f159bfff640 1 -- 172.21.15.31:0/3383674124 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8034040 con 0x556051de9240 2024-04-25T01:55:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.938+0000 7f15b61ce640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f159c0a2f40 0x7f159c0a5400 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15a804d000 tx=0x7f15a804b000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.938+0000 7f159bfff640 1 -- 172.21.15.31:0/3383674124 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15a804e070 con 0x556051de9240 2024-04-25T01:55:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.938+0000 7f159bfff640 1 -- 172.21.15.31:0/3383674124 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a80498d0 con 0x556051de9240 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.938+0000 7f15baa1a740 1 -- 172.21.15.31:0/3383674124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 msgr2=0x556051de9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.938+0000 7f15baa1a740 1 --2- 172.21.15.31:0/3383674124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 0x556051de9620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15a8003da0 tx=0x7f15a8003dd0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.939+0000 7f15baa1a740 1 -- 172.21.15.31:0/3383674124 shutdown_connections 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.939+0000 7f15baa1a740 1 --2- 172.21.15.31:0/3383674124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051df4230 0x556051df6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.939+0000 7f15baa1a740 1 --2- 172.21.15.31:0/3383674124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9b60 0x556051df3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.939+0000 7f15baa1a740 1 --2- 172.21.15.31:0/3383674124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 0x556051de9620 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.939+0000 7f15baa1a740 1 -- 172.21.15.31:0/3383674124 >> 172.21.15.31:0/3383674124 conn(0x556051de5560 msgr2=0x556051de5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.939+0000 7f15baa1a740 1 -- 172.21.15.31:0/3383674124 shutdown_connections 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.939+0000 7f15baa1a740 1 -- 172.21.15.31:0/3383674124 wait complete. 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.939+0000 7f15baa1a740 1 -- start start 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.939+0000 7f15baa1a740 1 Processor -- start 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051df1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9b60 0x556051df17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051df4230 0x556051df1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15baa1a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556051df7020 con 0x556051de9240 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15baa1a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556051d09b50 con 0x556051de9b60 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15baa1a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556051df71a0 con 0x556051df4230 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15b59cd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051df1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15baa1a740 10 client.0 _ll_register_callbacks cb 0x556051d44ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15b59cd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051df1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57282/0 (socket says 172.21.15.31:57282) 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15b59cd640 1 -- 172.21.15.31:0/3009055909 learned_addr learned my addr 172.21.15.31:0/3009055909 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15b51cc640 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9b60 0x556051df17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15b61ce640 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051df4230 0x556051df1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15b59cd640 1 -- 172.21.15.31:0/3009055909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051df4230 msgr2=0x556051df1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15b59cd640 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051df4230 0x556051df1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15b59cd640 1 -- 172.21.15.31:0/3009055909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9b60 msgr2=0x556051df17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15b59cd640 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9b60 0x556051df17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15b59cd640 1 -- 172.21.15.31:0/3009055909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556051b2e620 con 0x556051de9240 2024-04-25T01:55:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15b59cd640 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051df1270 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f15ac010bf0 tx=0x7f15ac00af80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7127d56c17e3711 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15997fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.940+0000 7f15997fa640 1 -- 172.21.15.31:0/3009055909 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac01e060 con 0x556051de9240 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15997fa640 1 -- 172.21.15.31:0/3009055909 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f15ac00fb60 con 0x556051de9240 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15baa1a740 1 -- 172.21.15.31:0/3009055909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15a804e070 con 0x556051de9240 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15997fa640 1 -- 172.21.15.31:0/3009055909 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f15ac00fd60 con 0x556051de9240 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15baa1a740 10 client.15810 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15baa1a740 1 -- 172.21.15.31:0/3009055909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556051e03130 con 0x556051de9240 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15997fa640 1 -- 172.21.15.31:0/3009055909 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15a804e070 con 0x556051de9240 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15997fa640 1 -- 172.21.15.31:0/3009055909 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f15ac018400 con 0x556051de9240 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15baa1a740 10 client.15810 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15baa1a740 10 client.15810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15baa1a740 20 client.15810 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15baa1a740 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556051e045a0 0x556051e06990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15baa1a740 4 client.15810 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.941+0000 7f15baa1a740 1 -- 172.21.15.31:0/3009055909 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556051d5b1f0 con 0x556051e045a0 2024-04-25T01:55:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.942+0000 7f15b51cc640 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556051e045a0 0x556051e06990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.942+0000 7f15b51cc640 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556051e045a0 0x556051e06990 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=698301e209d6e5cd server_cookie=bc35d00b7106d412 in_seq=0 out_seq=0 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.942+0000 7f15997fa640 10 client.15810 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15997fa640 1 -- 172.21.15.31:0/3009055909 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556051d5b1f0 con 0x556051e045a0 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15997fa640 10 client.15810 handle_command_reply: tid=0 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 1 client.15810 shutdown 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 1 -- 172.21.15.31:0/3009055909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 msgr2=0x556051df1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051df1270 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f15ac010bf0 tx=0x7f15ac00af80 comp rx=0 tx=0).stop 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 1 -- 172.21.15.31:0/3009055909 shutdown_connections 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556051e045a0 0x556051e06990 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051df4230 0x556051df1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15997fa640 0 client.15810 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9b60 0x556051df17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 1 --2- 172.21.15.31:0/3009055909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051df1270 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 1 -- 172.21.15.31:0/3009055909 >> 172.21.15.31:0/3009055909 conn(0x556051de5560 msgr2=0x556051deb260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 1 -- 172.21.15.31:0/3009055909 shutdown_connections 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 1 -- 172.21.15.31:0/3009055909 wait complete. 2024-04-25T01:55:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.943+0000 7f15baa1a740 20 client.15810 trim_cache size 0 max 16384 2024-04-25T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.972+0000 7f15baa1a740 1 Processor -- start 2024-04-25T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15baa1a740 1 -- start start 2024-04-25T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051dffee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051de9b60 0x556051deb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051df4230 0x556051e0c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15baa1a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01af430 con 0x556051de9240 2024-04-25T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15baa1a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01b0f70 con 0x556051df4230 2024-04-25T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15b59cd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051dffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15baa1a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01afb00 con 0x556051de9b60 2024-04-25T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15b59cd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051dffee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57298/0 (socket says 172.21.15.31:57298) 2024-04-25T01:55:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15b59cd640 1 -- 172.21.15.31:0/716349718 learned_addr learned my addr 172.21.15.31:0/716349718 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15b51cc640 1 --2- 172.21.15.31:0/716349718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051de9b60 0x556051deb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15b61ce640 1 --2- 172.21.15.31:0/716349718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051df4230 0x556051e0c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15b59cd640 1 -- 172.21.15.31:0/716349718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051de9b60 msgr2=0x556051deb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15b59cd640 1 --2- 172.21.15.31:0/716349718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051de9b60 0x556051deb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15b59cd640 1 -- 172.21.15.31:0/716349718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051df4230 msgr2=0x556051e0c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15b59cd640 1 --2- 172.21.15.31:0/716349718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051df4230 0x556051e0c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15b59cd640 1 -- 172.21.15.31:0/716349718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556051e06d00 con 0x556051de9240 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.973+0000 7f15b61ce640 1 --2- 172.21.15.31:0/716349718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051df4230 0x556051e0c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.974+0000 7f15b59cd640 1 --2- 172.21.15.31:0/716349718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051dffee0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f15ac005bd0 tx=0x7f15ac00fcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe4fd86a5f136a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.974+0000 7f159b7fe640 1 -- 172.21.15.31:0/716349718 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac028080 con 0x556051de9240 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.974+0000 7f159b7fe640 1 -- 172.21.15.31:0/716349718 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15ac00a340 con 0x556051de9240 2024-04-25T01:55:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.974+0000 7f159b7fe640 1 -- 172.21.15.31:0/716349718 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac0048c0 con 0x556051de9240 2024-04-25T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.975+0000 7f15baa1a740 1 -- 172.21.15.31:0/716349718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 msgr2=0x556051dffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.975+0000 7f15baa1a740 1 --2- 172.21.15.31:0/716349718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051dffee0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f15ac005bd0 tx=0x7f15ac00fcf0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.975+0000 7f15baa1a740 1 -- 172.21.15.31:0/716349718 shutdown_connections 2024-04-25T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.975+0000 7f15baa1a740 1 --2- 172.21.15.31:0/716349718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051df4230 0x556051e0c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.975+0000 7f15baa1a740 1 --2- 172.21.15.31:0/716349718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051de9b60 0x556051deb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.975+0000 7f15baa1a740 1 --2- 172.21.15.31:0/716349718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051de9240 0x556051dffee0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.975+0000 7f15baa1a740 1 -- 172.21.15.31:0/716349718 >> 172.21.15.31:0/716349718 conn(0x556051de5560 msgr2=0x556051dea9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.975+0000 7f15baa1a740 1 -- 172.21.15.31:0/716349718 shutdown_connections 2024-04-25T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.975+0000 7f15baa1a740 1 -- 172.21.15.31:0/716349718 wait complete. 2024-04-25T01:55:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.975+0000 7f15baa1a740 1 -- start start 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.976+0000 7f15baa1a740 1 Processor -- start 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.976+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 0x556051df8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.976+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051de9b60 0x556051df8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.976+0000 7f15baa1a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051df4230 0x556051e62a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.976+0000 7f15baa1a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01b08a0 con 0x556051df4230 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.976+0000 7f15baa1a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01b01d0 con 0x556051de9240 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.976+0000 7f15baa1a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01ad410 con 0x556051de9b60 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.976+0000 7f15b61ce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051df4230 0x556051e62a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.976+0000 7f15b61ce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051df4230 0x556051e62a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57302/0 (socket says 172.21.15.31:57302) 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.976+0000 7f15b61ce640 1 -- 172.21.15.31:0/1970655489 learned_addr learned my addr 172.21.15.31:0/1970655489 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.976+0000 7f15b59cd640 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 0x556051df8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15b51cc640 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051de9b60 0x556051df8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15b61ce640 1 -- 172.21.15.31:0/1970655489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051de9b60 msgr2=0x556051df8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:29.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15b61ce640 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051de9b60 0x556051df8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15b61ce640 1 -- 172.21.15.31:0/1970655489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 msgr2=0x556051df8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15b61ce640 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 0x556051df8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15b61ce640 1 -- 172.21.15.31:0/1970655489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556051e63280 con 0x556051df4230 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15baa1a740 10 client.0 _ll_register_callbacks cb 0x7f15ac00a800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15b59cd640 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 0x556051df8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15b51cc640 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051de9b60 0x556051df8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15b61ce640 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051df4230 0x556051e62a80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f15a802af10 tx=0x7f15a8009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e928935a29256750 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15baa1a740 1 -- 172.21.15.31:0/1970655489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15ac00a340 con 0x556051df4230 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15997fa640 10 client.15816 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15997fa640 1 -- 172.21.15.31:0/1970655489 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8004860 con 0x556051df4230 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15997fa640 1 -- 172.21.15.31:0/1970655489 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f15a80496f0 con 0x556051df4230 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15997fa640 1 -- 172.21.15.31:0/1970655489 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f15a8049930 con 0x556051df4230 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15baa1a740 10 client.15816 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.977+0000 7f15baa1a740 1 -- 172.21.15.31:0/1970655489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=27}) -- 0x556051e6b580 con 0x556051df4230 2024-04-25T01:55:29.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.978+0000 7f15997fa640 1 -- 172.21.15.31:0/1970655489 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f15a8049ad0 con 0x556051df4230 2024-04-25T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.978+0000 7f15baa1a740 10 client.15816 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.978+0000 7f15baa1a740 10 client.15816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.978+0000 7f15baa1a740 20 client.15816 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.978+0000 7f15baa1a740 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556051e045a0 0x556051e6d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.978+0000 7f15baa1a740 4 client.15816 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.978+0000 7f15baa1a740 1 -- 172.21.15.31:0/1970655489 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556051d55540 con 0x556051e045a0 2024-04-25T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.978+0000 7f15b59cd640 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556051e045a0 0x556051e6d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.979+0000 7f15997fa640 10 client.15816 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.979+0000 7f15b59cd640 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556051e045a0 0x556051e6d800 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b43fc32dbb51bb61 server_cookie=8ab2b6428aef1089 in_seq=0 out_seq=0 2024-04-25T01:55:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15997fa640 1 -- 172.21.15.31:0/1970655489 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556051d55540 con 0x556051e045a0 2024-04-25T01:55:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15997fa640 10 client.15816 handle_command_reply: tid=0 2024-04-25T01:55:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 1 client.15816 shutdown 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 1 -- 172.21.15.31:0/1970655489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051df4230 msgr2=0x556051e62a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051df4230 0x556051e62a80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f15a802af10 tx=0x7f15a8009630 comp rx=0 tx=0).stop 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 1 -- 172.21.15.31:0/1970655489 shutdown_connections 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556051e045a0 0x556051e6d800 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556051df4230 0x556051e62a80 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15997fa640 0 client.15816 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556051de9b60 0x556051df8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 1 --2- 172.21.15.31:0/1970655489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556051de9240 0x556051df8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 1 -- 172.21.15.31:0/1970655489 >> 172.21.15.31:0/1970655489 conn(0x556051de5560 msgr2=0x556051df1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 1 -- 172.21.15.31:0/1970655489 shutdown_connections 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 1 -- 172.21.15.31:0/1970655489 wait complete. 2024-04-25T01:55:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.983+0000 7f15baa1a740 20 client.15816 trim_cache size 0 max 16384 2024-04-25T01:55:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:55:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:55:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:55:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:55:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 -- 172.21.15.31:0/1939470241 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f159c0a2f40 msgr2=0x7f159c0a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f159c0a2f40 0x7f159c0a5400 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15a804d000 tx=0x7f15a804b000 comp rx=0 tx=0).stop 2024-04-25T01:55:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 -- 172.21.15.31:0/1939470241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b01032d0 msgr2=0x7f15b0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b01032d0 0x7f15b0100b40 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f15a000d950 tx=0x7f15a000dd50 comp rx=0 tx=0).stop 2024-04-25T01:55:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 -- 172.21.15.31:0/1939470241 shutdown_connections 2024-04-25T01:55:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f159c0a2f40 0x7f159c0a5400 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15b010bd80 0x7f15b0101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b01032d0 0x7f15b0100b40 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 --2- 172.21.15.31:0/1939470241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b01029b0 0x7f15b0100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 -- 172.21.15.31:0/1939470241 >> 172.21.15.31:0/1939470241 conn(0x7f15b0077850 msgr2=0x7f15b00fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 -- 172.21.15.31:0/1939470241 shutdown_connections 2024-04-25T01:55:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:29.984+0000 7f15b7c31640 1 -- 172.21.15.31:0/1939470241 wait complete. 2024-04-25T01:55:29.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:29 smithi031 ceph-mon[29204]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:30.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:29 smithi040 ceph-mon[32418]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:30.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:29 smithi134 ceph-mon[33098]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:31.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:31 smithi031 ceph-mon[29204]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T01:55:32.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:31 smithi040 ceph-mon[32418]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T01:55:32.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:31 smithi134 ceph-mon[33098]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T01:55:33.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:33 smithi031 ceph-mon[29204]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:55:33.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:33 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:55:34.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:33 smithi040 ceph-mon[32418]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:55:34.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:33 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:55:34.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:33 smithi134 ceph-mon[33098]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:55:34.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:33 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:55:35.661 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:35 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:55:35.661 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:35 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:55:35.661 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:35 smithi040 ceph-mon[32418]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-25T01:55:35.661 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:35 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:55:35.661 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:35 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:55:35.661 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:35 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:55:35.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:35 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:55:35.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:35 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:55:35.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:35 smithi134 ceph-mon[33098]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-25T01:55:35.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:35 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:55:35.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:35 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:55:35.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:35 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:55:35.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:35 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:55:35.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:35 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:55:35.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:35 smithi031 ceph-mon[29204]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-25T01:55:35.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:35 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:55:35.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:35 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:55:35.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:35 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:55:37.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:37 smithi134 ceph-mon[33098]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:55:37.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:37 smithi031 ceph-mon[29204]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:55:38.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:37 smithi040 ceph-mon[32418]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:55:38.593 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:55:38.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:55:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.711+0000 7f89630a5640 1 -- 172.21.15.31:0/806956659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f895c100610 msgr2=0x7f895c100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.711+0000 7f89630a5640 1 --2- 172.21.15.31:0/806956659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f895c100610 0x7f895c100a90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8948009930 tx=0x7f894802a420 comp rx=0 tx=0).stop 2024-04-25T01:55:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.711+0000 7f89630a5640 1 -- 172.21.15.31:0/806956659 shutdown_connections 2024-04-25T01:55:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.711+0000 7f89630a5640 1 --2- 172.21.15.31:0/806956659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f895c100fd0 0x7f895c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.711+0000 7f89630a5640 1 --2- 172.21.15.31:0/806956659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f895c100610 0x7f895c100a90 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.711+0000 7f89630a5640 1 --2- 172.21.15.31:0/806956659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f895c1066b0 0x7f895c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.711+0000 7f89630a5640 1 -- 172.21.15.31:0/806956659 >> 172.21.15.31:0/806956659 conn(0x7f895c077740 msgr2=0x7f895c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.712+0000 7f89630a5640 1 -- 172.21.15.31:0/806956659 shutdown_connections 2024-04-25T01:55:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.712+0000 7f89630a5640 1 -- 172.21.15.31:0/806956659 wait complete. 2024-04-25T01:55:38.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.712+0000 7f89630a5640 1 Processor -- start 2024-04-25T01:55:38.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.712+0000 7f89630a5640 1 -- start start 2024-04-25T01:55:38.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f89630a5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f895c100610 0x7f895c19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f89630a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f895c100fd0 0x7f895c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f89630a5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f895c1066b0 0x7f895c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f89630a5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f895c1108f0 con 0x7f895c100fd0 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f89630a5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f895c110770 con 0x7f895c1066b0 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f89630a5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f895c110a70 con 0x7f895c100610 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f8960e41640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f895c100610 0x7f895c19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f895bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f895c100fd0 0x7f895c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f8960e41640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f895c100610 0x7f895c19da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:34848/0 (socket says 172.21.15.31:34848) 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f8960e41640 1 -- 172.21.15.31:0/3742308190 learned_addr learned my addr 172.21.15.31:0/3742308190 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f895bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f895c100fd0 0x7f895c19dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33328/0 (socket says 172.21.15.31:33328) 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f8961642640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f895c1066b0 0x7f895c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f895bfff640 1 -- 172.21.15.31:0/3742308190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f895c100610 msgr2=0x7f895c19da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f895bfff640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f895c100610 0x7f895c19da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f895bfff640 1 -- 172.21.15.31:0/3742308190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f895c1066b0 msgr2=0x7f895c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f895bfff640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f895c1066b0 0x7f895c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.713+0000 7f895bfff640 1 -- 172.21.15.31:0/3742308190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895c1983e0 con 0x7f895c100fd0 2024-04-25T01:55:38.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.714+0000 7f895bfff640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f895c100fd0 0x7f895c19dfa0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8948009850 tx=0x7f8948030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.714+0000 7f8959ffb640 1 -- 172.21.15.31:0/3742308190 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894803f070 con 0x7f895c100fd0 2024-04-25T01:55:38.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.714+0000 7f8959ffb640 1 -- 172.21.15.31:0/3742308190 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8948004460 con 0x7f895c100fd0 2024-04-25T01:55:38.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.714+0000 7f8959ffb640 1 -- 172.21.15.31:0/3742308190 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948035030 con 0x7f895c100fd0 2024-04-25T01:55:38.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.714+0000 7f89630a5640 1 -- 172.21.15.31:0/3742308190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895c198670 con 0x7f895c100fd0 2024-04-25T01:55:38.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.714+0000 7f89630a5640 1 -- 172.21.15.31:0/3742308190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895c1021e0 con 0x7f895c100fd0 2024-04-25T01:55:38.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8959ffb640 1 -- 172.21.15.31:0/3742308190 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f8948042070 con 0x7f895c100fd0 2024-04-25T01:55:38.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8965dbc740 1 Processor -- start 2024-04-25T01:55:38.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8965dbc740 1 -- start start 2024-04-25T01:55:38.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 0x563fa7ac1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1d30 0x563fa7ad48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ad4e30 0x563fa7ad7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8965dbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563fa7a266b0 con 0x563fa7ac1380 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8965dbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563fa79ce880 con 0x563fa7ac1d30 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8965dbc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563fa7a22670 con 0x563fa7ad4e30 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f895bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ad4e30 0x563fa7ad7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8961642640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 0x563fa7ac1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f895bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ad4e30 0x563fa7ad7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:34850/0 (socket says 172.21.15.31:34850) 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f895bfff640 1 -- 172.21.15.31:0/322739453 learned_addr learned my addr 172.21.15.31:0/322739453 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8961642640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 0x563fa7ac1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33344/0 (socket says 172.21.15.31:33344) 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.718+0000 7f8960e41640 1 --2- 172.21.15.31:0/322739453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1d30 0x563fa7ad48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.719+0000 7f895bfff640 1 -- 172.21.15.31:0/322739453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1d30 msgr2=0x563fa7ad48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.719+0000 7f895bfff640 1 --2- 172.21.15.31:0/322739453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1d30 0x563fa7ad48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.719+0000 7f895bfff640 1 -- 172.21.15.31:0/322739453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 msgr2=0x563fa7ac1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.719+0000 7f895bfff640 1 --2- 172.21.15.31:0/322739453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 0x563fa7ac1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.719+0000 7f895bfff640 1 -- 172.21.15.31:0/322739453 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fa78047d0 con 0x563fa7ad4e30 2024-04-25T01:55:38.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.719+0000 7f895bfff640 1 --2- 172.21.15.31:0/322739453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ad4e30 0x563fa7ad7220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f89480ed770 tx=0x7f89480f8360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ebb27e707871bb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.719+0000 7f8946ffd640 1 -- 172.21.15.31:0/322739453 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89480f8b60 con 0x563fa7ad4e30 2024-04-25T01:55:38.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.719+0000 7f8946ffd640 1 -- 172.21.15.31:0/322739453 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89480fd030 con 0x563fa7ad4e30 2024-04-25T01:55:38.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.719+0000 7f8946ffd640 1 -- 172.21.15.31:0/322739453 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948101630 con 0x563fa7ad4e30 2024-04-25T01:55:38.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.719+0000 7f8959ffb640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f894c0a2ea0 0x7f894c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.719+0000 7f8959ffb640 1 -- 172.21.15.31:0/3742308190 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f89480f0040 con 0x7f895c100fd0 2024-04-25T01:55:38.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.720+0000 7f8961642640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f894c0a2ea0 0x7f894c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.720+0000 7f8965dbc740 1 -- 172.21.15.31:0/322739453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ad4e30 msgr2=0x563fa7ad7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.720+0000 7f8965dbc740 1 --2- 172.21.15.31:0/322739453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ad4e30 0x563fa7ad7220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f89480ed770 tx=0x7f89480f8360 comp rx=0 tx=0).stop 2024-04-25T01:55:38.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.720+0000 7f8961642640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f894c0a2ea0 0x7f894c0a5360 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8954005fd0 tx=0x7f8954009500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.720+0000 7f8965dbc740 1 -- 172.21.15.31:0/322739453 shutdown_connections 2024-04-25T01:55:38.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.721+0000 7f8965dbc740 1 --2- 172.21.15.31:0/322739453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ad4e30 0x563fa7ad7220 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.721+0000 7f8965dbc740 1 --2- 172.21.15.31:0/322739453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1d30 0x563fa7ad48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.721+0000 7f8965dbc740 1 --2- 172.21.15.31:0/322739453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 0x563fa7ac1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.721+0000 7f8965dbc740 1 -- 172.21.15.31:0/322739453 >> 172.21.15.31:0/322739453 conn(0x563fa7abd950 msgr2=0x563fa7abdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:38.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.721+0000 7f8965dbc740 1 -- 172.21.15.31:0/322739453 shutdown_connections 2024-04-25T01:55:38.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.721+0000 7f8965dbc740 1 -- 172.21.15.31:0/322739453 wait complete. 2024-04-25T01:55:38.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.721+0000 7f8965dbc740 1 -- start start 2024-04-25T01:55:38.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.721+0000 7f8965dbc740 1 Processor -- start 2024-04-25T01:55:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.722+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1380 0x563fa7ad1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.722+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 0x563fa7ad22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.722+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ad4e30 0x563fa7ad2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.722+0000 7f8965dbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563fa7ad79b0 con 0x563fa7ad4e30 2024-04-25T01:55:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.722+0000 7f895bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 0x563fa7ad22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.722+0000 7f8961642640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ad4e30 0x563fa7ad2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.722+0000 7f895bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 0x563fa7ad22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:34864/0 (socket says 172.21.15.31:34864) 2024-04-25T01:55:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.722+0000 7f895bfff640 1 -- 172.21.15.31:0/1794183616 learned_addr learned my addr 172.21.15.31:0/1794183616 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.722+0000 7f8960e41640 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1380 0x563fa7ad1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.722+0000 7f8965dbc740 1 -- 172.21.15.31:0/1794183616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563fa79e1f50 con 0x563fa7ac1380 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f8965dbc740 1 -- 172.21.15.31:0/1794183616 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563fa7ad7b30 con 0x563fa7ac1d30 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f8965dbc740 10 client.0 _ll_register_callbacks cb 0x563fa7a1d000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f895bfff640 1 -- 172.21.15.31:0/1794183616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1380 msgr2=0x563fa7ad1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f895bfff640 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1380 0x563fa7ad1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f895bfff640 1 -- 172.21.15.31:0/1794183616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ad4e30 msgr2=0x563fa7ad2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f895bfff640 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ad4e30 0x563fa7ad2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f895bfff640 1 -- 172.21.15.31:0/1794183616 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fa773a3f0 con 0x563fa7ac1d30 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f8961642640 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ad4e30 0x563fa7ad2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f895bfff640 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 0x563fa7ad22d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f89480edde0 tx=0x7f894802a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec27df64aaddd8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f893bfff640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f893bfff640 1 -- 172.21.15.31:0/1794183616 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948049070 con 0x563fa7ac1d30 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.723+0000 7f8965dbc740 1 -- 172.21.15.31:0/1794183616 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89480fd030 con 0x563fa7ac1d30 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.724+0000 7f893bfff640 1 -- 172.21.15.31:0/1794183616 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f89480ba4d0 con 0x563fa7ac1d30 2024-04-25T01:55:38.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.724+0000 7f893bfff640 1 -- 172.21.15.31:0/1794183616 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f89480b86d0 con 0x563fa7ac1d30 2024-04-25T01:55:38.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.724+0000 7f8965dbc740 10 client.25354 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:38.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.724+0000 7f8965dbc740 1 -- 172.21.15.31:0/1794183616 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({fsmap=27}) -- 0x563fa76fa0f0 con 0x563fa7ac1d30 2024-04-25T01:55:38.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.724+0000 7f893bfff640 1 -- 172.21.15.31:0/1794183616 <== mon.1 v2:172.21.15.134:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8948003cd0 con 0x563fa7ac1d30 2024-04-25T01:55:38.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.724+0000 7f8965dbc740 10 client.25354 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:38.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.724+0000 7f8965dbc740 10 client.25354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:38.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.724+0000 7f8965dbc740 20 client.25354 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:38.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.725+0000 7f8965dbc740 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563fa7ae4b20 0x563fa7ae6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.725+0000 7f8965dbc740 4 client.25354 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:38.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.725+0000 7f8965dbc740 1 -- 172.21.15.31:0/1794183616 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f89480fd030 con 0x563fa7ae4b20 2024-04-25T01:55:38.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.725+0000 7f8960e41640 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563fa7ae4b20 0x563fa7ae6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.726+0000 7f8960e41640 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563fa7ae4b20 0x563fa7ae6f10 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc75eb45f17e3673 server_cookie=98ca98d256c64c9e in_seq=0 out_seq=0 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.726+0000 7f893bfff640 10 client.25354 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.726+0000 7f893bfff640 1 -- 172.21.15.31:0/1794183616 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f89480fd030 con 0x563fa7ae4b20 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.726+0000 7f893bfff640 10 client.25354 handle_command_reply: tid=0 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.726+0000 7f8965dbc740 1 client.25354 shutdown 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.726+0000 7f8965dbc740 1 -- 172.21.15.31:0/1794183616 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 msgr2=0x563fa7ad22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.726+0000 7f8965dbc740 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 0x563fa7ad22d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f89480edde0 tx=0x7f894802a420 comp rx=0 tx=0).stop 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.727+0000 7f8965dbc740 1 -- 172.21.15.31:0/1794183616 shutdown_connections 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.727+0000 7f8965dbc740 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563fa7ae4b20 0x563fa7ae6f10 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.727+0000 7f8965dbc740 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ad4e30 0x563fa7ad2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.727+0000 7f8965dbc740 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 0x563fa7ad22d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.727+0000 7f8965dbc740 1 --2- 172.21.15.31:0/1794183616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1380 0x563fa7ad1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.727+0000 7f8965dbc740 1 -- 172.21.15.31:0/1794183616 >> 172.21.15.31:0/1794183616 conn(0x563fa7abd950 msgr2=0x563fa7ac37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.727+0000 7f893bfff640 0 client.25354 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:38.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.727+0000 7f8965dbc740 1 -- 172.21.15.31:0/1794183616 shutdown_connections 2024-04-25T01:55:38.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.727+0000 7f8965dbc740 1 -- 172.21.15.31:0/1794183616 wait complete. 2024-04-25T01:55:38.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.727+0000 7f8965dbc740 20 client.25354 trim_cache size 0 max 16384 2024-04-25T01:55:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8965dbc740 1 Processor -- start 2024-04-25T01:55:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8965dbc740 1 -- start start 2024-04-25T01:55:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7abd950 0x563fa7ad44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1380 0x563fa7ae0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 0x563fa7aec540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8960e41640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7abd950 0x563fa7ad44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8965dbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f895c1aade0 con 0x563fa7abd950 2024-04-25T01:55:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8965dbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f895c1ac920 con 0x563fa7ac1380 2024-04-25T01:55:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8965dbc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f895c1ab4b0 con 0x563fa7ac1d30 2024-04-25T01:55:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8960e41640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7abd950 0x563fa7ad44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33366/0 (socket says 172.21.15.31:33366) 2024-04-25T01:55:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8960e41640 1 -- 172.21.15.31:0/1527070992 learned_addr learned my addr 172.21.15.31:0/1527070992 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f8961642640 1 --2- 172.21.15.31:0/1527070992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 0x563fa7aec540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.762+0000 7f895bfff640 1 --2- 172.21.15.31:0/1527070992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1380 0x563fa7ae0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.763+0000 7f8960e41640 1 -- 172.21.15.31:0/1527070992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 msgr2=0x563fa7aec540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.763+0000 7f8960e41640 1 --2- 172.21.15.31:0/1527070992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 0x563fa7aec540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.763+0000 7f8960e41640 1 -- 172.21.15.31:0/1527070992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1380 msgr2=0x563fa7ae0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.763+0000 7f8960e41640 1 --2- 172.21.15.31:0/1527070992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1380 0x563fa7ae0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.763+0000 7f8960e41640 1 -- 172.21.15.31:0/1527070992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fa7ad2cf0 con 0x563fa7abd950 2024-04-25T01:55:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.763+0000 7f8960e41640 1 --2- 172.21.15.31:0/1527070992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7abd950 0x563fa7ad44c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f895000be40 tx=0x7f895000b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b046f0f741d55ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.763+0000 7f89467fc640 1 -- 172.21.15.31:0/1527070992 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950004220 con 0x563fa7abd950 2024-04-25T01:55:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.763+0000 7f89467fc640 1 -- 172.21.15.31:0/1527070992 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8950023030 con 0x563fa7abd950 2024-04-25T01:55:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.763+0000 7f89467fc640 1 -- 172.21.15.31:0/1527070992 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950003a10 con 0x563fa7abd950 2024-04-25T01:55:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.764+0000 7f8965dbc740 1 -- 172.21.15.31:0/1527070992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7abd950 msgr2=0x563fa7ad44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.764+0000 7f8965dbc740 1 --2- 172.21.15.31:0/1527070992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7abd950 0x563fa7ad44c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f895000be40 tx=0x7f895000b410 comp rx=0 tx=0).stop 2024-04-25T01:55:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.764+0000 7f8965dbc740 1 -- 172.21.15.31:0/1527070992 shutdown_connections 2024-04-25T01:55:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.764+0000 7f8965dbc740 1 --2- 172.21.15.31:0/1527070992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7ac1d30 0x563fa7aec540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.764+0000 7f8965dbc740 1 --2- 172.21.15.31:0/1527070992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1380 0x563fa7ae0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.764+0000 7f8965dbc740 1 --2- 172.21.15.31:0/1527070992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7abd950 0x563fa7ad44c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.764+0000 7f8965dbc740 1 -- 172.21.15.31:0/1527070992 >> 172.21.15.31:0/1527070992 conn(0x563fa7ad09d0 msgr2=0x563fa7ad25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.764+0000 7f8965dbc740 1 -- 172.21.15.31:0/1527070992 shutdown_connections 2024-04-25T01:55:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.764+0000 7f8965dbc740 1 -- 172.21.15.31:0/1527070992 wait complete. 2024-04-25T01:55:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.764+0000 7f8965dbc740 1 -- start start 2024-04-25T01:55:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.764+0000 7f8965dbc740 1 Processor -- start 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7abd950 0x563fa7ad9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 0x563fa7ad97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8965dbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1d30 0x563fa7adc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8965dbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f895c1ac250 con 0x563fa7ac1380 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8965dbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f895c1abb80 con 0x563fa7ac1d30 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8960e41640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7abd950 0x563fa7ad9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8960e41640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7abd950 0x563fa7ad9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:34886/0 (socket says 172.21.15.31:34886) 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f895bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 0x563fa7ad97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8960e41640 1 -- 172.21.15.31:0/602301422 learned_addr learned my addr 172.21.15.31:0/602301422 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8965dbc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f895c1a8dc0 con 0x563fa7abd950 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8965dbc740 10 client.0 _ll_register_callbacks cb 0x7f89540099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8961642640 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1d30 0x563fa7adc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8960e41640 1 -- 172.21.15.31:0/602301422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1d30 msgr2=0x563fa7adc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8960e41640 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1d30 0x563fa7adc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8960e41640 1 -- 172.21.15.31:0/602301422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 msgr2=0x563fa7ad97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8960e41640 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 0x563fa7ad97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f8960e41640 1 -- 172.21.15.31:0/602301422 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fa7adcdf0 con 0x563fa7abd950 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.765+0000 7f895bfff640 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 0x563fa7ad97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f8960e41640 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7abd950 0x563fa7ad9270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f89500036e0 tx=0x7f8950005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57187346b0100776 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f893bfff640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f893bfff640 1 -- 172.21.15.31:0/602301422 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895001f3b0 con 0x563fa7abd950 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f893bfff640 1 -- 172.21.15.31:0/602301422 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f895001f550 con 0x563fa7abd950 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f8965dbc740 1 -- 172.21.15.31:0/602301422 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8950023030 con 0x563fa7abd950 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f893bfff640 1 -- 172.21.15.31:0/602301422 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f895001f750 con 0x563fa7abd950 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f8965dbc740 10 client.25360 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f8965dbc740 1 -- 172.21.15.31:0/602301422 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563fa7b4b880 con 0x563fa7abd950 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f893bfff640 1 -- 172.21.15.31:0/602301422 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8950023030 con 0x563fa7abd950 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f893bfff640 1 -- 172.21.15.31:0/602301422 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f895003b5a0 con 0x563fa7abd950 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f8965dbc740 10 client.25360 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f8965dbc740 10 client.25360 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f8965dbc740 20 client.25360 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f8965dbc740 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563fa7ad4e30 0x563fa7b4de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f8965dbc740 4 client.25360 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.766+0000 7f8965dbc740 1 -- 172.21.15.31:0/602301422 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563fa7a33bf0 con 0x563fa7ad4e30 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.767+0000 7f895bfff640 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563fa7ad4e30 0x563fa7b4de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.767+0000 7f893bfff640 10 client.25360 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.767+0000 7f895bfff640 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563fa7ad4e30 0x563fa7b4de50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d5ebfcd198b10b0 server_cookie=947f9074b3ce04b0 in_seq=0 out_seq=0 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.767+0000 7f893bfff640 1 -- 172.21.15.31:0/602301422 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563fa7a33bf0 con 0x563fa7ad4e30 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.767+0000 7f893bfff640 10 client.25360 handle_command_reply: tid=0 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.767+0000 7f8965dbc740 1 client.25360 shutdown 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f8965dbc740 1 -- 172.21.15.31:0/602301422 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7abd950 msgr2=0x563fa7ad9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f8965dbc740 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7abd950 0x563fa7ad9270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f89500036e0 tx=0x7f8950005740 comp rx=0 tx=0).stop 2024-04-25T01:55:38.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f8965dbc740 1 -- 172.21.15.31:0/602301422 shutdown_connections 2024-04-25T01:55:38.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f8965dbc740 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563fa7ad4e30 0x563fa7b4de50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f893bfff640 0 client.25360 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:38.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f8965dbc740 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563fa7ac1d30 0x563fa7adc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f8965dbc740 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fa7ac1380 0x563fa7ad97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f8965dbc740 1 --2- 172.21.15.31:0/602301422 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563fa7abd950 0x563fa7ad9270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f8965dbc740 1 -- 172.21.15.31:0/602301422 >> 172.21.15.31:0/602301422 conn(0x563fa7ad09d0 msgr2=0x563fa7abc260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:38.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f8965dbc740 1 -- 172.21.15.31:0/602301422 shutdown_connections 2024-04-25T01:55:38.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f8965dbc740 1 -- 172.21.15.31:0/602301422 wait complete. 2024-04-25T01:55:38.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.768+0000 7f8965dbc740 20 client.25360 trim_cache size 0 max 16384 2024-04-25T01:55:38.775 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:55:38.776 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T01:55:38.776 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T01:55:38.776 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:55:38.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 -- 172.21.15.31:0/3742308190 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f894c0a2ea0 msgr2=0x7f894c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f894c0a2ea0 0x7f894c0a5360 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8954005fd0 tx=0x7f8954009500 comp rx=0 tx=0).stop 2024-04-25T01:55:38.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 -- 172.21.15.31:0/3742308190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f895c100fd0 msgr2=0x7f895c19dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f895c100fd0 0x7f895c19dfa0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8948009850 tx=0x7f8948030650 comp rx=0 tx=0).stop 2024-04-25T01:55:38.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 -- 172.21.15.31:0/3742308190 shutdown_connections 2024-04-25T01:55:38.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f894c0a2ea0 0x7f894c0a5360 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f895c1066b0 0x7f895c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f895c100fd0 0x7f895c19dfa0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 --2- 172.21.15.31:0/3742308190 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f895c100610 0x7f895c19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 -- 172.21.15.31:0/3742308190 >> 172.21.15.31:0/3742308190 conn(0x7f895c077740 msgr2=0x7f895c068a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:38.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 -- 172.21.15.31:0/3742308190 shutdown_connections 2024-04-25T01:55:38.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.769+0000 7f89630a5640 1 -- 172.21.15.31:0/3742308190 wait complete. 2024-04-25T01:55:38.781 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:55:38.782 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:55:38.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.884+0000 7fabe1d1b640 1 -- 172.21.15.31:0/1509379871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabdc10bd10 msgr2=0x7fabdc10e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.884+0000 7fabe1d1b640 1 --2- 172.21.15.31:0/1509379871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabdc10bd10 0x7fabdc10e1b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009ca0 tx=0x7fabd402a9e0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.884+0000 7fabe1d1b640 1 -- 172.21.15.31:0/1509379871 shutdown_connections 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.884+0000 7fabe1d1b640 1 --2- 172.21.15.31:0/1509379871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabdc10bd10 0x7fabdc10e1b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.884+0000 7fabe1d1b640 1 --2- 172.21.15.31:0/1509379871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabdc1032f0 0x7fabdc10b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.884+0000 7fabe1d1b640 1 --2- 172.21.15.31:0/1509379871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabdc1029d0 0x7fabdc102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.884+0000 7fabe1d1b640 1 -- 172.21.15.31:0/1509379871 >> 172.21.15.31:0/1509379871 conn(0x7fabdc077740 msgr2=0x7fabdc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.884+0000 7fabe1d1b640 1 -- 172.21.15.31:0/1509379871 shutdown_connections 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.884+0000 7fabe1d1b640 1 -- 172.21.15.31:0/1509379871 wait complete. 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabe1d1b640 1 Processor -- start 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabe1d1b640 1 -- start start 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabe1d1b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabdc1029d0 0x7fabdc197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabe1d1b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabdc1032f0 0x7fabdc198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabe1d1b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabdc10bd10 0x7fabdc19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabe1d1b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc1119c0 con 0x7fabdc10bd10 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabe1d1b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc111840 con 0x7fabdc1032f0 2024-04-25T01:55:38.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabe1d1b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc111b40 con 0x7fabdc1029d0 2024-04-25T01:55:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabdbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabdc10bd10 0x7fabdc19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabdbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabdc10bd10 0x7fabdc19c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33400/0 (socket says 172.21.15.31:33400) 2024-04-25T01:55:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabdbfff640 1 -- 172.21.15.31:0/398571831 learned_addr learned my addr 172.21.15.31:0/398571831 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.885+0000 7fabdb7fe640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabdc1029d0 0x7fabdc197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabdaffd640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabdc1032f0 0x7fabdc198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabdbfff640 1 -- 172.21.15.31:0/398571831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabdc1029d0 msgr2=0x7fabdc197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabdbfff640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabdc1029d0 0x7fabdc197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabdbfff640 1 -- 172.21.15.31:0/398571831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabdc1032f0 msgr2=0x7fabdc198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabdbfff640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabdc1032f0 0x7fabdc198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabdbfff640 1 -- 172.21.15.31:0/398571831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabdc19ceb0 con 0x7fabdc10bd10 2024-04-25T01:55:38.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabdbfff640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabdc10bd10 0x7fabdc19c780 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fabd4006010 tx=0x7fabd402fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabd8ff9640 1 -- 172.21.15.31:0/398571831 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4033040 con 0x7fabdc10bd10 2024-04-25T01:55:38.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabe1d1b640 1 -- 172.21.15.31:0/398571831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabdc19d140 con 0x7fabdc10bd10 2024-04-25T01:55:38.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabe1d1b640 1 -- 172.21.15.31:0/398571831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabdc1a4a30 con 0x7fabdc10bd10 2024-04-25T01:55:38.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabd8ff9640 1 -- 172.21.15.31:0/398571831 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabd402e7a0 con 0x7fabdc10bd10 2024-04-25T01:55:38.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.886+0000 7fabd8ff9640 1 -- 172.21.15.31:0/398571831 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4043720 con 0x7fabdc10bd10 2024-04-25T01:55:38.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.887+0000 7fabe1d1b640 1 -- 172.21.15.31:0/398571831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabdc068560 con 0x7fabdc10bd10 2024-04-25T01:55:38.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.890+0000 7fabd8ff9640 1 -- 172.21.15.31:0/398571831 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7fabd4036070 con 0x7fabdc10bd10 2024-04-25T01:55:38.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.891+0000 7fabd8ff9640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabc40a2e20 0x7fabc40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.891+0000 7fabd8ff9640 1 -- 172.21.15.31:0/398571831 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fabd40f0b20 con 0x7fabdc10bd10 2024-04-25T01:55:38.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.891+0000 7fabdb7fe640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabc40a2e20 0x7fabc40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.892+0000 7fabdb7fe640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabc40a2e20 0x7fabc40a52e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabcc005fd0 tx=0x7fabcc00b390 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:38.894+0000 7fabd8ff9640 1 -- 172.21.15.31:0/398571831 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fabd403e070 con 0x7fabdc10bd10 2024-04-25T01:55:39.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.121+0000 7fabe1d1b640 1 -- 172.21.15.31:0/398571831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabdc106ef0 con 0x7fabdc10bd10 2024-04-25T01:55:39.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.122+0000 7fabd8ff9640 1 -- 172.21.15.31:0/398571831 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11923 (secure 0 0 0) 0x7fabd40bcef0 con 0x7fabdc10bd10 2024-04-25T01:55:39.124 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:55:39.124 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:54:28.592746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24451,24463,24503,24487,24473]},"id":1}]} 2024-04-25T01:55:39.124 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-04-25T01:55:39.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.124+0000 7fabe1d1b640 1 -- 172.21.15.31:0/398571831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabc40a2e20 msgr2=0x7fabc40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.124+0000 7fabe1d1b640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabc40a2e20 0x7fabc40a52e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabcc005fd0 tx=0x7fabcc00b390 comp rx=0 tx=0).stop 2024-04-25T01:55:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.124+0000 7fabe1d1b640 1 -- 172.21.15.31:0/398571831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabdc10bd10 msgr2=0x7fabdc19c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.124+0000 7fabe1d1b640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabdc10bd10 0x7fabdc19c780 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fabd4006010 tx=0x7fabd402fbe0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.124+0000 7fabe1d1b640 1 -- 172.21.15.31:0/398571831 shutdown_connections 2024-04-25T01:55:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.124+0000 7fabe1d1b640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabc40a2e20 0x7fabc40a52e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.124+0000 7fabe1d1b640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabdc10bd10 0x7fabdc19c780 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.124+0000 7fabe1d1b640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabdc1032f0 0x7fabdc198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.124+0000 7fabe1d1b640 1 --2- 172.21.15.31:0/398571831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabdc1029d0 0x7fabdc197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.124+0000 7fabe1d1b640 1 -- 172.21.15.31:0/398571831 >> 172.21.15.31:0/398571831 conn(0x7fabdc077740 msgr2=0x7fabdc0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.125+0000 7fabe1d1b640 1 -- 172.21.15.31:0/398571831 shutdown_connections 2024-04-25T01:55:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.125+0000 7fabe1d1b640 1 -- 172.21.15.31:0/398571831 wait complete. 2024-04-25T01:55:39.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:55:39.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.242+0000 7fc21c9e9640 1 -- 172.21.15.31:0/1004414036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218107730 msgr2=0x7fc218107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.242+0000 7fc21c9e9640 1 --2- 172.21.15.31:0/1004414036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218107730 0x7fc218107b10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc208009930 tx=0x7fc20802a420 comp rx=0 tx=0).stop 2024-04-25T01:55:39.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.242+0000 7fc21c9e9640 1 -- 172.21.15.31:0/1004414036 shutdown_connections 2024-04-25T01:55:39.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.242+0000 7fc21c9e9640 1 --2- 172.21.15.31:0/1004414036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2181020e0 0x7fc21810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.242+0000 7fc21c9e9640 1 --2- 172.21.15.31:0/1004414036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc218101720 0x7fc218101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.242+0000 7fc21c9e9640 1 --2- 172.21.15.31:0/1004414036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218107730 0x7fc218107b10 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.242+0000 7fc21c9e9640 1 -- 172.21.15.31:0/1004414036 >> 172.21.15.31:0/1004414036 conn(0x7fc218078b40 msgr2=0x7fc218078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.242+0000 7fc21c9e9640 1 -- 172.21.15.31:0/1004414036 shutdown_connections 2024-04-25T01:55:39.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.242+0000 7fc21c9e9640 1 -- 172.21.15.31:0/1004414036 wait complete. 2024-04-25T01:55:39.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.243+0000 7fc21c9e9640 1 Processor -- start 2024-04-25T01:55:39.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.243+0000 7fc21c9e9640 1 -- start start 2024-04-25T01:55:39.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.243+0000 7fc21c9e9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101720 0x7fc218073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.243+0000 7fc21c9e9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2181020e0 0x7fc218073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.243+0000 7fc21c9e9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc218107730 0x7fc2180744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.243+0000 7fc21c9e9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc218111a40 con 0x7fc218101720 2024-04-25T01:55:39.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc21c9e9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181118c0 con 0x7fc2181020e0 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc21c9e9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc218111bc0 con 0x7fc218107730 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc215d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2181020e0 0x7fc218073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc215d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2181020e0 0x7fc218073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56210/0 (socket says 172.21.15.31:56210) 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc21659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101720 0x7fc218073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc215d9b640 1 -- 172.21.15.31:0/3721461360 learned_addr learned my addr 172.21.15.31:0/3721461360 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc216d9d640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc218107730 0x7fc2180744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc21659c640 1 -- 172.21.15.31:0/3721461360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc218107730 msgr2=0x7fc2180744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc21659c640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc218107730 0x7fc2180744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc21659c640 1 -- 172.21.15.31:0/3721461360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2181020e0 msgr2=0x7fc218073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc21659c640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2181020e0 0x7fc218073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc21659c640 1 -- 172.21.15.31:0/3721461360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2181a6460 con 0x7fc218101720 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.244+0000 7fc21659c640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101720 0x7fc218073a40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc20802a940 tx=0x7fc208003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.245+0000 7fc2077fe640 1 -- 172.21.15.31:0/3721461360 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20803f070 con 0x7fc218101720 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.245+0000 7fc2077fe640 1 -- 172.21.15.31:0/3721461360 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc208031b60 con 0x7fc218101720 2024-04-25T01:55:39.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.245+0000 7fc21c9e9640 1 -- 172.21.15.31:0/3721461360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2181a66f0 con 0x7fc218101720 2024-04-25T01:55:39.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.245+0000 7fc2077fe640 1 -- 172.21.15.31:0/3721461360 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208035030 con 0x7fc218101720 2024-04-25T01:55:39.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.245+0000 7fc21c9e9640 1 -- 172.21.15.31:0/3721461360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2181a6c00 con 0x7fc218101720 2024-04-25T01:55:39.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.248+0000 7fc21f7cc740 1 Processor -- start 2024-04-25T01:55:39.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.248+0000 7fc2077fe640 1 -- 172.21.15.31:0/3721461360 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7fc208042070 con 0x7fc218101720 2024-04-25T01:55:39.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.248+0000 7fc21f7cc740 1 -- start start 2024-04-25T01:55:39.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.248+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 0x558d2047d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.248+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2047dea0 0x558d20488030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.248+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d20488570 0x558d2048a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.248+0000 7fc21f7cc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558d203e2670 con 0x558d20488570 2024-04-25T01:55:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.248+0000 7fc21f7cc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558d2038a970 con 0x558d2047dea0 2024-04-25T01:55:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.248+0000 7fc21f7cc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x558d203de630 con 0x558d2047d580 2024-04-25T01:55:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc215d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d20488570 0x558d2048a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc21659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2047dea0 0x558d20488030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc21659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2047dea0 0x558d20488030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56224/0 (socket says 172.21.15.31:56224) 2024-04-25T01:55:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc215d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d20488570 0x558d2048a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33432/0 (socket says 172.21.15.31:33432) 2024-04-25T01:55:39.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc21659c640 1 -- 172.21.15.31:0/632368499 learned_addr learned my addr 172.21.15.31:0/632368499 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc215d9b640 1 -- 172.21.15.31:0/632368499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 msgr2=0x558d2047d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc215d9b640 1 --2- 172.21.15.31:0/632368499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 0x558d2047d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc215d9b640 1 -- 172.21.15.31:0/632368499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2047dea0 msgr2=0x558d20488030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc215d9b640 1 --2- 172.21.15.31:0/632368499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2047dea0 0x558d20488030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc215d9b640 1 -- 172.21.15.31:0/632368499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d203ae400 con 0x558d20488570 2024-04-25T01:55:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc2077fe640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2000a2ea0 0x7fc2000a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc2077fe640 1 -- 172.21.15.31:0/3721461360 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fc2080efb40 con 0x7fc218101720 2024-04-25T01:55:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc216d9d640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2000a2ea0 0x7fc2000a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.249+0000 7fc215d9b640 1 --2- 172.21.15.31:0/632368499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d20488570 0x558d2048a960 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc2180754d0 tx=0x7fc20c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a61d8e41247b765 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.250+0000 7fc204ff9640 1 -- 172.21.15.31:0/632368499 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c004230 con 0x558d20488570 2024-04-25T01:55:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.250+0000 7fc204ff9640 1 -- 172.21.15.31:0/632368499 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc20c0043d0 con 0x558d20488570 2024-04-25T01:55:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.250+0000 7fc204ff9640 1 -- 172.21.15.31:0/632368499 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c004d50 con 0x558d20488570 2024-04-25T01:55:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 -- 172.21.15.31:0/632368499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d20488570 msgr2=0x558d2048a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc216d9d640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2000a2ea0 0x7fc2000a5360 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc210005fd0 tx=0x7fc210009a60 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/632368499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d20488570 0x558d2048a960 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc2180754d0 tx=0x7fc20c011810 comp rx=0 tx=0).stop 2024-04-25T01:55:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 -- 172.21.15.31:0/632368499 shutdown_connections 2024-04-25T01:55:39.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/632368499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d20488570 0x558d2048a960 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/632368499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2047dea0 0x558d20488030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/632368499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 0x558d2047d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 -- 172.21.15.31:0/632368499 >> 172.21.15.31:0/632368499 conn(0x558d204798a0 msgr2=0x558d20479c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 -- 172.21.15.31:0/632368499 shutdown_connections 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 -- 172.21.15.31:0/632368499 wait complete. 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 -- start start 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 Processor -- start 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.251+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047d580 0x558d20485430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2047dea0 0x558d20485970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047d580 0x558d20485430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047d580 0x558d20485430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33438/0 (socket says 172.21.15.31:33438) 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d20485eb0 0x558d20494850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21659c640 1 -- 172.21.15.31:0/2237907043 learned_addr learned my addr 172.21.15.31:0/2237907043 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2237907043 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558d2048b360 con 0x558d2047d580 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2237907043 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558d2039e050 con 0x558d2047dea0 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2237907043 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x558d2048b4e0 con 0x558d20485eb0 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21f7cc740 10 client.0 _ll_register_callbacks cb 0x558d200f5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21659c640 1 -- 172.21.15.31:0/2237907043 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d20485eb0 msgr2=0x558d20494850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21659c640 1 --2- 172.21.15.31:0/2237907043 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d20485eb0 0x558d20494850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21659c640 1 -- 172.21.15.31:0/2237907043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2047dea0 msgr2=0x558d20485970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21659c640 1 --2- 172.21.15.31:0/2237907043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2047dea0 0x558d20485970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21659c640 1 -- 172.21.15.31:0/2237907043 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d2036fdc0 con 0x558d2047d580 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21659c640 1 --2- 172.21.15.31:0/2237907043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047d580 0x558d20485430 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc208005ab0 tx=0x7fc2080f3240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e38d871333b378 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc1fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc1fdffb640 1 -- 172.21.15.31:0/2237907043 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208049070 con 0x558d2047d580 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc1fdffb640 1 -- 172.21.15.31:0/2237907043 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fc20803b4f0 con 0x558d2047d580 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.252+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2237907043 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc20c0043d0 con 0x558d2047d580 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc1fdffb640 1 -- 172.21.15.31:0/2237907043 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc2080bbdb0 con 0x558d2047d580 2024-04-25T01:55:39.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc21f7cc740 10 client.15852 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2237907043 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558d200f63f0 con 0x558d2047d580 2024-04-25T01:55:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc1fdffb640 1 -- 172.21.15.31:0/2237907043 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2080bbfe0 con 0x558d2047d580 2024-04-25T01:55:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc1fdffb640 1 -- 172.21.15.31:0/2237907043 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc2080bf030 con 0x558d2047d580 2024-04-25T01:55:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc21f7cc740 10 client.15852 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc21f7cc740 10 client.15852 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:55:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc21f7cc740 20 client.15852 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/2237907043 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x558d20498c60 0x558d2049b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc21f7cc740 4 client.15852 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2237907043 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc20c0043d0 con 0x558d20498c60 2024-04-25T01:55:39.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.253+0000 7fc215d9b640 1 --2- 172.21.15.31:0/2237907043 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x558d20498c60 0x558d2049b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.254+0000 7fc215d9b640 1 --2- 172.21.15.31:0/2237907043 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x558d20498c60 0x558d2049b0f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=50b8dba96dd894e5 server_cookie=80fbde7e2bae6fc in_seq=0 out_seq=0 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.255+0000 7fc1fdffb640 10 client.15852 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc1fdffb640 1 -- 172.21.15.31:0/2237907043 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fc20c0043d0 con 0x558d20498c60 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc1fdffb640 10 client.15852 handle_command_reply: tid=0 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 1 client.15852 shutdown 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2237907043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047d580 msgr2=0x558d20485430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/2237907043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047d580 0x558d20485430 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc208005ab0 tx=0x7fc2080f3240 comp rx=0 tx=0).stop 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2237907043 shutdown_connections 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/2237907043 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x558d20498c60 0x558d2049b0f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/2237907043 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d20485eb0 0x558d20494850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/2237907043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d2047dea0 0x558d20485970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/2237907043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047d580 0x558d20485430 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2237907043 >> 172.21.15.31:0/2237907043 conn(0x558d204798a0 msgr2=0x558d20478b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc1fdffb640 0 client.15852 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:55:39.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2237907043 shutdown_connections 2024-04-25T01:55:39.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2237907043 wait complete. 2024-04-25T01:55:39.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.256+0000 7fc21f7cc740 20 client.15852 trim_cache size 0 max 16384 2024-04-25T01:55:39.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc21f7cc740 1 Processor -- start 2024-04-25T01:55:39.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc21f7cc740 1 -- start start 2024-04-25T01:55:39.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 0x558d204792c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 0x558d204942a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 0x558d204947e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc21f7cc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181af530 con 0x558d2047dea0 2024-04-25T01:55:39.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc21f7cc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181b1070 con 0x558d204798a0 2024-04-25T01:55:39.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc21f7cc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181afc00 con 0x558d2047d580 2024-04-25T01:55:39.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc216d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 0x558d204947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc216d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 0x558d204947e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33440/0 (socket says 172.21.15.31:33440) 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc216d9d640 1 -- 172.21.15.31:0/2054906782 learned_addr learned my addr 172.21.15.31:0/2054906782 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc215d9b640 1 --2- 172.21.15.31:0/2054906782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 0x558d204942a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.270+0000 7fc21659c640 1 --2- 172.21.15.31:0/2054906782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 0x558d204792c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.271+0000 7fc216d9d640 1 -- 172.21.15.31:0/2054906782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 msgr2=0x558d204942a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.271+0000 7fc216d9d640 1 --2- 172.21.15.31:0/2054906782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 0x558d204942a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.271+0000 7fc216d9d640 1 -- 172.21.15.31:0/2054906782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 msgr2=0x558d204792c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.271+0000 7fc216d9d640 1 --2- 172.21.15.31:0/2054906782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 0x558d204792c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.271+0000 7fc216d9d640 1 -- 172.21.15.31:0/2054906782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d201c07d0 con 0x558d2047dea0 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.271+0000 7fc216d9d640 1 --2- 172.21.15.31:0/2054906782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 0x558d204947e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc210011000 tx=0x7fc210009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3028610a9588aca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.271+0000 7fc1fffff640 1 -- 172.21.15.31:0/2054906782 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210023070 con 0x558d2047dea0 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.271+0000 7fc1fffff640 1 -- 172.21.15.31:0/2054906782 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc210012e20 con 0x558d2047dea0 2024-04-25T01:55:39.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.271+0000 7fc1fffff640 1 -- 172.21.15.31:0/2054906782 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210011a40 con 0x558d2047dea0 2024-04-25T01:55:39.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.272+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2054906782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 msgr2=0x558d204947e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.272+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/2054906782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 0x558d204947e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc210011000 tx=0x7fc210009600 comp rx=0 tx=0).stop 2024-04-25T01:55:39.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.272+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2054906782 shutdown_connections 2024-04-25T01:55:39.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.272+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/2054906782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 0x558d204947e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.272+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/2054906782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 0x558d204942a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.272+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/2054906782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 0x558d204792c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.272+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2054906782 >> 172.21.15.31:0/2054906782 conn(0x558d20485ad0 msgr2=0x558d20486c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.272+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2054906782 shutdown_connections 2024-04-25T01:55:39.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.272+0000 7fc21f7cc740 1 -- 172.21.15.31:0/2054906782 wait complete. 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21f7cc740 1 -- start start 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21f7cc740 1 Processor -- start 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 0x558d2048d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 0x558d2048deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21f7cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 0x558d204fd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21f7cc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181b09a0 con 0x558d2047dea0 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21f7cc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181b02d0 con 0x558d204798a0 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21f7cc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181ad510 con 0x558d2047d580 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21f7cc740 10 client.0 _ll_register_callbacks cb 0x7fc208005760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 0x558d2048d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 0x558d2048d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56258/0 (socket says 172.21.15.31:56258) 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc21659c640 1 -- 172.21.15.31:0/3875416960 learned_addr learned my addr 172.21.15.31:0/3875416960 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.273+0000 7fc216d9d640 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 0x558d204fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc21659c640 1 -- 172.21.15.31:0/3875416960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 msgr2=0x558d2048deb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc21659c640 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 0x558d2048deb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc21659c640 1 -- 172.21.15.31:0/3875416960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 msgr2=0x558d204fd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc21659c640 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 0x558d204fd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc21659c640 1 -- 172.21.15.31:0/3875416960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d204fd7e0 con 0x558d204798a0 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc216d9d640 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 0x558d204fd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc21659c640 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 0x558d2048d970 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc2080f32b0 tx=0x7fc20803b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbd7c7b3b794aa7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc1fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc1fdffb640 1 -- 172.21.15.31:0/3875416960 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208049070 con 0x558d204798a0 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc1fdffb640 1 -- 172.21.15.31:0/3875416960 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fc2080dad70 con 0x558d204798a0 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc21f7cc740 1 -- 172.21.15.31:0/3875416960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc210012e20 con 0x558d204798a0 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc1fdffb640 1 -- 172.21.15.31:0/3875416960 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc2080bb940 con 0x558d204798a0 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc21f7cc740 10 client.25466 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.274+0000 7fc21f7cc740 1 -- 172.21.15.31:0/3875416960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558d204fd7e0 con 0x558d204798a0 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.275+0000 7fc1fdffb640 1 -- 172.21.15.31:0/3875416960 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc210012e20 con 0x558d204798a0 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.275+0000 7fc1fdffb640 1 -- 172.21.15.31:0/3875416960 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc2080bf070 con 0x558d204798a0 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.275+0000 7fc21f7cc740 10 client.25466 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:39.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.275+0000 7fc21f7cc740 10 client.25466 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.275+0000 7fc21f7cc740 20 client.25466 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.275+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x558d20485eb0 0x558d20500f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.275+0000 7fc21f7cc740 4 client.25466 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.275+0000 7fc21f7cc740 1 -- 172.21.15.31:0/3875416960 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558d203efbb0 con 0x558d20485eb0 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.275+0000 7fc215d9b640 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x558d20485eb0 0x558d20500f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc1fdffb640 10 client.25466 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc215d9b640 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x558d20485eb0 0x558d20500f00 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f52d6ba59d3a9bf7 server_cookie=e03b0fe00a35eee4 in_seq=0 out_seq=0 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc1fdffb640 1 -- 172.21.15.31:0/3875416960 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558d203efbb0 con 0x558d20485eb0 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc1fdffb640 10 client.25466 handle_command_reply: tid=0 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 1 client.25466 shutdown 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 1 -- 172.21.15.31:0/3875416960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 msgr2=0x558d2048d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 0x558d2048d970 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc2080f32b0 tx=0x7fc20803b720 comp rx=0 tx=0).stop 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 1 -- 172.21.15.31:0/3875416960 shutdown_connections 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x558d20485eb0 0x558d20500f00 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc1fdffb640 0 client.25466 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:55:39.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558d2047dea0 0x558d204fd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d2047d580 0x558d2048deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 1 --2- 172.21.15.31:0/3875416960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d204798a0 0x558d2048d970 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 1 -- 172.21.15.31:0/3875416960 >> 172.21.15.31:0/3875416960 conn(0x558d20485ad0 msgr2=0x558d20486c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 1 -- 172.21.15.31:0/3875416960 shutdown_connections 2024-04-25T01:55:39.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 1 -- 172.21.15.31:0/3875416960 wait complete. 2024-04-25T01:55:39.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.276+0000 7fc21f7cc740 20 client.25466 trim_cache size 0 max 16384 2024-04-25T01:55:39.283 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:55:39.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 -- 172.21.15.31:0/3721461360 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2000a2ea0 msgr2=0x7fc2000a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2000a2ea0 0x7fc2000a5360 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc210005fd0 tx=0x7fc210009a60 comp rx=0 tx=0).stop 2024-04-25T01:55:39.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 -- 172.21.15.31:0/3721461360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101720 msgr2=0x7fc218073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101720 0x7fc218073a40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc20802a940 tx=0x7fc208003c10 comp rx=0 tx=0).stop 2024-04-25T01:55:39.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 -- 172.21.15.31:0/3721461360 shutdown_connections 2024-04-25T01:55:39.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2000a2ea0 0x7fc2000a5360 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc218107730 0x7fc2180744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2181020e0 0x7fc218073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 --2- 172.21.15.31:0/3721461360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc218101720 0x7fc218073a40 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 -- 172.21.15.31:0/3721461360 >> 172.21.15.31:0/3721461360 conn(0x7fc218078b40 msgr2=0x7fc218104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 -- 172.21.15.31:0/3721461360 shutdown_connections 2024-04-25T01:55:39.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.277+0000 7fc21c9e9640 1 -- 172.21.15.31:0/3721461360 wait complete. 2024-04-25T01:55:39.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:55:39.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.393+0000 7fe044761640 1 -- 172.21.15.31:0/2636346588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe03c1029c0 msgr2=0x7fe03c102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.393+0000 7fe044761640 1 --2- 172.21.15.31:0/2636346588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe03c1029c0 0x7fe03c102da0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe038009900 tx=0x7fe03802a510 comp rx=0 tx=0).stop 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.393+0000 7fe044761640 1 -- 172.21.15.31:0/2636346588 shutdown_connections 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.393+0000 7fe044761640 1 --2- 172.21.15.31:0/2636346588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe03c10bd00 0x7fe03c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.393+0000 7fe044761640 1 --2- 172.21.15.31:0/2636346588 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe03c1032e0 0x7fe03c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.393+0000 7fe044761640 1 --2- 172.21.15.31:0/2636346588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe03c1029c0 0x7fe03c102da0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.393+0000 7fe044761640 1 -- 172.21.15.31:0/2636346588 >> 172.21.15.31:0/2636346588 conn(0x7fe03c077740 msgr2=0x7fe03c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.393+0000 7fe044761640 1 -- 172.21.15.31:0/2636346588 shutdown_connections 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.393+0000 7fe044761640 1 -- 172.21.15.31:0/2636346588 wait complete. 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.393+0000 7fe044761640 1 Processor -- start 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.393+0000 7fe044761640 1 -- start start 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe044761640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe03c1029c0 0x7fe03c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe044761640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe03c1032e0 0x7fe03c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe044761640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe03c10bd00 0x7fe03c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe044761640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c1119b0 con 0x7fe03c1029c0 2024-04-25T01:55:39.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe044761640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c111830 con 0x7fe03c1032e0 2024-04-25T01:55:39.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe044761640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c111b30 con 0x7fe03c10bd00 2024-04-25T01:55:39.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe0424fd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe03c1029c0 0x7fe03c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe0424fd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe03c1029c0 0x7fe03c197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33474/0 (socket says 172.21.15.31:33474) 2024-04-25T01:55:39.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe0424fd640 1 -- 172.21.15.31:0/3346315780 learned_addr learned my addr 172.21.15.31:0/3346315780 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe041cfc640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe03c1032e0 0x7fe03c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe042cfe640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe03c10bd00 0x7fe03c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe0424fd640 1 -- 172.21.15.31:0/3346315780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe03c10bd00 msgr2=0x7fe03c19c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe0424fd640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe03c10bd00 0x7fe03c19c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe0424fd640 1 -- 172.21.15.31:0/3346315780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe03c1032e0 msgr2=0x7fe03c1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe0424fd640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe03c1032e0 0x7fe03c1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe0424fd640 1 -- 172.21.15.31:0/3346315780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03c19cf50 con 0x7fe03c1029c0 2024-04-25T01:55:39.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe042cfe640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe03c10bd00 0x7fe03c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:39.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe041cfc640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe03c1032e0 0x7fe03c1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:39.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.394+0000 7fe0424fd640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe03c1029c0 0x7fe03c197f60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe038005940 tx=0x7fe038004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.395+0000 7fe0337fe640 1 -- 172.21.15.31:0/3346315780 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03803f030 con 0x7fe03c1029c0 2024-04-25T01:55:39.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.395+0000 7fe0337fe640 1 -- 172.21.15.31:0/3346315780 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe038003aa0 con 0x7fe03c1029c0 2024-04-25T01:55:39.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.395+0000 7fe044761640 1 -- 172.21.15.31:0/3346315780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe03c19d1e0 con 0x7fe03c1029c0 2024-04-25T01:55:39.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.395+0000 7fe0337fe640 1 -- 172.21.15.31:0/3346315780 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe038035070 con 0x7fe03c1029c0 2024-04-25T01:55:39.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.395+0000 7fe044761640 1 -- 172.21.15.31:0/3346315780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe03c19d660 con 0x7fe03c1029c0 2024-04-25T01:55:39.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.397+0000 7fe0337fe640 1 -- 172.21.15.31:0/3346315780 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7fe038042030 con 0x7fe03c1029c0 2024-04-25T01:55:39.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.398+0000 7fe04746e740 1 Processor -- start 2024-04-25T01:55:39.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.398+0000 7fe04746e740 1 -- start start 2024-04-25T01:55:39.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.398+0000 7fe04746e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945e580 0x55a78945e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.398+0000 7fe04746e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945eea0 0x55a789469030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.398+0000 7fe04746e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a789469570 0x55a78946b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.398+0000 7fe04746e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a7893c3670 con 0x55a789469570 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.398+0000 7fe04746e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a78936b970 con 0x55a78945eea0 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.398+0000 7fe04746e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a7893bf630 con 0x55a78945e580 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.398+0000 7fe0424fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945eea0 0x55a789469030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.398+0000 7fe041cfc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a789469570 0x55a78946b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe0337fe640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0280a2ea0 0x7fe0280a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe0424fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945eea0 0x55a789469030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56278/0 (socket says 172.21.15.31:56278) 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe0424fd640 1 -- 172.21.15.31:0/513372600 learned_addr learned my addr 172.21.15.31:0/513372600 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe041cfc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a789469570 0x55a78946b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33490/0 (socket says 172.21.15.31:33490) 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe042cfe640 1 --2- 172.21.15.31:0/513372600 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945e580 0x55a78945e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe042cfe640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0280a2ea0 0x7fe0280a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe0424fd640 1 -- 172.21.15.31:0/513372600 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945e580 msgr2=0x55a78945e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe0424fd640 1 --2- 172.21.15.31:0/513372600 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945e580 0x55a78945e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe0424fd640 1 -- 172.21.15.31:0/513372600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a789469570 msgr2=0x55a78946b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe0424fd640 1 --2- 172.21.15.31:0/513372600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a789469570 0x55a78946b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe0424fd640 1 -- 172.21.15.31:0/513372600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a78938f400 con 0x55a78945eea0 2024-04-25T01:55:39.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe041cfc640 1 --2- 172.21.15.31:0/513372600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a789469570 0x55a78946b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:39.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe0337fe640 1 -- 172.21.15.31:0/3346315780 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fe0380efe00 con 0x7fe03c1029c0 2024-04-25T01:55:39.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe0424fd640 1 --2- 172.21.15.31:0/513372600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945eea0 0x55a789469030 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0380ed450 tx=0x7fe0380b1620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfc4fa7ffb235f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.399+0000 7fe042cfe640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0280a2ea0 0x7fe0280a5360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe034009370 tx=0x7fe034009240 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.400+0000 7fe030ff9640 1 -- 172.21.15.31:0/513372600 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe038049070 con 0x55a78945eea0 2024-04-25T01:55:39.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.400+0000 7fe030ff9640 1 -- 172.21.15.31:0/513372600 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe03803b460 con 0x55a78945eea0 2024-04-25T01:55:39.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.400+0000 7fe030ff9640 1 -- 172.21.15.31:0/513372600 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe038087220 con 0x55a78945eea0 2024-04-25T01:55:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.400+0000 7fe04746e740 1 -- 172.21.15.31:0/513372600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945eea0 msgr2=0x55a789469030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.400+0000 7fe04746e740 1 --2- 172.21.15.31:0/513372600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945eea0 0x55a789469030 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0380ed450 tx=0x7fe0380b1620 comp rx=0 tx=0).stop 2024-04-25T01:55:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.400+0000 7fe04746e740 1 -- 172.21.15.31:0/513372600 shutdown_connections 2024-04-25T01:55:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.400+0000 7fe04746e740 1 --2- 172.21.15.31:0/513372600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a789469570 0x55a78946b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.400+0000 7fe04746e740 1 --2- 172.21.15.31:0/513372600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945eea0 0x55a789469030 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.400+0000 7fe04746e740 1 --2- 172.21.15.31:0/513372600 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945e580 0x55a78945e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.400+0000 7fe04746e740 1 -- 172.21.15.31:0/513372600 >> 172.21.15.31:0/513372600 conn(0x55a78945a8a0 msgr2=0x55a78945ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe04746e740 1 -- 172.21.15.31:0/513372600 shutdown_connections 2024-04-25T01:55:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe04746e740 1 -- 172.21.15.31:0/513372600 wait complete. 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe04746e740 1 -- start start 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe04746e740 1 Processor -- start 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe04746e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945e580 0x55a789468e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe04746e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 0x55a789466160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe04746e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a789469570 0x55a7894666a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe04746e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a78946c120 con 0x55a78945eea0 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe04746e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a78937f050 con 0x55a78945e580 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe04746e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a78946c2a0 con 0x55a789469570 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe04746e740 10 client.0 _ll_register_callbacks cb 0x55a7890d6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.401+0000 7fe041cfc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 0x55a789466160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe041cfc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 0x55a789466160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33498/0 (socket says 172.21.15.31:33498) 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe041cfc640 1 -- 172.21.15.31:0/2160868335 learned_addr learned my addr 172.21.15.31:0/2160868335 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe042cfe640 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a789469570 0x55a7894666a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe0424fd640 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945e580 0x55a789468e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe041cfc640 1 -- 172.21.15.31:0/2160868335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a789469570 msgr2=0x55a7894666a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe041cfc640 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a789469570 0x55a7894666a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe041cfc640 1 -- 172.21.15.31:0/2160868335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945e580 msgr2=0x55a789468e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe041cfc640 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945e580 0x55a789468e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe041cfc640 1 -- 172.21.15.31:0/2160868335 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a789350dc0 con 0x55a78945eea0 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe042cfe640 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a789469570 0x55a7894666a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe041cfc640 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 0x55a789466160 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe02c00bfc0 tx=0x7fe02c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc6b1581431a65b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe025ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe025ffb640 1 -- 172.21.15.31:0/2160868335 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02c012650 con 0x55a78945eea0 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe025ffb640 1 -- 172.21.15.31:0/2160868335 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fe02c0127f0 con 0x55a78945eea0 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.402+0000 7fe04746e740 1 -- 172.21.15.31:0/2160868335 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a7893d0bb0 con 0x55a78945eea0 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.403+0000 7fe025ffb640 1 -- 172.21.15.31:0/2160868335 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe02c0129f0 con 0x55a78945eea0 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.403+0000 7fe04746e740 10 client.15870 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.403+0000 7fe04746e740 1 -- 172.21.15.31:0/2160868335 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7890d73f0 con 0x55a78945eea0 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.403+0000 7fe025ffb640 1 -- 172.21.15.31:0/2160868335 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a7893d0bb0 con 0x55a78945eea0 2024-04-25T01:55:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.403+0000 7fe025ffb640 1 -- 172.21.15.31:0/2160868335 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe02c01a3d0 con 0x55a78945eea0 2024-04-25T01:55:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.403+0000 7fe04746e740 10 client.15870 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.403+0000 7fe04746e740 10 client.15870 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:55:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.403+0000 7fe04746e740 20 client.15870 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.403+0000 7fe04746e740 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a789479310 0x55a78947b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.403+0000 7fe04746e740 4 client.15870 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.403+0000 7fe04746e740 1 -- 172.21.15.31:0/2160868335 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7893caf00 con 0x55a789479310 2024-04-25T01:55:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.404+0000 7fe0424fd640 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a789479310 0x55a78947b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.404+0000 7fe025ffb640 10 client.15870 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:55:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.404+0000 7fe0424fd640 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a789479310 0x55a78947b770 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a4718c6723ae00a4 server_cookie=bd9473d5c432cd83 in_seq=0 out_seq=0 2024-04-25T01:55:39.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.405+0000 7fe025ffb640 1 -- 172.21.15.31:0/2160868335 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7893caf00 con 0x55a789479310 2024-04-25T01:55:39.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.405+0000 7fe025ffb640 10 client.15870 handle_command_reply: tid=0 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.405+0000 7fe04746e740 1 client.15870 shutdown 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.405+0000 7fe04746e740 1 -- 172.21.15.31:0/2160868335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 msgr2=0x55a789466160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.405+0000 7fe04746e740 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 0x55a789466160 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe02c00bfc0 tx=0x7fe02c0117b0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.406+0000 7fe04746e740 1 -- 172.21.15.31:0/2160868335 shutdown_connections 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.406+0000 7fe04746e740 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a789479310 0x55a78947b770 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.406+0000 7fe025ffb640 0 client.15870 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.406+0000 7fe04746e740 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a789469570 0x55a7894666a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.406+0000 7fe04746e740 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 0x55a789466160 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.406+0000 7fe04746e740 1 --2- 172.21.15.31:0/2160868335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a78945e580 0x55a789468e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.406+0000 7fe04746e740 1 -- 172.21.15.31:0/2160868335 >> 172.21.15.31:0/2160868335 conn(0x55a78945a8a0 msgr2=0x55a789459b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.406+0000 7fe04746e740 1 -- 172.21.15.31:0/2160868335 shutdown_connections 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.406+0000 7fe04746e740 1 -- 172.21.15.31:0/2160868335 wait complete. 2024-04-25T01:55:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.406+0000 7fe04746e740 20 client.15870 trim_cache size 0 max 16384 2024-04-25T01:55:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.419+0000 7fe04746e740 1 Processor -- start 2024-04-25T01:55:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.419+0000 7fe04746e740 1 -- start start 2024-04-25T01:55:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.419+0000 7fe04746e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945e580 0x55a7894689a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe04746e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 0x55a7894603e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe04746e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a789469570 0x55a789460920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe04746e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c1af490 con 0x55a78945eea0 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe04746e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c1b0fd0 con 0x55a789469570 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe04746e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c1afb60 con 0x55a78945e580 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe041cfc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 0x55a7894603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe041cfc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 0x55a7894603e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33510/0 (socket says 172.21.15.31:33510) 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe041cfc640 1 -- 172.21.15.31:0/2002886657 learned_addr learned my addr 172.21.15.31:0/2002886657 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe0424fd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945e580 0x55a7894689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe042cfe640 1 --2- 172.21.15.31:0/2002886657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a789469570 0x55a789460920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe0424fd640 1 -- 172.21.15.31:0/2002886657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a789469570 msgr2=0x55a789460920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe0424fd640 1 --2- 172.21.15.31:0/2002886657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a789469570 0x55a789460920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe0424fd640 1 -- 172.21.15.31:0/2002886657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 msgr2=0x55a7894603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe0424fd640 1 --2- 172.21.15.31:0/2002886657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 0x55a7894603e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe0424fd640 1 -- 172.21.15.31:0/2002886657 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7891a17d0 con 0x55a78945e580 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.420+0000 7fe0424fd640 1 --2- 172.21.15.31:0/2002886657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945e580 0x55a7894689a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe0380f00f0 tx=0x7fe0380b8560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ccc76b1d3d8bad4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe027fff640 1 -- 172.21.15.31:0/2002886657 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe038049040 con 0x55a78945e580 2024-04-25T01:55:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe027fff640 1 -- 172.21.15.31:0/2002886657 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe03803be20 con 0x55a78945e580 2024-04-25T01:55:39.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe027fff640 1 -- 172.21.15.31:0/2002886657 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe038003c50 con 0x55a78945e580 2024-04-25T01:55:39.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe04746e740 1 -- 172.21.15.31:0/2002886657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945e580 msgr2=0x55a7894689a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe04746e740 1 --2- 172.21.15.31:0/2002886657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945e580 0x55a7894689a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe0380f00f0 tx=0x7fe0380b8560 comp rx=0 tx=0).stop 2024-04-25T01:55:39.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe04746e740 1 -- 172.21.15.31:0/2002886657 shutdown_connections 2024-04-25T01:55:39.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe04746e740 1 --2- 172.21.15.31:0/2002886657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a789469570 0x55a789460920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe04746e740 1 --2- 172.21.15.31:0/2002886657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945eea0 0x55a7894603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe04746e740 1 --2- 172.21.15.31:0/2002886657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945e580 0x55a7894689a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe04746e740 1 -- 172.21.15.31:0/2002886657 >> 172.21.15.31:0/2002886657 conn(0x55a78945a8a0 msgr2=0x55a78945fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe04746e740 1 -- 172.21.15.31:0/2002886657 shutdown_connections 2024-04-25T01:55:39.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.421+0000 7fe04746e740 1 -- 172.21.15.31:0/2002886657 wait complete. 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe04746e740 1 -- start start 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe04746e740 1 Processor -- start 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe04746e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945e580 0x55a789471360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe04746e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945eea0 0x55a7894718a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe04746e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a789469570 0x55a78946ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe0424fd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945e580 0x55a789471360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe0424fd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945e580 0x55a789471360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33516/0 (socket says 172.21.15.31:33516) 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe04746e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c1b0900 con 0x55a78945e580 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe0424fd640 1 -- 172.21.15.31:0/3523684846 learned_addr learned my addr 172.21.15.31:0/3523684846 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe04746e740 1 -- 172.21.15.31:0/3523684846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c1b0230 con 0x55a789469570 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe04746e740 1 -- 172.21.15.31:0/3523684846 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c1ad470 con 0x55a78945eea0 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe041cfc640 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945eea0 0x55a7894718a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.422+0000 7fe04746e740 10 client.0 _ll_register_callbacks cb 0x7fe02c00b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe042cfe640 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a789469570 0x55a78946ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe041cfc640 1 -- 172.21.15.31:0/3523684846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a789469570 msgr2=0x55a78946ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe041cfc640 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a789469570 0x55a78946ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe041cfc640 1 -- 172.21.15.31:0/3523684846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945e580 msgr2=0x55a789471360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe041cfc640 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945e580 0x55a789471360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe041cfc640 1 -- 172.21.15.31:0/3523684846 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a78946d230 con 0x55a78945eea0 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe042cfe640 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a789469570 0x55a78946ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe0424fd640 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945e580 0x55a789471360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe041cfc640 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945eea0 0x55a7894718a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55a789409a60 tx=0x7fe02c00bb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c55dc56709221fb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe025ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe025ffb640 1 -- 172.21.15.31:0/3523684846 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02c028040 con 0x55a78945eea0 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe04746e740 1 -- 172.21.15.31:0/3523684846 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe03803be20 con 0x55a78945eea0 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe025ffb640 1 -- 172.21.15.31:0/3523684846 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fe02c012d90 con 0x55a78945eea0 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe025ffb640 1 -- 172.21.15.31:0/3523684846 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe02c01aab0 con 0x55a78945eea0 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe04746e740 10 client.25393 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:39.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.423+0000 7fe04746e740 1 -- 172.21.15.31:0/3523684846 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7894df510 con 0x55a78945eea0 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.424+0000 7fe025ffb640 1 -- 172.21.15.31:0/3523684846 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe03803be20 con 0x55a78945eea0 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.424+0000 7fe025ffb640 1 -- 172.21.15.31:0/3523684846 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe02c01ac50 con 0x55a78945eea0 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.424+0000 7fe04746e740 10 client.25393 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.424+0000 7fe04746e740 10 client.25393 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.424+0000 7fe04746e740 20 client.25393 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.424+0000 7fe04746e740 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a789479310 0x55a7894e1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.424+0000 7fe04746e740 4 client.25393 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.424+0000 7fe04746e740 1 -- 172.21.15.31:0/3523684846 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a789348820 con 0x55a789479310 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe0424fd640 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a789479310 0x55a7894e1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe0424fd640 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a789479310 0x55a7894e1c90 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a5dadb6bef8e6889 server_cookie=d0c20ce6a4b514c0 in_seq=0 out_seq=0 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe025ffb640 10 client.25393 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe025ffb640 1 -- 172.21.15.31:0/3523684846 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a789348820 con 0x55a789479310 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe025ffb640 10 client.25393 handle_command_reply: tid=0 2024-04-25T01:55:39.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe04746e740 1 client.25393 shutdown 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe04746e740 1 -- 172.21.15.31:0/3523684846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945eea0 msgr2=0x55a7894718a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe04746e740 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945eea0 0x55a7894718a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55a789409a60 tx=0x7fe02c00bb80 comp rx=0 tx=0).stop 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe04746e740 1 -- 172.21.15.31:0/3523684846 shutdown_connections 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe04746e740 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a789479310 0x55a7894e1c90 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe04746e740 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a789469570 0x55a78946ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe04746e740 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a78945eea0 0x55a7894718a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe04746e740 1 --2- 172.21.15.31:0/3523684846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a78945e580 0x55a789471360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe04746e740 1 -- 172.21.15.31:0/3523684846 >> 172.21.15.31:0/3523684846 conn(0x55a78945a8a0 msgr2=0x55a789460560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe025ffb640 0 client.25393 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe04746e740 1 -- 172.21.15.31:0/3523684846 shutdown_connections 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.425+0000 7fe04746e740 1 -- 172.21.15.31:0/3523684846 wait complete. 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe04746e740 20 client.25393 trim_cache size 0 max 16384 2024-04-25T01:55:39.432 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe044761640 1 -- 172.21.15.31:0/3346315780 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0280a2ea0 msgr2=0x7fe0280a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe044761640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0280a2ea0 0x7fe0280a5360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe034009370 tx=0x7fe034009240 comp rx=0 tx=0).stop 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe044761640 1 -- 172.21.15.31:0/3346315780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe03c1029c0 msgr2=0x7fe03c197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe044761640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe03c1029c0 0x7fe03c197f60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe038005940 tx=0x7fe038004270 comp rx=0 tx=0).stop 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe044761640 1 -- 172.21.15.31:0/3346315780 shutdown_connections 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe044761640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0280a2ea0 0x7fe0280a5360 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe044761640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe03c10bd00 0x7fe03c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe044761640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe03c1032e0 0x7fe03c1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe044761640 1 --2- 172.21.15.31:0/3346315780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe03c1029c0 0x7fe03c197f60 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe044761640 1 -- 172.21.15.31:0/3346315780 >> 172.21.15.31:0/3346315780 conn(0x7fe03c077740 msgr2=0x7fe03c0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.426+0000 7fe044761640 1 -- 172.21.15.31:0/3346315780 shutdown_connections 2024-04-25T01:55:39.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.427+0000 7fe044761640 1 -- 172.21.15.31:0/3346315780 wait complete. 2024-04-25T01:55:39.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:55:39.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.543+0000 7f0daba9e640 1 -- 172.21.15.31:0/25072658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da4100600 msgr2=0x7f0da410e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.543+0000 7f0daba9e640 1 --2- 172.21.15.31:0/25072658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da4100600 0x7f0da410e5e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c009cd0 tx=0x7f0d9c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.543+0000 7f0daba9e640 1 -- 172.21.15.31:0/25072658 shutdown_connections 2024-04-25T01:55:39.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.543+0000 7f0daba9e640 1 --2- 172.21.15.31:0/25072658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da4100600 0x7f0da410e5e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.543+0000 7f0daba9e640 1 --2- 172.21.15.31:0/25072658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da40ffc40 0x7f0da41000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.543+0000 7f0daba9e640 1 --2- 172.21.15.31:0/25072658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0da4107720 0x7f0da4107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.543+0000 7f0daba9e640 1 -- 172.21.15.31:0/25072658 >> 172.21.15.31:0/25072658 conn(0x7f0da4076bd0 msgr2=0x7f0da40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.543+0000 7f0daba9e640 1 -- 172.21.15.31:0/25072658 shutdown_connections 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.543+0000 7f0daba9e640 1 -- 172.21.15.31:0/25072658 wait complete. 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.544+0000 7f0daba9e640 1 Processor -- start 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.544+0000 7f0daba9e640 1 -- start start 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.544+0000 7f0daba9e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0da40ffc40 0x7f0da4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.544+0000 7f0daba9e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da4100600 0x7f0da4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.544+0000 7f0daba9e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da4107720 0x7f0da419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.544+0000 7f0daba9e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0da41119e0 con 0x7f0da4107720 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.544+0000 7f0daba9e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0da4111860 con 0x7f0da40ffc40 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.544+0000 7f0daba9e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0da4111b60 con 0x7f0da4100600 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.544+0000 7f0daa03b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da4107720 0x7f0da419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.544+0000 7f0da983a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0da40ffc40 0x7f0da4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.544+0000 7f0da983a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0da40ffc40 0x7f0da4197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56336/0 (socket says 172.21.15.31:56336) 2024-04-25T01:55:39.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0da983a640 1 -- 172.21.15.31:0/2203163558 learned_addr learned my addr 172.21.15.31:0/2203163558 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0da9039640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da4100600 0x7f0da4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0daa03b640 1 -- 172.21.15.31:0/2203163558 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da4100600 msgr2=0x7f0da4198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0daa03b640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da4100600 0x7f0da4198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0daa03b640 1 -- 172.21.15.31:0/2203163558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0da40ffc40 msgr2=0x7f0da4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0daa03b640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0da40ffc40 0x7f0da4197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0daa03b640 1 -- 172.21.15.31:0/2203163558 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da419ced0 con 0x7f0da4107720 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0da983a640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0da40ffc40 0x7f0da4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0da9039640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da4100600 0x7f0da4198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0daa03b640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da4107720 0x7f0da419c7a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c02ae00 tx=0x7f0d9c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0d9affd640 1 -- 172.21.15.31:0/2203163558 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c004040 con 0x7f0da4107720 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0d9affd640 1 -- 172.21.15.31:0/2203163558 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d9c004690 con 0x7f0da4107720 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0daba9e640 1 -- 172.21.15.31:0/2203163558 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da419d160 con 0x7f0da4107720 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0d9affd640 1 -- 172.21.15.31:0/2203163558 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c030e20 con 0x7f0da4107720 2024-04-25T01:55:39.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.545+0000 7f0daba9e640 1 -- 172.21.15.31:0/2203163558 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da41a4a50 con 0x7f0da4107720 2024-04-25T01:55:39.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0d9affd640 1 -- 172.21.15.31:0/2203163558 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f0d9c0305a0 con 0x7f0da4107720 2024-04-25T01:55:39.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0dae7a9740 1 Processor -- start 2024-04-25T01:55:39.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0dae7a9740 1 -- start start 2024-04-25T01:55:39.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f52572b580 0x55f52572b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f525736030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f525736570 0x55f525738960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0dae7a9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f525690670 con 0x55f52572b580 2024-04-25T01:55:39.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0dae7a9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f525638970 con 0x55f525736570 2024-04-25T01:55:39.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0dae7a9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f52568c630 con 0x55f52572bea0 2024-04-25T01:55:39.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0da9039640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f52572b580 0x55f52572b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0da9039640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f52572b580 0x55f52572b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33548/0 (socket says 172.21.15.31:33548) 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.549+0000 7f0da9039640 1 -- 172.21.15.31:0/3762243563 learned_addr learned my addr 172.21.15.31:0/3762243563 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0da983a640 1 --2- 172.21.15.31:0/3762243563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f525736030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0da9039640 1 --2- 172.21.15.31:0/3762243563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f525736570 0x55f525738960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0d9affd640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0d900a2ea0 0x7f0d900a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0da9039640 1 -- 172.21.15.31:0/3762243563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 msgr2=0x55f525736030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0da9039640 1 --2- 172.21.15.31:0/3762243563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f525736030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0da9039640 1 -- 172.21.15.31:0/3762243563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f525736570 msgr2=0x55f525738960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0da9039640 1 --2- 172.21.15.31:0/3762243563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f525736570 0x55f525738960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0da9039640 1 -- 172.21.15.31:0/3762243563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f52565c400 con 0x55f52572b580 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0d9affd640 1 -- 172.21.15.31:0/2203163558 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0d9c02e030 con 0x7f0da4107720 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0daa03b640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0d900a2ea0 0x7f0d900a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0da9039640 1 --2- 172.21.15.31:0/3762243563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f52572b580 0x55f52572b960 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0d94013610 tx=0x7f0d940097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4583d09b0bdaebb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.550+0000 7f0d8ffff640 1 -- 172.21.15.31:0/3762243563 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9400f520 con 0x55f52572b580 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.551+0000 7f0d8ffff640 1 -- 172.21.15.31:0/3762243563 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d9400fb70 con 0x55f52572b580 2024-04-25T01:55:39.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.551+0000 7f0d8ffff640 1 -- 172.21.15.31:0/3762243563 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d940107f0 con 0x55f52572b580 2024-04-25T01:55:39.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.551+0000 7f0daa03b640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0d900a2ea0 0x7f0d900a5360 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c0ee380 tx=0x7f0d9c0095c0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.552+0000 7f0dae7a9740 1 -- 172.21.15.31:0/3762243563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f52572b580 msgr2=0x55f52572b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.552+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/3762243563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f52572b580 0x55f52572b960 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0d94013610 tx=0x7f0d940097d0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.552+0000 7f0dae7a9740 1 -- 172.21.15.31:0/3762243563 shutdown_connections 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.552+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/3762243563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f525736570 0x55f525738960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.552+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/3762243563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f525736030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.552+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/3762243563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f52572b580 0x55f52572b960 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.552+0000 7f0dae7a9740 1 -- 172.21.15.31:0/3762243563 >> 172.21.15.31:0/3762243563 conn(0x55f5257278a0 msgr2=0x55f525727c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.552+0000 7f0dae7a9740 1 -- 172.21.15.31:0/3762243563 shutdown_connections 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.552+0000 7f0dae7a9740 1 -- 172.21.15.31:0/3762243563 wait complete. 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.552+0000 7f0dae7a9740 1 -- start start 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.553+0000 7f0dae7a9740 1 Processor -- start 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.553+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572b580 0x55f525735ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.553+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572bea0 0x55f5257331e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.553+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525733760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.553+0000 7f0dae7a9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f52573d770 con 0x55f525736570 2024-04-25T01:55:39.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0dae7a9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f52564c050 con 0x55f52572bea0 2024-04-25T01:55:39.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da983a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572bea0 0x55f5257331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da983a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572bea0 0x55f5257331e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56360/0 (socket says 172.21.15.31:56360) 2024-04-25T01:55:39.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da983a640 1 -- 172.21.15.31:0/1440503729 learned_addr learned my addr 172.21.15.31:0/1440503729 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da9039640 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525733760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1440503729 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f52573d8f0 con 0x55f52572b580 2024-04-25T01:55:39.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0dae7a9740 10 client.0 _ll_register_callbacks cb 0x55f5253a3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:39.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da9039640 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572b580 0x55f525735ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da983a640 1 -- 172.21.15.31:0/1440503729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572b580 msgr2=0x55f525735ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da983a640 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572b580 0x55f525735ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da983a640 1 -- 172.21.15.31:0/1440503729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 msgr2=0x55f525733760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da983a640 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525733760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da983a640 1 -- 172.21.15.31:0/1440503729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f52561ddc0 con 0x55f52572bea0 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da9039640 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572b580 0x55f525735ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0da983a640 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572bea0 0x55f5257331e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0da00127d0 tx=0x7f0da0012c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d1edef917083b46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0d8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.554+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/1440503729 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0010950 con 0x55f52572bea0 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.555+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/1440503729 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0da0010af0 con 0x55f52572bea0 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.555+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1440503729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d9400fb70 con 0x55f52572bea0 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.555+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/1440503729 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0da0010cf0 con 0x55f52572bea0 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.555+0000 7f0dae7a9740 10 client.25496 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.555+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1440503729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f5253a43f0 con 0x55f52572bea0 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.555+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/1440503729 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d9400fb70 con 0x55f52572bea0 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.555+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/1440503729 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0da0019070 con 0x55f52572bea0 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.556+0000 7f0dae7a9740 10 client.25496 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.556+0000 7f0dae7a9740 10 client.25496 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:55:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.556+0000 7f0dae7a9740 20 client.25496 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.556+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55f52574ac70 0x55f52574d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.556+0000 7f0dae7a9740 4 client.25496 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.556+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1440503729 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f52569dbb0 con 0x55f52574ac70 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.556+0000 7f0da9039640 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55f52574ac70 0x55f52574d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.556+0000 7f0da9039640 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55f52574ac70 0x55f52574d0f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6d873b45dc6dacd4 server_cookie=c99a67aac9b19611 in_seq=0 out_seq=0 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.556+0000 7f0d8d7fa640 10 client.25496 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.557+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/1440503729 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f52569dbb0 con 0x55f52574ac70 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.557+0000 7f0d8d7fa640 10 client.25496 handle_command_reply: tid=0 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.557+0000 7f0dae7a9740 1 client.25496 shutdown 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.557+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1440503729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572bea0 msgr2=0x55f5257331e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.557+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572bea0 0x55f5257331e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0da00127d0 tx=0x7f0da0012c90 comp rx=0 tx=0).stop 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.558+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1440503729 shutdown_connections 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.558+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55f52574ac70 0x55f52574d0f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.558+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525733760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.558+0000 7f0d8d7fa640 0 client.25496 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:55:39.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.558+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572bea0 0x55f5257331e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.558+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/1440503729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572b580 0x55f525735ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.558+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1440503729 >> 172.21.15.31:0/1440503729 conn(0x55f5257278a0 msgr2=0x55f525726b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.558+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1440503729 shutdown_connections 2024-04-25T01:55:39.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.558+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1440503729 wait complete. 2024-04-25T01:55:39.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.558+0000 7f0dae7a9740 20 client.25496 trim_cache size 0 max 16384 2024-04-25T01:55:39.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.571+0000 7f0dae7a9740 1 Processor -- start 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.571+0000 7f0dae7a9740 1 -- start start 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525746690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f525752a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525752f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0dae7a9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0da41aaf80 con 0x55f525736570 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0dae7a9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0da41acac0 con 0x55f52572b580 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0dae7a9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0da41ab650 con 0x55f52572bea0 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0da9039640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525752f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0da9039640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525746690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0da9039640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525746690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56370/0 (socket says 172.21.15.31:56370) 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0da9039640 1 -- 172.21.15.31:0/1542190921 learned_addr learned my addr 172.21.15.31:0/1542190921 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0da983a640 1 --2- 172.21.15.31:0/1542190921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f525752a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0da9039640 1 -- 172.21.15.31:0/1542190921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 msgr2=0x55f525752a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0da9039640 1 --2- 172.21.15.31:0/1542190921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f525752a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0da9039640 1 -- 172.21.15.31:0/1542190921 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 msgr2=0x55f525752f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0da9039640 1 --2- 172.21.15.31:0/1542190921 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525752f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0da9039640 1 -- 172.21.15.31:0/1542190921 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f525733ad0 con 0x55f52572b580 2024-04-25T01:55:39.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.572+0000 7f0da983a640 1 --2- 172.21.15.31:0/1542190921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f525752a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:39.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.573+0000 7f0da9039640 1 --2- 172.21.15.31:0/1542190921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525746690 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0d94016010 tx=0x7f0d94022040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a97f46751fdaf7df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.573+0000 7f0d8f7fe640 1 -- 172.21.15.31:0/1542190921 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9402d540 con 0x55f52572b580 2024-04-25T01:55:39.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.573+0000 7f0d8f7fe640 1 -- 172.21.15.31:0/1542190921 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d94013250 con 0x55f52572b580 2024-04-25T01:55:39.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.573+0000 7f0d8f7fe640 1 -- 172.21.15.31:0/1542190921 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9400f720 con 0x55f52572b580 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.573+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1542190921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 msgr2=0x55f525746690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.573+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/1542190921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525746690 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0d94016010 tx=0x7f0d94022040 comp rx=0 tx=0).stop 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.573+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1542190921 shutdown_connections 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.573+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/1542190921 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525752f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.573+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/1542190921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f525752a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.573+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/1542190921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525746690 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.573+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1542190921 >> 172.21.15.31:0/1542190921 conn(0x55f5257278a0 msgr2=0x55f525727c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.574+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1542190921 shutdown_connections 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.574+0000 7f0dae7a9740 1 -- 172.21.15.31:0/1542190921 wait complete. 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.574+0000 7f0dae7a9740 1 -- start start 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.574+0000 7f0dae7a9740 1 Processor -- start 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.574+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525742ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.574+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f5257430e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.574+0000 7f0dae7a9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525740320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0dae7a9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0da41ac3f0 con 0x55f525736570 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0dae7a9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0da41abd20 con 0x55f52572b580 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0dae7a9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0da41a8f60 con 0x55f52572bea0 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0dae7a9740 10 client.0 _ll_register_callbacks cb 0x7f0da000bd50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0da9039640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525742ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0da9039640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525742ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56384/0 (socket says 172.21.15.31:56384) 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0da9039640 1 -- 172.21.15.31:0/2608823495 learned_addr learned my addr 172.21.15.31:0/2608823495 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0da9039640 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525740320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0da983a640 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f5257430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0da9039640 1 -- 172.21.15.31:0/2608823495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 msgr2=0x55f5257430e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0da9039640 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f5257430e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0da9039640 1 -- 172.21.15.31:0/2608823495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 msgr2=0x55f525740320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0da9039640 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525740320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0da9039640 1 -- 172.21.15.31:0/2608823495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f52546e7d0 con 0x55f52572b580 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0da9039640 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525742ba0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0d94009a10 tx=0x7f0d94009aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=255cb190bba8eee4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.575+0000 7f0d8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/2608823495 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d94005250 con 0x55f52572b580 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0dae7a9740 1 -- 172.21.15.31:0/2608823495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d94013250 con 0x55f52572b580 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/2608823495 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0d940053f0 con 0x55f52572b580 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/2608823495 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0d9400f520 con 0x55f52572b580 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0dae7a9740 10 client.25502 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0dae7a9740 1 -- 172.21.15.31:0/2608823495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f5257b0d40 con 0x55f52572b580 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/2608823495 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d94013250 con 0x55f52572b580 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/2608823495 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0d94004bd0 con 0x55f52572b580 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0dae7a9740 10 client.25502 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0dae7a9740 10 client.25502 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0dae7a9740 20 client.25502 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:39.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55f52574ac70 0x55f5257b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0dae7a9740 4 client.25502 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0dae7a9740 1 -- 172.21.15.31:0/2608823495 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f525697f00 con 0x55f52574ac70 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.576+0000 7f0da983a640 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55f52574ac70 0x55f5257b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0d8d7fa640 10 client.25502 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0da983a640 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55f52574ac70 0x55f5257b31d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d289b2f9526797 server_cookie=28556684dc261c78 in_seq=0 out_seq=0 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0d8d7fa640 1 -- 172.21.15.31:0/2608823495 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f525697f00 con 0x55f52574ac70 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0d8d7fa640 10 client.25502 handle_command_reply: tid=0 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0dae7a9740 1 client.25502 shutdown 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0dae7a9740 1 -- 172.21.15.31:0/2608823495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 msgr2=0x55f525742ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525742ba0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0d94009a10 tx=0x7f0d94009aa0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0dae7a9740 1 -- 172.21.15.31:0/2608823495 shutdown_connections 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55f52574ac70 0x55f5257b31d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f525736570 0x55f525740320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0d8d7fa640 0 client.25502 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:55:39.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f52572bea0 0x55f5257430e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0dae7a9740 1 --2- 172.21.15.31:0/2608823495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f52572b580 0x55f525742ba0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.577+0000 7f0dae7a9740 1 -- 172.21.15.31:0/2608823495 >> 172.21.15.31:0/2608823495 conn(0x55f5257278a0 msgr2=0x55f525731ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0dae7a9740 1 -- 172.21.15.31:0/2608823495 shutdown_connections 2024-04-25T01:55:39.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0dae7a9740 1 -- 172.21.15.31:0/2608823495 wait complete. 2024-04-25T01:55:39.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0dae7a9740 20 client.25502 trim_cache size 0 max 16384 2024-04-25T01:55:39.585 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:55:39.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0daba9e640 1 -- 172.21.15.31:0/2203163558 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0d900a2ea0 msgr2=0x7f0d900a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0daba9e640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0d900a2ea0 0x7f0d900a5360 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c0ee380 tx=0x7f0d9c0095c0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0daba9e640 1 -- 172.21.15.31:0/2203163558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da4107720 msgr2=0x7f0da419c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0daba9e640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da4107720 0x7f0da419c7a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c02ae00 tx=0x7f0d9c003970 comp rx=0 tx=0).stop 2024-04-25T01:55:39.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0daba9e640 1 -- 172.21.15.31:0/2203163558 shutdown_connections 2024-04-25T01:55:39.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0daba9e640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0d900a2ea0 0x7f0d900a5360 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0daba9e640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da4107720 0x7f0da419c7a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0daba9e640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da4100600 0x7f0da4198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0daba9e640 1 --2- 172.21.15.31:0/2203163558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0da40ffc40 0x7f0da4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0daba9e640 1 -- 172.21.15.31:0/2203163558 >> 172.21.15.31:0/2203163558 conn(0x7f0da4076bd0 msgr2=0x7f0da40fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.578+0000 7f0daba9e640 1 -- 172.21.15.31:0/2203163558 shutdown_connections 2024-04-25T01:55:39.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.579+0000 7f0daba9e640 1 -- 172.21.15.31:0/2203163558 wait complete. 2024-04-25T01:55:39.591 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:55:39.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.699+0000 7efd00683640 1 -- 172.21.15.31:0/1206586878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf8100fe0 msgr2=0x7efcf810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.699+0000 7efd00683640 1 --2- 172.21.15.31:0/1206586878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf8100fe0 0x7efcf810d4f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009cd0 tx=0x7efcf002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.699+0000 7efd00683640 1 -- 172.21.15.31:0/1206586878 shutdown_connections 2024-04-25T01:55:39.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.699+0000 7efd00683640 1 --2- 172.21.15.31:0/1206586878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf8100fe0 0x7efcf810d4f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.699+0000 7efd00683640 1 --2- 172.21.15.31:0/1206586878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf8100620 0x7efcf8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.699+0000 7efd00683640 1 --2- 172.21.15.31:0/1206586878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf8106630 0x7efcf8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.699+0000 7efd00683640 1 -- 172.21.15.31:0/1206586878 >> 172.21.15.31:0/1206586878 conn(0x7efcf80776a0 msgr2=0x7efcf8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.699+0000 7efd00683640 1 -- 172.21.15.31:0/1206586878 shutdown_connections 2024-04-25T01:55:39.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.699+0000 7efd00683640 1 -- 172.21.15.31:0/1206586878 wait complete. 2024-04-25T01:55:39.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efd00683640 1 Processor -- start 2024-04-25T01:55:39.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efd00683640 1 -- start start 2024-04-25T01:55:39.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efd00683640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf8100620 0x7efcf8195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efd00683640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf8100fe0 0x7efcf8196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efd00683640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf8106630 0x7efcf819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efd00683640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efcf810f7e0 con 0x7efcf8100620 2024-04-25T01:55:39.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efcfe41f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf8100620 0x7efcf8195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efcfe41f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf8100620 0x7efcf8195d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33584/0 (socket says 172.21.15.31:33584) 2024-04-25T01:55:39.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efcfe41f640 1 -- 172.21.15.31:0/850039980 learned_addr learned my addr 172.21.15.31:0/850039980 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efd00683640 1 -- 172.21.15.31:0/850039980 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efcf810f660 con 0x7efcf8100fe0 2024-04-25T01:55:39.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efd00683640 1 -- 172.21.15.31:0/850039980 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efcf810f960 con 0x7efcf8106630 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.700+0000 7efcfdc1e640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf8100fe0 0x7efcf8196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.701+0000 7efcfec20640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf8106630 0x7efcf819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.701+0000 7efcfe41f640 1 -- 172.21.15.31:0/850039980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf8106630 msgr2=0x7efcf819a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.701+0000 7efcfe41f640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf8106630 0x7efcf819a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.701+0000 7efcfe41f640 1 -- 172.21.15.31:0/850039980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf8100fe0 msgr2=0x7efcf8196280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.701+0000 7efcfe41f640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf8100fe0 0x7efcf8196280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.701+0000 7efcfe41f640 1 -- 172.21.15.31:0/850039980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf819ad30 con 0x7efcf8100620 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.701+0000 7efcfdc1e640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf8100fe0 0x7efcf8196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.701+0000 7efcfe41f640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf8100620 0x7efcf8195d40 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7efcf4004650 tx=0x7efcf4003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.701+0000 7efcef7fe640 1 -- 172.21.15.31:0/850039980 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4009b60 con 0x7efcf8100620 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.701+0000 7efcef7fe640 1 -- 172.21.15.31:0/850039980 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcf4009d00 con 0x7efcf8100620 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.701+0000 7efcef7fe640 1 -- 172.21.15.31:0/850039980 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4004d70 con 0x7efcf8100620 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.702+0000 7efd00683640 1 -- 172.21.15.31:0/850039980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf819b020 con 0x7efcf8100620 2024-04-25T01:55:39.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.702+0000 7efd00683640 1 -- 172.21.15.31:0/850039980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf81a28b0 con 0x7efcf8100620 2024-04-25T01:55:39.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.704+0000 7efd0338a740 1 Processor -- start 2024-04-25T01:55:39.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.704+0000 7efd0338a740 1 -- start start 2024-04-25T01:55:39.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efd0338a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563afface580 0x563afface960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efd0338a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 0x563affad9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efd0338a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563affad9570 0x563affadb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efd0338a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563affa33670 con 0x563afface580 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efd0338a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563aff9db970 con 0x563affad9570 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efd0338a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563affa2f630 con 0x563affaceea0 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efcfec20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563afface580 0x563afface960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efcfec20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563afface580 0x563afface960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33590/0 (socket says 172.21.15.31:33590) 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efcfec20640 1 -- 172.21.15.31:0/1347740714 learned_addr learned my addr 172.21.15.31:0/1347740714 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efcfdc1e640 1 --2- 172.21.15.31:0/1347740714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563affad9570 0x563affadb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efcfec20640 1 -- 172.21.15.31:0/1347740714 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 msgr2=0x563affad9030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efcfec20640 1 --2- 172.21.15.31:0/1347740714 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 0x563affad9030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efcfec20640 1 -- 172.21.15.31:0/1347740714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563affad9570 msgr2=0x563affadb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efcfec20640 1 --2- 172.21.15.31:0/1347740714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563affad9570 0x563affadb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efcfec20640 1 -- 172.21.15.31:0/1347740714 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aff9ff400 con 0x563afface580 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.705+0000 7efcef7fe640 1 -- 172.21.15.31:0/850039980 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7efcf401f020 con 0x7efcf8100620 2024-04-25T01:55:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.706+0000 7efcfec20640 1 --2- 172.21.15.31:0/1347740714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563afface580 0x563afface960 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009cd0 tx=0x7efcf0009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cebcab304e49d3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.706+0000 7efcecff9640 1 -- 172.21.15.31:0/1347740714 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0049070 con 0x563afface580 2024-04-25T01:55:39.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.706+0000 7efcecff9640 1 -- 172.21.15.31:0/1347740714 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcf0036040 con 0x563afface580 2024-04-25T01:55:39.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.706+0000 7efcecff9640 1 -- 172.21.15.31:0/1347740714 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf00042a0 con 0x563afface580 2024-04-25T01:55:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.706+0000 7efcef7fe640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efce40a2ea0 0x7efce40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efcfe41f640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efce40a2ea0 0x7efce40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efd0338a740 1 -- 172.21.15.31:0/1347740714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563afface580 msgr2=0x563afface960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efd0338a740 1 --2- 172.21.15.31:0/1347740714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563afface580 0x563afface960 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009cd0 tx=0x7efcf0009a20 comp rx=0 tx=0).stop 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efcef7fe640 1 -- 172.21.15.31:0/850039980 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7efcf40d2f00 con 0x7efcf8100620 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efd0338a740 1 -- 172.21.15.31:0/1347740714 shutdown_connections 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efd0338a740 1 --2- 172.21.15.31:0/1347740714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563affad9570 0x563affadb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efd0338a740 1 --2- 172.21.15.31:0/1347740714 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 0x563affad9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efd0338a740 1 --2- 172.21.15.31:0/1347740714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563afface580 0x563afface960 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efd0338a740 1 -- 172.21.15.31:0/1347740714 >> 172.21.15.31:0/1347740714 conn(0x563affaca8a0 msgr2=0x563affacac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efcfe41f640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efce40a2ea0 0x7efce40a5360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efcf40d0580 tx=0x7efcf40a2040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efd0338a740 1 -- 172.21.15.31:0/1347740714 shutdown_connections 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.707+0000 7efd0338a740 1 -- 172.21.15.31:0/1347740714 wait complete. 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.708+0000 7efd0338a740 1 -- start start 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.708+0000 7efd0338a740 1 Processor -- start 2024-04-25T01:55:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.708+0000 7efd0338a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affad8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.708+0000 7efd0338a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 0x563affad6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.708+0000 7efd0338a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affad6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.708+0000 7efd0338a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563affadc300 con 0x563affad9570 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.708+0000 7efd0338a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563aff9ef050 con 0x563afface580 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.708+0000 7efd0338a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563affadc480 con 0x563affaceea0 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.708+0000 7efd0338a740 10 client.0 _ll_register_callbacks cb 0x563aff746be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.708+0000 7efcfec20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affad6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efcfec20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affad6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33602/0 (socket says 172.21.15.31:33602) 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efcfec20640 1 -- 172.21.15.31:0/3193186079 learned_addr learned my addr 172.21.15.31:0/3193186079 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efcfec20640 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affad8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efcfdc1e640 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 0x563affad6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efcfec20640 1 -- 172.21.15.31:0/3193186079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 msgr2=0x563affad6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efcfec20640 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 0x563affad6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efcfec20640 1 -- 172.21.15.31:0/3193186079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 msgr2=0x563affad8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efcfec20640 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affad8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efcfec20640 1 -- 172.21.15.31:0/3193186079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aff9c0dc0 con 0x563affad9570 2024-04-25T01:55:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efcfec20640 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affad6850 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7efcf0006010 tx=0x7efcf002a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b78c32cfcca38e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efce1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efce1ffb640 1 -- 172.21.15.31:0/3193186079 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0049070 con 0x563affad9570 2024-04-25T01:55:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efd0338a740 1 -- 172.21.15.31:0/3193186079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcf0036040 con 0x563affad9570 2024-04-25T01:55:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efce1ffb640 1 -- 172.21.15.31:0/3193186079 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7efcf0004440 con 0x563affad9570 2024-04-25T01:55:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efce1ffb640 1 -- 172.21.15.31:0/3193186079 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efcf0004640 con 0x563affad9570 2024-04-25T01:55:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.709+0000 7efd0338a740 10 client.15906 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.710+0000 7efd0338a740 1 -- 172.21.15.31:0/3193186079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563aff7473f0 con 0x563affad9570 2024-04-25T01:55:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.710+0000 7efce1ffb640 1 -- 172.21.15.31:0/3193186079 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcf0036040 con 0x563affad9570 2024-04-25T01:55:39.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.710+0000 7efce1ffb640 1 -- 172.21.15.31:0/3193186079 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7efcf0030070 con 0x563affad9570 2024-04-25T01:55:39.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.710+0000 7efd0338a740 10 client.15906 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:39.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.710+0000 7efd0338a740 10 client.15906 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:55:39.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.710+0000 7efd0338a740 20 client.15906 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:39.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.710+0000 7efd0338a740 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563affae94a0 0x563affaeb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.710+0000 7efd0338a740 4 client.15906 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:39.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.710+0000 7efd0338a740 1 -- 172.21.15.31:0/3193186079 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563affa40bb0 con 0x563affae94a0 2024-04-25T01:55:39.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.711+0000 7efcfdc1e640 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563affae94a0 0x563affaeb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.711+0000 7efcfdc1e640 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563affae94a0 0x563affaeb890 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aed1801dbd20e189 server_cookie=485f101690b77ad6 in_seq=0 out_seq=0 2024-04-25T01:55:39.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.711+0000 7efce1ffb640 10 client.15906 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:55:39.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.712+0000 7efce1ffb640 1 -- 172.21.15.31:0/3193186079 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563affa40bb0 con 0x563affae94a0 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.712+0000 7efce1ffb640 10 client.15906 handle_command_reply: tid=0 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.712+0000 7efd0338a740 1 client.15906 shutdown 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.712+0000 7efd0338a740 1 -- 172.21.15.31:0/3193186079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 msgr2=0x563affad6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.712+0000 7efd0338a740 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affad6850 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7efcf0006010 tx=0x7efcf002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.712+0000 7efd0338a740 1 -- 172.21.15.31:0/3193186079 shutdown_connections 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.712+0000 7efd0338a740 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563affae94a0 0x563affaeb890 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.713+0000 7efd0338a740 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affad6850 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.713+0000 7efd0338a740 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 0x563affad6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.713+0000 7efd0338a740 1 --2- 172.21.15.31:0/3193186079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affad8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.713+0000 7efd0338a740 1 -- 172.21.15.31:0/3193186079 >> 172.21.15.31:0/3193186079 conn(0x563affaca8a0 msgr2=0x563affac9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.713+0000 7efce1ffb640 0 client.15906 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.713+0000 7efd0338a740 1 -- 172.21.15.31:0/3193186079 shutdown_connections 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.713+0000 7efd0338a740 1 -- 172.21.15.31:0/3193186079 wait complete. 2024-04-25T01:55:39.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.713+0000 7efd0338a740 20 client.15906 trim_cache size 0 max 16384 2024-04-25T01:55:39.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.728+0000 7efd0338a740 1 Processor -- start 2024-04-25T01:55:39.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.728+0000 7efd0338a740 1 -- start start 2024-04-25T01:55:39.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.728+0000 7efd0338a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affad8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.728+0000 7efd0338a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 0x563affaee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.728+0000 7efd0338a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affaeea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.728+0000 7efcfec20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affaeea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.728+0000 7efcfec20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affaeea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33608/0 (socket says 172.21.15.31:33608) 2024-04-25T01:55:39.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.728+0000 7efcfec20640 1 -- 172.21.15.31:0/3688109886 learned_addr learned my addr 172.21.15.31:0/3688109886 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.728+0000 7efd0338a740 1 -- 172.21.15.31:0/3688109886 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81aaf50 con 0x563affad9570 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.728+0000 7efd0338a740 1 -- 172.21.15.31:0/3688109886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81aca90 con 0x563afface580 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.728+0000 7efd0338a740 1 -- 172.21.15.31:0/3688109886 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81ab620 con 0x563affaceea0 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.729+0000 7efcfec20640 1 --2- 172.21.15.31:0/3688109886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affad8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.729+0000 7efcfdc1e640 1 --2- 172.21.15.31:0/3688109886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 0x563affaee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.729+0000 7efcfec20640 1 -- 172.21.15.31:0/3688109886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 msgr2=0x563affaee4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.729+0000 7efcfec20640 1 --2- 172.21.15.31:0/3688109886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 0x563affaee4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.729+0000 7efcfec20640 1 -- 172.21.15.31:0/3688109886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 msgr2=0x563affad8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.729+0000 7efcfec20640 1 --2- 172.21.15.31:0/3688109886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affad8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.729+0000 7efcfec20640 1 -- 172.21.15.31:0/3688109886 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563affadbcd0 con 0x563affad9570 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.729+0000 7efcfec20640 1 --2- 172.21.15.31:0/3688109886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affaeea10 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7efcf0000f80 tx=0x7efcf0009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b6705e803cfc799 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.729+0000 7efce3fff640 1 -- 172.21.15.31:0/3688109886 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0049040 con 0x563affad9570 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.729+0000 7efce3fff640 1 -- 172.21.15.31:0/3688109886 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcf0036040 con 0x563affad9570 2024-04-25T01:55:39.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.729+0000 7efce3fff640 1 -- 172.21.15.31:0/3688109886 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf00306c0 con 0x563affad9570 2024-04-25T01:55:39.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.730+0000 7efd0338a740 1 -- 172.21.15.31:0/3688109886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 msgr2=0x563affaeea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.730+0000 7efd0338a740 1 --2- 172.21.15.31:0/3688109886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affaeea10 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7efcf0000f80 tx=0x7efcf0009a20 comp rx=0 tx=0).stop 2024-04-25T01:55:39.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.730+0000 7efd0338a740 1 -- 172.21.15.31:0/3688109886 shutdown_connections 2024-04-25T01:55:39.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.730+0000 7efd0338a740 1 --2- 172.21.15.31:0/3688109886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affad9570 0x563affaeea10 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.730+0000 7efd0338a740 1 --2- 172.21.15.31:0/3688109886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affaceea0 0x563affaee4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.730+0000 7efd0338a740 1 --2- 172.21.15.31:0/3688109886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affad8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.730+0000 7efd0338a740 1 -- 172.21.15.31:0/3688109886 >> 172.21.15.31:0/3688109886 conn(0x563affaca8a0 msgr2=0x563affae4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.730+0000 7efd0338a740 1 -- 172.21.15.31:0/3688109886 shutdown_connections 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.730+0000 7efd0338a740 1 -- 172.21.15.31:0/3688109886 wait complete. 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.731+0000 7efd0338a740 1 -- start start 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.731+0000 7efd0338a740 1 Processor -- start 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.731+0000 7efd0338a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affaee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.731+0000 7efd0338a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affaceea0 0x563affaebc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.731+0000 7efd0338a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affad9570 0x563affaec1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.731+0000 7efcfdc1e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affaceea0 0x563affaebc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.731+0000 7efcfdc1e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affaceea0 0x563affaebc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33618/0 (socket says 172.21.15.31:33618) 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.731+0000 7efcfdc1e640 1 -- 172.21.15.31:0/2936979023 learned_addr learned my addr 172.21.15.31:0/2936979023 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.731+0000 7efd0338a740 1 -- 172.21.15.31:0/2936979023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81ac3c0 con 0x563affaceea0 2024-04-25T01:55:39.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.731+0000 7efd0338a740 1 -- 172.21.15.31:0/2936979023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81abcf0 con 0x563afface580 2024-04-25T01:55:39.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.731+0000 7efd0338a740 1 -- 172.21.15.31:0/2936979023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efcf81a8f50 con 0x563affad9570 2024-04-25T01:55:39.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efcfec20640 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affaee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efd0338a740 10 client.0 _ll_register_callbacks cb 0x7efcf0047070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:39.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efcfdc1e640 1 -- 172.21.15.31:0/2936979023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affad9570 msgr2=0x563affaec1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efcfdc1e640 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affad9570 0x563affaec1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efcfdc1e640 1 -- 172.21.15.31:0/2936979023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 msgr2=0x563affaee8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efcfdc1e640 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affaee8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efcfdc1e640 1 -- 172.21.15.31:0/2936979023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aff8117d0 con 0x563affaceea0 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efcfdc1e640 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affaceea0 0x563affaebc70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x563affacba30 tx=0x7efce80216d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee8139b11216087 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efce1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efce1ffb640 1 -- 172.21.15.31:0/2936979023 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce8003cc0 con 0x563affaceea0 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efce1ffb640 1 -- 172.21.15.31:0/2936979023 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7efce80047a0 con 0x563affaceea0 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efd0338a740 1 -- 172.21.15.31:0/2936979023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcf0036040 con 0x563affaceea0 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.732+0000 7efce1ffb640 1 -- 172.21.15.31:0/2936979023 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efce8003e60 con 0x563affaceea0 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.733+0000 7efd0338a740 10 client.15912 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.733+0000 7efd0338a740 1 -- 172.21.15.31:0/2936979023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=27}) -- 0x563affb4f490 con 0x563affaceea0 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.733+0000 7efce1ffb640 1 -- 172.21.15.31:0/2936979023 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7efce8004bb0 con 0x563affaceea0 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.733+0000 7efd0338a740 10 client.15912 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.733+0000 7efd0338a740 10 client.15912 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.733+0000 7efd0338a740 20 client.15912 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.733+0000 7efd0338a740 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563affb4fa30 0x563affb51e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.733+0000 7efd0338a740 4 client.15912 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:39.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.733+0000 7efd0338a740 1 -- 172.21.15.31:0/2936979023 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7efcf0036040 con 0x563affb4fa30 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.733+0000 7efcfec20640 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563affb4fa30 0x563affb51e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.734+0000 7efce1ffb640 10 client.15912 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.734+0000 7efcfec20640 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563affb4fa30 0x563affb51e50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e7922b5a3c7355af server_cookie=df97c073c95c735f in_seq=0 out_seq=0 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.734+0000 7efce1ffb640 1 -- 172.21.15.31:0/2936979023 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7efcf0036040 con 0x563affb4fa30 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.734+0000 7efce1ffb640 10 client.15912 handle_command_reply: tid=0 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.734+0000 7efd0338a740 1 client.15912 shutdown 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.734+0000 7efd0338a740 1 -- 172.21.15.31:0/2936979023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affaceea0 msgr2=0x563affaebc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.734+0000 7efd0338a740 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affaceea0 0x563affaebc70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x563affacba30 tx=0x7efce80216d0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.734+0000 7efd0338a740 1 -- 172.21.15.31:0/2936979023 shutdown_connections 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd0338a740 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563affb4fa30 0x563affb51e50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd0338a740 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563affad9570 0x563affaec1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efce1ffb640 0 client.15912 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd0338a740 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563affaceea0 0x563affaebc70 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd0338a740 1 --2- 172.21.15.31:0/2936979023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563afface580 0x563affaee8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd0338a740 1 -- 172.21.15.31:0/2936979023 >> 172.21.15.31:0/2936979023 conn(0x563affaca8a0 msgr2=0x563affad0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd0338a740 1 -- 172.21.15.31:0/2936979023 shutdown_connections 2024-04-25T01:55:39.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd0338a740 1 -- 172.21.15.31:0/2936979023 wait complete. 2024-04-25T01:55:39.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd0338a740 20 client.15912 trim_cache size 0 max 16384 2024-04-25T01:55:39.742 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd00683640 1 -- 172.21.15.31:0/850039980 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efce40a2ea0 msgr2=0x7efce40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd00683640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efce40a2ea0 0x7efce40a5360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efcf40d0580 tx=0x7efcf40a2040 comp rx=0 tx=0).stop 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd00683640 1 -- 172.21.15.31:0/850039980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf8100620 msgr2=0x7efcf8195d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.735+0000 7efd00683640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf8100620 0x7efcf8195d40 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7efcf4004650 tx=0x7efcf4003da0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.736+0000 7efd00683640 1 -- 172.21.15.31:0/850039980 shutdown_connections 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.736+0000 7efd00683640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efce40a2ea0 0x7efce40a5360 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.736+0000 7efd00683640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf8106630 0x7efcf819a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.736+0000 7efd00683640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcf8100fe0 0x7efcf8196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.736+0000 7efd00683640 1 --2- 172.21.15.31:0/850039980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcf8100620 0x7efcf8195d40 secure :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7efcf4004650 tx=0x7efcf4003da0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.736+0000 7efd00683640 1 -- 172.21.15.31:0/850039980 >> 172.21.15.31:0/850039980 conn(0x7efcf80776a0 msgr2=0x7efcf81031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.736+0000 7efd00683640 1 -- 172.21.15.31:0/850039980 shutdown_connections 2024-04-25T01:55:39.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.736+0000 7efd00683640 1 -- 172.21.15.31:0/850039980 wait complete. 2024-04-25T01:55:39.744 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:39 smithi031 ceph-mon[29204]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:39.744 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:39 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/398571831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:39.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:55:39.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.855+0000 7ffa0411b640 1 -- 172.21.15.31:0/1188989093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc10bd00 msgr2=0x7ff9fc10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.855+0000 7ffa0411b640 1 --2- 172.21.15.31:0/1188989093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc10bd00 0x7ff9fc10e1a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009ca0 tx=0x7ff9f402a9e0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.855+0000 7ffa0411b640 1 -- 172.21.15.31:0/1188989093 shutdown_connections 2024-04-25T01:55:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.855+0000 7ffa0411b640 1 --2- 172.21.15.31:0/1188989093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc10bd00 0x7ff9fc10e1a0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.855+0000 7ffa0411b640 1 --2- 172.21.15.31:0/1188989093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9fc1032e0 0x7ff9fc10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.855+0000 7ffa0411b640 1 --2- 172.21.15.31:0/1188989093 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9fc1029c0 0x7ff9fc102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.855+0000 7ffa0411b640 1 -- 172.21.15.31:0/1188989093 >> 172.21.15.31:0/1188989093 conn(0x7ff9fc077740 msgr2=0x7ff9fc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.855+0000 7ffa0411b640 1 -- 172.21.15.31:0/1188989093 shutdown_connections 2024-04-25T01:55:39.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.856+0000 7ffa0411b640 1 -- 172.21.15.31:0/1188989093 wait complete. 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.856+0000 7ffa0411b640 1 Processor -- start 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.856+0000 7ffa0411b640 1 -- start start 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.856+0000 7ffa0411b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9fc1029c0 0x7ff9fc19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.856+0000 7ffa0411b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9fc1032e0 0x7ff9fc19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.856+0000 7ffa0411b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc10bd00 0x7ff9fc19ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.856+0000 7ffa0411b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc1119b0 con 0x7ff9fc10bd00 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.856+0000 7ffa0411b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc111830 con 0x7ff9fc1029c0 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.856+0000 7ffa0411b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc111b30 con 0x7ff9fc1032e0 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa01eb7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9fc1029c0 0x7ff9fc19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa01eb7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9fc1029c0 0x7ff9fc19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56460/0 (socket says 172.21.15.31:56460) 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa01eb7640 1 -- 172.21.15.31:0/2414138456 learned_addr learned my addr 172.21.15.31:0/2414138456 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa026b8640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc10bd00 0x7ff9fc19ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa016b6640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9fc1032e0 0x7ff9fc19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa01eb7640 1 -- 172.21.15.31:0/2414138456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9fc1032e0 msgr2=0x7ff9fc19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa01eb7640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9fc1032e0 0x7ff9fc19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa01eb7640 1 -- 172.21.15.31:0/2414138456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc10bd00 msgr2=0x7ff9fc19ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa01eb7640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc10bd00 0x7ff9fc19ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa01eb7640 1 -- 172.21.15.31:0/2414138456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc19f150 con 0x7ff9fc1029c0 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa016b6640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9fc1032e0 0x7ff9fc19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa026b8640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc10bd00 0x7ff9fc19ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.857+0000 7ffa01eb7640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9fc1029c0 0x7ff9fc19a160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800ea60 tx=0x7ff9f800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.858+0000 7ff9f2ffd640 1 -- 172.21.15.31:0/2414138456 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f800cde0 con 0x7ff9fc1029c0 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.858+0000 7ffa0411b640 1 -- 172.21.15.31:0/2414138456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9fc19f440 con 0x7ff9fc1029c0 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.858+0000 7ff9f2ffd640 1 -- 172.21.15.31:0/2414138456 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9f80035f0 con 0x7ff9fc1029c0 2024-04-25T01:55:39.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.858+0000 7ffa0411b640 1 -- 172.21.15.31:0/2414138456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9fc1a6cd0 con 0x7ff9fc1029c0 2024-04-25T01:55:39.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.858+0000 7ff9f2ffd640 1 -- 172.21.15.31:0/2414138456 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8010690 con 0x7ff9fc1029c0 2024-04-25T01:55:39.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.860+0000 7ffa06e2f740 1 Processor -- start 2024-04-25T01:55:39.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.860+0000 7ffa06e2f740 1 -- start start 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 0x56155415b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415bea0 0x561554166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561554166570 0x561554168960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5615540c0670 con 0x56155415bea0 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561554068970 con 0x56155415b580 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5615540bc630 con 0x561554166570 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa026b8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 0x56155415b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa026b8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 0x56155415b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56474/0 (socket says 172.21.15.31:56474) 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa026b8640 1 -- 172.21.15.31:0/2446554572 learned_addr learned my addr 172.21.15.31:0/2446554572 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa016b6640 1 --2- 172.21.15.31:0/2446554572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561554166570 0x561554168960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa016b6640 1 -- 172.21.15.31:0/2446554572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 msgr2=0x56155415b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa016b6640 1 --2- 172.21.15.31:0/2446554572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 0x56155415b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa016b6640 1 -- 172.21.15.31:0/2446554572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415bea0 msgr2=0x561554166030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:55:39.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa016b6640 1 --2- 172.21.15.31:0/2446554572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415bea0 0x561554166030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa016b6640 1 -- 172.21.15.31:0/2446554572 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56155408c400 con 0x561554166570 2024-04-25T01:55:39.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa026b8640 1 --2- 172.21.15.31:0/2446554572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 0x56155415b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:39.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.861+0000 7ffa016b6640 1 --2- 172.21.15.31:0/2446554572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561554166570 0x561554168960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009f20 tx=0x7ff9ec0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0bba095996bae30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.862+0000 7ff9e7fff640 1 -- 172.21.15.31:0/2446554572 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec01e070 con 0x561554166570 2024-04-25T01:55:39.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.862+0000 7ff9e7fff640 1 -- 172.21.15.31:0/2446554572 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9ec0035f0 con 0x561554166570 2024-04-25T01:55:39.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.862+0000 7ff9e7fff640 1 -- 172.21.15.31:0/2446554572 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0044e0 con 0x561554166570 2024-04-25T01:55:39.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.862+0000 7ff9f2ffd640 1 -- 172.21.15.31:0/2414138456 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7ff9f80037a0 con 0x7ff9fc1029c0 2024-04-25T01:55:39.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.862+0000 7ffa06e2f740 1 -- 172.21.15.31:0/2446554572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561554166570 msgr2=0x561554168960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.862+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/2446554572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561554166570 0x561554168960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009f20 tx=0x7ff9ec0118d0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.863+0000 7ff9f2ffd640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff9e80a2e20 0x7ff9e80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.863+0000 7ff9f2ffd640 1 -- 172.21.15.31:0/2414138456 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7ff9f80cf920 con 0x7ff9fc1029c0 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.863+0000 7ffa06e2f740 1 -- 172.21.15.31:0/2446554572 shutdown_connections 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.863+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/2446554572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561554166570 0x561554168960 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.863+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/2446554572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415bea0 0x561554166030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.863+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/2446554572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 0x56155415b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.863+0000 7ffa06e2f740 1 -- 172.21.15.31:0/2446554572 >> 172.21.15.31:0/2446554572 conn(0x5615541578a0 msgr2=0x561554157c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.863+0000 7ffa026b8640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff9e80a2e20 0x7ff9e80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.863+0000 7ffa06e2f740 1 -- 172.21.15.31:0/2446554572 shutdown_connections 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.863+0000 7ffa06e2f740 1 -- 172.21.15.31:0/2446554572 wait complete. 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.864+0000 7ffa06e2f740 1 -- start start 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.864+0000 7ffa026b8640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff9e80a2e20 0x7ff9e80a52e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4006010 tx=0x7ff9f403a040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.864+0000 7ffa06e2f740 1 Processor -- start 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.864+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 0x561554165f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.864+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 0x561554163240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561554166570 0x561554163850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561554169360 con 0x561554166570 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56155407c050 con 0x56155415b580 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5615541694e0 con 0x56155415bea0 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa016b6640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 0x561554163240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa026b8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561554166570 0x561554163850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa016b6640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 0x561554163240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:35164/0 (socket says 172.21.15.31:35164) 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa016b6640 1 -- 172.21.15.31:0/188416780 learned_addr learned my addr 172.21.15.31:0/188416780 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa01eb7640 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 0x561554165f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa06e2f740 10 client.0 _ll_register_callbacks cb 0x561553dd3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa016b6640 1 -- 172.21.15.31:0/188416780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 msgr2=0x561554165f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa016b6640 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 0x561554165f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa016b6640 1 -- 172.21.15.31:0/188416780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561554166570 msgr2=0x561554163850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa016b6640 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561554166570 0x561554163850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa016b6640 1 -- 172.21.15.31:0/188416780 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56155404ddc0 con 0x56155415bea0 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ffa016b6640 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 0x561554163240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec000c00 tx=0x7ff9ec004240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da3b0ecb4da7981 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ff9e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.865+0000 7ff9e57fa640 1 -- 172.21.15.31:0/188416780 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec01e040 con 0x56155415bea0 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ff9e57fa640 1 -- 172.21.15.31:0/188416780 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7ff9ec004430 con 0x56155415bea0 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ffa06e2f740 1 -- 172.21.15.31:0/188416780 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9ec0035f0 con 0x56155415bea0 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ff9e57fa640 1 -- 172.21.15.31:0/188416780 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff9ec032050 con 0x56155415bea0 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ffa06e2f740 10 client.25432 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ffa06e2f740 1 -- 172.21.15.31:0/188416780 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561553dd43f0 con 0x56155415bea0 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ff9e57fa640 1 -- 172.21.15.31:0/188416780 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9ec029070 con 0x56155415bea0 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ff9e57fa640 1 -- 172.21.15.31:0/188416780 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff9ec021030 con 0x56155415bea0 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ffa06e2f740 10 client.25432 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ffa06e2f740 10 client.25432 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ffa06e2f740 20 client.25432 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:39.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561554176a80 0x561554178ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ffa06e2f740 4 client.25432 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:39.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.866+0000 7ffa06e2f740 1 -- 172.21.15.31:0/188416780 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff9ec0035f0 con 0x561554176a80 2024-04-25T01:55:39.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.867+0000 7ffa01eb7640 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561554176a80 0x561554178ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.867+0000 7ffa01eb7640 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561554176a80 0x561554178ec0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7a77597c7978f83 server_cookie=7b8aaf63add8a5e0 in_seq=0 out_seq=0 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.867+0000 7ff9e57fa640 10 client.25432 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ff9e57fa640 1 -- 172.21.15.31:0/188416780 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7ff9ec0035f0 con 0x561554176a80 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ff9e57fa640 10 client.25432 handle_command_reply: tid=0 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ffa06e2f740 1 client.25432 shutdown 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ffa06e2f740 1 -- 172.21.15.31:0/188416780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 msgr2=0x561554163240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 0x561554163240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec000c00 tx=0x7ff9ec004240 comp rx=0 tx=0).stop 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ffa06e2f740 1 -- 172.21.15.31:0/188416780 shutdown_connections 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561554176a80 0x561554178ec0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561554166570 0x561554163850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 0x561554163240 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/188416780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56155415b580 0x561554165f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ffa06e2f740 1 -- 172.21.15.31:0/188416780 >> 172.21.15.31:0/188416780 conn(0x5615541578a0 msgr2=0x561554156b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ff9e57fa640 0 client.25432 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:39.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ffa06e2f740 1 -- 172.21.15.31:0/188416780 shutdown_connections 2024-04-25T01:55:39.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.868+0000 7ffa06e2f740 1 -- 172.21.15.31:0/188416780 wait complete. 2024-04-25T01:55:39.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.869+0000 7ffa06e2f740 20 client.25432 trim_cache size 0 max 16384 2024-04-25T01:55:39.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa06e2f740 1 Processor -- start 2024-04-25T01:55:39.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa06e2f740 1 -- start start 2024-04-25T01:55:39.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415b580 0x5615541572c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415bea0 0x561554165a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561554166570 0x56155417e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc1af580 con 0x56155415bea0 2024-04-25T01:55:39.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc1b10c0 con 0x561554166570 2024-04-25T01:55:39.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc1afc50 con 0x56155415b580 2024-04-25T01:55:39.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa016b6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415bea0 0x561554165a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa016b6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415bea0 0x561554165a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33648/0 (socket says 172.21.15.31:33648) 2024-04-25T01:55:39.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa016b6640 1 -- 172.21.15.31:0/435504266 learned_addr learned my addr 172.21.15.31:0/435504266 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa026b8640 1 --2- 172.21.15.31:0/435504266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561554166570 0x56155417e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.882+0000 7ffa01eb7640 1 --2- 172.21.15.31:0/435504266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415b580 0x5615541572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.883+0000 7ffa016b6640 1 -- 172.21.15.31:0/435504266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415b580 msgr2=0x5615541572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.883+0000 7ffa016b6640 1 --2- 172.21.15.31:0/435504266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415b580 0x5615541572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.883+0000 7ffa016b6640 1 -- 172.21.15.31:0/435504266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561554166570 msgr2=0x56155417e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.883+0000 7ffa016b6640 1 --2- 172.21.15.31:0/435504266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561554166570 0x56155417e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.883+0000 7ffa016b6640 1 -- 172.21.15.31:0/435504266 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561554168cd0 con 0x56155415bea0 2024-04-25T01:55:39.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.883+0000 7ffa016b6640 1 --2- 172.21.15.31:0/435504266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415bea0 0x561554165a80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009fb0 tx=0x7ff9ec005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211694d71eb1b063 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.883+0000 7ff9e77fe640 1 -- 172.21.15.31:0/435504266 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec000e60 con 0x56155415bea0 2024-04-25T01:55:39.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.883+0000 7ff9e77fe640 1 -- 172.21.15.31:0/435504266 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9ec029040 con 0x56155415bea0 2024-04-25T01:55:39.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.883+0000 7ff9e77fe640 1 -- 172.21.15.31:0/435504266 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec01e070 con 0x56155415bea0 2024-04-25T01:55:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.884+0000 7ffa06e2f740 1 -- 172.21.15.31:0/435504266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415bea0 msgr2=0x561554165a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.884+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/435504266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415bea0 0x561554165a80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009fb0 tx=0x7ff9ec005db0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.884+0000 7ffa06e2f740 1 -- 172.21.15.31:0/435504266 shutdown_connections 2024-04-25T01:55:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.884+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/435504266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561554166570 0x56155417e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.884+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/435504266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415bea0 0x561554165a80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.884+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/435504266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415b580 0x5615541572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.884+0000 7ffa06e2f740 1 -- 172.21.15.31:0/435504266 >> 172.21.15.31:0/435504266 conn(0x5615541578a0 msgr2=0x56155415d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.884+0000 7ffa06e2f740 1 -- 172.21.15.31:0/435504266 shutdown_connections 2024-04-25T01:55:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.884+0000 7ffa06e2f740 1 -- 172.21.15.31:0/435504266 wait complete. 2024-04-25T01:55:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.884+0000 7ffa06e2f740 1 -- start start 2024-04-25T01:55:39.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa06e2f740 1 Processor -- start 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415b580 0x561554169920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 0x561554169e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa06e2f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561554166570 0x5615541db030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc1b09f0 con 0x56155415b580 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc1b0320 con 0x561554166570 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa06e2f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc1ad580 con 0x56155415bea0 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa01eb7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415b580 0x561554169920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa06e2f740 10 client.0 _ll_register_callbacks cb 0x7ff9ec01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa01eb7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415b580 0x561554169920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33660/0 (socket says 172.21.15.31:33660) 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa01eb7640 1 -- 172.21.15.31:0/3542847548 learned_addr learned my addr 172.21.15.31:0/3542847548 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa026b8640 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561554166570 0x5615541db030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.885+0000 7ffa016b6640 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 0x561554169e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ffa01eb7640 1 -- 172.21.15.31:0/3542847548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 msgr2=0x561554169e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ffa01eb7640 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 0x561554169e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ffa01eb7640 1 -- 172.21.15.31:0/3542847548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561554166570 msgr2=0x5615541db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ffa01eb7640 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561554166570 0x5615541db030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ffa01eb7640 1 -- 172.21.15.31:0/3542847548 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561553e9e7d0 con 0x56155415b580 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ffa01eb7640 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415b580 0x561554169920 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009920 tx=0x7ff9f80096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b718fb59b5da7f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ff9e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ff9e57fa640 1 -- 172.21.15.31:0/3542847548 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f80b3070 con 0x56155415b580 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ffa06e2f740 1 -- 172.21.15.31:0/3542847548 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9ec029040 con 0x56155415b580 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ff9e57fa640 1 -- 172.21.15.31:0/3542847548 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7ff9f80cd220 con 0x56155415b580 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ff9e57fa640 1 -- 172.21.15.31:0/3542847548 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff9f809fff0 con 0x56155415b580 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ffa06e2f740 10 client.15927 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.886+0000 7ffa06e2f740 1 -- 172.21.15.31:0/3542847548 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5615541dc780 con 0x56155415b580 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.887+0000 7ff9e57fa640 1 -- 172.21.15.31:0/3542847548 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9ec029040 con 0x56155415b580 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.887+0000 7ff9e57fa640 1 -- 172.21.15.31:0/3542847548 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff9f8097990 con 0x56155415b580 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.887+0000 7ffa06e2f740 10 client.15927 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.887+0000 7ffa06e2f740 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.887+0000 7ffa06e2f740 20 client.15927 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:39.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.887+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615541dce50 0x5615541df240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.887+0000 7ffa06e2f740 4 client.15927 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.887+0000 7ffa06e2f740 1 -- 172.21.15.31:0/3542847548 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615540cdbb0 con 0x5615541dce50 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.887+0000 7ffa016b6640 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615541dce50 0x5615541df240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa016b6640 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615541dce50 0x5615541df240 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3389073a044f7ced server_cookie=560656881c1dfa26 in_seq=0 out_seq=0 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ff9e57fa640 10 client.15927 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ff9e57fa640 1 -- 172.21.15.31:0/3542847548 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615540cdbb0 con 0x5615541dce50 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ff9e57fa640 10 client.15927 handle_command_reply: tid=0 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 1 client.15927 shutdown 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 1 -- 172.21.15.31:0/3542847548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415b580 msgr2=0x561554169920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415b580 0x561554169920 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009920 tx=0x7ff9f80096c0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 1 -- 172.21.15.31:0/3542847548 shutdown_connections 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615541dce50 0x5615541df240 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561554166570 0x5615541db030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56155415bea0 0x561554169e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 1 --2- 172.21.15.31:0/3542847548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56155415b580 0x561554169920 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 1 -- 172.21.15.31:0/3542847548 >> 172.21.15.31:0/3542847548 conn(0x5615541578a0 msgr2=0x56155415dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ff9e57fa640 0 client.15927 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:39.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 1 -- 172.21.15.31:0/3542847548 shutdown_connections 2024-04-25T01:55:39.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 1 -- 172.21.15.31:0/3542847548 wait complete. 2024-04-25T01:55:39.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.888+0000 7ffa06e2f740 20 client.15927 trim_cache size 0 max 16384 2024-04-25T01:55:39.895 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:55:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 -- 172.21.15.31:0/2414138456 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff9e80a2e20 msgr2=0x7ff9e80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff9e80a2e20 0x7ff9e80a52e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4006010 tx=0x7ff9f403a040 comp rx=0 tx=0).stop 2024-04-25T01:55:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 -- 172.21.15.31:0/2414138456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9fc1029c0 msgr2=0x7ff9fc19a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9fc1029c0 0x7ff9fc19a160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800ea60 tx=0x7ff9f800ef20 comp rx=0 tx=0).stop 2024-04-25T01:55:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 -- 172.21.15.31:0/2414138456 shutdown_connections 2024-04-25T01:55:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff9e80a2e20 0x7ff9e80a52e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9fc10bd00 0x7ff9fc19ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9fc1032e0 0x7ff9fc19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 --2- 172.21.15.31:0/2414138456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9fc1029c0 0x7ff9fc19a160 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 -- 172.21.15.31:0/2414138456 >> 172.21.15.31:0/2414138456 conn(0x7ff9fc077740 msgr2=0x7ff9fc107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 -- 172.21.15.31:0/2414138456 shutdown_connections 2024-04-25T01:55:39.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:39.889+0000 7ffa0411b640 1 -- 172.21.15.31:0/2414138456 wait complete. 2024-04-25T01:55:39.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:39 smithi134 ceph-mon[33098]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:39.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:39 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/398571831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:39.963 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:39 smithi040 ceph-mon[32418]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:39.963 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:39 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/398571831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:40.903 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:55:40.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:40.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:55:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.010+0000 7f991e612640 1 -- 172.21.15.31:0/2123009170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9918100c90 msgr2=0x7f99181063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.010+0000 7f991e612640 1 --2- 172.21.15.31:0/2123009170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9918100c90 0x7f99181063c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f990c009860 tx=0x7f990c02a340 comp rx=0 tx=0).stop 2024-04-25T01:55:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.010+0000 7f991e612640 1 -- 172.21.15.31:0/2123009170 shutdown_connections 2024-04-25T01:55:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.010+0000 7f991e612640 1 --2- 172.21.15.31:0/2123009170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9918106aa0 0x7f991810e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.010+0000 7f991e612640 1 --2- 172.21.15.31:0/2123009170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9918100c90 0x7f99181063c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.010+0000 7f991e612640 1 --2- 172.21.15.31:0/2123009170 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99181002e0 0x7f99181006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.010+0000 7f991e612640 1 -- 172.21.15.31:0/2123009170 >> 172.21.15.31:0/2123009170 conn(0x7f99180758d0 msgr2=0x7f9918075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.010+0000 7f991e612640 1 -- 172.21.15.31:0/2123009170 shutdown_connections 2024-04-25T01:55:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.010+0000 7f991e612640 1 -- 172.21.15.31:0/2123009170 wait complete. 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f991e612640 1 Processor -- start 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f991e612640 1 -- start start 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f991e612640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99181002e0 0x7f9918198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f991e612640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9918106aa0 0x7f9918198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f991e612640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991819c8c0 0x7f991819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f991e612640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9918111a40 con 0x7f991819c8c0 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f991e612640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99181118c0 con 0x7f9918106aa0 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f991e612640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9918111bc0 con 0x7f99181002e0 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f991cbaf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991819c8c0 0x7f991819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f9917fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99181002e0 0x7f9918198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f991cbaf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991819c8c0 0x7f991819cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33680/0 (socket says 172.21.15.31:33680) 2024-04-25T01:55:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f991cbaf640 1 -- 172.21.15.31:0/936448591 learned_addr learned my addr 172.21.15.31:0/936448591 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.011+0000 7f99177fe640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9918106aa0 0x7f9918198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f99177fe640 1 -- 172.21.15.31:0/936448591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99181002e0 msgr2=0x7f9918198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f99177fe640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99181002e0 0x7f9918198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f99177fe640 1 -- 172.21.15.31:0/936448591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991819c8c0 msgr2=0x7f991819cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f99177fe640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991819c8c0 0x7f991819cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f99177fe640 1 -- 172.21.15.31:0/936448591 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991819d3b0 con 0x7f9918106aa0 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f991cbaf640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991819c8c0 0x7f991819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f9917fff640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99181002e0 0x7f9918198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f99177fe640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9918106aa0 0x7f9918198540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f990c035040 tx=0x7f990c030e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f99157fa640 1 -- 172.21.15.31:0/936448591 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c037050 con 0x7f9918106aa0 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f991e612640 1 -- 172.21.15.31:0/936448591 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991819d640 con 0x7f9918106aa0 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f99157fa640 1 -- 172.21.15.31:0/936448591 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f990c038040 con 0x7f9918106aa0 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f99157fa640 1 -- 172.21.15.31:0/936448591 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c0039f0 con 0x7f9918106aa0 2024-04-25T01:55:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.012+0000 7f991e612640 1 -- 172.21.15.31:0/936448591 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99181a4e60 con 0x7f9918106aa0 2024-04-25T01:55:41.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.014+0000 7f99157fa640 1 -- 172.21.15.31:0/936448591 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f990c045030 con 0x7f9918106aa0 2024-04-25T01:55:41.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.015+0000 7f99157fa640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f99000a2e20 0x7f99000a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.015+0000 7f99157fa640 1 -- 172.21.15.31:0/936448591 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f990c0ef230 con 0x7f9918106aa0 2024-04-25T01:55:41.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.015+0000 7f9917fff640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f99000a2e20 0x7f99000a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f99213f6740 1 Processor -- start 2024-04-25T01:55:41.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f99213f6740 1 -- start start 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f9917fff640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f99000a2e20 0x7f99000a52e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9908009b40 tx=0x7f9908005e50 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f99213f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef8980 0x5638dfef6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f99213f6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef7290 0x5638dfef7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f99213f6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff04400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f9917fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef8980 0x5638dfef6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f99213f6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5638dfe5b2c0 con 0x5638dfef7290 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f99213f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5638dfe03910 con 0x5638dfef8980 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f99213f6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5638dfe57280 con 0x5638dff02010 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f99177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef7290 0x5638dfef7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f9917fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef8980 0x5638dfef6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56536/0 (socket says 172.21.15.31:56536) 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f9917fff640 1 -- 172.21.15.31:0/975319088 learned_addr learned my addr 172.21.15.31:0/975319088 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.016+0000 7f991cbaf640 1 --2- 172.21.15.31:0/975319088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff04400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.017+0000 7f9917fff640 1 -- 172.21.15.31:0/975319088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 msgr2=0x5638dff04400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.017+0000 7f9917fff640 1 --2- 172.21.15.31:0/975319088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff04400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.017+0000 7f9917fff640 1 -- 172.21.15.31:0/975319088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef7290 msgr2=0x5638dfef7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.017+0000 7f9917fff640 1 --2- 172.21.15.31:0/975319088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef7290 0x5638dfef7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.017+0000 7f9917fff640 1 -- 172.21.15.31:0/975319088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638dfe14fb0 con 0x5638dfef8980 2024-04-25T01:55:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.017+0000 7f99177fe640 1 --2- 172.21.15.31:0/975319088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef7290 0x5638dfef7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.017+0000 7f9917fff640 1 --2- 172.21.15.31:0/975319088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef8980 0x5638dfef6cc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f990800ef30 tx=0x7f9908011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=727f17a621d885ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.017+0000 7f98fe7fc640 1 -- 172.21.15.31:0/975319088 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990801e640 con 0x5638dfef8980 2024-04-25T01:55:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.017+0000 7f98fe7fc640 1 -- 172.21.15.31:0/975319088 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9908023070 con 0x5638dfef8980 2024-04-25T01:55:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.017+0000 7f98fe7fc640 1 -- 172.21.15.31:0/975319088 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99080109e0 con 0x5638dfef8980 2024-04-25T01:55:41.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.018+0000 7f99213f6740 1 -- 172.21.15.31:0/975319088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef8980 msgr2=0x5638dfef6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.018+0000 7f99213f6740 1 --2- 172.21.15.31:0/975319088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef8980 0x5638dfef6cc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f990800ef30 tx=0x7f9908011000 comp rx=0 tx=0).stop 2024-04-25T01:55:41.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.018+0000 7f99213f6740 1 -- 172.21.15.31:0/975319088 shutdown_connections 2024-04-25T01:55:41.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.018+0000 7f99213f6740 1 --2- 172.21.15.31:0/975319088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff04400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.018+0000 7f99213f6740 1 --2- 172.21.15.31:0/975319088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef7290 0x5638dfef7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.018+0000 7f99213f6740 1 --2- 172.21.15.31:0/975319088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef8980 0x5638dfef6cc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.018+0000 7f99213f6740 1 -- 172.21.15.31:0/975319088 >> 172.21.15.31:0/975319088 conn(0x5638dfebf950 msgr2=0x5638dfef37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.018+0000 7f99213f6740 1 -- 172.21.15.31:0/975319088 shutdown_connections 2024-04-25T01:55:41.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.018+0000 7f99213f6740 1 -- 172.21.15.31:0/975319088 wait complete. 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.018+0000 7f99213f6740 1 -- start start 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99213f6740 1 Processor -- start 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99213f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff0c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99213f6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 0x5638dff0c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99213f6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff06600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99213f6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5638dff04e40 con 0x5638dfef8980 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99213f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5638dfe16e90 con 0x5638dfef7290 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 0x5638dff0c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 0x5638dff0c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33698/0 (socket says 172.21.15.31:33698) 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99177fe640 1 -- 172.21.15.31:0/2619920011 learned_addr learned my addr 172.21.15.31:0/2619920011 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99213f6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5638dff04fc0 con 0x5638dff02010 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f9917fff640 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff0c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99213f6740 10 client.0 _ll_register_callbacks cb 0x5638dfe52970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f991cbaf640 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff06600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99177fe640 1 -- 172.21.15.31:0/2619920011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 msgr2=0x5638dff06600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.019+0000 7f99177fe640 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff06600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f99177fe640 1 -- 172.21.15.31:0/2619920011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 msgr2=0x5638dff0c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f99177fe640 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff0c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f99177fe640 1 -- 172.21.15.31:0/2619920011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638dfc377e0 con 0x5638dfef8980 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f991cbaf640 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff06600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f9917fff640 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff0c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f99177fe640 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 0x5638dff0c7d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f990c0b8060 tx=0x7f990c02a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5f691e7060b6c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f98f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f98f77fe640 1 -- 172.21.15.31:0/2619920011 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c0baff0 con 0x5638dfef8980 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f99213f6740 1 -- 172.21.15.31:0/2619920011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9908023070 con 0x5638dfef8980 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f98f77fe640 1 -- 172.21.15.31:0/2619920011 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f990c0c0dd0 con 0x5638dfef8980 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f98f77fe640 1 -- 172.21.15.31:0/2619920011 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f990c0d3030 con 0x5638dfef8980 2024-04-25T01:55:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f99213f6740 10 client.15939 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f99213f6740 1 -- 172.21.15.31:0/2619920011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5638dfc97f70 con 0x5638dfef8980 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.020+0000 7f98f77fe640 1 -- 172.21.15.31:0/2619920011 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9908023070 con 0x5638dfef8980 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.021+0000 7f98f77fe640 1 -- 172.21.15.31:0/2619920011 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f990c0a4bd0 con 0x5638dfef8980 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.021+0000 7f99213f6740 10 client.15939 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.021+0000 7f99213f6740 10 client.15939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.021+0000 7f99213f6740 20 client.15939 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.021+0000 7f99213f6740 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5638dff16410 0x5638dff18860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.021+0000 7f99213f6740 4 client.15939 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.021+0000 7f99213f6740 1 -- 172.21.15.31:0/2619920011 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638dfe2f5a0 con 0x5638dff16410 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.021+0000 7f9917fff640 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5638dff16410 0x5638dff18860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.021+0000 7f98f77fe640 10 client.15939 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.021+0000 7f9917fff640 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5638dff16410 0x5638dff18860 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e19761dd2cda1a6b server_cookie=cb2a45155b95599a in_seq=0 out_seq=0 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f98f77fe640 1 -- 172.21.15.31:0/2619920011 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638dfe2f5a0 con 0x5638dff16410 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f98f77fe640 10 client.15939 handle_command_reply: tid=0 2024-04-25T01:55:41.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f99213f6740 1 client.15939 shutdown 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f99213f6740 1 -- 172.21.15.31:0/2619920011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 msgr2=0x5638dff0c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f99213f6740 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 0x5638dff0c7d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f990c0b8060 tx=0x7f990c02a340 comp rx=0 tx=0).stop 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f99213f6740 1 -- 172.21.15.31:0/2619920011 shutdown_connections 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f99213f6740 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5638dff16410 0x5638dff18860 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f99213f6740 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff06600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f98f77fe640 0 client.15939 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f99213f6740 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 0x5638dff0c7d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f99213f6740 1 --2- 172.21.15.31:0/2619920011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff0c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.022+0000 7f99213f6740 1 -- 172.21.15.31:0/2619920011 >> 172.21.15.31:0/2619920011 conn(0x5638dfebf950 msgr2=0x5638dfef95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.023+0000 7f99213f6740 1 -- 172.21.15.31:0/2619920011 shutdown_connections 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.023+0000 7f99213f6740 1 -- 172.21.15.31:0/2619920011 wait complete. 2024-04-25T01:55:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.023+0000 7f99213f6740 20 client.15939 trim_cache size 0 max 16384 2024-04-25T01:55:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f99213f6740 1 Processor -- start 2024-04-25T01:55:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f99213f6740 1 -- start start 2024-04-25T01:55:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f99213f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff05880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f99213f6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 0x5638dff0c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f99213f6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff1e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f99213f6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f99181ab110 con 0x5638dfef8980 2024-04-25T01:55:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f99213f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99181acc50 con 0x5638dfef7290 2024-04-25T01:55:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f99213f6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f99181ab7e0 con 0x5638dff02010 2024-04-25T01:55:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f99177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 0x5638dff0c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f9917fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff05880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f9917fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff05880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56554/0 (socket says 172.21.15.31:56554) 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f99177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 0x5638dff0c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33700/0 (socket says 172.21.15.31:33700) 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.041+0000 7f9917fff640 1 -- 172.21.15.31:0/3180029811 learned_addr learned my addr 172.21.15.31:0/3180029811 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.042+0000 7f991cbaf640 1 --2- 172.21.15.31:0/3180029811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff1e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.042+0000 7f9917fff640 1 -- 172.21.15.31:0/3180029811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 msgr2=0x5638dff1e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.042+0000 7f9917fff640 1 --2- 172.21.15.31:0/3180029811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff1e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.042+0000 7f9917fff640 1 -- 172.21.15.31:0/3180029811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 msgr2=0x5638dff0c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.042+0000 7f9917fff640 1 --2- 172.21.15.31:0/3180029811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 0x5638dff0c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.042+0000 7f9917fff640 1 -- 172.21.15.31:0/3180029811 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638dfef3a00 con 0x5638dfef7290 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.042+0000 7f991cbaf640 1 --2- 172.21.15.31:0/3180029811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff1e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.042+0000 7f9917fff640 1 --2- 172.21.15.31:0/3180029811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff05880 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f990800ef30 tx=0x7f9908011f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa538c077cd10c92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.042+0000 7f98fdffb640 1 -- 172.21.15.31:0/3180029811 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9908037070 con 0x5638dfef7290 2024-04-25T01:55:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.042+0000 7f98fdffb640 1 -- 172.21.15.31:0/3180029811 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9908023050 con 0x5638dfef7290 2024-04-25T01:55:41.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.042+0000 7f98fdffb640 1 -- 172.21.15.31:0/3180029811 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99080318a0 con 0x5638dfef7290 2024-04-25T01:55:41.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.043+0000 7f99213f6740 1 -- 172.21.15.31:0/3180029811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 msgr2=0x5638dff05880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.043+0000 7f99213f6740 1 --2- 172.21.15.31:0/3180029811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff05880 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f990800ef30 tx=0x7f9908011f40 comp rx=0 tx=0).stop 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.043+0000 7f99213f6740 1 -- 172.21.15.31:0/3180029811 shutdown_connections 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.043+0000 7f99213f6740 1 --2- 172.21.15.31:0/3180029811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dff02010 0x5638dff1e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.043+0000 7f99213f6740 1 --2- 172.21.15.31:0/3180029811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dfef8980 0x5638dff0c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.043+0000 7f99213f6740 1 --2- 172.21.15.31:0/3180029811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff05880 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.043+0000 7f99213f6740 1 -- 172.21.15.31:0/3180029811 >> 172.21.15.31:0/3180029811 conn(0x5638dfebf950 msgr2=0x5638dff0be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.043+0000 7f99213f6740 1 -- 172.21.15.31:0/3180029811 shutdown_connections 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.043+0000 7f99213f6740 1 -- 172.21.15.31:0/3180029811 wait complete. 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f99213f6740 1 -- start start 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f99213f6740 1 Processor -- start 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f99213f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff104a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f99213f6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dfef8980 0x5638dff109e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f99213f6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dff02010 0x5638dff10f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f99213f6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f99181ac580 con 0x5638dff02010 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f99213f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99181abeb0 con 0x5638dfef7290 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f99213f6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f99181a9110 con 0x5638dfef8980 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f99213f6740 10 client.0 _ll_register_callbacks cb 0x7f990c0097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f9917fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff104a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f9917fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff104a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56570/0 (socket says 172.21.15.31:56570) 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f9917fff640 1 -- 172.21.15.31:0/3572660438 learned_addr learned my addr 172.21.15.31:0/3572660438 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.044+0000 7f991cbaf640 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dff02010 0x5638dff10f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f99177fe640 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dfef8980 0x5638dff109e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f9917fff640 1 -- 172.21.15.31:0/3572660438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dfef8980 msgr2=0x5638dff109e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f9917fff640 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dfef8980 0x5638dff109e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f9917fff640 1 -- 172.21.15.31:0/3572660438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dff02010 msgr2=0x5638dff10f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f9917fff640 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dff02010 0x5638dff10f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f9917fff640 1 -- 172.21.15.31:0/3572660438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638dfb2f170 con 0x5638dfef7290 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f9917fff640 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff104a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9908005b50 tx=0x7f990800ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e61799a296744ea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f98f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f98f77fe640 1 -- 172.21.15.31:0/3572660438 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9908040060 con 0x5638dfef7290 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f99213f6740 1 -- 172.21.15.31:0/3572660438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9908023050 con 0x5638dfef7290 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f98f77fe640 1 -- 172.21.15.31:0/3572660438 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9908005550 con 0x5638dfef7290 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f98f77fe640 1 -- 172.21.15.31:0/3572660438 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9908037070 con 0x5638dfef7290 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f99213f6740 10 client.25556 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f99213f6740 1 -- 172.21.15.31:0/3572660438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5638dff7cbe0 con 0x5638dfef7290 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.045+0000 7f98f77fe640 1 -- 172.21.15.31:0/3572660438 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9908023050 con 0x5638dfef7290 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.046+0000 7f98f77fe640 1 -- 172.21.15.31:0/3572660438 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f990800f040 con 0x5638dfef7290 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.046+0000 7f99213f6740 10 client.25556 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.046+0000 7f99213f6740 10 client.25556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.046+0000 7f99213f6740 20 client.25556 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.046+0000 7f99213f6740 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5638dff7d090 0x5638dff7f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.046+0000 7f99213f6740 4 client.25556 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:55:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.046+0000 7f99213f6740 1 -- 172.21.15.31:0/3572660438 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5638dfe68800 con 0x5638dff7d090 2024-04-25T01:55:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.046+0000 7f99177fe640 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5638dff7d090 0x5638dff7f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.047+0000 7f98f77fe640 10 client.25556 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.047+0000 7f99177fe640 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5638dff7d090 0x5638dff7f4b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cc8f62ca8129c60 server_cookie=8cb71e0ee152b938 in_seq=0 out_seq=0 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f98f77fe640 1 -- 172.21.15.31:0/3572660438 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5638dfe68800 con 0x5638dff7d090 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f98f77fe640 10 client.25556 handle_command_reply: tid=0 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 1 client.25556 shutdown 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 1 -- 172.21.15.31:0/3572660438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 msgr2=0x5638dff104a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff104a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9908005b50 tx=0x7f990800ec60 comp rx=0 tx=0).stop 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 1 -- 172.21.15.31:0/3572660438 shutdown_connections 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5638dff7d090 0x5638dff7f4b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5638dff02010 0x5638dff10f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5638dfef8980 0x5638dff109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 1 --2- 172.21.15.31:0/3572660438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5638dfef7290 0x5638dff104a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 1 -- 172.21.15.31:0/3572660438 >> 172.21.15.31:0/3572660438 conn(0x5638dfebf950 msgr2=0x5638dfef20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f98f77fe640 0 client.25556 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:41.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 1 -- 172.21.15.31:0/3572660438 shutdown_connections 2024-04-25T01:55:41.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 1 -- 172.21.15.31:0/3572660438 wait complete. 2024-04-25T01:55:41.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.048+0000 7f99213f6740 20 client.25556 trim_cache size 0 max 16384 2024-04-25T01:55:41.054 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:55:41.054 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T01:55:41.054 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "7195ef74-69b8-4b8f-95f9-b6334ec58562", 2024-04-25T01:55:41.054 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T01:55:41.054 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:55:41.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 -- 172.21.15.31:0/936448591 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f99000a2e20 msgr2=0x7f99000a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f99000a2e20 0x7f99000a52e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9908009b40 tx=0x7f9908005e50 comp rx=0 tx=0).stop 2024-04-25T01:55:41.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 -- 172.21.15.31:0/936448591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9918106aa0 msgr2=0x7f9918198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9918106aa0 0x7f9918198540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f990c035040 tx=0x7f990c030e00 comp rx=0 tx=0).stop 2024-04-25T01:55:41.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 -- 172.21.15.31:0/936448591 shutdown_connections 2024-04-25T01:55:41.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f99000a2e20 0x7f99000a52e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991819c8c0 0x7f991819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9918106aa0 0x7f9918198540 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 --2- 172.21.15.31:0/936448591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99181002e0 0x7f9918198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 -- 172.21.15.31:0/936448591 >> 172.21.15.31:0/936448591 conn(0x7f99180758d0 msgr2=0x7f9918107c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 -- 172.21.15.31:0/936448591 shutdown_connections 2024-04-25T01:55:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.049+0000 7f991e612640 1 -- 172.21.15.31:0/936448591 wait complete. 2024-04-25T01:55:41.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:55:41.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:55:41.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.170+0000 7f8542e93640 1 -- 172.21.15.31:0/2272427653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f853c100540 msgr2=0x7f853c100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.170+0000 7f8542e93640 1 --2- 172.21.15.31:0/2272427653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f853c100540 0x7f853c100920 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8530009900 tx=0x7f853002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.170+0000 7f8542e93640 1 -- 172.21.15.31:0/2272427653 shutdown_connections 2024-04-25T01:55:41.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.170+0000 7f8542e93640 1 --2- 172.21.15.31:0/2272427653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f853c10e0f0 0x7f853c1104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.170+0000 7f8542e93640 1 --2- 172.21.15.31:0/2272427653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f853c100e60 0x7f853c10dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.170+0000 7f8542e93640 1 --2- 172.21.15.31:0/2272427653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f853c100540 0x7f853c100920 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.170+0000 7f8542e93640 1 -- 172.21.15.31:0/2272427653 >> 172.21.15.31:0/2272427653 conn(0x7f853c077740 msgr2=0x7f853c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.170+0000 7f8542e93640 1 -- 172.21.15.31:0/2272427653 shutdown_connections 2024-04-25T01:55:41.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.170+0000 7f8542e93640 1 -- 172.21.15.31:0/2272427653 wait complete. 2024-04-25T01:55:41.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8542e93640 1 Processor -- start 2024-04-25T01:55:41.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8542e93640 1 -- start start 2024-04-25T01:55:41.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8542e93640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f853c100e60 0x7f853c19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8542e93640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f853c10e0f0 0x7f853c19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8542e93640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f853c19eb60 0x7f853c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8542e93640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f853c113ba0 con 0x7f853c100e60 2024-04-25T01:55:41.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8542e93640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f853c113a20 con 0x7f853c19eb60 2024-04-25T01:55:41.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8542e93640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f853c113d20 con 0x7f853c10e0f0 2024-04-25T01:55:41.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8540c2f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f853c100e60 0x7f853c19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8540c2f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f853c100e60 0x7f853c19a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33728/0 (socket says 172.21.15.31:33728) 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8540c2f640 1 -- 172.21.15.31:0/1381158829 learned_addr learned my addr 172.21.15.31:0/1381158829 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f8541430640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f853c19eb60 0x7f853c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.171+0000 7f853bfff640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f853c10e0f0 0x7f853c19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f8540c2f640 1 -- 172.21.15.31:0/1381158829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f853c10e0f0 msgr2=0x7f853c19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f8540c2f640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f853c10e0f0 0x7f853c19a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f8540c2f640 1 -- 172.21.15.31:0/1381158829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f853c19eb60 msgr2=0x7f853c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f8540c2f640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f853c19eb60 0x7f853c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f8540c2f640 1 -- 172.21.15.31:0/1381158829 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853c19f650 con 0x7f853c100e60 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f853bfff640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f853c10e0f0 0x7f853c19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f8540c2f640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f853c100e60 0x7f853c19a2a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8530006010 tx=0x7f85300040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f8539ffb640 1 -- 172.21.15.31:0/1381158829 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853003f030 con 0x7f853c100e60 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f8539ffb640 1 -- 172.21.15.31:0/1381158829 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85300044c0 con 0x7f853c100e60 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f8542e93640 1 -- 172.21.15.31:0/1381158829 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853c19f8e0 con 0x7f853c100e60 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f8539ffb640 1 -- 172.21.15.31:0/1381158829 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530035070 con 0x7f853c100e60 2024-04-25T01:55:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.172+0000 7f8542e93640 1 -- 172.21.15.31:0/1381158829 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853c1a7050 con 0x7f853c100e60 2024-04-25T01:55:41.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.175+0000 7f8539ffb640 1 -- 172.21.15.31:0/1381158829 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f8530042030 con 0x7f853c100e60 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.175+0000 7f8539ffb640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f852c0a2e20 0x7f852c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.175+0000 7f8539ffb640 1 -- 172.21.15.31:0/1381158829 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f85300ef8b0 con 0x7f853c100e60 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.175+0000 7f853bfff640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f852c0a2e20 0x7f852c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.175+0000 7f8545b9d740 1 Processor -- start 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f8545b9d740 1 -- start start 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 0x564f79e4da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4df50 0x564f79e580e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e58620 0x564f79e5aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f8545b9d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564f79db26b0 con 0x564f79e4d630 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f8545b9d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564f79d5a880 con 0x564f79e4df50 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f8545b9d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564f79dae670 con 0x564f79e58620 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f8540c2f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 0x564f79e4da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f853bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4df50 0x564f79e580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f853bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4df50 0x564f79e580e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56604/0 (socket says 172.21.15.31:56604) 2024-04-25T01:55:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f853bfff640 1 -- 172.21.15.31:0/2266528987 learned_addr learned my addr 172.21.15.31:0/2266528987 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f853bfff640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f852c0a2e20 0x7f852c0a52e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f853c10efa0 tx=0x7f8528005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f8541430640 1 --2- 172.21.15.31:0/2266528987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e58620 0x564f79e5aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f853bfff640 1 -- 172.21.15.31:0/2266528987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e58620 msgr2=0x564f79e5aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f853bfff640 1 --2- 172.21.15.31:0/2266528987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e58620 0x564f79e5aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f853bfff640 1 -- 172.21.15.31:0/2266528987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 msgr2=0x564f79e4da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f853bfff640 1 --2- 172.21.15.31:0/2266528987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 0x564f79e4da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.176+0000 7f853bfff640 1 -- 172.21.15.31:0/2266528987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f79b907d0 con 0x564f79e4df50 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.177+0000 7f8540c2f640 1 --2- 172.21.15.31:0/2266528987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 0x564f79e4da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.177+0000 7f853bfff640 1 --2- 172.21.15.31:0/2266528987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4df50 0x564f79e580e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8528012480 tx=0x7f8528014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaebb879878c3c22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.177+0000 7f8526ffd640 1 -- 172.21.15.31:0/2266528987 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8528004220 con 0x564f79e4df50 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.177+0000 7f8526ffd640 1 -- 172.21.15.31:0/2266528987 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85280043c0 con 0x564f79e4df50 2024-04-25T01:55:41.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.177+0000 7f8526ffd640 1 -- 172.21.15.31:0/2266528987 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852801f9a0 con 0x564f79e4df50 2024-04-25T01:55:41.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.178+0000 7f8545b9d740 1 -- 172.21.15.31:0/2266528987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4df50 msgr2=0x564f79e580e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.178+0000 7f8545b9d740 1 --2- 172.21.15.31:0/2266528987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4df50 0x564f79e580e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8528012480 tx=0x7f8528014000 comp rx=0 tx=0).stop 2024-04-25T01:55:41.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.178+0000 7f8545b9d740 1 -- 172.21.15.31:0/2266528987 shutdown_connections 2024-04-25T01:55:41.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.178+0000 7f8545b9d740 1 --2- 172.21.15.31:0/2266528987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e58620 0x564f79e5aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.178+0000 7f8545b9d740 1 --2- 172.21.15.31:0/2266528987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4df50 0x564f79e580e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.178+0000 7f8545b9d740 1 --2- 172.21.15.31:0/2266528987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 0x564f79e4da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.178+0000 7f8545b9d740 1 -- 172.21.15.31:0/2266528987 >> 172.21.15.31:0/2266528987 conn(0x564f79e49950 msgr2=0x564f79e49d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.178+0000 7f8545b9d740 1 -- 172.21.15.31:0/2266528987 shutdown_connections 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.178+0000 7f8545b9d740 1 -- 172.21.15.31:0/2266528987 wait complete. 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.178+0000 7f8545b9d740 1 -- start start 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.178+0000 7f8545b9d740 1 Processor -- start 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e555c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4df50 0x564f79e55b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e58620 0x564f79e56040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8545b9d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564f79e5b410 con 0x564f79e4df50 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8545b9d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564f79d6df50 con 0x564f79e4d630 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8545b9d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564f79e5b590 con 0x564f79e58620 2024-04-25T01:55:41.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8545b9d740 10 client.0 _ll_register_callbacks cb 0x564f79da9000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f853bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4df50 0x564f79e55b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8540c2f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f853bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4df50 0x564f79e55b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33738/0 (socket says 172.21.15.31:33738) 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8540c2f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e555c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56616/0 (socket says 172.21.15.31:56616) 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f853bfff640 1 -- 172.21.15.31:0/2474840568 learned_addr learned my addr 172.21.15.31:0/2474840568 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8541430640 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e58620 0x564f79e56040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f853bfff640 1 -- 172.21.15.31:0/2474840568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e58620 msgr2=0x564f79e56040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f853bfff640 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e58620 0x564f79e56040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f853bfff640 1 -- 172.21.15.31:0/2474840568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 msgr2=0x564f79e555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f853bfff640 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f853bfff640 1 -- 172.21.15.31:0/2474840568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f79ac63f0 con 0x564f79e4df50 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8541430640 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e58620 0x564f79e56040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.179+0000 7f8540c2f640 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.180+0000 7f853bfff640 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4df50 0x564f79e55b00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8528012450 tx=0x7f85280056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a92d09eaf327ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.180+0000 7f851bfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:41.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.180+0000 7f851bfff640 1 -- 172.21.15.31:0/2474840568 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8528037060 con 0x564f79e4df50 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.180+0000 7f851bfff640 1 -- 172.21.15.31:0/2474840568 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f8528003bc0 con 0x564f79e4df50 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.180+0000 7f8545b9d740 1 -- 172.21.15.31:0/2474840568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85280043c0 con 0x564f79e4df50 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.180+0000 7f851bfff640 1 -- 172.21.15.31:0/2474840568 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f852801f720 con 0x564f79e4df50 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.180+0000 7f8545b9d740 10 client.15957 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.180+0000 7f8545b9d740 1 -- 172.21.15.31:0/2474840568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564f79a860f0 con 0x564f79e4df50 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.180+0000 7f851bfff640 1 -- 172.21.15.31:0/2474840568 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85280043c0 con 0x564f79e4df50 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.181+0000 7f851bfff640 1 -- 172.21.15.31:0/2474840568 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8528027900 con 0x564f79e4df50 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.181+0000 7f8545b9d740 10 client.15957 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.181+0000 7f8545b9d740 10 client.15957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.181+0000 7f8545b9d740 20 client.15957 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.181+0000 7f8545b9d740 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564f79e688d0 0x564f79e6acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.181+0000 7f8545b9d740 4 client.15957 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.181+0000 7f8545b9d740 1 -- 172.21.15.31:0/2474840568 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f79dbfbf0 con 0x564f79e688d0 2024-04-25T01:55:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.181+0000 7f8540c2f640 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564f79e688d0 0x564f79e6acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.181+0000 7f8540c2f640 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564f79e688d0 0x564f79e6acc0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dd1d455fe4fc319 server_cookie=68f9ae0b9c3de5bf in_seq=0 out_seq=0 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.181+0000 7f851bfff640 10 client.15957 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f851bfff640 1 -- 172.21.15.31:0/2474840568 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f79dbfbf0 con 0x564f79e688d0 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f851bfff640 10 client.15957 handle_command_reply: tid=0 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f8545b9d740 1 client.15957 shutdown 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f8545b9d740 1 -- 172.21.15.31:0/2474840568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4df50 msgr2=0x564f79e55b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f8545b9d740 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4df50 0x564f79e55b00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8528012450 tx=0x7f85280056d0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f8545b9d740 1 -- 172.21.15.31:0/2474840568 shutdown_connections 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f8545b9d740 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564f79e688d0 0x564f79e6acc0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f8545b9d740 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e58620 0x564f79e56040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f851bfff640 0 client.15957 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f8545b9d740 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4df50 0x564f79e55b00 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f8545b9d740 1 --2- 172.21.15.31:0/2474840568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f8545b9d740 1 -- 172.21.15.31:0/2474840568 >> 172.21.15.31:0/2474840568 conn(0x564f79e49950 msgr2=0x564f79e4f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f8545b9d740 1 -- 172.21.15.31:0/2474840568 shutdown_connections 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.182+0000 7f8545b9d740 1 -- 172.21.15.31:0/2474840568 wait complete. 2024-04-25T01:55:41.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.183+0000 7f8545b9d740 20 client.15957 trim_cache size 0 max 16384 2024-04-25T01:55:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.200+0000 7f8545b9d740 1 Processor -- start 2024-04-25T01:55:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.200+0000 7f8545b9d740 1 -- start start 2024-04-25T01:55:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.200+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 0x564f79e57cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.200+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e4df50 0x564f79e48af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.200+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e58620 0x564f79e4fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.200+0000 7f8545b9d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1afa60 con 0x564f79e4d630 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.200+0000 7f8545b9d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1b15a0 con 0x564f79e58620 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.200+0000 7f8545b9d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1b0130 con 0x564f79e4df50 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f8540c2f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 0x564f79e57cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f8540c2f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 0x564f79e57cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33744/0 (socket says 172.21.15.31:33744) 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f8540c2f640 1 -- 172.21.15.31:0/3217280705 learned_addr learned my addr 172.21.15.31:0/3217280705 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f8541430640 1 --2- 172.21.15.31:0/3217280705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e58620 0x564f79e4fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f853bfff640 1 --2- 172.21.15.31:0/3217280705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e4df50 0x564f79e48af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f8540c2f640 1 -- 172.21.15.31:0/3217280705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e4df50 msgr2=0x564f79e48af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f8540c2f640 1 --2- 172.21.15.31:0/3217280705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e4df50 0x564f79e48af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f8540c2f640 1 -- 172.21.15.31:0/3217280705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e58620 msgr2=0x564f79e4fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f8540c2f640 1 --2- 172.21.15.31:0/3217280705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e58620 0x564f79e4fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f8540c2f640 1 -- 172.21.15.31:0/3217280705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f79e3b090 con 0x564f79e4d630 2024-04-25T01:55:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f8540c2f640 1 --2- 172.21.15.31:0/3217280705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 0x564f79e57cf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f85300ed650 tx=0x7f85300ef040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=714e0bd1b09d5b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f85267fc640 1 -- 172.21.15.31:0/3217280705 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85300c4910 con 0x564f79e4d630 2024-04-25T01:55:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f85267fc640 1 -- 172.21.15.31:0/3217280705 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85300a4070 con 0x564f79e4d630 2024-04-25T01:55:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.201+0000 7f85267fc640 1 -- 172.21.15.31:0/3217280705 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85300c46e0 con 0x564f79e4d630 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.202+0000 7f8545b9d740 1 -- 172.21.15.31:0/3217280705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 msgr2=0x564f79e57cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.202+0000 7f8545b9d740 1 --2- 172.21.15.31:0/3217280705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 0x564f79e57cf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f85300ed650 tx=0x7f85300ef040 comp rx=0 tx=0).stop 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.202+0000 7f8545b9d740 1 -- 172.21.15.31:0/3217280705 shutdown_connections 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.202+0000 7f8545b9d740 1 --2- 172.21.15.31:0/3217280705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e58620 0x564f79e4fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.202+0000 7f8545b9d740 1 --2- 172.21.15.31:0/3217280705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e4df50 0x564f79e48af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.202+0000 7f8545b9d740 1 --2- 172.21.15.31:0/3217280705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e4d630 0x564f79e57cf0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.202+0000 7f8545b9d740 1 -- 172.21.15.31:0/3217280705 >> 172.21.15.31:0/3217280705 conn(0x564f79e49950 msgr2=0x564f79e55580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.202+0000 7f8545b9d740 1 -- 172.21.15.31:0/3217280705 shutdown_connections 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.202+0000 7f8545b9d740 1 -- 172.21.15.31:0/3217280705 wait complete. 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8545b9d740 1 -- start start 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8545b9d740 1 Processor -- start 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e52560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e4df50 0x564f79e52aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8545b9d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e58620 0x564f79ecdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8545b9d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1b0ed0 con 0x564f79e58620 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8545b9d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1b0800 con 0x564f79e4d630 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8545b9d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1ada40 con 0x564f79e4df50 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8545b9d740 10 client.0 _ll_register_callbacks cb 0x7f8534007880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8540c2f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e52560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8540c2f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e52560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56634/0 (socket says 172.21.15.31:56634) 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8540c2f640 1 -- 172.21.15.31:0/867908509 learned_addr learned my addr 172.21.15.31:0/867908509 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f853bfff640 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e4df50 0x564f79e52aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.203+0000 7f8541430640 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e58620 0x564f79ecdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f8540c2f640 1 -- 172.21.15.31:0/867908509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e4df50 msgr2=0x564f79e52aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f8540c2f640 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e4df50 0x564f79e52aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f8540c2f640 1 -- 172.21.15.31:0/867908509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e58620 msgr2=0x564f79ecdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f8540c2f640 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e58620 0x564f79ecdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f8540c2f640 1 -- 172.21.15.31:0/867908509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f79ece190 con 0x564f79e4d630 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f8540c2f640 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e52560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f85300b8100 tx=0x7f85300bbe20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ad2e9848e78a7be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f851bfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f851bfff640 1 -- 172.21.15.31:0/867908509 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85300c4910 con 0x564f79e4d630 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f851bfff640 1 -- 172.21.15.31:0/867908509 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f85300b9d40 con 0x564f79e4d630 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f8545b9d740 1 -- 172.21.15.31:0/867908509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85300a4070 con 0x564f79e4d630 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f851bfff640 1 -- 172.21.15.31:0/867908509 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f85300b9fa0 con 0x564f79e4d630 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f8545b9d740 10 client.25574 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.204+0000 7f8545b9d740 1 -- 172.21.15.31:0/867908509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564f79ecf200 con 0x564f79e4d630 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.205+0000 7f851bfff640 1 -- 172.21.15.31:0/867908509 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85300a4070 con 0x564f79e4d630 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.205+0000 7f851bfff640 1 -- 172.21.15.31:0/867908509 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f85300c4b60 con 0x564f79e4d630 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.205+0000 7f8545b9d740 10 client.25574 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.205+0000 7f8545b9d740 10 client.25574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.205+0000 7f8545b9d740 20 client.25574 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.205+0000 7f8545b9d740 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564f79e688d0 0x564f79ed18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.205+0000 7f8545b9d740 4 client.25574 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:55:41.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.205+0000 7f8545b9d740 1 -- 172.21.15.31:0/867908509 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f79db9f40 con 0x564f79e688d0 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.205+0000 7f853bfff640 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564f79e688d0 0x564f79ed18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.205+0000 7f853bfff640 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564f79e688d0 0x564f79ed18e0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41d643cdaca30c08 server_cookie=77e9d71dd2ae4beb in_seq=0 out_seq=0 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.205+0000 7f851bfff640 10 client.25574 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f851bfff640 1 -- 172.21.15.31:0/867908509 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564f79db9f40 con 0x564f79e688d0 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f851bfff640 10 client.25574 handle_command_reply: tid=0 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 1 client.25574 shutdown 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 1 -- 172.21.15.31:0/867908509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 msgr2=0x564f79e52560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e52560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f85300b8100 tx=0x7f85300bbe20 comp rx=0 tx=0).stop 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 1 -- 172.21.15.31:0/867908509 shutdown_connections 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564f79e688d0 0x564f79ed18e0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f79e58620 0x564f79ecdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f851bfff640 0 client.25574 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564f79e4df50 0x564f79e52aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 1 --2- 172.21.15.31:0/867908509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564f79e4d630 0x564f79e52560 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 1 -- 172.21.15.31:0/867908509 >> 172.21.15.31:0/867908509 conn(0x564f79e49950 msgr2=0x564f79e48970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 1 -- 172.21.15.31:0/867908509 shutdown_connections 2024-04-25T01:55:41.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 1 -- 172.21.15.31:0/867908509 wait complete. 2024-04-25T01:55:41.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.206+0000 7f8545b9d740 20 client.25574 trim_cache size 0 max 16384 2024-04-25T01:55:41.213 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:55:41.213 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:55:41.213 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T01:55:41.213 INFO:teuthology.orchestra.run.smithi031.stdout: "7195ef74-69b8-4b8f-95f9-b6334ec58562": { 2024-04-25T01:55:41.213 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T01:55:41.213 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "7195ef74-69b8-4b8f-95f9-b6334ec58562", 2024-04-25T01:55:41.213 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T01:55:41.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T01:55:41.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T01:55:41.213 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:55:41.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 -- 172.21.15.31:0/1381158829 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f852c0a2e20 msgr2=0x7f852c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f852c0a2e20 0x7f852c0a52e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f853c10efa0 tx=0x7f8528005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:41.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 -- 172.21.15.31:0/1381158829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f853c100e60 msgr2=0x7f853c19a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:41.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f853c100e60 0x7f853c19a2a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8530006010 tx=0x7f85300040c0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 -- 172.21.15.31:0/1381158829 shutdown_connections 2024-04-25T01:55:41.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f852c0a2e20 0x7f852c0a52e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f853c19eb60 0x7f853c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f853c10e0f0 0x7f853c19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 --2- 172.21.15.31:0/1381158829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f853c100e60 0x7f853c19a2a0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:41.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 -- 172.21.15.31:0/1381158829 >> 172.21.15.31:0/1381158829 conn(0x7f853c077740 msgr2=0x7f853c10f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:41.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 -- 172.21.15.31:0/1381158829 shutdown_connections 2024-04-25T01:55:41.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:55:41.207+0000 7f8542e93640 1 -- 172.21.15.31:0/1381158829 wait complete. 2024-04-25T01:55:41.220 INFO:tasks.cephfs.filesystem:scrub status for tag:7195ef74-69b8-4b8f-95f9-b6334ec58562 - {'path': '/', 'tag': '7195ef74-69b8-4b8f-95f9-b6334ec58562', 'options': 'recursive,force'} 2024-04-25T01:55:41.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:41 smithi134 ceph-mon[33098]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T01:55:41.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:41 smithi134 ceph-mon[33098]: scrub queued for path: / 2024-04-25T01:55:41.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:41 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:41.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:41 smithi134 ceph-mon[33098]: scrub summary: active paths [/] 2024-04-25T01:55:41.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:41 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:41.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:41 smithi134 ceph-mon[33098]: scrub summary: idle 2024-04-25T01:55:41.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:41 smithi031 ceph-mon[29204]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T01:55:41.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:41 smithi031 ceph-mon[29204]: scrub queued for path: / 2024-04-25T01:55:41.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:41 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:41.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:41 smithi031 ceph-mon[29204]: scrub summary: active paths [/] 2024-04-25T01:55:41.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:41 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:41.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:41 smithi031 ceph-mon[29204]: scrub summary: idle 2024-04-25T01:55:42.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:41 smithi040 ceph-mon[32418]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T01:55:42.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:41 smithi040 ceph-mon[32418]: scrub queued for path: / 2024-04-25T01:55:42.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:41 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:42.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:41 smithi040 ceph-mon[32418]: scrub summary: active paths [/] 2024-04-25T01:55:42.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:41 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:42.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:41 smithi040 ceph-mon[32418]: scrub summary: idle 2024-04-25T01:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.091+0000 7f66cb970640 1 -- 172.21.15.31:0/2614068020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c41032f0 msgr2=0x7f66c410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.091+0000 7f66cb970640 1 --2- 172.21.15.31:0/2614068020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c41032f0 0x7f66c410b7d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009900 tx=0x7f66b402a510 comp rx=0 tx=0).stop 2024-04-25T01:55:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.091+0000 7f66cb970640 1 -- 172.21.15.31:0/2614068020 shutdown_connections 2024-04-25T01:55:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.091+0000 7f66cb970640 1 --2- 172.21.15.31:0/2614068020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c410bd10 0x7f66c410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.091+0000 7f66cb970640 1 --2- 172.21.15.31:0/2614068020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c41032f0 0x7f66c410b7d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.091+0000 7f66cb970640 1 --2- 172.21.15.31:0/2614068020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c41029d0 0x7f66c4102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.091+0000 7f66cb970640 1 -- 172.21.15.31:0/2614068020 >> 172.21.15.31:0/2614068020 conn(0x7f66c40777c0 msgr2=0x7f66c4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.091+0000 7f66cb970640 1 -- 172.21.15.31:0/2614068020 shutdown_connections 2024-04-25T01:55:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.091+0000 7f66cb970640 1 -- 172.21.15.31:0/2614068020 wait complete. 2024-04-25T01:55:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.092+0000 7f66cb970640 1 Processor -- start 2024-04-25T01:55:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.092+0000 7f66cb970640 1 -- start start 2024-04-25T01:55:43.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.092+0000 7f66cb970640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c41029d0 0x7f66c4100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.092+0000 7f66cb970640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c41032f0 0x7f66c4100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.092+0000 7f66cb970640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c410bd10 0x7f66c41010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.092+0000 7f66cb970640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66c41119c0 con 0x7f66c41032f0 2024-04-25T01:55:43.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.092+0000 7f66cb970640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f66c4111840 con 0x7f66c410bd10 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.092+0000 7f66cb970640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f66c4111b40 con 0x7f66c41029d0 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.092+0000 7f66c8f0b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c41032f0 0x7f66c4100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.092+0000 7f66c8f0b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c41032f0 0x7f66c4100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35536/0 (socket says 172.21.15.31:35536) 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.092+0000 7f66c8f0b640 1 -- 172.21.15.31:0/4205203835 learned_addr learned my addr 172.21.15.31:0/4205203835 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66c970c640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c41029d0 0x7f66c4100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66c9f0d640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c410bd10 0x7f66c41010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66c8f0b640 1 -- 172.21.15.31:0/4205203835 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c41029d0 msgr2=0x7f66c4100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66c8f0b640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c41029d0 0x7f66c4100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66c8f0b640 1 -- 172.21.15.31:0/4205203835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c410bd10 msgr2=0x7f66c41010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66c8f0b640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c410bd10 0x7f66c41010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66c8f0b640 1 -- 172.21.15.31:0/4205203835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c41a63c0 con 0x7f66c41032f0 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66c8f0b640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c41032f0 0x7f66c4100b60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f66b4005940 tx=0x7f66b40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66ba7fc640 1 -- 172.21.15.31:0/4205203835 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b403f030 con 0x7f66c41032f0 2024-04-25T01:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66ba7fc640 1 -- 172.21.15.31:0/4205203835 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66b4003cb0 con 0x7f66c41032f0 2024-04-25T01:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66cb970640 1 -- 172.21.15.31:0/4205203835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c41a6650 con 0x7f66c41032f0 2024-04-25T01:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66ba7fc640 1 -- 172.21.15.31:0/4205203835 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b4035070 con 0x7f66c41032f0 2024-04-25T01:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.093+0000 7f66cb970640 1 -- 172.21.15.31:0/4205203835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c41a6b40 con 0x7f66c41032f0 2024-04-25T01:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.097+0000 7f66ba7fc640 1 -- 172.21.15.31:0/4205203835 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f66b4042030 con 0x7f66c41032f0 2024-04-25T01:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.097+0000 7f66ce322740 1 Processor -- start 2024-04-25T01:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.097+0000 7f66ce322740 1 -- start start 2024-04-25T01:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.097+0000 7f66ce322740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 0x56351a822620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.097+0000 7f66ce322740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a82ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66ce322740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a82d230 0x56351a82f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66ba7fc640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66b00a2ec0 0x7f66b00a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66ce322740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56351a786cb0 con 0x56351a822b60 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66ce322740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56351a72f5b0 con 0x56351a822240 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66ce322740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56351a782c70 con 0x56351a82d230 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c970c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a82ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66ba7fc640 1 -- 172.21.15.31:0/4205203835 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f66b40ef560 con 0x7f66c41032f0 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c970c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a82ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35548/0 (socket says 172.21.15.31:35548) 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c970c640 1 -- 172.21.15.31:0/2085430295 learned_addr learned my addr 172.21.15.31:0/2085430295 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c9f0d640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66b00a2ec0 0x7f66b00a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c8f0b640 1 --2- 172.21.15.31:0/2085430295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a82d230 0x56351a82f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c970c640 1 -- 172.21.15.31:0/2085430295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a82d230 msgr2=0x56351a82f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c970c640 1 --2- 172.21.15.31:0/2085430295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a82d230 0x56351a82f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c970c640 1 -- 172.21.15.31:0/2085430295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 msgr2=0x56351a822620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c970c640 1 --2- 172.21.15.31:0/2085430295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 0x56351a822620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c970c640 1 -- 172.21.15.31:0/2085430295 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56351a49b210 con 0x56351a822b60 2024-04-25T01:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c8f0b640 1 --2- 172.21.15.31:0/2085430295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a82d230 0x56351a82f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.098+0000 7f66c970c640 1 --2- 172.21.15.31:0/2085430295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a82ccf0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f66c000bcd0 tx=0x7f66c00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9d0c4eaeb36fcb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.099+0000 7f66af7fe640 1 -- 172.21.15.31:0/2085430295 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0005220 con 0x56351a822b60 2024-04-25T01:55:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.099+0000 7f66c9f0d640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66b00a2ec0 0x7f66b00a5380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f66bc005fd0 tx=0x7f66bc009500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.099+0000 7f66af7fe640 1 -- 172.21.15.31:0/2085430295 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66c0004c50 con 0x56351a822b60 2024-04-25T01:55:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.099+0000 7f66af7fe640 1 -- 172.21.15.31:0/2085430295 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0011730 con 0x56351a822b60 2024-04-25T01:55:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.099+0000 7f66ce322740 1 -- 172.21.15.31:0/2085430295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 msgr2=0x56351a82ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.099+0000 7f66ce322740 1 --2- 172.21.15.31:0/2085430295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a82ccf0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f66c000bcd0 tx=0x7f66c00107a0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.100+0000 7f66ce322740 1 -- 172.21.15.31:0/2085430295 shutdown_connections 2024-04-25T01:55:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.100+0000 7f66ce322740 1 --2- 172.21.15.31:0/2085430295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a82d230 0x56351a82f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.100+0000 7f66ce322740 1 --2- 172.21.15.31:0/2085430295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a82ccf0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.100+0000 7f66ce322740 1 --2- 172.21.15.31:0/2085430295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 0x56351a822620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.100+0000 7f66ce322740 1 -- 172.21.15.31:0/2085430295 >> 172.21.15.31:0/2085430295 conn(0x56351a81e560 msgr2=0x56351a81e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.100+0000 7f66ce322740 1 -- 172.21.15.31:0/2085430295 shutdown_connections 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.100+0000 7f66ce322740 1 -- 172.21.15.31:0/2085430295 wait complete. 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.100+0000 7f66ce322740 1 -- start start 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.100+0000 7f66ce322740 1 Processor -- start 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66ce322740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 0x56351a82a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66ce322740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a82a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66ce322740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 0x56351a82aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66ce322740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56351a830020 con 0x56351a82d230 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66ce322740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56351a742b50 con 0x56351a822240 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66ce322740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56351a8301a0 con 0x56351a822b60 2024-04-25T01:55:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66ce322740 10 client.0 _ll_register_callbacks cb 0x56351a77dca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c9f0d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 0x56351a82aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c8f0b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a82a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c8f0b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a82a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:49906/0 (socket says 172.21.15.31:49906) 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c9f0d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 0x56351a82aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35560/0 (socket says 172.21.15.31:35560) 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c8f0b640 1 -- 172.21.15.31:0/4050565769 learned_addr learned my addr 172.21.15.31:0/4050565769 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c970c640 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 0x56351a82a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c9f0d640 1 -- 172.21.15.31:0/4050565769 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 msgr2=0x56351a82a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c9f0d640 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a82a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c9f0d640 1 -- 172.21.15.31:0/4050565769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 msgr2=0x56351a82a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c9f0d640 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 0x56351a82a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c9f0d640 1 -- 172.21.15.31:0/4050565769 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56351a567620 con 0x56351a82d230 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.101+0000 7f66c8f0b640 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a82a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.102+0000 7f66c9f0d640 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 0x56351a82aca0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f66bc004280 tx=0x7f66bc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cfa40bfaf79b959 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.102+0000 7f66acff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.102+0000 7f66acff9640 1 -- 172.21.15.31:0/4050565769 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc022070 con 0x56351a82d230 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.102+0000 7f66ce322740 1 -- 172.21.15.31:0/4050565769 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66c0004c50 con 0x56351a82d230 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.102+0000 7f66acff9640 1 -- 172.21.15.31:0/4050565769 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f66bc00fad0 con 0x56351a82d230 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.102+0000 7f66acff9640 1 -- 172.21.15.31:0/4050565769 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f66bc02e050 con 0x56351a82d230 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.102+0000 7f66ce322740 10 client.15975 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.102+0000 7f66ce322740 1 -- 172.21.15.31:0/4050565769 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56351a83c290 con 0x56351a82d230 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.102+0000 7f66acff9640 1 -- 172.21.15.31:0/4050565769 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66c0004c50 con 0x56351a82d230 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.103+0000 7f66acff9640 1 -- 172.21.15.31:0/4050565769 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f66bc0263b0 con 0x56351a82d230 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.103+0000 7f66ce322740 10 client.15975 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.103+0000 7f66ce322740 10 client.15975 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.103+0000 7f66ce322740 20 client.15975 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.103+0000 7f66ce322740 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56351a83dae0 0x56351a83ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.103+0000 7f66ce322740 4 client.15975 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.103+0000 7f66ce322740 1 -- 172.21.15.31:0/4050565769 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56351a7941f0 con 0x56351a83dae0 2024-04-25T01:55:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.103+0000 7f66c970c640 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56351a83dae0 0x56351a83ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.104+0000 7f66acff9640 10 client.15975 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.104+0000 7f66c970c640 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56351a83dae0 0x56351a83ff60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3f896963aee5765 server_cookie=14982423ce79352f in_seq=0 out_seq=0 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.104+0000 7f66acff9640 1 -- 172.21.15.31:0/4050565769 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56351a7941f0 con 0x56351a83dae0 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.104+0000 7f66acff9640 10 client.15975 handle_command_reply: tid=0 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.104+0000 7f66ce322740 1 client.15975 shutdown 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66ce322740 1 -- 172.21.15.31:0/4050565769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 msgr2=0x56351a82aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66ce322740 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 0x56351a82aca0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f66bc004280 tx=0x7f66bc004ab0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66ce322740 1 -- 172.21.15.31:0/4050565769 shutdown_connections 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66ce322740 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56351a83dae0 0x56351a83ff60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66ce322740 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 0x56351a82aca0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66ce322740 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a82a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66ce322740 1 --2- 172.21.15.31:0/4050565769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 0x56351a82a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66ce322740 1 -- 172.21.15.31:0/4050565769 >> 172.21.15.31:0/4050565769 conn(0x56351a81e560 msgr2=0x56351a824360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66acff9640 0 client.15975 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66ce322740 1 -- 172.21.15.31:0/4050565769 shutdown_connections 2024-04-25T01:55:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66ce322740 1 -- 172.21.15.31:0/4050565769 wait complete. 2024-04-25T01:55:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.105+0000 7f66ce322740 20 client.15975 trim_cache size 0 max 16384 2024-04-25T01:55:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.133+0000 7f66ce322740 1 Processor -- start 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.133+0000 7f66ce322740 1 -- start start 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.133+0000 7f66ce322740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822240 0x56351a82c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.133+0000 7f66ce322740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a824ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.133+0000 7f66ce322740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a82d230 0x56351a82a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.133+0000 7f66ce322740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66c41af400 con 0x56351a822b60 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66ce322740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f66c41b0f40 con 0x56351a82d230 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66ce322740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f66c41afad0 con 0x56351a822240 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c8f0b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a824ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c8f0b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a824ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35566/0 (socket says 172.21.15.31:35566) 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c8f0b640 1 -- 172.21.15.31:0/1483055571 learned_addr learned my addr 172.21.15.31:0/1483055571 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c9f0d640 1 --2- 172.21.15.31:0/1483055571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a82d230 0x56351a82a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c970c640 1 --2- 172.21.15.31:0/1483055571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822240 0x56351a82c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c8f0b640 1 -- 172.21.15.31:0/1483055571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822240 msgr2=0x56351a82c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c8f0b640 1 --2- 172.21.15.31:0/1483055571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822240 0x56351a82c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c8f0b640 1 -- 172.21.15.31:0/1483055571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a82d230 msgr2=0x56351a82a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c8f0b640 1 --2- 172.21.15.31:0/1483055571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a82d230 0x56351a82a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c8f0b640 1 -- 172.21.15.31:0/1483055571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56351a822990 con 0x56351a822b60 2024-04-25T01:55:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c9f0d640 1 --2- 172.21.15.31:0/1483055571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a82d230 0x56351a82a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66c8f0b640 1 --2- 172.21.15.31:0/1483055571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a824ab0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f66b40bb6a0 tx=0x7f66b4003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9aaac3c8191152 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.134+0000 7f66aeffd640 1 -- 172.21.15.31:0/1483055571 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b4049040 con 0x56351a822b60 2024-04-25T01:55:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.135+0000 7f66aeffd640 1 -- 172.21.15.31:0/1483055571 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66b403b460 con 0x56351a822b60 2024-04-25T01:55:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.135+0000 7f66aeffd640 1 -- 172.21.15.31:0/1483055571 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b4049ad0 con 0x56351a822b60 2024-04-25T01:55:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.136+0000 7f66ce322740 1 -- 172.21.15.31:0/1483055571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 msgr2=0x56351a824ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.136+0000 7f66ce322740 1 --2- 172.21.15.31:0/1483055571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a824ab0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f66b40bb6a0 tx=0x7f66b4003e60 comp rx=0 tx=0).stop 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.136+0000 7f66ce322740 1 -- 172.21.15.31:0/1483055571 shutdown_connections 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.136+0000 7f66ce322740 1 --2- 172.21.15.31:0/1483055571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a82d230 0x56351a82a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.136+0000 7f66ce322740 1 --2- 172.21.15.31:0/1483055571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a822b60 0x56351a824ab0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.136+0000 7f66ce322740 1 --2- 172.21.15.31:0/1483055571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822240 0x56351a82c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.136+0000 7f66ce322740 1 -- 172.21.15.31:0/1483055571 >> 172.21.15.31:0/1483055571 conn(0x56351a81e560 msgr2=0x56351a82a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.136+0000 7f66ce322740 1 -- 172.21.15.31:0/1483055571 shutdown_connections 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.136+0000 7f66ce322740 1 -- 172.21.15.31:0/1483055571 wait complete. 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.136+0000 7f66ce322740 1 -- start start 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.136+0000 7f66ce322740 1 Processor -- start 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66ce322740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 0x56351a826e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66ce322740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a8273c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66ce322740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 0x56351a827900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66ce322740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66c41b0870 con 0x56351a82d230 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66ce322740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f66c41b01a0 con 0x56351a822240 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66ce322740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f66c41ad3e0 con 0x56351a822b60 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66c9f0d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 0x56351a827900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66c8f0b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a8273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66c8f0b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a8273c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:49912/0 (socket says 172.21.15.31:49912) 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66c8f0b640 1 -- 172.21.15.31:0/45777337 learned_addr learned my addr 172.21.15.31:0/45777337 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66ce322740 10 client.0 _ll_register_callbacks cb 0x7f66bc003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66c970c640 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 0x56351a826e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66c9f0d640 1 -- 172.21.15.31:0/45777337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 msgr2=0x56351a8273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66c9f0d640 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a8273c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66c9f0d640 1 -- 172.21.15.31:0/45777337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 msgr2=0x56351a826e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66c9f0d640 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 0x56351a826e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66c9f0d640 1 -- 172.21.15.31:0/45777337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56351a8a30e0 con 0x56351a82d230 2024-04-25T01:55:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.137+0000 7f66c8f0b640 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a8273c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66ce322740 1 -- 172.21.15.31:0/45777337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56351a78e540 con 0x56351a82d230 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66c9f0d640 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 0x56351a827900 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f66bc016040 tx=0x7f66bc005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac86f2e16d4e57d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66acff9640 10 client.15981 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66acff9640 1 -- 172.21.15.31:0/45777337 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc036030 con 0x56351a82d230 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66acff9640 1 -- 172.21.15.31:0/45777337 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f66bc009040 con 0x56351a82d230 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66acff9640 1 -- 172.21.15.31:0/45777337 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f66bc022050 con 0x56351a82d230 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66ce322740 10 client.15981 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66ce322740 1 -- 172.21.15.31:0/45777337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=27}) -- 0x56351a8a4210 con 0x56351a82d230 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66acff9640 1 -- 172.21.15.31:0/45777337 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f66bc026950 con 0x56351a82d230 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66ce322740 10 client.15981 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66ce322740 10 client.15981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66ce322740 20 client.15981 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66ce322740 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56351a8a48a0 0x56351a8a6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66ce322740 4 client.15981 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.138+0000 7f66ce322740 1 -- 172.21.15.31:0/45777337 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56351a70c8e0 con 0x56351a8a48a0 2024-04-25T01:55:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.139+0000 7f66c970c640 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56351a8a48a0 0x56351a8a6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.139+0000 7f66acff9640 10 client.15981 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.139+0000 7f66c970c640 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56351a8a48a0 0x56351a8a6cc0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41849806ca208f2e server_cookie=82e6515ab2372579 in_seq=0 out_seq=0 2024-04-25T01:55:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66acff9640 1 -- 172.21.15.31:0/45777337 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56351a70c8e0 con 0x56351a8a48a0 2024-04-25T01:55:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66acff9640 10 client.15981 handle_command_reply: tid=0 2024-04-25T01:55:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66ce322740 1 client.15981 shutdown 2024-04-25T01:55:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66ce322740 1 -- 172.21.15.31:0/45777337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 msgr2=0x56351a827900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66ce322740 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 0x56351a827900 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f66bc016040 tx=0x7f66bc005cf0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66ce322740 1 -- 172.21.15.31:0/45777337 shutdown_connections 2024-04-25T01:55:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66ce322740 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56351a8a48a0 0x56351a8a6cc0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66ce322740 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56351a82d230 0x56351a827900 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66ce322740 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56351a822b60 0x56351a8273c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66ce322740 1 --2- 172.21.15.31:0/45777337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a822240 0x56351a826e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66acff9640 0 client.15981 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66ce322740 1 -- 172.21.15.31:0/45777337 >> 172.21.15.31:0/45777337 conn(0x56351a81e560 msgr2=0x56351a82aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66ce322740 1 -- 172.21.15.31:0/45777337 shutdown_connections 2024-04-25T01:55:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.143+0000 7f66ce322740 1 -- 172.21.15.31:0/45777337 wait complete. 2024-04-25T01:55:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66ce322740 20 client.15981 trim_cache size 0 max 16384 2024-04-25T01:55:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:55:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:55:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:55:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:55:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 -- 172.21.15.31:0/4205203835 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66b00a2ec0 msgr2=0x7f66b00a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66b00a2ec0 0x7f66b00a5380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f66bc005fd0 tx=0x7f66bc009500 comp rx=0 tx=0).stop 2024-04-25T01:55:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 -- 172.21.15.31:0/4205203835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c41032f0 msgr2=0x7f66c4100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c41032f0 0x7f66c4100b60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f66b4005940 tx=0x7f66b40042f0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 -- 172.21.15.31:0/4205203835 shutdown_connections 2024-04-25T01:55:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66b00a2ec0 0x7f66b00a5380 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c410bd10 0x7f66c41010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c41032f0 0x7f66c4100b60 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 --2- 172.21.15.31:0/4205203835 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c41029d0 0x7f66c4100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 -- 172.21.15.31:0/4205203835 >> 172.21.15.31:0/4205203835 conn(0x7f66c40777c0 msgr2=0x7f66c40fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 -- 172.21.15.31:0/4205203835 shutdown_connections 2024-04-25T01:55:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:43.144+0000 7f66cb970640 1 -- 172.21.15.31:0/4205203835 wait complete. 2024-04-25T01:55:43.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:43 smithi134 ceph-mon[33098]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:43.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:43 smithi031 ceph-mon[29204]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:44.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:43 smithi040 ceph-mon[32418]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T01:55:45.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:45 smithi134 ceph-mon[33098]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:55:45.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:45 smithi031 ceph-mon[29204]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:55:46.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:45 smithi040 ceph-mon[32418]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:55:47.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:47 smithi134 ceph-mon[33098]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T01:55:47.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:47 smithi031 ceph-mon[29204]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T01:55:48.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:47 smithi040 ceph-mon[32418]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T01:55:49.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:49 smithi134 ceph-mon[33098]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:55:49.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:49 smithi031 ceph-mon[29204]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:55:50.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:49 smithi040 ceph-mon[32418]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:55:51.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:51 smithi134 ceph-mon[33098]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T01:55:51.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:51 smithi031 ceph-mon[29204]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T01:55:52.009 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 96% (12004/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-25T01:55:52.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:51 smithi040 ceph-mon[32418]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T01:55:52.203 INFO:tasks.workunit.client.0.smithi031.stderr:Note: switching to 'd2c45d977142e024a59b68a8f1dce8f0b9840551'. 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr: git switch -c 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr:Or undo this operation with: 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr: git switch - 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T01:55:52.204 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T01:55:52.205 INFO:tasks.workunit.client.0.smithi031.stderr:HEAD is now at d2c45d97714 Merge pull request #56528 from adk3798/squid-mypy-bump 2024-04-25T01:55:52.210 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T01:55:52.275 INFO:tasks.workunit.client.0.smithi031.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T01:55:52.277 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T01:55:52.278 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T01:55:52.926 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T01:55:52.984 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T01:55:53.025 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T01:55:53.026 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T01:55:53.026 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T01:55:53.066 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T01:55:53.070 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T01:55:53.070 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T01:55:53.126 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-25T01:55:53.127 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-25T01:55:53.128 DEBUG:teuthology.orchestra.run.smithi031:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2c45d977142e024a59b68a8f1dce8f0b9840551 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-25T01:55:53.191 INFO:tasks.workunit.client.0.smithi031.stderr:+ dbench 1 2024-04-25T01:55:53.197 INFO:tasks.workunit.client.0.smithi031.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-25T01:55:53.197 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T01:55:53.197 INFO:tasks.workunit.client.0.smithi031.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-25T01:55:53.198 INFO:tasks.workunit.client.0.smithi031.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-25T01:55:53.298 INFO:tasks.workunit.client.0.smithi031.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-25T01:55:53.298 INFO:tasks.workunit.client.0.smithi031.stdout:releasing clients 2024-04-25T01:55:53.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:53 smithi134 ceph-mon[33098]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:55:53.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:53 smithi031 ceph-mon[29204]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:55:54.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:53 smithi040 ceph-mon[32418]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:55:54.298 INFO:tasks.workunit.client.0.smithi031.stdout: 1 902 21.68 MB/sec warmup 1 sec latency 11.387 ms 2024-04-25T01:55:55.298 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1526 13.05 MB/sec warmup 2 sec latency 17.495 ms 2024-04-25T01:55:55.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:55 smithi031 ceph-mon[29204]: pgmap v46: 97 pgs: 97 active+clean; 12 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 919 KiB/s wr, 15 op/s 2024-04-25T01:55:56.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:55 smithi040 ceph-mon[32418]: pgmap v46: 97 pgs: 97 active+clean; 12 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 919 KiB/s wr, 15 op/s 2024-04-25T01:55:56.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:55 smithi134 ceph-mon[33098]: pgmap v46: 97 pgs: 97 active+clean; 12 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 919 KiB/s wr, 15 op/s 2024-04-25T01:55:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.240+0000 7f1415970640 1 -- 172.21.15.31:0/4204959060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101029d0 msgr2=0x7f1410102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.240+0000 7f1415970640 1 --2- 172.21.15.31:0/4204959060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101029d0 0x7f1410102db0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1400009900 tx=0x7f140002a510 comp rx=0 tx=0).stop 2024-04-25T01:55:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.240+0000 7f1415970640 1 -- 172.21.15.31:0/4204959060 shutdown_connections 2024-04-25T01:55:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.240+0000 7f1415970640 1 --2- 172.21.15.31:0/4204959060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141010bd10 0x7f141010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.240+0000 7f1415970640 1 --2- 172.21.15.31:0/4204959060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14101032f0 0x7f141010b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.240+0000 7f1415970640 1 --2- 172.21.15.31:0/4204959060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101029d0 0x7f1410102db0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.240+0000 7f1415970640 1 -- 172.21.15.31:0/4204959060 >> 172.21.15.31:0/4204959060 conn(0x7f14100777c0 msgr2=0x7f1410077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.240+0000 7f1415970640 1 -- 172.21.15.31:0/4204959060 shutdown_connections 2024-04-25T01:55:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.240+0000 7f1415970640 1 -- 172.21.15.31:0/4204959060 wait complete. 2024-04-25T01:55:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.241+0000 7f1415970640 1 Processor -- start 2024-04-25T01:55:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.241+0000 7f1415970640 1 -- start start 2024-04-25T01:55:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.241+0000 7f1415970640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14101029d0 0x7f141019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.241+0000 7f1415970640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101032f0 0x7f141019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.241+0000 7f1415970640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141010bd10 0x7f141019e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.241+0000 7f1415970640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14101119c0 con 0x7f14101032f0 2024-04-25T01:55:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.241+0000 7f1415970640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1410111840 con 0x7f141010bd10 2024-04-25T01:55:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.241+0000 7f1415970640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1410111b40 con 0x7f14101029d0 2024-04-25T01:55:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.241+0000 7f14067fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101032f0 0x7f141019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.241+0000 7f140f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141010bd10 0x7f141019e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f14067fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101032f0 0x7f141019a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48890/0 (socket says 172.21.15.31:48890) 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f14067fc640 1 -- 172.21.15.31:0/174822545 learned_addr learned my addr 172.21.15.31:0/174822545 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f140effd640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14101029d0 0x7f141019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f14067fc640 1 -- 172.21.15.31:0/174822545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14101029d0 msgr2=0x7f141019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f14067fc640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14101029d0 0x7f141019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f14067fc640 1 -- 172.21.15.31:0/174822545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141010bd10 msgr2=0x7f141019e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f14067fc640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141010bd10 0x7f141019e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f14067fc640 1 -- 172.21.15.31:0/174822545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141019f0f0 con 0x7f14101032f0 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f140f7fe640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141010bd10 0x7f141019e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f14067fc640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101032f0 0x7f141019a640 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f13f800d950 tx=0x7f13f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f141496e640 1 -- 172.21.15.31:0/174822545 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f8017030 con 0x7f14101032f0 2024-04-25T01:55:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f1415970640 1 -- 172.21.15.31:0/174822545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141019f3e0 con 0x7f14101032f0 2024-04-25T01:55:56.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f141496e640 1 -- 172.21.15.31:0/174822545 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13f8004020 con 0x7f14101032f0 2024-04-25T01:55:56.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f141496e640 1 -- 172.21.15.31:0/174822545 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f80047a0 con 0x7f14101032f0 2024-04-25T01:55:56.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.242+0000 7f1415970640 1 -- 172.21.15.31:0/174822545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14101a6c70 con 0x7f14101032f0 2024-04-25T01:55:56.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f141496e640 1 -- 172.21.15.31:0/174822545 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 25) ==== 138066+0+0 (secure 0 0 0) 0x7f13f80041d0 con 0x7f14101032f0 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f1418684740 1 Processor -- start 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f1418684740 1 -- start start 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f1418684740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1f240 0x56447bd1f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f141496e640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f13fc0a2ec0 0x7f13fc0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f1418684740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 0x56447bd29cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f1418684740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd2a230 0x56447bd2c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f1418684740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56447bc83cb0 con 0x56447bd1fb60 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f1418684740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56447bc2c5b0 con 0x56447bd1f240 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f1418684740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56447bc7fc70 con 0x56447bd2a230 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f14067fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 0x56447bd29cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f140effd640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f13fc0a2ec0 0x7f13fc0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f140f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1f240 0x56447bd1f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f141496e640 1 -- 172.21.15.31:0/174822545 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f13f80cf0b0 con 0x7f14101032f0 2024-04-25T01:55:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f140f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1f240 0x56447bd1f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:43466/0 (socket says 172.21.15.31:43466) 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.245+0000 7f140f7fe640 1 -- 172.21.15.31:0/302874188 learned_addr learned my addr 172.21.15.31:0/302874188 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.246+0000 7f140f7fe640 1 --2- 172.21.15.31:0/302874188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd2a230 0x56447bd2c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.246+0000 7f140f7fe640 1 -- 172.21.15.31:0/302874188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd2a230 msgr2=0x56447bd2c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.246+0000 7f140f7fe640 1 --2- 172.21.15.31:0/302874188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd2a230 0x56447bd2c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.246+0000 7f140f7fe640 1 -- 172.21.15.31:0/302874188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 msgr2=0x56447bd29cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.246+0000 7f140f7fe640 1 --2- 172.21.15.31:0/302874188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 0x56447bd29cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.246+0000 7f140f7fe640 1 -- 172.21.15.31:0/302874188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56447b998210 con 0x56447bd1f240 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.246+0000 7f140effd640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f13fc0a2ec0 0x7f13fc0a5380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1400031440 tx=0x7f14000095f0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.246+0000 7f140f7fe640 1 --2- 172.21.15.31:0/302874188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1f240 0x56447bd1f620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f14080099e0 tx=0x7f1408005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b71c69ecf86dae86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.246+0000 7f1405ffb640 1 -- 172.21.15.31:0/302874188 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408011020 con 0x56447bd1f240 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.246+0000 7f1405ffb640 1 -- 172.21.15.31:0/302874188 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14080111c0 con 0x56447bd1f240 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.246+0000 7f1405ffb640 1 -- 172.21.15.31:0/302874188 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408010c00 con 0x56447bd1f240 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.247+0000 7f1418684740 1 -- 172.21.15.31:0/302874188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1f240 msgr2=0x56447bd1f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.247+0000 7f1418684740 1 --2- 172.21.15.31:0/302874188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1f240 0x56447bd1f620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f14080099e0 tx=0x7f1408005bd0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.247+0000 7f1418684740 1 -- 172.21.15.31:0/302874188 shutdown_connections 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.247+0000 7f1418684740 1 --2- 172.21.15.31:0/302874188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd2a230 0x56447bd2c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.247+0000 7f1418684740 1 --2- 172.21.15.31:0/302874188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 0x56447bd29cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.247+0000 7f1418684740 1 --2- 172.21.15.31:0/302874188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1f240 0x56447bd1f620 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.247+0000 7f1418684740 1 -- 172.21.15.31:0/302874188 >> 172.21.15.31:0/302874188 conn(0x56447bd1b560 msgr2=0x56447bd1b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.247+0000 7f1418684740 1 -- 172.21.15.31:0/302874188 shutdown_connections 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.247+0000 7f1418684740 1 -- 172.21.15.31:0/302874188 wait complete. 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.247+0000 7f1418684740 1 -- start start 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f1418684740 1 Processor -- start 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f1418684740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd271a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f1418684740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 0x56447bd276e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f1418684740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd2a230 0x56447bd27c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f1418684740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56447bd2d020 con 0x56447bd1fb60 2024-04-25T01:55:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f1418684740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56447bc3fb50 con 0x56447bd2a230 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f1418684740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56447bd2d1a0 con 0x56447bd1f240 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f1418684740 10 client.0 _ll_register_callbacks cb 0x56447bc7aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f14067fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 0x56447bd276e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f14067fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 0x56447bd276e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48912/0 (socket says 172.21.15.31:48912) 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f14067fc640 1 -- 172.21.15.31:0/1908446178 learned_addr learned my addr 172.21.15.31:0/1908446178 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f140effd640 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd271a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f140f7fe640 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd2a230 0x56447bd27c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f14067fc640 1 -- 172.21.15.31:0/1908446178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 msgr2=0x56447bd271a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f14067fc640 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd271a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f14067fc640 1 -- 172.21.15.31:0/1908446178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd2a230 msgr2=0x56447bd27c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f14067fc640 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd2a230 0x56447bd27c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.248+0000 7f14067fc640 1 -- 172.21.15.31:0/1908446178 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56447ba64620 con 0x56447bd1fb60 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f14067fc640 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 0x56447bd276e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009610 tx=0x7f13f8021c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870b75b8668fc6e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f13eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f13eeffd640 1 -- 172.21.15.31:0/1908446178 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f809add0 con 0x56447bd1fb60 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f13eeffd640 1 -- 172.21.15.31:0/1908446178 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f13f8021ca0 con 0x56447bd1fb60 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f1418684740 1 -- 172.21.15.31:0/1908446178 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14080111c0 con 0x56447bd1fb60 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f13eeffd640 1 -- 172.21.15.31:0/1908446178 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f13f80a09b0 con 0x56447bd1fb60 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f1418684740 10 client.15993 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f1418684740 1 -- 172.21.15.31:0/1908446178 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56447bd39250 con 0x56447bd1fb60 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f13eeffd640 1 -- 172.21.15.31:0/1908446178 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14080111c0 con 0x56447bd1fb60 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f13eeffd640 1 -- 172.21.15.31:0/1908446178 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f13f80b4240 con 0x56447bd1fb60 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f1418684740 10 client.15993 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f1418684740 10 client.15993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f1418684740 20 client.15993 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f1418684740 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56447bd3a760 0x56447bd3cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f1418684740 4 client.15993 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.249+0000 7f1418684740 1 -- 172.21.15.31:0/1908446178 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56447bc911f0 con 0x56447bd3a760 2024-04-25T01:55:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.250+0000 7f140effd640 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56447bd3a760 0x56447bd3cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.250+0000 7f13eeffd640 10 client.15993 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.250+0000 7f140effd640 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56447bd3a760 0x56447bd3cb50 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79ed9a6ed4b73666 server_cookie=8100b3bca62fb9db in_seq=0 out_seq=0 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f13eeffd640 1 -- 172.21.15.31:0/1908446178 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56447bc911f0 con 0x56447bd3a760 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f13eeffd640 10 client.15993 handle_command_reply: tid=0 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 1 client.15993 shutdown 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 1 -- 172.21.15.31:0/1908446178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 msgr2=0x56447bd276e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 0x56447bd276e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009610 tx=0x7f13f8021c30 comp rx=0 tx=0).stop 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 1 -- 172.21.15.31:0/1908446178 shutdown_connections 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56447bd3a760 0x56447bd3cb50 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd2a230 0x56447bd27c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd1fb60 0x56447bd276e0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f13eeffd640 0 client.15993 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 1 --2- 172.21.15.31:0/1908446178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd271a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 1 -- 172.21.15.31:0/1908446178 >> 172.21.15.31:0/1908446178 conn(0x56447bd1b560 msgr2=0x56447bd21260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 1 -- 172.21.15.31:0/1908446178 shutdown_connections 2024-04-25T01:55:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 1 -- 172.21.15.31:0/1908446178 wait complete. 2024-04-25T01:55:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.251+0000 7f1418684740 20 client.15993 trim_cache size 0 max 16384 2024-04-25T01:55:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f1418684740 1 Processor -- start 2024-04-25T01:55:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f1418684740 1 -- start start 2024-04-25T01:55:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f1418684740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd35ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f1418684740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1fb60 0x56447bd219a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f1418684740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd1a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f1418684740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14101af460 con 0x56447bd2a230 2024-04-25T01:55:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f1418684740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f14101b0fa0 con 0x56447bd1fb60 2024-04-25T01:55:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f1418684740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f14101afb30 con 0x56447bd1f240 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f140f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd1a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f140f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd1a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48928/0 (socket says 172.21.15.31:48928) 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f140f7fe640 1 -- 172.21.15.31:0/2046023139 learned_addr learned my addr 172.21.15.31:0/2046023139 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f14067fc640 1 --2- 172.21.15.31:0/2046023139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1fb60 0x56447bd219a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f140effd640 1 --2- 172.21.15.31:0/2046023139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd35ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f140f7fe640 1 -- 172.21.15.31:0/2046023139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 msgr2=0x56447bd35ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f140f7fe640 1 --2- 172.21.15.31:0/2046023139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd35ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f140f7fe640 1 -- 172.21.15.31:0/2046023139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1fb60 msgr2=0x56447bd219a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f140f7fe640 1 --2- 172.21.15.31:0/2046023139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1fb60 0x56447bd219a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f140f7fe640 1 -- 172.21.15.31:0/2046023139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56447bd1f990 con 0x56447bd2a230 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.269+0000 7f140effd640 1 --2- 172.21.15.31:0/2046023139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd35ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.270+0000 7f140f7fe640 1 --2- 172.21.15.31:0/2046023139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd1a440 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f14080099e0 tx=0x7f14080096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b532d1276347ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.270+0000 7f14057fa640 1 -- 172.21.15.31:0/2046023139 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140800fdc0 con 0x56447bd2a230 2024-04-25T01:55:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.270+0000 7f14057fa640 1 -- 172.21.15.31:0/2046023139 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1408004a00 con 0x56447bd2a230 2024-04-25T01:55:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.270+0000 7f14057fa640 1 -- 172.21.15.31:0/2046023139 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408005610 con 0x56447bd2a230 2024-04-25T01:55:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.271+0000 7f1418684740 1 -- 172.21.15.31:0/2046023139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 msgr2=0x56447bd1a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.271+0000 7f1418684740 1 --2- 172.21.15.31:0/2046023139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd1a440 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f14080099e0 tx=0x7f14080096c0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.271+0000 7f1418684740 1 -- 172.21.15.31:0/2046023139 shutdown_connections 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.271+0000 7f1418684740 1 --2- 172.21.15.31:0/2046023139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd1a440 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.271+0000 7f1418684740 1 --2- 172.21.15.31:0/2046023139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1fb60 0x56447bd219a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.271+0000 7f1418684740 1 --2- 172.21.15.31:0/2046023139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd35ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.271+0000 7f1418684740 1 -- 172.21.15.31:0/2046023139 >> 172.21.15.31:0/2046023139 conn(0x56447bd1b560 msgr2=0x56447bd19d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.271+0000 7f1418684740 1 -- 172.21.15.31:0/2046023139 shutdown_connections 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.271+0000 7f1418684740 1 -- 172.21.15.31:0/2046023139 wait complete. 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.271+0000 7f1418684740 1 -- start start 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.271+0000 7f1418684740 1 Processor -- start 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f1418684740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd23df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f1418684740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1fb60 0x56447bd24330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f1418684740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd24870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f1418684740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14101b08d0 con 0x56447bd2a230 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f1418684740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f14101b0200 con 0x56447bd1fb60 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f1418684740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f14101ad440 con 0x56447bd1f240 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f1418684740 10 client.0 _ll_register_callbacks cb 0x7f13f80050c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f140f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd24870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f140f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd24870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48944/0 (socket says 172.21.15.31:48944) 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f140f7fe640 1 -- 172.21.15.31:0/3498157068 learned_addr learned my addr 172.21.15.31:0/3498157068 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f14067fc640 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1fb60 0x56447bd24330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f140effd640 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd23df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f140f7fe640 1 -- 172.21.15.31:0/3498157068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 msgr2=0x56447bd23df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f140f7fe640 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd23df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f140f7fe640 1 -- 172.21.15.31:0/3498157068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1fb60 msgr2=0x56447bd24330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f140f7fe640 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1fb60 0x56447bd24330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f140f7fe640 1 -- 172.21.15.31:0/3498157068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56447bd9fcb0 con 0x56447bd2a230 2024-04-25T01:55:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f14067fc640 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1fb60 0x56447bd24330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f140f7fe640 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd24870 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1408009560 tx=0x7f140800fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97141272f172964 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f13eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.272+0000 7f13eeffd640 1 -- 172.21.15.31:0/3498157068 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408030070 con 0x56447bd2a230 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f13eeffd640 1 -- 172.21.15.31:0/3498157068 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f1408004bb0 con 0x56447bd2a230 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f1418684740 1 -- 172.21.15.31:0/3498157068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1408004a00 con 0x56447bd2a230 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f13eeffd640 1 -- 172.21.15.31:0/3498157068 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f140803c050 con 0x56447bd2a230 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f1418684740 10 client.15999 fetch_fsmap learned FSMap version 27 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f1418684740 1 -- 172.21.15.31:0/3498157068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56447bda0f20 con 0x56447bd2a230 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f13eeffd640 1 -- 172.21.15.31:0/3498157068 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1408004a00 con 0x56447bd2a230 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f13eeffd640 1 -- 172.21.15.31:0/3498157068 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f140801e560 con 0x56447bd2a230 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f1418684740 10 client.15999 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f1418684740 10 client.15999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f1418684740 20 client.15999 populate_metadata read hostname 'smithi031' 2024-04-25T01:55:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f1418684740 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56447bd3a760 0x56447bda3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f1418684740 4 client.15999 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:55:56.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f1418684740 1 -- 172.21.15.31:0/3498157068 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56447bc8b540 con 0x56447bd3a760 2024-04-25T01:55:56.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.273+0000 7f140effd640 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56447bd3a760 0x56447bda3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.274+0000 7f13eeffd640 10 client.15999 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:56.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.274+0000 7f140effd640 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56447bd3a760 0x56447bda3400 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af7a8a776e49ee3b server_cookie=f21701bdc08e8562 in_seq=0 out_seq=0 2024-04-25T01:55:56.298 INFO:tasks.workunit.client.0.smithi031.stdout: 1 2221 8.92 MB/sec warmup 3 sec latency 19.187 ms 2024-04-25T01:55:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.313+0000 7f13eeffd640 1 -- 172.21.15.31:0/3498157068 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56447bc8b540 con 0x56447bd3a760 2024-04-25T01:55:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f13eeffd640 10 client.15999 handle_command_reply: tid=0 2024-04-25T01:55:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 1 client.15999 shutdown 2024-04-25T01:55:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 1 -- 172.21.15.31:0/3498157068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 msgr2=0x56447bd24870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd24870 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1408009560 tx=0x7f140800fa40 comp rx=0 tx=0).stop 2024-04-25T01:55:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 1 -- 172.21.15.31:0/3498157068 shutdown_connections 2024-04-25T01:55:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56447bd3a760 0x56447bda3400 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f13eeffd640 0 client.15999 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:55:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56447bd2a230 0x56447bd24870 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56447bd1fb60 0x56447bd24330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 1 --2- 172.21.15.31:0/3498157068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56447bd1f240 0x56447bd23df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 1 -- 172.21.15.31:0/3498157068 >> 172.21.15.31:0/3498157068 conn(0x56447bd1b560 msgr2=0x56447bd1a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 1 -- 172.21.15.31:0/3498157068 shutdown_connections 2024-04-25T01:55:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 1 -- 172.21.15.31:0/3498157068 wait complete. 2024-04-25T01:55:56.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.314+0000 7f1418684740 20 client.15999 trim_cache size 0 max 16384 2024-04-25T01:55:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:55:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:55:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:55:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:55:56.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 -- 172.21.15.31:0/174822545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f13fc0a2ec0 msgr2=0x7f13fc0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f13fc0a2ec0 0x7f13fc0a5380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1400031440 tx=0x7f14000095f0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 -- 172.21.15.31:0/174822545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101032f0 msgr2=0x7f141019a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101032f0 0x7f141019a640 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f13f800d950 tx=0x7f13f800de10 comp rx=0 tx=0).stop 2024-04-25T01:55:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 -- 172.21.15.31:0/174822545 shutdown_connections 2024-04-25T01:55:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f13fc0a2ec0 0x7f13fc0a5380 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f141010bd10 0x7f141019e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14101032f0 0x7f141019a640 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 --2- 172.21.15.31:0/174822545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14101029d0 0x7f141019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 -- 172.21.15.31:0/174822545 >> 172.21.15.31:0/174822545 conn(0x7f14100777c0 msgr2=0x7f1410107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 -- 172.21.15.31:0/174822545 shutdown_connections 2024-04-25T01:55:56.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:55:56.315+0000 7f1415970640 1 -- 172.21.15.31:0/174822545 wait complete. 2024-04-25T01:55:57.298 INFO:tasks.workunit.client.0.smithi031.stdout: 1 2973 8.38 MB/sec warmup 4 sec latency 37.166 ms 2024-04-25T01:55:57.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:57 smithi031 ceph-mon[29204]: pgmap v47: 97 pgs: 97 active+clean; 22 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.8 MiB/s wr, 27 op/s 2024-04-25T01:55:58.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:57 smithi040 ceph-mon[32418]: pgmap v47: 97 pgs: 97 active+clean; 22 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.8 MiB/s wr, 27 op/s 2024-04-25T01:55:58.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:57 smithi134 ceph-mon[33098]: pgmap v47: 97 pgs: 97 active+clean; 22 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.8 MiB/s wr, 27 op/s 2024-04-25T01:55:58.298 INFO:tasks.workunit.client.0.smithi031.stdout: 1 3573 7.69 MB/sec warmup 5 sec latency 16.323 ms 2024-04-25T01:55:59.298 INFO:tasks.workunit.client.0.smithi031.stdout: 1 4028 6.68 MB/sec warmup 6 sec latency 16.236 ms 2024-04-25T01:55:59.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:55:59 smithi031 ceph-mon[29204]: pgmap v48: 97 pgs: 97 active+clean; 22 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.8 MiB/s wr, 24 op/s 2024-04-25T01:56:00.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:55:59 smithi040 ceph-mon[32418]: pgmap v48: 97 pgs: 97 active+clean; 22 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.8 MiB/s wr, 24 op/s 2024-04-25T01:56:00.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:55:59 smithi134 ceph-mon[33098]: pgmap v48: 97 pgs: 97 active+clean; 22 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.8 MiB/s wr, 24 op/s 2024-04-25T01:56:00.298 INFO:tasks.workunit.client.0.smithi031.stdout: 1 4563 5.86 MB/sec warmup 7 sec latency 19.873 ms 2024-04-25T01:56:01.298 INFO:tasks.workunit.client.0.smithi031.stdout: 1 5216 5.67 MB/sec warmup 8 sec latency 16.869 ms 2024-04-25T01:56:01.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:01 smithi031 ceph-mon[29204]: pgmap v49: 97 pgs: 97 active+clean; 26 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 2.4 MiB/s wr, 77 op/s 2024-04-25T01:56:01.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:01 smithi031 ceph-mon[29204]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T01:56:02.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:01 smithi040 ceph-mon[32418]: pgmap v49: 97 pgs: 97 active+clean; 26 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 2.4 MiB/s wr, 77 op/s 2024-04-25T01:56:02.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:01 smithi040 ceph-mon[32418]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T01:56:02.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:01 smithi134 ceph-mon[33098]: pgmap v49: 97 pgs: 97 active+clean; 26 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 2.4 MiB/s wr, 77 op/s 2024-04-25T01:56:02.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:01 smithi134 ceph-mon[33098]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T01:56:02.298 INFO:tasks.workunit.client.0.smithi031.stdout: 1 5900 5.22 MB/sec warmup 9 sec latency 18.813 ms 2024-04-25T01:56:03.298 INFO:tasks.workunit.client.0.smithi031.stdout: 1 6743 5.34 MB/sec warmup 10 sec latency 19.481 ms 2024-04-25T01:56:03.988 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:03 smithi031 ceph-mon[29204]: pgmap v50: 97 pgs: 97 active+clean; 26 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 2.4 MiB/s wr, 73 op/s 2024-04-25T01:56:04.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:03 smithi040 ceph-mon[32418]: pgmap v50: 97 pgs: 97 active+clean; 26 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 2.4 MiB/s wr, 73 op/s 2024-04-25T01:56:04.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:03 smithi134 ceph-mon[33098]: pgmap v50: 97 pgs: 97 active+clean; 26 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 2.4 MiB/s wr, 73 op/s 2024-04-25T01:56:04.298 INFO:tasks.workunit.client.0.smithi031.stdout: 1 7217 5.26 MB/sec warmup 11 sec latency 59.191 ms 2024-04-25T01:56:05.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 7615 4.93 MB/sec warmup 12 sec latency 17.039 ms 2024-04-25T01:56:05.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:05 smithi031 ceph-mon[29204]: pgmap v51: 97 pgs: 97 active+clean; 26 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 2.7 MiB/s wr, 104 op/s 2024-04-25T01:56:06.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:05 smithi040 ceph-mon[32418]: pgmap v51: 97 pgs: 97 active+clean; 26 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 2.7 MiB/s wr, 104 op/s 2024-04-25T01:56:06.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:05 smithi134 ceph-mon[33098]: pgmap v51: 97 pgs: 97 active+clean; 26 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 2.7 MiB/s wr, 104 op/s 2024-04-25T01:56:06.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 8147 4.70 MB/sec warmup 13 sec latency 18.367 ms 2024-04-25T01:56:07.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 8487 4.59 MB/sec warmup 14 sec latency 537.329 ms 2024-04-25T01:56:07.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:07 smithi031 ceph-mon[29204]: pgmap v52: 97 pgs: 97 active+clean; 28 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-04-25T01:56:07.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:07 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:56:07.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:07 smithi134 ceph-mon[33098]: pgmap v52: 97 pgs: 97 active+clean; 28 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-04-25T01:56:07.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:07 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:56:08.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:07 smithi040 ceph-mon[32418]: pgmap v52: 97 pgs: 97 active+clean; 28 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-04-25T01:56:08.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:07 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:56:08.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 8487 4.28 MB/sec warmup 15 sec latency 1537.423 ms 2024-04-25T01:56:09.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 8487 4.01 MB/sec warmup 16 sec latency 2537.532 ms 2024-04-25T01:56:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.414+0000 7fafcbba2640 1 -- 172.21.15.31:0/270782775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc4103310 msgr2=0x7fafc410b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.414+0000 7fafcbba2640 1 --2- 172.21.15.31:0/270782775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc4103310 0x7fafc410b7f0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009900 tx=0x7fafb402a480 comp rx=0 tx=0).stop 2024-04-25T01:56:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.414+0000 7fafcbba2640 1 -- 172.21.15.31:0/270782775 shutdown_connections 2024-04-25T01:56:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.414+0000 7fafcbba2640 1 --2- 172.21.15.31:0/270782775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc410bdc0 0x7fafc410e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.414+0000 7fafcbba2640 1 --2- 172.21.15.31:0/270782775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc4103310 0x7fafc410b7f0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.414+0000 7fafcbba2640 1 --2- 172.21.15.31:0/270782775 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc41029f0 0x7fafc4102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.414+0000 7fafcbba2640 1 -- 172.21.15.31:0/270782775 >> 172.21.15.31:0/270782775 conn(0x7fafc40769c0 msgr2=0x7fafc4076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.414+0000 7fafcbba2640 1 -- 172.21.15.31:0/270782775 shutdown_connections 2024-04-25T01:56:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.414+0000 7fafcbba2640 1 -- 172.21.15.31:0/270782775 wait complete. 2024-04-25T01:56:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.415+0000 7fafcbba2640 1 Processor -- start 2024-04-25T01:56:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.415+0000 7fafcbba2640 1 -- start start 2024-04-25T01:56:09.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.415+0000 7fafcbba2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc41029f0 0x7fafc4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.415+0000 7fafcbba2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc4103310 0x7fafc4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.415+0000 7fafcbba2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc410bdc0 0x7fafc419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.415+0000 7fafcbba2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fafc4111a00 con 0x7fafc410bdc0 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.415+0000 7fafcbba2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fafc4111880 con 0x7fafc41029f0 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.415+0000 7fafcbba2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fafc4111b80 con 0x7fafc4103310 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.415+0000 7fafca13f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc410bdc0 0x7fafc419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.415+0000 7fafca13f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc410bdc0 0x7fafc419c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46534/0 (socket says 172.21.15.31:46534) 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.415+0000 7fafca13f640 1 -- 172.21.15.31:0/3038921840 learned_addr learned my addr 172.21.15.31:0/3038921840 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafc993e640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc41029f0 0x7fafc4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafc913d640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc4103310 0x7fafc4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafca13f640 1 -- 172.21.15.31:0/3038921840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc4103310 msgr2=0x7fafc4198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafca13f640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc4103310 0x7fafc4198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafca13f640 1 -- 172.21.15.31:0/3038921840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc41029f0 msgr2=0x7fafc4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafca13f640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc41029f0 0x7fafc4197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafca13f640 1 -- 172.21.15.31:0/3038921840 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc419ced0 con 0x7fafc410bdc0 2024-04-25T01:56:09.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafc993e640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc41029f0 0x7fafc4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:09.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafca13f640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc410bdc0 0x7fafc419c7a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00eb40 tx=0x7fafbc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:09.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafbaffd640 1 -- 172.21.15.31:0/3038921840 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc004b20 con 0x7fafc410bdc0 2024-04-25T01:56:09.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafcbba2640 1 -- 172.21.15.31:0/3038921840 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc419d1c0 con 0x7fafc410bdc0 2024-04-25T01:56:09.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafcbba2640 1 -- 172.21.15.31:0/3038921840 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc41a4a50 con 0x7fafc410bdc0 2024-04-25T01:56:09.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafbaffd640 1 -- 172.21.15.31:0/3038921840 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafbc004cc0 con 0x7fafc410bdc0 2024-04-25T01:56:09.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.416+0000 7fafbaffd640 1 -- 172.21.15.31:0/3038921840 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc010690 con 0x7fafc410bdc0 2024-04-25T01:56:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.418+0000 7fafbaffd640 1 -- 172.21.15.31:0/3038921840 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fafbc005020 con 0x7fafc410bdc0 2024-04-25T01:56:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.419+0000 7fafce992740 1 Processor -- start 2024-04-25T01:56:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.419+0000 7fafce992740 1 -- start start 2024-04-25T01:56:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.419+0000 7fafce992740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f083b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.419+0000 7fafce992740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f083bb60 0x5568f0845cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.419+0000 7fafce992740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f0846230 0x5568f0848620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.419+0000 7fafce992740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5568f079fcb0 con 0x5568f0846230 2024-04-25T01:56:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafce992740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5568f07485b0 con 0x5568f083bb60 2024-04-25T01:56:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafce992740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5568f079bc70 con 0x5568f083b240 2024-04-25T01:56:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafbaffd640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fafb00a2e20 0x7fafb00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc993e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f083bb60 0x5568f0845cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc913d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f0846230 0x5568f0848620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc913d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f0846230 0x5568f0848620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46548/0 (socket says 172.21.15.31:46548) 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc993e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f083bb60 0x5568f0845cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36276/0 (socket says 172.21.15.31:36276) 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc913d640 1 -- 172.21.15.31:0/1962267358 learned_addr learned my addr 172.21.15.31:0/1962267358 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafca13f640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fafb00a2e20 0x7fafb00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc913d640 1 --2- 172.21.15.31:0/1962267358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f083b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafbaffd640 1 -- 172.21.15.31:0/3038921840 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fafbc0cfb80 con 0x7fafc410bdc0 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc993e640 1 -- 172.21.15.31:0/1962267358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 msgr2=0x5568f083b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc993e640 1 --2- 172.21.15.31:0/1962267358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f083b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc993e640 1 -- 172.21.15.31:0/1962267358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f0846230 msgr2=0x5568f0848620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc993e640 1 --2- 172.21.15.31:0/1962267358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f0846230 0x5568f0848620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc993e640 1 -- 172.21.15.31:0/1962267358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f04b4210 con 0x5568f083bb60 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc913d640 1 --2- 172.21.15.31:0/1962267358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f0846230 0x5568f0848620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafca13f640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fafb00a2e20 0x7fafb00a52e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fafbc09c020 tx=0x7fafbc086bc0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.420+0000 7fafc993e640 1 --2- 172.21.15.31:0/1962267358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f083bb60 0x5568f0845cf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fafc000bc80 tx=0x7fafc000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22c70969e75a5bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.421+0000 7fafaffff640 1 -- 172.21.15.31:0/1962267358 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0015070 con 0x5568f083bb60 2024-04-25T01:56:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.421+0000 7fafaffff640 1 -- 172.21.15.31:0/1962267358 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafc00118f0 con 0x5568f083bb60 2024-04-25T01:56:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.421+0000 7fafaffff640 1 -- 172.21.15.31:0/1962267358 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc00107d0 con 0x5568f083bb60 2024-04-25T01:56:09.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.421+0000 7fafce992740 1 -- 172.21.15.31:0/1962267358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f083bb60 msgr2=0x5568f0845cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:09.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.421+0000 7fafce992740 1 --2- 172.21.15.31:0/1962267358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f083bb60 0x5568f0845cf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fafc000bc80 tx=0x7fafc000f730 comp rx=0 tx=0).stop 2024-04-25T01:56:09.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.421+0000 7fafce992740 1 -- 172.21.15.31:0/1962267358 shutdown_connections 2024-04-25T01:56:09.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.421+0000 7fafce992740 1 --2- 172.21.15.31:0/1962267358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f0846230 0x5568f0848620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.421+0000 7fafce992740 1 --2- 172.21.15.31:0/1962267358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f083bb60 0x5568f0845cf0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.421+0000 7fafce992740 1 --2- 172.21.15.31:0/1962267358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f083b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.421+0000 7fafce992740 1 -- 172.21.15.31:0/1962267358 >> 172.21.15.31:0/1962267358 conn(0x5568f0837560 msgr2=0x5568f0837940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:09.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.422+0000 7fafce992740 1 -- 172.21.15.31:0/1962267358 shutdown_connections 2024-04-25T01:56:09.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.422+0000 7fafce992740 1 -- 172.21.15.31:0/1962267358 wait complete. 2024-04-25T01:56:09.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.422+0000 7fafce992740 1 -- start start 2024-04-25T01:56:09.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.422+0000 7fafce992740 1 Processor -- start 2024-04-25T01:56:09.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafce992740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f0843160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafce992740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08436a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafce992740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f0843be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafce992740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5568f0849020 con 0x5568f083bb60 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafce992740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5568f075bb50 con 0x5568f0846230 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafce992740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5568f08491a0 con 0x5568f083b240 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafce992740 10 client.0 _ll_register_callbacks cb 0x5568f0796ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc993e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc993e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08436a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46558/0 (socket says 172.21.15.31:46558) 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc993e640 1 -- 172.21.15.31:0/2115592134 learned_addr learned my addr 172.21.15.31:0/2115592134 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc913d640 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f0843160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc913d640 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f0843be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc993e640 1 -- 172.21.15.31:0/2115592134 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 msgr2=0x5568f0843160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc993e640 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f0843160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc993e640 1 -- 172.21.15.31:0/2115592134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 msgr2=0x5568f0843be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc993e640 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f0843be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc993e640 1 -- 172.21.15.31:0/2115592134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f0580620 con 0x5568f083bb60 2024-04-25T01:56:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc913d640 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f0843160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafc993e640 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08436a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fafc0010410 tx=0x7fafc000bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa073049f3eebab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafad7fa640 1 -- 172.21.15.31:0/2115592134 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0029060 con 0x5568f083bb60 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafad7fa640 1 -- 172.21.15.31:0/2115592134 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fafc0011aa0 con 0x5568f083bb60 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.423+0000 7fafce992740 1 -- 172.21.15.31:0/2115592134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc00118f0 con 0x5568f083bb60 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafad7fa640 1 -- 172.21.15.31:0/2115592134 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fafc0015050 con 0x5568f083bb60 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafce992740 10 client.16011 fetch_fsmap learned FSMap version 28 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafce992740 1 -- 172.21.15.31:0/2115592134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5568f0855280 con 0x5568f083bb60 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafad7fa640 1 -- 172.21.15.31:0/2115592134 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafc00118f0 con 0x5568f083bb60 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafad7fa640 1 -- 172.21.15.31:0/2115592134 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fafc0004840 con 0x5568f083bb60 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafce992740 10 client.16011 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafce992740 10 client.16011 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafce992740 20 client.16011 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:09.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafce992740 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5568f08566e0 0x5568f0858ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafce992740 4 client.16011 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafce992740 1 -- 172.21.15.31:0/2115592134 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568f07ad1f0 con 0x5568f08566e0 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.424+0000 7fafc913d640 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5568f08566e0 0x5568f0858ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.425+0000 7fafc913d640 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5568f08566e0 0x5568f0858ad0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3450011f8fb42a90 server_cookie=f45cfdd1b8c7a17f in_seq=0 out_seq=0 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.425+0000 7fafad7fa640 10 client.16011 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafad7fa640 1 -- 172.21.15.31:0/2115592134 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5568f07ad1f0 con 0x5568f08566e0 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafad7fa640 10 client.16011 handle_command_reply: tid=0 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 1 client.16011 shutdown 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 1 -- 172.21.15.31:0/2115592134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 msgr2=0x5568f08436a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08436a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fafc0010410 tx=0x7fafc000bf80 comp rx=0 tx=0).stop 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 1 -- 172.21.15.31:0/2115592134 shutdown_connections 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5568f08566e0 0x5568f0858ad0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f0843be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08436a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafad7fa640 0 client.16011 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:09.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 1 --2- 172.21.15.31:0/2115592134 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f0843160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 1 -- 172.21.15.31:0/2115592134 >> 172.21.15.31:0/2115592134 conn(0x5568f0837560 msgr2=0x5568f083d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 1 -- 172.21.15.31:0/2115592134 shutdown_connections 2024-04-25T01:56:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 1 -- 172.21.15.31:0/2115592134 wait complete. 2024-04-25T01:56:09.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.426+0000 7fafce992740 20 client.16011 trim_cache size 0 max 16384 2024-04-25T01:56:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.441+0000 7fafce992740 1 Processor -- start 2024-04-25T01:56:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.441+0000 7fafce992740 1 -- start start 2024-04-25T01:56:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.441+0000 7fafce992740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f0851f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafce992740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f084e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafce992740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f08422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafce992740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fafa800e650 con 0x5568f083bb60 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafce992740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fafa80101c0 con 0x5568f0846230 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafce992740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fafa800ed20 con 0x5568f083b240 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafc993e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f084e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafc993e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f084e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46566/0 (socket says 172.21.15.31:46566) 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafc993e640 1 -- 172.21.15.31:0/3292387211 learned_addr learned my addr 172.21.15.31:0/3292387211 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafc913d640 1 --2- 172.21.15.31:0/3292387211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f08422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafc993e640 1 -- 172.21.15.31:0/3292387211 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 msgr2=0x5568f0851f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafc993e640 1 --2- 172.21.15.31:0/3292387211 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f0851f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafc993e640 1 -- 172.21.15.31:0/3292387211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 msgr2=0x5568f08422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafc993e640 1 --2- 172.21.15.31:0/3292387211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f08422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafc993e640 1 -- 172.21.15.31:0/3292387211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f083c700 con 0x5568f083bb60 2024-04-25T01:56:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafc913d640 1 --2- 172.21.15.31:0/3292387211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f08422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafc993e640 1 --2- 172.21.15.31:0/3292387211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f084e860 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fafc0006010 tx=0x7fafc0004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f1786aa66c9fb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafaf7fe640 1 -- 172.21.15.31:0/3292387211 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0029070 con 0x5568f083bb60 2024-04-25T01:56:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafaf7fe640 1 -- 172.21.15.31:0/3292387211 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafc000fc30 con 0x5568f083bb60 2024-04-25T01:56:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.442+0000 7fafaf7fe640 1 -- 172.21.15.31:0/3292387211 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc00107a0 con 0x5568f083bb60 2024-04-25T01:56:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.443+0000 7fafce992740 1 -- 172.21.15.31:0/3292387211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 msgr2=0x5568f084e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.443+0000 7fafce992740 1 --2- 172.21.15.31:0/3292387211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f084e860 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fafc0006010 tx=0x7fafc0004f90 comp rx=0 tx=0).stop 2024-04-25T01:56:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.443+0000 7fafce992740 1 -- 172.21.15.31:0/3292387211 shutdown_connections 2024-04-25T01:56:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.443+0000 7fafce992740 1 --2- 172.21.15.31:0/3292387211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f08422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.443+0000 7fafce992740 1 --2- 172.21.15.31:0/3292387211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f084e860 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.443+0000 7fafce992740 1 --2- 172.21.15.31:0/3292387211 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f0851f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.443+0000 7fafce992740 1 -- 172.21.15.31:0/3292387211 >> 172.21.15.31:0/3292387211 conn(0x5568f0837560 msgr2=0x5568f0835d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.443+0000 7fafce992740 1 -- 172.21.15.31:0/3292387211 shutdown_connections 2024-04-25T01:56:09.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.443+0000 7fafce992740 1 -- 172.21.15.31:0/3292387211 wait complete. 2024-04-25T01:56:09.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.443+0000 7fafce992740 1 -- start start 2024-04-25T01:56:09.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.443+0000 7fafce992740 1 Processor -- start 2024-04-25T01:56:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafce992740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f08b9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafce992740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08b96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafce992740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f085d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafce992740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fafa800fac0 con 0x5568f083bb60 2024-04-25T01:56:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafce992740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fafa800f3f0 con 0x5568f0846230 2024-04-25T01:56:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafce992740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fafa800c630 con 0x5568f083b240 2024-04-25T01:56:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafce992740 10 client.0 _ll_register_callbacks cb 0x7fafbc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafc993e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08b96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafc993e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08b96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46572/0 (socket says 172.21.15.31:46572) 2024-04-25T01:56:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafc993e640 1 -- 172.21.15.31:0/1771308750 learned_addr learned my addr 172.21.15.31:0/1771308750 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafc913d640 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f08b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafc913d640 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f085d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafc993e640 1 -- 172.21.15.31:0/1771308750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 msgr2=0x5568f08b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafc993e640 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f08b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafc993e640 1 -- 172.21.15.31:0/1771308750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 msgr2=0x5568f085d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafc993e640 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f085d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafc993e640 1 -- 172.21.15.31:0/1771308750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f085df60 con 0x5568f083bb60 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafc993e640 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08b96d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fafc000b4c0 tx=0x7fafc000bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb63413d354bdb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.444+0000 7fafad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafad7fa640 1 -- 172.21.15.31:0/1771308750 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc003c060 con 0x5568f083bb60 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafad7fa640 1 -- 172.21.15.31:0/1771308750 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fafc0004950 con 0x5568f083bb60 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafce992740 1 -- 172.21.15.31:0/1771308750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc000fc30 con 0x5568f083bb60 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafad7fa640 1 -- 172.21.15.31:0/1771308750 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fafc0029070 con 0x5568f083bb60 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafce992740 10 client.16017 fetch_fsmap learned FSMap version 28 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafce992740 1 -- 172.21.15.31:0/1771308750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5568f08c09a0 con 0x5568f083bb60 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafad7fa640 1 -- 172.21.15.31:0/1771308750 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafc000fc30 con 0x5568f083bb60 2024-04-25T01:56:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafad7fa640 1 -- 172.21.15.31:0/1771308750 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fafc00246e0 con 0x5568f083bb60 2024-04-25T01:56:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafce992740 10 client.16017 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:56:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafce992740 10 client.16017 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafce992740 20 client.16017 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafce992740 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5568f08c1060 0x5568f08c3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafce992740 4 client.16017 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:56:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.445+0000 7fafce992740 1 -- 172.21.15.31:0/1771308750 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5568f07a7540 con 0x5568f08c1060 2024-04-25T01:56:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.446+0000 7fafc913d640 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5568f08c1060 0x5568f08c3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.446+0000 7fafc913d640 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5568f08c1060 0x5568f08c3480 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48011eb5e30b5311 server_cookie=db10daf5f2ee9e4e in_seq=0 out_seq=0 2024-04-25T01:56:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:09.446+0000 7fafad7fa640 10 client.16017 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:09.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:09 smithi031 ceph-mon[29204]: pgmap v53: 97 pgs: 97 active+clean; 28 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-04-25T01:56:10.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:09 smithi040 ceph-mon[32418]: pgmap v53: 97 pgs: 97 active+clean; 28 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-04-25T01:56:10.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:09 smithi134 ceph-mon[33098]: pgmap v53: 97 pgs: 97 active+clean; 28 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 1.4 MiB/s wr, 109 op/s 2024-04-25T01:56:10.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 8934 3.80 MB/sec warmup 17 sec latency 2743.061 ms 2024-04-25T01:56:11.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:56:11.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:56:11.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 9536 3.69 MB/sec warmup 18 sec latency 18.518 ms 2024-04-25T01:56:11.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.335+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/2378014090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc1029d0 msgr2=0x7fd0dc102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.335+0000 7fd0e1a3f640 1 --2- 172.21.15.31:0/2378014090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc1029d0 0x7fd0dc102db0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009900 tx=0x7fd0cc02a480 comp rx=0 tx=0).stop 2024-04-25T01:56:11.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.335+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/2378014090 shutdown_connections 2024-04-25T01:56:11.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.335+0000 7fd0e1a3f640 1 --2- 172.21.15.31:0/2378014090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0dc10bd10 0x7fd0dc10e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.335+0000 7fd0e1a3f640 1 --2- 172.21.15.31:0/2378014090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0dc1032f0 0x7fd0dc10b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.335+0000 7fd0e1a3f640 1 --2- 172.21.15.31:0/2378014090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc1029d0 0x7fd0dc102db0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.335+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/2378014090 >> 172.21.15.31:0/2378014090 conn(0x7fd0dc0777a0 msgr2=0x7fd0dc077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:11.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.335+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/2378014090 shutdown_connections 2024-04-25T01:56:11.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.335+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/2378014090 wait complete. 2024-04-25T01:56:11.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0e1a3f640 1 Processor -- start 2024-04-25T01:56:11.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0e1a3f640 1 -- start start 2024-04-25T01:56:11.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0e1a3f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0dc1029d0 0x7fd0dc197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0e1a3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc1032f0 0x7fd0dc198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0e1a3f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0dc10bd10 0x7fd0dc19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0e1a3f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc1119a0 con 0x7fd0dc1032f0 2024-04-25T01:56:11.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0e1a3f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc111820 con 0x7fd0dc10bd10 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0e1a3f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc111b20 con 0x7fd0dc1029d0 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0daffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0dc1029d0 0x7fd0dc197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0da7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc1032f0 0x7fd0dc198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0daffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0dc1029d0 0x7fd0dc197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36906/0 (socket says 172.21.15.31:36906) 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0db7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0dc10bd10 0x7fd0dc19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.336+0000 7fd0daffd640 1 -- 172.21.15.31:0/841733657 learned_addr learned my addr 172.21.15.31:0/841733657 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0da7fc640 1 -- 172.21.15.31:0/841733657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0dc1029d0 msgr2=0x7fd0dc197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0da7fc640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0dc1029d0 0x7fd0dc197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0da7fc640 1 -- 172.21.15.31:0/841733657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0dc10bd10 msgr2=0x7fd0dc19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0da7fc640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0dc10bd10 0x7fd0dc19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0da7fc640 1 -- 172.21.15.31:0/841733657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0dc19cf30 con 0x7fd0dc1032f0 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0db7fe640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0dc10bd10 0x7fd0dc19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0da7fc640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc1032f0 0x7fd0dc198480 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000e9f0 tx=0x7fd0d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0e0a3d640 1 -- 172.21.15.31:0/841733657 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d000ccc0 con 0x7fd0dc1032f0 2024-04-25T01:56:11.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/841733657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0dc19d220 con 0x7fd0dc1032f0 2024-04-25T01:56:11.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0e0a3d640 1 -- 172.21.15.31:0/841733657 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0d0005020 con 0x7fd0dc1032f0 2024-04-25T01:56:11.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0e0a3d640 1 -- 172.21.15.31:0/841733657 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0004a00 con 0x7fd0dc1032f0 2024-04-25T01:56:11.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.337+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/841733657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0dc1a4ab0 con 0x7fd0dc1032f0 2024-04-25T01:56:11.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.340+0000 7fd0e0a3d640 1 -- 172.21.15.31:0/841733657 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd0d0004ba0 con 0x7fd0dc1032f0 2024-04-25T01:56:11.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.340+0000 7fd0e482b740 1 Processor -- start 2024-04-25T01:56:11.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0e0a3d640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd0c40a2d00 0x7fd0c40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0e482b740 1 -- start start 2024-04-25T01:56:11.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0db7fe640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd0c40a2d00 0x7fd0c40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0e0a3d640 1 -- 172.21.15.31:0/841733657 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd0d00cf360 con 0x7fd0dc1032f0 2024-04-25T01:56:11.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 0x5631cb72e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 0x5631cb73d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb73da30 0x5631cb73fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0e482b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5631cb6936b0 con 0x5631cb72ee30 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0e482b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5631cb63b880 con 0x5631cb72e510 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0e482b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5631cb68f670 con 0x5631cb73da30 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0daffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 0x5631cb72e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0daffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 0x5631cb72e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36332/0 (socket says 172.21.15.31:36332) 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0daffd640 1 -- 172.21.15.31:0/4198546142 learned_addr learned my addr 172.21.15.31:0/4198546142 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0db7fe640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd0c40a2d00 0x7fd0c40a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4005fd0 tx=0x7fd0d4009500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.341+0000 7fd0da7fc640 1 --2- 172.21.15.31:0/4198546142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 0x5631cb73d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.342+0000 7fd0db7fe640 1 --2- 172.21.15.31:0/4198546142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb73da30 0x5631cb73fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.342+0000 7fd0daffd640 1 -- 172.21.15.31:0/4198546142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb73da30 msgr2=0x5631cb73fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.342+0000 7fd0daffd640 1 --2- 172.21.15.31:0/4198546142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb73da30 0x5631cb73fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.342+0000 7fd0daffd640 1 -- 172.21.15.31:0/4198546142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 msgr2=0x5631cb73d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.342+0000 7fd0daffd640 1 --2- 172.21.15.31:0/4198546142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 0x5631cb73d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.342+0000 7fd0daffd640 1 -- 172.21.15.31:0/4198546142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631cb4717d0 con 0x5631cb72e510 2024-04-25T01:56:11.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.342+0000 7fd0db7fe640 1 --2- 172.21.15.31:0/4198546142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb73da30 0x5631cb73fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:11.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.342+0000 7fd0daffd640 1 --2- 172.21.15.31:0/4198546142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 0x5631cb72e8f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc000c00 tx=0x7fd0cc031af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d6362707a1ab7e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:11.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.342+0000 7fd0c1ffb640 1 -- 172.21.15.31:0/4198546142 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc040070 con 0x5631cb72e510 2024-04-25T01:56:11.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.342+0000 7fd0c1ffb640 1 -- 172.21.15.31:0/4198546142 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0cc004030 con 0x5631cb72e510 2024-04-25T01:56:11.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.342+0000 7fd0c1ffb640 1 -- 172.21.15.31:0/4198546142 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc035040 con 0x5631cb72e510 2024-04-25T01:56:11.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.343+0000 7fd0e482b740 1 -- 172.21.15.31:0/4198546142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 msgr2=0x5631cb72e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.343+0000 7fd0e482b740 1 --2- 172.21.15.31:0/4198546142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 0x5631cb72e8f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc000c00 tx=0x7fd0cc031af0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.343+0000 7fd0e482b740 1 -- 172.21.15.31:0/4198546142 shutdown_connections 2024-04-25T01:56:11.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.343+0000 7fd0e482b740 1 --2- 172.21.15.31:0/4198546142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb73da30 0x5631cb73fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.343+0000 7fd0e482b740 1 --2- 172.21.15.31:0/4198546142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 0x5631cb73d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.343+0000 7fd0e482b740 1 --2- 172.21.15.31:0/4198546142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 0x5631cb72e8f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.343+0000 7fd0e482b740 1 -- 172.21.15.31:0/4198546142 >> 172.21.15.31:0/4198546142 conn(0x5631cb72a950 msgr2=0x5631cb72ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.343+0000 7fd0e482b740 1 -- 172.21.15.31:0/4198546142 shutdown_connections 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.343+0000 7fd0e482b740 1 -- 172.21.15.31:0/4198546142 wait complete. 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.343+0000 7fd0e482b740 1 -- start start 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.343+0000 7fd0e482b740 1 Processor -- start 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 0x5631cb73a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 0x5631cb73aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb73da30 0x5631cb73b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0e482b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5631cb740810 con 0x5631cb72ee30 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0e482b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5631cb64ef50 con 0x5631cb72e510 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0e482b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5631cb740990 con 0x5631cb73da30 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0e482b740 10 client.0 _ll_register_callbacks cb 0x5631cb68a000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0da7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 0x5631cb73aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0daffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 0x5631cb73a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0db7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb73da30 0x5631cb73b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0da7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 0x5631cb73aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46604/0 (socket says 172.21.15.31:46604) 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0da7fc640 1 -- 172.21.15.31:0/235810381 learned_addr learned my addr 172.21.15.31:0/235810381 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0daffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 0x5631cb73a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36348/0 (socket says 172.21.15.31:36348) 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0da7fc640 1 -- 172.21.15.31:0/235810381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb73da30 msgr2=0x5631cb73b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0da7fc640 1 --2- 172.21.15.31:0/235810381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb73da30 0x5631cb73b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0da7fc640 1 -- 172.21.15.31:0/235810381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 msgr2=0x5631cb73a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0da7fc640 1 --2- 172.21.15.31:0/235810381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 0x5631cb73a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0da7fc640 1 -- 172.21.15.31:0/235810381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631cb3a73f0 con 0x5631cb72ee30 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0da7fc640 1 --2- 172.21.15.31:0/235810381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 0x5631cb73aea0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd0d009b5d0 tx=0x7fd0d009b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f8bd22d8b0f2e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0baffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.344+0000 7fd0baffd640 1 -- 172.21.15.31:0/235810381 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d00b3070 con 0x5631cb72ee30 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0baffd640 1 -- 172.21.15.31:0/235810381 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd0d00051d0 con 0x5631cb72ee30 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0e482b740 1 -- 172.21.15.31:0/235810381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0cc004030 con 0x5631cb72ee30 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0baffd640 1 -- 172.21.15.31:0/235810381 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd0d001c040 con 0x5631cb72ee30 2024-04-25T01:56:11.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0e482b740 10 client.16029 fetch_fsmap learned FSMap version 28 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0e482b740 1 -- 172.21.15.31:0/235810381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5631cb3670f0 con 0x5631cb72ee30 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0baffd640 1 -- 172.21.15.31:0/235810381 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0cc004030 con 0x5631cb72ee30 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0baffd640 1 -- 172.21.15.31:0/235810381 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fd0d00a40b0 con 0x5631cb72ee30 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0e482b740 10 client.16029 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0e482b740 10 client.16029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0e482b740 20 client.16029 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0e482b740 1 --2- 172.21.15.31:0/235810381 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631cb74dc60 0x5631cb750090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0e482b740 4 client.16029 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0e482b740 1 -- 172.21.15.31:0/235810381 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631cb6a0bf0 con 0x5631cb74dc60 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.345+0000 7fd0daffd640 1 --2- 172.21.15.31:0/235810381 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631cb74dc60 0x5631cb750090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.346+0000 7fd0baffd640 10 client.16029 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:11.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.346+0000 7fd0daffd640 1 --2- 172.21.15.31:0/235810381 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631cb74dc60 0x5631cb750090 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da12bb6a9f555e1c server_cookie=d35e40e4d4775a7f in_seq=0 out_seq=0 2024-04-25T01:56:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafad7fa640 1 -- 172.21.15.31:0/1771308750 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5568f07a7540 con 0x5568f08c1060 2024-04-25T01:56:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafad7fa640 10 client.16017 handle_command_reply: tid=0 2024-04-25T01:56:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafce992740 1 client.16017 shutdown 2024-04-25T01:56:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafce992740 1 -- 172.21.15.31:0/1771308750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 msgr2=0x5568f08b96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafce992740 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08b96d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fafc000b4c0 tx=0x7fafc000bc40 comp rx=0 tx=0).stop 2024-04-25T01:56:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafce992740 1 -- 172.21.15.31:0/1771308750 shutdown_connections 2024-04-25T01:56:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafce992740 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5568f08c1060 0x5568f08c3480 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafce992740 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568f0846230 0x5568f085d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafce992740 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5568f083bb60 0x5568f08b96d0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafce992740 1 --2- 172.21.15.31:0/1771308750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5568f083b240 0x5568f08b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafce992740 1 -- 172.21.15.31:0/1771308750 >> 172.21.15.31:0/1771308750 conn(0x5568f0837560 msgr2=0x5568f0835d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafad7fa640 0 client.16017 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafce992740 1 -- 172.21.15.31:0/1771308750 shutdown_connections 2024-04-25T01:56:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.642+0000 7fafce992740 1 -- 172.21.15.31:0/1771308750 wait complete. 2024-04-25T01:56:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafce992740 20 client.16017 trim_cache size 0 max 16384 2024-04-25T01:56:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0baffd640 1 -- 172.21.15.31:0/235810381 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631cb6a0bf0 con 0x5631cb74dc60 2024-04-25T01:56:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0baffd640 10 client.16029 handle_command_reply: tid=0 2024-04-25T01:56:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 1 client.16029 shutdown 2024-04-25T01:56:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:56:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:56:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:56:11.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:56:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 1 -- 172.21.15.31:0/235810381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 msgr2=0x5631cb73aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 1 --2- 172.21.15.31:0/235810381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 0x5631cb73aea0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd0d009b5d0 tx=0x7fd0d009b990 comp rx=0 tx=0).stop 2024-04-25T01:56:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 1 -- 172.21.15.31:0/235810381 shutdown_connections 2024-04-25T01:56:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 1 --2- 172.21.15.31:0/235810381 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631cb74dc60 0x5631cb750090 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 1 --2- 172.21.15.31:0/235810381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb73da30 0x5631cb73b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 1 --2- 172.21.15.31:0/235810381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72ee30 0x5631cb73aea0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 1 --2- 172.21.15.31:0/235810381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72e510 0x5631cb73a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0baffd640 0 client.16029 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:11.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 1 -- 172.21.15.31:0/235810381 >> 172.21.15.31:0/235810381 conn(0x5631cb72a950 msgr2=0x5631cb730700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:11.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 1 -- 172.21.15.31:0/235810381 shutdown_connections 2024-04-25T01:56:11.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 1 -- 172.21.15.31:0/235810381 wait complete. 2024-04-25T01:56:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 -- 172.21.15.31:0/3038921840 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fafb00a2e20 msgr2=0x7fafb00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fafb00a2e20 0x7fafb00a52e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fafbc09c020 tx=0x7fafbc086bc0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 -- 172.21.15.31:0/3038921840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc410bdc0 msgr2=0x7fafc419c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc410bdc0 0x7fafc419c7a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00eb40 tx=0x7fafbc00c2d0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fd0e482b740 20 client.16029 trim_cache size 0 max 16384 2024-04-25T01:56:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 -- 172.21.15.31:0/3038921840 shutdown_connections 2024-04-25T01:56:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fafb00a2e20 0x7fafb00a52e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc410bdc0 0x7fafc419c7a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc4103310 0x7fafc4198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 --2- 172.21.15.31:0/3038921840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc41029f0 0x7fafc4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 -- 172.21.15.31:0/3038921840 >> 172.21.15.31:0/3038921840 conn(0x7fafc40769c0 msgr2=0x7fafc40fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 -- 172.21.15.31:0/3038921840 shutdown_connections 2024-04-25T01:56:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:11.643+0000 7fafcbba2640 1 -- 172.21.15.31:0/3038921840 wait complete. 2024-04-25T01:56:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.664+0000 7fd0e482b740 1 Processor -- start 2024-04-25T01:56:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.664+0000 7fd0e482b740 1 -- start start 2024-04-25T01:56:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.664+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72e510 0x5631cb73d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.664+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb72ee30 0x5631cb749770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.664+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb73da30 0x5631cb73b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.664+0000 7fd0e482b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc1ab0c0 con 0x5631cb72e510 2024-04-25T01:56:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.664+0000 7fd0e482b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc1acc30 con 0x5631cb73da30 2024-04-25T01:56:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.664+0000 7fd0e482b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc1ab790 con 0x5631cb72ee30 2024-04-25T01:56:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0daffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72e510 0x5631cb73d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0daffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72e510 0x5631cb73d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46614/0 (socket says 172.21.15.31:46614) 2024-04-25T01:56:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0daffd640 1 -- 172.21.15.31:0/765331656 learned_addr learned my addr 172.21.15.31:0/765331656 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0db7fe640 1 --2- 172.21.15.31:0/765331656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb73da30 0x5631cb73b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0da7fc640 1 --2- 172.21.15.31:0/765331656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb72ee30 0x5631cb749770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0daffd640 1 -- 172.21.15.31:0/765331656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb72ee30 msgr2=0x5631cb749770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0daffd640 1 --2- 172.21.15.31:0/765331656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb72ee30 0x5631cb749770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0daffd640 1 -- 172.21.15.31:0/765331656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb73da30 msgr2=0x5631cb73b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0daffd640 1 --2- 172.21.15.31:0/765331656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb73da30 0x5631cb73b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0daffd640 1 -- 172.21.15.31:0/765331656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631cb71c090 con 0x5631cb72e510 2024-04-25T01:56:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0daffd640 1 --2- 172.21.15.31:0/765331656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72e510 0x5631cb73d090 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc005090 tx=0x7fd0cc02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9224fa432ac301d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0c17fa640 1 -- 172.21.15.31:0/765331656 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc040040 con 0x5631cb72e510 2024-04-25T01:56:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0c17fa640 1 -- 172.21.15.31:0/765331656 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0cc031950 con 0x5631cb72e510 2024-04-25T01:56:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.665+0000 7fd0c17fa640 1 -- 172.21.15.31:0/765331656 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc035040 con 0x5631cb72e510 2024-04-25T01:56:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.666+0000 7fd0e482b740 1 -- 172.21.15.31:0/765331656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72e510 msgr2=0x5631cb73d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.666+0000 7fd0e482b740 1 --2- 172.21.15.31:0/765331656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72e510 0x5631cb73d090 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc005090 tx=0x7fd0cc02a480 comp rx=0 tx=0).stop 2024-04-25T01:56:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.666+0000 7fd0e482b740 1 -- 172.21.15.31:0/765331656 shutdown_connections 2024-04-25T01:56:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.666+0000 7fd0e482b740 1 --2- 172.21.15.31:0/765331656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb73da30 0x5631cb73b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.666+0000 7fd0e482b740 1 --2- 172.21.15.31:0/765331656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb72ee30 0x5631cb749770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.666+0000 7fd0e482b740 1 --2- 172.21.15.31:0/765331656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb72e510 0x5631cb73d090 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.666+0000 7fd0e482b740 1 -- 172.21.15.31:0/765331656 >> 172.21.15.31:0/765331656 conn(0x5631cb72a950 msgr2=0x5631cb7398a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.666+0000 7fd0e482b740 1 -- 172.21.15.31:0/765331656 shutdown_connections 2024-04-25T01:56:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.666+0000 7fd0e482b740 1 -- 172.21.15.31:0/765331656 wait complete. 2024-04-25T01:56:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.666+0000 7fd0e482b740 1 -- start start 2024-04-25T01:56:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0e482b740 1 Processor -- start 2024-04-25T01:56:11.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb72e510 0x5631cb745fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72ee30 0x5631cb741340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0e482b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb73da30 0x5631cb741880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0e482b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc1ac530 con 0x5631cb73da30 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0e482b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc1abe60 con 0x5631cb72ee30 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0e482b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc1a90a0 con 0x5631cb72e510 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0e482b740 10 client.0 _ll_register_callbacks cb 0x7fd0d009b750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0db7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb73da30 0x5631cb741880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0daffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb72e510 0x5631cb745fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0db7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb73da30 0x5631cb741880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46622/0 (socket says 172.21.15.31:46622) 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0db7fe640 1 -- 172.21.15.31:0/4213116212 learned_addr learned my addr 172.21.15.31:0/4213116212 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.667+0000 7fd0da7fc640 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72ee30 0x5631cb741340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0db7fe640 1 -- 172.21.15.31:0/4213116212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb72e510 msgr2=0x5631cb745fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0db7fe640 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb72e510 0x5631cb745fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0db7fe640 1 -- 172.21.15.31:0/4213116212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72ee30 msgr2=0x5631cb741340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0db7fe640 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72ee30 0x5631cb741340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0db7fe640 1 -- 172.21.15.31:0/4213116212 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631cb742080 con 0x5631cb73da30 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0daffd640 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb72e510 0x5631cb745fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0db7fe640 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb73da30 0x5631cb741880 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400fab0 tx=0x7fd0d4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c755b8fc7c740b72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0baffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0baffd640 1 -- 172.21.15.31:0/4213116212 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4004860 con 0x5631cb73da30 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0baffd640 1 -- 172.21.15.31:0/4213116212 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd0d401c600 con 0x5631cb73da30 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0e482b740 1 -- 172.21.15.31:0/4213116212 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0cc031950 con 0x5631cb73da30 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0baffd640 1 -- 172.21.15.31:0/4213116212 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd0d4035050 con 0x5631cb73da30 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0e482b740 10 client.16035 fetch_fsmap learned FSMap version 28 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.668+0000 7fd0e482b740 1 -- 172.21.15.31:0/4213116212 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5631cb7b41d0 con 0x5631cb73da30 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.669+0000 7fd0baffd640 1 -- 172.21.15.31:0/4213116212 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0cc031950 con 0x5631cb73da30 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.669+0000 7fd0baffd640 1 -- 172.21.15.31:0/4213116212 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fd0d40278f0 con 0x5631cb73da30 2024-04-25T01:56:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.669+0000 7fd0e482b740 10 client.16035 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.669+0000 7fd0e482b740 10 client.16035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.669+0000 7fd0e482b740 20 client.16035 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.669+0000 7fd0e482b740 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631cb7b4a00 0x5631cb7b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.669+0000 7fd0e482b740 4 client.16035 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.669+0000 7fd0e482b740 1 -- 172.21.15.31:0/4213116212 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5631cb69af40 con 0x5631cb7b4a00 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.669+0000 7fd0daffd640 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631cb7b4a00 0x5631cb7b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.669+0000 7fd0baffd640 10 client.16035 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.669+0000 7fd0daffd640 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631cb7b4a00 0x5631cb7b6e20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5daa7a57cfe3cc1d server_cookie=457e3d362fd1a17d in_seq=0 out_seq=0 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0baffd640 1 -- 172.21.15.31:0/4213116212 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5631cb69af40 con 0x5631cb7b4a00 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0baffd640 10 client.16035 handle_command_reply: tid=0 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 1 client.16035 shutdown 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 1 -- 172.21.15.31:0/4213116212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb73da30 msgr2=0x5631cb741880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb73da30 0x5631cb741880 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400fab0 tx=0x7fd0d4004ab0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 1 -- 172.21.15.31:0/4213116212 shutdown_connections 2024-04-25T01:56:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631cb7b4a00 0x5631cb7b6e20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631cb73da30 0x5631cb741880 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631cb72ee30 0x5631cb741340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0baffd640 0 client.16035 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 1 --2- 172.21.15.31:0/4213116212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631cb72e510 0x5631cb745fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 1 -- 172.21.15.31:0/4213116212 >> 172.21.15.31:0/4213116212 conn(0x5631cb72a950 msgr2=0x5631cb73b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 1 -- 172.21.15.31:0/4213116212 shutdown_connections 2024-04-25T01:56:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 1 -- 172.21.15.31:0/4213116212 wait complete. 2024-04-25T01:56:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.670+0000 7fd0e482b740 20 client.16035 trim_cache size 0 max 16384 2024-04-25T01:56:11.680 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:56:11.680 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T01:56:11.681 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T01:56:11.681 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:56:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/841733657 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd0c40a2d00 msgr2=0x7fd0c40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd0c40a2d00 0x7fd0c40a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4005fd0 tx=0x7fd0d4009500 comp rx=0 tx=0).stop 2024-04-25T01:56:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/841733657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc1032f0 msgr2=0x7fd0dc198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc1032f0 0x7fd0dc198480 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000e9f0 tx=0x7fd0d000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/841733657 shutdown_connections 2024-04-25T01:56:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd0c40a2d00 0x7fd0c40a51c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0dc10bd10 0x7fd0dc19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc1032f0 0x7fd0dc198480 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 --2- 172.21.15.31:0/841733657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0dc1029d0 0x7fd0dc197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/841733657 >> 172.21.15.31:0/841733657 conn(0x7fd0dc0777a0 msgr2=0x7fd0dc0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:11.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/841733657 shutdown_connections 2024-04-25T01:56:11.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.671+0000 7fd0e1a3f640 1 -- 172.21.15.31:0/841733657 wait complete. 2024-04-25T01:56:11.685 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:56:11.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:56:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.793+0000 7f314c007640 1 -- 172.21.15.31:0/384190009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3144101740 msgr2=0x7f3144101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.793+0000 7f314c007640 1 --2- 172.21.15.31:0/384190009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3144101740 0x7f3144101bc0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f3134009930 tx=0x7f313402a420 comp rx=0 tx=0).stop 2024-04-25T01:56:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.793+0000 7f314c007640 1 -- 172.21.15.31:0/384190009 shutdown_connections 2024-04-25T01:56:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.793+0000 7f314c007640 1 --2- 172.21.15.31:0/384190009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3144102100 0x7f314410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.793+0000 7f314c007640 1 --2- 172.21.15.31:0/384190009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3144101740 0x7f3144101bc0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.793+0000 7f314c007640 1 --2- 172.21.15.31:0/384190009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3144107750 0x7f3144107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.793+0000 7f314c007640 1 -- 172.21.15.31:0/384190009 >> 172.21.15.31:0/384190009 conn(0x7f3144078bd0 msgr2=0x7f3144078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.793+0000 7f314c007640 1 -- 172.21.15.31:0/384190009 shutdown_connections 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.793+0000 7f314c007640 1 -- 172.21.15.31:0/384190009 wait complete. 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f314c007640 1 Processor -- start 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f314c007640 1 -- start start 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f314c007640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3144101740 0x7f3144197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f314c007640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3144102100 0x7f3144198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f314c007640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3144107750 0x7f314419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f314c007640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3144111a10 con 0x7f3144102100 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f314c007640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3144111890 con 0x7f3144101740 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f314c007640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3144111b90 con 0x7f3144107750 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f31495a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3144102100 0x7f3144198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f31495a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3144102100 0x7f3144198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46636/0 (socket says 172.21.15.31:46636) 2024-04-25T01:56:11.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f31495a2640 1 -- 172.21.15.31:0/2197621114 learned_addr learned my addr 172.21.15.31:0/2197621114 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f3149da3640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3144101740 0x7f3144197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f314a5a4640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3144107750 0x7f314419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f31495a2640 1 -- 172.21.15.31:0/2197621114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3144107750 msgr2=0x7f314419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f31495a2640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3144107750 0x7f314419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.794+0000 7f31495a2640 1 -- 172.21.15.31:0/2197621114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3144101740 msgr2=0x7f3144197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.795+0000 7f31495a2640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3144101740 0x7f3144197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.795+0000 7f31495a2640 1 -- 172.21.15.31:0/2197621114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314419cf40 con 0x7f3144102100 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.795+0000 7f31495a2640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3144102100 0x7f3144198490 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3134005940 tx=0x7f31340040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.795+0000 7f313affd640 1 -- 172.21.15.31:0/2197621114 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313403f030 con 0x7f3144102100 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.795+0000 7f313affd640 1 -- 172.21.15.31:0/2197621114 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31340044c0 con 0x7f3144102100 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.795+0000 7f314c007640 1 -- 172.21.15.31:0/2197621114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314419d1d0 con 0x7f3144102100 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.795+0000 7f313affd640 1 -- 172.21.15.31:0/2197621114 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3134035070 con 0x7f3144102100 2024-04-25T01:56:11.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.795+0000 7f314c007640 1 -- 172.21.15.31:0/2197621114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31441a4ac0 con 0x7f3144102100 2024-04-25T01:56:11.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.796+0000 7f314c007640 1 -- 172.21.15.31:0/2197621114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3144068560 con 0x7f3144102100 2024-04-25T01:56:11.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.797+0000 7f313affd640 1 -- 172.21.15.31:0/2197621114 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3134042030 con 0x7f3144102100 2024-04-25T01:56:11.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.798+0000 7f313affd640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f31300a2d00 0x7f31300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:11.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.798+0000 7f3149da3640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f31300a2d00 0x7f31300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:11.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.798+0000 7f313affd640 1 -- 172.21.15.31:0/2197621114 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f31340ef690 con 0x7f3144102100 2024-04-25T01:56:11.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.798+0000 7f3149da3640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f31300a2d00 0x7f31300a51c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3140009b50 tx=0x7f3140005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:11.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:11.803+0000 7f313affd640 1 -- 172.21.15.31:0/2197621114 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f31340bbcb0 con 0x7f3144102100 2024-04-25T01:56:11.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:11 smithi031 ceph-mon[29204]: pgmap v54: 97 pgs: 97 active+clean; 27 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 1.6 MiB/s wr, 143 op/s 2024-04-25T01:56:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.063+0000 7f314c007640 1 -- 172.21.15.31:0/2197621114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31441992a0 con 0x7f3144102100 2024-04-25T01:56:12.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.064+0000 7f313affd640 1 -- 172.21.15.31:0/2197621114 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11930 (secure 0 0 0) 0x7f31340bffb0 con 0x7f3144102100 2024-04-25T01:56:12.066 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:56:12.066 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:56:11.299762+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24463,24451,24503,24473,24487]},"id":1}]} 2024-04-25T01:56:12.066 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T01:56:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 -- 172.21.15.31:0/2197621114 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f31300a2d00 msgr2=0x7f31300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f31300a2d00 0x7f31300a51c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3140009b50 tx=0x7f3140005f90 comp rx=0 tx=0).stop 2024-04-25T01:56:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 -- 172.21.15.31:0/2197621114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3144102100 msgr2=0x7f3144198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3144102100 0x7f3144198490 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3134005940 tx=0x7f31340040c0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 -- 172.21.15.31:0/2197621114 shutdown_connections 2024-04-25T01:56:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f31300a2d00 0x7f31300a51c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3144107750 0x7f314419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3144102100 0x7f3144198490 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 --2- 172.21.15.31:0/2197621114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3144101740 0x7f3144197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 -- 172.21.15.31:0/2197621114 >> 172.21.15.31:0/2197621114 conn(0x7f3144078bd0 msgr2=0x7f31440ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 -- 172.21.15.31:0/2197621114 shutdown_connections 2024-04-25T01:56:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.066+0000 7f314c007640 1 -- 172.21.15.31:0/2197621114 wait complete. 2024-04-25T01:56:12.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:56:12.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:11 smithi040 ceph-mon[32418]: pgmap v54: 97 pgs: 97 active+clean; 27 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 1.6 MiB/s wr, 143 op/s 2024-04-25T01:56:12.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:11 smithi134 ceph-mon[33098]: pgmap v54: 97 pgs: 97 active+clean; 27 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 1.6 MiB/s wr, 143 op/s 2024-04-25T01:56:12.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.189+0000 7fa44e0ba640 1 -- 172.21.15.31:0/1562158512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448106810 msgr2=0x7fa448106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.189+0000 7fa44e0ba640 1 --2- 172.21.15.31:0/1562158512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448106810 0x7fa448106bf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa438009900 tx=0x7fa43802a410 comp rx=0 tx=0).stop 2024-04-25T01:56:12.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.189+0000 7fa44e0ba640 1 -- 172.21.15.31:0/1562158512 shutdown_connections 2024-04-25T01:56:12.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.189+0000 7fa44e0ba640 1 --2- 172.21.15.31:0/1562158512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4481012e0 0x7fa44810d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.189+0000 7fa44e0ba640 1 --2- 172.21.15.31:0/1562158512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa448100920 0x7fa448100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.189+0000 7fa44e0ba640 1 --2- 172.21.15.31:0/1562158512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448106810 0x7fa448106bf0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.189+0000 7fa44e0ba640 1 -- 172.21.15.31:0/1562158512 >> 172.21.15.31:0/1562158512 conn(0x7fa448077620 msgr2=0x7fa448077a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.189+0000 7fa44e0ba640 1 -- 172.21.15.31:0/1562158512 shutdown_connections 2024-04-25T01:56:12.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.189+0000 7fa44e0ba640 1 -- 172.21.15.31:0/1562158512 wait complete. 2024-04-25T01:56:12.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.190+0000 7fa44e0ba640 1 Processor -- start 2024-04-25T01:56:12.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.190+0000 7fa44e0ba640 1 -- start start 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.190+0000 7fa44e0ba640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100920 0x7fa448197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.190+0000 7fa44e0ba640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4481012e0 0x7fa4481984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.190+0000 7fa44e0ba640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa448106810 0x7fa44819c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.190+0000 7fa44e0ba640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa448111a50 con 0x7fa448100920 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.190+0000 7fa44e0ba640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481118d0 con 0x7fa4481012e0 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.190+0000 7fa44e0ba640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa448111bd0 con 0x7fa448106810 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.190+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100920 0x7fa448197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100920 0x7fa448197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46650/0 (socket says 172.21.15.31:46650) 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa4477fe640 1 -- 172.21.15.31:0/527941146 learned_addr learned my addr 172.21.15.31:0/527941146 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa446ffd640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4481012e0 0x7fa4481984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa447fff640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa448106810 0x7fa44819c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa4477fe640 1 -- 172.21.15.31:0/527941146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa448106810 msgr2=0x7fa44819c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa4477fe640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa448106810 0x7fa44819c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa4477fe640 1 -- 172.21.15.31:0/527941146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4481012e0 msgr2=0x7fa4481984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa4477fe640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4481012e0 0x7fa4481984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa4477fe640 1 -- 172.21.15.31:0/527941146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44819cf60 con 0x7fa448100920 2024-04-25T01:56:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa446ffd640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4481012e0 0x7fa4481984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa4477fe640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100920 0x7fa448197f70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa438005940 tx=0x7fa4380040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa444ff9640 1 -- 172.21.15.31:0/527941146 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43803f030 con 0x7fa448100920 2024-04-25T01:56:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa444ff9640 1 -- 172.21.15.31:0/527941146 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4380044c0 con 0x7fa448100920 2024-04-25T01:56:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa44e0ba640 1 -- 172.21.15.31:0/527941146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa44819d1f0 con 0x7fa448100920 2024-04-25T01:56:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa444ff9640 1 -- 172.21.15.31:0/527941146 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438035070 con 0x7fa448100920 2024-04-25T01:56:12.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.191+0000 7fa44e0ba640 1 -- 172.21.15.31:0/527941146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4481a4ae0 con 0x7fa448100920 2024-04-25T01:56:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.193+0000 7fa444ff9640 1 -- 172.21.15.31:0/527941146 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa438042030 con 0x7fa448100920 2024-04-25T01:56:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.194+0000 7fa444ff9640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4300a2d00 0x7fa4300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.194+0000 7fa450dc6740 1 Processor -- start 2024-04-25T01:56:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.194+0000 7fa450dc6740 1 -- start start 2024-04-25T01:56:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.194+0000 7fa446ffd640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4300a2d00 0x7fa4300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.194+0000 7fa444ff9640 1 -- 172.21.15.31:0/527941146 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa4380ef710 con 0x7fa448100920 2024-04-25T01:56:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.194+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b0485860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.194+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 0x5597b0490030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.194+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0490570 0x5597b0492960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.194+0000 7fa450dc6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5597b03ea670 con 0x5597b0485da0 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.194+0000 7fa450dc6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5597b0392970 con 0x5597b0485480 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.194+0000 7fa450dc6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5597b03e6630 con 0x5597b0490570 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa446ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 0x5597b0490030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b0485860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b0485860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36436/0 (socket says 172.21.15.31:36436) 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa4477fe640 1 -- 172.21.15.31:0/2898478900 learned_addr learned my addr 172.21.15.31:0/2898478900 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa4477fe640 1 -- 172.21.15.31:0/2898478900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0490570 msgr2=0x5597b0492960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa4477fe640 1 --2- 172.21.15.31:0/2898478900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0490570 0x5597b0492960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa4477fe640 1 -- 172.21.15.31:0/2898478900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 msgr2=0x5597b0490030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa4477fe640 1 --2- 172.21.15.31:0/2898478900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 0x5597b0490030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa4477fe640 1 -- 172.21.15.31:0/2898478900 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597b03b6400 con 0x5597b0485480 2024-04-25T01:56:12.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa446ffd640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4300a2d00 0x7fa4300a51c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009fd0 tx=0x7fa43c00e780 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa4477fe640 1 --2- 172.21.15.31:0/2898478900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b0485860 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa4380bbcc0 tx=0x7fa4380bbd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5898b0adbeb9995 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa42dffb640 1 -- 172.21.15.31:0/2898478900 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438049070 con 0x5597b0485480 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa42dffb640 1 -- 172.21.15.31:0/2898478900 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa438003d60 con 0x5597b0485480 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.195+0000 7fa42dffb640 1 -- 172.21.15.31:0/2898478900 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4380bc1f0 con 0x5597b0485480 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.196+0000 7fa450dc6740 1 -- 172.21.15.31:0/2898478900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 msgr2=0x5597b0485860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.196+0000 7fa450dc6740 1 --2- 172.21.15.31:0/2898478900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b0485860 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa4380bbcc0 tx=0x7fa4380bbd30 comp rx=0 tx=0).stop 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.196+0000 7fa450dc6740 1 -- 172.21.15.31:0/2898478900 shutdown_connections 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.196+0000 7fa450dc6740 1 --2- 172.21.15.31:0/2898478900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0490570 0x5597b0492960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.196+0000 7fa450dc6740 1 --2- 172.21.15.31:0/2898478900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 0x5597b0490030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.196+0000 7fa450dc6740 1 --2- 172.21.15.31:0/2898478900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b0485860 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.196+0000 7fa450dc6740 1 -- 172.21.15.31:0/2898478900 >> 172.21.15.31:0/2898478900 conn(0x5597b04818a0 msgr2=0x5597b0481c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.197+0000 7fa450dc6740 1 -- 172.21.15.31:0/2898478900 shutdown_connections 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.197+0000 7fa450dc6740 1 -- 172.21.15.31:0/2898478900 wait complete. 2024-04-25T01:56:12.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.197+0000 7fa450dc6740 1 -- start start 2024-04-25T01:56:12.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.197+0000 7fa450dc6740 1 Processor -- start 2024-04-25T01:56:12.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.197+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b048fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.197+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 0x5597b048d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.197+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0490570 0x5597b048d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.197+0000 7fa450dc6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5597b04930f0 con 0x5597b0490570 2024-04-25T01:56:12.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.197+0000 7fa450dc6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5597b03a6050 con 0x5597b0485480 2024-04-25T01:56:12.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.197+0000 7fa450dc6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5597b0493270 con 0x5597b0485da0 2024-04-25T01:56:12.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.197+0000 7fa450dc6740 10 client.0 _ll_register_callbacks cb 0x5597b00fdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:12.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa447fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0490570 0x5597b048d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa446ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 0x5597b048d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b048fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa447fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0490570 0x5597b048d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46672/0 (socket says 172.21.15.31:46672) 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa446ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 0x5597b048d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:37000/0 (socket says 172.21.15.31:37000) 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa447fff640 1 -- 172.21.15.31:0/3813771326 learned_addr learned my addr 172.21.15.31:0/3813771326 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa4477fe640 1 -- 172.21.15.31:0/3813771326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 msgr2=0x5597b048d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa4477fe640 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 0x5597b048d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa4477fe640 1 -- 172.21.15.31:0/3813771326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0490570 msgr2=0x5597b048d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa4477fe640 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0490570 0x5597b048d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa4477fe640 1 -- 172.21.15.31:0/3813771326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597b0377dc0 con 0x5597b0485480 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa447fff640 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0490570 0x5597b048d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa446ffd640 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 0x5597b048d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa4477fe640 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b048fee0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa4380bbcc0 tx=0x7fa4380c4a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e03bf8c981beccb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa426ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa426ffd640 1 -- 172.21.15.31:0/3813771326 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438049050 con 0x5597b0485480 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa426ffd640 1 -- 172.21.15.31:0/3813771326 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa4380bc190 con 0x5597b0485480 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.198+0000 7fa450dc6740 1 -- 172.21.15.31:0/3813771326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa438003d60 con 0x5597b0485480 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa426ffd640 1 -- 172.21.15.31:0/3813771326 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa4380c4460 con 0x5597b0485480 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa450dc6740 10 client.25658 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa450dc6740 1 -- 172.21.15.31:0/3813771326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5597b00fe3f0 con 0x5597b0485480 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa426ffd640 1 -- 172.21.15.31:0/3813771326 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa438003d60 con 0x5597b0485480 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa426ffd640 1 -- 172.21.15.31:0/3813771326 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa4380d4070 con 0x5597b0485480 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa450dc6740 10 client.25658 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa450dc6740 10 client.25658 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa450dc6740 20 client.25658 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa450dc6740 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x5597b04a01e0 0x5597b04a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa450dc6740 4 client.25658 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa450dc6740 1 -- 172.21.15.31:0/3813771326 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597b03f7bb0 con 0x5597b04a01e0 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.199+0000 7fa446ffd640 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x5597b04a01e0 0x5597b04a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.200+0000 7fa446ffd640 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x5597b04a01e0 0x5597b04a25d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=efb1f2d3d8b67549 server_cookie=cebf198b8d532962 in_seq=0 out_seq=0 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.200+0000 7fa426ffd640 10 client.25658 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.200+0000 7fa426ffd640 1 -- 172.21.15.31:0/3813771326 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5597b03f7bb0 con 0x5597b04a01e0 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.200+0000 7fa426ffd640 10 client.25658 handle_command_reply: tid=0 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.200+0000 7fa450dc6740 1 client.25658 shutdown 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa450dc6740 1 -- 172.21.15.31:0/3813771326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 msgr2=0x5597b048fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa450dc6740 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b048fee0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa4380bbcc0 tx=0x7fa4380c4a70 comp rx=0 tx=0).stop 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa450dc6740 1 -- 172.21.15.31:0/3813771326 shutdown_connections 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa450dc6740 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x5597b04a01e0 0x5597b04a25d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa450dc6740 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0490570 0x5597b048d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa450dc6740 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 0x5597b048d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa450dc6740 1 --2- 172.21.15.31:0/3813771326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b048fee0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa426ffd640 0 client.25658 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:56:12.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa450dc6740 1 -- 172.21.15.31:0/3813771326 >> 172.21.15.31:0/3813771326 conn(0x5597b04818a0 msgr2=0x5597b0480b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa450dc6740 1 -- 172.21.15.31:0/3813771326 shutdown_connections 2024-04-25T01:56:12.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa450dc6740 1 -- 172.21.15.31:0/3813771326 wait complete. 2024-04-25T01:56:12.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.201+0000 7fa450dc6740 20 client.25658 trim_cache size 0 max 16384 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.215+0000 7fa450dc6740 1 Processor -- start 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.215+0000 7fa450dc6740 1 -- start start 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b049bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 0x5597b047ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0490570 0x5597b048fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa450dc6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481ab050 con 0x5597b0485da0 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa450dc6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481acc50 con 0x5597b0485480 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa450dc6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481ab720 con 0x5597b0490570 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa446ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 0x5597b047ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b049bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b049bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36454/0 (socket says 172.21.15.31:36454) 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa446ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 0x5597b047ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46676/0 (socket says 172.21.15.31:46676) 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa4477fe640 1 -- 172.21.15.31:0/534187756 learned_addr learned my addr 172.21.15.31:0/534187756 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa447fff640 1 --2- 172.21.15.31:0/534187756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0490570 0x5597b048fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa446ffd640 1 -- 172.21.15.31:0/534187756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0490570 msgr2=0x5597b048fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa446ffd640 1 --2- 172.21.15.31:0/534187756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0490570 0x5597b048fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa446ffd640 1 -- 172.21.15.31:0/534187756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 msgr2=0x5597b049bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa446ffd640 1 --2- 172.21.15.31:0/534187756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b049bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa446ffd640 1 -- 172.21.15.31:0/534187756 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597b01c87d0 con 0x5597b0485da0 2024-04-25T01:56:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa4477fe640 1 --2- 172.21.15.31:0/534187756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b049bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa446ffd640 1 --2- 172.21.15.31:0/534187756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 0x5597b047ca80 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x5597b048ca30 tx=0x7fa43c027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb67b40d3d20c86b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa42d7fa640 1 -- 172.21.15.31:0/534187756 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c035e60 con 0x5597b0485da0 2024-04-25T01:56:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa42d7fa640 1 -- 172.21.15.31:0/534187756 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa43c03b070 con 0x5597b0485da0 2024-04-25T01:56:12.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.216+0000 7fa42d7fa640 1 -- 172.21.15.31:0/534187756 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c0355f0 con 0x5597b0485da0 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.217+0000 7fa450dc6740 1 -- 172.21.15.31:0/534187756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 msgr2=0x5597b047ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.217+0000 7fa450dc6740 1 --2- 172.21.15.31:0/534187756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 0x5597b047ca80 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x5597b048ca30 tx=0x7fa43c027000 comp rx=0 tx=0).stop 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.217+0000 7fa450dc6740 1 -- 172.21.15.31:0/534187756 shutdown_connections 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.217+0000 7fa450dc6740 1 --2- 172.21.15.31:0/534187756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0490570 0x5597b048fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.217+0000 7fa450dc6740 1 --2- 172.21.15.31:0/534187756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485da0 0x5597b047ca80 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.217+0000 7fa450dc6740 1 --2- 172.21.15.31:0/534187756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0485480 0x5597b049bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.217+0000 7fa450dc6740 1 -- 172.21.15.31:0/534187756 >> 172.21.15.31:0/534187756 conn(0x5597b04818a0 msgr2=0x5597b0488840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.217+0000 7fa450dc6740 1 -- 172.21.15.31:0/534187756 shutdown_connections 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.217+0000 7fa450dc6740 1 -- 172.21.15.31:0/534187756 wait complete. 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa450dc6740 1 -- start start 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa450dc6740 1 Processor -- start 2024-04-25T01:56:12.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485480 0x5597b04936d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 0x5597b0493c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa450dc6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0490570 0x5597b05044f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa450dc6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481ac4c0 con 0x5597b0485480 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa450dc6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481abdf0 con 0x5597b0490570 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa450dc6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481a9050 con 0x5597b0485da0 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa450dc6740 10 client.0 _ll_register_callbacks cb 0x7fa43c009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485480 0x5597b04936d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485480 0x5597b04936d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46690/0 (socket says 172.21.15.31:46690) 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa4477fe640 1 -- 172.21.15.31:0/4238727182 learned_addr learned my addr 172.21.15.31:0/4238727182 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa447fff640 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0490570 0x5597b05044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa446ffd640 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 0x5597b0493c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa4477fe640 1 -- 172.21.15.31:0/4238727182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 msgr2=0x5597b0493c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa4477fe640 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 0x5597b0493c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa4477fe640 1 -- 172.21.15.31:0/4238727182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0490570 msgr2=0x5597b05044f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa4477fe640 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0490570 0x5597b05044f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.218+0000 7fa4477fe640 1 -- 172.21.15.31:0/4238727182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597b0504b00 con 0x5597b0485480 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa4477fe640 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485480 0x5597b04936d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa4380bba30 tx=0x7fa4380bbd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854eacf1cb3fe697 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa426ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa426ffd640 1 -- 172.21.15.31:0/4238727182 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438049020 con 0x5597b0485480 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa426ffd640 1 -- 172.21.15.31:0/4238727182 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fa4380b8460 con 0x5597b0485480 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa450dc6740 1 -- 172.21.15.31:0/4238727182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa43c03b070 con 0x5597b0485480 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa426ffd640 1 -- 172.21.15.31:0/4238727182 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa4380c4600 con 0x5597b0485480 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa450dc6740 10 client.16059 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa450dc6740 1 -- 172.21.15.31:0/4238727182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5597b0504b00 con 0x5597b0485480 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa426ffd640 1 -- 172.21.15.31:0/4238727182 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa43c03b070 con 0x5597b0485480 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa426ffd640 1 -- 172.21.15.31:0/4238727182 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa4380d4040 con 0x5597b0485480 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa450dc6740 10 client.16059 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa450dc6740 10 client.16059 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:56:12.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa450dc6740 20 client.16059 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa450dc6740 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x5597b0506110 0x5597b0508500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa450dc6740 4 client.16059 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.219+0000 7fa450dc6740 1 -- 172.21.15.31:0/4238727182 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5597b03f1f00 con 0x5597b0506110 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa446ffd640 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x5597b0506110 0x5597b0508500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa446ffd640 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x5597b0506110 0x5597b0508500 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7bdb02cadfbe76f8 server_cookie=f7ad4108405ff626 in_seq=0 out_seq=0 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa426ffd640 10 client.16059 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa426ffd640 1 -- 172.21.15.31:0/4238727182 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5597b03f1f00 con 0x5597b0506110 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa426ffd640 10 client.16059 handle_command_reply: tid=0 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa450dc6740 1 client.16059 shutdown 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa450dc6740 1 -- 172.21.15.31:0/4238727182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485480 msgr2=0x5597b04936d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa450dc6740 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485480 0x5597b04936d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa4380bba30 tx=0x7fa4380bbd30 comp rx=0 tx=0).stop 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa450dc6740 1 -- 172.21.15.31:0/4238727182 shutdown_connections 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa450dc6740 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x5597b0506110 0x5597b0508500 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa450dc6740 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597b0490570 0x5597b05044f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa450dc6740 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5597b0485da0 0x5597b0493c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa450dc6740 1 --2- 172.21.15.31:0/4238727182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b0485480 0x5597b04936d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa450dc6740 1 -- 172.21.15.31:0/4238727182 >> 172.21.15.31:0/4238727182 conn(0x5597b04818a0 msgr2=0x5597b0488840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa426ffd640 0 client.16059 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:56:12.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.220+0000 7fa450dc6740 1 -- 172.21.15.31:0/4238727182 shutdown_connections 2024-04-25T01:56:12.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa450dc6740 1 -- 172.21.15.31:0/4238727182 wait complete. 2024-04-25T01:56:12.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa450dc6740 20 client.16059 trim_cache size 0 max 16384 2024-04-25T01:56:12.229 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:56:12.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 -- 172.21.15.31:0/527941146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4300a2d00 msgr2=0x7fa4300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4300a2d00 0x7fa4300a51c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009fd0 tx=0x7fa43c00e780 comp rx=0 tx=0).stop 2024-04-25T01:56:12.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 -- 172.21.15.31:0/527941146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100920 msgr2=0x7fa448197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100920 0x7fa448197f70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa438005940 tx=0x7fa4380040c0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 -- 172.21.15.31:0/527941146 shutdown_connections 2024-04-25T01:56:12.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa4300a2d00 0x7fa4300a51c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa448106810 0x7fa44819c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4481012e0 0x7fa4481984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 --2- 172.21.15.31:0/527941146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa448100920 0x7fa448197f70 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 -- 172.21.15.31:0/527941146 >> 172.21.15.31:0/527941146 conn(0x7fa448077620 msgr2=0x7fa4480ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 -- 172.21.15.31:0/527941146 shutdown_connections 2024-04-25T01:56:12.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.221+0000 7fa44e0ba640 1 -- 172.21.15.31:0/527941146 wait complete. 2024-04-25T01:56:12.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:56:12.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 10374 3.87 MB/sec warmup 19 sec latency 22.107 ms 2024-04-25T01:56:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.346+0000 7f0a1021b640 1 -- 172.21.15.31:0/3122082254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a0810bd00 msgr2=0x7f0a0810e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.346+0000 7f0a1021b640 1 --2- 172.21.15.31:0/3122082254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a0810bd00 0x7f0a0810e1a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009ca0 tx=0x7f0a0002a9e0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.346+0000 7f0a1021b640 1 -- 172.21.15.31:0/3122082254 shutdown_connections 2024-04-25T01:56:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.346+0000 7f0a1021b640 1 --2- 172.21.15.31:0/3122082254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a0810bd00 0x7f0a0810e1a0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.346+0000 7f0a1021b640 1 --2- 172.21.15.31:0/3122082254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a081032e0 0x7f0a0810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.346+0000 7f0a1021b640 1 --2- 172.21.15.31:0/3122082254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a081029c0 0x7f0a08102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.346+0000 7f0a1021b640 1 -- 172.21.15.31:0/3122082254 >> 172.21.15.31:0/3122082254 conn(0x7f0a08077740 msgr2=0x7f0a08077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.346+0000 7f0a1021b640 1 -- 172.21.15.31:0/3122082254 shutdown_connections 2024-04-25T01:56:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.346+0000 7f0a1021b640 1 -- 172.21.15.31:0/3122082254 wait complete. 2024-04-25T01:56:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a1021b640 1 Processor -- start 2024-04-25T01:56:12.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a1021b640 1 -- start start 2024-04-25T01:56:12.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a1021b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a081029c0 0x7f0a08197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a1021b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a081032e0 0x7f0a08198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a1021b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a0810bd00 0x7f0a0819c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a1021b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0a08111a00 con 0x7f0a0810bd00 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a1021b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0a08111880 con 0x7f0a081029c0 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a1021b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0a08111b80 con 0x7f0a081032e0 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a0e7b8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a0810bd00 0x7f0a0819c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a0dfb7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a081029c0 0x7f0a08197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a0e7b8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a0810bd00 0x7f0a0819c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46698/0 (socket says 172.21.15.31:46698) 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a0e7b8640 1 -- 172.21.15.31:0/2451152916 learned_addr learned my addr 172.21.15.31:0/2451152916 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.347+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a081032e0 0x7f0a08198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f0a0e7b8640 1 -- 172.21.15.31:0/2451152916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a081032e0 msgr2=0x7f0a08198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f0a0e7b8640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a081032e0 0x7f0a08198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f0a0e7b8640 1 -- 172.21.15.31:0/2451152916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a081029c0 msgr2=0x7f0a08197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f0a0e7b8640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a081029c0 0x7f0a08197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f0a0e7b8640 1 -- 172.21.15.31:0/2451152916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0819ced0 con 0x7f0a0810bd00 2024-04-25T01:56:12.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f0a0dfb7640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a081029c0 0x7f0a08197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a081032e0 0x7f0a08198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:12.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f0a0e7b8640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a0810bd00 0x7f0a0819c7a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0a0002f040 tx=0x7f0a0002fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f09faffd640 1 -- 172.21.15.31:0/2451152916 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a00033040 con 0x7f0a0810bd00 2024-04-25T01:56:12.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f0a1021b640 1 -- 172.21.15.31:0/2451152916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a0819d160 con 0x7f0a0810bd00 2024-04-25T01:56:12.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f09faffd640 1 -- 172.21.15.31:0/2451152916 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a0002e7a0 con 0x7f0a0810bd00 2024-04-25T01:56:12.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f09faffd640 1 -- 172.21.15.31:0/2451152916 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a00043650 con 0x7f0a0810bd00 2024-04-25T01:56:12.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.348+0000 7f0a1021b640 1 -- 172.21.15.31:0/2451152916 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a08100540 con 0x7f0a0810bd00 2024-04-25T01:56:12.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.351+0000 7f09faffd640 1 -- 172.21.15.31:0/2451152916 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0a00036070 con 0x7f0a0810bd00 2024-04-25T01:56:12.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.351+0000 7f0a12ffd740 1 Processor -- start 2024-04-25T01:56:12.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.351+0000 7f0a12ffd740 1 -- start start 2024-04-25T01:56:12.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 0x55a41d939960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d939ea0 0x55a41d944030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d944570 0x55a41d946960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a41d89e670 con 0x55a41d939580 2024-04-25T01:56:12.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a41d846970 con 0x55a41d944570 2024-04-25T01:56:12.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a41d89a630 con 0x55a41d939ea0 2024-04-25T01:56:12.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0d7b6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 0x55a41d939960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0d7b6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 0x55a41d939960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46708/0 (socket says 172.21.15.31:46708) 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0d7b6640 1 -- 172.21.15.31:0/2334483030 learned_addr learned my addr 172.21.15.31:0/2334483030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f09faffd640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09f40a2d00 0x7f09f40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0dfb7640 1 --2- 172.21.15.31:0/2334483030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d939ea0 0x55a41d944030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/2334483030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d944570 0x55a41d946960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0e7b8640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09f40a2d00 0x7f09f40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0d7b6640 1 -- 172.21.15.31:0/2334483030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d939ea0 msgr2=0x55a41d944030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/2334483030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d939ea0 0x55a41d944030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f09faffd640 1 -- 172.21.15.31:0/2451152916 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0a000f01b0 con 0x7f0a0810bd00 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0d7b6640 1 -- 172.21.15.31:0/2334483030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d944570 msgr2=0x55a41d946960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/2334483030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d944570 0x55a41d946960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0d7b6640 1 -- 172.21.15.31:0/2334483030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a41d86a400 con 0x55a41d939580 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.352+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/2334483030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 0x55a41d939960 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f09fc015600 tx=0x7f09fc009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5eb0833db25c2d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.353+0000 7f0a0e7b8640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09f40a2d00 0x7f09f40a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0a00006010 tx=0x7f0a000095c0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.353+0000 7f09f3fff640 1 -- 172.21.15.31:0/2334483030 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc01b020 con 0x55a41d939580 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.353+0000 7f09f3fff640 1 -- 172.21.15.31:0/2334483030 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09fc0035f0 con 0x55a41d939580 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.353+0000 7f09f3fff640 1 -- 172.21.15.31:0/2334483030 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc0044e0 con 0x55a41d939580 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.353+0000 7f0a12ffd740 1 -- 172.21.15.31:0/2334483030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 msgr2=0x55a41d939960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.353+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/2334483030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 0x55a41d939960 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f09fc015600 tx=0x7f09fc009920 comp rx=0 tx=0).stop 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.353+0000 7f0a12ffd740 1 -- 172.21.15.31:0/2334483030 shutdown_connections 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.354+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/2334483030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d944570 0x55a41d946960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.354+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/2334483030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d939ea0 0x55a41d944030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.354+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/2334483030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 0x55a41d939960 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.354+0000 7f0a12ffd740 1 -- 172.21.15.31:0/2334483030 >> 172.21.15.31:0/2334483030 conn(0x55a41d9358a0 msgr2=0x55a41d935c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.354+0000 7f0a12ffd740 1 -- 172.21.15.31:0/2334483030 shutdown_connections 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.354+0000 7f0a12ffd740 1 -- 172.21.15.31:0/2334483030 wait complete. 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.354+0000 7f0a12ffd740 1 -- start start 2024-04-25T01:56:12.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.354+0000 7f0a12ffd740 1 Processor -- start 2024-04-25T01:56:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.354+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d939580 0x55a41d943ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 0x55a41d9411b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d944570 0x55a41d941710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a41d947360 con 0x55a41d944570 2024-04-25T01:56:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a41d85a050 con 0x55a41d939ea0 2024-04-25T01:56:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a41d9474e0 con 0x55a41d939580 2024-04-25T01:56:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a12ffd740 10 client.0 _ll_register_callbacks cb 0x55a41d5b1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0dfb7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 0x55a41d9411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0d7b6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d944570 0x55a41d941710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0dfb7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 0x55a41d9411b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36492/0 (socket says 172.21.15.31:36492) 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0dfb7640 1 -- 172.21.15.31:0/1982799580 learned_addr learned my addr 172.21.15.31:0/1982799580 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d939580 0x55a41d943ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0d7b6640 1 -- 172.21.15.31:0/1982799580 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d939580 msgr2=0x55a41d943ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d939580 0x55a41d943ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0d7b6640 1 -- 172.21.15.31:0/1982799580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 msgr2=0x55a41d9411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 0x55a41d9411b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0d7b6640 1 -- 172.21.15.31:0/1982799580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a41d82bdc0 con 0x55a41d944570 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0dfb7640 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 0x55a41d9411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d944570 0x55a41d941710 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f09fc015600 tx=0x7f09fc003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8138dc19733b807 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f09f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f09f17fa640 1 -- 172.21.15.31:0/1982799580 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc01b030 con 0x55a41d944570 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f09f17fa640 1 -- 172.21.15.31:0/1982799580 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f09fc004200 con 0x55a41d944570 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.355+0000 7f0a12ffd740 1 -- 172.21.15.31:0/1982799580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09fc0035f0 con 0x55a41d944570 2024-04-25T01:56:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f09f17fa640 1 -- 172.21.15.31:0/1982799580 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f09fc004400 con 0x55a41d944570 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f0a12ffd740 10 client.16071 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f0a12ffd740 1 -- 172.21.15.31:0/1982799580 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a41d5b23f0 con 0x55a41d944570 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f09f17fa640 1 -- 172.21.15.31:0/1982799580 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09fc0035f0 con 0x55a41d944570 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f09f17fa640 1 -- 172.21.15.31:0/1982799580 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f09fc004aa0 con 0x55a41d944570 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f0a12ffd740 10 client.16071 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f0a12ffd740 10 client.16071 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f0a12ffd740 20 client.16071 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a41d954450 0x55a41d956840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f0a12ffd740 4 client.16071 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f0a12ffd740 1 -- 172.21.15.31:0/1982799580 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a41d8abbb0 con 0x55a41d954450 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.356+0000 7f0a0dfb7640 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a41d954450 0x55a41d956840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.357+0000 7f0a0dfb7640 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a41d954450 0x55a41d956840 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ccb28013032620f3 server_cookie=635a084a4a45f18b in_seq=0 out_seq=0 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.357+0000 7f09f17fa640 10 client.16071 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.357+0000 7f09f17fa640 1 -- 172.21.15.31:0/1982799580 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a41d8abbb0 con 0x55a41d954450 2024-04-25T01:56:12.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.357+0000 7f09f17fa640 10 client.16071 handle_command_reply: tid=0 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.357+0000 7f0a12ffd740 1 client.16071 shutdown 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.357+0000 7f0a12ffd740 1 -- 172.21.15.31:0/1982799580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d944570 msgr2=0x55a41d941710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.358+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d944570 0x55a41d941710 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f09fc015600 tx=0x7f09fc003e20 comp rx=0 tx=0).stop 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.358+0000 7f0a12ffd740 1 -- 172.21.15.31:0/1982799580 shutdown_connections 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.358+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a41d954450 0x55a41d956840 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.358+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d944570 0x55a41d941710 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.358+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 0x55a41d9411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.358+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/1982799580 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d939580 0x55a41d943ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.358+0000 7f0a12ffd740 1 -- 172.21.15.31:0/1982799580 >> 172.21.15.31:0/1982799580 conn(0x55a41d9358a0 msgr2=0x55a41d93b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.358+0000 7f09f17fa640 0 client.16071 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.358+0000 7f0a12ffd740 1 -- 172.21.15.31:0/1982799580 shutdown_connections 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.358+0000 7f0a12ffd740 1 -- 172.21.15.31:0/1982799580 wait complete. 2024-04-25T01:56:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.358+0000 7f0a12ffd740 20 client.16071 trim_cache size 0 max 16384 2024-04-25T01:56:12.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a12ffd740 1 Processor -- start 2024-04-25T01:56:12.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a12ffd740 1 -- start start 2024-04-25T01:56:12.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 0x55a41d950220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 0x55a41d95be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d944570 0x55a41d95c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081af4c0 con 0x55a41d939580 2024-04-25T01:56:12.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081b10c0 con 0x55a41d939ea0 2024-04-25T01:56:12.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081afb90 con 0x55a41d944570 2024-04-25T01:56:12.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a0d7b6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 0x55a41d950220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a0dfb7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 0x55a41d95be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a0d7b6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 0x55a41d950220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46720/0 (socket says 172.21.15.31:46720) 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a0d7b6640 1 -- 172.21.15.31:0/2635522059 learned_addr learned my addr 172.21.15.31:0/2635522059 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a0dfb7640 1 -- 172.21.15.31:0/2635522059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d944570 msgr2=0x55a41d95c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a0dfb7640 1 --2- 172.21.15.31:0/2635522059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d944570 0x55a41d95c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a0dfb7640 1 -- 172.21.15.31:0/2635522059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 msgr2=0x55a41d950220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a0dfb7640 1 --2- 172.21.15.31:0/2635522059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 0x55a41d950220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a0dfb7640 1 -- 172.21.15.31:0/2635522059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a41d67c7d0 con 0x55a41d939ea0 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.375+0000 7f0a0dfb7640 1 --2- 172.21.15.31:0/2635522059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 0x55a41d95be70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55a41d956db0 tx=0x7f0a04005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9ea84fdbfdb3751 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f09f37fe640 1 -- 172.21.15.31:0/2635522059 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a040120d0 con 0x55a41d939ea0 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f09f37fe640 1 -- 172.21.15.31:0/2635522059 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a04012270 con 0x55a41d939ea0 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f09f37fe640 1 -- 172.21.15.31:0/2635522059 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a04005620 con 0x55a41d939ea0 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f0a12ffd740 1 -- 172.21.15.31:0/2635522059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 msgr2=0x55a41d95be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/2635522059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 0x55a41d95be70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55a41d956db0 tx=0x7f0a04005f90 comp rx=0 tx=0).stop 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f0a12ffd740 1 -- 172.21.15.31:0/2635522059 shutdown_connections 2024-04-25T01:56:12.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/2635522059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d944570 0x55a41d95c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/2635522059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939ea0 0x55a41d95be70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/2635522059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939580 0x55a41d950220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f0a12ffd740 1 -- 172.21.15.31:0/2635522059 >> 172.21.15.31:0/2635522059 conn(0x55a41d9358a0 msgr2=0x55a41d9342b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f0a12ffd740 1 -- 172.21.15.31:0/2635522059 shutdown_connections 2024-04-25T01:56:12.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.376+0000 7f0a12ffd740 1 -- 172.21.15.31:0/2635522059 wait complete. 2024-04-25T01:56:12.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a12ffd740 1 -- start start 2024-04-25T01:56:12.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a12ffd740 1 Processor -- start 2024-04-25T01:56:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939580 0x55a41d94c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939ea0 0x55a41d94cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a12ffd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d944570 0x55a41d949cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081b0930 con 0x55a41d939ea0 2024-04-25T01:56:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081b0260 con 0x55a41d939580 2024-04-25T01:56:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a12ffd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081ad4a0 con 0x55a41d944570 2024-04-25T01:56:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a12ffd740 10 client.0 _ll_register_callbacks cb 0x7f0a04009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a0dfb7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939ea0 0x55a41d94cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a0d7b6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939580 0x55a41d94c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a0dfb7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939ea0 0x55a41d94cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46726/0 (socket says 172.21.15.31:46726) 2024-04-25T01:56:12.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a0dfb7640 1 -- 172.21.15.31:0/67864195 learned_addr learned my addr 172.21.15.31:0/67864195 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.377+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d944570 0x55a41d949cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a0dfb7640 1 -- 172.21.15.31:0/67864195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d944570 msgr2=0x55a41d949cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a0dfb7640 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d944570 0x55a41d949cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a0dfb7640 1 -- 172.21.15.31:0/67864195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939580 msgr2=0x55a41d94c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a0dfb7640 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939580 0x55a41d94c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a0dfb7640 1 -- 172.21.15.31:0/67864195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a41d94a4c0 con 0x55a41d939ea0 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939580 0x55a41d94c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a0dfb7640 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939ea0 0x55a41d94cc70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0a04013000 tx=0x7f0a04012160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33cdf6575ed4f7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f09f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f09f17fa640 1 -- 172.21.15.31:0/67864195 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a04004250 con 0x55a41d939ea0 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f09f17fa640 1 -- 172.21.15.31:0/67864195 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f0a040043f0 con 0x55a41d939ea0 2024-04-25T01:56:12.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a12ffd740 1 -- 172.21.15.31:0/67864195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a04012270 con 0x55a41d939ea0 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f09f17fa640 1 -- 172.21.15.31:0/67864195 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0a0403a050 con 0x55a41d939ea0 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a12ffd740 10 client.16077 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a12ffd740 1 -- 172.21.15.31:0/67864195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a41d9ba330 con 0x55a41d939ea0 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f09f17fa640 1 -- 172.21.15.31:0/67864195 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a04012270 con 0x55a41d939ea0 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f09f17fa640 1 -- 172.21.15.31:0/67864195 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0a04022560 con 0x55a41d939ea0 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a12ffd740 10 client.16077 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a12ffd740 10 client.16077 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a12ffd740 20 client.16077 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.378+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a41d9baa80 0x55a41d9bcea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.379+0000 7f0a12ffd740 4 client.16077 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.379+0000 7f0a12ffd740 1 -- 172.21.15.31:0/67864195 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a41d8a5f00 con 0x55a41d9baa80 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.379+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a41d9baa80 0x55a41d9bcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.379+0000 7f09f17fa640 10 client.16077 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.379+0000 7f0a0d7b6640 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a41d9baa80 0x55a41d9bcea0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=71ca539ced473f59 server_cookie=dea605ecd1097261 in_seq=0 out_seq=0 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.379+0000 7f09f17fa640 1 -- 172.21.15.31:0/67864195 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a41d8a5f00 con 0x55a41d9baa80 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.379+0000 7f09f17fa640 10 client.16077 handle_command_reply: tid=0 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.379+0000 7f0a12ffd740 1 client.16077 shutdown 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.379+0000 7f0a12ffd740 1 -- 172.21.15.31:0/67864195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939ea0 msgr2=0x55a41d94cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.379+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939ea0 0x55a41d94cc70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0a04013000 tx=0x7f0a04012160 comp rx=0 tx=0).stop 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a12ffd740 1 -- 172.21.15.31:0/67864195 shutdown_connections 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55a41d9baa80 0x55a41d9bcea0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a41d944570 0x55a41d949cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f09f17fa640 0 client.16077 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a41d939ea0 0x55a41d94cc70 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a12ffd740 1 --2- 172.21.15.31:0/67864195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a41d939580 0x55a41d94c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a12ffd740 1 -- 172.21.15.31:0/67864195 >> 172.21.15.31:0/67864195 conn(0x55a41d9358a0 msgr2=0x55a41d934b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a12ffd740 1 -- 172.21.15.31:0/67864195 shutdown_connections 2024-04-25T01:56:12.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a12ffd740 1 -- 172.21.15.31:0/67864195 wait complete. 2024-04-25T01:56:12.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a12ffd740 20 client.16077 trim_cache size 0 max 16384 2024-04-25T01:56:12.389 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:56:12.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 -- 172.21.15.31:0/2451152916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09f40a2d00 msgr2=0x7f09f40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09f40a2d00 0x7f09f40a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0a00006010 tx=0x7f0a000095c0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 -- 172.21.15.31:0/2451152916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a0810bd00 msgr2=0x7f0a0819c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a0810bd00 0x7f0a0819c7a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0a0002f040 tx=0x7f0a0002fbe0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 -- 172.21.15.31:0/2451152916 shutdown_connections 2024-04-25T01:56:12.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09f40a2d00 0x7f09f40a51c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a0810bd00 0x7f0a0819c7a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a081032e0 0x7f0a08198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 --2- 172.21.15.31:0/2451152916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a081029c0 0x7f0a08197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 -- 172.21.15.31:0/2451152916 >> 172.21.15.31:0/2451152916 conn(0x7f0a08077740 msgr2=0x7f0a080fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 -- 172.21.15.31:0/2451152916 shutdown_connections 2024-04-25T01:56:12.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.380+0000 7f0a1021b640 1 -- 172.21.15.31:0/2451152916 wait complete. 2024-04-25T01:56:12.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:56:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.510+0000 7f941c714640 1 -- 172.21.15.31:0/2921820572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94141029c0 msgr2=0x7f9414102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.510+0000 7f941c714640 1 --2- 172.21.15.31:0/2921820572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94141029c0 0x7f9414102da0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f940c009ca0 tx=0x7f940c02a9e0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.510+0000 7f941c714640 1 -- 172.21.15.31:0/2921820572 shutdown_connections 2024-04-25T01:56:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.510+0000 7f941c714640 1 --2- 172.21.15.31:0/2921820572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f941410bd00 0x7f941410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.510+0000 7f941c714640 1 --2- 172.21.15.31:0/2921820572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94141032e0 0x7f941410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.510+0000 7f941c714640 1 --2- 172.21.15.31:0/2921820572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94141029c0 0x7f9414102da0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.510+0000 7f941c714640 1 -- 172.21.15.31:0/2921820572 >> 172.21.15.31:0/2921820572 conn(0x7f9414077740 msgr2=0x7f9414077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.510+0000 7f941c714640 1 -- 172.21.15.31:0/2921820572 shutdown_connections 2024-04-25T01:56:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.510+0000 7f941c714640 1 -- 172.21.15.31:0/2921820572 wait complete. 2024-04-25T01:56:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.511+0000 7f941c714640 1 Processor -- start 2024-04-25T01:56:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.511+0000 7f941c714640 1 -- start start 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.511+0000 7f941c714640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94141029c0 0x7f9414197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.511+0000 7f941c714640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94141032e0 0x7f9414198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.511+0000 7f941c714640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941410bd00 0x7f941419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.511+0000 7f941c714640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f94141119b0 con 0x7f94141032e0 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.511+0000 7f941c714640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9414111830 con 0x7f941410bd00 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.511+0000 7f941c714640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9414111b30 con 0x7f94141029c0 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.511+0000 7f9419caf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94141032e0 0x7f9414198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f941acb1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941410bd00 0x7f941419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f9419caf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94141032e0 0x7f9414198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46738/0 (socket says 172.21.15.31:46738) 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f9419caf640 1 -- 172.21.15.31:0/1315228784 learned_addr learned my addr 172.21.15.31:0/1315228784 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f941a4b0640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94141029c0 0x7f9414197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f9419caf640 1 -- 172.21.15.31:0/1315228784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94141029c0 msgr2=0x7f9414197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f9419caf640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94141029c0 0x7f9414197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f9419caf640 1 -- 172.21.15.31:0/1315228784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941410bd00 msgr2=0x7f941419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f9419caf640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941410bd00 0x7f941419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f9419caf640 1 -- 172.21.15.31:0/1315228784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941419ceb0 con 0x7f94141032e0 2024-04-25T01:56:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f941acb1640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941410bd00 0x7f941419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f9419caf640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94141032e0 0x7f9414198450 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f940800e9f0 tx=0x7f940800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f94137fe640 1 -- 172.21.15.31:0/1315228784 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940800cd80 con 0x7f94141032e0 2024-04-25T01:56:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f94137fe640 1 -- 172.21.15.31:0/1315228784 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94080108e0 con 0x7f94141032e0 2024-04-25T01:56:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f94137fe640 1 -- 172.21.15.31:0/1315228784 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408018920 con 0x7f94141032e0 2024-04-25T01:56:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f941c714640 1 -- 172.21.15.31:0/1315228784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f941419d1a0 con 0x7f94141032e0 2024-04-25T01:56:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.512+0000 7f941c714640 1 -- 172.21.15.31:0/1315228784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94141a4a30 con 0x7f94141032e0 2024-04-25T01:56:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.515+0000 7f94137fe640 1 -- 172.21.15.31:0/1315228784 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9408010a90 con 0x7f94141032e0 2024-04-25T01:56:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.515+0000 7f941f429740 1 Processor -- start 2024-04-25T01:56:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.515+0000 7f941f429740 1 -- start start 2024-04-25T01:56:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.515+0000 7f941f429740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 0x55ee37b10960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.515+0000 7f941f429740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 0x55ee37b1b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.515+0000 7f941f429740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b1b570 0x55ee37b1d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.515+0000 7f941f429740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ee37a75670 con 0x55ee37b10ea0 2024-04-25T01:56:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.515+0000 7f941f429740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ee37a1d970 con 0x55ee37b10580 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.515+0000 7f941f429740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ee37a71630 con 0x55ee37b1b570 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f941a4b0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 0x55ee37b1b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f941acb1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 0x55ee37b10960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f941acb1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 0x55ee37b10960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36550/0 (socket says 172.21.15.31:36550) 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f941acb1640 1 -- 172.21.15.31:0/2478383127 learned_addr learned my addr 172.21.15.31:0/2478383127 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f9419caf640 1 --2- 172.21.15.31:0/2478383127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b1b570 0x55ee37b1d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f941acb1640 1 -- 172.21.15.31:0/2478383127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b1b570 msgr2=0x55ee37b1d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f941acb1640 1 --2- 172.21.15.31:0/2478383127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b1b570 0x55ee37b1d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f94137fe640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93fc0a2d00 0x7f93fc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f941acb1640 1 -- 172.21.15.31:0/2478383127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 msgr2=0x55ee37b1b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f941acb1640 1 --2- 172.21.15.31:0/2478383127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 0x55ee37b1b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f941acb1640 1 -- 172.21.15.31:0/2478383127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee37a41400 con 0x55ee37b10580 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f941acb1640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93fc0a2d00 0x7f93fc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f94137fe640 1 -- 172.21.15.31:0/1315228784 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f94080cf6d0 con 0x7f94141032e0 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f941acb1640 1 --2- 172.21.15.31:0/2478383127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 0x55ee37b10960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f940400af90 tx=0x7f940400f850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c832c417d910064 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f9410ff9640 1 -- 172.21.15.31:0/2478383127 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9404010b20 con 0x55ee37b10580 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f9410ff9640 1 -- 172.21.15.31:0/2478383127 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9404010d40 con 0x55ee37b10580 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.516+0000 7f9410ff9640 1 -- 172.21.15.31:0/2478383127 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9404017890 con 0x55ee37b10580 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.517+0000 7f941f429740 1 -- 172.21.15.31:0/2478383127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 msgr2=0x55ee37b10960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.517+0000 7f941acb1640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93fc0a2d00 0x7f93fc0a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f940401c000 tx=0x7f9404017ee0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.517+0000 7f941f429740 1 --2- 172.21.15.31:0/2478383127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 0x55ee37b10960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f940400af90 tx=0x7f940400f850 comp rx=0 tx=0).stop 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.517+0000 7f941f429740 1 -- 172.21.15.31:0/2478383127 shutdown_connections 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.517+0000 7f941f429740 1 --2- 172.21.15.31:0/2478383127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b1b570 0x55ee37b1d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.517+0000 7f941f429740 1 --2- 172.21.15.31:0/2478383127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 0x55ee37b1b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.517+0000 7f941f429740 1 --2- 172.21.15.31:0/2478383127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 0x55ee37b10960 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.517+0000 7f941f429740 1 -- 172.21.15.31:0/2478383127 >> 172.21.15.31:0/2478383127 conn(0x55ee37b0c8a0 msgr2=0x55ee37b0cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.517+0000 7f941f429740 1 -- 172.21.15.31:0/2478383127 shutdown_connections 2024-04-25T01:56:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.517+0000 7f941f429740 1 -- 172.21.15.31:0/2478383127 wait complete. 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941f429740 1 -- start start 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941f429740 1 Processor -- start 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941f429740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10580 0x55ee37b1aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941f429740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10ea0 0x55ee37b181a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941f429740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 0x55ee37b186e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941f429740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ee37b1e300 con 0x55ee37b10580 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941f429740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ee37a31050 con 0x55ee37b1b570 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941f429740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ee37b1e480 con 0x55ee37b10ea0 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941f429740 10 client.0 _ll_register_callbacks cb 0x55ee37788be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941a4b0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10580 0x55ee37b1aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941acb1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 0x55ee37b186e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941a4b0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10580 0x55ee37b1aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46756/0 (socket says 172.21.15.31:46756) 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f941a4b0640 1 -- 172.21.15.31:0/1478238222 learned_addr learned my addr 172.21.15.31:0/1478238222 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.518+0000 7f9419caf640 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10ea0 0x55ee37b181a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.519+0000 7f941a4b0640 1 -- 172.21.15.31:0/1478238222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10ea0 msgr2=0x55ee37b181a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.519+0000 7f941a4b0640 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10ea0 0x55ee37b181a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.519+0000 7f941a4b0640 1 -- 172.21.15.31:0/1478238222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 msgr2=0x55ee37b186e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.519+0000 7f941a4b0640 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 0x55ee37b186e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.519+0000 7f941a4b0640 1 -- 172.21.15.31:0/1478238222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee37a02dc0 con 0x55ee37b10580 2024-04-25T01:56:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.519+0000 7f941a4b0640 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10580 0x55ee37b1aeb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f940c035000 tx=0x7f940c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c426b0f2827d7e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.519+0000 7f93f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.519+0000 7f93f9ffb640 1 -- 172.21.15.31:0/1478238222 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c033040 con 0x55ee37b10580 2024-04-25T01:56:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.519+0000 7f93f9ffb640 1 -- 172.21.15.31:0/1478238222 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f940c02e590 con 0x55ee37b10580 2024-04-25T01:56:12.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.519+0000 7f941f429740 1 -- 172.21.15.31:0/1478238222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9404010d40 con 0x55ee37b10580 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f93f9ffb640 1 -- 172.21.15.31:0/1478238222 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f940c02e790 con 0x55ee37b10580 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f941f429740 10 client.16089 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f941f429740 1 -- 172.21.15.31:0/1478238222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ee377893f0 con 0x55ee37b10580 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f93f9ffb640 1 -- 172.21.15.31:0/1478238222 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9404010d40 con 0x55ee37b10580 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f93f9ffb640 1 -- 172.21.15.31:0/1478238222 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f940c02f870 con 0x55ee37b10580 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f941f429740 10 client.16089 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f941f429740 10 client.16089 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f941f429740 20 client.16089 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f941f429740 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ee37b2f8a0 0x55ee37b31d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f941f429740 4 client.16089 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f941f429740 1 -- 172.21.15.31:0/1478238222 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee37a82bb0 con 0x55ee37b2f8a0 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.520+0000 7f9419caf640 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ee37b2f8a0 0x55ee37b31d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f9419caf640 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ee37b2f8a0 0x55ee37b31d40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b70bf9f90a6375ed server_cookie=4f7da70a46ce00ac in_seq=0 out_seq=0 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f93f9ffb640 10 client.16089 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f93f9ffb640 1 -- 172.21.15.31:0/1478238222 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee37a82bb0 con 0x55ee37b2f8a0 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f93f9ffb640 10 client.16089 handle_command_reply: tid=0 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f941f429740 1 client.16089 shutdown 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f941f429740 1 -- 172.21.15.31:0/1478238222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10580 msgr2=0x55ee37b1aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f941f429740 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10580 0x55ee37b1aeb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f940c035000 tx=0x7f940c003ba0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f941f429740 1 -- 172.21.15.31:0/1478238222 shutdown_connections 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f941f429740 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ee37b2f8a0 0x55ee37b31d40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f941f429740 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 0x55ee37b186e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f941f429740 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10ea0 0x55ee37b181a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.521+0000 7f93f9ffb640 0 client.16089 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.522+0000 7f941f429740 1 --2- 172.21.15.31:0/1478238222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10580 0x55ee37b1aeb0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.522+0000 7f941f429740 1 -- 172.21.15.31:0/1478238222 >> 172.21.15.31:0/1478238222 conn(0x55ee37b0c8a0 msgr2=0x55ee37b126a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.522+0000 7f941f429740 1 -- 172.21.15.31:0/1478238222 shutdown_connections 2024-04-25T01:56:12.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.522+0000 7f941f429740 1 -- 172.21.15.31:0/1478238222 wait complete. 2024-04-25T01:56:12.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.522+0000 7f941f429740 20 client.16089 trim_cache size 0 max 16384 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.537+0000 7f941f429740 1 Processor -- start 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.537+0000 7f941f429740 1 -- start start 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941f429740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10580 0x55ee37b271c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941f429740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 0x55ee37b1a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941f429740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 0x55ee37b37660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941f429740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f94141ab100 con 0x55ee37b10ea0 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941f429740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94141acd00 con 0x55ee37b1b570 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941f429740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f94141ab7d0 con 0x55ee37b10580 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f9419caf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 0x55ee37b1a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941acb1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 0x55ee37b37660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f9419caf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 0x55ee37b1a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46766/0 (socket says 172.21.15.31:46766) 2024-04-25T01:56:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f9419caf640 1 -- 172.21.15.31:0/986274458 learned_addr learned my addr 172.21.15.31:0/986274458 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941acb1640 1 --2- 172.21.15.31:0/986274458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 0x55ee37b37660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36570/0 (socket says 172.21.15.31:36570) 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941a4b0640 1 --2- 172.21.15.31:0/986274458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10580 0x55ee37b271c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941acb1640 1 -- 172.21.15.31:0/986274458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10580 msgr2=0x55ee37b271c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941acb1640 1 --2- 172.21.15.31:0/986274458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10580 0x55ee37b271c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941acb1640 1 -- 172.21.15.31:0/986274458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 msgr2=0x55ee37b1a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941acb1640 1 --2- 172.21.15.31:0/986274458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 0x55ee37b1a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941acb1640 1 -- 172.21.15.31:0/986274458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee378537d0 con 0x55ee37b1b570 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f9419caf640 1 --2- 172.21.15.31:0/986274458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 0x55ee37b1a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.538+0000 7f941acb1640 1 --2- 172.21.15.31:0/986274458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 0x55ee37b37660 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55ee37b11a00 tx=0x7f9404027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=937948d931587c8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f93fbfff640 1 -- 172.21.15.31:0/986274458 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940400aaf0 con 0x55ee37b1b570 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f93fbfff640 1 -- 172.21.15.31:0/986274458 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9404017860 con 0x55ee37b1b570 2024-04-25T01:56:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f93fbfff640 1 -- 172.21.15.31:0/986274458 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9404021c00 con 0x55ee37b1b570 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f941f429740 1 -- 172.21.15.31:0/986274458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 msgr2=0x55ee37b37660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f941f429740 1 --2- 172.21.15.31:0/986274458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 0x55ee37b37660 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55ee37b11a00 tx=0x7f9404027000 comp rx=0 tx=0).stop 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f941f429740 1 -- 172.21.15.31:0/986274458 shutdown_connections 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f941f429740 1 --2- 172.21.15.31:0/986274458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b1b570 0x55ee37b37660 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f941f429740 1 --2- 172.21.15.31:0/986274458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b10ea0 0x55ee37b1a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f941f429740 1 --2- 172.21.15.31:0/986274458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10580 0x55ee37b271c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f941f429740 1 -- 172.21.15.31:0/986274458 >> 172.21.15.31:0/986274458 conn(0x55ee37b0c8a0 msgr2=0x55ee37b13740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f941f429740 1 -- 172.21.15.31:0/986274458 shutdown_connections 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.539+0000 7f941f429740 1 -- 172.21.15.31:0/986274458 wait complete. 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941f429740 1 -- start start 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941f429740 1 Processor -- start 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941f429740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 0x55ee37b23930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941f429740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10ea0 0x55ee37b1ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941f429740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b1b570 0x55ee37b1f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941f429740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f94141ac570 con 0x55ee37b1b570 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941f429740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94141abea0 con 0x55ee37b10580 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941f429740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f94141a90e0 con 0x55ee37b10ea0 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941f429740 10 client.0 _ll_register_callbacks cb 0x7f940c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941acb1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b1b570 0x55ee37b1f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941a4b0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 0x55ee37b23930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941acb1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b1b570 0x55ee37b1f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46768/0 (socket says 172.21.15.31:46768) 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941acb1640 1 -- 172.21.15.31:0/543979195 learned_addr learned my addr 172.21.15.31:0/543979195 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f9419caf640 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10ea0 0x55ee37b1ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941acb1640 1 -- 172.21.15.31:0/543979195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10ea0 msgr2=0x55ee37b1ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941acb1640 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10ea0 0x55ee37b1ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941acb1640 1 -- 172.21.15.31:0/543979195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 msgr2=0x55ee37b23930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941acb1640 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 0x55ee37b23930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f941acb1640 1 -- 172.21.15.31:0/543979195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee37b1f8d0 con 0x55ee37b1b570 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.540+0000 7f9419caf640 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10ea0 0x55ee37b1ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f941acb1640 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b1b570 0x55ee37b1f230 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f940400f890 tx=0x7f940400fe00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2421eed5cf443c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f93f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f93f9ffb640 1 -- 172.21.15.31:0/543979195 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94040175c0 con 0x55ee37b1b570 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f93f9ffb640 1 -- 172.21.15.31:0/543979195 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f940401e040 con 0x55ee37b1b570 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f941f429740 1 -- 172.21.15.31:0/543979195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9404017860 con 0x55ee37b1b570 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f93f9ffb640 1 -- 172.21.15.31:0/543979195 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9404004020 con 0x55ee37b1b570 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f941f429740 10 client.16095 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f941f429740 1 -- 172.21.15.31:0/543979195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ee37b95d60 con 0x55ee37b1b570 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f93f9ffb640 1 -- 172.21.15.31:0/543979195 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9404017860 con 0x55ee37b1b570 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f93f9ffb640 1 -- 172.21.15.31:0/543979195 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f94040035f0 con 0x55ee37b1b570 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f941f429740 10 client.16095 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f941f429740 10 client.16095 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f941f429740 20 client.16095 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f941f429740 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ee37b96260 0x55ee37b98680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f941f429740 4 client.16095 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.541+0000 7f941f429740 1 -- 172.21.15.31:0/543979195 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee37a7cf00 con 0x55ee37b96260 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941a4b0640 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ee37b96260 0x55ee37b98680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941a4b0640 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ee37b96260 0x55ee37b98680 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=39e5f3c9683c25a server_cookie=23becddd7880827e in_seq=0 out_seq=0 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f93f9ffb640 10 client.16095 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f93f9ffb640 1 -- 172.21.15.31:0/543979195 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee37a7cf00 con 0x55ee37b96260 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f93f9ffb640 10 client.16095 handle_command_reply: tid=0 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 1 client.16095 shutdown 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 1 -- 172.21.15.31:0/543979195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b1b570 msgr2=0x55ee37b1f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b1b570 0x55ee37b1f230 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f940400f890 tx=0x7f940400fe00 comp rx=0 tx=0).stop 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 1 -- 172.21.15.31:0/543979195 shutdown_connections 2024-04-25T01:56:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55ee37b96260 0x55ee37b98680 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee37b1b570 0x55ee37b1f230 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee37b10ea0 0x55ee37b1ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 1 --2- 172.21.15.31:0/543979195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee37b10580 0x55ee37b23930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 1 -- 172.21.15.31:0/543979195 >> 172.21.15.31:0/543979195 conn(0x55ee37b0c8a0 msgr2=0x55ee37b11cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f93f9ffb640 0 client.16095 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:56:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 1 -- 172.21.15.31:0/543979195 shutdown_connections 2024-04-25T01:56:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 1 -- 172.21.15.31:0/543979195 wait complete. 2024-04-25T01:56:12.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.542+0000 7f941f429740 20 client.16095 trim_cache size 0 max 16384 2024-04-25T01:56:12.551 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:56:12.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 -- 172.21.15.31:0/1315228784 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93fc0a2d00 msgr2=0x7f93fc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93fc0a2d00 0x7f93fc0a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f940401c000 tx=0x7f9404017ee0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 -- 172.21.15.31:0/1315228784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94141032e0 msgr2=0x7f9414198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94141032e0 0x7f9414198450 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f940800e9f0 tx=0x7f940800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 -- 172.21.15.31:0/1315228784 shutdown_connections 2024-04-25T01:56:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f93fc0a2d00 0x7f93fc0a51c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941410bd00 0x7f941419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94141032e0 0x7f9414198450 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 --2- 172.21.15.31:0/1315228784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94141029c0 0x7f9414197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 -- 172.21.15.31:0/1315228784 >> 172.21.15.31:0/1315228784 conn(0x7f9414077740 msgr2=0x7f94140fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 -- 172.21.15.31:0/1315228784 shutdown_connections 2024-04-25T01:56:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.543+0000 7f941c714640 1 -- 172.21.15.31:0/1315228784 wait complete. 2024-04-25T01:56:12.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:56:12.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.666+0000 7f4913529640 1 -- 172.21.15.31:0/352528014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490c101710 msgr2=0x7f490c101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.666+0000 7f4913529640 1 --2- 172.21.15.31:0/352528014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490c101710 0x7f490c101b90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009930 tx=0x7f48fc02a420 comp rx=0 tx=0).stop 2024-04-25T01:56:12.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.666+0000 7f4913529640 1 -- 172.21.15.31:0/352528014 shutdown_connections 2024-04-25T01:56:12.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.666+0000 7f4913529640 1 --2- 172.21.15.31:0/352528014 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f490c1020d0 0x7f490c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.666+0000 7f4913529640 1 --2- 172.21.15.31:0/352528014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490c101710 0x7f490c101b90 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.666+0000 7f4913529640 1 --2- 172.21.15.31:0/352528014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c107720 0x7f490c107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.666+0000 7f4913529640 1 -- 172.21.15.31:0/352528014 >> 172.21.15.31:0/352528014 conn(0x7f490c078b90 msgr2=0x7f490c078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.666+0000 7f4913529640 1 -- 172.21.15.31:0/352528014 shutdown_connections 2024-04-25T01:56:12.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.666+0000 7f4913529640 1 -- 172.21.15.31:0/352528014 wait complete. 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4913529640 1 Processor -- start 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4913529640 1 -- start start 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4913529640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c101710 0x7f490c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4913529640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490c1020d0 0x7f490c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4913529640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f490c107720 0x7f490c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4913529640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1119e0 con 0x7f490c1020d0 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4913529640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f490c111860 con 0x7f490c101710 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4913529640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f490c111b60 con 0x7f490c107720 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4910ac4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490c1020d0 0x7f490c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4910ac4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490c1020d0 0x7f490c1984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47512/0 (socket says 172.21.15.31:47512) 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4910ac4640 1 -- 172.21.15.31:0/97049013 learned_addr learned my addr 172.21.15.31:0/97049013 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f49112c5640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c101710 0x7f490c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4910ac4640 1 -- 172.21.15.31:0/97049013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f490c107720 msgr2=0x7f490c19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4910ac4640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f490c107720 0x7f490c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4910ac4640 1 -- 172.21.15.31:0/97049013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c101710 msgr2=0x7f490c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4910ac4640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c101710 0x7f490c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.667+0000 7f4910ac4640 1 -- 172.21.15.31:0/97049013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490c19cf80 con 0x7f490c1020d0 2024-04-25T01:56:12.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.668+0000 7f4910ac4640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490c1020d0 0x7f490c1984d0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f48fc005940 tx=0x7f48fc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.668+0000 7f49027fc640 1 -- 172.21.15.31:0/97049013 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc03f030 con 0x7f490c1020d0 2024-04-25T01:56:12.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.668+0000 7f4913529640 1 -- 172.21.15.31:0/97049013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f490c19d210 con 0x7f490c1020d0 2024-04-25T01:56:12.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.668+0000 7f49027fc640 1 -- 172.21.15.31:0/97049013 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48fc0044c0 con 0x7f490c1020d0 2024-04-25T01:56:12.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.668+0000 7f49027fc640 1 -- 172.21.15.31:0/97049013 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc035070 con 0x7f490c1020d0 2024-04-25T01:56:12.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.668+0000 7f4913529640 1 -- 172.21.15.31:0/97049013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f490c1a4b00 con 0x7f490c1020d0 2024-04-25T01:56:12.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.671+0000 7f49027fc640 1 -- 172.21.15.31:0/97049013 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f48fc042030 con 0x7f490c1020d0 2024-04-25T01:56:12.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.671+0000 7f491623b740 1 Processor -- start 2024-04-25T01:56:12.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.671+0000 7f491623b740 1 -- start start 2024-04-25T01:56:12.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.671+0000 7f491623b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb41960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f491623b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb4c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f491623b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb4c570 0x55bb3fb4e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f491623b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bb3faa6670 con 0x55bb3fb4c570 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f491623b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bb3fa4e970 con 0x55bb3fb41580 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f491623b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bb3faa2630 con 0x55bb3fb41ea0 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f4911ac6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb41960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f4910ac4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb4c570 0x55bb3fb4e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f4911ac6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb41960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44490/0 (socket says 172.21.15.31:44490) 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f4911ac6640 1 -- 172.21.15.31:0/2929123175 learned_addr learned my addr 172.21.15.31:0/2929123175 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f49027fc640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48f80a2d00 0x7f48f80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f49112c5640 1 --2- 172.21.15.31:0/2929123175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb4c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f4911ac6640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48f80a2d00 0x7f48f80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f4911ac6640 1 -- 172.21.15.31:0/2929123175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb41ea0 msgr2=0x55bb3fb4c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f4911ac6640 1 --2- 172.21.15.31:0/2929123175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb4c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f4911ac6640 1 -- 172.21.15.31:0/2929123175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb4c570 msgr2=0x55bb3fb4e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f4911ac6640 1 --2- 172.21.15.31:0/2929123175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb4c570 0x55bb3fb4e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f4911ac6640 1 -- 172.21.15.31:0/2929123175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb3fa72400 con 0x55bb3fb41580 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f49027fc640 1 -- 172.21.15.31:0/97049013 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f48fc0ef750 con 0x7f490c1020d0 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.672+0000 7f4911ac6640 1 --2- 172.21.15.31:0/2929123175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb41960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f49040128e0 tx=0x7f4904012da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cebd215f0ac2a703 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.673+0000 7f4911ac6640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48f80a2d00 0x7f48f80a51c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4904010740 tx=0x7f490401c000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.673+0000 7f48f77fe640 1 -- 172.21.15.31:0/2929123175 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490401b7a0 con 0x55bb3fb41580 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.673+0000 7f48f77fe640 1 -- 172.21.15.31:0/2929123175 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4904020030 con 0x55bb3fb41580 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.673+0000 7f48f77fe640 1 -- 172.21.15.31:0/2929123175 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490401cae0 con 0x55bb3fb41580 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.673+0000 7f491623b740 1 -- 172.21.15.31:0/2929123175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 msgr2=0x55bb3fb41960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.673+0000 7f491623b740 1 --2- 172.21.15.31:0/2929123175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb41960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f49040128e0 tx=0x7f4904012da0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.674+0000 7f491623b740 1 -- 172.21.15.31:0/2929123175 shutdown_connections 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.674+0000 7f491623b740 1 --2- 172.21.15.31:0/2929123175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb4c570 0x55bb3fb4e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.674+0000 7f491623b740 1 --2- 172.21.15.31:0/2929123175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb4c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.674+0000 7f491623b740 1 --2- 172.21.15.31:0/2929123175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb41960 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.674+0000 7f491623b740 1 -- 172.21.15.31:0/2929123175 >> 172.21.15.31:0/2929123175 conn(0x55bb3fb3d8a0 msgr2=0x55bb3fb3dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.674+0000 7f491623b740 1 -- 172.21.15.31:0/2929123175 shutdown_connections 2024-04-25T01:56:12.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.674+0000 7f491623b740 1 -- 172.21.15.31:0/2929123175 wait complete. 2024-04-25T01:56:12.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.674+0000 7f491623b740 1 -- start start 2024-04-25T01:56:12.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.674+0000 7f491623b740 1 Processor -- start 2024-04-25T01:56:12.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f491623b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb4bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f491623b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb49250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f491623b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 0x55bb3fb49860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f491623b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bb3fb4f300 con 0x55bb3fb41580 2024-04-25T01:56:12.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f491623b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bb3fa62050 con 0x55bb3fb41ea0 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f491623b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bb3fb4f480 con 0x55bb3fb4c570 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f491623b740 10 client.0 _ll_register_callbacks cb 0x55bb3f7b9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f49112c5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb4bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f4910ac4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb49250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f49112c5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb4bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47522/0 (socket says 172.21.15.31:47522) 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f49112c5640 1 -- 172.21.15.31:0/1897523118 learned_addr learned my addr 172.21.15.31:0/1897523118 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f4911ac6640 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 0x55bb3fb49860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f49112c5640 1 -- 172.21.15.31:0/1897523118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 msgr2=0x55bb3fb49860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f49112c5640 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 0x55bb3fb49860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f49112c5640 1 -- 172.21.15.31:0/1897523118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41ea0 msgr2=0x55bb3fb49250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f49112c5640 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb49250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f49112c5640 1 -- 172.21.15.31:0/1897523118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb3fa33dc0 con 0x55bb3fb41580 2024-04-25T01:56:12.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f4910ac4640 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb49250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f49112c5640 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb4bf60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f49080126d0 tx=0x7f4908012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165d407caef4f3ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f48f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1897523118 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490800f9e0 con 0x55bb3fb41580 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1897523118 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f490800fb80 con 0x55bb3fb41580 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.675+0000 7f491623b740 1 -- 172.21.15.31:0/1897523118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4904020030 con 0x55bb3fb41580 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1897523118 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f490800fd80 con 0x55bb3fb41580 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f491623b740 10 client.16107 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f491623b740 1 -- 172.21.15.31:0/1897523118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bb3f7ba3f0 con 0x55bb3fb41580 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1897523118 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4904020030 con 0x55bb3fb41580 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1897523118 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f490801c420 con 0x55bb3fb41580 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f491623b740 10 client.16107 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f491623b740 10 client.16107 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f491623b740 20 client.16107 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:12.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f491623b740 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bb3fb5c640 0x55bb3fb5eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f491623b740 4 client.16107 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f491623b740 1 -- 172.21.15.31:0/1897523118 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb3fab3bb0 con 0x55bb3fb5c640 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.676+0000 7f4910ac4640 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bb3fb5c640 0x55bb3fb5eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.677+0000 7f4910ac4640 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bb3fb5c640 0x55bb3fb5eac0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=de8a0078f226896 server_cookie=dc651bca72761a98 in_seq=0 out_seq=0 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.677+0000 7f48f4ff9640 10 client.16107 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.677+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1897523118 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb3fab3bb0 con 0x55bb3fb5c640 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.677+0000 7f48f4ff9640 10 client.16107 handle_command_reply: tid=0 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.677+0000 7f491623b740 1 client.16107 shutdown 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f491623b740 1 -- 172.21.15.31:0/1897523118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 msgr2=0x55bb3fb4bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f491623b740 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb4bf60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f49080126d0 tx=0x7f4908012b90 comp rx=0 tx=0).stop 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f491623b740 1 -- 172.21.15.31:0/1897523118 shutdown_connections 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f491623b740 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bb3fb5c640 0x55bb3fb5eac0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f491623b740 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 0x55bb3fb49860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f491623b740 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb49250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f491623b740 1 --2- 172.21.15.31:0/1897523118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb4bf60 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f491623b740 1 -- 172.21.15.31:0/1897523118 >> 172.21.15.31:0/1897523118 conn(0x55bb3fb3d8a0 msgr2=0x55bb3fb436a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f48f4ff9640 0 client.16107 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f491623b740 1 -- 172.21.15.31:0/1897523118 shutdown_connections 2024-04-25T01:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f491623b740 1 -- 172.21.15.31:0/1897523118 wait complete. 2024-04-25T01:56:12.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.678+0000 7f491623b740 20 client.16107 trim_cache size 0 max 16384 2024-04-25T01:56:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.692+0000 7f491623b740 1 Processor -- start 2024-04-25T01:56:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.692+0000 7f491623b740 1 -- start start 2024-04-25T01:56:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.692+0000 7f491623b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb581c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.692+0000 7f491623b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb483b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.692+0000 7f491623b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 0x55bb3fb4ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.692+0000 7f491623b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1ad370 con 0x55bb3fb41580 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.692+0000 7f491623b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1aef70 con 0x55bb3fb41ea0 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.692+0000 7f491623b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1ada40 con 0x55bb3fb4c570 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f49112c5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f49112c5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47530/0 (socket says 172.21.15.31:47530) 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f49112c5640 1 -- 172.21.15.31:0/3467830088 learned_addr learned my addr 172.21.15.31:0/3467830088 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f4910ac4640 1 --2- 172.21.15.31:0/3467830088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f49112c5640 1 -- 172.21.15.31:0/3467830088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 msgr2=0x55bb3fb4ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f49112c5640 1 --2- 172.21.15.31:0/3467830088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 0x55bb3fb4ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f49112c5640 1 -- 172.21.15.31:0/3467830088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41ea0 msgr2=0x55bb3fb483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f49112c5640 1 --2- 172.21.15.31:0/3467830088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f49112c5640 1 -- 172.21.15.31:0/3467830088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb3fb41cd0 con 0x55bb3fb41580 2024-04-25T01:56:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f49112c5640 1 --2- 172.21.15.31:0/3467830088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb581c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f490801ca80 tx=0x7f490800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603c91ead02ed83e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f48f6ffd640 1 -- 172.21.15.31:0/3467830088 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908029030 con 0x55bb3fb41580 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f48f6ffd640 1 -- 172.21.15.31:0/3467830088 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f490801cb00 con 0x55bb3fb41580 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.693+0000 7f48f6ffd640 1 -- 172.21.15.31:0/3467830088 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490801c420 con 0x55bb3fb41580 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.694+0000 7f491623b740 1 -- 172.21.15.31:0/3467830088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 msgr2=0x55bb3fb581c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.694+0000 7f491623b740 1 --2- 172.21.15.31:0/3467830088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb581c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f490801ca80 tx=0x7f490800f790 comp rx=0 tx=0).stop 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.694+0000 7f491623b740 1 -- 172.21.15.31:0/3467830088 shutdown_connections 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.694+0000 7f491623b740 1 --2- 172.21.15.31:0/3467830088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 0x55bb3fb4ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.694+0000 7f491623b740 1 --2- 172.21.15.31:0/3467830088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.694+0000 7f491623b740 1 --2- 172.21.15.31:0/3467830088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41580 0x55bb3fb581c0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.694+0000 7f491623b740 1 -- 172.21.15.31:0/3467830088 >> 172.21.15.31:0/3467830088 conn(0x55bb3fb3d8a0 msgr2=0x55bb3fb42d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.694+0000 7f491623b740 1 -- 172.21.15.31:0/3467830088 shutdown_connections 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.694+0000 7f491623b740 1 -- 172.21.15.31:0/3467830088 wait complete. 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.694+0000 7f491623b740 1 -- start start 2024-04-25T01:56:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.694+0000 7f491623b740 1 Processor -- start 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f491623b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb4fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f491623b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb501f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f491623b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 0x55bb3fb54120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f491623b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1ae7e0 con 0x55bb3fb41ea0 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f491623b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1ae110 con 0x55bb3fb41580 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f491623b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f490c1ab350 con 0x55bb3fb4c570 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f491623b740 10 client.0 _ll_register_callbacks cb 0x7f49080097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f4910ac4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f49112c5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb4fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f49112c5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb4fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44516/0 (socket says 172.21.15.31:44516) 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f4910ac4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb501f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47534/0 (socket says 172.21.15.31:47534) 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f49112c5640 1 -- 172.21.15.31:0/1442321268 learned_addr learned my addr 172.21.15.31:0/1442321268 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f4911ac6640 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 0x55bb3fb54120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f49112c5640 1 -- 172.21.15.31:0/1442321268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 msgr2=0x55bb3fb54120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f49112c5640 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 0x55bb3fb54120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f49112c5640 1 -- 172.21.15.31:0/1442321268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41ea0 msgr2=0x55bb3fb501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f49112c5640 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f49112c5640 1 -- 172.21.15.31:0/1442321268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb3f8847d0 con 0x55bb3fb41580 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.695+0000 7f4910ac4640 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f49112c5640 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb4fcb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f49080126d0 tx=0x7f4908005300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30363a672f904b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f48f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1442321268 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908031070 con 0x55bb3fb41580 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1442321268 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f49080054f0 con 0x55bb3fb41580 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f491623b740 1 -- 172.21.15.31:0/1442321268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f490801cb00 con 0x55bb3fb41580 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1442321268 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4908029040 con 0x55bb3fb41580 2024-04-25T01:56:12.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f491623b740 10 client.25718 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f491623b740 1 -- 172.21.15.31:0/1442321268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bb3fbc2590 con 0x55bb3fb41580 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1442321268 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f490801cb00 con 0x55bb3fb41580 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1442321268 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4908003ba0 con 0x55bb3fb41580 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f491623b740 10 client.25718 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f491623b740 10 client.25718 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f491623b740 20 client.25718 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f491623b740 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bb3fbc2bd0 0x55bb3fbc4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f491623b740 4 client.25718 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.696+0000 7f491623b740 1 -- 172.21.15.31:0/1442321268 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb3faadf00 con 0x55bb3fbc2bd0 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f4910ac4640 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bb3fbc2bd0 0x55bb3fbc4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f48f4ff9640 10 client.25718 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f4910ac4640 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bb3fbc2bd0 0x55bb3fbc4ff0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ba4e125a4df869f4 server_cookie=7d285f45566de337 in_seq=0 out_seq=0 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f48f4ff9640 1 -- 172.21.15.31:0/1442321268 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb3faadf00 con 0x55bb3fbc2bd0 2024-04-25T01:56:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f48f4ff9640 10 client.25718 handle_command_reply: tid=0 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f491623b740 1 client.25718 shutdown 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f491623b740 1 -- 172.21.15.31:0/1442321268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 msgr2=0x55bb3fb4fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f491623b740 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb4fcb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f49080126d0 tx=0x7f4908005300 comp rx=0 tx=0).stop 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f491623b740 1 -- 172.21.15.31:0/1442321268 shutdown_connections 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f491623b740 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x55bb3fbc2bd0 0x55bb3fbc4ff0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f491623b740 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb3fb4c570 0x55bb3fb54120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f491623b740 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb3fb41ea0 0x55bb3fb501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f48f4ff9640 0 client.25718 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f491623b740 1 --2- 172.21.15.31:0/1442321268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bb3fb41580 0x55bb3fb4fcb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.697+0000 7f491623b740 1 -- 172.21.15.31:0/1442321268 >> 172.21.15.31:0/1442321268 conn(0x55bb3fb3d8a0 msgr2=0x55bb3fb43eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f491623b740 1 -- 172.21.15.31:0/1442321268 shutdown_connections 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f491623b740 1 -- 172.21.15.31:0/1442321268 wait complete. 2024-04-25T01:56:12.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f491623b740 20 client.25718 trim_cache size 0 max 16384 2024-04-25T01:56:12.707 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:56:12.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 -- 172.21.15.31:0/97049013 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48f80a2d00 msgr2=0x7f48f80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48f80a2d00 0x7f48f80a51c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4904010740 tx=0x7f490401c000 comp rx=0 tx=0).stop 2024-04-25T01:56:12.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 -- 172.21.15.31:0/97049013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490c1020d0 msgr2=0x7f490c1984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490c1020d0 0x7f490c1984d0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f48fc005940 tx=0x7f48fc0040c0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 -- 172.21.15.31:0/97049013 shutdown_connections 2024-04-25T01:56:12.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f48f80a2d00 0x7f48f80a51c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f490c107720 0x7f490c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490c1020d0 0x7f490c1984d0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 --2- 172.21.15.31:0/97049013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c101710 0x7f490c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 -- 172.21.15.31:0/97049013 >> 172.21.15.31:0/97049013 conn(0x7f490c078b90 msgr2=0x7f490c0ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 -- 172.21.15.31:0/97049013 shutdown_connections 2024-04-25T01:56:12.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.698+0000 7f4913529640 1 -- 172.21.15.31:0/97049013 wait complete. 2024-04-25T01:56:12.712 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:56:12.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.830+0000 7f3ae9709640 1 -- 172.21.15.31:0/3003480279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ae41066b0 msgr2=0x7f3ae4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.830+0000 7f3ae9709640 1 --2- 172.21.15.31:0/3003480279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ae41066b0 0x7f3ae4106a90 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009930 tx=0x7f3ad402a420 comp rx=0 tx=0).stop 2024-04-25T01:56:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.830+0000 7f3ae9709640 1 -- 172.21.15.31:0/3003480279 shutdown_connections 2024-04-25T01:56:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.830+0000 7f3ae9709640 1 --2- 172.21.15.31:0/3003480279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ae4100fd0 0x7f3ae410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.830+0000 7f3ae9709640 1 --2- 172.21.15.31:0/3003480279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae4100610 0x7f3ae4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.830+0000 7f3ae9709640 1 --2- 172.21.15.31:0/3003480279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ae41066b0 0x7f3ae4106a90 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.830+0000 7f3ae9709640 1 -- 172.21.15.31:0/3003480279 >> 172.21.15.31:0/3003480279 conn(0x7f3ae4077740 msgr2=0x7f3ae4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.830+0000 7f3ae9709640 1 -- 172.21.15.31:0/3003480279 shutdown_connections 2024-04-25T01:56:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.830+0000 7f3ae9709640 1 -- 172.21.15.31:0/3003480279 wait complete. 2024-04-25T01:56:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3ae9709640 1 Processor -- start 2024-04-25T01:56:12.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3ae9709640 1 -- start start 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3ae9709640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ae4100610 0x7f3ae4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3ae9709640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ae4100fd0 0x7f3ae4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3ae9709640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae41066b0 0x7f3ae419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3ae9709640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae41119e0 con 0x7f3ae4100fd0 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3ae9709640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae4111860 con 0x7f3ae4100610 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3ae9709640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae4111b60 con 0x7f3ae41066b0 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3adbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ae4100fd0 0x7f3ae4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3adbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ae4100fd0 0x7f3ae4198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47562/0 (socket says 172.21.15.31:47562) 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3adbfff640 1 -- 172.21.15.31:0/3160139016 learned_addr learned my addr 172.21.15.31:0/3160139016 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ae4100610 0x7f3ae4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3ae37fe640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae41066b0 0x7f3ae419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3adbfff640 1 -- 172.21.15.31:0/3160139016 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae41066b0 msgr2=0x7f3ae419c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3adbfff640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae41066b0 0x7f3ae419c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3adbfff640 1 -- 172.21.15.31:0/3160139016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ae4100610 msgr2=0x7f3ae4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3adbfff640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ae4100610 0x7f3ae4197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3adbfff640 1 -- 172.21.15.31:0/3160139016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae419ced0 con 0x7f3ae4100fd0 2024-04-25T01:56:12.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.831+0000 7f3ae37fe640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae41066b0 0x7f3ae419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:12.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.832+0000 7f3adbfff640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ae4100fd0 0x7f3ae4198420 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00e9f0 tx=0x7f3acc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.832+0000 7f3ae0ff9640 1 -- 172.21.15.31:0/3160139016 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc00cd80 con 0x7f3ae4100fd0 2024-04-25T01:56:12.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.832+0000 7f3ae0ff9640 1 -- 172.21.15.31:0/3160139016 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3acc0054d0 con 0x7f3ae4100fd0 2024-04-25T01:56:12.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.832+0000 7f3ae9709640 1 -- 172.21.15.31:0/3160139016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae419d1c0 con 0x7f3ae4100fd0 2024-04-25T01:56:12.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.832+0000 7f3ae0ff9640 1 -- 172.21.15.31:0/3160139016 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc010490 con 0x7f3ae4100fd0 2024-04-25T01:56:12.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.832+0000 7f3ae9709640 1 -- 172.21.15.31:0/3160139016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae41a4a50 con 0x7f3ae4100fd0 2024-04-25T01:56:12.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.834+0000 7f3ae0ff9640 1 -- 172.21.15.31:0/3160139016 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3acc005020 con 0x7f3ae4100fd0 2024-04-25T01:56:12.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3aec410740 1 Processor -- start 2024-04-25T01:56:12.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3aec410740 1 -- start start 2024-04-25T01:56:12.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3aec410740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 0x561b445f8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3aec410740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8ea0 0x561b44603030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3aec410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b44603570 0x561b44605960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3aec410740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561b4455d670 con 0x561b44603570 2024-04-25T01:56:12.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3aec410740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561b44505970 con 0x561b445f8ea0 2024-04-25T01:56:12.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3aec410740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561b44559630 con 0x561b445f8580 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3ae0ff9640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ad00a2d00 0x7f3ad00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3ae2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8ea0 0x561b44603030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3adbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b44603570 0x561b44605960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3ae2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8ea0 0x561b44603030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44550/0 (socket says 172.21.15.31:44550) 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3ae2ffd640 1 -- 172.21.15.31:0/332982841 learned_addr learned my addr 172.21.15.31:0/332982841 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3ae37fe640 1 --2- 172.21.15.31:0/332982841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 0x561b445f8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.835+0000 7f3ae37fe640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ad00a2d00 0x7f3ad00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.836+0000 7f3ae0ff9640 1 -- 172.21.15.31:0/3160139016 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f3acc0cf810 con 0x7f3ae4100fd0 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.836+0000 7f3ae2ffd640 1 -- 172.21.15.31:0/332982841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 msgr2=0x561b445f8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.836+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/332982841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 0x561b445f8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.836+0000 7f3ae2ffd640 1 -- 172.21.15.31:0/332982841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b44603570 msgr2=0x561b44605960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.836+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/332982841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b44603570 0x561b44605960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.836+0000 7f3ae2ffd640 1 -- 172.21.15.31:0/332982841 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b44529400 con 0x561b445f8ea0 2024-04-25T01:56:12.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.836+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/332982841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8ea0 0x561b44603030 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009900 tx=0x7f3ad4005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56165cc253efb6a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.836+0000 7f3ae37fe640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ad00a2d00 0x7f3ad00a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3adc005c40 tx=0x7f3adc009500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.836+0000 7f3ad97fa640 1 -- 172.21.15.31:0/332982841 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4040070 con 0x561b445f8ea0 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.836+0000 7f3ad97fa640 1 -- 172.21.15.31:0/332982841 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ad4004030 con 0x561b445f8ea0 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.836+0000 7f3ad97fa640 1 -- 172.21.15.31:0/332982841 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4035040 con 0x561b445f8ea0 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.837+0000 7f3aec410740 1 -- 172.21.15.31:0/332982841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8ea0 msgr2=0x561b44603030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.837+0000 7f3aec410740 1 --2- 172.21.15.31:0/332982841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8ea0 0x561b44603030 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009900 tx=0x7f3ad4005be0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.837+0000 7f3aec410740 1 -- 172.21.15.31:0/332982841 shutdown_connections 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.837+0000 7f3aec410740 1 --2- 172.21.15.31:0/332982841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b44603570 0x561b44605960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.837+0000 7f3aec410740 1 --2- 172.21.15.31:0/332982841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8ea0 0x561b44603030 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.837+0000 7f3aec410740 1 --2- 172.21.15.31:0/332982841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 0x561b445f8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.837+0000 7f3aec410740 1 -- 172.21.15.31:0/332982841 >> 172.21.15.31:0/332982841 conn(0x561b445f48a0 msgr2=0x561b445f4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.837+0000 7f3aec410740 1 -- 172.21.15.31:0/332982841 shutdown_connections 2024-04-25T01:56:12.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.837+0000 7f3aec410740 1 -- 172.21.15.31:0/332982841 wait complete. 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.837+0000 7f3aec410740 1 -- start start 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.837+0000 7f3aec410740 1 Processor -- start 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3aec410740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 0x561b44602e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3aec410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44600150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3aec410740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b44603570 0x561b44600690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3aec410740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561b44606300 con 0x561b445f8ea0 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3aec410740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561b44519050 con 0x561b445f8580 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3aec410740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561b44606480 con 0x561b44603570 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3aec410740 10 client.0 _ll_register_callbacks cb 0x561b44270be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3adbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44600150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3adbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44600150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47584/0 (socket says 172.21.15.31:47584) 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3adbfff640 1 -- 172.21.15.31:0/2380747992 learned_addr learned my addr 172.21.15.31:0/2380747992 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 0x561b44602e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3ae37fe640 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b44603570 0x561b44600690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3adbfff640 1 -- 172.21.15.31:0/2380747992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b44603570 msgr2=0x561b44600690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3adbfff640 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b44603570 0x561b44600690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3adbfff640 1 -- 172.21.15.31:0/2380747992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 msgr2=0x561b44602e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3adbfff640 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 0x561b44602e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3adbfff640 1 -- 172.21.15.31:0/2380747992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b444eadc0 con 0x561b445f8ea0 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 0x561b44602e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3adbfff640 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44600150 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0981e0 tx=0x7f3acc09a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5571764742542569 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.838+0000 7f3aca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aca7fc640 1 -- 172.21.15.31:0/2380747992 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc09bd80 con 0x561b445f8ea0 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aca7fc640 1 -- 172.21.15.31:0/2380747992 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f3acc0a4560 con 0x561b445f8ea0 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aec410740 1 -- 172.21.15.31:0/2380747992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ad4004030 con 0x561b445f8ea0 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aca7fc640 1 -- 172.21.15.31:0/2380747992 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3acc0a0130 con 0x561b445f8ea0 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aec410740 10 client.16125 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:12.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aec410740 1 -- 172.21.15.31:0/2380747992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b442713f0 con 0x561b445f8ea0 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aca7fc640 1 -- 172.21.15.31:0/2380747992 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ad4004030 con 0x561b445f8ea0 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aca7fc640 1 -- 172.21.15.31:0/2380747992 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3acc097b60 con 0x561b445f8ea0 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aec410740 10 client.16125 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aec410740 10 client.16125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aec410740 20 client.16125 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aec410740 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561b44613580 0x561b446159e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aec410740 4 client.16125 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3aec410740 1 -- 172.21.15.31:0/2380747992 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b4456abb0 con 0x561b44613580 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.839+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561b44613580 0x561b446159e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.840+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561b44613580 0x561b446159e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d40612de656cb9ca server_cookie=c49c2ce440210011 in_seq=0 out_seq=0 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.840+0000 7f3aca7fc640 10 client.16125 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.840+0000 7f3aca7fc640 1 -- 172.21.15.31:0/2380747992 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b4456abb0 con 0x561b44613580 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.840+0000 7f3aca7fc640 10 client.16125 handle_command_reply: tid=0 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.840+0000 7f3aec410740 1 client.16125 shutdown 2024-04-25T01:56:12.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aec410740 1 -- 172.21.15.31:0/2380747992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 msgr2=0x561b44600150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aec410740 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44600150 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0981e0 tx=0x7f3acc09a290 comp rx=0 tx=0).stop 2024-04-25T01:56:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aec410740 1 -- 172.21.15.31:0/2380747992 shutdown_connections 2024-04-25T01:56:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aec410740 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561b44613580 0x561b446159e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aec410740 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b44603570 0x561b44600690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aec410740 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44600150 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aec410740 1 --2- 172.21.15.31:0/2380747992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 0x561b44602e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aca7fc640 0 client.16125 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aec410740 1 -- 172.21.15.31:0/2380747992 >> 172.21.15.31:0/2380747992 conn(0x561b445f48a0 msgr2=0x561b445f3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aec410740 1 -- 172.21.15.31:0/2380747992 shutdown_connections 2024-04-25T01:56:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aec410740 1 -- 172.21.15.31:0/2380747992 wait complete. 2024-04-25T01:56:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.841+0000 7f3aec410740 20 client.16125 trim_cache size 0 max 16384 2024-04-25T01:56:12.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.854+0000 7f3aec410740 1 Processor -- start 2024-04-25T01:56:12.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3aec410740 1 -- start start 2024-04-25T01:56:12.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3aec410740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 0x561b4460f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3aec410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b4461b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3aec410740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b44603570 0x561b4461b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3aec410740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae41ad1c0 con 0x561b445f8ea0 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3aec410740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae41aedc0 con 0x561b44603570 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3aec410740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae41ad890 con 0x561b445f8580 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3adbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b4461b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3ae2ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 0x561b4460f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3ae2ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 0x561b4460f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:54322/0 (socket says 172.21.15.31:54322) 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3ae2ffd640 1 -- 172.21.15.31:0/2156685977 learned_addr learned my addr 172.21.15.31:0/2156685977 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3adbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b4461b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47588/0 (socket says 172.21.15.31:47588) 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3ae37fe640 1 --2- 172.21.15.31:0/2156685977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b44603570 0x561b4461b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3ae2ffd640 1 -- 172.21.15.31:0/2156685977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b44603570 msgr2=0x561b4461b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/2156685977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b44603570 0x561b4461b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3ae2ffd640 1 -- 172.21.15.31:0/2156685977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 msgr2=0x561b4461b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/2156685977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b4461b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.855+0000 7f3ae2ffd640 1 -- 172.21.15.31:0/2156685977 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b4433b7d0 con 0x561b445f8580 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.856+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/2156685977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 0x561b4460f1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4005090 tx=0x7f3ad4005020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3721b5542f8963c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.856+0000 7f3ad8ff9640 1 -- 172.21.15.31:0/2156685977 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4040040 con 0x561b445f8580 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.856+0000 7f3ad8ff9640 1 -- 172.21.15.31:0/2156685977 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ad40314a0 con 0x561b445f8580 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.856+0000 7f3ad8ff9640 1 -- 172.21.15.31:0/2156685977 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4035040 con 0x561b445f8580 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.856+0000 7f3aec410740 1 -- 172.21.15.31:0/2156685977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 msgr2=0x561b4460f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.856+0000 7f3aec410740 1 --2- 172.21.15.31:0/2156685977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 0x561b4460f1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4005090 tx=0x7f3ad4005020 comp rx=0 tx=0).stop 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.856+0000 7f3aec410740 1 -- 172.21.15.31:0/2156685977 shutdown_connections 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.856+0000 7f3aec410740 1 --2- 172.21.15.31:0/2156685977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b44603570 0x561b4461b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.856+0000 7f3aec410740 1 --2- 172.21.15.31:0/2156685977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b4461b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.856+0000 7f3aec410740 1 --2- 172.21.15.31:0/2156685977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b445f8580 0x561b4460f1c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 1 -- 172.21.15.31:0/2156685977 >> 172.21.15.31:0/2156685977 conn(0x561b445f48a0 msgr2=0x561b445f32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 1 -- 172.21.15.31:0/2156685977 shutdown_connections 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 1 -- 172.21.15.31:0/2156685977 wait complete. 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 1 -- start start 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 1 Processor -- start 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 0x561b44608880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44608dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b44603570 0x561b44677b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae41ae630 con 0x561b445f8ea0 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae41adf60 con 0x561b445f8580 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae41ab1a0 con 0x561b44603570 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3aec410740 10 client.0 _ll_register_callbacks cb 0x7f3acc00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3adbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44608dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3ae2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 0x561b44608880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3adbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44608dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47600/0 (socket says 172.21.15.31:47600) 2024-04-25T01:56:12.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3adbfff640 1 -- 172.21.15.31:0/1578941015 learned_addr learned my addr 172.21.15.31:0/1578941015 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.857+0000 7f3ae37fe640 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b44603570 0x561b44677b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3adbfff640 1 -- 172.21.15.31:0/1578941015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b44603570 msgr2=0x561b44677b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3adbfff640 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b44603570 0x561b44677b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3adbfff640 1 -- 172.21.15.31:0/1578941015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 msgr2=0x561b44608880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3adbfff640 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 0x561b44608880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3adbfff640 1 -- 172.21.15.31:0/1578941015 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b446780b0 con 0x561b445f8ea0 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3ae37fe640 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b44603570 0x561b44677b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 0x561b44608880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3adbfff640 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44608dc0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3acc098750 tx=0x7f3acc097ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31171f3905d3f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aca7fc640 1 -- 172.21.15.31:0/1578941015 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc09b8c0 con 0x561b445f8ea0 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aca7fc640 1 -- 172.21.15.31:0/1578941015 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f3acc0a4760 con 0x561b445f8ea0 2024-04-25T01:56:12.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aec410740 1 -- 172.21.15.31:0/1578941015 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ad40314a0 con 0x561b445f8ea0 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aca7fc640 1 -- 172.21.15.31:0/1578941015 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3acc09aea0 con 0x561b445f8ea0 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aec410740 10 client.16131 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aec410740 1 -- 172.21.15.31:0/1578941015 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b446793e0 con 0x561b445f8ea0 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aca7fc640 1 -- 172.21.15.31:0/1578941015 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ad40314a0 con 0x561b445f8ea0 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aca7fc640 1 -- 172.21.15.31:0/1578941015 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3acc0a12c0 con 0x561b445f8ea0 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aec410740 10 client.16131 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aec410740 10 client.16131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.858+0000 7f3aec410740 20 client.16131 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561b44679bd0 0x561b4467bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 4 client.16131 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 1 -- 172.21.15.31:0/1578941015 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b44600a20 con 0x561b44679bd0 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561b44679bd0 0x561b4467bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3ae2ffd640 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561b44679bd0 0x561b4467bff0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36bab8c55e873507 server_cookie=e5acdff15a94cc63 in_seq=0 out_seq=0 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aca7fc640 10 client.16131 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aca7fc640 1 -- 172.21.15.31:0/1578941015 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b44600a20 con 0x561b44679bd0 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aca7fc640 10 client.16131 handle_command_reply: tid=0 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 1 client.16131 shutdown 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 1 -- 172.21.15.31:0/1578941015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 msgr2=0x561b44608dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44608dc0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3acc098750 tx=0x7f3acc097ce0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 1 -- 172.21.15.31:0/1578941015 shutdown_connections 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561b44679bd0 0x561b4467bff0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561b44603570 0x561b44677b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b445f8ea0 0x561b44608dc0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 1 --2- 172.21.15.31:0/1578941015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b445f8580 0x561b44608880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.859+0000 7f3aec410740 1 -- 172.21.15.31:0/1578941015 >> 172.21.15.31:0/1578941015 conn(0x561b445f48a0 msgr2=0x561b445f3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3aca7fc640 0 client.16131 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3aec410740 1 -- 172.21.15.31:0/1578941015 shutdown_connections 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3aec410740 1 -- 172.21.15.31:0/1578941015 wait complete. 2024-04-25T01:56:12.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3aec410740 20 client.16131 trim_cache size 0 max 16384 2024-04-25T01:56:12.869 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 -- 172.21.15.31:0/3160139016 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ad00a2d00 msgr2=0x7f3ad00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ad00a2d00 0x7f3ad00a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3adc005c40 tx=0x7f3adc009500 comp rx=0 tx=0).stop 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 -- 172.21.15.31:0/3160139016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ae4100fd0 msgr2=0x7f3ae4198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ae4100fd0 0x7f3ae4198420 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00e9f0 tx=0x7f3acc00edf0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 -- 172.21.15.31:0/3160139016 shutdown_connections 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ad00a2d00 0x7f3ad00a51c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ae41066b0 0x7f3ae419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ae4100fd0 0x7f3ae4198420 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 --2- 172.21.15.31:0/3160139016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ae4100610 0x7f3ae4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 -- 172.21.15.31:0/3160139016 >> 172.21.15.31:0/3160139016 conn(0x7f3ae4077740 msgr2=0x7f3ae40ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 -- 172.21.15.31:0/3160139016 shutdown_connections 2024-04-25T01:56:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:12.860+0000 7f3ae9709640 1 -- 172.21.15.31:0/3160139016 wait complete. 2024-04-25T01:56:12.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:12 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:56:12.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:12 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2197621114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:56:13.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:12 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:56:13.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:12 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2197621114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:56:13.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:12 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:56:13.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:12 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2197621114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:56:13.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 10763 3.85 MB/sec warmup 20 sec latency 12.524 ms 2024-04-25T01:56:13.874 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:56:13.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:56:13.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:56:13.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:13 smithi031 ceph-mon[29204]: pgmap v55: 97 pgs: 97 active+clean; 27 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.0 MiB/s wr, 90 op/s 2024-04-25T01:56:13.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.988+0000 7f9fa1110640 1 -- 172.21.15.31:0/3467495043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f9c10bd00 msgr2=0x7f9f9c10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:13.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.988+0000 7f9fa1110640 1 --2- 172.21.15.31:0/3467495043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f9c10bd00 0x7f9f9c10e1a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009ca0 tx=0x7f9f8c02a9e0 comp rx=0 tx=0).stop 2024-04-25T01:56:13.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.988+0000 7f9fa1110640 1 -- 172.21.15.31:0/3467495043 shutdown_connections 2024-04-25T01:56:13.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.988+0000 7f9fa1110640 1 --2- 172.21.15.31:0/3467495043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f9c10bd00 0x7f9f9c10e1a0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:13.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.988+0000 7f9fa1110640 1 --2- 172.21.15.31:0/3467495043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f9c1032e0 0x7f9f9c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:13.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.988+0000 7f9fa1110640 1 --2- 172.21.15.31:0/3467495043 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f9c1029c0 0x7f9f9c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:13.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.988+0000 7f9fa1110640 1 -- 172.21.15.31:0/3467495043 >> 172.21.15.31:0/3467495043 conn(0x7f9f9c077740 msgr2=0x7f9f9c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:13.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.988+0000 7f9fa1110640 1 -- 172.21.15.31:0/3467495043 shutdown_connections 2024-04-25T01:56:13.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.989+0000 7f9fa1110640 1 -- 172.21.15.31:0/3467495043 wait complete. 2024-04-25T01:56:13.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.989+0000 7f9fa1110640 1 Processor -- start 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.989+0000 7f9fa1110640 1 -- start start 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9fa1110640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f9c1029c0 0x7f9f9c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9fa1110640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f9c1032e0 0x7f9f9c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9fa1110640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f9c10bd00 0x7f9f9c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9fa1110640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c1119b0 con 0x7f9f9c1029c0 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9fa1110640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c111830 con 0x7f9f9c1032e0 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9fa1110640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c111b30 con 0x7f9f9c10bd00 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f9c1029c0 0x7f9f9c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f9c1032e0 0x7f9f9c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f9c1029c0 0x7f9f9c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47626/0 (socket says 172.21.15.31:47626) 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9ad9d640 1 -- 172.21.15.31:0/1951804140 learned_addr learned my addr 172.21.15.31:0/1951804140 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:13.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9b59e640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f9c10bd00 0x7f9f9c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9a59c640 1 -- 172.21.15.31:0/1951804140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f9c10bd00 msgr2=0x7f9f9c19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f9c10bd00 0x7f9f9c19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9a59c640 1 -- 172.21.15.31:0/1951804140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f9c1029c0 msgr2=0x7f9f9c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f9c1029c0 0x7f9f9c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9a59c640 1 -- 172.21.15.31:0/1951804140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f9c19cf60 con 0x7f9f9c1032e0 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9ad9d640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f9c1029c0 0x7f9f9c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.990+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f9c1032e0 0x7f9f9c1984b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400e9f0 tx=0x7f9f8400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.991+0000 7f9f93fff640 1 -- 172.21.15.31:0/1951804140 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8400ccc0 con 0x7f9f9c1032e0 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.991+0000 7f9fa1110640 1 -- 172.21.15.31:0/1951804140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f9c19d250 con 0x7f9f9c1032e0 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.991+0000 7f9f93fff640 1 -- 172.21.15.31:0/1951804140 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f840108e0 con 0x7f9f9c1032e0 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.991+0000 7f9f93fff640 1 -- 172.21.15.31:0/1951804140 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84018880 con 0x7f9f9c1032e0 2024-04-25T01:56:13.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.991+0000 7f9fa1110640 1 -- 172.21.15.31:0/1951804140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f9c1a4ae0 con 0x7f9f9c1032e0 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.993+0000 7f9fa3e1e740 1 Processor -- start 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.993+0000 7f9f93fff640 1 -- 172.21.15.31:0/1951804140 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9f84010a90 con 0x7f9f9c1032e0 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.993+0000 7f9fa3e1e740 1 -- start start 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 0x564c2a5cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5ce290 0x564c2a5ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5d9010 0x564c2a5db400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564c2a5322c0 con 0x564c2a5d9010 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564c2a4da910 con 0x564c2a5cf980 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564c2a52e280 con 0x564c2a5ce290 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9b59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 0x564c2a5cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5d9010 0x564c2a5db400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9b59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 0x564c2a5cdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44608/0 (socket says 172.21.15.31:44608) 2024-04-25T01:56:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9b59e640 1 -- 172.21.15.31:0/3534028051 learned_addr learned my addr 172.21.15.31:0/3534028051 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9ad9d640 1 --2- 172.21.15.31:0/3534028051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5ce290 0x564c2a5ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f93fff640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f880a2d00 0x7f9f880a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9a59c640 1 -- 172.21.15.31:0/3534028051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5ce290 msgr2=0x564c2a5ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/3534028051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5ce290 0x564c2a5ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9a59c640 1 -- 172.21.15.31:0/3534028051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 msgr2=0x564c2a5cdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/3534028051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 0x564c2a5cdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9a59c640 1 -- 172.21.15.31:0/3534028051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c2a4ebfb0 con 0x564c2a5d9010 2024-04-25T01:56:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9ad9d640 1 --2- 172.21.15.31:0/3534028051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5ce290 0x564c2a5ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9b59e640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f880a2d00 0x7f9f880a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f93fff640 1 -- 172.21.15.31:0/1951804140 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9f840cf6c0 con 0x7f9f9c1032e0 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.994+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/3534028051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5d9010 0x564c2a5db400 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9f840cce60 tx=0x7f9f84091170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410d99a5438f2205 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.995+0000 7f9f917fa640 1 -- 172.21.15.31:0/3534028051 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8409b820 con 0x564c2a5d9010 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.995+0000 7f9f9b59e640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f880a2d00 0x7f9f880a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009a60 tx=0x7f9f8c03a040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.995+0000 7f9f917fa640 1 -- 172.21.15.31:0/3534028051 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f84010e00 con 0x564c2a5d9010 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.995+0000 7f9f917fa640 1 -- 172.21.15.31:0/3534028051 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8409b640 con 0x564c2a5d9010 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.995+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3534028051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5d9010 msgr2=0x564c2a5db400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.995+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/3534028051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5d9010 0x564c2a5db400 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9f840cce60 tx=0x7f9f84091170 comp rx=0 tx=0).stop 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.995+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3534028051 shutdown_connections 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.995+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/3534028051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5d9010 0x564c2a5db400 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.995+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/3534028051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5ce290 0x564c2a5ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.995+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/3534028051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 0x564c2a5cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3534028051 >> 172.21.15.31:0/3534028051 conn(0x564c2a596950 msgr2=0x564c2a5ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3534028051 shutdown_connections 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3534028051 wait complete. 2024-04-25T01:56:14.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 1 -- start start 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 1 Processor -- start 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5d5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 0x564c2a5d6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5d9010 0x564c2a5d6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564c2a5dbde0 con 0x564c2a5ce290 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564c2a4ede90 con 0x564c2a5cf980 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564c2a5dbf60 con 0x564c2a5d9010 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9f9ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5d5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9fa3e1e740 10 client.0 _ll_register_callbacks cb 0x564c2a529970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9f9ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5d5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47650/0 (socket says 172.21.15.31:47650) 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9f9ad9d640 1 -- 172.21.15.31:0/3411295782 learned_addr learned my addr 172.21.15.31:0/3411295782 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 0x564c2a5d6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.996+0000 7f9f9b59e640 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5d9010 0x564c2a5d6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f9ad9d640 1 -- 172.21.15.31:0/3411295782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5d9010 msgr2=0x564c2a5d6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f9ad9d640 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5d9010 0x564c2a5d6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f9ad9d640 1 -- 172.21.15.31:0/3411295782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 msgr2=0x564c2a5d6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f9ad9d640 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 0x564c2a5d6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f9ad9d640 1 -- 172.21.15.31:0/3411295782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c2a30e7e0 con 0x564c2a5ce290 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 0x564c2a5d6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f9ad9d640 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5d5f10 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9f94011880 tx=0x7f9f94011d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c101cc2db692fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f827fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f827fc640 1 -- 172.21.15.31:0/3411295782 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9401f060 con 0x564c2a5ce290 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f827fc640 1 -- 172.21.15.31:0/3411295782 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9f94010ba0 con 0x564c2a5ce290 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3411295782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564c2a5065a0 con 0x564c2a5ce290 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f827fc640 1 -- 172.21.15.31:0/3411295782 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f94010da0 con 0x564c2a5ce290 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9fa3e1e740 10 client.16143 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:14.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3411295782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564c2a36ef70 con 0x564c2a5ce290 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f827fc640 1 -- 172.21.15.31:0/3411295782 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564c2a5065a0 con 0x564c2a5ce290 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.997+0000 7f9f827fc640 1 -- 172.21.15.31:0/3411295782 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9f94022360 con 0x564c2a5ce290 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.998+0000 7f9fa3e1e740 10 client.16143 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.998+0000 7f9fa3e1e740 10 client.16143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.998+0000 7f9fa3e1e740 20 client.16143 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.998+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564c2a5e91a0 0x564c2a5eb640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.998+0000 7f9fa3e1e740 4 client.16143 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.998+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3411295782 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c2a53f800 con 0x564c2a5e91a0 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.998+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564c2a5e91a0 0x564c2a5eb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.998+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564c2a5e91a0 0x564c2a5eb640 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be71dd51d12dc03e server_cookie=8cc4313615e7378b in_seq=0 out_seq=0 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.998+0000 7f9f827fc640 10 client.16143 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9f827fc640 1 -- 172.21.15.31:0/3411295782 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c2a53f800 con 0x564c2a5e91a0 2024-04-25T01:56:14.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9f827fc640 10 client.16143 handle_command_reply: tid=0 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 1 client.16143 shutdown 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3411295782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 msgr2=0x564c2a5d5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5d5f10 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9f94011880 tx=0x7f9f94011d40 comp rx=0 tx=0).stop 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3411295782 shutdown_connections 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564c2a5e91a0 0x564c2a5eb640 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5d9010 0x564c2a5d6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5cf980 0x564c2a5d6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9f827fc640 0 client.16143 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/3411295782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5d5f10 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3411295782 >> 172.21.15.31:0/3411295782 conn(0x564c2a596950 msgr2=0x564c2a5d0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3411295782 shutdown_connections 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/3411295782 wait complete. 2024-04-25T01:56:14.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:13.999+0000 7f9fa3e1e740 20 client.16143 trim_cache size 0 max 16384 2024-04-25T01:56:14.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.019+0000 7f9fa3e1e740 1 Processor -- start 2024-04-25T01:56:14.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.019+0000 7f9fa3e1e740 1 -- start start 2024-04-25T01:56:14.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.019+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5ce290 0x564c2a5e4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5cf980 0x564c2a5c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5d9010 0x564c2a5f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c1ab080 con 0x564c2a5cf980 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c1acc80 con 0x564c2a5ce290 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c1ab750 con 0x564c2a5d9010 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f9a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5cf980 0x564c2a5c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f9a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5cf980 0x564c2a5c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47654/0 (socket says 172.21.15.31:47654) 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f9a59c640 1 -- 172.21.15.31:0/2563777363 learned_addr learned my addr 172.21.15.31:0/2563777363 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f9ad9d640 1 --2- 172.21.15.31:0/2563777363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5ce290 0x564c2a5e4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f9b59e640 1 --2- 172.21.15.31:0/2563777363 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5d9010 0x564c2a5f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f9a59c640 1 -- 172.21.15.31:0/2563777363 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5d9010 msgr2=0x564c2a5f0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/2563777363 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5d9010 0x564c2a5f0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f9a59c640 1 -- 172.21.15.31:0/2563777363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5ce290 msgr2=0x564c2a5e4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/2563777363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5ce290 0x564c2a5e4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f9a59c640 1 -- 172.21.15.31:0/2563777363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c2a5d4ec0 con 0x564c2a5cf980 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/2563777363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5cf980 0x564c2a5c97d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9f840ccf10 tx=0x7f9f84009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb7e102db4ea35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f90ff9640 1 -- 172.21.15.31:0/2563777363 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8409b820 con 0x564c2a5cf980 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f90ff9640 1 -- 172.21.15.31:0/2563777363 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f8409ffe0 con 0x564c2a5cf980 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.020+0000 7f9f90ff9640 1 -- 172.21.15.31:0/2563777363 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84098350 con 0x564c2a5cf980 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.021+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/2563777363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5cf980 msgr2=0x564c2a5c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.021+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/2563777363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5cf980 0x564c2a5c97d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9f840ccf10 tx=0x7f9f84009700 comp rx=0 tx=0).stop 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.021+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/2563777363 shutdown_connections 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.021+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/2563777363 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5d9010 0x564c2a5f0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.021+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/2563777363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5cf980 0x564c2a5c97d0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.021+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/2563777363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5ce290 0x564c2a5e4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.021+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/2563777363 >> 172.21.15.31:0/2563777363 conn(0x564c2a596950 msgr2=0x564c2a5c90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.021+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/2563777363 shutdown_connections 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.021+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/2563777363 wait complete. 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.021+0000 7f9fa3e1e740 1 -- start start 2024-04-25T01:56:14.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.021+0000 7f9fa3e1e740 1 Processor -- start 2024-04-25T01:56:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5dcec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5cf980 0x564c2a5dd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9fa3e1e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5d9010 0x564c2a5dd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c1ac4f0 con 0x564c2a5ce290 2024-04-25T01:56:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c1abe20 con 0x564c2a5d9010 2024-04-25T01:56:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9fa3e1e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c1a9060 con 0x564c2a5cf980 2024-04-25T01:56:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9fa3e1e740 10 client.0 _ll_register_callbacks cb 0x7f9f94011460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f9ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5dcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f9ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5dcec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47658/0 (socket says 172.21.15.31:47658) 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f9b59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5d9010 0x564c2a5dd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f9a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5cf980 0x564c2a5dd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f9ad9d640 1 -- 172.21.15.31:0/202484338 learned_addr learned my addr 172.21.15.31:0/202484338 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f9ad9d640 1 -- 172.21.15.31:0/202484338 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5cf980 msgr2=0x564c2a5dd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f9ad9d640 1 --2- 172.21.15.31:0/202484338 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5cf980 0x564c2a5dd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f9ad9d640 1 -- 172.21.15.31:0/202484338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5d9010 msgr2=0x564c2a5dd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f9ad9d640 1 --2- 172.21.15.31:0/202484338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5d9010 0x564c2a5dd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f9ad9d640 1 -- 172.21.15.31:0/202484338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c2a206170 con 0x564c2a5ce290 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f9ad9d640 1 --2- 172.21.15.31:0/202484338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5dcec0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f9f94022730 tx=0x7f9f9400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e148333711310f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f827fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f827fc640 1 -- 172.21.15.31:0/202484338 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94005850 con 0x564c2a5ce290 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9f827fc640 1 -- 172.21.15.31:0/202484338 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9f94010ba0 con 0x564c2a5ce290 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.022+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/202484338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f8409ffe0 con 0x564c2a5ce290 2024-04-25T01:56:14.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9f827fc640 1 -- 172.21.15.31:0/202484338 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f94005040 con 0x564c2a5ce290 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9fa3e1e740 10 client.16149 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/202484338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564c2a64fbf0 con 0x564c2a5ce290 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9f827fc640 1 -- 172.21.15.31:0/202484338 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f8409ffe0 con 0x564c2a5ce290 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9f827fc640 1 -- 172.21.15.31:0/202484338 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9f940049d0 con 0x564c2a5ce290 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9fa3e1e740 10 client.16149 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9fa3e1e740 10 client.16149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9fa3e1e740 20 client.16149 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/202484338 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564c2a6503d0 0x564c2a6527f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9fa3e1e740 4 client.16149 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/202484338 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564c2a539b50 con 0x564c2a6503d0 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.023+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/202484338 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564c2a6503d0 0x564c2a6527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.024+0000 7f9f827fc640 10 client.16149 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.024+0000 7f9f9a59c640 1 --2- 172.21.15.31:0/202484338 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564c2a6503d0 0x564c2a6527f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9877f7977662803b server_cookie=dd7ff3d880c9021 in_seq=0 out_seq=0 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9f827fc640 1 -- 172.21.15.31:0/202484338 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564c2a539b50 con 0x564c2a6503d0 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9f827fc640 10 client.16149 handle_command_reply: tid=0 2024-04-25T01:56:14.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9fa3e1e740 1 client.16149 shutdown 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/202484338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 msgr2=0x564c2a5dcec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/202484338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5dcec0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f9f94022730 tx=0x7f9f9400bf90 comp rx=0 tx=0).stop 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/202484338 shutdown_connections 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/202484338 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564c2a6503d0 0x564c2a6527f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/202484338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c2a5d9010 0x564c2a5dd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/202484338 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564c2a5cf980 0x564c2a5dd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9fa3e1e740 1 --2- 172.21.15.31:0/202484338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564c2a5ce290 0x564c2a5dcec0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/202484338 >> 172.21.15.31:0/202484338 conn(0x564c2a596950 msgr2=0x564c2a5c9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9f827fc640 0 client.16149 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/202484338 shutdown_connections 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.025+0000 7f9fa3e1e740 1 -- 172.21.15.31:0/202484338 wait complete. 2024-04-25T01:56:14.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa3e1e740 20 client.16149 trim_cache size 0 max 16384 2024-04-25T01:56:14.033 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:56:14.033 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T01:56:14.033 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "4a50b5d1-e9d8-4c94-ba97-ace3fbe06664", 2024-04-25T01:56:14.033 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T01:56:14.033 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:56:14.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 -- 172.21.15.31:0/1951804140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f880a2d00 msgr2=0x7f9f880a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f880a2d00 0x7f9f880a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009a60 tx=0x7f9f8c03a040 comp rx=0 tx=0).stop 2024-04-25T01:56:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 -- 172.21.15.31:0/1951804140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f9c1032e0 msgr2=0x7f9f9c1984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f9c1032e0 0x7f9f9c1984b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400e9f0 tx=0x7f9f8400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 -- 172.21.15.31:0/1951804140 shutdown_connections 2024-04-25T01:56:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9f880a2d00 0x7f9f880a51c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f9c10bd00 0x7f9f9c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f9c1032e0 0x7f9f9c1984b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 --2- 172.21.15.31:0/1951804140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f9c1029c0 0x7f9f9c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 -- 172.21.15.31:0/1951804140 >> 172.21.15.31:0/1951804140 conn(0x7f9f9c077740 msgr2=0x7f9f9c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 -- 172.21.15.31:0/1951804140 shutdown_connections 2024-04-25T01:56:14.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.026+0000 7f9fa1110640 1 -- 172.21.15.31:0/1951804140 wait complete. 2024-04-25T01:56:14.040 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:56:14.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:56:14.041 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:13 smithi040 ceph-mon[32418]: pgmap v55: 97 pgs: 97 active+clean; 27 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.0 MiB/s wr, 90 op/s 2024-04-25T01:56:14.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.149+0000 7fb42126b640 1 -- 172.21.15.31:0/2258237096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41c1066b0 msgr2=0x7fb41c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.149+0000 7fb42126b640 1 --2- 172.21.15.31:0/2258237096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41c1066b0 0x7fb41c106a90 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009930 tx=0x7fb40c02a420 comp rx=0 tx=0).stop 2024-04-25T01:56:14.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.149+0000 7fb42126b640 1 -- 172.21.15.31:0/2258237096 shutdown_connections 2024-04-25T01:56:14.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.149+0000 7fb42126b640 1 --2- 172.21.15.31:0/2258237096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb41c100fd0 0x7fb41c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.149+0000 7fb42126b640 1 --2- 172.21.15.31:0/2258237096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb41c100610 0x7fb41c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.149+0000 7fb42126b640 1 --2- 172.21.15.31:0/2258237096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41c1066b0 0x7fb41c106a90 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.149+0000 7fb42126b640 1 -- 172.21.15.31:0/2258237096 >> 172.21.15.31:0/2258237096 conn(0x7fb41c077740 msgr2=0x7fb41c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.149+0000 7fb42126b640 1 -- 172.21.15.31:0/2258237096 shutdown_connections 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.149+0000 7fb42126b640 1 -- 172.21.15.31:0/2258237096 wait complete. 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.149+0000 7fb42126b640 1 Processor -- start 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb42126b640 1 -- start start 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb42126b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb41c100610 0x7fb41c1022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb42126b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41c100fd0 0x7fb41c1027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb42126b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb41c1066b0 0x7fb41c102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb42126b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c1108a0 con 0x7fb41c100fd0 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb42126b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c110720 con 0x7fb41c1066b0 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb42126b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c110a20 con 0x7fb41c100610 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb413fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41c100fd0 0x7fb41c1027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb41b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb41c1066b0 0x7fb41c102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb413fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41c100fd0 0x7fb41c1027f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47678/0 (socket says 172.21.15.31:47678) 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb41b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb41c1066b0 0x7fb41c102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44636/0 (socket says 172.21.15.31:44636) 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb41affd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb41c100610 0x7fb41c1022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb413fff640 1 -- 172.21.15.31:0/3996458818 learned_addr learned my addr 172.21.15.31:0/3996458818 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb413fff640 1 -- 172.21.15.31:0/3996458818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb41c100610 msgr2=0x7fb41c1022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb413fff640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb41c100610 0x7fb41c1022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb413fff640 1 -- 172.21.15.31:0/3996458818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb41c1066b0 msgr2=0x7fb41c102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb413fff640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb41c1066b0 0x7fb41c102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb413fff640 1 -- 172.21.15.31:0/3996458818 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41c1a63f0 con 0x7fb41c100fd0 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb41affd640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb41c100610 0x7fb41c1022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.150+0000 7fb413fff640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41c100fd0 0x7fb41c1027f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb40400e9f0 tx=0x7fb40400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.151+0000 7fb418ff9640 1 -- 172.21.15.31:0/3996458818 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40400cd80 con 0x7fb41c100fd0 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.151+0000 7fb418ff9640 1 -- 172.21.15.31:0/3996458818 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb4040108e0 con 0x7fb41c100fd0 2024-04-25T01:56:14.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.151+0000 7fb42126b640 1 -- 172.21.15.31:0/3996458818 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb41c1a66e0 con 0x7fb41c100fd0 2024-04-25T01:56:14.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.151+0000 7fb418ff9640 1 -- 172.21.15.31:0/3996458818 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb404018920 con 0x7fb41c100fd0 2024-04-25T01:56:14.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.151+0000 7fb42126b640 1 -- 172.21.15.31:0/3996458818 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb41c1a6c80 con 0x7fb41c100fd0 2024-04-25T01:56:14.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.153+0000 7fb418ff9640 1 -- 172.21.15.31:0/3996458818 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb404010a90 con 0x7fb41c100fd0 2024-04-25T01:56:14.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.153+0000 7fb423f7a740 1 Processor -- start 2024-04-25T01:56:14.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.153+0000 7fb423f7a740 1 -- start start 2024-04-25T01:56:14.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13630 0x5625edf13a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf13f50 0x5625edf1e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf1e620 0x5625edf20a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb423f7a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5625ede786b0 con 0x5625edf1e620 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb423f7a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5625ede20880 con 0x5625edf13f50 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb423f7a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5625ede74670 con 0x5625edf13630 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb418ff9640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb4080a2d80 0x7fb4080a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb413fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf1e620 0x5625edf20a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb41affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf13f50 0x5625edf1e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb41affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf13f50 0x5625edf1e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44638/0 (socket says 172.21.15.31:44638) 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb41affd640 1 -- 172.21.15.31:0/2911944890 learned_addr learned my addr 172.21.15.31:0/2911944890 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb41b7fe640 1 --2- 172.21.15.31:0/2911944890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13630 0x5625edf13a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb41b7fe640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb4080a2d80 0x7fb4080a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb418ff9640 1 -- 172.21.15.31:0/3996458818 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fb4040cf670 con 0x7fb41c100fd0 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb41b7fe640 1 -- 172.21.15.31:0/2911944890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf13f50 msgr2=0x5625edf1e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb41b7fe640 1 --2- 172.21.15.31:0/2911944890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf13f50 0x5625edf1e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb41b7fe640 1 -- 172.21.15.31:0/2911944890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf1e620 msgr2=0x5625edf20a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.154+0000 7fb41b7fe640 1 --2- 172.21.15.31:0/2911944890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf1e620 0x5625edf20a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.155+0000 7fb41b7fe640 1 -- 172.21.15.31:0/2911944890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625edc567d0 con 0x5625edf13630 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.155+0000 7fb41b7fe640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb4080a2d80 0x7fb4080a5240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb41400fc90 tx=0x7fb414018000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.155+0000 7fb41b7fe640 1 --2- 172.21.15.31:0/2911944890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13630 0x5625edf13a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb414011cc0 tx=0x7fb41400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92370e20f0275a10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.155+0000 7fb4117fa640 1 -- 172.21.15.31:0/2911944890 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414004b20 con 0x5625edf13630 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.155+0000 7fb4117fa640 1 -- 172.21.15.31:0/2911944890 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb414009ce0 con 0x5625edf13630 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.155+0000 7fb4117fa640 1 -- 172.21.15.31:0/2911944890 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414005460 con 0x5625edf13630 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.156+0000 7fb423f7a740 1 -- 172.21.15.31:0/2911944890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13630 msgr2=0x5625edf13a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.156+0000 7fb423f7a740 1 --2- 172.21.15.31:0/2911944890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13630 0x5625edf13a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb414011cc0 tx=0x7fb41400f790 comp rx=0 tx=0).stop 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.156+0000 7fb423f7a740 1 -- 172.21.15.31:0/2911944890 shutdown_connections 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.156+0000 7fb423f7a740 1 --2- 172.21.15.31:0/2911944890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf1e620 0x5625edf20a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.156+0000 7fb423f7a740 1 --2- 172.21.15.31:0/2911944890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf13f50 0x5625edf1e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.156+0000 7fb423f7a740 1 --2- 172.21.15.31:0/2911944890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13630 0x5625edf13a10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.156+0000 7fb423f7a740 1 -- 172.21.15.31:0/2911944890 >> 172.21.15.31:0/2911944890 conn(0x5625edf0f950 msgr2=0x5625edf0fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.156+0000 7fb423f7a740 1 -- 172.21.15.31:0/2911944890 shutdown_connections 2024-04-25T01:56:14.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.156+0000 7fb423f7a740 1 -- 172.21.15.31:0/2911944890 wait complete. 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.156+0000 7fb423f7a740 1 -- start start 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb423f7a740 1 Processor -- start 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 0x5625edf1b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13f50 0x5625edf1ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf1bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb423f7a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5625edf213b0 con 0x5625edf13630 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb423f7a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5625ede33f50 con 0x5625edf1e620 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb423f7a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5625edf21530 con 0x5625edf13f50 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb423f7a740 10 client.0 _ll_register_callbacks cb 0x5625ede6f000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb41affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 0x5625edf1b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb41b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf1bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb41affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 0x5625edf1b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47700/0 (socket says 172.21.15.31:47700) 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb41affd640 1 -- 172.21.15.31:0/3196638414 learned_addr learned my addr 172.21.15.31:0/3196638414 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:14.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb413fff640 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13f50 0x5625edf1ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb41b7fe640 1 -- 172.21.15.31:0/3196638414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13f50 msgr2=0x5625edf1ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb41b7fe640 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13f50 0x5625edf1ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb41b7fe640 1 -- 172.21.15.31:0/3196638414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 msgr2=0x5625edf1b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb41b7fe640 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 0x5625edf1b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb41b7fe640 1 -- 172.21.15.31:0/3196638414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625edb8c3f0 con 0x5625edf1e620 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb41b7fe640 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf1bfb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb414005100 tx=0x7fb414004f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fed01186213332a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.157+0000 7fb4027fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb4027fc640 1 -- 172.21.15.31:0/3196638414 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414010a40 con 0x5625edf1e620 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb4027fc640 1 -- 172.21.15.31:0/3196638414 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fb414010be0 con 0x5625edf1e620 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb423f7a740 1 -- 172.21.15.31:0/3196638414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb414009ce0 con 0x5625edf1e620 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb4027fc640 1 -- 172.21.15.31:0/3196638414 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb414010de0 con 0x5625edf1e620 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb423f7a740 10 client.25766 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb423f7a740 1 -- 172.21.15.31:0/3196638414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625edb4c0f0 con 0x5625edf1e620 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb4027fc640 1 -- 172.21.15.31:0/3196638414 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb414009ce0 con 0x5625edf1e620 2024-04-25T01:56:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb4027fc640 1 -- 172.21.15.31:0/3196638414 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb41402e070 con 0x5625edf1e620 2024-04-25T01:56:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb423f7a740 10 client.25766 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb423f7a740 10 client.25766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb423f7a740 20 client.25766 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb423f7a740 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5625edf2e780 0x5625edf30b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb423f7a740 4 client.25766 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.158+0000 7fb423f7a740 1 -- 172.21.15.31:0/3196638414 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625ede85bf0 con 0x5625edf2e780 2024-04-25T01:56:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.159+0000 7fb41affd640 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5625edf2e780 0x5625edf30b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.159+0000 7fb41affd640 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5625edf2e780 0x5625edf30b70 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=692f258e07da392f server_cookie=c9630f96e7a025b6 in_seq=0 out_seq=0 2024-04-25T01:56:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.159+0000 7fb4027fc640 10 client.25766 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:14.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb4027fc640 1 -- 172.21.15.31:0/3196638414 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625ede85bf0 con 0x5625edf2e780 2024-04-25T01:56:14.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb4027fc640 10 client.25766 handle_command_reply: tid=0 2024-04-25T01:56:14.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 1 client.25766 shutdown 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 1 -- 172.21.15.31:0/3196638414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 msgr2=0x5625edf1bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf1bfb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb414005100 tx=0x7fb414004f20 comp rx=0 tx=0).stop 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 1 -- 172.21.15.31:0/3196638414 shutdown_connections 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5625edf2e780 0x5625edf30b70 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf1bfb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13f50 0x5625edf1ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb4027fc640 0 client.25766 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 1 --2- 172.21.15.31:0/3196638414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 0x5625edf1b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 1 -- 172.21.15.31:0/3196638414 >> 172.21.15.31:0/3196638414 conn(0x5625edf0f950 msgr2=0x5625edf15750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 1 -- 172.21.15.31:0/3196638414 shutdown_connections 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 1 -- 172.21.15.31:0/3196638414 wait complete. 2024-04-25T01:56:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.160+0000 7fb423f7a740 20 client.25766 trim_cache size 0 max 16384 2024-04-25T01:56:14.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.179+0000 7fb423f7a740 1 Processor -- start 2024-04-25T01:56:14.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.179+0000 7fb423f7a740 1 -- start start 2024-04-25T01:56:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 0x5625edf1dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13f50 0x5625edf2a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf363f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb423f7a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c1af540 con 0x5625edf13630 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb423f7a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c1b1140 con 0x5625edf1e620 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb423f7a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c1afc10 con 0x5625edf13f50 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb41affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 0x5625edf1dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb41affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 0x5625edf1dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47716/0 (socket says 172.21.15.31:47716) 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb41affd640 1 -- 172.21.15.31:0/3140163895 learned_addr learned my addr 172.21.15.31:0/3140163895 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb413fff640 1 --2- 172.21.15.31:0/3140163895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13f50 0x5625edf2a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb41b7fe640 1 --2- 172.21.15.31:0/3140163895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf363f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb41affd640 1 -- 172.21.15.31:0/3140163895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13f50 msgr2=0x5625edf2a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb41affd640 1 --2- 172.21.15.31:0/3140163895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13f50 0x5625edf2a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb41affd640 1 -- 172.21.15.31:0/3140163895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 msgr2=0x5625edf363f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb41affd640 1 --2- 172.21.15.31:0/3140163895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf363f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb41affd640 1 -- 172.21.15.31:0/3140163895 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625edf13d80 con 0x5625edf13630 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb413fff640 1 --2- 172.21.15.31:0/3140163895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13f50 0x5625edf2a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb41affd640 1 --2- 172.21.15.31:0/3140163895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 0x5625edf1dc60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb40c0096c0 tx=0x7fb40c004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93d58132fd0df76 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb410ff9640 1 -- 172.21.15.31:0/3140163895 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c035040 con 0x5625edf13630 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb410ff9640 1 -- 172.21.15.31:0/3140163895 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb40c047870 con 0x5625edf13630 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.180+0000 7fb410ff9640 1 -- 172.21.15.31:0/3140163895 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c0049d0 con 0x5625edf13630 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.181+0000 7fb423f7a740 1 -- 172.21.15.31:0/3140163895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 msgr2=0x5625edf1dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.181+0000 7fb423f7a740 1 --2- 172.21.15.31:0/3140163895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 0x5625edf1dc60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb40c0096c0 tx=0x7fb40c004280 comp rx=0 tx=0).stop 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.181+0000 7fb423f7a740 1 -- 172.21.15.31:0/3140163895 shutdown_connections 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.181+0000 7fb423f7a740 1 --2- 172.21.15.31:0/3140163895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf363f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.181+0000 7fb423f7a740 1 --2- 172.21.15.31:0/3140163895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13f50 0x5625edf2a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.181+0000 7fb423f7a740 1 --2- 172.21.15.31:0/3140163895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13630 0x5625edf1dc60 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.181+0000 7fb423f7a740 1 -- 172.21.15.31:0/3140163895 >> 172.21.15.31:0/3140163895 conn(0x5625edf0f950 msgr2=0x5625edf15720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.181+0000 7fb423f7a740 1 -- 172.21.15.31:0/3140163895 shutdown_connections 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.181+0000 7fb423f7a740 1 -- 172.21.15.31:0/3140163895 wait complete. 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.181+0000 7fb423f7a740 1 -- start start 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb423f7a740 1 Processor -- start 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13630 0x5625edf26780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13f50 0x5625edf26cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb423f7a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf17bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb423f7a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c1b09b0 con 0x5625edf13f50 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb423f7a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c1b02e0 con 0x5625edf1e620 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb423f7a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c1ad520 con 0x5625edf13630 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb423f7a740 10 client.0 _ll_register_callbacks cb 0x7fb40400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb413fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13f50 0x5625edf26cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb413fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13f50 0x5625edf26cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47718/0 (socket says 172.21.15.31:47718) 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb413fff640 1 -- 172.21.15.31:0/2358901531 learned_addr learned my addr 172.21.15.31:0/2358901531 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb41affd640 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13630 0x5625edf26780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb41b7fe640 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf17bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb413fff640 1 -- 172.21.15.31:0/2358901531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13630 msgr2=0x5625edf26780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb413fff640 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13630 0x5625edf26780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb413fff640 1 -- 172.21.15.31:0/2358901531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 msgr2=0x5625edf17bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb413fff640 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf17bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb413fff640 1 -- 172.21.15.31:0/2358901531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625edf183e0 con 0x5625edf13f50 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb413fff640 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13f50 0x5625edf26cc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb4040980f0 tx=0x7fb404098030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9005218441d619a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.182+0000 7fb4027fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb4027fc640 1 -- 172.21.15.31:0/2358901531 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40409bc90 con 0x5625edf13f50 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb4027fc640 1 -- 172.21.15.31:0/2358901531 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fb404010430 con 0x5625edf13f50 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb423f7a740 1 -- 172.21.15.31:0/2358901531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb40c047870 con 0x5625edf13f50 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb4027fc640 1 -- 172.21.15.31:0/2358901531 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb4040b8020 con 0x5625edf13f50 2024-04-25T01:56:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb423f7a740 10 client.16167 fetch_fsmap learned FSMap version 29 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb423f7a740 1 -- 172.21.15.31:0/2358901531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625edf947f0 con 0x5625edf13f50 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb4027fc640 1 -- 172.21.15.31:0/2358901531 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb40c047870 con 0x5625edf13f50 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb4027fc640 1 -- 172.21.15.31:0/2358901531 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb4040cfb20 con 0x5625edf13f50 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb423f7a740 10 client.16167 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb423f7a740 10 client.16167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb423f7a740 20 client.16167 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb423f7a740 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5625edf95050 0x5625edf97470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb423f7a740 4 client.16167 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb423f7a740 1 -- 172.21.15.31:0/2358901531 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5625ede7ff40 con 0x5625edf95050 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.183+0000 7fb41affd640 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5625edf95050 0x5625edf97470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb41affd640 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5625edf95050 0x5625edf97470 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf5ab4aedab12cc5 server_cookie=c424527ec8b9edbf in_seq=0 out_seq=0 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb4027fc640 10 client.16167 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:14.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb4027fc640 1 -- 172.21.15.31:0/2358901531 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5625ede7ff40 con 0x5625edf95050 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb4027fc640 10 client.16167 handle_command_reply: tid=0 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 1 client.16167 shutdown 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 1 -- 172.21.15.31:0/2358901531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13f50 msgr2=0x5625edf26cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13f50 0x5625edf26cc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb4040980f0 tx=0x7fb404098030 comp rx=0 tx=0).stop 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 1 -- 172.21.15.31:0/2358901531 shutdown_connections 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5625edf95050 0x5625edf97470 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5625edf1e620 0x5625edf17bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625edf13f50 0x5625edf26cc0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb4027fc640 0 client.16167 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 1 --2- 172.21.15.31:0/2358901531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5625edf13630 0x5625edf26780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 1 -- 172.21.15.31:0/2358901531 >> 172.21.15.31:0/2358901531 conn(0x5625edf0f950 msgr2=0x5625edf19cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 1 -- 172.21.15.31:0/2358901531 shutdown_connections 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 1 -- 172.21.15.31:0/2358901531 wait complete. 2024-04-25T01:56:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.184+0000 7fb423f7a740 20 client.16167 trim_cache size 0 max 16384 2024-04-25T01:56:14.194 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:56:14.194 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:56:14.194 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T01:56:14.194 INFO:teuthology.orchestra.run.smithi031.stdout: "4a50b5d1-e9d8-4c94-ba97-ace3fbe06664": { 2024-04-25T01:56:14.194 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T01:56:14.194 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "4a50b5d1-e9d8-4c94-ba97-ace3fbe06664", 2024-04-25T01:56:14.194 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T01:56:14.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T01:56:14.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T01:56:14.194 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:56:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 -- 172.21.15.31:0/3996458818 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb4080a2d80 msgr2=0x7fb4080a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb4080a2d80 0x7fb4080a5240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb41400fc90 tx=0x7fb414018000 comp rx=0 tx=0).stop 2024-04-25T01:56:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 -- 172.21.15.31:0/3996458818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41c100fd0 msgr2=0x7fb41c1027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41c100fd0 0x7fb41c1027f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb40400e9f0 tx=0x7fb40400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 -- 172.21.15.31:0/3996458818 shutdown_connections 2024-04-25T01:56:14.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb4080a2d80 0x7fb4080a5240 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb41c1066b0 0x7fb41c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb41c100fd0 0x7fb41c1027f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 --2- 172.21.15.31:0/3996458818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb41c100610 0x7fb41c1022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 -- 172.21.15.31:0/3996458818 >> 172.21.15.31:0/3996458818 conn(0x7fb41c077740 msgr2=0x7fb41c0ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 -- 172.21.15.31:0/3996458818 shutdown_connections 2024-04-25T01:56:14.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:14.185+0000 7fb42126b640 1 -- 172.21.15.31:0/3996458818 wait complete. 2024-04-25T01:56:14.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:13 smithi134 ceph-mon[33098]: pgmap v55: 97 pgs: 97 active+clean; 27 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.0 MiB/s wr, 90 op/s 2024-04-25T01:56:14.198 INFO:tasks.cephfs.filesystem:scrub status for tag:4a50b5d1-e9d8-4c94-ba97-ace3fbe06664 - {'path': '/', 'tag': '4a50b5d1-e9d8-4c94-ba97-ace3fbe06664', 'options': 'recursive,force'} 2024-04-25T01:56:14.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 11047 3.73 MB/sec warmup 21 sec latency 31.025 ms 2024-04-25T01:56:14.411 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-h[62279]: 2024-04-25T01:56:14.038+0000 7f9cae6c3700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000034f [4,head] /volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/TIPS.DOC auth v360 ap=1 s=14984 n(v0 rc2024-04-25T01:56:06.681311+0000 b14984 1=1+0) caps={15375=pAsLsXsFsc/-@1},l=15375(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d73e7f0100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000034f:[<0x10000000208/TIPS.DOC v338>,<0x10000000009/WORD v2706>,<0x10000000008/~dmtmp v74>,<0x10000000007/client0 v58>,<0x10000000006/clients v36>,<0x10000000005/tmp v36>,<0x10000000003/client.0 v37>,<0x10000000002/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d v51>,<0x10000000001/sv_0 v71>,<0x10000000000/qa v57>,<0x1/volumes v53>]//[]","memoryvalue":"(3)0x1000000034f:[<0x10000000208/TIPS.DOC v360>,<0x10000000009/WORD v2935>,<0x10000000008/~dmtmp v76>,<0x10000000007/client0 v42>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x10000000003/client.0 v39>,<0x10000000002/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d v47>,<0x10000000001/sv_0 v73>,<0x10000000000/qa v60>,<0x1/volumes v55>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-25T01:56:14.411 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-h[62279]: 2024-04-25T01:56:14.039+0000 7f9cae6c3700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000034b [4,head] /volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/HANDOUT.DOC auth v358 ap=1 s=19054 n(v0 rc2024-04-25T01:56:06.681190+0000 b19054 1=1+0) caps={15375=pAsLsXsFsc/-@1},l=15375(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d73e7f1700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x1000000034b:[<0x10000000208/HANDOUT.DOC v292>,<0x10000000009/WORD v2706>,<0x10000000008/~dmtmp v74>,<0x10000000007/client0 v58>,<0x10000000006/clients v36>,<0x10000000005/tmp v36>,<0x10000000003/client.0 v37>,<0x10000000002/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d v51>,<0x10000000001/sv_0 v71>,<0x10000000000/qa v57>,<0x1/volumes v53>]//[]","memoryvalue":"(3)0x1000000034b:[<0x10000000208/HANDOUT.DOC v358>,<0x10000000009/WORD v2935>,<0x10000000008/~dmtmp v76>,<0x10000000007/client0 v42>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x10000000003/client.0 v39>,<0x10000000002/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d v47>,<0x10000000001/sv_0 v73>,<0x10000000000/qa v60>,<0x1/volumes v55>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-25T01:56:14.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:14 smithi031 ceph-mon[29204]: scrub queued for path: / 2024-04-25T01:56:14.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:14 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:14.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:14 smithi031 ceph-mon[29204]: scrub summary: active paths [/] 2024-04-25T01:56:14.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:14 smithi031 ceph-mon[29204]: scrub summary: active paths [/] 2024-04-25T01:56:14.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:14 smithi031 ceph-mon[29204]: scrub summary: active paths [/] 2024-04-25T01:56:14.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:14 smithi031 ceph-mon[29204]: Scrub error on inode 0x1000000034f (/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/TIPS.DOC) see mds.h log and `damage ls` output for details 2024-04-25T01:56:14.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:14 smithi031 ceph-mon[29204]: Scrub error on inode 0x1000000034b (/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/HANDOUT.DOC) see mds.h log and `damage ls` output for details 2024-04-25T01:56:14.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:14 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:14.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:14 smithi031 ceph-mon[29204]: scrub summary: active paths [/] 2024-04-25T01:56:14.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:14 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:14.990 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:14 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-mon[32418]: scrub queued for path: / 2024-04-25T01:56:15.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-mon[32418]: scrub summary: active paths [/] 2024-04-25T01:56:15.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-mon[32418]: scrub summary: active paths [/] 2024-04-25T01:56:15.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-mon[32418]: scrub summary: active paths [/] 2024-04-25T01:56:15.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-mon[32418]: Scrub error on inode 0x1000000034f (/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/TIPS.DOC) see mds.h log and `damage ls` output for details 2024-04-25T01:56:15.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-mon[32418]: Scrub error on inode 0x1000000034b (/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/HANDOUT.DOC) see mds.h log and `damage ls` output for details 2024-04-25T01:56:15.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-mon[32418]: scrub summary: active paths [/] 2024-04-25T01:56:15.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.162 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:14 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:14 smithi134 ceph-mon[33098]: scrub queued for path: / 2024-04-25T01:56:15.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:14 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:14 smithi134 ceph-mon[33098]: scrub summary: active paths [/] 2024-04-25T01:56:15.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:14 smithi134 ceph-mon[33098]: scrub summary: active paths [/] 2024-04-25T01:56:15.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:14 smithi134 ceph-mon[33098]: scrub summary: active paths [/] 2024-04-25T01:56:15.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:14 smithi134 ceph-mon[33098]: Scrub error on inode 0x1000000034f (/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/TIPS.DOC) see mds.h log and `damage ls` output for details 2024-04-25T01:56:15.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:14 smithi134 ceph-mon[33098]: Scrub error on inode 0x1000000034b (/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/HANDOUT.DOC) see mds.h log and `damage ls` output for details 2024-04-25T01:56:15.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:14 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:14 smithi134 ceph-mon[33098]: scrub summary: active paths [/] 2024-04-25T01:56:15.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:14 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:14 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 11404 3.58 MB/sec warmup 22 sec latency 12.615 ms 2024-04-25T01:56:15.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:15 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:15 smithi031 ceph-mon[29204]: scrub summary: active paths [/] 2024-04-25T01:56:15.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:15 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:15 smithi031 ceph-mon[29204]: scrub summary: active paths [/] 2024-04-25T01:56:15.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:15 smithi031 ceph-mon[29204]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:15.989 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:15 smithi031 ceph-mon[29204]: pgmap v56: 97 pgs: 97 active+clean; 27 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-04-25T01:56:16.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:15 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:16.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:15 smithi040 ceph-mon[32418]: scrub summary: active paths [/] 2024-04-25T01:56:16.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:15 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:16.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:15 smithi040 ceph-mon[32418]: scrub summary: active paths [/] 2024-04-25T01:56:16.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:15 smithi040 ceph-mon[32418]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:16.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:15 smithi040 ceph-mon[32418]: pgmap v56: 97 pgs: 97 active+clean; 27 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-04-25T01:56:16.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:15 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:16.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:15 smithi134 ceph-mon[33098]: scrub summary: active paths [/] 2024-04-25T01:56:16.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:15 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:16.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:15 smithi134 ceph-mon[33098]: scrub summary: active paths [/] 2024-04-25T01:56:16.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:15 smithi134 ceph-mon[33098]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:16.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:15 smithi134 ceph-mon[33098]: pgmap v56: 97 pgs: 97 active+clean; 27 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-04-25T01:56:16.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 11900 3.50 MB/sec warmup 23 sec latency 18.457 ms 2024-04-25T01:56:17.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 12527 3.49 MB/sec warmup 24 sec latency 16.468 ms 2024-04-25T01:56:18.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:17 smithi040 ceph-mon[32418]: pgmap v57: 97 pgs: 97 active+clean; 27 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:56:18.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:17 smithi134 ceph-mon[33098]: pgmap v57: 97 pgs: 97 active+clean; 27 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:56:18.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:17 smithi031 ceph-mon[29204]: pgmap v57: 97 pgs: 97 active+clean; 27 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:56:18.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 13107 3.42 MB/sec warmup 25 sec latency 12.969 ms 2024-04-25T01:56:19.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:18 smithi040 ceph-mon[32418]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-25T01:56:19.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:18 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] up:active 2024-04-25T01:56:19.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:18 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:56:19.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:18 smithi134 ceph-mon[33098]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-25T01:56:19.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:18 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] up:active 2024-04-25T01:56:19.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:18 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:56:19.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:18 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a[29200]: 2024-04-25T01:56:18.738+0000 7fc418dd9700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-25T01:56:19.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:18 smithi031 ceph-mon[29204]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-25T01:56:19.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:18 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] up:active 2024-04-25T01:56:19.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:18 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:56:19.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 13838 3.52 MB/sec warmup 26 sec latency 18.679 ms 2024-04-25T01:56:20.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:19 smithi040 ceph-mon[32418]: pgmap v58: 97 pgs: 97 active+clean; 27 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 903 KiB/s wr, 96 op/s 2024-04-25T01:56:20.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:19 smithi134 ceph-mon[33098]: pgmap v58: 97 pgs: 97 active+clean; 27 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 903 KiB/s wr, 96 op/s 2024-04-25T01:56:20.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:19 smithi031 ceph-mon[29204]: pgmap v58: 97 pgs: 97 active+clean; 27 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 903 KiB/s wr, 96 op/s 2024-04-25T01:56:20.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 14269 3.55 MB/sec warmup 27 sec latency 17.933 ms 2024-04-25T01:56:21.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 14587 3.48 MB/sec warmup 28 sec latency 13.260 ms 2024-04-25T01:56:22.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:21 smithi040 ceph-mon[32418]: pgmap v59: 97 pgs: 97 active+clean; 28 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-04-25T01:56:22.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:21 smithi040 ceph-mon[32418]: scrub summary: idle 2024-04-25T01:56:22.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:21 smithi040 ceph-mon[32418]: scrub summary: idle 2024-04-25T01:56:22.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:21 smithi134 ceph-mon[33098]: pgmap v59: 97 pgs: 97 active+clean; 28 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-04-25T01:56:22.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:21 smithi134 ceph-mon[33098]: scrub summary: idle 2024-04-25T01:56:22.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:21 smithi134 ceph-mon[33098]: scrub summary: idle 2024-04-25T01:56:22.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:21 smithi031 ceph-mon[29204]: pgmap v59: 97 pgs: 97 active+clean; 28 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-04-25T01:56:22.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:21 smithi031 ceph-mon[29204]: scrub summary: idle 2024-04-25T01:56:22.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:21 smithi031 ceph-mon[29204]: scrub summary: idle 2024-04-25T01:56:22.299 INFO:tasks.workunit.client.0.smithi031.stdout: 1 14913 3.37 MB/sec warmup 29 sec latency 19.689 ms 2024-04-25T01:56:23.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:22 smithi040 ceph-mon[32418]: scrub summary: idle 2024-04-25T01:56:23.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:22 smithi134 ceph-mon[33098]: scrub summary: idle 2024-04-25T01:56:23.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:22 smithi031 ceph-mon[29204]: scrub summary: idle 2024-04-25T01:56:23.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 15427 3.32 MB/sec warmup 30 sec latency 16.614 ms 2024-04-25T01:56:24.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:23 smithi040 ceph-mon[32418]: pgmap v60: 97 pgs: 97 active+clean; 28 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T01:56:24.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:23 smithi134 ceph-mon[33098]: pgmap v60: 97 pgs: 97 active+clean; 28 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T01:56:24.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:23 smithi031 ceph-mon[29204]: pgmap v60: 97 pgs: 97 active+clean; 28 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T01:56:24.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16042 3.32 MB/sec warmup 31 sec latency 19.157 ms 2024-04-25T01:56:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.744+0000 7f388c09a640 1 -- 172.21.15.31:0/2344827142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3884106630 msgr2=0x7f3884106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.744+0000 7f388c09a640 1 --2- 172.21.15.31:0/2344827142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3884106630 0x7f3884106a10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3880009930 tx=0x7f388002a420 comp rx=0 tx=0).stop 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.744+0000 7f388c09a640 1 -- 172.21.15.31:0/2344827142 shutdown_connections 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.744+0000 7f388c09a640 1 --2- 172.21.15.31:0/2344827142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3884101030 0x7f388410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.744+0000 7f388c09a640 1 --2- 172.21.15.31:0/2344827142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3884100670 0x7f3884100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.744+0000 7f388c09a640 1 --2- 172.21.15.31:0/2344827142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3884106630 0x7f3884106a10 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.744+0000 7f388c09a640 1 -- 172.21.15.31:0/2344827142 >> 172.21.15.31:0/2344827142 conn(0x7f38840777c0 msgr2=0x7f3884077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.744+0000 7f388c09a640 1 -- 172.21.15.31:0/2344827142 shutdown_connections 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.744+0000 7f388c09a640 1 -- 172.21.15.31:0/2344827142 wait complete. 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.744+0000 7f388c09a640 1 Processor -- start 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f388c09a640 1 -- start start 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f388c09a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3884100670 0x7f38840fd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f388c09a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3884101030 0x7f38840fdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f388c09a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3884106630 0x7f38840fe080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f388c09a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38841108b0 con 0x7f3884100670 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f388c09a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3884110730 con 0x7f3884106630 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f388c09a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3884110a30 con 0x7f3884101030 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f3889e36640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3884100670 0x7f38840fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f3889e36640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3884100670 0x7f38840fd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40410/0 (socket says 172.21.15.31:40410) 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f3889e36640 1 -- 172.21.15.31:0/1693267446 learned_addr learned my addr 172.21.15.31:0/1693267446 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f388a637640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3884106630 0x7f38840fe080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f3889635640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3884101030 0x7f38840fdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f3889e36640 1 -- 172.21.15.31:0/1693267446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3884101030 msgr2=0x7f38840fdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f3889e36640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3884101030 0x7f38840fdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f3889e36640 1 -- 172.21.15.31:0/1693267446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3884106630 msgr2=0x7f38840fe080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f3889e36640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3884106630 0x7f38840fe080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f3889e36640 1 -- 172.21.15.31:0/1693267446 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38841a6470 con 0x7f3884100670 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f388a637640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3884106630 0x7f38840fe080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.745+0000 7f3889e36640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3884100670 0x7f38840fd600 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3880005940 tx=0x7f38800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.746+0000 7f387affd640 1 -- 172.21.15.31:0/1693267446 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388003f030 con 0x7f3884100670 2024-04-25T01:56:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.746+0000 7f387affd640 1 -- 172.21.15.31:0/1693267446 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3880003cb0 con 0x7f3884100670 2024-04-25T01:56:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.746+0000 7f387affd640 1 -- 172.21.15.31:0/1693267446 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880035070 con 0x7f3884100670 2024-04-25T01:56:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.746+0000 7f388c09a640 1 -- 172.21.15.31:0/1693267446 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38841a6700 con 0x7f3884100670 2024-04-25T01:56:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.746+0000 7f388c09a640 1 -- 172.21.15.31:0/1693267446 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38841a6bf0 con 0x7f3884100670 2024-04-25T01:56:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.748+0000 7f387affd640 1 -- 172.21.15.31:0/1693267446 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3880042030 con 0x7f3884100670 2024-04-25T01:56:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f388eda2740 1 Processor -- start 2024-04-25T01:56:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f387affd640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38700a2da0 0x7f38700a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f388eda2740 1 -- start start 2024-04-25T01:56:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f388a637640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38700a2da0 0x7f38700a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f388eda2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4249620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f388eda2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a4253cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f387affd640 1 -- 172.21.15.31:0/1693267446 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f38800ef880 con 0x7f3884100670 2024-04-25T01:56:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f388eda2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a4256620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f388eda2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5629a41adcb0 con 0x5629a4249240 2024-04-25T01:56:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f388eda2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5629a41565b0 con 0x5629a4249b60 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f388eda2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5629a41a9c70 con 0x5629a4254230 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.749+0000 7f3889e36640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4249620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f3889e36640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4249620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40422/0 (socket says 172.21.15.31:40422) 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f3889e36640 1 -- 172.21.15.31:0/4290959646 learned_addr learned my addr 172.21.15.31:0/4290959646 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f3889635640 1 --2- 172.21.15.31:0/4290959646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a4253cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f388a637640 1 --2- 172.21.15.31:0/4290959646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a4256620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f3889635640 1 -- 172.21.15.31:0/4290959646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 msgr2=0x5629a4256620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f3889635640 1 --2- 172.21.15.31:0/4290959646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a4256620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f3889635640 1 -- 172.21.15.31:0/4290959646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 msgr2=0x5629a4249620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f3889635640 1 --2- 172.21.15.31:0/4290959646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4249620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f3889635640 1 -- 172.21.15.31:0/4290959646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629a3ec2210 con 0x5629a4249b60 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f3889e36640 1 --2- 172.21.15.31:0/4290959646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4249620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f388a637640 1 --2- 172.21.15.31:0/4290959646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a4256620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f388a637640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38700a2da0 0x7f38700a5260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f387c00f5d0 tx=0x7f387c00f9d0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f3889635640 1 --2- 172.21.15.31:0/4290959646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a4253cf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f38740058c0 tx=0x7f387400f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6389ccd049cf1f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f386ffff640 1 -- 172.21.15.31:0/4290959646 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874004020 con 0x5629a4249b60 2024-04-25T01:56:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f386ffff640 1 -- 172.21.15.31:0/4290959646 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38740041c0 con 0x5629a4249b60 2024-04-25T01:56:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.750+0000 7f386ffff640 1 -- 172.21.15.31:0/4290959646 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874018680 con 0x5629a4249b60 2024-04-25T01:56:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.751+0000 7f388eda2740 1 -- 172.21.15.31:0/4290959646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 msgr2=0x5629a4253cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.751+0000 7f388eda2740 1 --2- 172.21.15.31:0/4290959646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a4253cf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f38740058c0 tx=0x7f387400f3c0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.751+0000 7f388eda2740 1 -- 172.21.15.31:0/4290959646 shutdown_connections 2024-04-25T01:56:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.751+0000 7f388eda2740 1 --2- 172.21.15.31:0/4290959646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a4256620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.751+0000 7f388eda2740 1 --2- 172.21.15.31:0/4290959646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a4253cf0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.751+0000 7f388eda2740 1 --2- 172.21.15.31:0/4290959646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4249620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.751+0000 7f388eda2740 1 -- 172.21.15.31:0/4290959646 >> 172.21.15.31:0/4290959646 conn(0x5629a4245560 msgr2=0x5629a4245940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.751+0000 7f388eda2740 1 -- 172.21.15.31:0/4290959646 shutdown_connections 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.751+0000 7f388eda2740 1 -- 172.21.15.31:0/4290959646 wait complete. 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f388eda2740 1 -- start start 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f388eda2740 1 Processor -- start 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f388eda2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4251170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f388eda2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a42516b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f388eda2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a4251bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f388eda2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5629a4257020 con 0x5629a4249240 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f388eda2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5629a4169b50 con 0x5629a4249b60 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f388eda2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5629a42571a0 con 0x5629a4254230 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f388eda2740 10 client.0 _ll_register_callbacks cb 0x5629a41a4ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f3889e36640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4251170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f3889635640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a42516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f3889e36640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4251170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40432/0 (socket says 172.21.15.31:40432) 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.752+0000 7f3889e36640 1 -- 172.21.15.31:0/1614973353 learned_addr learned my addr 172.21.15.31:0/1614973353 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f388a637640 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a4251bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f3889635640 1 -- 172.21.15.31:0/1614973353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 msgr2=0x5629a4251bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f3889635640 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a4251bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f3889635640 1 -- 172.21.15.31:0/1614973353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 msgr2=0x5629a4251170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f3889635640 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4251170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f3889635640 1 -- 172.21.15.31:0/1614973353 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629a3f8e620 con 0x5629a4249b60 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f3889e36640 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4251170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f3889635640 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a42516b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3874003ea0 tx=0x7f3874003cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f0fc657cd0f216c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f386d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f386d7fa640 1 -- 172.21.15.31:0/1614973353 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874027020 con 0x5629a4249b60 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f386d7fa640 1 -- 172.21.15.31:0/1614973353 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f3874004370 con 0x5629a4249b60 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f388eda2740 1 -- 172.21.15.31:0/1614973353 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38740041c0 con 0x5629a4249b60 2024-04-25T01:56:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f386d7fa640 1 -- 172.21.15.31:0/1614973353 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3874032050 con 0x5629a4249b60 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f388eda2740 10 client.25781 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.753+0000 7f388eda2740 1 -- 172.21.15.31:0/1614973353 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5629a4263150 con 0x5629a4249b60 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.754+0000 7f386d7fa640 1 -- 172.21.15.31:0/1614973353 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38740041c0 con 0x5629a4249b60 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.754+0000 7f386d7fa640 1 -- 172.21.15.31:0/1614973353 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f3874004960 con 0x5629a4249b60 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.754+0000 7f388eda2740 10 client.25781 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.754+0000 7f388eda2740 10 client.25781 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.754+0000 7f388eda2740 20 client.25781 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.754+0000 7f388eda2740 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5629a42645b0 0x5629a42669a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.754+0000 7f388eda2740 4 client.25781 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.754+0000 7f388eda2740 1 -- 172.21.15.31:0/1614973353 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629a41bb1f0 con 0x5629a42645b0 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.754+0000 7f3889e36640 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5629a42645b0 0x5629a42669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.754+0000 7f3889e36640 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5629a42645b0 0x5629a42669a0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=286e1bff0828417b server_cookie=ca6f7e2011c83563 in_seq=0 out_seq=0 2024-04-25T01:56:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.754+0000 7f386d7fa640 10 client.25781 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.755+0000 7f386d7fa640 1 -- 172.21.15.31:0/1614973353 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5629a41bb1f0 con 0x5629a42645b0 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f386d7fa640 10 client.25781 handle_command_reply: tid=0 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 1 client.25781 shutdown 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 1 -- 172.21.15.31:0/1614973353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 msgr2=0x5629a42516b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a42516b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3874003ea0 tx=0x7f3874003cc0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 1 -- 172.21.15.31:0/1614973353 shutdown_connections 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5629a42645b0 0x5629a42669a0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a4251bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a42516b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f386d7fa640 0 client.25781 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 1 --2- 172.21.15.31:0/1614973353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a4251170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 1 -- 172.21.15.31:0/1614973353 >> 172.21.15.31:0/1614973353 conn(0x5629a4245560 msgr2=0x5629a424b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 1 -- 172.21.15.31:0/1614973353 shutdown_connections 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 1 -- 172.21.15.31:0/1614973353 wait complete. 2024-04-25T01:56:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.756+0000 7f388eda2740 20 client.25781 trim_cache size 0 max 16384 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.773+0000 7f388eda2740 1 Processor -- start 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.773+0000 7f388eda2740 1 -- start start 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.773+0000 7f388eda2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a42538a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.773+0000 7f388eda2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a425fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.773+0000 7f388eda2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a42502d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.773+0000 7f388eda2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38841af400 con 0x5629a4249240 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.773+0000 7f388eda2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f38841b1000 con 0x5629a4249b60 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.773+0000 7f388eda2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f38841afad0 con 0x5629a4254230 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.773+0000 7f3889e36640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a42538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f3889635640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a425fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f3889e36640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a42538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40438/0 (socket says 172.21.15.31:40438) 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f3889e36640 1 -- 172.21.15.31:0/325308610 learned_addr learned my addr 172.21.15.31:0/325308610 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f388a637640 1 --2- 172.21.15.31:0/325308610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a42502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f3889635640 1 -- 172.21.15.31:0/325308610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 msgr2=0x5629a42502d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f3889635640 1 --2- 172.21.15.31:0/325308610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a42502d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f3889635640 1 -- 172.21.15.31:0/325308610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 msgr2=0x5629a42538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f3889635640 1 --2- 172.21.15.31:0/325308610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a42538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f3889635640 1 -- 172.21.15.31:0/325308610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629a4266d10 con 0x5629a4249b60 2024-04-25T01:56:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f3889e36640 1 --2- 172.21.15.31:0/325308610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a42538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f3889635640 1 --2- 172.21.15.31:0/325308610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a425fee0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3874004e20 tx=0x7f387400a590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=231f93714f9418fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f386f7fe640 1 -- 172.21.15.31:0/325308610 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387403a070 con 0x5629a4249b60 2024-04-25T01:56:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f386f7fe640 1 -- 172.21.15.31:0/325308610 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3874004b10 con 0x5629a4249b60 2024-04-25T01:56:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.774+0000 7f386f7fe640 1 -- 172.21.15.31:0/325308610 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874027070 con 0x5629a4249b60 2024-04-25T01:56:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.775+0000 7f388eda2740 1 -- 172.21.15.31:0/325308610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 msgr2=0x5629a425fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.775+0000 7f388eda2740 1 --2- 172.21.15.31:0/325308610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a425fee0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3874004e20 tx=0x7f387400a590 comp rx=0 tx=0).stop 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.775+0000 7f388eda2740 1 -- 172.21.15.31:0/325308610 shutdown_connections 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.775+0000 7f388eda2740 1 --2- 172.21.15.31:0/325308610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a42502d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.775+0000 7f388eda2740 1 --2- 172.21.15.31:0/325308610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249b60 0x5629a425fee0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.775+0000 7f388eda2740 1 --2- 172.21.15.31:0/325308610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249240 0x5629a42538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.775+0000 7f388eda2740 1 -- 172.21.15.31:0/325308610 >> 172.21.15.31:0/325308610 conn(0x5629a4245560 msgr2=0x5629a4243d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.775+0000 7f388eda2740 1 -- 172.21.15.31:0/325308610 shutdown_connections 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.775+0000 7f388eda2740 1 -- 172.21.15.31:0/325308610 wait complete. 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.775+0000 7f388eda2740 1 -- start start 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f388eda2740 1 Processor -- start 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f388eda2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249240 0x5629a425c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f388eda2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249b60 0x5629a425c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f388eda2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a424d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f388eda2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38841b0870 con 0x5629a4249b60 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f388eda2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f38841b01a0 con 0x5629a4249240 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f388eda2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f38841ad3e0 con 0x5629a4254230 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f388eda2740 10 client.0 _ll_register_callbacks cb 0x7f38800044e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f3889635640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249b60 0x5629a425c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f3889e36640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249240 0x5629a425c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f3889635640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249b60 0x5629a425c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40440/0 (socket says 172.21.15.31:40440) 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f3889635640 1 -- 172.21.15.31:0/1138421210 learned_addr learned my addr 172.21.15.31:0/1138421210 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f388a637640 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a424d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f3889635640 1 -- 172.21.15.31:0/1138421210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 msgr2=0x5629a424d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f3889635640 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a424d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f3889635640 1 -- 172.21.15.31:0/1138421210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249240 msgr2=0x5629a425c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f3889635640 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249240 0x5629a425c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f3889635640 1 -- 172.21.15.31:0/1138421210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629a424ddc0 con 0x5629a4249b60 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.776+0000 7f3889e36640 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249240 0x5629a425c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f3889635640 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249b60 0x5629a425c930 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f387401a040 tx=0x7f3874003ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a97dfe407912de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f386d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f386d7fa640 1 -- 172.21.15.31:0/1138421210 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387403a070 con 0x5629a4249b60 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f386d7fa640 1 -- 172.21.15.31:0/1138421210 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f3874004660 con 0x5629a4249b60 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f388eda2740 1 -- 172.21.15.31:0/1138421210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5629a41b5540 con 0x5629a4249b60 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f386d7fa640 1 -- 172.21.15.31:0/1138421210 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3874027040 con 0x5629a4249b60 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f388eda2740 10 client.16185 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f388eda2740 1 -- 172.21.15.31:0/1138421210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5629a42ca540 con 0x5629a4249b60 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f386d7fa640 1 -- 172.21.15.31:0/1138421210 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5629a41b5540 con 0x5629a4249b60 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f386d7fa640 1 -- 172.21.15.31:0/1138421210 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f3874023470 con 0x5629a4249b60 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f388eda2740 10 client.16185 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f388eda2740 10 client.16185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f388eda2740 20 client.16185 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f388eda2740 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5629a42cac10 0x5629a42cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f388eda2740 4 client.16185 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.777+0000 7f388eda2740 1 -- 172.21.15.31:0/1138421210 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5629a41338e0 con 0x5629a42cac10 2024-04-25T01:56:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.778+0000 7f3889e36640 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5629a42cac10 0x5629a42cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.778+0000 7f3889e36640 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5629a42cac10 0x5629a42cd030 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb6c1ec18100b5df server_cookie=e6abe4b623bcfd5b in_seq=0 out_seq=0 2024-04-25T01:56:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.778+0000 7f386d7fa640 10 client.16185 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.801+0000 7f386d7fa640 1 -- 172.21.15.31:0/1138421210 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5629a41338e0 con 0x5629a42cac10 2024-04-25T01:56:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.801+0000 7f386d7fa640 10 client.16185 handle_command_reply: tid=0 2024-04-25T01:56:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.801+0000 7f388eda2740 1 client.16185 shutdown 2024-04-25T01:56:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.801+0000 7f388eda2740 1 -- 172.21.15.31:0/1138421210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249b60 msgr2=0x5629a425c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.801+0000 7f388eda2740 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249b60 0x5629a425c930 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f387401a040 tx=0x7f3874003ed0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388eda2740 1 -- 172.21.15.31:0/1138421210 shutdown_connections 2024-04-25T01:56:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388eda2740 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5629a42cac10 0x5629a42cd030 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388eda2740 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5629a4254230 0x5629a424d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388eda2740 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629a4249b60 0x5629a425c930 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388eda2740 1 --2- 172.21.15.31:0/1138421210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5629a4249240 0x5629a425c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f386d7fa640 0 client.16185 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388eda2740 1 -- 172.21.15.31:0/1138421210 >> 172.21.15.31:0/1138421210 conn(0x5629a4245560 msgr2=0x5629a42445c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388eda2740 1 -- 172.21.15.31:0/1138421210 shutdown_connections 2024-04-25T01:56:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388eda2740 1 -- 172.21.15.31:0/1138421210 wait complete. 2024-04-25T01:56:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388eda2740 20 client.16185 trim_cache size 0 max 16384 2024-04-25T01:56:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:56:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:56:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:56:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:56:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388c09a640 1 -- 172.21.15.31:0/1693267446 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38700a2da0 msgr2=0x7f38700a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388c09a640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38700a2da0 0x7f38700a5260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f387c00f5d0 tx=0x7f387c00f9d0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388c09a640 1 -- 172.21.15.31:0/1693267446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3884100670 msgr2=0x7f38840fd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388c09a640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3884100670 0x7f38840fd600 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3880005940 tx=0x7f38800042f0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388c09a640 1 -- 172.21.15.31:0/1693267446 shutdown_connections 2024-04-25T01:56:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388c09a640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38700a2da0 0x7f38700a5260 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388c09a640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3884106630 0x7f38840fe080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388c09a640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3884101030 0x7f38840fdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388c09a640 1 --2- 172.21.15.31:0/1693267446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3884100670 0x7f38840fd600 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388c09a640 1 -- 172.21.15.31:0/1693267446 >> 172.21.15.31:0/1693267446 conn(0x7f38840777c0 msgr2=0x7f3884069230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.802+0000 7f388c09a640 1 -- 172.21.15.31:0/1693267446 shutdown_connections 2024-04-25T01:56:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:24.803+0000 7f388c09a640 1 -- 172.21.15.31:0/1693267446 wait complete. 2024-04-25T01:56:25.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 16596 3.26 MB/sec warmup 32 sec latency 18.633 ms 2024-04-25T01:56:26.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:25 smithi040 ceph-mon[32418]: pgmap v61: 97 pgs: 97 active+clean; 28 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-04-25T01:56:26.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:25 smithi134 ceph-mon[33098]: pgmap v61: 97 pgs: 97 active+clean; 28 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-04-25T01:56:26.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:25 smithi031 ceph-mon[29204]: pgmap v61: 97 pgs: 97 active+clean; 28 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-04-25T01:56:26.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 17201 3.33 MB/sec warmup 33 sec latency 19.304 ms 2024-04-25T01:56:27.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 17715 3.38 MB/sec warmup 34 sec latency 18.198 ms 2024-04-25T01:56:28.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:27 smithi040 ceph-mon[32418]: pgmap v62: 97 pgs: 97 active+clean; 28 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-04-25T01:56:28.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:27 smithi134 ceph-mon[33098]: pgmap v62: 97 pgs: 97 active+clean; 28 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-04-25T01:56:28.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:27 smithi031 ceph-mon[29204]: pgmap v62: 97 pgs: 97 active+clean; 28 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-04-25T01:56:28.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 18079 3.33 MB/sec warmup 35 sec latency 14.178 ms 2024-04-25T01:56:29.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 18346 3.24 MB/sec warmup 36 sec latency 18.221 ms 2024-04-25T01:56:30.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:29 smithi040 ceph-mon[32418]: pgmap v63: 97 pgs: 97 active+clean; 28 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.0 MiB/s wr, 124 op/s 2024-04-25T01:56:30.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:29 smithi134 ceph-mon[33098]: pgmap v63: 97 pgs: 97 active+clean; 28 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.0 MiB/s wr, 124 op/s 2024-04-25T01:56:30.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:29 smithi031 ceph-mon[29204]: pgmap v63: 97 pgs: 97 active+clean; 28 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.0 MiB/s wr, 124 op/s 2024-04-25T01:56:30.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 18865 3.20 MB/sec warmup 37 sec latency 12.443 ms 2024-04-25T01:56:31.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 19436 3.21 MB/sec warmup 38 sec latency 18.436 ms 2024-04-25T01:56:32.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:31 smithi040 ceph-mon[32418]: pgmap v64: 97 pgs: 97 active+clean; 32 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 1.7 MiB/s wr, 188 op/s 2024-04-25T01:56:32.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:31 smithi134 ceph-mon[33098]: pgmap v64: 97 pgs: 97 active+clean; 32 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 1.7 MiB/s wr, 188 op/s 2024-04-25T01:56:32.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:31 smithi031 ceph-mon[29204]: pgmap v64: 97 pgs: 97 active+clean; 32 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 1.7 MiB/s wr, 188 op/s 2024-04-25T01:56:32.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 20127 3.17 MB/sec warmup 39 sec latency 12.494 ms 2024-04-25T01:56:33.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 20705 3.23 MB/sec warmup 40 sec latency 19.453 ms 2024-04-25T01:56:34.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:33 smithi040 ceph-mon[32418]: pgmap v65: 97 pgs: 97 active+clean; 32 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-04-25T01:56:34.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:33 smithi134 ceph-mon[33098]: pgmap v65: 97 pgs: 97 active+clean; 32 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-04-25T01:56:34.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:33 smithi031 ceph-mon[29204]: pgmap v65: 97 pgs: 97 active+clean; 32 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-04-25T01:56:34.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 21166 3.20 MB/sec warmup 41 sec latency 17.715 ms 2024-04-25T01:56:35.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 21627 3.22 MB/sec warmup 42 sec latency 17.788 ms 2024-04-25T01:56:36.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:35 smithi040 ceph-mon[32418]: pgmap v66: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.7 MiB/s wr, 188 op/s 2024-04-25T01:56:36.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:35 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:56:36.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:35 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:36.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:35 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:36.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:35 smithi134 ceph-mon[33098]: pgmap v66: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.7 MiB/s wr, 188 op/s 2024-04-25T01:56:36.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:35 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:56:36.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:35 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:36.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:35 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:36.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:35 smithi031 ceph-mon[29204]: pgmap v66: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.7 MiB/s wr, 188 op/s 2024-04-25T01:56:36.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:35 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:56:36.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:35 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:36.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:35 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:36.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 21875 3.16 MB/sec warmup 43 sec latency 14.301 ms 2024-04-25T01:56:37.140 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:36 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:37.140 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:36 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:37.141 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:36 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:56:37.141 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:36 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:56:37.141 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:36 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:37.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:36 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:37.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:36 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:37.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:36 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:56:37.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:36 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:56:37.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:36 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:37.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:36 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:37.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:36 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:37.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:36 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:56:37.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:36 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:56:37.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:36 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:56:37.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 22303 3.10 MB/sec warmup 44 sec latency 18.909 ms 2024-04-25T01:56:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.905+0000 7f6dc2363640 1 -- 172.21.15.31:0/3922102202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10e100 msgr2=0x7f6dbc1104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.905+0000 7f6dc2363640 1 --2- 172.21.15.31:0/3922102202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10e100 0x7f6dbc1104f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6db4009ca0 tx=0x7f6db402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.905+0000 7f6dc2363640 1 -- 172.21.15.31:0/3922102202 shutdown_connections 2024-04-25T01:56:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.905+0000 7f6dc2363640 1 --2- 172.21.15.31:0/3922102202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10e100 0x7f6dbc1104f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.905+0000 7f6dc2363640 1 --2- 172.21.15.31:0/3922102202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc100e70 0x7f6dbc10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.905+0000 7f6dc2363640 1 --2- 172.21.15.31:0/3922102202 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6dbc100550 0x7f6dbc100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.905+0000 7f6dc2363640 1 -- 172.21.15.31:0/3922102202 >> 172.21.15.31:0/3922102202 conn(0x7f6dbc0777c0 msgr2=0x7f6dbc077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.905+0000 7f6dc2363640 1 -- 172.21.15.31:0/3922102202 shutdown_connections 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.905+0000 7f6dc2363640 1 -- 172.21.15.31:0/3922102202 wait complete. 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.905+0000 7f6dc2363640 1 Processor -- start 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.905+0000 7f6dc2363640 1 -- start start 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc2363640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc100550 0x7f6dbc19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc2363640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6dbc100e70 0x7f6dbc19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc2363640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10e100 0x7f6dbc19e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc2363640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc113c30 con 0x7f6dbc10e100 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc2363640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc113ab0 con 0x7f6dbc100550 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc2363640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc113db0 con 0x7f6dbc100e70 2024-04-25T01:56:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc0900640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10e100 0x7f6dbc19e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dbbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc100550 0x7f6dbc19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc0900640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10e100 0x7f6dbc19e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46310/0 (socket says 172.21.15.31:46310) 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc0900640 1 -- 172.21.15.31:0/3517595 learned_addr learned my addr 172.21.15.31:0/3517595 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6db37fe640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6dbc100e70 0x7f6dbc19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc0900640 1 -- 172.21.15.31:0/3517595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6dbc100e70 msgr2=0x7f6dbc19a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc0900640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6dbc100e70 0x7f6dbc19a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc0900640 1 -- 172.21.15.31:0/3517595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc100550 msgr2=0x7f6dbc19a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc0900640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc100550 0x7f6dbc19a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc0900640 1 -- 172.21.15.31:0/3517595 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dbc19f020 con 0x7f6dbc10e100 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc100550 0x7f6dbc19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc0900640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10e100 0x7f6dbc19e8f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6db4031040 tx=0x7f6db4032f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6db9ffb640 1 -- 172.21.15.31:0/3517595 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db4003b70 con 0x7f6dbc10e100 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6db9ffb640 1 -- 172.21.15.31:0/3517595 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6db4003d10 con 0x7f6dbc10e100 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6dc2363640 1 -- 172.21.15.31:0/3517595 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dbc19f2b0 con 0x7f6dbc10e100 2024-04-25T01:56:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.906+0000 7f6db9ffb640 1 -- 172.21.15.31:0/3517595 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db4030ad0 con 0x7f6dbc10e100 2024-04-25T01:56:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.907+0000 7f6dc2363640 1 -- 172.21.15.31:0/3517595 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dbc1a6ba0 con 0x7f6dbc10e100 2024-04-25T01:56:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.909+0000 7f6db9ffb640 1 -- 172.21.15.31:0/3517595 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6db4030c70 con 0x7f6dbc10e100 2024-04-25T01:56:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dc5076740 1 Processor -- start 2024-04-25T01:56:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dc5076740 1 -- start start 2024-04-25T01:56:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67240 0x55fd65c67620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c71cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c72230 0x55fd65c74620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dc5076740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fd65bcbcb0 con 0x55fd65c67b60 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dc5076740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fd65b745b0 con 0x55fd65c72230 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dc5076740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fd65bc7c70 con 0x55fd65c67240 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dbbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c71cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dbbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c71cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46322/0 (socket says 172.21.15.31:46322) 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3043508716 learned_addr learned my addr 172.21.15.31:0/3043508716 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6db37fe640 1 --2- 172.21.15.31:0/3043508716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c72230 0x55fd65c74620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6db37fe640 1 --2- 172.21.15.31:0/3043508716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67240 0x55fd65c67620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6db9ffb640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6da80a2e20 0x7f6da80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3043508716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67240 msgr2=0x55fd65c67620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/3043508716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67240 0x55fd65c67620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3043508716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c72230 msgr2=0x55fd65c74620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/3043508716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c72230 0x55fd65c74620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3043508716 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd658e0210 con 0x55fd65c67b60 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.910+0000 7f6dc0900640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6da80a2e20 0x7f6da80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.911+0000 7f6db9ffb640 1 -- 172.21.15.31:0/3517595 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f6db40f0660 con 0x7f6dbc10e100 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.911+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/3043508716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c71cf0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0115f0 tx=0x7f6dac011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e394c086e8d282 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.911+0000 7f6db27fc640 1 -- 172.21.15.31:0/3043508716 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dac01e070 con 0x55fd65c67b60 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.911+0000 7f6db27fc640 1 -- 172.21.15.31:0/3043508716 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dac00faf0 con 0x55fd65c67b60 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.911+0000 7f6db27fc640 1 -- 172.21.15.31:0/3043508716 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dac019b60 con 0x55fd65c67b60 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.912+0000 7f6dc0900640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6da80a2e20 0x7f6da80a52e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6db40edfc0 tx=0x7f6db408bc10 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.912+0000 7f6dc5076740 1 -- 172.21.15.31:0/3043508716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 msgr2=0x55fd65c71cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.912+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3043508716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c71cf0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0115f0 tx=0x7f6dac011ab0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.912+0000 7f6dc5076740 1 -- 172.21.15.31:0/3043508716 shutdown_connections 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.912+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3043508716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c72230 0x55fd65c74620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.912+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3043508716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c71cf0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.912+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3043508716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67240 0x55fd65c67620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.912+0000 7f6dc5076740 1 -- 172.21.15.31:0/3043508716 >> 172.21.15.31:0/3043508716 conn(0x55fd65c63560 msgr2=0x55fd65c63940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.912+0000 7f6dc5076740 1 -- 172.21.15.31:0/3043508716 shutdown_connections 2024-04-25T01:56:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.912+0000 7f6dc5076740 1 -- 172.21.15.31:0/3043508716 wait complete. 2024-04-25T01:56:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.912+0000 7f6dc5076740 1 -- start start 2024-04-25T01:56:37.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dc5076740 1 Processor -- start 2024-04-25T01:56:37.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c67240 0x55fd65c6f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c6f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c72230 0x55fd65c6fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dc5076740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fd65c75020 con 0x55fd65c67b60 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dc5076740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fd65b87b50 con 0x55fd65c67240 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dc5076740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fd65c751a0 con 0x55fd65c72230 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dc5076740 10 client.0 _ll_register_callbacks cb 0x55fd65bc2ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dbbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c6f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dbbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c6f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46330/0 (socket says 172.21.15.31:46330) 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3484275151 learned_addr learned my addr 172.21.15.31:0/3484275151 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6db37fe640 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c67240 0x55fd65c6f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6db37fe640 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c72230 0x55fd65c6fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3484275151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c72230 msgr2=0x55fd65c6fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c72230 0x55fd65c6fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3484275151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c67240 msgr2=0x55fd65c6f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c67240 0x55fd65c6f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3484275151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd659ac620 con 0x55fd65c67b60 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c6f670 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6dac01c040 tx=0x7f6dac00f8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7501c2d3b9368d8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.913+0000 7f6d9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/3484275151 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dac01e040 con 0x55fd65c67b60 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/3484275151 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f6dac004040 con 0x55fd65c67b60 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6dc5076740 1 -- 172.21.15.31:0/3484275151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dac00faf0 con 0x55fd65c67b60 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/3484275151 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6dac030050 con 0x55fd65c67b60 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6dc5076740 10 client.16197 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6dc5076740 1 -- 172.21.15.31:0/3484275151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fd65c811c0 con 0x55fd65c67b60 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/3484275151 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dac00faf0 con 0x55fd65c67b60 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/3484275151 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f6dac021070 con 0x55fd65c67b60 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6dc5076740 10 client.16197 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6dc5076740 10 client.16197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6dc5076740 20 client.16197 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd65c825c0 0x55fd65c849b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6dc5076740 4 client.16197 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6dc5076740 1 -- 172.21.15.31:0/3484275151 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd65bd91f0 con 0x55fd65c825c0 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.914+0000 7f6db37fe640 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd65c825c0 0x55fd65c849b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.915+0000 7f6db37fe640 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd65c825c0 0x55fd65c849b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b6e2ec0afc85fb5 server_cookie=85b749b7f1b6cbe4 in_seq=0 out_seq=0 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.915+0000 7f6d9b7fe640 10 client.16197 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/3484275151 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd65bd91f0 con 0x55fd65c825c0 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6d9b7fe640 10 client.16197 handle_command_reply: tid=0 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 1 client.16197 shutdown 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 1 -- 172.21.15.31:0/3484275151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 msgr2=0x55fd65c6f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c6f670 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6dac01c040 tx=0x7f6dac00f8f0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 1 -- 172.21.15.31:0/3484275151 shutdown_connections 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd65c825c0 0x55fd65c849b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6d9b7fe640 0 client.16197 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c72230 0x55fd65c6fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c6f670 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3484275151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c67240 0x55fd65c6f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 1 -- 172.21.15.31:0/3484275151 >> 172.21.15.31:0/3484275151 conn(0x55fd65c63560 msgr2=0x55fd65c69260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 1 -- 172.21.15.31:0/3484275151 shutdown_connections 2024-04-25T01:56:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 1 -- 172.21.15.31:0/3484275151 wait complete. 2024-04-25T01:56:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.916+0000 7f6dc5076740 20 client.16197 trim_cache size 0 max 16384 2024-04-25T01:56:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.932+0000 7f6dc5076740 1 Processor -- start 2024-04-25T01:56:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.932+0000 7f6dc5076740 1 -- start start 2024-04-25T01:56:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.932+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67240 0x55fd65c71860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.932+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c7df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.932+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c72230 0x55fd65c6e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.932+0000 7f6dc5076740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc1af4a0 con 0x55fd65c67b60 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.932+0000 7f6dc5076740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc1b10a0 con 0x55fd65c72230 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.932+0000 7f6dc5076740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc1afb70 con 0x55fd65c67240 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.932+0000 7f6dbbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c7df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.932+0000 7f6dbbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c7df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46346/0 (socket says 172.21.15.31:46346) 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.932+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3765447694 learned_addr learned my addr 172.21.15.31:0/3765447694 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6db37fe640 1 --2- 172.21.15.31:0/3765447694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c72230 0x55fd65c6e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6db37fe640 1 --2- 172.21.15.31:0/3765447694 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67240 0x55fd65c71860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3765447694 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67240 msgr2=0x55fd65c71860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/3765447694 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67240 0x55fd65c71860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3765447694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c72230 msgr2=0x55fd65c6e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/3765447694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c72230 0x55fd65c6e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6dbbfff640 1 -- 172.21.15.31:0/3765447694 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd65c67990 con 0x55fd65c67b60 2024-04-25T01:56:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6db37fe640 1 --2- 172.21.15.31:0/3765447694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c72230 0x55fd65c6e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/3765447694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c7df10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6dac000c00 tx=0x7f6dac00f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f0873f74bff67c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6db1ffb640 1 -- 172.21.15.31:0/3765447694 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dac01e040 con 0x55fd65c67b60 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6db1ffb640 1 -- 172.21.15.31:0/3765447694 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dac016070 con 0x55fd65c67b60 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.933+0000 7f6db1ffb640 1 -- 172.21.15.31:0/3765447694 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dac0198b0 con 0x55fd65c67b60 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.934+0000 7f6dc5076740 1 -- 172.21.15.31:0/3765447694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 msgr2=0x55fd65c7df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.934+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3765447694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c7df10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6dac000c00 tx=0x7f6dac00f850 comp rx=0 tx=0).stop 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.934+0000 7f6dc5076740 1 -- 172.21.15.31:0/3765447694 shutdown_connections 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.934+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3765447694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c72230 0x55fd65c6e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.934+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3765447694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c67b60 0x55fd65c7df10 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.934+0000 7f6dc5076740 1 --2- 172.21.15.31:0/3765447694 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67240 0x55fd65c71860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.934+0000 7f6dc5076740 1 -- 172.21.15.31:0/3765447694 >> 172.21.15.31:0/3765447694 conn(0x55fd65c63560 msgr2=0x55fd65c63940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.934+0000 7f6dc5076740 1 -- 172.21.15.31:0/3765447694 shutdown_connections 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.934+0000 7f6dc5076740 1 -- 172.21.15.31:0/3765447694 wait complete. 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.934+0000 7f6dc5076740 1 -- start start 2024-04-25T01:56:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.934+0000 7f6dc5076740 1 Processor -- start 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c67240 0x55fd65c753e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67b60 0x55fd65c75920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dc5076740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c72230 0x55fd65ce6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dc5076740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc1b0910 con 0x55fd65c72230 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dc5076740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc1b0240 con 0x55fd65c67240 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dc5076740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc1ad480 con 0x55fd65c67b60 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dc5076740 10 client.0 _ll_register_callbacks cb 0x7f6dac01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dbbfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67b60 0x55fd65c75920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c67240 0x55fd65c753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dbbfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67b60 0x55fd65c75920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:40240/0 (socket says 172.21.15.31:40240) 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dbbfff640 1 -- 172.21.15.31:0/1549959947 learned_addr learned my addr 172.21.15.31:0/1549959947 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6db37fe640 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c72230 0x55fd65ce6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dbbfff640 1 -- 172.21.15.31:0/1549959947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c67240 msgr2=0x55fd65c753e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c67240 0x55fd65c753e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dbbfff640 1 -- 172.21.15.31:0/1549959947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c72230 msgr2=0x55fd65ce6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c72230 0x55fd65ce6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dbbfff640 1 -- 172.21.15.31:0/1549959947 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd65c761b0 con 0x55fd65c67b60 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6dbbfff640 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67b60 0x55fd65c75920 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6dac010010 tx=0x7f6dac00fa20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dc5116cf2e3374a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.935+0000 7f6d9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/1549959947 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dac004240 con 0x55fd65c67b60 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6dc5076740 1 -- 172.21.15.31:0/1549959947 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dac016070 con 0x55fd65c67b60 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/1549959947 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f6dac019410 con 0x55fd65c67b60 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/1549959947 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6dac019630 con 0x55fd65c67b60 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6dc5076740 10 client.25663 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6dc5076740 1 -- 172.21.15.31:0/1549959947 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fd65ce8060 con 0x55fd65c67b60 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/1549959947 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dac016070 con 0x55fd65c67b60 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/1549959947 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f6dac021070 con 0x55fd65c67b60 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6dc5076740 10 client.25663 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6dc5076740 10 client.25663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6dc5076740 20 client.25663 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6dc5076740 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd65ce8830 0x55fd65ceac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6dc5076740 4 client.25663 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:56:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6dc5076740 1 -- 172.21.15.31:0/1549959947 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fd65bd3540 con 0x55fd65ce8830 2024-04-25T01:56:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.936+0000 7f6db37fe640 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd65ce8830 0x55fd65ceac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.937+0000 7f6d9b7fe640 10 client.25663 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.937+0000 7f6db37fe640 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd65ce8830 0x55fd65ceac50 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=132f27602cc7d714 server_cookie=67f951d622f3dcc1 in_seq=0 out_seq=0 2024-04-25T01:56:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6d9b7fe640 1 -- 172.21.15.31:0/1549959947 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fd65bd3540 con 0x55fd65ce8830 2024-04-25T01:56:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6d9b7fe640 10 client.25663 handle_command_reply: tid=0 2024-04-25T01:56:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 1 client.25663 shutdown 2024-04-25T01:56:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 1 -- 172.21.15.31:0/1549959947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67b60 msgr2=0x55fd65c75920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67b60 0x55fd65c75920 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6dac010010 tx=0x7f6dac00fa20 comp rx=0 tx=0).stop 2024-04-25T01:56:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 1 -- 172.21.15.31:0/1549959947 shutdown_connections 2024-04-25T01:56:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd65ce8830 0x55fd65ceac50 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd65c72230 0x55fd65ce6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6d9b7fe640 0 client.25663 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd65c67b60 0x55fd65c75920 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 1 --2- 172.21.15.31:0/1549959947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd65c67240 0x55fd65c753e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 1 -- 172.21.15.31:0/1549959947 >> 172.21.15.31:0/1549959947 conn(0x55fd65c63560 msgr2=0x55fd65c6e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 1 -- 172.21.15.31:0/1549959947 shutdown_connections 2024-04-25T01:56:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 1 -- 172.21.15.31:0/1549959947 wait complete. 2024-04-25T01:56:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.960+0000 7f6dc5076740 20 client.25663 trim_cache size 0 max 16384 2024-04-25T01:56:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:56:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:56:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:56:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:56:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 -- 172.21.15.31:0/3517595 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6da80a2e20 msgr2=0x7f6da80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6da80a2e20 0x7f6da80a52e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6db40edfc0 tx=0x7f6db408bc10 comp rx=0 tx=0).stop 2024-04-25T01:56:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 -- 172.21.15.31:0/3517595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10e100 msgr2=0x7f6dbc19e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10e100 0x7f6dbc19e8f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6db4031040 tx=0x7f6db4032f80 comp rx=0 tx=0).stop 2024-04-25T01:56:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 -- 172.21.15.31:0/3517595 shutdown_connections 2024-04-25T01:56:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6da80a2e20 0x7f6da80a52e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dbc10e100 0x7f6dbc19e8f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6dbc100e70 0x7f6dbc19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 --2- 172.21.15.31:0/3517595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dbc100550 0x7f6dbc19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 -- 172.21.15.31:0/3517595 >> 172.21.15.31:0/3517595 conn(0x7f6dbc0777c0 msgr2=0x7f6dbc10f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 -- 172.21.15.31:0/3517595 shutdown_connections 2024-04-25T01:56:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:37.961+0000 7f6dc2363640 1 -- 172.21.15.31:0/3517595 wait complete. 2024-04-25T01:56:38.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:37 smithi040 ceph-mon[32418]: pgmap v67: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-25T01:56:38.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:37 smithi134 ceph-mon[33098]: pgmap v67: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-25T01:56:38.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:37 smithi031 ceph-mon[29204]: pgmap v67: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-25T01:56:38.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 22857 3.13 MB/sec warmup 45 sec latency 24.760 ms 2024-04-25T01:56:39.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 23503 3.08 MB/sec warmup 46 sec latency 18.808 ms 2024-04-25T01:56:40.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:39 smithi040 ceph-mon[32418]: pgmap v68: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-25T01:56:40.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:39 smithi134 ceph-mon[33098]: pgmap v68: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-25T01:56:40.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:39 smithi031 ceph-mon[29204]: pgmap v68: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-25T01:56:40.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 24043 3.12 MB/sec warmup 47 sec latency 18.768 ms 2024-04-25T01:56:41.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 24635 3.13 MB/sec warmup 48 sec latency 19.217 ms 2024-04-25T01:56:42.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:41 smithi040 ceph-mon[32418]: pgmap v69: 97 pgs: 97 active+clean; 28 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.7 MiB/s wr, 190 op/s 2024-04-25T01:56:42.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:41 smithi134 ceph-mon[33098]: pgmap v69: 97 pgs: 97 active+clean; 28 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.7 MiB/s wr, 190 op/s 2024-04-25T01:56:42.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:41 smithi031 ceph-mon[29204]: pgmap v69: 97 pgs: 97 active+clean; 28 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.7 MiB/s wr, 190 op/s 2024-04-25T01:56:42.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 25095 3.15 MB/sec warmup 49 sec latency 19.540 ms 2024-04-25T01:56:43.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 25340 3.10 MB/sec warmup 50 sec latency 13.760 ms 2024-04-25T01:56:44.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:43 smithi040 ceph-mon[32418]: pgmap v70: 97 pgs: 97 active+clean; 28 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-25T01:56:44.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:43 smithi134 ceph-mon[33098]: pgmap v70: 97 pgs: 97 active+clean; 28 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-25T01:56:44.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:56:44.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:56:44.222 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:43 smithi031 ceph-mon[29204]: pgmap v70: 97 pgs: 97 active+clean; 28 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-25T01:56:44.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 25748 3.05 MB/sec warmup 51 sec latency 18.246 ms 2024-04-25T01:56:44.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.312+0000 7f831ece1640 1 -- 172.21.15.31:0/587839161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318069d40 msgr2=0x7f831806a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.312+0000 7f831ece1640 1 --2- 172.21.15.31:0/587839161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318069d40 0x7f831806a1c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8308009860 tx=0x7f830802a3d0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.312+0000 7f831ece1640 1 -- 172.21.15.31:0/587839161 shutdown_connections 2024-04-25T01:56:44.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.312+0000 7f831ece1640 1 --2- 172.21.15.31:0/587839161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f831806a700 0x7f831810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.312+0000 7f831ece1640 1 --2- 172.21.15.31:0/587839161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318069d40 0x7f831806a1c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.312+0000 7f831ece1640 1 --2- 172.21.15.31:0/587839161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8318107940 0x7f8318107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.312+0000 7f831ece1640 1 -- 172.21.15.31:0/587839161 >> 172.21.15.31:0/587839161 conn(0x7f8318076440 msgr2=0x7f8318076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.312+0000 7f831ece1640 1 -- 172.21.15.31:0/587839161 shutdown_connections 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.312+0000 7f831ece1640 1 -- 172.21.15.31:0/587839161 wait complete. 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ece1640 1 Processor -- start 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ece1640 1 -- start start 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ece1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318069d40 0x7f8318100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ece1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831806a700 0x7f8318100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ece1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8318107940 0x7f83181014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ece1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f83181119a0 con 0x7f831806a700 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ece1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8318111820 con 0x7f8318069d40 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ece1640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8318111b20 con 0x7f8318107940 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f8317fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831806a700 0x7f8318100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ca7d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318069d40 0x7f8318100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ca7d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318069d40 0x7f8318100a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33566/0 (socket says 172.21.15.31:33566) 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ca7d640 1 -- 172.21.15.31:0/2341207704 learned_addr learned my addr 172.21.15.31:0/2341207704 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831d27e640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8318107940 0x7f83181014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ca7d640 1 -- 172.21.15.31:0/2341207704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8318107940 msgr2=0x7f83181014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ca7d640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8318107940 0x7f83181014e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ca7d640 1 -- 172.21.15.31:0/2341207704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831806a700 msgr2=0x7f8318100fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ca7d640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831806a700 0x7f8318100fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f831ca7d640 1 -- 172.21.15.31:0/2341207704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83181a6440 con 0x7f8318069d40 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.313+0000 7f8317fff640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831806a700 0x7f8318100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.314+0000 7f831ca7d640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318069d40 0x7f8318100a60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f830000e9c0 tx=0x7f830000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.314+0000 7f8315ffb640 1 -- 172.21.15.31:0/2341207704 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830000ccc0 con 0x7f8318069d40 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.314+0000 7f8315ffb640 1 -- 172.21.15.31:0/2341207704 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83000054d0 con 0x7f8318069d40 2024-04-25T01:56:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.314+0000 7f831ece1640 1 -- 172.21.15.31:0/2341207704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83181a6730 con 0x7f8318069d40 2024-04-25T01:56:44.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.314+0000 7f8315ffb640 1 -- 172.21.15.31:0/2341207704 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300010430 con 0x7f8318069d40 2024-04-25T01:56:44.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.314+0000 7f831ece1640 1 -- 172.21.15.31:0/2341207704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83181a6cd0 con 0x7f8318069d40 2024-04-25T01:56:44.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.316+0000 7f8315ffb640 1 -- 172.21.15.31:0/2341207704 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f83000046c0 con 0x7f8318069d40 2024-04-25T01:56:44.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.317+0000 7f83219f3740 1 Processor -- start 2024-04-25T01:56:44.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.317+0000 7f8315ffb640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f83040a2d80 0x7f83040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.317+0000 7f83219f3740 1 -- start start 2024-04-25T01:56:44.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.317+0000 7f831d27e640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f83040a2d80 0x7f83040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.317+0000 7f8315ffb640 1 -- 172.21.15.31:0/2341207704 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f83000cf570 con 0x7f8318069d40 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.317+0000 7f83219f3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fdca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.317+0000 7f83219f3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fdcf50 0x563c78fe70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.317+0000 7f83219f3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fe7620 0x563c78fe9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.317+0000 7f83219f3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563c78f416b0 con 0x563c78fe7620 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.317+0000 7f83219f3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563c78ee9880 con 0x563c78fdcf50 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.317+0000 7f83219f3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c78f3d670 con 0x563c78fdc630 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f831d27e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fe7620 0x563c78fe9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f8317fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fdcf50 0x563c78fe70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f8317fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fdcf50 0x563c78fe70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33572/0 (socket says 172.21.15.31:33572) 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f831d27e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fe7620 0x563c78fe9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55714/0 (socket says 172.21.15.31:55714) 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f8317fff640 1 -- 172.21.15.31:0/145303718 learned_addr learned my addr 172.21.15.31:0/145303718 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f831ca7d640 1 --2- 172.21.15.31:0/145303718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fdca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f8317fff640 1 -- 172.21.15.31:0/145303718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 msgr2=0x563c78fdca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f8317fff640 1 --2- 172.21.15.31:0/145303718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fdca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f8317fff640 1 -- 172.21.15.31:0/145303718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fe7620 msgr2=0x563c78fe9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f8317fff640 1 --2- 172.21.15.31:0/145303718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fe7620 0x563c78fe9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f8317fff640 1 -- 172.21.15.31:0/145303718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c78d1f7d0 con 0x563c78fdcf50 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f831d27e640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f83040a2d80 0x7f83040a5240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8310005fd0 tx=0x7f8310013750 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f8317fff640 1 --2- 172.21.15.31:0/145303718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fdcf50 0x563c78fe70e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f830802a8f0 tx=0x7f83080307e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4c5a6d8e58712f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f830effd640 1 -- 172.21.15.31:0/145303718 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8308030c70 con 0x563c78fdcf50 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f830effd640 1 -- 172.21.15.31:0/145303718 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f830804a030 con 0x563c78fdcf50 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.318+0000 7f830effd640 1 -- 172.21.15.31:0/145303718 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83080052a0 con 0x563c78fdcf50 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.319+0000 7f83219f3740 1 -- 172.21.15.31:0/145303718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fdcf50 msgr2=0x563c78fe70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.319+0000 7f83219f3740 1 --2- 172.21.15.31:0/145303718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fdcf50 0x563c78fe70e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f830802a8f0 tx=0x7f83080307e0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.319+0000 7f83219f3740 1 -- 172.21.15.31:0/145303718 shutdown_connections 2024-04-25T01:56:44.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.319+0000 7f83219f3740 1 --2- 172.21.15.31:0/145303718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fe7620 0x563c78fe9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.319+0000 7f83219f3740 1 --2- 172.21.15.31:0/145303718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fdcf50 0x563c78fe70e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.319+0000 7f83219f3740 1 --2- 172.21.15.31:0/145303718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fdca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.319+0000 7f83219f3740 1 -- 172.21.15.31:0/145303718 >> 172.21.15.31:0/145303718 conn(0x563c78fd8950 msgr2=0x563c78fd8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.319+0000 7f83219f3740 1 -- 172.21.15.31:0/145303718 shutdown_connections 2024-04-25T01:56:44.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.319+0000 7f83219f3740 1 -- 172.21.15.31:0/145303718 wait complete. 2024-04-25T01:56:44.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f83219f3740 1 -- start start 2024-04-25T01:56:44.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f83219f3740 1 Processor -- start 2024-04-25T01:56:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f83219f3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fe45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f83219f3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fe4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f83219f3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78fe5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f83219f3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563c78fea3b0 con 0x563c78fdcf50 2024-04-25T01:56:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f83219f3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563c78efcf50 con 0x563c78fe7620 2024-04-25T01:56:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f83219f3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c78fea530 con 0x563c78fdc630 2024-04-25T01:56:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f83219f3740 10 client.0 _ll_register_callbacks cb 0x563c78f38000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f8317fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fe4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f831d27e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78fe5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f8317fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fe4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55728/0 (socket says 172.21.15.31:55728) 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.320+0000 7f8317fff640 1 -- 172.21.15.31:0/1070741369 learned_addr learned my addr 172.21.15.31:0/1070741369 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f831ca7d640 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fe45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f831d27e640 1 -- 172.21.15.31:0/1070741369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 msgr2=0x563c78fe45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f831d27e640 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fe45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f831d27e640 1 -- 172.21.15.31:0/1070741369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 msgr2=0x563c78fe4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f831d27e640 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fe4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f831d27e640 1 -- 172.21.15.31:0/1070741369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c78c553f0 con 0x563c78fe7620 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f8317fff640 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fe4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f831d27e640 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78fe5060 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f831000fc20 tx=0x7f831001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1252ceebac0d1ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f82f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f82f7fff640 1 -- 172.21.15.31:0/1070741369 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310025070 con 0x563c78fe7620 2024-04-25T01:56:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f83219f3740 1 -- 172.21.15.31:0/1070741369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f830804a030 con 0x563c78fe7620 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f82f7fff640 1 -- 172.21.15.31:0/1070741369 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f831001c040 con 0x563c78fe7620 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f82f7fff640 1 -- 172.21.15.31:0/1070741369 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8310031050 con 0x563c78fe7620 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f83219f3740 10 client.25817 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.321+0000 7f83219f3740 1 -- 172.21.15.31:0/1070741369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563c78c150f0 con 0x563c78fe7620 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.322+0000 7f82f7fff640 1 -- 172.21.15.31:0/1070741369 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f830804a030 con 0x563c78fe7620 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.322+0000 7f82f7fff640 1 -- 172.21.15.31:0/1070741369 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8310020050 con 0x563c78fe7620 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.322+0000 7f83219f3740 10 client.25817 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.322+0000 7f83219f3740 10 client.25817 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.322+0000 7f83219f3740 20 client.25817 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.322+0000 7f83219f3740 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c78ff7960 0x563c78ff9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.322+0000 7f83219f3740 4 client.25817 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.322+0000 7f83219f3740 1 -- 172.21.15.31:0/1070741369 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c78f4ebf0 con 0x563c78ff7960 2024-04-25T01:56:44.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.322+0000 7f831ca7d640 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c78ff7960 0x563c78ff9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.322+0000 7f82f7fff640 10 client.25817 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:44.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.322+0000 7f831ca7d640 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c78ff7960 0x563c78ff9dc0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f67d72ae71f640ea server_cookie=a5a786bc9c621f19 in_seq=0 out_seq=0 2024-04-25T01:56:44.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.323+0000 7f82f7fff640 1 -- 172.21.15.31:0/1070741369 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c78f4ebf0 con 0x563c78ff7960 2024-04-25T01:56:44.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.323+0000 7f82f7fff640 10 client.25817 handle_command_reply: tid=0 2024-04-25T01:56:44.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.323+0000 7f83219f3740 1 client.25817 shutdown 2024-04-25T01:56:44.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.323+0000 7f83219f3740 1 -- 172.21.15.31:0/1070741369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 msgr2=0x563c78fe5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.323+0000 7f83219f3740 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78fe5060 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f831000fc20 tx=0x7f831001d000 comp rx=0 tx=0).stop 2024-04-25T01:56:44.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.324+0000 7f83219f3740 1 -- 172.21.15.31:0/1070741369 shutdown_connections 2024-04-25T01:56:44.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.324+0000 7f83219f3740 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c78ff7960 0x563c78ff9dc0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.324+0000 7f83219f3740 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78fe5060 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.324+0000 7f83219f3740 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fe4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.324+0000 7f82f7fff640 0 client.25817 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:44.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.324+0000 7f83219f3740 1 --2- 172.21.15.31:0/1070741369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fe45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.324+0000 7f83219f3740 1 -- 172.21.15.31:0/1070741369 >> 172.21.15.31:0/1070741369 conn(0x563c78fd8950 msgr2=0x563c78fde750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.324+0000 7f83219f3740 1 -- 172.21.15.31:0/1070741369 shutdown_connections 2024-04-25T01:56:44.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.324+0000 7f83219f3740 1 -- 172.21.15.31:0/1070741369 wait complete. 2024-04-25T01:56:44.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.324+0000 7f83219f3740 20 client.25817 trim_cache size 0 max 16384 2024-04-25T01:56:44.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f83219f3740 1 Processor -- start 2024-04-25T01:56:44.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f83219f3740 1 -- start start 2024-04-25T01:56:44.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f83219f3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdc630 0x563c78fe3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f83219f3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdcf50 0x563c78ff3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f83219f3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78fdfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f83219f3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f83181ab090 con 0x563c78fdc630 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f83219f3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f83181acc90 con 0x563c78fe7620 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f83219f3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f83181ab760 con 0x563c78fdcf50 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f831ca7d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdc630 0x563c78fe3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f831ca7d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdc630 0x563c78fe3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55734/0 (socket says 172.21.15.31:55734) 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f831ca7d640 1 -- 172.21.15.31:0/1191300978 learned_addr learned my addr 172.21.15.31:0/1191300978 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f8317fff640 1 --2- 172.21.15.31:0/1191300978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdcf50 0x563c78ff3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f831d27e640 1 --2- 172.21.15.31:0/1191300978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78fdfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f831ca7d640 1 -- 172.21.15.31:0/1191300978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdcf50 msgr2=0x563c78ff3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f831ca7d640 1 --2- 172.21.15.31:0/1191300978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdcf50 0x563c78ff3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f831ca7d640 1 -- 172.21.15.31:0/1191300978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 msgr2=0x563c78fdfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.342+0000 7f831ca7d640 1 --2- 172.21.15.31:0/1191300978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78fdfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.343+0000 7f831ca7d640 1 -- 172.21.15.31:0/1191300978 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c78fca090 con 0x563c78fdc630 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.343+0000 7f8317fff640 1 --2- 172.21.15.31:0/1191300978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdcf50 0x563c78ff3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:44.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.343+0000 7f831ca7d640 1 --2- 172.21.15.31:0/1191300978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdc630 0x563c78fe3520 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f8300097f40 tx=0x7f830009b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d97b53c6fb615c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.343+0000 7f830e7fc640 1 -- 172.21.15.31:0/1191300978 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83000a4750 con 0x563c78fdc630 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.343+0000 7f830e7fc640 1 -- 172.21.15.31:0/1191300978 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8300099de0 con 0x563c78fdc630 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.343+0000 7f830e7fc640 1 -- 172.21.15.31:0/1191300978 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83000a4520 con 0x563c78fdc630 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.343+0000 7f83219f3740 1 -- 172.21.15.31:0/1191300978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdc630 msgr2=0x563c78fe3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.343+0000 7f83219f3740 1 --2- 172.21.15.31:0/1191300978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdc630 0x563c78fe3520 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f8300097f40 tx=0x7f830009b9e0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 -- 172.21.15.31:0/1191300978 shutdown_connections 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 --2- 172.21.15.31:0/1191300978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78fdfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 --2- 172.21.15.31:0/1191300978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdcf50 0x563c78ff3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 --2- 172.21.15.31:0/1191300978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdc630 0x563c78fe3520 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 -- 172.21.15.31:0/1191300978 >> 172.21.15.31:0/1191300978 conn(0x563c78fd8950 msgr2=0x563c78fd7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 -- 172.21.15.31:0/1191300978 shutdown_connections 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 -- 172.21.15.31:0/1191300978 wait complete. 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 -- start start 2024-04-25T01:56:44.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 Processor -- start 2024-04-25T01:56:44.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fef900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fead60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78feb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f83181ac500 con 0x563c78fdcf50 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f83181abe30 con 0x563c78fe7620 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f83181a9070 con 0x563c78fdc630 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f83219f3740 10 client.0 _ll_register_callbacks cb 0x7f8310013540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.344+0000 7f8317fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f831ca7d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f831d27e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78feb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f8317fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55744/0 (socket says 172.21.15.31:55744) 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f831d27e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78feb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33588/0 (socket says 172.21.15.31:33588) 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f831ca7d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fef900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:37384/0 (socket says 172.21.15.31:37384) 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f8317fff640 1 -- 172.21.15.31:0/3537608567 learned_addr learned my addr 172.21.15.31:0/3537608567 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f8317fff640 1 -- 172.21.15.31:0/3537608567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 msgr2=0x563c78fef900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f8317fff640 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fef900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f8317fff640 1 -- 172.21.15.31:0/3537608567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 msgr2=0x563c78feb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f8317fff640 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78feb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f8317fff640 1 -- 172.21.15.31:0/3537608567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c78feb890 con 0x563c78fdcf50 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f831d27e640 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78feb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f831ca7d640 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fef900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f8317fff640 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fead60 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f830802a8f0 tx=0x7f8308030eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54920060a866dabd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f82f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f82f7fff640 1 -- 172.21.15.31:0/3537608567 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8308005250 con 0x563c78fdcf50 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f82f7fff640 1 -- 172.21.15.31:0/3537608567 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f83080053f0 con 0x563c78fdcf50 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f83219f3740 1 -- 172.21.15.31:0/3537608567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8300099de0 con 0x563c78fdcf50 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f82f7fff640 1 -- 172.21.15.31:0/3537608567 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f83080055f0 con 0x563c78fdcf50 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f83219f3740 10 client.16221 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.345+0000 7f83219f3740 1 -- 172.21.15.31:0/3537608567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563c7905dd30 con 0x563c78fdcf50 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.346+0000 7f82f7fff640 1 -- 172.21.15.31:0/3537608567 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8300099de0 con 0x563c78fdcf50 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.346+0000 7f82f7fff640 1 -- 172.21.15.31:0/3537608567 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f830804f7f0 con 0x563c78fdcf50 2024-04-25T01:56:44.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.346+0000 7f83219f3740 10 client.16221 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.346+0000 7f83219f3740 10 client.16221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.346+0000 7f83219f3740 20 client.16221 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.346+0000 7f83219f3740 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c7905e180 0x563c79060570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.346+0000 7f83219f3740 4 client.16221 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.346+0000 7f83219f3740 1 -- 172.21.15.31:0/3537608567 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563c78f48f40 con 0x563c7905e180 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.346+0000 7f831ca7d640 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c7905e180 0x563c79060570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.346+0000 7f82f7fff640 10 client.16221 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.346+0000 7f831ca7d640 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c7905e180 0x563c79060570 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a854cbbf0cc85c11 server_cookie=fa51696b047e0cd4 in_seq=0 out_seq=0 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.347+0000 7f82f7fff640 1 -- 172.21.15.31:0/3537608567 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563c78f48f40 con 0x563c7905e180 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.347+0000 7f82f7fff640 10 client.16221 handle_command_reply: tid=0 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.347+0000 7f83219f3740 1 client.16221 shutdown 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.347+0000 7f83219f3740 1 -- 172.21.15.31:0/3537608567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 msgr2=0x563c78fead60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.347+0000 7f83219f3740 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fead60 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f830802a8f0 tx=0x7f8308030eb0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.347+0000 7f83219f3740 1 -- 172.21.15.31:0/3537608567 shutdown_connections 2024-04-25T01:56:44.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.347+0000 7f83219f3740 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c7905e180 0x563c79060570 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.347+0000 7f83219f3740 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c78fe7620 0x563c78feb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.347+0000 7f83219f3740 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c78fdcf50 0x563c78fead60 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.347+0000 7f83219f3740 1 --2- 172.21.15.31:0/3537608567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c78fdc630 0x563c78fef900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.347+0000 7f83219f3740 1 -- 172.21.15.31:0/3537608567 >> 172.21.15.31:0/3537608567 conn(0x563c78fd8950 msgr2=0x563c78fe45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.348+0000 7f82f7fff640 0 client.16221 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.348+0000 7f83219f3740 1 -- 172.21.15.31:0/3537608567 shutdown_connections 2024-04-25T01:56:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.348+0000 7f83219f3740 1 -- 172.21.15.31:0/3537608567 wait complete. 2024-04-25T01:56:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.348+0000 7f83219f3740 20 client.16221 trim_cache size 0 max 16384 2024-04-25T01:56:44.356 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T01:56:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T01:56:44.357 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T01:56:44.357 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T01:56:44.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 -- 172.21.15.31:0/2341207704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f83040a2d80 msgr2=0x7f83040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f83040a2d80 0x7f83040a5240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8310005fd0 tx=0x7f8310013750 comp rx=0 tx=0).stop 2024-04-25T01:56:44.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 -- 172.21.15.31:0/2341207704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318069d40 msgr2=0x7f8318100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318069d40 0x7f8318100a60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f830000e9c0 tx=0x7f830000ee80 comp rx=0 tx=0).stop 2024-04-25T01:56:44.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 -- 172.21.15.31:0/2341207704 shutdown_connections 2024-04-25T01:56:44.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f83040a2d80 0x7f83040a5240 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8318107940 0x7f83181014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f831806a700 0x7f8318100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 --2- 172.21.15.31:0/2341207704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8318069d40 0x7f8318100a60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 -- 172.21.15.31:0/2341207704 >> 172.21.15.31:0/2341207704 conn(0x7f8318076440 msgr2=0x7f83180ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 -- 172.21.15.31:0/2341207704 shutdown_connections 2024-04-25T01:56:44.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.349+0000 7f831ece1640 1 -- 172.21.15.31:0/2341207704 wait complete. 2024-04-25T01:56:44.362 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:56:44.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:56:44.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.511+0000 7f03a07fb640 1 -- 172.21.15.31:0/232842939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0398100fe0 msgr2=0x7f039810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.511+0000 7f03a07fb640 1 --2- 172.21.15.31:0/232842939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0398100fe0 0x7f039810d4f0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0390009930 tx=0x7f039002a420 comp rx=0 tx=0).stop 2024-04-25T01:56:44.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.511+0000 7f03a07fb640 1 -- 172.21.15.31:0/232842939 shutdown_connections 2024-04-25T01:56:44.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.511+0000 7f03a07fb640 1 --2- 172.21.15.31:0/232842939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0398100fe0 0x7f039810d4f0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.511+0000 7f03a07fb640 1 --2- 172.21.15.31:0/232842939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0398100620 0x7f0398100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.511+0000 7f03a07fb640 1 --2- 172.21.15.31:0/232842939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0398106630 0x7f0398106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.511+0000 7f03a07fb640 1 -- 172.21.15.31:0/232842939 >> 172.21.15.31:0/232842939 conn(0x7f0398077740 msgr2=0x7f0398077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.511+0000 7f03a07fb640 1 -- 172.21.15.31:0/232842939 shutdown_connections 2024-04-25T01:56:44.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.511+0000 7f03a07fb640 1 -- 172.21.15.31:0/232842939 wait complete. 2024-04-25T01:56:44.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f03a07fb640 1 Processor -- start 2024-04-25T01:56:44.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f03a07fb640 1 -- start start 2024-04-25T01:56:44.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f03a07fb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0398100620 0x7f0398197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f03a07fb640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0398100fe0 0x7f03981983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f03a07fb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0398106630 0x7f039819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f03a07fb640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f03981108b0 con 0x7f0398106630 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f03a07fb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0398110730 con 0x7f0398100620 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f03a07fb640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0398110a30 con 0x7f0398100fe0 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039ed98640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0398106630 0x7f039819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039dd96640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0398100fe0 0x7f03981983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039ed98640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0398106630 0x7f039819c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55756/0 (socket says 172.21.15.31:55756) 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039dd96640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0398100fe0 0x7f03981983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:37394/0 (socket says 172.21.15.31:37394) 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039ed98640 1 -- 172.21.15.31:0/2749007700 learned_addr learned my addr 172.21.15.31:0/2749007700 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039e597640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0398100620 0x7f0398197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039dd96640 1 -- 172.21.15.31:0/2749007700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0398100620 msgr2=0x7f0398197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039dd96640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0398100620 0x7f0398197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039dd96640 1 -- 172.21.15.31:0/2749007700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0398106630 msgr2=0x7f039819c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039dd96640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0398106630 0x7f039819c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039dd96640 1 -- 172.21.15.31:0/2749007700 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039819cea0 con 0x7f0398100fe0 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039e597640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0398100620 0x7f0398197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.512+0000 7f039ed98640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0398106630 0x7f039819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.513+0000 7f039dd96640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0398100fe0 0x7f03981983f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f038800e9f0 tx=0x7f038800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.513+0000 7f038f7fe640 1 -- 172.21.15.31:0/2749007700 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038800ccc0 con 0x7f0398100fe0 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.513+0000 7f03a07fb640 1 -- 172.21.15.31:0/2749007700 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f039819d190 con 0x7f0398100fe0 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.513+0000 7f038f7fe640 1 -- 172.21.15.31:0/2749007700 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03880108e0 con 0x7f0398100fe0 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.513+0000 7f038f7fe640 1 -- 172.21.15.31:0/2749007700 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0388018880 con 0x7f0398100fe0 2024-04-25T01:56:44.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.513+0000 7f03a07fb640 1 -- 172.21.15.31:0/2749007700 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03981a4a20 con 0x7f0398100fe0 2024-04-25T01:56:44.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.514+0000 7f03a07fb640 1 -- 172.21.15.31:0/2749007700 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0398068560 con 0x7f0398100fe0 2024-04-25T01:56:44.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.515+0000 7f038f7fe640 1 -- 172.21.15.31:0/2749007700 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0388010a90 con 0x7f0398100fe0 2024-04-25T01:56:44.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.516+0000 7f038f7fe640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f03840a2d00 0x7f03840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.516+0000 7f039e597640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f03840a2d00 0x7f03840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.516+0000 7f038f7fe640 1 -- 172.21.15.31:0/2749007700 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f03880cf9d0 con 0x7f0398100fe0 2024-04-25T01:56:44.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.516+0000 7f039e597640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f03840a2d00 0x7f03840a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0394005fd0 tx=0x7f0394009500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.519+0000 7f038f7fe640 1 -- 172.21.15.31:0/2749007700 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f038809bff0 con 0x7f0398100fe0 2024-04-25T01:56:44.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.775+0000 7f03a07fb640 1 -- 172.21.15.31:0/2749007700 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0398109c20 con 0x7f0398100fe0 2024-04-25T01:56:44.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.776+0000 7f038f7fe640 1 -- 172.21.15.31:0/2749007700 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11931 (secure 0 0 0) 0x7f03880a02f0 con 0x7f0398100fe0 2024-04-25T01:56:44.779 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T01:56:44.779 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3566811410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3566811410},{"type":"v1","addr":"172.21.15.31:6835","nonce":3566811410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/2948364489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":2948364489},{"type":"v1","addr":"172.21.15.31:6837","nonce":2948364489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:24.898441+0000","modified":"2024-04-25T01:56:17.753044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24473,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3704800465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3704800465},{"type":"v1","addr":"172.21.15.31:6839","nonce":3704800465}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6841/4277792607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":4277792607},{"type":"v1","addr":"172.21.15.31:6841","nonce":4277792607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24407":{"gid":24407,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6833/3875547004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3875547004},{"type":"v1","addr":"172.21.15.40:6833","nonce":3875547004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6833/400138993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":400138993},{"type":"v1","addr":"172.21.15.134:6833","nonce":400138993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6835/538218297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":538218297},{"type":"v1","addr":"172.21.15.40:6835","nonce":538218297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6835/1266939870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":1266939870},{"type":"v1","addr":"172.21.15.134:6835","nonce":1266939870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.134:6837/2058394187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":2058394187},{"type":"v1","addr":"172.21.15.134:6837","nonce":2058394187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":63,"addr":"172.21.15.40:6837/1901184087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":1901184087},{"type":"v1","addr":"172.21.15.40:6837","nonce":1901184087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.134:6839/450925094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":450925094},{"type":"v1","addr":"172.21.15.134:6839","nonce":450925094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6839/3236110559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":3236110559},{"type":"v1","addr":"172.21.15.40:6839","nonce":3236110559}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24487,24473,24503,24463,24451]},"id":1}]} 2024-04-25T01:56:44.779 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2024-04-25T01:56:44.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.778+0000 7f03a07fb640 1 -- 172.21.15.31:0/2749007700 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f03840a2d00 msgr2=0x7f03840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.778+0000 7f03a07fb640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f03840a2d00 0x7f03840a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0394005fd0 tx=0x7f0394009500 comp rx=0 tx=0).stop 2024-04-25T01:56:44.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.778+0000 7f03a07fb640 1 -- 172.21.15.31:0/2749007700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0398100fe0 msgr2=0x7f03981983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.778+0000 7f03a07fb640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0398100fe0 0x7f03981983f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f038800e9f0 tx=0x7f038800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.778+0000 7f03a07fb640 1 -- 172.21.15.31:0/2749007700 shutdown_connections 2024-04-25T01:56:44.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.778+0000 7f03a07fb640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f03840a2d00 0x7f03840a51c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.778+0000 7f03a07fb640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0398106630 0x7f039819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.778+0000 7f03a07fb640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0398100fe0 0x7f03981983f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.778+0000 7f03a07fb640 1 --2- 172.21.15.31:0/2749007700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0398100620 0x7f0398197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.778+0000 7f03a07fb640 1 -- 172.21.15.31:0/2749007700 >> 172.21.15.31:0/2749007700 conn(0x7f0398077740 msgr2=0x7f0398068c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.779+0000 7f03a07fb640 1 -- 172.21.15.31:0/2749007700 shutdown_connections 2024-04-25T01:56:44.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.779+0000 7f03a07fb640 1 -- 172.21.15.31:0/2749007700 wait complete. 2024-04-25T01:56:44.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:56:44.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.906+0000 7f29566f7640 1 -- 172.21.15.31:0/2100717401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295010bd00 msgr2=0x7f295010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.906+0000 7f29566f7640 1 --2- 172.21.15.31:0/2100717401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295010bd00 0x7f295010e1a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2948009ca0 tx=0x7f294802a9e0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.906+0000 7f29566f7640 1 -- 172.21.15.31:0/2100717401 shutdown_connections 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.906+0000 7f29566f7640 1 --2- 172.21.15.31:0/2100717401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295010bd00 0x7f295010e1a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.906+0000 7f29566f7640 1 --2- 172.21.15.31:0/2100717401 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29501032e0 0x7f295010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.906+0000 7f29566f7640 1 --2- 172.21.15.31:0/2100717401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29501029c0 0x7f2950102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.906+0000 7f29566f7640 1 -- 172.21.15.31:0/2100717401 >> 172.21.15.31:0/2100717401 conn(0x7f2950077740 msgr2=0x7f2950077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.906+0000 7f29566f7640 1 -- 172.21.15.31:0/2100717401 shutdown_connections 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.906+0000 7f29566f7640 1 -- 172.21.15.31:0/2100717401 wait complete. 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.906+0000 7f29566f7640 1 Processor -- start 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.906+0000 7f29566f7640 1 -- start start 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f29566f7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29501029c0 0x7f2950100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f29566f7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29501032e0 0x7f2950100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f29566f7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295010bd00 0x7f2950101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f29566f7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f29501119b0 con 0x7f295010bd00 2024-04-25T01:56:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f29566f7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2950111830 con 0x7f29501032e0 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f29566f7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2950111b30 con 0x7f29501029c0 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2954c94640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295010bd00 0x7f2950101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2947fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29501032e0 0x7f2950100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2947fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29501032e0 0x7f2950100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33630/0 (socket says 172.21.15.31:33630) 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f294ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29501029c0 0x7f2950100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2947fff640 1 -- 172.21.15.31:0/2291933265 learned_addr learned my addr 172.21.15.31:0/2291933265 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2954c94640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295010bd00 0x7f2950101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55782/0 (socket says 172.21.15.31:55782) 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2954c94640 1 -- 172.21.15.31:0/2291933265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29501029c0 msgr2=0x7f2950100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2954c94640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29501029c0 0x7f2950100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2954c94640 1 -- 172.21.15.31:0/2291933265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29501032e0 msgr2=0x7f2950100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2954c94640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29501032e0 0x7f2950100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2954c94640 1 -- 172.21.15.31:0/2291933265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29501a63d0 con 0x7f295010bd00 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2947fff640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29501032e0 0x7f2950100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.907+0000 7f2954c94640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295010bd00 0x7f2950101090 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f294802af00 tx=0x7f294802fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.908+0000 7f294dffb640 1 -- 172.21.15.31:0/2291933265 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948033040 con 0x7f295010bd00 2024-04-25T01:56:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.908+0000 7f29566f7640 1 -- 172.21.15.31:0/2291933265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29501a6660 con 0x7f295010bd00 2024-04-25T01:56:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.908+0000 7f294dffb640 1 -- 172.21.15.31:0/2291933265 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f294802e7a0 con 0x7f295010bd00 2024-04-25T01:56:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.908+0000 7f294dffb640 1 -- 172.21.15.31:0/2291933265 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948043720 con 0x7f295010bd00 2024-04-25T01:56:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.908+0000 7f29566f7640 1 -- 172.21.15.31:0/2291933265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29501a6b40 con 0x7f295010bd00 2024-04-25T01:56:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f294dffb640 1 -- 172.21.15.31:0/2291933265 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2948036070 con 0x7f295010bd00 2024-04-25T01:56:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f29593fd740 1 Processor -- start 2024-04-25T01:56:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f29593fd740 1 -- start start 2024-04-25T01:56:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f29593fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8580 0x564318ca8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f29593fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8ea0 0x564318cb3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f29593fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318cb3570 0x564318cb5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f29593fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564318c0d670 con 0x564318cb3570 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f29593fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564318bb5970 con 0x564318ca8ea0 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f29593fd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564318c09630 con 0x564318ca8580 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f294ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8ea0 0x564318cb3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f294ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8ea0 0x564318cb3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33642/0 (socket says 172.21.15.31:33642) 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f294ffff640 1 -- 172.21.15.31:0/859751636 learned_addr learned my addr 172.21.15.31:0/859751636 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f2947fff640 1 --2- 172.21.15.31:0/859751636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318cb3570 0x564318cb5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f2947fff640 1 --2- 172.21.15.31:0/859751636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8580 0x564318ca8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.911+0000 7f294dffb640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f293c0a2d30 0x7f293c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.912+0000 7f294ffff640 1 -- 172.21.15.31:0/859751636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8580 msgr2=0x564318ca8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.912+0000 7f294ffff640 1 --2- 172.21.15.31:0/859751636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8580 0x564318ca8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.912+0000 7f294ffff640 1 -- 172.21.15.31:0/859751636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318cb3570 msgr2=0x564318cb5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.912+0000 7f294ffff640 1 --2- 172.21.15.31:0/859751636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318cb3570 0x564318cb5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.912+0000 7f294ffff640 1 -- 172.21.15.31:0/859751636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564318bd9400 con 0x564318ca8ea0 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.912+0000 7f2954c94640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f293c0a2d30 0x7f293c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.912+0000 7f294dffb640 1 -- 172.21.15.31:0/2291933265 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f29480f0030 con 0x7f295010bd00 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.912+0000 7f294ffff640 1 --2- 172.21.15.31:0/859751636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8ea0 0x564318cb3030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2940009cf0 tx=0x7f2940011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=217a957967ecc59d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.912+0000 7f29467fc640 1 -- 172.21.15.31:0/859751636 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2940009de0 con 0x564318ca8ea0 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.912+0000 7f29467fc640 1 -- 172.21.15.31:0/859751636 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2940005020 con 0x564318ca8ea0 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.912+0000 7f29467fc640 1 -- 172.21.15.31:0/859751636 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2940004ba0 con 0x564318ca8ea0 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f29593fd740 1 -- 172.21.15.31:0/859751636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8ea0 msgr2=0x564318cb3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f2954c94640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f293c0a2d30 0x7f293c0a51f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f29480ed880 tx=0x7f294808e6a0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f29593fd740 1 --2- 172.21.15.31:0/859751636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8ea0 0x564318cb3030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2940009cf0 tx=0x7f2940011810 comp rx=0 tx=0).stop 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f29593fd740 1 -- 172.21.15.31:0/859751636 shutdown_connections 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f29593fd740 1 --2- 172.21.15.31:0/859751636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318cb3570 0x564318cb5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f29593fd740 1 --2- 172.21.15.31:0/859751636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8ea0 0x564318cb3030 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f29593fd740 1 --2- 172.21.15.31:0/859751636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8580 0x564318ca8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f29593fd740 1 -- 172.21.15.31:0/859751636 >> 172.21.15.31:0/859751636 conn(0x564318ca48a0 msgr2=0x564318ca4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f29593fd740 1 -- 172.21.15.31:0/859751636 shutdown_connections 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f29593fd740 1 -- 172.21.15.31:0/859751636 wait complete. 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f29593fd740 1 -- start start 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.913+0000 7f29593fd740 1 Processor -- start 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f29593fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8580 0x564318cb2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f29593fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318cb01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f29593fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318cb3570 0x564318cb0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f29593fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564318cb6360 con 0x564318ca8ea0 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f29593fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564318bc9050 con 0x564318cb3570 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f29593fd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564318cb64e0 con 0x564318ca8580 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f29593fd740 10 client.0 _ll_register_callbacks cb 0x564318920be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f294ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318cb01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f2947fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8580 0x564318cb2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f294ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318cb01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55808/0 (socket says 172.21.15.31:55808) 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f294ffff640 1 -- 172.21.15.31:0/1665823415 learned_addr learned my addr 172.21.15.31:0/1665823415 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f2947fff640 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318cb3570 0x564318cb0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f2947fff640 1 -- 172.21.15.31:0/1665823415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318cb3570 msgr2=0x564318cb0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f2947fff640 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318cb3570 0x564318cb0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f2947fff640 1 -- 172.21.15.31:0/1665823415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 msgr2=0x564318cb01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f2947fff640 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318cb01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f2947fff640 1 -- 172.21.15.31:0/1665823415 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564318b9adc0 con 0x564318ca8580 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f294ffff640 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318cb01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f2947fff640 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8580 0x564318cb2ec0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2950102050 tx=0x7f2938014040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d40da43546fc41f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f292f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.914+0000 7f292f7fe640 1 -- 172.21.15.31:0/1665823415 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938004030 con 0x564318ca8580 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f29593fd740 1 -- 172.21.15.31:0/1665823415 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2940005020 con 0x564318ca8580 2024-04-25T01:56:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f292f7fe640 1 -- 172.21.15.31:0/1665823415 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f29380041d0 con 0x564318ca8580 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f292f7fe640 1 -- 172.21.15.31:0/1665823415 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f29380043d0 con 0x564318ca8580 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f29593fd740 10 client.25699 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f29593fd740 1 -- 172.21.15.31:0/1665823415 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5643189213f0 con 0x564318ca8580 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f292f7fe640 1 -- 172.21.15.31:0/1665823415 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2940005020 con 0x564318ca8580 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f292f7fe640 1 -- 172.21.15.31:0/1665823415 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f293802b480 con 0x564318ca8580 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f29593fd740 10 client.25699 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f29593fd740 10 client.25699 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f29593fd740 20 client.25699 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f29593fd740 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x564318cc3570 0x564318cc5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f29593fd740 4 client.25699 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f29593fd740 1 -- 172.21.15.31:0/1665823415 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564318c1abb0 con 0x564318cc3570 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.915+0000 7f294ffff640 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x564318cc3570 0x564318cc5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.916+0000 7f292f7fe640 10 client.25699 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:56:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.916+0000 7f294ffff640 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x564318cc3570 0x564318cc5980 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fbddd66352d8348 server_cookie=55b8e5cee50cb95d in_seq=0 out_seq=0 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.916+0000 7f292f7fe640 1 -- 172.21.15.31:0/1665823415 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564318c1abb0 con 0x564318cc3570 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.916+0000 7f292f7fe640 10 client.25699 handle_command_reply: tid=0 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.916+0000 7f29593fd740 1 client.25699 shutdown 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f29593fd740 1 -- 172.21.15.31:0/1665823415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8580 msgr2=0x564318cb2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f29593fd740 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8580 0x564318cb2ec0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2950102050 tx=0x7f2938014040 comp rx=0 tx=0).stop 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f29593fd740 1 -- 172.21.15.31:0/1665823415 shutdown_connections 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f29593fd740 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x564318cc3570 0x564318cc5980 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f29593fd740 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318cb3570 0x564318cb0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f29593fd740 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318cb01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f29593fd740 1 --2- 172.21.15.31:0/1665823415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8580 0x564318cb2ec0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f292f7fe640 0 client.25699 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f29593fd740 1 -- 172.21.15.31:0/1665823415 >> 172.21.15.31:0/1665823415 conn(0x564318ca48a0 msgr2=0x564318ca3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f29593fd740 1 -- 172.21.15.31:0/1665823415 shutdown_connections 2024-04-25T01:56:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f29593fd740 1 -- 172.21.15.31:0/1665823415 wait complete. 2024-04-25T01:56:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.917+0000 7f29593fd740 20 client.25699 trim_cache size 0 max 16384 2024-04-25T01:56:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f29593fd740 1 Processor -- start 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f29593fd740 1 -- start start 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f29593fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 0x564318cbf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f29593fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318ccb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f29593fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318cb3570 0x564318ccb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f29593fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f29501af510 con 0x564318ca8ea0 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f29593fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f29501b1110 con 0x564318ca8580 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f29593fd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f29501afbe0 con 0x564318cb3570 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f294ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318ccb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f294ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318ccb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55824/0 (socket says 172.21.15.31:55824) 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f294ffff640 1 -- 172.21.15.31:0/4030882028 learned_addr learned my addr 172.21.15.31:0/4030882028 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f2947fff640 1 --2- 172.21.15.31:0/4030882028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 0x564318cbf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f2947fff640 1 --2- 172.21.15.31:0/4030882028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318cb3570 0x564318ccb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f294ffff640 1 -- 172.21.15.31:0/4030882028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318cb3570 msgr2=0x564318ccb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f294ffff640 1 --2- 172.21.15.31:0/4030882028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318cb3570 0x564318ccb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f294ffff640 1 -- 172.21.15.31:0/4030882028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 msgr2=0x564318cbf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f294ffff640 1 --2- 172.21.15.31:0/4030882028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 0x564318cbf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f294ffff640 1 -- 172.21.15.31:0/4030882028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564318caf160 con 0x564318ca8ea0 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.932+0000 7f294ffff640 1 --2- 172.21.15.31:0/4030882028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318ccb200 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2940009cf0 tx=0x7f2940004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b7a14b94a0f879 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f2945ffb640 1 -- 172.21.15.31:0/4030882028 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294002bb50 con 0x564318ca8ea0 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f2945ffb640 1 -- 172.21.15.31:0/4030882028 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29400044f0 con 0x564318ca8ea0 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f2945ffb640 1 -- 172.21.15.31:0/4030882028 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294002b800 con 0x564318ca8ea0 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f29593fd740 1 -- 172.21.15.31:0/4030882028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 msgr2=0x564318ccb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f29593fd740 1 --2- 172.21.15.31:0/4030882028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318ccb200 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2940009cf0 tx=0x7f2940004840 comp rx=0 tx=0).stop 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f29593fd740 1 -- 172.21.15.31:0/4030882028 shutdown_connections 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f29593fd740 1 --2- 172.21.15.31:0/4030882028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318cb3570 0x564318ccb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f29593fd740 1 --2- 172.21.15.31:0/4030882028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318ca8ea0 0x564318ccb200 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f29593fd740 1 --2- 172.21.15.31:0/4030882028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 0x564318cbf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f29593fd740 1 -- 172.21.15.31:0/4030882028 >> 172.21.15.31:0/4030882028 conn(0x564318ca48a0 msgr2=0x564318ca3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f29593fd740 1 -- 172.21.15.31:0/4030882028 shutdown_connections 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.933+0000 7f29593fd740 1 -- 172.21.15.31:0/4030882028 wait complete. 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f29593fd740 1 -- start start 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f29593fd740 1 Processor -- start 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f29593fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 0x564318cbba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f29593fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8ea0 0x564318cb8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f29593fd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318cb3570 0x564318cb9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f29593fd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f29501b0980 con 0x564318cb3570 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f29593fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f29501b02b0 con 0x564318ca8580 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f29593fd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f29501ad510 con 0x564318ca8ea0 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f29593fd740 10 client.0 _ll_register_callbacks cb 0x7f29380126a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f2947fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318cb3570 0x564318cb9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f2947fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 0x564318cbba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f294ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8ea0 0x564318cb8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f2947fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 0x564318cbba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33664/0 (socket says 172.21.15.31:33664) 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.934+0000 7f2947fff640 1 -- 172.21.15.31:0/3851770541 learned_addr learned my addr 172.21.15.31:0/3851770541 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f2947fff640 1 -- 172.21.15.31:0/3851770541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8ea0 msgr2=0x564318cb8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f2947fff640 1 --2- 172.21.15.31:0/3851770541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8ea0 0x564318cb8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f2947fff640 1 -- 172.21.15.31:0/3851770541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318cb3570 msgr2=0x564318cb9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f2947fff640 1 --2- 172.21.15.31:0/3851770541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318cb3570 0x564318cb9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f2947fff640 1 -- 172.21.15.31:0/3851770541 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643189eb7d0 con 0x564318ca8580 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f2947fff640 1 --2- 172.21.15.31:0/3851770541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 0x564318cbba80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f293802b9f0 tx=0x7f293800b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0bcbb174a4e2fd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f292f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f292f7fe640 1 -- 172.21.15.31:0/3851770541 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938040060 con 0x564318ca8580 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f29593fd740 1 -- 172.21.15.31:0/3851770541 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29400044f0 con 0x564318ca8580 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f292f7fe640 1 -- 172.21.15.31:0/3851770541 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f293802ba70 con 0x564318ca8580 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f292f7fe640 1 -- 172.21.15.31:0/3851770541 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f293802b680 con 0x564318ca8580 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f29593fd740 10 client.25847 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.935+0000 7f29593fd740 1 -- 172.21.15.31:0/3851770541 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564318d29840 con 0x564318ca8580 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f292f7fe640 1 -- 172.21.15.31:0/3851770541 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29400044f0 con 0x564318ca8580 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f292f7fe640 1 -- 172.21.15.31:0/3851770541 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2938045b10 con 0x564318ca8580 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f29593fd740 10 client.25847 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f29593fd740 10 client.25847 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f29593fd740 20 client.25847 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f29593fd740 1 --2- 172.21.15.31:0/3851770541 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x564318d29f90 0x564318d2c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f29593fd740 4 client.25847 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f29593fd740 1 -- 172.21.15.31:0/3851770541 --> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564318c14f00 con 0x564318d29f90 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f294ffff640 1 --2- 172.21.15.31:0/3851770541 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x564318d29f90 0x564318d2c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f294ffff640 1 --2- 172.21.15.31:0/3851770541 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x564318d29f90 0x564318d2c3b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=93cd7620dcd96bb1 server_cookie=4d4fe89e3d236d92 in_seq=0 out_seq=0 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f292f7fe640 10 client.25847 ms_handle_connect on v2:172.21.15.134:6834/1266939870 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f292f7fe640 1 -- 172.21.15.31:0/3851770541 <== mds.4 v2:172.21.15.134:6834/1266939870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564318c14f00 con 0x564318d29f90 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f292f7fe640 10 client.25847 handle_command_reply: tid=0 2024-04-25T01:56:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.936+0000 7f29593fd740 1 client.25847 shutdown 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29593fd740 1 -- 172.21.15.31:0/3851770541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 msgr2=0x564318cbba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29593fd740 1 --2- 172.21.15.31:0/3851770541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 0x564318cbba80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f293802b9f0 tx=0x7f293800b480 comp rx=0 tx=0).stop 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29593fd740 1 -- 172.21.15.31:0/3851770541 shutdown_connections 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29593fd740 1 --2- 172.21.15.31:0/3851770541 >> [v2:172.21.15.134:6834/1266939870,v1:172.21.15.134:6835/1266939870] conn(0x564318d29f90 0x564318d2c3b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29593fd740 1 --2- 172.21.15.31:0/3851770541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564318cb3570 0x564318cb9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29593fd740 1 --2- 172.21.15.31:0/3851770541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564318ca8ea0 0x564318cb8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f292f7fe640 0 client.25847 ms_handle_reset on v2:172.21.15.134:6834/1266939870 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29593fd740 1 --2- 172.21.15.31:0/3851770541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564318ca8580 0x564318cbba80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29593fd740 1 -- 172.21.15.31:0/3851770541 >> 172.21.15.31:0/3851770541 conn(0x564318ca48a0 msgr2=0x564318cbe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29593fd740 1 -- 172.21.15.31:0/3851770541 shutdown_connections 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29593fd740 1 -- 172.21.15.31:0/3851770541 wait complete. 2024-04-25T01:56:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29593fd740 20 client.25847 trim_cache size 0 max 16384 2024-04-25T01:56:44.946 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:56:44.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29566f7640 1 -- 172.21.15.31:0/2291933265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f293c0a2d30 msgr2=0x7f293c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29566f7640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f293c0a2d30 0x7f293c0a51f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f29480ed880 tx=0x7f294808e6a0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29566f7640 1 -- 172.21.15.31:0/2291933265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295010bd00 msgr2=0x7f2950101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29566f7640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295010bd00 0x7f2950101090 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f294802af00 tx=0x7f294802fb70 comp rx=0 tx=0).stop 2024-04-25T01:56:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29566f7640 1 -- 172.21.15.31:0/2291933265 shutdown_connections 2024-04-25T01:56:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29566f7640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f293c0a2d30 0x7f293c0a51f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29566f7640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295010bd00 0x7f2950101090 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29566f7640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29501032e0 0x7f2950100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29566f7640 1 --2- 172.21.15.31:0/2291933265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29501029c0 0x7f2950100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.937+0000 7f29566f7640 1 -- 172.21.15.31:0/2291933265 >> 172.21.15.31:0/2291933265 conn(0x7f2950077740 msgr2=0x7f29500fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.938+0000 7f29566f7640 1 -- 172.21.15.31:0/2291933265 shutdown_connections 2024-04-25T01:56:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:44.938+0000 7f29566f7640 1 -- 172.21.15.31:0/2291933265 wait complete. 2024-04-25T01:56:44.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:56:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.060+0000 7f5ae3129640 1 -- 172.21.15.31:0/3190589395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5adc107720 msgr2=0x7f5adc107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.060+0000 7f5ae3129640 1 --2- 172.21.15.31:0/3190589395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5adc107720 0x7f5adc107b00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009860 tx=0x7f5acc02a340 comp rx=0 tx=0).stop 2024-04-25T01:56:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.060+0000 7f5ae3129640 1 -- 172.21.15.31:0/3190589395 shutdown_connections 2024-04-25T01:56:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.060+0000 7f5ae3129640 1 --2- 172.21.15.31:0/3190589395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5adc1020d0 0x7f5adc10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.060+0000 7f5ae3129640 1 --2- 172.21.15.31:0/3190589395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5adc101710 0x7f5adc101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.060+0000 7f5ae3129640 1 --2- 172.21.15.31:0/3190589395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5adc107720 0x7f5adc107b00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.060+0000 7f5ae3129640 1 -- 172.21.15.31:0/3190589395 >> 172.21.15.31:0/3190589395 conn(0x7f5adc0fd180 msgr2=0x7f5adc0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.060+0000 7f5ae3129640 1 -- 172.21.15.31:0/3190589395 shutdown_connections 2024-04-25T01:56:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.060+0000 7f5ae3129640 1 -- 172.21.15.31:0/3190589395 wait complete. 2024-04-25T01:56:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.060+0000 7f5ae3129640 1 Processor -- start 2024-04-25T01:56:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.060+0000 7f5ae3129640 1 -- start start 2024-04-25T01:56:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5ae3129640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5adc101710 0x7f5adc197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5ae3129640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5adc1020d0 0x7f5adc198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5ae3129640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5adc107720 0x7f5adc19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5ae3129640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1119a0 con 0x7f5adc107720 2024-04-25T01:56:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5ae3129640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc111820 con 0x7f5adc1020d0 2024-04-25T01:56:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5ae3129640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc111b20 con 0x7f5adc101710 2024-04-25T01:56:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5ae16c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5adc107720 0x7f5adc19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5adc1020d0 0x7f5adc198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5ae0ec5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5adc101710 0x7f5adc197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5adc1020d0 0x7f5adc198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33670/0 (socket says 172.21.15.31:33670) 2024-04-25T01:56:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5adbfff640 1 -- 172.21.15.31:0/574235425 learned_addr learned my addr 172.21.15.31:0/574235425 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5adbfff640 1 -- 172.21.15.31:0/574235425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5adc101710 msgr2=0x7f5adc197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5adbfff640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5adc101710 0x7f5adc197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5adbfff640 1 -- 172.21.15.31:0/574235425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5adc107720 msgr2=0x7f5adc19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5adbfff640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5adc107720 0x7f5adc19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5adbfff640 1 -- 172.21.15.31:0/574235425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc19cef0 con 0x7f5adc1020d0 2024-04-25T01:56:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5ae16c6640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5adc107720 0x7f5adc19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.061+0000 7f5adbfff640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5adc1020d0 0x7f5adc198440 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000e9f0 tx=0x7f5ad000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.062+0000 7f5ad9ffb640 1 -- 172.21.15.31:0/574235425 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad000cd80 con 0x7f5adc1020d0 2024-04-25T01:56:45.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.062+0000 7f5ad9ffb640 1 -- 172.21.15.31:0/574235425 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ad00108e0 con 0x7f5adc1020d0 2024-04-25T01:56:45.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.062+0000 7f5ae3129640 1 -- 172.21.15.31:0/574235425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5adc19d1e0 con 0x7f5adc1020d0 2024-04-25T01:56:45.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.062+0000 7f5ad9ffb640 1 -- 172.21.15.31:0/574235425 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0018920 con 0x7f5adc1020d0 2024-04-25T01:56:45.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.062+0000 7f5ae3129640 1 -- 172.21.15.31:0/574235425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5adc1a4a70 con 0x7f5adc1020d0 2024-04-25T01:56:45.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.064+0000 7f5ae5e38740 1 Processor -- start 2024-04-25T01:56:45.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae5e38740 1 -- start start 2024-04-25T01:56:45.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ad9ffb640 1 -- 172.21.15.31:0/574235425 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5ad0010a90 con 0x7f5adc1020d0 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231580 0x55ece4231960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece423c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece423c570 0x55ece423e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ece4196670 con 0x55ece4231580 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ece413e970 con 0x55ece423c570 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ece4192630 con 0x55ece4231ea0 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae16c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231580 0x55ece4231960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae16c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231580 0x55ece4231960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55870/0 (socket says 172.21.15.31:55870) 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae16c6640 1 -- 172.21.15.31:0/1737451523 learned_addr learned my addr 172.21.15.31:0/1737451523 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5adbfff640 1 --2- 172.21.15.31:0/1737451523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece423c570 0x55ece423e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae0ec5640 1 --2- 172.21.15.31:0/1737451523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece423c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae16c6640 1 -- 172.21.15.31:0/1737451523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 msgr2=0x55ece423c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae16c6640 1 --2- 172.21.15.31:0/1737451523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece423c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae16c6640 1 -- 172.21.15.31:0/1737451523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece423c570 msgr2=0x55ece423e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae16c6640 1 --2- 172.21.15.31:0/1737451523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece423c570 0x55ece423e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.065+0000 7f5ae16c6640 1 -- 172.21.15.31:0/1737451523 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ece4162400 con 0x55ece4231580 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.066+0000 7f5ad9ffb640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ac40a2d80 0x7f5ac40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.066+0000 7f5ae16c6640 1 --2- 172.21.15.31:0/1737451523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231580 0x55ece4231960 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400e870 tx=0x7f5ad40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352160c5f02c0508 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.066+0000 7f5ae0ec5640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ac40a2d80 0x7f5ac40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.066+0000 7f5ad9ffb640 1 -- 172.21.15.31:0/574235425 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5ad00cf630 con 0x7f5adc1020d0 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.066+0000 7f5ac2ffd640 1 -- 172.21.15.31:0/1737451523 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad400ddd0 con 0x55ece4231580 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.066+0000 7f5ac2ffd640 1 -- 172.21.15.31:0/1737451523 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ad401c070 con 0x55ece4231580 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.066+0000 7f5ac2ffd640 1 -- 172.21.15.31:0/1737451523 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad40165e0 con 0x55ece4231580 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.066+0000 7f5ae5e38740 1 -- 172.21.15.31:0/1737451523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231580 msgr2=0x55ece4231960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/1737451523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231580 0x55ece4231960 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400e870 tx=0x7f5ad40095c0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae0ec5640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ac40a2d80 0x7f5ac40a5240 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009830 tx=0x7f5acc0058d0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 -- 172.21.15.31:0/1737451523 shutdown_connections 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/1737451523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece423c570 0x55ece423e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/1737451523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece423c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/1737451523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231580 0x55ece4231960 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 -- 172.21.15.31:0/1737451523 >> 172.21.15.31:0/1737451523 conn(0x55ece422d8a0 msgr2=0x55ece422dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 -- 172.21.15.31:0/1737451523 shutdown_connections 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 -- 172.21.15.31:0/1737451523 wait complete. 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 -- start start 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 Processor -- start 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece423bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece42391c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4239720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.067+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ece4243770 con 0x55ece423c570 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ece4152050 con 0x55ece4231580 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ece42438f0 con 0x55ece4231ea0 2024-04-25T01:56:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae5e38740 10 client.0 _ll_register_callbacks cb 0x55ece3ea9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae16c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4239720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae0ec5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece423bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae16c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4239720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55882/0 (socket says 172.21.15.31:55882) 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae0ec5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece423bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33684/0 (socket says 172.21.15.31:33684) 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae16c6640 1 -- 172.21.15.31:0/3270634003 learned_addr learned my addr 172.21.15.31:0/3270634003 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5adbfff640 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece42391c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae16c6640 1 -- 172.21.15.31:0/3270634003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 msgr2=0x55ece42391c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae16c6640 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece42391c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae16c6640 1 -- 172.21.15.31:0/3270634003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 msgr2=0x55ece423bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae16c6640 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece423bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae16c6640 1 -- 172.21.15.31:0/3270634003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ece4123dc0 con 0x55ece423c570 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae0ec5640 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece423bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae16c6640 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4239720 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400e840 tx=0x7f5ad400de70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c93701c550f609 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5abbfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5abbfff640 1 -- 172.21.15.31:0/3270634003 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad40176a0 con 0x55ece423c570 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5abbfff640 1 -- 172.21.15.31:0/3270634003 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5ad4017840 con 0x55ece423c570 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.068+0000 7f5ae5e38740 1 -- 172.21.15.31:0/3270634003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ad401c070 con 0x55ece423c570 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5abbfff640 1 -- 172.21.15.31:0/3270634003 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5ad4017a40 con 0x55ece423c570 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5ae5e38740 10 client.16257 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5ae5e38740 1 -- 172.21.15.31:0/3270634003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ece3eaa3f0 con 0x55ece423c570 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5abbfff640 1 -- 172.21.15.31:0/3270634003 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ad401c070 con 0x55ece423c570 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5abbfff640 1 -- 172.21.15.31:0/3270634003 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5ad400b040 con 0x55ece423c570 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5ae5e38740 10 client.16257 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5ae5e38740 10 client.16257 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5ae5e38740 20 client.16257 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55ece4250a20 0x55ece4252e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5ae5e38740 4 client.16257 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5ae5e38740 1 -- 172.21.15.31:0/3270634003 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ece41a3bb0 con 0x55ece4250a20 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.069+0000 7f5ae0ec5640 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55ece4250a20 0x55ece4252e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.070+0000 7f5ae0ec5640 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55ece4250a20 0x55ece4252e10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6539c7e2d3fe5f0c server_cookie=eef3a496bfaa5e3e in_seq=0 out_seq=0 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.070+0000 7f5abbfff640 10 client.16257 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.070+0000 7f5abbfff640 1 -- 172.21.15.31:0/3270634003 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ece41a3bb0 con 0x55ece4250a20 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.070+0000 7f5abbfff640 10 client.16257 handle_command_reply: tid=0 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.070+0000 7f5ae5e38740 1 client.16257 shutdown 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.070+0000 7f5ae5e38740 1 -- 172.21.15.31:0/3270634003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 msgr2=0x55ece4239720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.070+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4239720 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400e840 tx=0x7f5ad400de70 comp rx=0 tx=0).stop 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.070+0000 7f5ae5e38740 1 -- 172.21.15.31:0/3270634003 shutdown_connections 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.070+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55ece4250a20 0x55ece4252e10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.070+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4239720 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.071+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece42391c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.071+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/3270634003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece423bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.071+0000 7f5abbfff640 0 client.16257 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.071+0000 7f5ae5e38740 1 -- 172.21.15.31:0/3270634003 >> 172.21.15.31:0/3270634003 conn(0x55ece422d8a0 msgr2=0x55ece42336a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.071+0000 7f5ae5e38740 1 -- 172.21.15.31:0/3270634003 shutdown_connections 2024-04-25T01:56:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.071+0000 7f5ae5e38740 1 -- 172.21.15.31:0/3270634003 wait complete. 2024-04-25T01:56:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.071+0000 7f5ae5e38740 20 client.16257 trim_cache size 0 max 16384 2024-04-25T01:56:45.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.084+0000 7f5ae5e38740 1 Processor -- start 2024-04-25T01:56:45.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5ae5e38740 1 -- start start 2024-04-25T01:56:45.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece424c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231ea0 0x55ece42585f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece423c570 0x55ece4258b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1aafe0 con 0x55ece4231ea0 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1acbe0 con 0x55ece4231580 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1ab6b0 con 0x55ece423c570 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231ea0 0x55ece42585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231ea0 0x55ece42585f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55890/0 (socket says 172.21.15.31:55890) 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5adbfff640 1 -- 172.21.15.31:0/2903462976 learned_addr learned my addr 172.21.15.31:0/2903462976 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5ae16c6640 1 --2- 172.21.15.31:0/2903462976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece423c570 0x55ece4258b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5ae0ec5640 1 --2- 172.21.15.31:0/2903462976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece424c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5adbfff640 1 -- 172.21.15.31:0/2903462976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece423c570 msgr2=0x55ece4258b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5adbfff640 1 --2- 172.21.15.31:0/2903462976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece423c570 0x55ece4258b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5adbfff640 1 -- 172.21.15.31:0/2903462976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 msgr2=0x55ece424c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5adbfff640 1 --2- 172.21.15.31:0/2903462976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece424c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5adbfff640 1 -- 172.21.15.31:0/2903462976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ece3f747d0 con 0x55ece4231ea0 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5adbfff640 1 --2- 172.21.15.31:0/2903462976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231ea0 0x55ece42585f0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5ad009bbe0 tx=0x7f5ad0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9500aa1fe941f1a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.085+0000 7f5ac27fc640 1 -- 172.21.15.31:0/2903462976 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad009b5b0 con 0x55ece4231ea0 2024-04-25T01:56:45.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.086+0000 7f5ac27fc640 1 -- 172.21.15.31:0/2903462976 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ad0010de0 con 0x55ece4231ea0 2024-04-25T01:56:45.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.086+0000 7f5ac27fc640 1 -- 172.21.15.31:0/2903462976 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad009b0a0 con 0x55ece4231ea0 2024-04-25T01:56:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.086+0000 7f5ae5e38740 1 -- 172.21.15.31:0/2903462976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231ea0 msgr2=0x55ece42585f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.086+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/2903462976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231ea0 0x55ece42585f0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5ad009bbe0 tx=0x7f5ad0009310 comp rx=0 tx=0).stop 2024-04-25T01:56:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.086+0000 7f5ae5e38740 1 -- 172.21.15.31:0/2903462976 shutdown_connections 2024-04-25T01:56:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.086+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/2903462976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece423c570 0x55ece4258b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.086+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/2903462976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece4231ea0 0x55ece42585f0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.086+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/2903462976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece424c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.086+0000 7f5ae5e38740 1 -- 172.21.15.31:0/2903462976 >> 172.21.15.31:0/2903462976 conn(0x55ece422d8a0 msgr2=0x55ece422cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.086+0000 7f5ae5e38740 1 -- 172.21.15.31:0/2903462976 shutdown_connections 2024-04-25T01:56:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.086+0000 7f5ae5e38740 1 -- 172.21.15.31:0/2903462976 wait complete. 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae5e38740 1 -- start start 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae5e38740 1 Processor -- start 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece4246590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece4246ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae5e38740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4247010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1ac450 con 0x55ece423c570 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1abd80 con 0x55ece4231580 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae5e38740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc1a8fc0 con 0x55ece4231ea0 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae5e38740 10 client.0 _ll_register_callbacks cb 0x7f5ad00d5420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae16c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4247010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae0ec5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece4246590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae16c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4247010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55896/0 (socket says 172.21.15.31:55896) 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5ae16c6640 1 -- 172.21.15.31:0/48388856 learned_addr learned my addr 172.21.15.31:0/48388856 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.087+0000 7f5adbfff640 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece4246ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5ae0ec5640 1 -- 172.21.15.31:0/48388856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 msgr2=0x55ece4246ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5ae0ec5640 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece4246ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5ae0ec5640 1 -- 172.21.15.31:0/48388856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 msgr2=0x55ece4247010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5ae0ec5640 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4247010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5ae0ec5640 1 -- 172.21.15.31:0/48388856 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ece42b5ea0 con 0x55ece4231580 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5ae16c6640 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4247010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5ae0ec5640 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece4246590 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5acc005940 tx=0x7f5acc005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef60deb843fc9283 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5abbfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5abbfff640 1 -- 172.21.15.31:0/48388856 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc037040 con 0x55ece4231580 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5abbfff640 1 -- 172.21.15.31:0/48388856 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5acc003b80 con 0x55ece4231580 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5ae5e38740 1 -- 172.21.15.31:0/48388856 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ece419df00 con 0x55ece4231580 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5abbfff640 1 -- 172.21.15.31:0/48388856 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5acc05d020 con 0x55ece4231580 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5ae5e38740 10 client.25865 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.088+0000 7f5ae5e38740 1 -- 172.21.15.31:0/48388856 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ece42b7160 con 0x55ece4231580 2024-04-25T01:56:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.089+0000 7f5abbfff640 1 -- 172.21.15.31:0/48388856 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ece419df00 con 0x55ece4231580 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.089+0000 7f5abbfff640 1 -- 172.21.15.31:0/48388856 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5acc04a040 con 0x55ece4231580 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.089+0000 7f5ae5e38740 10 client.25865 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.089+0000 7f5ae5e38740 10 client.25865 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.089+0000 7f5ae5e38740 20 client.25865 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.089+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55ece4250a20 0x55ece42b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.089+0000 7f5ae5e38740 4 client.25865 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.089+0000 7f5ae5e38740 1 -- 172.21.15.31:0/48388856 --> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ece411b820 con 0x55ece4250a20 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.089+0000 7f5adbfff640 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55ece4250a20 0x55ece42b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.089+0000 7f5adbfff640 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55ece4250a20 0x55ece42b98a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f8dc46a227d563e1 server_cookie=9f30f6af12658c11 in_seq=0 out_seq=0 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.089+0000 7f5abbfff640 10 client.25865 ms_handle_connect on v2:172.21.15.134:6836/2058394187 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5abbfff640 1 -- 172.21.15.31:0/48388856 <== mds.3 v2:172.21.15.134:6836/2058394187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ece411b820 con 0x55ece4250a20 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5abbfff640 10 client.25865 handle_command_reply: tid=0 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 1 client.25865 shutdown 2024-04-25T01:56:45.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 1 -- 172.21.15.31:0/48388856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 msgr2=0x55ece4246590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece4246590 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5acc005940 tx=0x7f5acc005f90 comp rx=0 tx=0).stop 2024-04-25T01:56:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 1 -- 172.21.15.31:0/48388856 shutdown_connections 2024-04-25T01:56:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.134:6836/2058394187,v1:172.21.15.134:6837/2058394187] conn(0x55ece4250a20 0x55ece42b98a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ece423c570 0x55ece4247010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ece4231ea0 0x55ece4246ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 1 --2- 172.21.15.31:0/48388856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ece4231580 0x55ece4246590 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5abbfff640 0 client.25865 ms_handle_reset on v2:172.21.15.134:6836/2058394187 2024-04-25T01:56:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 1 -- 172.21.15.31:0/48388856 >> 172.21.15.31:0/48388856 conn(0x55ece422d8a0 msgr2=0x55ece424b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 1 -- 172.21.15.31:0/48388856 shutdown_connections 2024-04-25T01:56:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 1 -- 172.21.15.31:0/48388856 wait complete. 2024-04-25T01:56:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.090+0000 7f5ae5e38740 20 client.25865 trim_cache size 0 max 16384 2024-04-25T01:56:45.099 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:56:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 -- 172.21.15.31:0/574235425 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ac40a2d80 msgr2=0x7f5ac40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ac40a2d80 0x7f5ac40a5240 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009830 tx=0x7f5acc0058d0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 -- 172.21.15.31:0/574235425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5adc1020d0 msgr2=0x7f5adc198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5adc1020d0 0x7f5adc198440 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000e9f0 tx=0x7f5ad000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 -- 172.21.15.31:0/574235425 shutdown_connections 2024-04-25T01:56:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5ac40a2d80 0x7f5ac40a5240 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5adc107720 0x7f5adc19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5adc1020d0 0x7f5adc198440 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 --2- 172.21.15.31:0/574235425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5adc101710 0x7f5adc197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 -- 172.21.15.31:0/574235425 >> 172.21.15.31:0/574235425 conn(0x7f5adc0fd180 msgr2=0x7f5adc0fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 -- 172.21.15.31:0/574235425 shutdown_connections 2024-04-25T01:56:45.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.091+0000 7f5ae3129640 1 -- 172.21.15.31:0/574235425 wait complete. 2024-04-25T01:56:45.103 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:44 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2749007700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:56:45.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:56:45.160 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:44 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2749007700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:56:45.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:44 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2749007700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:56:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.213+0000 7f36a4bc1640 1 -- 172.21.15.31:0/3022182963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a0100fd0 msgr2=0x7f36a010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.213+0000 7f36a4bc1640 1 --2- 172.21.15.31:0/3022182963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a0100fd0 0x7f36a010d4e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3698009cd0 tx=0x7f369802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.213+0000 7f36a4bc1640 1 -- 172.21.15.31:0/3022182963 shutdown_connections 2024-04-25T01:56:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.213+0000 7f36a4bc1640 1 --2- 172.21.15.31:0/3022182963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a0100fd0 0x7f36a010d4e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.213+0000 7f36a4bc1640 1 --2- 172.21.15.31:0/3022182963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a0100610 0x7f36a0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.213+0000 7f36a4bc1640 1 --2- 172.21.15.31:0/3022182963 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36a01066b0 0x7f36a0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.213+0000 7f36a4bc1640 1 -- 172.21.15.31:0/3022182963 >> 172.21.15.31:0/3022182963 conn(0x7f36a0077740 msgr2=0x7f36a0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.213+0000 7f36a4bc1640 1 -- 172.21.15.31:0/3022182963 shutdown_connections 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.213+0000 7f36a4bc1640 1 -- 172.21.15.31:0/3022182963 wait complete. 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.214+0000 7f36a4bc1640 1 Processor -- start 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.214+0000 7f36a4bc1640 1 -- start start 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.214+0000 7f36a4bc1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a0100610 0x7f36a0198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.214+0000 7f36a4bc1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36a0100fd0 0x7f36a01991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.214+0000 7f36a4bc1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01066b0 0x7f36a01a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.214+0000 7f36a4bc1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f36a01108a0 con 0x7f36a0100610 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.214+0000 7f36a4bc1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f36a0110720 con 0x7f36a01066b0 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.214+0000 7f36a4bc1640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f36a0110a20 con 0x7f36a0100fd0 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f369e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a0100610 0x7f36a0198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f369e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a0100610 0x7f36a0198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55918/0 (socket says 172.21.15.31:55918) 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f369e59c640 1 -- 172.21.15.31:0/1957389843 learned_addr learned my addr 172.21.15.31:0/1957389843 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f369ed9d640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01066b0 0x7f36a01a3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f3697fff640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36a0100fd0 0x7f36a01991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f369e59c640 1 -- 172.21.15.31:0/1957389843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36a0100fd0 msgr2=0x7f36a01991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f369e59c640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36a0100fd0 0x7f36a01991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f369e59c640 1 -- 172.21.15.31:0/1957389843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01066b0 msgr2=0x7f36a01a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f369e59c640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01066b0 0x7f36a01a3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f369e59c640 1 -- 172.21.15.31:0/1957389843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a01a4260 con 0x7f36a0100610 2024-04-25T01:56:45.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f369e59c640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a0100610 0x7f36a0198ca0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f369000e9c0 tx=0x7f369000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f36977fe640 1 -- 172.21.15.31:0/1957389843 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369000cd80 con 0x7f36a0100610 2024-04-25T01:56:45.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f36977fe640 1 -- 172.21.15.31:0/1957389843 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36900035f0 con 0x7f36a0100610 2024-04-25T01:56:45.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f36977fe640 1 -- 172.21.15.31:0/1957389843 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3690010690 con 0x7f36a0100610 2024-04-25T01:56:45.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f36a4bc1640 1 -- 172.21.15.31:0/1957389843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a01a4430 con 0x7f36a0100610 2024-04-25T01:56:45.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.215+0000 7f36a4bc1640 1 -- 172.21.15.31:0/1957389843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a01a4900 con 0x7f36a0100610 2024-04-25T01:56:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.218+0000 7f36977fe640 1 -- 172.21.15.31:0/1957389843 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f36900037a0 con 0x7f36a0100610 2024-04-25T01:56:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.218+0000 7f36a79ad740 1 Processor -- start 2024-04-25T01:56:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.218+0000 7f36a79ad740 1 -- start start 2024-04-25T01:56:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7460 0x55d7f9ac7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 0x55d7f9ad6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f36a79ad740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9a2c670 con 0x55d7f9ac7d80 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f36a79ad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d7f99d4970 con 0x55d7f9ad6980 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f36a79ad740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9a28630 con 0x55d7f9ac7460 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f36977fe640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f368c0a2d00 0x7f368c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f369e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 0x55d7f9ad6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f3697fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f369ed9d640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f368c0a2d00 0x7f368c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f3697fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33722/0 (socket says 172.21.15.31:33722) 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f3697fff640 1 -- 172.21.15.31:0/3338512310 learned_addr learned my addr 172.21.15.31:0/3338512310 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f369ed9d640 1 --2- 172.21.15.31:0/3338512310 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7460 0x55d7f9ac7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f36977fe640 1 -- 172.21.15.31:0/1957389843 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f36900cfa00 con 0x7f36a0100610 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f3697fff640 1 -- 172.21.15.31:0/3338512310 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7460 msgr2=0x55d7f9ac7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f3697fff640 1 --2- 172.21.15.31:0/3338512310 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7460 0x55d7f9ac7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f3697fff640 1 -- 172.21.15.31:0/3338512310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 msgr2=0x55d7f9ad6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f3697fff640 1 --2- 172.21.15.31:0/3338512310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 0x55d7f9ad6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f3697fff640 1 -- 172.21.15.31:0/3338512310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f99f8400 con 0x55d7f9ad6980 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f369ed9d640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f368c0a2d00 0x7f368c0a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3698004af0 tx=0x7f3698009910 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.219+0000 7f3697fff640 1 --2- 172.21.15.31:0/3338512310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad8d70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3688009c10 tx=0x7f36880117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a551c65ca22d48f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.220+0000 7f3694ff9640 1 -- 172.21.15.31:0/3338512310 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3688012450 con 0x55d7f9ad6980 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.220+0000 7f3694ff9640 1 -- 172.21.15.31:0/3338512310 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3688012aa0 con 0x55d7f9ad6980 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.220+0000 7f3694ff9640 1 -- 172.21.15.31:0/3338512310 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368800f9b0 con 0x55d7f9ad6980 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.220+0000 7f36a79ad740 1 -- 172.21.15.31:0/3338512310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 msgr2=0x55d7f9ad8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.220+0000 7f36a79ad740 1 --2- 172.21.15.31:0/3338512310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad8d70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3688009c10 tx=0x7f36880117b0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.220+0000 7f36a79ad740 1 -- 172.21.15.31:0/3338512310 shutdown_connections 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.220+0000 7f36a79ad740 1 --2- 172.21.15.31:0/3338512310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad8d70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.220+0000 7f36a79ad740 1 --2- 172.21.15.31:0/3338512310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 0x55d7f9ad6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.220+0000 7f36a79ad740 1 --2- 172.21.15.31:0/3338512310 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7460 0x55d7f9ac7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.220+0000 7f36a79ad740 1 -- 172.21.15.31:0/3338512310 >> 172.21.15.31:0/3338512310 conn(0x55d7f9ac38a0 msgr2=0x55d7f9ac3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f36a79ad740 1 -- 172.21.15.31:0/3338512310 shutdown_connections 2024-04-25T01:56:45.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f36a79ad740 1 -- 172.21.15.31:0/3338512310 wait complete. 2024-04-25T01:56:45.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f36a79ad740 1 -- start start 2024-04-25T01:56:45.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f36a79ad740 1 Processor -- start 2024-04-25T01:56:45.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7460 0x55d7f9ad61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 0x55d7f9ad34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f36a79ad740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9ad97c0 con 0x55d7f9ac7d80 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f36a79ad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d7f99e8050 con 0x55d7f9ad6980 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f36a79ad740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9ad9940 con 0x55d7f9ac7460 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f36a79ad740 10 client.0 _ll_register_callbacks cb 0x55d7f973fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f3697fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 0x55d7f9ad34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f3697fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 0x55d7f9ad34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55930/0 (socket says 172.21.15.31:55930) 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.221+0000 7f3697fff640 1 -- 172.21.15.31:0/1250631744 learned_addr learned my addr 172.21.15.31:0/1250631744 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f369ed9d640 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f369e59c640 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7460 0x55d7f9ad61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f3697fff640 1 -- 172.21.15.31:0/1250631744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7460 msgr2=0x55d7f9ad61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f3697fff640 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7460 0x55d7f9ad61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f3697fff640 1 -- 172.21.15.31:0/1250631744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 msgr2=0x55d7f9ad3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f3697fff640 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f3697fff640 1 -- 172.21.15.31:0/1250631744 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7f99b9dc0 con 0x55d7f9ac7d80 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f369ed9d640 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f369e59c640 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7460 0x55d7f9ad61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f3697fff640 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 0x55d7f9ad34e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f3688009510 tx=0x7f368800f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76595b9f6fb8dc3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f3685ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f3685ffb640 1 -- 172.21.15.31:0/1250631744 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3688027030 con 0x55d7f9ac7d80 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f36a79ad740 1 -- 172.21.15.31:0/1250631744 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3688012aa0 con 0x55d7f9ac7d80 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f3685ffb640 1 -- 172.21.15.31:0/1250631744 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f368801f070 con 0x55d7f9ac7d80 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f3685ffb640 1 -- 172.21.15.31:0/1250631744 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3688031050 con 0x55d7f9ac7d80 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f36a79ad740 10 client.16275 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.222+0000 7f36a79ad740 1 -- 172.21.15.31:0/1250631744 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d7f97403f0 con 0x55d7f9ac7d80 2024-04-25T01:56:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.223+0000 7f3685ffb640 1 -- 172.21.15.31:0/1250631744 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3688012aa0 con 0x55d7f9ac7d80 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.223+0000 7f3685ffb640 1 -- 172.21.15.31:0/1250631744 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f368800f730 con 0x55d7f9ac7d80 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.223+0000 7f36a79ad740 10 client.16275 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.223+0000 7f36a79ad740 10 client.16275 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.223+0000 7f36a79ad740 20 client.16275 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.223+0000 7f36a79ad740 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55d7f9ae6880 0x55d7f9ae8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.223+0000 7f36a79ad740 4 client.16275 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.223+0000 7f36a79ad740 1 -- 172.21.15.31:0/1250631744 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7f9a39bb0 con 0x55d7f9ae6880 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.223+0000 7f369e59c640 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55d7f9ae6880 0x55d7f9ae8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.223+0000 7f369e59c640 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55d7f9ae6880 0x55d7f9ae8c70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7064fc95aa733f3f server_cookie=7ae1a7a87e07767f in_seq=0 out_seq=0 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.223+0000 7f3685ffb640 10 client.16275 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f3685ffb640 1 -- 172.21.15.31:0/1250631744 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7f9a39bb0 con 0x55d7f9ae6880 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f3685ffb640 10 client.16275 handle_command_reply: tid=0 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 1 client.16275 shutdown 2024-04-25T01:56:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 1 -- 172.21.15.31:0/1250631744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 msgr2=0x55d7f9ad34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 0x55d7f9ad34e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f3688009510 tx=0x7f368800f440 comp rx=0 tx=0).stop 2024-04-25T01:56:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 1 -- 172.21.15.31:0/1250631744 shutdown_connections 2024-04-25T01:56:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55d7f9ae6880 0x55d7f9ae8c70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7d80 0x55d7f9ad34e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f3685ffb640 0 client.16275 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:56:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 1 --2- 172.21.15.31:0/1250631744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7460 0x55d7f9ad61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 1 -- 172.21.15.31:0/1250631744 >> 172.21.15.31:0/1250631744 conn(0x55d7f9ac38a0 msgr2=0x55d7f9ac9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 1 -- 172.21.15.31:0/1250631744 shutdown_connections 2024-04-25T01:56:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 1 -- 172.21.15.31:0/1250631744 wait complete. 2024-04-25T01:56:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.224+0000 7f36a79ad740 20 client.16275 trim_cache size 0 max 16384 2024-04-25T01:56:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.240+0000 7f36a79ad740 1 Processor -- start 2024-04-25T01:56:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.240+0000 7f36a79ad740 1 -- start start 2024-04-25T01:56:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.240+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7460 0x55d7f9ae26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.240+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7d80 0x55d7f9acec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.240+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.240+0000 7f36a79ad740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f36a01ab060 con 0x55d7f9ac7460 2024-04-25T01:56:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.240+0000 7f36a79ad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f36a01acc60 con 0x55d7f9ac7d80 2024-04-25T01:56:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.240+0000 7f36a79ad740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f36a01ab730 con 0x55d7f9ad6980 2024-04-25T01:56:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f369e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7460 0x55d7f9ae26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f369e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7460 0x55d7f9ae26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55936/0 (socket says 172.21.15.31:55936) 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f3697fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7d80 0x55d7f9acec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f369e59c640 1 -- 172.21.15.31:0/679470470 learned_addr learned my addr 172.21.15.31:0/679470470 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f369ed9d640 1 --2- 172.21.15.31:0/679470470 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f369e59c640 1 -- 172.21.15.31:0/679470470 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ad6980 msgr2=0x55d7f9ad24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f369e59c640 1 --2- 172.21.15.31:0/679470470 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f369e59c640 1 -- 172.21.15.31:0/679470470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7d80 msgr2=0x55d7f9acec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f369e59c640 1 --2- 172.21.15.31:0/679470470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7d80 0x55d7f9acec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f369e59c640 1 -- 172.21.15.31:0/679470470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f980a7d0 con 0x55d7f9ac7460 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f369ed9d640 1 --2- 172.21.15.31:0/679470470 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f3697fff640 1 --2- 172.21.15.31:0/679470470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7d80 0x55d7f9acec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f369e59c640 1 --2- 172.21.15.31:0/679470470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7460 0x55d7f9ae26f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3690009fd0 tx=0x7f369006ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a158edf3a57b1594 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f3687fff640 1 -- 172.21.15.31:0/679470470 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3690003b10 con 0x55d7f9ac7460 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f3687fff640 1 -- 172.21.15.31:0/679470470 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3690066fa0 con 0x55d7f9ac7460 2024-04-25T01:56:45.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.241+0000 7f3687fff640 1 -- 172.21.15.31:0/679470470 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36900cd3c0 con 0x55d7f9ac7460 2024-04-25T01:56:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.242+0000 7f36a79ad740 1 -- 172.21.15.31:0/679470470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7460 msgr2=0x55d7f9ae26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.242+0000 7f36a79ad740 1 --2- 172.21.15.31:0/679470470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7460 0x55d7f9ae26f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3690009fd0 tx=0x7f369006ded0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.242+0000 7f36a79ad740 1 -- 172.21.15.31:0/679470470 shutdown_connections 2024-04-25T01:56:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.242+0000 7f36a79ad740 1 --2- 172.21.15.31:0/679470470 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ad6980 0x55d7f9ad24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.242+0000 7f36a79ad740 1 --2- 172.21.15.31:0/679470470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7d80 0x55d7f9acec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.242+0000 7f36a79ad740 1 --2- 172.21.15.31:0/679470470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ac7460 0x55d7f9ae26f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.242+0000 7f36a79ad740 1 -- 172.21.15.31:0/679470470 >> 172.21.15.31:0/679470470 conn(0x55d7f9ac38a0 msgr2=0x55d7f9ad1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.242+0000 7f36a79ad740 1 -- 172.21.15.31:0/679470470 shutdown_connections 2024-04-25T01:56:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.242+0000 7f36a79ad740 1 -- 172.21.15.31:0/679470470 wait complete. 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.242+0000 7f36a79ad740 1 -- start start 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.242+0000 7f36a79ad740 1 Processor -- start 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7460 0x55d7f9adec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7d80 0x55d7f9adf140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f36a79ad740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ad6980 0x55d7f9acfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f36a79ad740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f36a01ac4d0 con 0x55d7f9ad6980 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f36a79ad740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f36a01abe00 con 0x55d7f9ac7460 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f36a79ad740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f36a01a9060 con 0x55d7f9ac7d80 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f36a79ad740 10 client.0 _ll_register_callbacks cb 0x7f3690003d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f369ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ad6980 0x55d7f9acfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f369ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ad6980 0x55d7f9acfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55942/0 (socket says 172.21.15.31:55942) 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f369ed9d640 1 -- 172.21.15.31:0/4021654328 learned_addr learned my addr 172.21.15.31:0/4021654328 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f369e59c640 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7460 0x55d7f9adec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f3697fff640 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7d80 0x55d7f9adf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f369ed9d640 1 -- 172.21.15.31:0/4021654328 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7d80 msgr2=0x55d7f9adf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f369ed9d640 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7d80 0x55d7f9adf140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f369ed9d640 1 -- 172.21.15.31:0/4021654328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7460 msgr2=0x55d7f9adec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f369ed9d640 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7460 0x55d7f9adec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f369ed9d640 1 -- 172.21.15.31:0/4021654328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7f9ad0320 con 0x55d7f9ad6980 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f369e59c640 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7460 0x55d7f9adec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f369ed9d640 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ad6980 0x55d7f9acfb20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f369802ae00 tx=0x7f3698005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e568cdf771b4ed7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.243+0000 7f3685ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f3685ffb640 1 -- 172.21.15.31:0/4021654328 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3698059060 con 0x55d7f9ad6980 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f3685ffb640 1 -- 172.21.15.31:0/4021654328 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f3698045030 con 0x55d7f9ad6980 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f36a79ad740 1 -- 172.21.15.31:0/4021654328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3690066fa0 con 0x55d7f9ad6980 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f3685ffb640 1 -- 172.21.15.31:0/4021654328 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3698031610 con 0x55d7f9ad6980 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f36a79ad740 10 client.16281 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f36a79ad740 1 -- 172.21.15.31:0/4021654328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d7f9b4c610 con 0x55d7f9ad6980 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f3685ffb640 1 -- 172.21.15.31:0/4021654328 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3690066fa0 con 0x55d7f9ad6980 2024-04-25T01:56:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f3685ffb640 1 -- 172.21.15.31:0/4021654328 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f3698053070 con 0x55d7f9ad6980 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f36a79ad740 10 client.16281 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f36a79ad740 10 client.16281 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f36a79ad740 20 client.16281 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f36a79ad740 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55d7f9b4cd90 0x55d7f9b4f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f36a79ad740 4 client.16281 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.244+0000 7f36a79ad740 1 -- 172.21.15.31:0/4021654328 --> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d7f9ad3db0 con 0x55d7f9b4cd90 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f369e59c640 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55d7f9b4cd90 0x55d7f9b4f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f369e59c640 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55d7f9b4cd90 0x55d7f9b4f1b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=913fc4589dea6b9d server_cookie=a7853c111a42c8f0 in_seq=0 out_seq=0 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f3685ffb640 10 client.16281 ms_handle_connect on v2:172.21.15.40:6836/1901184087 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f3685ffb640 1 -- 172.21.15.31:0/4021654328 <== mds.2 v2:172.21.15.40:6836/1901184087 1 ==== command_reply(tid 0: 0 ) ==== 8+0+459 (crc 0 0 0) 0x55d7f9ad3db0 con 0x55d7f9b4cd90 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f3685ffb640 10 client.16281 handle_command_reply: tid=0 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 1 client.16281 shutdown 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 1 -- 172.21.15.31:0/4021654328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ad6980 msgr2=0x55d7f9acfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ad6980 0x55d7f9acfb20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f369802ae00 tx=0x7f3698005da0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 1 -- 172.21.15.31:0/4021654328 shutdown_connections 2024-04-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.40:6836/1901184087,v1:172.21.15.40:6837/1901184087] conn(0x55d7f9b4cd90 0x55d7f9b4f1b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7f9ad6980 0x55d7f9acfb20 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d7f9ac7d80 0x55d7f9adf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f3685ffb640 0 client.16281 ms_handle_reset on v2:172.21.15.40:6836/1901184087 2024-04-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 1 --2- 172.21.15.31:0/4021654328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d7f9ac7460 0x55d7f9adec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 1 -- 172.21.15.31:0/4021654328 >> 172.21.15.31:0/4021654328 conn(0x55d7f9ac38a0 msgr2=0x55d7f9aceaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 1 -- 172.21.15.31:0/4021654328 shutdown_connections 2024-04-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 1 -- 172.21.15.31:0/4021654328 wait complete. 2024-04-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.245+0000 7f36a79ad740 20 client.16281 trim_cache size 0 max 16384 2024-04-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "damage_type": "backtrace", 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 725033003, 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 1099511628623, 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/TIPS.DOC" 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "damage_type": "backtrace", 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 3601234302, 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 1099511628619, 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/HANDOUT.DOC" 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-04-25T01:56:45.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 -- 172.21.15.31:0/1957389843 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f368c0a2d00 msgr2=0x7f368c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f368c0a2d00 0x7f368c0a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3698004af0 tx=0x7f3698009910 comp rx=0 tx=0).stop 2024-04-25T01:56:45.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 -- 172.21.15.31:0/1957389843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a0100610 msgr2=0x7f36a0198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a0100610 0x7f36a0198ca0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f369000e9c0 tx=0x7f369000ee80 comp rx=0 tx=0).stop 2024-04-25T01:56:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 -- 172.21.15.31:0/1957389843 shutdown_connections 2024-04-25T01:56:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f368c0a2d00 0x7f368c0a51c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a01066b0 0x7f36a01a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f36a0100fd0 0x7f36a01991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 --2- 172.21.15.31:0/1957389843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a0100610 0x7f36a0198ca0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 -- 172.21.15.31:0/1957389843 >> 172.21.15.31:0/1957389843 conn(0x7f36a0077740 msgr2=0x7f36a00ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 -- 172.21.15.31:0/1957389843 shutdown_connections 2024-04-25T01:56:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.246+0000 7f36a4bc1640 1 -- 172.21.15.31:0/1957389843 wait complete. 2024-04-25T01:56:45.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:56:45.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 26266 3.07 MB/sec warmup 52 sec latency 19.440 ms 2024-04-25T01:56:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.368+0000 7f1457210640 1 -- 172.21.15.31:0/1653742579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f145010e0f0 msgr2=0x7f14501104e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.368+0000 7f1457210640 1 --2- 172.21.15.31:0/1653742579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f145010e0f0 0x7f14501104e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1448009ca0 tx=0x7f144802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 -- 172.21.15.31:0/1653742579 shutdown_connections 2024-04-25T01:56:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 --2- 172.21.15.31:0/1653742579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f145010e0f0 0x7f14501104e0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 --2- 172.21.15.31:0/1653742579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1450100e60 0x7f145010dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 --2- 172.21.15.31:0/1653742579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1450100540 0x7f1450100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 -- 172.21.15.31:0/1653742579 >> 172.21.15.31:0/1653742579 conn(0x7f1450077740 msgr2=0x7f1450077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 -- 172.21.15.31:0/1653742579 shutdown_connections 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 -- 172.21.15.31:0/1653742579 wait complete. 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 Processor -- start 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 -- start start 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1450100540 0x7f145019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1450100e60 0x7f145019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f145010e0f0 0x7f145019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1450113c20 con 0x7f145010e0f0 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1450113aa0 con 0x7f1450100e60 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.369+0000 7f1457210640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1450113da0 con 0x7f1450100540 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f14557ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f145010e0f0 0x7f145019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f14557ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f145010e0f0 0x7f145019eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55962/0 (socket says 172.21.15.31:55962) 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f14557ad640 1 -- 172.21.15.31:0/3601087349 learned_addr learned my addr 172.21.15.31:0/3601087349 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f1454fac640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1450100540 0x7f145019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f144ffff640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1450100e60 0x7f145019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f14557ad640 1 -- 172.21.15.31:0/3601087349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1450100540 msgr2=0x7f145019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f14557ad640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1450100540 0x7f145019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f14557ad640 1 -- 172.21.15.31:0/3601087349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1450100e60 msgr2=0x7f145019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f14557ad640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1450100e60 0x7f145019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f14557ad640 1 -- 172.21.15.31:0/3601087349 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145019f1f0 con 0x7f145010e0f0 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f144ffff640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1450100e60 0x7f145019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f14557ad640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f145010e0f0 0x7f145019eac0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1448005e30 tx=0x7f14480093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f144dffb640 1 -- 172.21.15.31:0/3601087349 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1448003c70 con 0x7f145010e0f0 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f144dffb640 1 -- 172.21.15.31:0/3601087349 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1448003e10 con 0x7f145010e0f0 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f1457210640 1 -- 172.21.15.31:0/3601087349 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f145019f480 con 0x7f145010e0f0 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.370+0000 7f144dffb640 1 -- 172.21.15.31:0/3601087349 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1448030c60 con 0x7f145010e0f0 2024-04-25T01:56:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.371+0000 7f1457210640 1 -- 172.21.15.31:0/3601087349 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14501a6d70 con 0x7f145010e0f0 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.373+0000 7f144dffb640 1 -- 172.21.15.31:0/3601087349 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1448004040 con 0x7f145010e0f0 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.373+0000 7f1459ff4740 1 Processor -- start 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.373+0000 7f1459ff4740 1 -- start start 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.373+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a125580 0x563d1a125960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.373+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a130030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.373+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a130570 0x563d1a132960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.373+0000 7f1459ff4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563d1a08a670 con 0x563d1a130570 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.373+0000 7f1459ff4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563d1a032970 con 0x563d1a125580 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.373+0000 7f1459ff4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563d1a086630 con 0x563d1a125ea0 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.373+0000 7f144ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a125580 0x563d1a125960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.373+0000 7f144ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a125580 0x563d1a125960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33764/0 (socket says 172.21.15.31:33764) 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f144ffff640 1 -- 172.21.15.31:0/4185344857 learned_addr learned my addr 172.21.15.31:0/4185344857 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f144ffff640 1 --2- 172.21.15.31:0/4185344857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a130570 0x563d1a132960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f144dffb640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f14400a2d80 0x7f14400a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f1454fac640 1 --2- 172.21.15.31:0/4185344857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a130030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f14557ad640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f14400a2d80 0x7f14400a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f144ffff640 1 -- 172.21.15.31:0/4185344857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 msgr2=0x563d1a130030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f144ffff640 1 --2- 172.21.15.31:0/4185344857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a130030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f144ffff640 1 -- 172.21.15.31:0/4185344857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a130570 msgr2=0x563d1a132960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f144ffff640 1 --2- 172.21.15.31:0/4185344857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a130570 0x563d1a132960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f144ffff640 1 -- 172.21.15.31:0/4185344857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d1a056400 con 0x563d1a125580 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f144dffb640 1 -- 172.21.15.31:0/3601087349 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f14480f0550 con 0x7f145010e0f0 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f144ffff640 1 --2- 172.21.15.31:0/4185344857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a125580 0x563d1a125960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f143c011b60 tx=0x7f143c005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69a087202d8e23d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f143affd640 1 -- 172.21.15.31:0/4185344857 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c00fc50 con 0x563d1a125580 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f14557ad640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f14400a2d80 0x7f14400a5240 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f14480bcb60 tx=0x7f144802a8c0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f143affd640 1 -- 172.21.15.31:0/4185344857 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f143c00fdf0 con 0x563d1a125580 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.374+0000 7f143affd640 1 -- 172.21.15.31:0/4185344857 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c00e620 con 0x563d1a125580 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.375+0000 7f1459ff4740 1 -- 172.21.15.31:0/4185344857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a125580 msgr2=0x563d1a125960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.375+0000 7f1459ff4740 1 --2- 172.21.15.31:0/4185344857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a125580 0x563d1a125960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f143c011b60 tx=0x7f143c005850 comp rx=0 tx=0).stop 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.375+0000 7f1459ff4740 1 -- 172.21.15.31:0/4185344857 shutdown_connections 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.375+0000 7f1459ff4740 1 --2- 172.21.15.31:0/4185344857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a130570 0x563d1a132960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.375+0000 7f1459ff4740 1 --2- 172.21.15.31:0/4185344857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a130030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.375+0000 7f1459ff4740 1 --2- 172.21.15.31:0/4185344857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a125580 0x563d1a125960 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.375+0000 7f1459ff4740 1 -- 172.21.15.31:0/4185344857 >> 172.21.15.31:0/4185344857 conn(0x563d1a1218a0 msgr2=0x563d1a121c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.375+0000 7f1459ff4740 1 -- 172.21.15.31:0/4185344857 shutdown_connections 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.375+0000 7f1459ff4740 1 -- 172.21.15.31:0/4185344857 wait complete. 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.375+0000 7f1459ff4740 1 -- start start 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f1459ff4740 1 Processor -- start 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a12fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a12d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a12d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f1459ff4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563d1a133360 con 0x563d1a125580 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f1459ff4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563d1a046050 con 0x563d1a130570 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f1459ff4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563d1a1334e0 con 0x563d1a125ea0 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f1459ff4740 10 client.0 _ll_register_callbacks cb 0x563d19d9dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f144ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a12fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f144ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a12fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55984/0 (socket says 172.21.15.31:55984) 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f144ffff640 1 -- 172.21.15.31:0/3515813705 learned_addr learned my addr 172.21.15.31:0/3515813705 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f1454fac640 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a12d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f144ffff640 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a12d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f144ffff640 1 -- 172.21.15.31:0/3515813705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 msgr2=0x563d1a12d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f144ffff640 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a12d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f144ffff640 1 -- 172.21.15.31:0/3515813705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 msgr2=0x563d1a12d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f144ffff640 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a12d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.376+0000 7f144ffff640 1 -- 172.21.15.31:0/3515813705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d1a017dc0 con 0x563d1a125580 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.377+0000 7f144ffff640 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a12fec0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f143c0057c0 tx=0x7f143c01fe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b297a6f1ba46dc56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.377+0000 7f142ffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.377+0000 7f142ffff640 1 -- 172.21.15.31:0/3515813705 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c00fc50 con 0x563d1a125580 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.377+0000 7f142ffff640 1 -- 172.21.15.31:0/3515813705 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f143c005230 con 0x563d1a125580 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.377+0000 7f1459ff4740 1 -- 172.21.15.31:0/3515813705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f143c00fdf0 con 0x563d1a125580 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.377+0000 7f142ffff640 1 -- 172.21.15.31:0/3515813705 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f143c030020 con 0x563d1a125580 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.377+0000 7f1459ff4740 10 client.16293 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.377+0000 7f1459ff4740 1 -- 172.21.15.31:0/3515813705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563d19d9e3f0 con 0x563d1a125580 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.377+0000 7f142ffff640 1 -- 172.21.15.31:0/3515813705 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f143c00fdf0 con 0x563d1a125580 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.377+0000 7f142ffff640 1 -- 172.21.15.31:0/3515813705 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f143c01d020 con 0x563d1a125580 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.378+0000 7f1459ff4740 10 client.16293 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.378+0000 7f1459ff4740 10 client.16293 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.378+0000 7f1459ff4740 20 client.16293 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.378+0000 7f1459ff4740 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563d1a1404b0 0x563d1a1428a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.378+0000 7f1459ff4740 4 client.16293 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:45.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.378+0000 7f1459ff4740 1 -- 172.21.15.31:0/3515813705 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d1a097bb0 con 0x563d1a1404b0 2024-04-25T01:56:45.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.378+0000 7f1454fac640 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563d1a1404b0 0x563d1a1428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.378+0000 7f1454fac640 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563d1a1404b0 0x563d1a1428a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2cbc6f0a5b4b29b9 server_cookie=47caf7eb1d8a3b2b in_seq=0 out_seq=0 2024-04-25T01:56:45.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.378+0000 7f142ffff640 10 client.16293 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:56:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f142ffff640 1 -- 172.21.15.31:0/3515813705 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563d1a097bb0 con 0x563d1a1404b0 2024-04-25T01:56:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f142ffff640 10 client.16293 handle_command_reply: tid=0 2024-04-25T01:56:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 1 client.16293 shutdown 2024-04-25T01:56:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 1 -- 172.21.15.31:0/3515813705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 msgr2=0x563d1a12fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a12fec0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f143c0057c0 tx=0x7f143c01fe60 comp rx=0 tx=0).stop 2024-04-25T01:56:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 1 -- 172.21.15.31:0/3515813705 shutdown_connections 2024-04-25T01:56:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563d1a1404b0 0x563d1a1428a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a12d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a12d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 1 --2- 172.21.15.31:0/3515813705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a12fec0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f142ffff640 0 client.16293 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:56:45.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 1 -- 172.21.15.31:0/3515813705 >> 172.21.15.31:0/3515813705 conn(0x563d1a1218a0 msgr2=0x563d1a1276a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 1 -- 172.21.15.31:0/3515813705 shutdown_connections 2024-04-25T01:56:45.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 1 -- 172.21.15.31:0/3515813705 wait complete. 2024-04-25T01:56:45.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.379+0000 7f1459ff4740 20 client.16293 trim_cache size 0 max 16384 2024-04-25T01:56:45.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1459ff4740 1 Processor -- start 2024-04-25T01:56:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1459ff4740 1 -- start start 2024-04-25T01:56:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125580 0x563d1a128980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125ea0 0x563d1a12f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a148110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1459ff4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14501af460 con 0x563d1a125ea0 2024-04-25T01:56:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1459ff4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f14501b1060 con 0x563d1a130570 2024-04-25T01:56:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1459ff4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f14501afb30 con 0x563d1a125580 2024-04-25T01:56:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1454fac640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125ea0 0x563d1a12f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1454fac640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125ea0 0x563d1a12f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55992/0 (socket says 172.21.15.31:55992) 2024-04-25T01:56:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1454fac640 1 -- 172.21.15.31:0/166263644 learned_addr learned my addr 172.21.15.31:0/166263644 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f144ffff640 1 --2- 172.21.15.31:0/166263644 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125580 0x563d1a128980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f144ffff640 1 --2- 172.21.15.31:0/166263644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1454fac640 1 -- 172.21.15.31:0/166263644 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125580 msgr2=0x563d1a128980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1454fac640 1 --2- 172.21.15.31:0/166263644 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125580 0x563d1a128980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1454fac640 1 -- 172.21.15.31:0/166263644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 msgr2=0x563d1a148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1454fac640 1 --2- 172.21.15.31:0/166263644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a148110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.393+0000 7f1454fac640 1 -- 172.21.15.31:0/166263644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d19e687d0 con 0x563d1a125ea0 2024-04-25T01:56:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.394+0000 7f1454fac640 1 --2- 172.21.15.31:0/166263644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125ea0 0x563d1a12f9f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x563d1a12ca10 tx=0x7f144400a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd729902e780c1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.394+0000 7f143a7fc640 1 -- 172.21.15.31:0/166263644 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444005220 con 0x563d1a125ea0 2024-04-25T01:56:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.394+0000 7f143a7fc640 1 -- 172.21.15.31:0/166263644 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1444004c50 con 0x563d1a125ea0 2024-04-25T01:56:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.394+0000 7f143a7fc640 1 -- 172.21.15.31:0/166263644 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14440204f0 con 0x563d1a125ea0 2024-04-25T01:56:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 -- 172.21.15.31:0/166263644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125ea0 msgr2=0x563d1a12f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 --2- 172.21.15.31:0/166263644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125ea0 0x563d1a12f9f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x563d1a12ca10 tx=0x7f144400a250 comp rx=0 tx=0).stop 2024-04-25T01:56:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 -- 172.21.15.31:0/166263644 shutdown_connections 2024-04-25T01:56:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 --2- 172.21.15.31:0/166263644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a148110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 --2- 172.21.15.31:0/166263644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125ea0 0x563d1a12f9f0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 --2- 172.21.15.31:0/166263644 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125580 0x563d1a128980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 -- 172.21.15.31:0/166263644 >> 172.21.15.31:0/166263644 conn(0x563d1a1218a0 msgr2=0x563d1a127560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 -- 172.21.15.31:0/166263644 shutdown_connections 2024-04-25T01:56:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 -- 172.21.15.31:0/166263644 wait complete. 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 -- start start 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 Processor -- start 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a138a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a133e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a1343b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14501b08d0 con 0x563d1a125580 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f14501b0200 con 0x563d1a130570 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.395+0000 7f1459ff4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f14501ad440 con 0x563d1a125ea0 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f1459ff4740 10 client.0 _ll_register_callbacks cb 0x7f143c020f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f144ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a138a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f144ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a138a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56002/0 (socket says 172.21.15.31:56002) 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f144ffff640 1 -- 172.21.15.31:0/1011617529 learned_addr learned my addr 172.21.15.31:0/1011617529 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f1454fac640 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f144ffff640 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a1343b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f144ffff640 1 -- 172.21.15.31:0/1011617529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 msgr2=0x563d1a133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f144ffff640 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a133e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f144ffff640 1 -- 172.21.15.31:0/1011617529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 msgr2=0x563d1a1343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f144ffff640 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a1343b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f144ffff640 1 -- 172.21.15.31:0/1011617529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d1a134aa0 con 0x563d1a125580 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f144ffff640 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a138a60 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f143c0059f0 tx=0x7f143c01fe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3aa771c2a3d8e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f142ffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f142ffff640 1 -- 172.21.15.31:0/1011617529 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c01f850 con 0x563d1a125580 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f142ffff640 1 -- 172.21.15.31:0/1011617529 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f143c01f9f0 con 0x563d1a125580 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.396+0000 7f1459ff4740 1 -- 172.21.15.31:0/1011617529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1444004c50 con 0x563d1a125580 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f142ffff640 1 -- 172.21.15.31:0/1011617529 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f143c01fbf0 con 0x563d1a125580 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f1459ff4740 10 client.16299 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f1459ff4740 1 -- 172.21.15.31:0/1011617529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563d1a1a6880 con 0x563d1a125580 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f142ffff640 1 -- 172.21.15.31:0/1011617529 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1444004c50 con 0x563d1a125580 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f142ffff640 1 -- 172.21.15.31:0/1011617529 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f143c01d040 con 0x563d1a125580 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f1459ff4740 10 client.16299 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f1459ff4740 10 client.16299 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f1459ff4740 20 client.16299 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f1459ff4740 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563d1a1a6fb0 0x563d1a1a93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f1459ff4740 4 client.16299 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f1459ff4740 1 -- 172.21.15.31:0/1011617529 --> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563d1a091f00 con 0x563d1a1a6fb0 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.397+0000 7f1454fac640 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563d1a1a6fb0 0x563d1a1a93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f142ffff640 10 client.16299 ms_handle_connect on v2:172.21.15.134:6838/450925094 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1454fac640 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563d1a1a6fb0 0x563d1a1a93d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1c6625b47ad2a275 server_cookie=ca49a2f1a7ae0e4b in_seq=0 out_seq=0 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f142ffff640 1 -- 172.21.15.31:0/1011617529 <== mds.1 v2:172.21.15.134:6838/450925094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563d1a091f00 con 0x563d1a1a6fb0 2024-04-25T01:56:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f142ffff640 10 client.16299 handle_command_reply: tid=0 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 1 client.16299 shutdown 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 1 -- 172.21.15.31:0/1011617529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 msgr2=0x563d1a138a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a138a60 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f143c0059f0 tx=0x7f143c01fe60 comp rx=0 tx=0).stop 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 1 -- 172.21.15.31:0/1011617529 shutdown_connections 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.134:6838/450925094,v1:172.21.15.134:6839/450925094] conn(0x563d1a1a6fb0 0x563d1a1a93d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d1a130570 0x563d1a1343b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563d1a125ea0 0x563d1a133e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 1 --2- 172.21.15.31:0/1011617529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d1a125580 0x563d1a138a60 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 1 -- 172.21.15.31:0/1011617529 >> 172.21.15.31:0/1011617529 conn(0x563d1a1218a0 msgr2=0x563d1a1202b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f142ffff640 0 client.16299 ms_handle_reset on v2:172.21.15.134:6838/450925094 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 1 -- 172.21.15.31:0/1011617529 shutdown_connections 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 1 -- 172.21.15.31:0/1011617529 wait complete. 2024-04-25T01:56:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.398+0000 7f1459ff4740 20 client.16299 trim_cache size 0 max 16384 2024-04-25T01:56:45.408 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:56:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 -- 172.21.15.31:0/3601087349 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f14400a2d80 msgr2=0x7f14400a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f14400a2d80 0x7f14400a5240 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f14480bcb60 tx=0x7f144802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 -- 172.21.15.31:0/3601087349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f145010e0f0 msgr2=0x7f145019eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f145010e0f0 0x7f145019eac0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1448005e30 tx=0x7f14480093d0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 -- 172.21.15.31:0/3601087349 shutdown_connections 2024-04-25T01:56:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f14400a2d80 0x7f14400a5240 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f145010e0f0 0x7f145019eac0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1450100e60 0x7f145019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 --2- 172.21.15.31:0/3601087349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1450100540 0x7f145019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 -- 172.21.15.31:0/3601087349 >> 172.21.15.31:0/3601087349 conn(0x7f1450077740 msgr2=0x7f145010f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 -- 172.21.15.31:0/3601087349 shutdown_connections 2024-04-25T01:56:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.399+0000 7f1457210640 1 -- 172.21.15.31:0/3601087349 wait complete. 2024-04-25T01:56:45.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:56:45.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.559+0000 7f21ed48c640 1 -- 172.21.15.31:0/2772911222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e8100610 msgr2=0x7f21e8100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.559+0000 7f21ed48c640 1 --2- 172.21.15.31:0/2772911222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e8100610 0x7f21e8100a90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009900 tx=0x7f21d002a410 comp rx=0 tx=0).stop 2024-04-25T01:56:45.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.559+0000 7f21ed48c640 1 -- 172.21.15.31:0/2772911222 shutdown_connections 2024-04-25T01:56:45.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.559+0000 7f21ed48c640 1 --2- 172.21.15.31:0/2772911222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e8100fd0 0x7f21e810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.559+0000 7f21ed48c640 1 --2- 172.21.15.31:0/2772911222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e8100610 0x7f21e8100a90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.559+0000 7f21ed48c640 1 --2- 172.21.15.31:0/2772911222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21e81066b0 0x7f21e8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.559+0000 7f21ed48c640 1 -- 172.21.15.31:0/2772911222 >> 172.21.15.31:0/2772911222 conn(0x7f21e8077740 msgr2=0x7f21e8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.559+0000 7f21ed48c640 1 -- 172.21.15.31:0/2772911222 shutdown_connections 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.559+0000 7f21ed48c640 1 -- 172.21.15.31:0/2772911222 wait complete. 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21ed48c640 1 Processor -- start 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21ed48c640 1 -- start start 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21ed48c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e8100610 0x7f21e810df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21ed48c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21e8100fd0 0x7f21e810e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21ed48c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e81066b0 0x7f21e810e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21ed48c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f21e81119e0 con 0x7f21e8100fd0 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21ed48c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21e8111860 con 0x7f21e81066b0 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21ed48c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f21e8111b60 con 0x7f21e8100610 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21e8100fd0 0x7f21e810e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21e8100fd0 0x7f21e810e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56022/0 (socket says 172.21.15.31:56022) 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21dffff640 1 -- 172.21.15.31:0/2208248969 learned_addr learned my addr 172.21.15.31:0/2208248969 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21e77fe640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e81066b0 0x7f21e810e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.560+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e8100610 0x7f21e810df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21dffff640 1 -- 172.21.15.31:0/2208248969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e8100610 msgr2=0x7f21e810df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21dffff640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e8100610 0x7f21e810df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21dffff640 1 -- 172.21.15.31:0/2208248969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e81066b0 msgr2=0x7f21e810e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21dffff640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e81066b0 0x7f21e810e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21dffff640 1 -- 172.21.15.31:0/2208248969 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e81a6320 con 0x7f21e8100fd0 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21e77fe640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e81066b0 0x7f21e810e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e8100610 0x7f21e810df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21dffff640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21e8100fd0 0x7f21e810e4b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f21d0006010 tx=0x7f21d00040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21e4ff9640 1 -- 172.21.15.31:0/2208248969 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d003f030 con 0x7f21e8100fd0 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21ed48c640 1 -- 172.21.15.31:0/2208248969 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21e81a65b0 con 0x7f21e8100fd0 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21ed48c640 1 -- 172.21.15.31:0/2208248969 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e81a6af0 con 0x7f21e8100fd0 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21e4ff9640 1 -- 172.21.15.31:0/2208248969 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21d00044c0 con 0x7f21e8100fd0 2024-04-25T01:56:45.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.561+0000 7f21e4ff9640 1 -- 172.21.15.31:0/2208248969 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0035070 con 0x7f21e8100fd0 2024-04-25T01:56:45.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.564+0000 7f21e4ff9640 1 -- 172.21.15.31:0/2208248969 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f21d0042030 con 0x7f21e8100fd0 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.564+0000 7f21f0199740 1 Processor -- start 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.564+0000 7f21f0199740 1 -- start start 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.564+0000 7f21e4ff9640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21d40a2db0 0x7f21d40a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e4ff9640 1 -- 172.21.15.31:0/2208248969 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f21d00ef5f0 con 0x7f21e8100fd0 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e77fe640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21d40a2db0 0x7f21d40a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21f0199740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dacfd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21f0199740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfdea0 0x55c4dad08030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21f0199740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 0x55c4dad0a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21f0199740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c4dac62670 con 0x55c4dacfd580 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21f0199740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c4dac0a970 con 0x55c4dacfdea0 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21f0199740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c4dac5e630 con 0x55c4dad08570 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dacfd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dacfd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56032/0 (socket says 172.21.15.31:56032) 2024-04-25T01:56:45.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e6ffd640 1 -- 172.21.15.31:0/1045495042 learned_addr learned my addr 172.21.15.31:0/1045495042 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21dffff640 1 --2- 172.21.15.31:0/1045495042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfdea0 0x55c4dad08030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e77fe640 1 --2- 172.21.15.31:0/1045495042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 0x55c4dad0a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e6ffd640 1 -- 172.21.15.31:0/1045495042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 msgr2=0x55c4dad0a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/1045495042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 0x55c4dad0a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e6ffd640 1 -- 172.21.15.31:0/1045495042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfdea0 msgr2=0x55c4dad08030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/1045495042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfdea0 0x55c4dad08030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e6ffd640 1 -- 172.21.15.31:0/1045495042 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4dac2e400 con 0x55c4dacfd580 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e77fe640 1 --2- 172.21.15.31:0/1045495042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 0x55c4dad0a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21dffff640 1 --2- 172.21.15.31:0/1045495042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfdea0 0x55c4dad08030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e77fe640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21d40a2db0 0x7f21d40a5270 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f21e0009fd0 tx=0x7f21e000fb40 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.565+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/1045495042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dacfd960 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f21d8011440 tx=0x7f21d8011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4504e39d769228 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.566+0000 7f21dd7fa640 1 -- 172.21.15.31:0/1045495042 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d8004310 con 0x55c4dacfd580 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.566+0000 7f21dd7fa640 1 -- 172.21.15.31:0/1045495042 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21d80044b0 con 0x55c4dacfd580 2024-04-25T01:56:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.566+0000 7f21dd7fa640 1 -- 172.21.15.31:0/1045495042 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d80035f0 con 0x55c4dacfd580 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.566+0000 7f21f0199740 1 -- 172.21.15.31:0/1045495042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 msgr2=0x55c4dacfd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.566+0000 7f21f0199740 1 --2- 172.21.15.31:0/1045495042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dacfd960 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f21d8011440 tx=0x7f21d8011900 comp rx=0 tx=0).stop 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.566+0000 7f21f0199740 1 -- 172.21.15.31:0/1045495042 shutdown_connections 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.566+0000 7f21f0199740 1 --2- 172.21.15.31:0/1045495042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 0x55c4dad0a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.566+0000 7f21f0199740 1 --2- 172.21.15.31:0/1045495042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfdea0 0x55c4dad08030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.566+0000 7f21f0199740 1 --2- 172.21.15.31:0/1045495042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dacfd960 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.566+0000 7f21f0199740 1 -- 172.21.15.31:0/1045495042 >> 172.21.15.31:0/1045495042 conn(0x55c4dacf98a0 msgr2=0x55c4dacf9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.566+0000 7f21f0199740 1 -- 172.21.15.31:0/1045495042 shutdown_connections 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21f0199740 1 -- 172.21.15.31:0/1045495042 wait complete. 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21f0199740 1 -- start start 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21f0199740 1 Processor -- start 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21f0199740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfd580 0x55c4dad07f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21f0199740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dacfdea0 0x55c4dad051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21f0199740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dad08570 0x55c4dad05770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21f0199740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c4dad0b360 con 0x55c4dad08570 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21f0199740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c4dac1e050 con 0x55c4dacfd580 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21f0199740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c4dad0b4e0 con 0x55c4dacfdea0 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21f0199740 10 client.0 _ll_register_callbacks cb 0x55c4da975be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21e77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dad08570 0x55c4dad05770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21e6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfd580 0x55c4dad07f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21dffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dacfdea0 0x55c4dad051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21e77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dad08570 0x55c4dad05770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56038/0 (socket says 172.21.15.31:56038) 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21e77fe640 1 -- 172.21.15.31:0/3858821604 learned_addr learned my addr 172.21.15.31:0/3858821604 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.567+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfd580 0x55c4dad07f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33832/0 (socket says 172.21.15.31:33832) 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21e77fe640 1 -- 172.21.15.31:0/3858821604 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dacfdea0 msgr2=0x55c4dad051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21e77fe640 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dacfdea0 0x55c4dad051f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21e77fe640 1 -- 172.21.15.31:0/3858821604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfd580 msgr2=0x55c4dad07f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21e77fe640 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfd580 0x55c4dad07f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21e77fe640 1 -- 172.21.15.31:0/3858821604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4dabefdc0 con 0x55c4dad08570 2024-04-25T01:56:45.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21e77fe640 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dad08570 0x55c4dad05770 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f21e0003bd0 tx=0x7f21e0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad188bb198bf376 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21ce7fc640 1 -- 172.21.15.31:0/3858821604 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e0017070 con 0x55c4dad08570 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21ce7fc640 1 -- 172.21.15.31:0/3858821604 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f21e001d400 con 0x55c4dad08570 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21f0199740 1 -- 172.21.15.31:0/3858821604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21d80044b0 con 0x55c4dad08570 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21ce7fc640 1 -- 172.21.15.31:0/3858821604 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f21e0027670 con 0x55c4dad08570 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21f0199740 10 client.16311 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21f0199740 1 -- 172.21.15.31:0/3858821604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c4da9763f0 con 0x55c4dad08570 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.568+0000 7f21ce7fc640 1 -- 172.21.15.31:0/3858821604 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21d80044b0 con 0x55c4dad08570 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.569+0000 7f21ce7fc640 1 -- 172.21.15.31:0/3858821604 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f21e001da30 con 0x55c4dad08570 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.569+0000 7f21f0199740 10 client.16311 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.569+0000 7f21f0199740 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.569+0000 7f21f0199740 20 client.16311 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.569+0000 7f21f0199740 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c4dad185a0 0x55c4dad1aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.569+0000 7f21f0199740 4 client.16311 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.569+0000 7f21f0199740 1 -- 172.21.15.31:0/3858821604 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4dac6fbb0 con 0x55c4dad185a0 2024-04-25T01:56:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.569+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c4dad185a0 0x55c4dad1aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.569+0000 7f21ce7fc640 10 client.16311 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.569+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c4dad185a0 0x55c4dad1aa10 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86a9006568106383 server_cookie=ae69b7d783a52ae3 in_seq=0 out_seq=0 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21ce7fc640 1 -- 172.21.15.31:0/3858821604 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c4dac6fbb0 con 0x55c4dad185a0 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21ce7fc640 10 client.16311 handle_command_reply: tid=0 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 1 client.16311 shutdown 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 1 -- 172.21.15.31:0/3858821604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dad08570 msgr2=0x55c4dad05770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dad08570 0x55c4dad05770 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f21e0003bd0 tx=0x7f21e0004ab0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 1 -- 172.21.15.31:0/3858821604 shutdown_connections 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c4dad185a0 0x55c4dad1aa10 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dad08570 0x55c4dad05770 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dacfdea0 0x55c4dad051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 1 --2- 172.21.15.31:0/3858821604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfd580 0x55c4dad07f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21ce7fc640 0 client.16311 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:45.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 1 -- 172.21.15.31:0/3858821604 >> 172.21.15.31:0/3858821604 conn(0x55c4dacf98a0 msgr2=0x55c4dacf8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 1 -- 172.21.15.31:0/3858821604 shutdown_connections 2024-04-25T01:56:45.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 1 -- 172.21.15.31:0/3858821604 wait complete. 2024-04-25T01:56:45.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.570+0000 7f21f0199740 20 client.16311 trim_cache size 0 max 16384 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.584+0000 7f21f0199740 1 Processor -- start 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.584+0000 7f21f0199740 1 -- start start 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21f0199740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dad14220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21f0199740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dacfdea0 0x55c4dad00580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21f0199740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dad08570 0x55c4dad07a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21f0199740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f21e81af430 con 0x55c4dacfd580 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21f0199740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21e81b1030 con 0x55c4dad08570 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21f0199740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f21e81afb00 con 0x55c4dacfdea0 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21e6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dad14220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21e6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dad14220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56044/0 (socket says 172.21.15.31:56044) 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21e6ffd640 1 -- 172.21.15.31:0/2253773536 learned_addr learned my addr 172.21.15.31:0/2253773536 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21dffff640 1 --2- 172.21.15.31:0/2253773536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dacfdea0 0x55c4dad00580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21e77fe640 1 --2- 172.21.15.31:0/2253773536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dad08570 0x55c4dad07a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21e6ffd640 1 -- 172.21.15.31:0/2253773536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dacfdea0 msgr2=0x55c4dad00580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/2253773536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dacfdea0 0x55c4dad00580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21e6ffd640 1 -- 172.21.15.31:0/2253773536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dad08570 msgr2=0x55c4dad07a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/2253773536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dad08570 0x55c4dad07a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21e6ffd640 1 -- 172.21.15.31:0/2253773536 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4daa407d0 con 0x55c4dacfd580 2024-04-25T01:56:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/2253773536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dad14220 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f21d80058c0 tx=0x7f21d8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa53ea2dcd1b0e35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.585+0000 7f21dcff9640 1 -- 172.21.15.31:0/2253773536 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d8031070 con 0x55c4dacfd580 2024-04-25T01:56:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.586+0000 7f21dcff9640 1 -- 172.21.15.31:0/2253773536 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21d802b330 con 0x55c4dacfd580 2024-04-25T01:56:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.586+0000 7f21dcff9640 1 -- 172.21.15.31:0/2253773536 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d801d040 con 0x55c4dacfd580 2024-04-25T01:56:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.586+0000 7f21f0199740 1 -- 172.21.15.31:0/2253773536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 msgr2=0x55c4dad14220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.586+0000 7f21f0199740 1 --2- 172.21.15.31:0/2253773536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dad14220 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f21d80058c0 tx=0x7f21d8009310 comp rx=0 tx=0).stop 2024-04-25T01:56:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.586+0000 7f21f0199740 1 -- 172.21.15.31:0/2253773536 shutdown_connections 2024-04-25T01:56:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.586+0000 7f21f0199740 1 --2- 172.21.15.31:0/2253773536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dad08570 0x55c4dad07a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.586+0000 7f21f0199740 1 --2- 172.21.15.31:0/2253773536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dacfdea0 0x55c4dad00580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.586+0000 7f21f0199740 1 --2- 172.21.15.31:0/2253773536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfd580 0x55c4dad14220 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.586+0000 7f21f0199740 1 -- 172.21.15.31:0/2253773536 >> 172.21.15.31:0/2253773536 conn(0x55c4dacf98a0 msgr2=0x55c4dacffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.586+0000 7f21f0199740 1 -- 172.21.15.31:0/2253773536 shutdown_connections 2024-04-25T01:56:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.586+0000 7f21f0199740 1 -- 172.21.15.31:0/2253773536 wait complete. 2024-04-25T01:56:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21f0199740 1 -- start start 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21f0199740 1 Processor -- start 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21f0199740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfd580 0x55c4dad10b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21f0199740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfdea0 0x55c4dad0bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21f0199740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 0x55c4dad0c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21f0199740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f21e81b08a0 con 0x55c4dacfdea0 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21f0199740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21e81b01d0 con 0x55c4dacfd580 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21f0199740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f21e81ad410 con 0x55c4dad08570 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21f0199740 10 client.0 _ll_register_callbacks cb 0x7f21e000fe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfdea0 0x55c4dad0bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfdea0 0x55c4dad0bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56050/0 (socket says 172.21.15.31:56050) 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21dffff640 1 -- 172.21.15.31:0/671536766 learned_addr learned my addr 172.21.15.31:0/671536766 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21e6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfd580 0x55c4dad10b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21e77fe640 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 0x55c4dad0c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21dffff640 1 -- 172.21.15.31:0/671536766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 msgr2=0x55c4dad0c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21dffff640 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 0x55c4dad0c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21dffff640 1 -- 172.21.15.31:0/671536766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfd580 msgr2=0x55c4dad10b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21dffff640 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfd580 0x55c4dad10b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21dffff640 1 -- 172.21.15.31:0/671536766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4dad0ccb0 con 0x55c4dacfdea0 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.587+0000 7f21e77fe640 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 0x55c4dad0c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21dffff640 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfdea0 0x55c4dad0bf00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f21d00bb910 tx=0x7f21d00bffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587e87f1542541ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21ce7fc640 1 -- 172.21.15.31:0/671536766 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0049040 con 0x55c4dacfdea0 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21ce7fc640 1 -- 172.21.15.31:0/671536766 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f21d00bbe70 con 0x55c4dacfdea0 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21f0199740 1 -- 172.21.15.31:0/671536766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21d802b330 con 0x55c4dacfdea0 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21ce7fc640 1 -- 172.21.15.31:0/671536766 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f21d0003df0 con 0x55c4dacfdea0 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21f0199740 10 client.16317 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21f0199740 1 -- 172.21.15.31:0/671536766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c4dad7ec50 con 0x55c4dacfdea0 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21ce7fc640 1 -- 172.21.15.31:0/671536766 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21d802b330 con 0x55c4dacfdea0 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21ce7fc640 1 -- 172.21.15.31:0/671536766 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f21d00bb0f0 con 0x55c4dacfdea0 2024-04-25T01:56:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21f0199740 10 client.16317 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21f0199740 10 client.16317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21f0199740 20 client.16317 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21f0199740 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c4dad7f460 0x55c4dad81880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21f0199740 4 client.16317 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.588+0000 7f21f0199740 1 -- 172.21.15.31:0/671536766 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4dac69f00 con 0x55c4dad7f460 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.589+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c4dad7f460 0x55c4dad81880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.589+0000 7f21e6ffd640 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c4dad7f460 0x55c4dad81880 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d03963fb8ee9b1a server_cookie=a0d6639ab2f38c54 in_seq=0 out_seq=0 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.589+0000 7f21ce7fc640 10 client.16317 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21ce7fc640 1 -- 172.21.15.31:0/671536766 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4dac69f00 con 0x55c4dad7f460 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21ce7fc640 10 client.16317 handle_command_reply: tid=0 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 1 client.16317 shutdown 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 1 -- 172.21.15.31:0/671536766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfdea0 msgr2=0x55c4dad0bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfdea0 0x55c4dad0bf00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f21d00bb910 tx=0x7f21d00bffd0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 1 -- 172.21.15.31:0/671536766 shutdown_connections 2024-04-25T01:56:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c4dad7f460 0x55c4dad81880 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4dad08570 0x55c4dad0c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21ce7fc640 0 client.16317 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:45.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dacfdea0 0x55c4dad0bf00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 1 --2- 172.21.15.31:0/671536766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4dacfd580 0x55c4dad10b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 1 -- 172.21.15.31:0/671536766 >> 172.21.15.31:0/671536766 conn(0x55c4dacf98a0 msgr2=0x55c4dad00700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 1 -- 172.21.15.31:0/671536766 shutdown_connections 2024-04-25T01:56:45.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 1 -- 172.21.15.31:0/671536766 wait complete. 2024-04-25T01:56:45.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21f0199740 20 client.16317 trim_cache size 0 max 16384 2024-04-25T01:56:45.597 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T01:56:45.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21ed48c640 1 -- 172.21.15.31:0/2208248969 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21d40a2db0 msgr2=0x7f21d40a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.590+0000 7f21ed48c640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21d40a2db0 0x7f21d40a5270 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f21e0009fd0 tx=0x7f21e000fb40 comp rx=0 tx=0).stop 2024-04-25T01:56:45.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.591+0000 7f21ed48c640 1 -- 172.21.15.31:0/2208248969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21e8100fd0 msgr2=0x7f21e810e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.591+0000 7f21ed48c640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21e8100fd0 0x7f21e810e4b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f21d0006010 tx=0x7f21d00040c0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.591+0000 7f21ed48c640 1 -- 172.21.15.31:0/2208248969 shutdown_connections 2024-04-25T01:56:45.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.591+0000 7f21ed48c640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21d40a2db0 0x7f21d40a5270 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.591+0000 7f21ed48c640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e81066b0 0x7f21e810e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.591+0000 7f21ed48c640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21e8100fd0 0x7f21e810e4b0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.591+0000 7f21ed48c640 1 --2- 172.21.15.31:0/2208248969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e8100610 0x7f21e810df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.591+0000 7f21ed48c640 1 -- 172.21.15.31:0/2208248969 >> 172.21.15.31:0/2208248969 conn(0x7f21e8077740 msgr2=0x7f21e80ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.591+0000 7f21ed48c640 1 -- 172.21.15.31:0/2208248969 shutdown_connections 2024-04-25T01:56:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T01:56:45.591+0000 7f21ed48c640 1 -- 172.21.15.31:0/2208248969 wait complete. 2024-04-25T01:56:45.607 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 725033003, 'ino': 1099511628623, 'path': '/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/TIPS.DOC'}, {'damage_type': 'backtrace', 'id': 3601234302, 'ino': 1099511628619, 'path': '/volumes/qa/sv_0/4cb36737-c3f4-42ff-b28f-ef38ae1b3b6d/client.0/tmp/clients/client0/~dmtmp/WORD/HANDOUT.DOC'}] 2024-04-25T01:56:45.607 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-04-25T01:56:46.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:45 smithi040 ceph-mon[32418]: pgmap v71: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-25T01:56:46.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:45 smithi134 ceph-mon[33098]: pgmap v71: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-25T01:56:46.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:45 smithi031 ceph-mon[29204]: pgmap v71: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-25T01:56:46.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 26856 3.03 MB/sec warmup 53 sec latency 17.813 ms 2024-04-25T01:56:47.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 27425 3.01 MB/sec warmup 54 sec latency 14.160 ms 2024-04-25T01:56:48.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:47 smithi040 ceph-mon[32418]: pgmap v72: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-04-25T01:56:48.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:47 smithi134 ceph-mon[33098]: pgmap v72: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-04-25T01:56:48.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:47 smithi031 ceph-mon[29204]: pgmap v72: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-04-25T01:56:48.300 INFO:tasks.workunit.client.0.smithi031.stdout: 1 28162 3.08 MB/sec warmup 55 sec latency 14.612 ms 2024-04-25T01:56:49.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 28575 3.08 MB/sec warmup 56 sec latency 14.000 ms 2024-04-25T01:56:50.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:49 smithi040 ceph-mon[32418]: pgmap v73: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-04-25T01:56:50.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:49 smithi134 ceph-mon[33098]: pgmap v73: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-04-25T01:56:50.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:49 smithi031 ceph-mon[29204]: pgmap v73: 97 pgs: 97 active+clean; 33 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.1 MiB/s wr, 129 op/s 2024-04-25T01:56:50.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 28850 3.05 MB/sec warmup 57 sec latency 17.964 ms 2024-04-25T01:56:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.058+0000 7f982c342640 1 -- 172.21.15.31:0/2562960729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9824100e90 msgr2=0x7f982410dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.058+0000 7f982c342640 1 --2- 172.21.15.31:0/2562960729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9824100e90 0x7f982410dbe0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f9814009900 tx=0x7f981402a410 comp rx=0 tx=0).stop 2024-04-25T01:56:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.058+0000 7f982c342640 1 -- 172.21.15.31:0/2562960729 shutdown_connections 2024-04-25T01:56:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.058+0000 7f982c342640 1 --2- 172.21.15.31:0/2562960729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f982410e120 0x7f9824110510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.058+0000 7f982c342640 1 --2- 172.21.15.31:0/2562960729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9824100e90 0x7f982410dbe0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.058+0000 7f982c342640 1 --2- 172.21.15.31:0/2562960729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9824100570 0x7f9824100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.058+0000 7f982c342640 1 -- 172.21.15.31:0/2562960729 >> 172.21.15.31:0/2562960729 conn(0x7f98240777c0 msgr2=0x7f9824077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.058+0000 7f982c342640 1 -- 172.21.15.31:0/2562960729 shutdown_connections 2024-04-25T01:56:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.058+0000 7f982c342640 1 -- 172.21.15.31:0/2562960729 wait complete. 2024-04-25T01:56:51.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982c342640 1 Processor -- start 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982c342640 1 -- start start 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982c342640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9824100570 0x7f9824102850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982c342640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9824100e90 0x7f9824102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982c342640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982410e120 0x7f98241032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982c342640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9824113be0 con 0x7f982410e120 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982c342640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9824113a60 con 0x7f9824100570 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982c342640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9824113d60 con 0x7f9824100e90 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982a8df640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982410e120 0x7f98241032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982a8df640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982410e120 0x7f98241032d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56070/0 (socket says 172.21.15.31:56070) 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982a8df640 1 -- 172.21.15.31:0/2453384704 learned_addr learned my addr 172.21.15.31:0/2453384704 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f98298dd640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9824100e90 0x7f9824102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.059+0000 7f982a0de640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9824100570 0x7f9824102850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.060+0000 7f982a8df640 1 -- 172.21.15.31:0/2453384704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9824100e90 msgr2=0x7f9824102d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.060+0000 7f982a8df640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9824100e90 0x7f9824102d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.060+0000 7f982a8df640 1 -- 172.21.15.31:0/2453384704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9824100570 msgr2=0x7f9824102850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.060+0000 7f982a8df640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9824100570 0x7f9824102850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.060+0000 7f982a8df640 1 -- 172.21.15.31:0/2453384704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98241aa850 con 0x7f982410e120 2024-04-25T01:56:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.060+0000 7f982a8df640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982410e120 0x7f98241032d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f981c00eb70 tx=0x7f981c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.060+0000 7f981b7fe640 1 -- 172.21.15.31:0/2453384704 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c004d20 con 0x7f982410e120 2024-04-25T01:56:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.060+0000 7f981b7fe640 1 -- 172.21.15.31:0/2453384704 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f981c00ce70 con 0x7f982410e120 2024-04-25T01:56:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.060+0000 7f981b7fe640 1 -- 172.21.15.31:0/2453384704 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c0107a0 con 0x7f982410e120 2024-04-25T01:56:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.060+0000 7f982c342640 1 -- 172.21.15.31:0/2453384704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98241aab40 con 0x7f982410e120 2024-04-25T01:56:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.060+0000 7f982c342640 1 -- 172.21.15.31:0/2453384704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98241ab010 con 0x7f982410e120 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.063+0000 7f981b7fe640 1 -- 172.21.15.31:0/2453384704 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f981c004fc0 con 0x7f982410e120 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.063+0000 7f982f050740 1 Processor -- start 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.063+0000 7f982f050740 1 -- start start 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.063+0000 7f982f050740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988240 0x559c74988620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.063+0000 7f982f050740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 0x559c74992cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.063+0000 7f982f050740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74993230 0x559c74995620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.063+0000 7f982f050740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559c748eccb0 con 0x559c74988240 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.063+0000 7f982f050740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559c748955b0 con 0x559c74993230 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.063+0000 7f982f050740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559c748e8c70 con 0x559c74988b60 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.063+0000 7f98298dd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988240 0x559c74988620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f982a0de640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 0x559c74992cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f982a0de640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 0x559c74992cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:37680/0 (socket says 172.21.15.31:37680) 2024-04-25T01:56:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f982a0de640 1 -- 172.21.15.31:0/3518457459 learned_addr learned my addr 172.21.15.31:0/3518457459 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f98298dd640 1 --2- 172.21.15.31:0/3518457459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74993230 0x559c74995620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f981b7fe640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f98100a2da0 0x7f98100a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f982a0de640 1 -- 172.21.15.31:0/3518457459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74993230 msgr2=0x559c74995620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f982a0de640 1 --2- 172.21.15.31:0/3518457459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74993230 0x559c74995620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f982a0de640 1 -- 172.21.15.31:0/3518457459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988240 msgr2=0x559c74988620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f982a0de640 1 --2- 172.21.15.31:0/3518457459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988240 0x559c74988620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f982a0de640 1 -- 172.21.15.31:0/3518457459 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c74601210 con 0x559c74988b60 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f982a8df640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f98100a2da0 0x7f98100a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f98298dd640 1 --2- 172.21.15.31:0/3518457459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74993230 0x559c74995620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f981b7fe640 1 -- 172.21.15.31:0/2453384704 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f981c0cfb60 con 0x7f982410e120 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f982a0de640 1 --2- 172.21.15.31:0/3518457459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 0x559c74992cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f982000bd10 tx=0x7f98200107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0e54a97d776cf1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f9818bfe640 1 -- 172.21.15.31:0/3518457459 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820004030 con 0x559c74988b60 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f982a8df640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f98100a2da0 0x7f98100a5260 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f981c098530 tx=0x7f981c06f140 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f9818bfe640 1 -- 172.21.15.31:0/3518457459 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98200035f0 con 0x559c74988b60 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.064+0000 7f9818bfe640 1 -- 172.21.15.31:0/3518457459 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820011640 con 0x559c74988b60 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.065+0000 7f982f050740 1 -- 172.21.15.31:0/3518457459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 msgr2=0x559c74992cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.065+0000 7f982f050740 1 --2- 172.21.15.31:0/3518457459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 0x559c74992cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f982000bd10 tx=0x7f98200107a0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.065+0000 7f982f050740 1 -- 172.21.15.31:0/3518457459 shutdown_connections 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.065+0000 7f982f050740 1 --2- 172.21.15.31:0/3518457459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74993230 0x559c74995620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.065+0000 7f982f050740 1 --2- 172.21.15.31:0/3518457459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 0x559c74992cf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.065+0000 7f982f050740 1 --2- 172.21.15.31:0/3518457459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988240 0x559c74988620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.065+0000 7f982f050740 1 -- 172.21.15.31:0/3518457459 >> 172.21.15.31:0/3518457459 conn(0x559c74984560 msgr2=0x559c74984940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.065+0000 7f982f050740 1 -- 172.21.15.31:0/3518457459 shutdown_connections 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.065+0000 7f982f050740 1 -- 172.21.15.31:0/3518457459 wait complete. 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.065+0000 7f982f050740 1 -- start start 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982f050740 1 Processor -- start 2024-04-25T01:56:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982f050740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988240 0x559c74990210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982f050740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74990750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982f050740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74993230 0x559c74990c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982f050740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559c74996020 con 0x559c74988b60 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982f050740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559c748a8b50 con 0x559c74993230 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982f050740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559c749961a0 con 0x559c74988240 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982f050740 10 client.0 _ll_register_callbacks cb 0x559c748e3ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982a0de640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74990750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982a0de640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74990750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56094/0 (socket says 172.21.15.31:56094) 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982a0de640 1 -- 172.21.15.31:0/1820052007 learned_addr learned my addr 172.21.15.31:0/1820052007 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f98298dd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988240 0x559c74990210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f98298dd640 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74993230 0x559c74990c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982a0de640 1 -- 172.21.15.31:0/1820052007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988240 msgr2=0x559c74990210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982a0de640 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988240 0x559c74990210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982a0de640 1 -- 172.21.15.31:0/1820052007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74993230 msgr2=0x559c74990c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982a0de640 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74993230 0x559c74990c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982a0de640 1 -- 172.21.15.31:0/1820052007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c746cd620 con 0x559c74988b60 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f982a0de640 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74990750 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9820010ca0 tx=0x7f982000b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883d06aa1c0f426e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.066+0000 7f980dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f980dffb640 1 -- 172.21.15.31:0/1820052007 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820029060 con 0x559c74988b60 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f980dffb640 1 -- 172.21.15.31:0/1820052007 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f9820004000 con 0x559c74988b60 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f982f050740 1 -- 172.21.15.31:0/1820052007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98200035f0 con 0x559c74988b60 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f980dffb640 1 -- 172.21.15.31:0/1820052007 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9820004200 con 0x559c74988b60 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f982f050740 10 client.16329 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f982f050740 1 -- 172.21.15.31:0/1820052007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559c749a2270 con 0x559c74988b60 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f980dffb640 1 -- 172.21.15.31:0/1820052007 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98200035f0 con 0x559c74988b60 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f980dffb640 1 -- 172.21.15.31:0/1820052007 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9820004750 con 0x559c74988b60 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f982f050740 10 client.16329 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f982f050740 10 client.16329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f982f050740 20 client.16329 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:51.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f982f050740 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c749a36d0 0x559c749a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f982f050740 4 client.16329 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:51.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f982f050740 1 -- 172.21.15.31:0/1820052007 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c748fa1f0 con 0x559c749a36d0 2024-04-25T01:56:51.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.067+0000 7f98298dd640 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c749a36d0 0x559c749a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.068+0000 7f98298dd640 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c749a36d0 0x559c749a5ac0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fafa5a75a8ac5bb0 server_cookie=7e05c52161bc1a in_seq=0 out_seq=0 2024-04-25T01:56:51.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.068+0000 7f980dffb640 10 client.16329 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:51.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f980dffb640 1 -- 172.21.15.31:0/1820052007 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559c748fa1f0 con 0x559c749a36d0 2024-04-25T01:56:51.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f980dffb640 10 client.16329 handle_command_reply: tid=0 2024-04-25T01:56:51.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 1 client.16329 shutdown 2024-04-25T01:56:51.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 1 -- 172.21.15.31:0/1820052007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 msgr2=0x559c74990750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:51.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74990750 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9820010ca0 tx=0x7f982000b950 comp rx=0 tx=0).stop 2024-04-25T01:56:51.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 1 -- 172.21.15.31:0/1820052007 shutdown_connections 2024-04-25T01:56:51.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c749a36d0 0x559c749a5ac0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74993230 0x559c74990c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f980dffb640 0 client.16329 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:51.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74990750 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 1 --2- 172.21.15.31:0/1820052007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988240 0x559c74990210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 1 -- 172.21.15.31:0/1820052007 >> 172.21.15.31:0/1820052007 conn(0x559c74984560 msgr2=0x559c7498a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:51.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 1 -- 172.21.15.31:0/1820052007 shutdown_connections 2024-04-25T01:56:51.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 1 -- 172.21.15.31:0/1820052007 wait complete. 2024-04-25T01:56:51.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.069+0000 7f982f050740 20 client.16329 trim_cache size 0 max 16384 2024-04-25T01:56:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f982f050740 1 Processor -- start 2024-04-25T01:56:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f982f050740 1 -- start start 2024-04-25T01:56:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f982f050740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74988240 0x559c7499eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f982f050740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74983440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f982f050740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74993230 0x559c749ab290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f982f050740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f98241b3790 con 0x559c74988b60 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f982f050740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f98241b5390 con 0x559c74988240 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f982f050740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f98241b3e60 con 0x559c74993230 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f982a0de640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74983440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f982a0de640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74983440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56098/0 (socket says 172.21.15.31:56098) 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f982a0de640 1 -- 172.21.15.31:0/3984493708 learned_addr learned my addr 172.21.15.31:0/3984493708 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.086+0000 7f98298dd640 1 --2- 172.21.15.31:0/3984493708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74988240 0x559c7499eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.087+0000 7f98298dd640 1 --2- 172.21.15.31:0/3984493708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74993230 0x559c749ab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.087+0000 7f982a0de640 1 -- 172.21.15.31:0/3984493708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74993230 msgr2=0x559c749ab290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.087+0000 7f982a0de640 1 --2- 172.21.15.31:0/3984493708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74993230 0x559c749ab290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.087+0000 7f982a0de640 1 -- 172.21.15.31:0/3984493708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74988240 msgr2=0x559c7499eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.087+0000 7f982a0de640 1 --2- 172.21.15.31:0/3984493708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74988240 0x559c7499eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.087+0000 7f982a0de640 1 -- 172.21.15.31:0/3984493708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c74988040 con 0x559c74988b60 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.087+0000 7f982a0de640 1 --2- 172.21.15.31:0/3984493708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74983440 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9820006010 tx=0x7f982000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307901f16ef9d954 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.087+0000 7f980ffff640 1 -- 172.21.15.31:0/3984493708 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820029070 con 0x559c74988b60 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.087+0000 7f980ffff640 1 -- 172.21.15.31:0/3984493708 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9820004850 con 0x559c74988b60 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.087+0000 7f980ffff640 1 -- 172.21.15.31:0/3984493708 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820004ec0 con 0x559c74988b60 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 -- 172.21.15.31:0/3984493708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 msgr2=0x559c74983440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 --2- 172.21.15.31:0/3984493708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74983440 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9820006010 tx=0x7f982000b390 comp rx=0 tx=0).stop 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 -- 172.21.15.31:0/3984493708 shutdown_connections 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 --2- 172.21.15.31:0/3984493708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74993230 0x559c749ab290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 --2- 172.21.15.31:0/3984493708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74988b60 0x559c74983440 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 --2- 172.21.15.31:0/3984493708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74988240 0x559c7499eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 -- 172.21.15.31:0/3984493708 >> 172.21.15.31:0/3984493708 conn(0x559c74984560 msgr2=0x559c74982d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 -- 172.21.15.31:0/3984493708 shutdown_connections 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 -- 172.21.15.31:0/3984493708 wait complete. 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 -- start start 2024-04-25T01:56:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 Processor -- start 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74988240 0x559c7499b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 0x559c74996bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74993230 0x559c74997170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f98241b4c00 con 0x559c74993230 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f98241b4530 con 0x559c74988240 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f98241b1770 con 0x559c74988b60 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.088+0000 7f982f050740 10 client.0 _ll_register_callbacks cb 0x7f9814009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f98298dd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74993230 0x559c74997170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f982a0de640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 0x559c74996bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f98298dd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74993230 0x559c74997170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56110/0 (socket says 172.21.15.31:56110) 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f98298dd640 1 -- 172.21.15.31:0/4187934093 learned_addr learned my addr 172.21.15.31:0/4187934093 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f98298dd640 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74988240 0x559c7499b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f98298dd640 1 -- 172.21.15.31:0/4187934093 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 msgr2=0x559c74996bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f98298dd640 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 0x559c74996bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f98298dd640 1 -- 172.21.15.31:0/4187934093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74988240 msgr2=0x559c7499b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f98298dd640 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74988240 0x559c7499b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f98298dd640 1 -- 172.21.15.31:0/4187934093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c749979a0 con 0x559c74993230 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f98298dd640 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74993230 0x559c74997170 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f981402a930 tx=0x7f9814005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df32ccb38372bd78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f980dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f980dffb640 1 -- 172.21.15.31:0/4187934093 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9814035040 con 0x559c74993230 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f982f050740 1 -- 172.21.15.31:0/4187934093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9820004850 con 0x559c74993230 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f980dffb640 1 -- 172.21.15.31:0/4187934093 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f981404c540 con 0x559c74993230 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f980dffb640 1 -- 172.21.15.31:0/4187934093 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f981404c740 con 0x559c74993230 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f982f050740 10 client.16335 fetch_fsmap learned FSMap version 30 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.089+0000 7f982f050740 1 -- 172.21.15.31:0/4187934093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559c74a09c30 con 0x559c74993230 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.090+0000 7f980dffb640 1 -- 172.21.15.31:0/4187934093 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9820004850 con 0x559c74993230 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.090+0000 7f980dffb640 1 -- 172.21.15.31:0/4187934093 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f981404dce0 con 0x559c74993230 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.090+0000 7f982f050740 10 client.16335 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.090+0000 7f982f050740 10 client.16335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.090+0000 7f982f050740 20 client.16335 populate_metadata read hostname 'smithi031' 2024-04-25T01:56:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.090+0000 7f982f050740 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c749a36d0 0x559c74a0c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.090+0000 7f982f050740 4 client.16335 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:56:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.090+0000 7f982f050740 1 -- 172.21.15.31:0/4187934093 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559c748f4540 con 0x559c749a36d0 2024-04-25T01:56:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.090+0000 7f982a0de640 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c749a36d0 0x559c74a0c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.090+0000 7f980dffb640 10 client.16335 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.090+0000 7f982a0de640 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c749a36d0 0x559c74a0c460 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67e1766666c7f71 server_cookie=c7414f8d8499c447 in_seq=0 out_seq=0 2024-04-25T01:56:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f980dffb640 1 -- 172.21.15.31:0/4187934093 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559c748f4540 con 0x559c749a36d0 2024-04-25T01:56:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f980dffb640 10 client.16335 handle_command_reply: tid=0 2024-04-25T01:56:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f982f050740 1 client.16335 shutdown 2024-04-25T01:56:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f982f050740 1 -- 172.21.15.31:0/4187934093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74993230 msgr2=0x559c74997170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f982f050740 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74993230 0x559c74997170 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f981402a930 tx=0x7f9814005b50 comp rx=0 tx=0).stop 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f982f050740 1 -- 172.21.15.31:0/4187934093 shutdown_connections 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f982f050740 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c749a36d0 0x559c74a0c460 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f982f050740 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c74993230 0x559c74997170 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f980dffb640 0 client.16335 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f982f050740 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c74988b60 0x559c74996bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f982f050740 1 --2- 172.21.15.31:0/4187934093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c74988240 0x559c7499b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f982f050740 1 -- 172.21.15.31:0/4187934093 >> 172.21.15.31:0/4187934093 conn(0x559c74984560 msgr2=0x559c749835c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f982f050740 1 -- 172.21.15.31:0/4187934093 shutdown_connections 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.109+0000 7f982f050740 1 -- 172.21.15.31:0/4187934093 wait complete. 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982f050740 20 client.16335 trim_cache size 0 max 16384 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:56:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:56:51.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:56:51.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:56:51.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 -- 172.21.15.31:0/2453384704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f98100a2da0 msgr2=0x7f98100a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:51.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f98100a2da0 0x7f98100a5260 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f981c098530 tx=0x7f981c06f140 comp rx=0 tx=0).stop 2024-04-25T01:56:51.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 -- 172.21.15.31:0/2453384704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982410e120 msgr2=0x7f98241032d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:51.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982410e120 0x7f98241032d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f981c00eb70 tx=0x7f981c00c6a0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 -- 172.21.15.31:0/2453384704 shutdown_connections 2024-04-25T01:56:51.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f98100a2da0 0x7f98100a5260 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982410e120 0x7f98241032d0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9824100e90 0x7f9824102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 --2- 172.21.15.31:0/2453384704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9824100570 0x7f9824102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:51.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 -- 172.21.15.31:0/2453384704 >> 172.21.15.31:0/2453384704 conn(0x7f98240777c0 msgr2=0x7f982410f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:51.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 -- 172.21.15.31:0/2453384704 shutdown_connections 2024-04-25T01:56:51.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:56:51.110+0000 7f982c342640 1 -- 172.21.15.31:0/2453384704 wait complete. 2024-04-25T01:56:51.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 29202 3.01 MB/sec warmup 58 sec latency 19.430 ms 2024-04-25T01:56:52.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:51 smithi040 ceph-mon[32418]: pgmap v74: 97 pgs: 97 active+clean; 33 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-25T01:56:52.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:51 smithi134 ceph-mon[33098]: pgmap v74: 97 pgs: 97 active+clean; 33 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-25T01:56:52.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:51 smithi031 ceph-mon[29204]: pgmap v74: 97 pgs: 97 active+clean; 33 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-25T01:56:52.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 29697 2.99 MB/sec warmup 59 sec latency 18.941 ms 2024-04-25T01:56:53.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 30333 2.99 MB/sec warmup 60 sec latency 17.293 ms 2024-04-25T01:56:54.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:53 smithi040 ceph-mon[32418]: pgmap v75: 97 pgs: 97 active+clean; 33 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:56:54.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:53 smithi134 ceph-mon[33098]: pgmap v75: 97 pgs: 97 active+clean; 33 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:56:54.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:53 smithi031 ceph-mon[29204]: pgmap v75: 97 pgs: 97 active+clean; 33 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:56:54.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 30903 2.98 MB/sec warmup 61 sec latency 17.747 ms 2024-04-25T01:56:55.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 31605 3.02 MB/sec warmup 62 sec latency 20.124 ms 2024-04-25T01:56:56.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:55 smithi040 ceph-mon[32418]: pgmap v76: 97 pgs: 97 active+clean; 33 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-25T01:56:56.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:55 smithi134 ceph-mon[33098]: pgmap v76: 97 pgs: 97 active+clean; 33 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-25T01:56:56.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:55 smithi031 ceph-mon[29204]: pgmap v76: 97 pgs: 97 active+clean; 33 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-25T01:56:56.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 32046 3.04 MB/sec warmup 63 sec latency 17.815 ms 2024-04-25T01:56:57.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 32353 3.02 MB/sec warmup 64 sec latency 18.549 ms 2024-04-25T01:56:58.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:57 smithi040 ceph-mon[32418]: pgmap v77: 97 pgs: 97 active+clean; 33 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-25T01:56:58.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:57 smithi134 ceph-mon[33098]: pgmap v77: 97 pgs: 97 active+clean; 33 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-25T01:56:58.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:57 smithi031 ceph-mon[29204]: pgmap v77: 97 pgs: 97 active+clean; 33 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-25T01:56:58.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 32657 2.97 MB/sec warmup 65 sec latency 18.563 ms 2024-04-25T01:56:59.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 33146 2.96 MB/sec warmup 66 sec latency 19.853 ms 2024-04-25T01:57:00.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:56:59 smithi040 ceph-mon[32418]: pgmap v78: 97 pgs: 97 active+clean; 33 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-25T01:57:00.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:56:59 smithi134 ceph-mon[33098]: pgmap v78: 97 pgs: 97 active+clean; 33 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-25T01:57:00.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:56:59 smithi031 ceph-mon[29204]: pgmap v78: 97 pgs: 97 active+clean; 33 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-04-25T01:57:00.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 33740 2.96 MB/sec warmup 67 sec latency 20.799 ms 2024-04-25T01:57:01.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 34332 2.94 MB/sec warmup 68 sec latency 20.761 ms 2024-04-25T01:57:02.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:01 smithi040 ceph-mon[32418]: pgmap v79: 97 pgs: 97 active+clean; 37 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-25T01:57:02.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:01 smithi134 ceph-mon[33098]: pgmap v79: 97 pgs: 97 active+clean; 37 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-25T01:57:02.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:01 smithi031 ceph-mon[29204]: pgmap v79: 97 pgs: 97 active+clean; 37 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-25T01:57:02.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 34951 2.98 MB/sec warmup 69 sec latency 15.537 ms 2024-04-25T01:57:03.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 35473 3.01 MB/sec warmup 70 sec latency 14.401 ms 2024-04-25T01:57:04.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:03 smithi040 ceph-mon[32418]: pgmap v80: 97 pgs: 97 active+clean; 37 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-25T01:57:04.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:03 smithi134 ceph-mon[33098]: pgmap v80: 97 pgs: 97 active+clean; 37 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-25T01:57:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.206+0000 7f5cfa85d640 1 -- 172.21.15.31:0/2558974085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cf4107760 msgr2=0x7f5cf4107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.206+0000 7f5cfa85d640 1 --2- 172.21.15.31:0/2558974085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cf4107760 0x7f5cf4107b40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4009930 tx=0x7f5ce402a420 comp rx=0 tx=0).stop 2024-04-25T01:57:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.206+0000 7f5cfa85d640 1 -- 172.21.15.31:0/2558974085 shutdown_connections 2024-04-25T01:57:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.206+0000 7f5cfa85d640 1 --2- 172.21.15.31:0/2558974085 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cf4102110 0x7f5cf410e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.206+0000 7f5cfa85d640 1 --2- 172.21.15.31:0/2558974085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf4101750 0x7f5cf4101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.206+0000 7f5cfa85d640 1 --2- 172.21.15.31:0/2558974085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cf4107760 0x7f5cf4107b40 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.206+0000 7f5cfa85d640 1 -- 172.21.15.31:0/2558974085 >> 172.21.15.31:0/2558974085 conn(0x7f5cf40fd200 msgr2=0x7f5cf40ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.206+0000 7f5cfa85d640 1 -- 172.21.15.31:0/2558974085 shutdown_connections 2024-04-25T01:57:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.206+0000 7f5cfa85d640 1 -- 172.21.15.31:0/2558974085 wait complete. 2024-04-25T01:57:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cfa85d640 1 Processor -- start 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cfa85d640 1 -- start start 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cfa85d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cf4101750 0x7f5cf4197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cfa85d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cf4102110 0x7f5cf41983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cfa85d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf4107760 0x7f5cf419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cfa85d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf4111a20 con 0x7f5cf4102110 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cfa85d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41118a0 con 0x7f5cf4107760 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cfa85d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf4111ba0 con 0x7f5cf4101750 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cf37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cf4102110 0x7f5cf41983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cf4101750 0x7f5cf4197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cf37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cf4102110 0x7f5cf41983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51250/0 (socket says 172.21.15.31:51250) 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cf8dfa640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf4107760 0x7f5cf419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cf37fe640 1 -- 172.21.15.31:0/1494494199 learned_addr learned my addr 172.21.15.31:0/1494494199 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cf37fe640 1 -- 172.21.15.31:0/1494494199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cf4101750 msgr2=0x7f5cf4197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cf4101750 0x7f5cf4197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cf37fe640 1 -- 172.21.15.31:0/1494494199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf4107760 msgr2=0x7f5cf419c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.207+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf4107760 0x7f5cf419c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.208+0000 7f5cf37fe640 1 -- 172.21.15.31:0/1494494199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf419ce00 con 0x7f5cf4102110 2024-04-25T01:57:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.208+0000 7f5cf3fff640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cf4101750 0x7f5cf4197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.208+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cf4102110 0x7f5cf41983a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c920 tx=0x7f5ce800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.208+0000 7f5cf17fa640 1 -- 172.21.15.31:0/1494494199 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8007c70 con 0x7f5cf4102110 2024-04-25T01:57:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.208+0000 7f5cfa85d640 1 -- 172.21.15.31:0/1494494199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf419d0f0 con 0x7f5cf4102110 2024-04-25T01:57:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.208+0000 7f5cf17fa640 1 -- 172.21.15.31:0/1494494199 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ce8007e10 con 0x7f5cf4102110 2024-04-25T01:57:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.208+0000 7f5cf17fa640 1 -- 172.21.15.31:0/1494494199 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8018820 con 0x7f5cf4102110 2024-04-25T01:57:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.208+0000 7f5cfa85d640 1 -- 172.21.15.31:0/1494494199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf41a4980 con 0x7f5cf4102110 2024-04-25T01:57:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.210+0000 7f5cf17fa640 1 -- 172.21.15.31:0/1494494199 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5ce80189c0 con 0x7f5cf4102110 2024-04-25T01:57:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cfd56d740 1 Processor -- start 2024-04-25T01:57:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cfd56d740 1 -- start start 2024-04-25T01:57:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd768620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 0x55f5dd772cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd773230 0x55f5dd775620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f5dd6cccb0 con 0x55f5dd768240 2024-04-25T01:57:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f5dd6755b0 con 0x55f5dd768b60 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f5dd6c8c70 con 0x55f5dd773230 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf17fa640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cdc0a2da0 0x7f5cdc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 0x55f5dd772cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf8dfa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd768620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 0x55f5dd772cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:34396/0 (socket says 172.21.15.31:34396) 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf3fff640 1 -- 172.21.15.31:0/3728021227 learned_addr learned my addr 172.21.15.31:0/3728021227 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/3728021227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd773230 0x55f5dd775620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf8dfa640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cdc0a2da0 0x7f5cdc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf17fa640 1 -- 172.21.15.31:0/1494494199 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5ce80cf4e0 con 0x7f5cf4102110 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf3fff640 1 -- 172.21.15.31:0/3728021227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd773230 msgr2=0x55f5dd775620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf3fff640 1 --2- 172.21.15.31:0/3728021227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd773230 0x55f5dd775620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf3fff640 1 -- 172.21.15.31:0/3728021227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 msgr2=0x55f5dd768620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf3fff640 1 --2- 172.21.15.31:0/3728021227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd768620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.211+0000 7f5cf3fff640 1 -- 172.21.15.31:0/3728021227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5dd3e1210 con 0x55f5dd768b60 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.212+0000 7f5cf8dfa640 1 --2- 172.21.15.31:0/3728021227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd768620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.212+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/3728021227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd773230 0x55f5dd775620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.212+0000 7f5cf3fff640 1 --2- 172.21.15.31:0/3728021227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 0x55f5dd772cf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4006010 tx=0x7f5ce4031550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e814c2dfb5dfd6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.212+0000 7f5cf8dfa640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cdc0a2da0 0x7f5cdc0a5260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5cec005fd0 tx=0x7f5cec009500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.212+0000 7f5cda7fc640 1 -- 172.21.15.31:0/3728021227 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4035040 con 0x55f5dd768b60 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.212+0000 7f5cda7fc640 1 -- 172.21.15.31:0/3728021227 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ce40318c0 con 0x55f5dd768b60 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.212+0000 7f5cda7fc640 1 -- 172.21.15.31:0/3728021227 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4004a00 con 0x55f5dd768b60 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.213+0000 7f5cfd56d740 1 -- 172.21.15.31:0/3728021227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 msgr2=0x55f5dd772cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.213+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/3728021227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 0x55f5dd772cf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4006010 tx=0x7f5ce4031550 comp rx=0 tx=0).stop 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.213+0000 7f5cfd56d740 1 -- 172.21.15.31:0/3728021227 shutdown_connections 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.213+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/3728021227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd773230 0x55f5dd775620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.213+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/3728021227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 0x55f5dd772cf0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.213+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/3728021227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd768620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.213+0000 7f5cfd56d740 1 -- 172.21.15.31:0/3728021227 >> 172.21.15.31:0/3728021227 conn(0x55f5dd764560 msgr2=0x55f5dd764940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.213+0000 7f5cfd56d740 1 -- 172.21.15.31:0/3728021227 shutdown_connections 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.213+0000 7f5cfd56d740 1 -- 172.21.15.31:0/3728021227 wait complete. 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.213+0000 7f5cfd56d740 1 -- start start 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.213+0000 7f5cfd56d740 1 Processor -- start 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd770110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768b60 0x55f5dd770650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd773230 0x55f5dd770b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f5dd775fc0 con 0x55f5dd768240 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f5dd688b50 con 0x55f5dd773230 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f5dd776140 con 0x55f5dd768b60 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd770110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd770110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51270/0 (socket says 172.21.15.31:51270) 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf3fff640 1 -- 172.21.15.31:0/1859822770 learned_addr learned my addr 172.21.15.31:0/1859822770 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768b60 0x55f5dd770650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cfd56d740 10 client.0 _ll_register_callbacks cb 0x55f5dd6c3ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf8dfa640 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd773230 0x55f5dd770b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf37fe640 1 -- 172.21.15.31:0/1859822770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd773230 msgr2=0x55f5dd770b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd773230 0x55f5dd770b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf37fe640 1 -- 172.21.15.31:0/1859822770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 msgr2=0x55f5dd770110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd770110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf37fe640 1 -- 172.21.15.31:0/1859822770 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5dd4ad620 con 0x55f5dd768b60 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf3fff640 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd770110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768b60 0x55f5dd770650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80ccc20 tx=0x7f5ce806b850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec32dccbcdb3b9bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.214+0000 7f5cd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cd37fe640 1 -- 172.21.15.31:0/1859822770 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce80cd0b0 con 0x55f5dd768b60 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cd37fe640 1 -- 172.21.15.31:0/1859822770 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5ce80a43d0 con 0x55f5dd768b60 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cfd56d740 1 -- 172.21.15.31:0/1859822770 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ce40318c0 con 0x55f5dd768b60 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cd37fe640 1 -- 172.21.15.31:0/1859822770 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5ce80a4660 con 0x55f5dd768b60 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cfd56d740 10 client.25798 fetch_fsmap learned FSMap version 30 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cfd56d740 1 -- 172.21.15.31:0/1859822770 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f5dd782170 con 0x55f5dd768b60 2024-04-25T01:57:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cd37fe640 1 -- 172.21.15.31:0/1859822770 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ce40318c0 con 0x55f5dd768b60 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cd37fe640 1 -- 172.21.15.31:0/1859822770 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5ce80860b0 con 0x55f5dd768b60 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cfd56d740 10 client.25798 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cfd56d740 10 client.25798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cfd56d740 20 client.25798 populate_metadata read hostname 'smithi031' 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f5dd783630 0x55f5dd785ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cfd56d740 4 client.25798 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cfd56d740 1 -- 172.21.15.31:0/1859822770 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5dd6da1f0 con 0x55f5dd783630 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.215+0000 7f5cf3fff640 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f5dd783630 0x55f5dd785ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.216+0000 7f5cd37fe640 10 client.25798 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.216+0000 7f5cf3fff640 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f5dd783630 0x55f5dd785ad0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84d9557c9d93bf4d server_cookie=f01b49475e668d98 in_seq=0 out_seq=0 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.216+0000 7f5cd37fe640 1 -- 172.21.15.31:0/1859822770 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f5dd6da1f0 con 0x55f5dd783630 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.216+0000 7f5cd37fe640 10 client.25798 handle_command_reply: tid=0 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.216+0000 7f5cfd56d740 1 client.25798 shutdown 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cfd56d740 1 -- 172.21.15.31:0/1859822770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768b60 msgr2=0x55f5dd770650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768b60 0x55f5dd770650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80ccc20 tx=0x7f5ce806b850 comp rx=0 tx=0).stop 2024-04-25T01:57:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cfd56d740 1 -- 172.21.15.31:0/1859822770 shutdown_connections 2024-04-25T01:57:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f5dd783630 0x55f5dd785ad0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd773230 0x55f5dd770b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768b60 0x55f5dd770650 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/1859822770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd770110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cd37fe640 0 client.25798 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cfd56d740 1 -- 172.21.15.31:0/1859822770 >> 172.21.15.31:0/1859822770 conn(0x55f5dd764560 msgr2=0x55f5dd76a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cfd56d740 1 -- 172.21.15.31:0/1859822770 shutdown_connections 2024-04-25T01:57:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cfd56d740 1 -- 172.21.15.31:0/1859822770 wait complete. 2024-04-25T01:57:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.217+0000 7f5cfd56d740 20 client.25798 trim_cache size 0 max 16384 2024-04-25T01:57:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cfd56d740 1 Processor -- start 2024-04-25T01:57:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cfd56d740 1 -- start start 2024-04-25T01:57:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768240 0x55f5dd77ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 0x55f5dd78b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd773230 0x55f5dd78b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41af340 con 0x55f5dd773230 2024-04-25T01:57:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41b0f40 con 0x55f5dd768b60 2024-04-25T01:57:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41afa10 con 0x55f5dd768240 2024-04-25T01:57:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cf8dfa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd773230 0x55f5dd78b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cf8dfa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd773230 0x55f5dd78b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51276/0 (socket says 172.21.15.31:51276) 2024-04-25T01:57:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cf8dfa640 1 -- 172.21.15.31:0/2233062236 learned_addr learned my addr 172.21.15.31:0/2233062236 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/2233062236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 0x55f5dd78b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.234+0000 7f5cf3fff640 1 --2- 172.21.15.31:0/2233062236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768240 0x55f5dd77ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.235+0000 7f5cf8dfa640 1 -- 172.21.15.31:0/2233062236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768240 msgr2=0x55f5dd77ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.235+0000 7f5cf8dfa640 1 --2- 172.21.15.31:0/2233062236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768240 0x55f5dd77ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.235+0000 7f5cf8dfa640 1 -- 172.21.15.31:0/2233062236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 msgr2=0x55f5dd78b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.235+0000 7f5cf8dfa640 1 --2- 172.21.15.31:0/2233062236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 0x55f5dd78b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.235+0000 7f5cf8dfa640 1 -- 172.21.15.31:0/2233062236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5dd775990 con 0x55f5dd773230 2024-04-25T01:57:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.235+0000 7f5cf8dfa640 1 --2- 172.21.15.31:0/2233062236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd773230 0x55f5dd78b8d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5cec011aa0 tx=0x7f5cec008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e139eba3be5680ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.235+0000 7f5cd9ffb640 1 -- 172.21.15.31:0/2233062236 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cec004b20 con 0x55f5dd773230 2024-04-25T01:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.235+0000 7f5cd9ffb640 1 -- 172.21.15.31:0/2233062236 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cec021de0 con 0x55f5dd773230 2024-04-25T01:57:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.235+0000 7f5cd9ffb640 1 -- 172.21.15.31:0/2233062236 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cec005640 con 0x55f5dd773230 2024-04-25T01:57:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.236+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2233062236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd773230 msgr2=0x55f5dd78b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.236+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/2233062236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd773230 0x55f5dd78b8d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5cec011aa0 tx=0x7f5cec008000 comp rx=0 tx=0).stop 2024-04-25T01:57:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.236+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2233062236 shutdown_connections 2024-04-25T01:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.236+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/2233062236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd773230 0x55f5dd78b8d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.236+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/2233062236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd768b60 0x55f5dd78b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.236+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/2233062236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768240 0x55f5dd77ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.236+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2233062236 >> 172.21.15.31:0/2233062236 conn(0x55f5dd764560 msgr2=0x55f5dd76a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.236+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2233062236 shutdown_connections 2024-04-25T01:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.236+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2233062236 wait complete. 2024-04-25T01:57:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.236+0000 7f5cfd56d740 1 -- start start 2024-04-25T01:57:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.236+0000 7f5cfd56d740 1 Processor -- start 2024-04-25T01:57:04.247 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:03 smithi031 ceph-mon[29204]: pgmap v80: 97 pgs: 97 active+clean; 37 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-25T01:57:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd77b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768b60 0x55f5dd76c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cfd56d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd773230 0x55f5dd76cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41b07b0 con 0x55f5dd768240 2024-04-25T01:57:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41b00e0 con 0x55f5dd773230 2024-04-25T01:57:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cfd56d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41ad320 con 0x55f5dd768b60 2024-04-25T01:57:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cfd56d740 10 client.0 _ll_register_callbacks cb 0x7f5ce800a5f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd77b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cf3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd77b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51284/0 (socket says 172.21.15.31:51284) 2024-04-25T01:57:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cf3fff640 1 -- 172.21.15.31:0/2201394232 learned_addr learned my addr 172.21.15.31:0/2201394232 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cf8dfa640 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd773230 0x55f5dd76cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768b60 0x55f5dd76c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cf3fff640 1 -- 172.21.15.31:0/2201394232 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768b60 msgr2=0x55f5dd76c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cf3fff640 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768b60 0x55f5dd76c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cf3fff640 1 -- 172.21.15.31:0/2201394232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd773230 msgr2=0x55f5dd76cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cf3fff640 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd773230 0x55f5dd76cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cf3fff640 1 -- 172.21.15.31:0/2201394232 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5dd76d440 con 0x55f5dd768240 2024-04-25T01:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cf3fff640 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd77b580 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4009930 tx=0x7f5ce402ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c16a7097631fb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:57:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cd37fe640 1 -- 172.21.15.31:0/2201394232 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4035040 con 0x55f5dd768240 2024-04-25T01:57:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cd37fe640 1 -- 172.21.15.31:0/2201394232 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f5ce40042c0 con 0x55f5dd768240 2024-04-25T01:57:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.237+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2201394232 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cec021de0 con 0x55f5dd768240 2024-04-25T01:57:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cd37fe640 1 -- 172.21.15.31:0/2201394232 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5ce405d020 con 0x55f5dd768240 2024-04-25T01:57:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cfd56d740 10 client.16353 fetch_fsmap learned FSMap version 30 2024-04-25T01:57:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2201394232 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f5dd7e99b0 con 0x55f5dd768240 2024-04-25T01:57:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cd37fe640 1 -- 172.21.15.31:0/2201394232 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cec021de0 con 0x55f5dd768240 2024-04-25T01:57:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cd37fe640 1 -- 172.21.15.31:0/2201394232 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5ce40035f0 con 0x55f5dd768240 2024-04-25T01:57:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cfd56d740 10 client.16353 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:57:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cfd56d740 10 client.16353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cfd56d740 20 client.16353 populate_metadata read hostname 'smithi031' 2024-04-25T01:57:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f5dd7e9db0 0x55f5dd7ec1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cfd56d740 4 client.16353 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:57:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2201394232 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f5dd6d4540 con 0x55f5dd7e9db0 2024-04-25T01:57:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.238+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f5dd7e9db0 0x55f5dd7ec1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.239+0000 7f5cf37fe640 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f5dd7e9db0 0x55f5dd7ec1a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18b015d53e12eb31 server_cookie=d7d8e077df91b895 in_seq=0 out_seq=0 2024-04-25T01:57:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.239+0000 7f5cd37fe640 10 client.16353 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.254+0000 7f5cd37fe640 1 -- 172.21.15.31:0/2201394232 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f5dd6d4540 con 0x55f5dd7e9db0 2024-04-25T01:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.254+0000 7f5cd37fe640 10 client.16353 handle_command_reply: tid=0 2024-04-25T01:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.254+0000 7f5cfd56d740 1 client.16353 shutdown 2024-04-25T01:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2201394232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 msgr2=0x55f5dd77b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd77b580 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4009930 tx=0x7f5ce402ab70 comp rx=0 tx=0).stop 2024-04-25T01:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2201394232 shutdown_connections 2024-04-25T01:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f5dd7e9db0 0x55f5dd7ec1a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f5dd773230 0x55f5dd76cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cd37fe640 0 client.16353 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5dd768b60 0x55f5dd76c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfd56d740 1 --2- 172.21.15.31:0/2201394232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f5dd768240 0x55f5dd77b580 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2201394232 >> 172.21.15.31:0/2201394232 conn(0x55f5dd764560 msgr2=0x55f5dd76e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2201394232 shutdown_connections 2024-04-25T01:57:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfd56d740 1 -- 172.21.15.31:0/2201394232 wait complete. 2024-04-25T01:57:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfd56d740 20 client.16353 trim_cache size 0 max 16384 2024-04-25T01:57:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:57:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:57:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:57:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:57:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfa85d640 1 -- 172.21.15.31:0/1494494199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cdc0a2da0 msgr2=0x7f5cdc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfa85d640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cdc0a2da0 0x7f5cdc0a5260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5cec005fd0 tx=0x7f5cec009500 comp rx=0 tx=0).stop 2024-04-25T01:57:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfa85d640 1 -- 172.21.15.31:0/1494494199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cf4102110 msgr2=0x7f5cf41983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.255+0000 7f5cfa85d640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cf4102110 0x7f5cf41983a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c920 tx=0x7f5ce800cde0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.256+0000 7f5cfa85d640 1 -- 172.21.15.31:0/1494494199 shutdown_connections 2024-04-25T01:57:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.256+0000 7f5cfa85d640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cdc0a2da0 0x7f5cdc0a5260 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.256+0000 7f5cfa85d640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf4107760 0x7f5cf419c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.256+0000 7f5cfa85d640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cf4102110 0x7f5cf41983a0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.256+0000 7f5cfa85d640 1 --2- 172.21.15.31:0/1494494199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cf4101750 0x7f5cf4197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.256+0000 7f5cfa85d640 1 -- 172.21.15.31:0/1494494199 >> 172.21.15.31:0/1494494199 conn(0x7f5cf40fd200 msgr2=0x7f5cf40fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.256+0000 7f5cfa85d640 1 -- 172.21.15.31:0/1494494199 shutdown_connections 2024-04-25T01:57:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:04.256+0000 7f5cfa85d640 1 -- 172.21.15.31:0/1494494199 wait complete. 2024-04-25T01:57:04.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 35835 2.98 MB/sec warmup 71 sec latency 18.486 ms 2024-04-25T01:57:05.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 36108 2.95 MB/sec warmup 72 sec latency 13.641 ms 2024-04-25T01:57:06.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:05 smithi040 ceph-mon[32418]: pgmap v81: 97 pgs: 97 active+clean; 39 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.7 MiB/s wr, 206 op/s 2024-04-25T01:57:06.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:05 smithi134 ceph-mon[33098]: pgmap v81: 97 pgs: 97 active+clean; 39 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.7 MiB/s wr, 206 op/s 2024-04-25T01:57:06.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:05 smithi031 ceph-mon[29204]: pgmap v81: 97 pgs: 97 active+clean; 39 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.7 MiB/s wr, 206 op/s 2024-04-25T01:57:06.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 36616 2.93 MB/sec warmup 73 sec latency 13.099 ms 2024-04-25T01:57:07.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 37196 2.94 MB/sec warmup 74 sec latency 12.895 ms 2024-04-25T01:57:08.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:07 smithi040 ceph-mon[32418]: pgmap v82: 97 pgs: 97 active+clean; 39 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-04-25T01:57:08.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:07 smithi134 ceph-mon[33098]: pgmap v82: 97 pgs: 97 active+clean; 39 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-04-25T01:57:08.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:07 smithi031 ceph-mon[29204]: pgmap v82: 97 pgs: 97 active+clean; 39 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-04-25T01:57:08.301 INFO:tasks.workunit.client.0.smithi031.stdout: 1 37833 2.92 MB/sec warmup 75 sec latency 18.390 ms 2024-04-25T01:57:09.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 38407 2.95 MB/sec warmup 76 sec latency 19.503 ms 2024-04-25T01:57:10.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:09 smithi040 ceph-mon[32418]: pgmap v83: 97 pgs: 97 active+clean; 39 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-04-25T01:57:10.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:09 smithi134 ceph-mon[33098]: pgmap v83: 97 pgs: 97 active+clean; 39 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-04-25T01:57:10.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:09 smithi031 ceph-mon[29204]: pgmap v83: 97 pgs: 97 active+clean; 39 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-04-25T01:57:10.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 38863 2.94 MB/sec warmup 77 sec latency 18.956 ms 2024-04-25T01:57:11.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 39335 2.96 MB/sec warmup 78 sec latency 18.122 ms 2024-04-25T01:57:12.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:11 smithi040 ceph-mon[32418]: pgmap v84: 97 pgs: 97 active+clean; 40 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 1.8 MiB/s wr, 201 op/s 2024-04-25T01:57:12.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:11 smithi134 ceph-mon[33098]: pgmap v84: 97 pgs: 97 active+clean; 40 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 1.8 MiB/s wr, 201 op/s 2024-04-25T01:57:12.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:11 smithi031 ceph-mon[29204]: pgmap v84: 97 pgs: 97 active+clean; 40 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 1.8 MiB/s wr, 201 op/s 2024-04-25T01:57:12.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 39603 2.93 MB/sec warmup 79 sec latency 18.865 ms 2024-04-25T01:57:13.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 40059 2.90 MB/sec warmup 80 sec latency 11.918 ms 2024-04-25T01:57:14.161 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:13 smithi040 ceph-mon[32418]: pgmap v85: 97 pgs: 97 active+clean; 40 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 1.2 MiB/s wr, 141 op/s 2024-04-25T01:57:14.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:13 smithi134 ceph-mon[33098]: pgmap v85: 97 pgs: 97 active+clean; 40 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 1.2 MiB/s wr, 141 op/s 2024-04-25T01:57:14.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:13 smithi031 ceph-mon[29204]: pgmap v85: 97 pgs: 97 active+clean; 40 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 1.2 MiB/s wr, 141 op/s 2024-04-25T01:57:14.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 40616 2.92 MB/sec warmup 81 sec latency 17.746 ms 2024-04-25T01:57:15.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 41239 2.89 MB/sec warmup 82 sec latency 19.005 ms 2024-04-25T01:57:16.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:15 smithi134 ceph-mon[33098]: pgmap v86: 97 pgs: 97 active+clean; 40 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.7 MiB/s wr, 207 op/s 2024-04-25T01:57:16.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:15 smithi031 ceph-mon[29204]: pgmap v86: 97 pgs: 97 active+clean; 40 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.7 MiB/s wr, 207 op/s 2024-04-25T01:57:16.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 41801 2.92 MB/sec warmup 83 sec latency 17.764 ms 2024-04-25T01:57:16.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:15 smithi040 ceph-mon[32418]: pgmap v86: 97 pgs: 97 active+clean; 40 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.7 MiB/s wr, 207 op/s 2024-04-25T01:57:17.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 42379 2.93 MB/sec warmup 84 sec latency 12.707 ms 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.355+0000 7f4cd7df9640 1 -- 172.21.15.31:0/3841282424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd0106630 msgr2=0x7f4cd0106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.355+0000 7f4cd7df9640 1 --2- 172.21.15.31:0/3841282424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd0106630 0x7f4cd0106a10 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009cd0 tx=0x7f4ccc02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.355+0000 7f4cd7df9640 1 -- 172.21.15.31:0/3841282424 shutdown_connections 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.355+0000 7f4cd7df9640 1 --2- 172.21.15.31:0/3841282424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd0101030 0x7f4cd010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.355+0000 7f4cd7df9640 1 --2- 172.21.15.31:0/3841282424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd0100670 0x7f4cd0100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.355+0000 7f4cd7df9640 1 --2- 172.21.15.31:0/3841282424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd0106630 0x7f4cd0106a10 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.355+0000 7f4cd7df9640 1 -- 172.21.15.31:0/3841282424 >> 172.21.15.31:0/3841282424 conn(0x7f4cd00777c0 msgr2=0x7f4cd0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.355+0000 7f4cd7df9640 1 -- 172.21.15.31:0/3841282424 shutdown_connections 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.355+0000 7f4cd7df9640 1 -- 172.21.15.31:0/3841282424 wait complete. 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd7df9640 1 Processor -- start 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd7df9640 1 -- start start 2024-04-25T01:57:17.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd7df9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd0100670 0x7f4cd0197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd7df9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd0101030 0x7f4cd01983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd7df9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd0106630 0x7f4cd019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd7df9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd01108b0 con 0x7f4cd0101030 2024-04-25T01:57:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd7df9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd0110730 con 0x7f4cd0106630 2024-04-25T01:57:17.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd7df9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd0110a30 con 0x7f4cd0100670 2024-04-25T01:57:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd5394640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd0101030 0x7f4cd01983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd5b95640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd0100670 0x7f4cd0197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd5394640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd0101030 0x7f4cd01983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54132/0 (socket says 172.21.15.31:54132) 2024-04-25T01:57:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd5b95640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd0100670 0x7f4cd0197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:45534/0 (socket says 172.21.15.31:45534) 2024-04-25T01:57:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd5394640 1 -- 172.21.15.31:0/2924491068 learned_addr learned my addr 172.21.15.31:0/2924491068 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd6396640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd0106630 0x7f4cd019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd5394640 1 -- 172.21.15.31:0/2924491068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd0100670 msgr2=0x7f4cd0197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd5394640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd0100670 0x7f4cd0197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd5394640 1 -- 172.21.15.31:0/2924491068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd0106630 msgr2=0x7f4cd019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd5394640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd0106630 0x7f4cd019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd5394640 1 -- 172.21.15.31:0/2924491068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd019ce90 con 0x7f4cd0101030 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.356+0000 7f4cd6396640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd0106630 0x7f4cd019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.357+0000 7f4cd5394640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd0101030 0x7f4cd01983e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000e9f0 tx=0x7f4cc000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.357+0000 7f4cc6ffd640 1 -- 172.21.15.31:0/2924491068 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc000cd80 con 0x7f4cd0101030 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.357+0000 7f4cc6ffd640 1 -- 172.21.15.31:0/2924491068 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cc00108e0 con 0x7f4cd0101030 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.357+0000 7f4cd7df9640 1 -- 172.21.15.31:0/2924491068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd019d180 con 0x7f4cd0101030 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.357+0000 7f4cc6ffd640 1 -- 172.21.15.31:0/2924491068 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0018920 con 0x7f4cd0101030 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.357+0000 7f4cd7df9640 1 -- 172.21.15.31:0/2924491068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd01a4a10 con 0x7f4cd0101030 2024-04-25T01:57:17.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.359+0000 7f4cc6ffd640 1 -- 172.21.15.31:0/2924491068 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4cc0010a90 con 0x7f4cd0101030 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.360+0000 7f4cdabe4740 1 Processor -- start 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.360+0000 7f4cc6ffd640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cbc0a2da0 0x7f4cbc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.360+0000 7f4cdabe4740 1 -- start start 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.360+0000 7f4cd6396640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cbc0a2da0 0x7f4cbc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.360+0000 7f4cc6ffd640 1 -- 172.21.15.31:0/2924491068 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4cc00cf6d0 con 0x7f4cd0101030 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.360+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1a240 0x55c084d1a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.360+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d24cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.360+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d25230 0x55c084d27620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.360+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c084c7ecb0 con 0x55c084d1ab60 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.360+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c084c275b0 con 0x55c084d25230 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.360+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c084c7ac70 con 0x55c084d1a240 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd5394640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d24cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd5394640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d24cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54136/0 (socket says 172.21.15.31:54136) 2024-04-25T01:57:17.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd6396640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d25230 0x55c084d27620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd5394640 1 -- 172.21.15.31:0/263868677 learned_addr learned my addr 172.21.15.31:0/263868677 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd5b95640 1 --2- 172.21.15.31:0/263868677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1a240 0x55c084d1a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd5394640 1 -- 172.21.15.31:0/263868677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1a240 msgr2=0x55c084d1a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd5394640 1 --2- 172.21.15.31:0/263868677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1a240 0x55c084d1a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd5394640 1 -- 172.21.15.31:0/263868677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d25230 msgr2=0x55c084d27620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd5394640 1 --2- 172.21.15.31:0/263868677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d25230 0x55c084d27620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd5394640 1 -- 172.21.15.31:0/263868677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c084993210 con 0x55c084d1ab60 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd6396640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cbc0a2da0 0x7f4cbc0a5260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80117b0 tx=0x7f4cc8011bb0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cd5394640 1 --2- 172.21.15.31:0/263868677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d24cf0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00986c0 tx=0x7f4cc0010f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa91c8a3e809051 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cbbfff640 1 -- 172.21.15.31:0/263868677 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc00a48b0 con 0x55c084d1ab60 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cbbfff640 1 -- 172.21.15.31:0/263868677 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cc00cd0f0 con 0x55c084d1ab60 2024-04-25T01:57:17.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.361+0000 7f4cbbfff640 1 -- 172.21.15.31:0/263868677 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0097a40 con 0x55c084d1ab60 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.362+0000 7f4cdabe4740 1 -- 172.21.15.31:0/263868677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 msgr2=0x55c084d24cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.362+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/263868677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d24cf0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00986c0 tx=0x7f4cc0010f20 comp rx=0 tx=0).stop 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.362+0000 7f4cdabe4740 1 -- 172.21.15.31:0/263868677 shutdown_connections 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.362+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/263868677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d25230 0x55c084d27620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.362+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/263868677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d24cf0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.362+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/263868677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1a240 0x55c084d1a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.362+0000 7f4cdabe4740 1 -- 172.21.15.31:0/263868677 >> 172.21.15.31:0/263868677 conn(0x55c084d16560 msgr2=0x55c084d16940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.362+0000 7f4cdabe4740 1 -- 172.21.15.31:0/263868677 shutdown_connections 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.362+0000 7f4cdabe4740 1 -- 172.21.15.31:0/263868677 wait complete. 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cdabe4740 1 -- start start 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cdabe4740 1 Processor -- start 2024-04-25T01:57:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1a240 0x55c084d22160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1ab60 0x55c084d226a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d25230 0x55c084d22be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c084d27fc0 con 0x55c084d1a240 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c084c3ab50 con 0x55c084d1ab60 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c084d28140 con 0x55c084d25230 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cdabe4740 10 client.0 _ll_register_callbacks cb 0x55c084c75ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cd5b95640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1a240 0x55c084d22160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cd5b95640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1a240 0x55c084d22160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54142/0 (socket says 172.21.15.31:54142) 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cd5b95640 1 -- 172.21.15.31:0/2508540561 learned_addr learned my addr 172.21.15.31:0/2508540561 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cd5394640 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1ab60 0x55c084d226a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.363+0000 7f4cd6396640 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d25230 0x55c084d22be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cd5b95640 1 -- 172.21.15.31:0/2508540561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d25230 msgr2=0x55c084d22be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cd5b95640 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d25230 0x55c084d22be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cd5b95640 1 -- 172.21.15.31:0/2508540561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1ab60 msgr2=0x55c084d226a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cd5b95640 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1ab60 0x55c084d226a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cd5b95640 1 -- 172.21.15.31:0/2508540561 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c084a5f620 con 0x55c084d1a240 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cd5394640 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1ab60 0x55c084d226a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cd6396640 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d25230 0x55c084d22be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cd5b95640 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1a240 0x55c084d22160 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009cd0 tx=0x7f4ccc031a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201dca1407054a06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cb97fa640 1 -- 172.21.15.31:0/2508540561 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc003b70 con 0x55c084d1a240 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cdabe4740 1 -- 172.21.15.31:0/2508540561 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cc00cd0f0 con 0x55c084d1a240 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cb97fa640 1 -- 172.21.15.31:0/2508540561 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4ccc003d10 con 0x55c084d1a240 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cb97fa640 1 -- 172.21.15.31:0/2508540561 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4ccc004040 con 0x55c084d1a240 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cdabe4740 10 client.16365 fetch_fsmap learned FSMap version 30 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cdabe4740 1 -- 172.21.15.31:0/2508540561 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c084d38570 con 0x55c084d1a240 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.364+0000 7f4cb97fa640 1 -- 172.21.15.31:0/2508540561 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cc00cd0f0 con 0x55c084d1a240 2024-04-25T01:57:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.365+0000 7f4cb97fa640 1 -- 172.21.15.31:0/2508540561 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f4ccc045070 con 0x55c084d1a240 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.365+0000 7f4cdabe4740 10 client.16365 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.365+0000 7f4cdabe4740 10 client.16365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.365+0000 7f4cdabe4740 20 client.16365 populate_metadata read hostname 'smithi031' 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.365+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c084d39aa0 0x55c084d3bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.365+0000 7f4cdabe4740 4 client.16365 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.365+0000 7f4cdabe4740 1 -- 172.21.15.31:0/2508540561 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c084c8c1f0 con 0x55c084d39aa0 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.365+0000 7f4cd5394640 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c084d39aa0 0x55c084d3bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.365+0000 7f4cb97fa640 10 client.16365 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.365+0000 7f4cd5394640 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c084d39aa0 0x55c084d3bf00 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9856ba186a4ebc63 server_cookie=ea1da9d6da7fdf6e in_seq=0 out_seq=0 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cb97fa640 1 -- 172.21.15.31:0/2508540561 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c084c8c1f0 con 0x55c084d39aa0 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cb97fa640 10 client.16365 handle_command_reply: tid=0 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 1 client.16365 shutdown 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 1 -- 172.21.15.31:0/2508540561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1a240 msgr2=0x55c084d22160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1a240 0x55c084d22160 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009cd0 tx=0x7f4ccc031a80 comp rx=0 tx=0).stop 2024-04-25T01:57:17.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 1 -- 172.21.15.31:0/2508540561 shutdown_connections 2024-04-25T01:57:17.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c084d39aa0 0x55c084d3bf00 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d25230 0x55c084d22be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cb97fa640 0 client.16365 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:17.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1ab60 0x55c084d226a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/2508540561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1a240 0x55c084d22160 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 1 -- 172.21.15.31:0/2508540561 >> 172.21.15.31:0/2508540561 conn(0x55c084d16560 msgr2=0x55c084d1c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:17.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 1 -- 172.21.15.31:0/2508540561 shutdown_connections 2024-04-25T01:57:17.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 1 -- 172.21.15.31:0/2508540561 wait complete. 2024-04-25T01:57:17.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.366+0000 7f4cdabe4740 20 client.16365 trim_cache size 0 max 16384 2024-04-25T01:57:17.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cdabe4740 1 Processor -- start 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cdabe4740 1 -- start start 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1a240 0x55c084d24890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d1c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d25230 0x55c084d21010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd01ab040 con 0x55c084d1ab60 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd01acc40 con 0x55c084d25230 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd01ab710 con 0x55c084d1a240 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cd5394640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d1c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cd5394640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d1c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54156/0 (socket says 172.21.15.31:54156) 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cd5394640 1 -- 172.21.15.31:0/1072957080 learned_addr learned my addr 172.21.15.31:0/1072957080 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cd6396640 1 --2- 172.21.15.31:0/1072957080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d25230 0x55c084d21010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cd5b95640 1 --2- 172.21.15.31:0/1072957080 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1a240 0x55c084d24890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cd5394640 1 -- 172.21.15.31:0/1072957080 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1a240 msgr2=0x55c084d24890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cd5394640 1 --2- 172.21.15.31:0/1072957080 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1a240 0x55c084d24890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cd5394640 1 -- 172.21.15.31:0/1072957080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d25230 msgr2=0x55c084d21010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cd5394640 1 --2- 172.21.15.31:0/1072957080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d25230 0x55c084d21010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.385+0000 7f4cd5394640 1 -- 172.21.15.31:0/1072957080 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c084d1a990 con 0x55c084d1ab60 2024-04-25T01:57:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.386+0000 7f4cd5394640 1 --2- 172.21.15.31:0/1072957080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d1c9a0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0098150 tx=0x7f4cc00980b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39523fae285d720a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.386+0000 7f4cbb7fe640 1 -- 172.21.15.31:0/1072957080 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc00b8070 con 0x55c084d1ab60 2024-04-25T01:57:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.386+0000 7f4cbb7fe640 1 -- 172.21.15.31:0/1072957080 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cc00cd0f0 con 0x55c084d1ab60 2024-04-25T01:57:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.386+0000 7f4cbb7fe640 1 -- 172.21.15.31:0/1072957080 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0097ca0 con 0x55c084d1ab60 2024-04-25T01:57:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.386+0000 7f4cdabe4740 1 -- 172.21.15.31:0/1072957080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 msgr2=0x55c084d1c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/1072957080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d1c9a0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0098150 tx=0x7f4cc00980b0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 -- 172.21.15.31:0/1072957080 shutdown_connections 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/1072957080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d25230 0x55c084d21010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/1072957080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d1ab60 0x55c084d1c9a0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/1072957080 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1a240 0x55c084d24890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 -- 172.21.15.31:0/1072957080 >> 172.21.15.31:0/1072957080 conn(0x55c084d16560 msgr2=0x55c084d20900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 -- 172.21.15.31:0/1072957080 shutdown_connections 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 -- 172.21.15.31:0/1072957080 wait complete. 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 -- start start 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 Processor -- start 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1a240 0x55c084d31ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1ab60 0x55c084d2cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d25230 0x55c084d2d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd01ac4b0 con 0x55c084d25230 2024-04-25T01:57:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd01abde0 con 0x55c084d1a240 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd01a9040 con 0x55c084d1ab60 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.387+0000 7f4cdabe4740 10 client.0 _ll_register_callbacks cb 0x7f4ccc009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5b95640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1a240 0x55c084d31ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd6396640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d25230 0x55c084d2d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5394640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1ab60 0x55c084d2cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5b95640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1a240 0x55c084d31ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58136/0 (socket says 172.21.15.31:58136) 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5b95640 1 -- 172.21.15.31:0/3915256754 learned_addr learned my addr 172.21.15.31:0/3915256754 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5394640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1ab60 0x55c084d2cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:45570/0 (socket says 172.21.15.31:45570) 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5b95640 1 -- 172.21.15.31:0/3915256754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1ab60 msgr2=0x55c084d2cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5b95640 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1ab60 0x55c084d2cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5b95640 1 -- 172.21.15.31:0/3915256754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d25230 msgr2=0x55c084d2d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5b95640 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d25230 0x55c084d2d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5b95640 1 -- 172.21.15.31:0/3915256754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c084d2dc30 con 0x55c084d1a240 2024-04-25T01:57:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5394640 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1ab60 0x55c084d2cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cd5b95640 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1a240 0x55c084d31ba0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc030960 tx=0x7f4ccc0095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ed10320dd052bcd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cb97fa640 1 -- 172.21.15.31:0/3915256754 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc05d060 con 0x55c084d1a240 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cdabe4740 1 -- 172.21.15.31:0/3915256754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cc00cd0f0 con 0x55c084d1a240 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.388+0000 7f4cb97fa640 1 -- 172.21.15.31:0/3915256754 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f4ccc02e040 con 0x55c084d1a240 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cb97fa640 1 -- 172.21.15.31:0/3915256754 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4ccc0306a0 con 0x55c084d1a240 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cdabe4740 10 client.25970 fetch_fsmap learned FSMap version 30 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cdabe4740 1 -- 172.21.15.31:0/3915256754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c084d9fff0 con 0x55c084d1a240 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cb97fa640 1 -- 172.21.15.31:0/3915256754 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cc00cd0f0 con 0x55c084d1a240 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cb97fa640 1 -- 172.21.15.31:0/3915256754 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f4ccc045040 con 0x55c084d1a240 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cdabe4740 10 client.25970 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cdabe4740 10 client.25970 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cdabe4740 20 client.25970 populate_metadata read hostname 'smithi031' 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c084da0800 0x55c084da2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cdabe4740 4 client.25970 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:57:17.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cdabe4740 1 -- 172.21.15.31:0/3915256754 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c084c86540 con 0x55c084da0800 2024-04-25T01:57:17.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.389+0000 7f4cd5394640 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c084da0800 0x55c084da2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:17.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.390+0000 7f4cd5394640 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c084da0800 0x55c084da2c20 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f2743dc7914cb07 server_cookie=2ecad3db510b433f in_seq=0 out_seq=0 2024-04-25T01:57:17.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.390+0000 7f4cb97fa640 10 client.25970 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.414+0000 7f4cb97fa640 1 -- 172.21.15.31:0/3915256754 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c084c86540 con 0x55c084da0800 2024-04-25T01:57:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.414+0000 7f4cb97fa640 10 client.25970 handle_command_reply: tid=0 2024-04-25T01:57:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.414+0000 7f4cdabe4740 1 client.25970 shutdown 2024-04-25T01:57:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cdabe4740 1 -- 172.21.15.31:0/3915256754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1a240 msgr2=0x55c084d31ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1a240 0x55c084d31ba0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc030960 tx=0x7f4ccc0095b0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cdabe4740 1 -- 172.21.15.31:0/3915256754 shutdown_connections 2024-04-25T01:57:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c084da0800 0x55c084da2c20 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cb97fa640 0 client.25970 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c084d25230 0x55c084d2d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c084d1ab60 0x55c084d2cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cdabe4740 1 --2- 172.21.15.31:0/3915256754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c084d1a240 0x55c084d31ba0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cdabe4740 1 -- 172.21.15.31:0/3915256754 >> 172.21.15.31:0/3915256754 conn(0x55c084d16560 msgr2=0x55c084d21190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cdabe4740 1 -- 172.21.15.31:0/3915256754 shutdown_connections 2024-04-25T01:57:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cdabe4740 1 -- 172.21.15.31:0/3915256754 wait complete. 2024-04-25T01:57:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.415+0000 7f4cdabe4740 20 client.25970 trim_cache size 0 max 16384 2024-04-25T01:57:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:57:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:57:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:57:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:57:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 -- 172.21.15.31:0/2924491068 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cbc0a2da0 msgr2=0x7f4cbc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cbc0a2da0 0x7f4cbc0a5260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80117b0 tx=0x7f4cc8011bb0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 -- 172.21.15.31:0/2924491068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd0101030 msgr2=0x7f4cd01983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd0101030 0x7f4cd01983e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000e9f0 tx=0x7f4cc000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 -- 172.21.15.31:0/2924491068 shutdown_connections 2024-04-25T01:57:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cbc0a2da0 0x7f4cbc0a5260 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd0106630 0x7f4cd019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd0101030 0x7f4cd01983e0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 --2- 172.21.15.31:0/2924491068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd0100670 0x7f4cd0197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 -- 172.21.15.31:0/2924491068 >> 172.21.15.31:0/2924491068 conn(0x7f4cd00777c0 msgr2=0x7f4cd0069190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 -- 172.21.15.31:0/2924491068 shutdown_connections 2024-04-25T01:57:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:17.416+0000 7f4cd7df9640 1 -- 172.21.15.31:0/2924491068 wait complete. 2024-04-25T01:57:18.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:17 smithi134 ceph-mon[33098]: pgmap v87: 97 pgs: 97 active+clean; 40 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-25T01:57:18.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:17 smithi031 ceph-mon[29204]: pgmap v87: 97 pgs: 97 active+clean; 40 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-25T01:57:18.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 42850 2.94 MB/sec warmup 85 sec latency 23.099 ms 2024-04-25T01:57:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:17 smithi040 ceph-mon[32418]: pgmap v87: 97 pgs: 97 active+clean; 40 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-25T01:57:19.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 43104 2.91 MB/sec warmup 86 sec latency 17.551 ms 2024-04-25T01:57:20.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:19 smithi134 ceph-mon[33098]: pgmap v88: 97 pgs: 97 active+clean; 40 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-25T01:57:20.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:19 smithi031 ceph-mon[29204]: pgmap v88: 97 pgs: 97 active+clean; 40 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-25T01:57:20.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 43518 2.89 MB/sec warmup 87 sec latency 18.605 ms 2024-04-25T01:57:20.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:19 smithi040 ceph-mon[32418]: pgmap v88: 97 pgs: 97 active+clean; 40 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-25T01:57:21.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 44020 2.90 MB/sec warmup 88 sec latency 86.602 ms 2024-04-25T01:57:22.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:21 smithi134 ceph-mon[33098]: pgmap v89: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.7 MiB/s wr, 190 op/s 2024-04-25T01:57:22.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:21 smithi031 ceph-mon[29204]: pgmap v89: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.7 MiB/s wr, 190 op/s 2024-04-25T01:57:22.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 44635 2.88 MB/sec warmup 89 sec latency 18.560 ms 2024-04-25T01:57:22.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:21 smithi040 ceph-mon[32418]: pgmap v89: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.7 MiB/s wr, 190 op/s 2024-04-25T01:57:23.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 45156 2.87 MB/sec warmup 90 sec latency 18.880 ms 2024-04-25T01:57:24.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:23 smithi134 ceph-mon[33098]: pgmap v90: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-04-25T01:57:24.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:23 smithi031 ceph-mon[29204]: pgmap v90: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-04-25T01:57:24.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 45890 2.91 MB/sec warmup 91 sec latency 18.287 ms 2024-04-25T01:57:24.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:23 smithi040 ceph-mon[32418]: pgmap v90: 97 pgs: 97 active+clean; 40 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-04-25T01:57:25.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 46322 2.92 MB/sec warmup 92 sec latency 11.280 ms 2024-04-25T01:57:26.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:25 smithi134 ceph-mon[33098]: pgmap v91: 97 pgs: 97 active+clean; 42 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-25T01:57:26.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:25 smithi031 ceph-mon[29204]: pgmap v91: 97 pgs: 97 active+clean; 42 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-25T01:57:26.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 46590 2.90 MB/sec warmup 93 sec latency 16.224 ms 2024-04-25T01:57:26.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:25 smithi040 ceph-mon[32418]: pgmap v91: 97 pgs: 97 active+clean; 42 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-25T01:57:27.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 46972 2.87 MB/sec warmup 94 sec latency 18.509 ms 2024-04-25T01:57:28.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:27 smithi134 ceph-mon[33098]: pgmap v92: 97 pgs: 97 active+clean; 42 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T01:57:28.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:27 smithi031 ceph-mon[29204]: pgmap v92: 97 pgs: 97 active+clean; 42 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T01:57:28.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 47532 2.89 MB/sec warmup 95 sec latency 19.104 ms 2024-04-25T01:57:28.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:27 smithi040 ceph-mon[32418]: pgmap v92: 97 pgs: 97 active+clean; 42 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T01:57:29.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 48073 2.87 MB/sec warmup 96 sec latency 18.145 ms 2024-04-25T01:57:30.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:29 smithi134 ceph-mon[33098]: pgmap v93: 97 pgs: 97 active+clean; 42 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T01:57:30.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:29 smithi031 ceph-mon[29204]: pgmap v93: 97 pgs: 97 active+clean; 42 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T01:57:30.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 48615 2.86 MB/sec warmup 97 sec latency 18.292 ms 2024-04-25T01:57:30.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:29 smithi040 ceph-mon[32418]: pgmap v93: 97 pgs: 97 active+clean; 42 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T01:57:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.512+0000 7f37a02cd640 1 -- 172.21.15.31:0/2858361333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3798100670 msgr2=0x7f3798100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.512+0000 7f37a02cd640 1 --2- 172.21.15.31:0/2858361333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3798100670 0x7f3798100af0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3788009930 tx=0x7f378802a420 comp rx=0 tx=0).stop 2024-04-25T01:57:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.512+0000 7f37a02cd640 1 -- 172.21.15.31:0/2858361333 shutdown_connections 2024-04-25T01:57:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.512+0000 7f37a02cd640 1 --2- 172.21.15.31:0/2858361333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3798101030 0x7f379810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.512+0000 7f37a02cd640 1 --2- 172.21.15.31:0/2858361333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3798100670 0x7f3798100af0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.512+0000 7f37a02cd640 1 --2- 172.21.15.31:0/2858361333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3798106630 0x7f3798106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.512+0000 7f37a02cd640 1 -- 172.21.15.31:0/2858361333 >> 172.21.15.31:0/2858361333 conn(0x7f37980777c0 msgr2=0x7f3798077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.512+0000 7f37a02cd640 1 -- 172.21.15.31:0/2858361333 shutdown_connections 2024-04-25T01:57:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.512+0000 7f37a02cd640 1 -- 172.21.15.31:0/2858361333 wait complete. 2024-04-25T01:57:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.513+0000 7f37a02cd640 1 Processor -- start 2024-04-25T01:57:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.513+0000 7f37a02cd640 1 -- start start 2024-04-25T01:57:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.513+0000 7f37a02cd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3798100670 0x7f379819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.513+0000 7f37a02cd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3798101030 0x7f379819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.513+0000 7f37a02cd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3798106630 0x7f379819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.513+0000 7f37a02cd640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f37981108b0 con 0x7f3798101030 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.513+0000 7f37a02cd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3798110730 con 0x7f3798106630 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.513+0000 7f37a02cd640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3798110a30 con 0x7f3798100670 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.513+0000 7f379d868640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3798101030 0x7f379819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f379d868640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3798101030 0x7f379819a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51918/0 (socket says 172.21.15.31:51918) 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f379d868640 1 -- 172.21.15.31:0/3612180039 learned_addr learned my addr 172.21.15.31:0/3612180039 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f379e86a640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3798106630 0x7f379819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f379e069640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3798100670 0x7f379819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f379d868640 1 -- 172.21.15.31:0/3612180039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3798100670 msgr2=0x7f379819a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f379d868640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3798100670 0x7f379819a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f379d868640 1 -- 172.21.15.31:0/3612180039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3798106630 msgr2=0x7f379819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f379d868640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3798106630 0x7f379819e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f379d868640 1 -- 172.21.15.31:0/3612180039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f379819f0b0 con 0x7f3798101030 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f379e069640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3798100670 0x7f379819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f379d868640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3798101030 0x7f379819a600 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3788005940 tx=0x7f37880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f378f7fe640 1 -- 172.21.15.31:0/3612180039 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378803f030 con 0x7f3798101030 2024-04-25T01:57:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f378f7fe640 1 -- 172.21.15.31:0/3612180039 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37880318a0 con 0x7f3798101030 2024-04-25T01:57:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f37a02cd640 1 -- 172.21.15.31:0/3612180039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f379819f340 con 0x7f3798101030 2024-04-25T01:57:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f378f7fe640 1 -- 172.21.15.31:0/3612180039 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788035070 con 0x7f3798101030 2024-04-25T01:57:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.514+0000 7f37a02cd640 1 -- 172.21.15.31:0/3612180039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37981a6c30 con 0x7f3798101030 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.517+0000 7f378f7fe640 1 -- 172.21.15.31:0/3612180039 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3788042030 con 0x7f3798101030 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.517+0000 7f37a2fda740 1 Processor -- start 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.517+0000 7f37a2fda740 1 -- start start 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 0x5592ce50e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 0x5592ce518cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce519230 0x5592ce51b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f37a2fda740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5592ce472cb0 con 0x5592ce50e240 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f37a2fda740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5592ce41b5b0 con 0x5592ce50eb60 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f37a2fda740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5592ce46ec70 con 0x5592ce519230 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f378f7fe640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37840a2e20 0x7f37840a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e86a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 0x5592ce50e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e069640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 0x5592ce518cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e86a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 0x5592ce50e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51922/0 (socket says 172.21.15.31:51922) 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e86a640 1 -- 172.21.15.31:0/3701538893 learned_addr learned my addr 172.21.15.31:0/3701538893 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e069640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 0x5592ce518cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33120/0 (socket says 172.21.15.31:33120) 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379d868640 1 --2- 172.21.15.31:0/3701538893 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce519230 0x5592ce51b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f378f7fe640 1 -- 172.21.15.31:0/3612180039 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f37880efb20 con 0x7f3798101030 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e86a640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37840a2e20 0x7f37840a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e069640 1 -- 172.21.15.31:0/3701538893 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce519230 msgr2=0x5592ce51b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e069640 1 --2- 172.21.15.31:0/3701538893 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce519230 0x5592ce51b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e069640 1 -- 172.21.15.31:0/3701538893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 msgr2=0x5592ce50e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e069640 1 --2- 172.21.15.31:0/3701538893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 0x5592ce50e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e069640 1 -- 172.21.15.31:0/3701538893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592ce187210 con 0x5592ce50eb60 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e86a640 1 --2- 172.21.15.31:0/3701538893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 0x5592ce50e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.518+0000 7f379e069640 1 --2- 172.21.15.31:0/3701538893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 0x5592ce518cf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3794005fd0 tx=0x7f379400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61788dc1dd64aaa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.519+0000 7f379e86a640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37840a2e20 0x7f37840a52e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3790009fd0 tx=0x7f3790004ab0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.519+0000 7f378cbfe640 1 -- 172.21.15.31:0/3701538893 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3794015070 con 0x5592ce50eb60 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.519+0000 7f378cbfe640 1 -- 172.21.15.31:0/3701538893 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37940035f0 con 0x5592ce50eb60 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.519+0000 7f378cbfe640 1 -- 172.21.15.31:0/3701538893 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37940116a0 con 0x5592ce50eb60 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.519+0000 7f37a2fda740 1 -- 172.21.15.31:0/3701538893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 msgr2=0x5592ce518cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.519+0000 7f37a2fda740 1 --2- 172.21.15.31:0/3701538893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 0x5592ce518cf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3794005fd0 tx=0x7f379400f730 comp rx=0 tx=0).stop 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.519+0000 7f37a2fda740 1 -- 172.21.15.31:0/3701538893 shutdown_connections 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.519+0000 7f37a2fda740 1 --2- 172.21.15.31:0/3701538893 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce519230 0x5592ce51b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.519+0000 7f37a2fda740 1 --2- 172.21.15.31:0/3701538893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 0x5592ce518cf0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.519+0000 7f37a2fda740 1 --2- 172.21.15.31:0/3701538893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 0x5592ce50e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.519+0000 7f37a2fda740 1 -- 172.21.15.31:0/3701538893 >> 172.21.15.31:0/3701538893 conn(0x5592ce50a560 msgr2=0x5592ce50a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f37a2fda740 1 -- 172.21.15.31:0/3701538893 shutdown_connections 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f37a2fda740 1 -- 172.21.15.31:0/3701538893 wait complete. 2024-04-25T01:57:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f37a2fda740 1 -- start start 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f37a2fda740 1 Processor -- start 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50e240 0x5592ce516170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 0x5592ce5166b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 0x5592ce516bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f37a2fda740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5592ce51c020 con 0x5592ce519230 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f37a2fda740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5592ce42eb50 con 0x5592ce50eb60 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f37a2fda740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5592ce51c1a0 con 0x5592ce50e240 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f37a2fda740 10 client.0 _ll_register_callbacks cb 0x5592ce469ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f379e86a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 0x5592ce516bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f379e86a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 0x5592ce516bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51926/0 (socket says 172.21.15.31:51926) 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f379e86a640 1 -- 172.21.15.31:0/1231526827 learned_addr learned my addr 172.21.15.31:0/1231526827 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f379d868640 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 0x5592ce5166b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.520+0000 7f379e069640 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50e240 0x5592ce516170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f379d868640 1 -- 172.21.15.31:0/1231526827 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50e240 msgr2=0x5592ce516170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f379d868640 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50e240 0x5592ce516170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f379d868640 1 -- 172.21.15.31:0/1231526827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 msgr2=0x5592ce516bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f379d868640 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 0x5592ce516bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f379d868640 1 -- 172.21.15.31:0/1231526827 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592ce253620 con 0x5592ce50eb60 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f379e86a640 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 0x5592ce516bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f379d868640 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 0x5592ce5166b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f37880ed2c0 tx=0x7f378808afc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef94bcff5db9283e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f3781ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f3781ffb640 1 -- 172.21.15.31:0/1231526827 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788049040 con 0x5592ce50eb60 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f37a2fda740 1 -- 172.21.15.31:0/1231526827 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37940035f0 con 0x5592ce50eb60 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f3781ffb640 1 -- 172.21.15.31:0/1231526827 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f37880b8700 con 0x5592ce50eb60 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f3781ffb640 1 -- 172.21.15.31:0/1231526827 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f378803b460 con 0x5592ce50eb60 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f37a2fda740 10 client.25982 fetch_fsmap learned FSMap version 30 2024-04-25T01:57:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f37a2fda740 1 -- 172.21.15.31:0/1231526827 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5592ce528210 con 0x5592ce50eb60 2024-04-25T01:57:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.521+0000 7f3781ffb640 1 -- 172.21.15.31:0/1231526827 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37940035f0 con 0x5592ce50eb60 2024-04-25T01:57:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.522+0000 7f3781ffb640 1 -- 172.21.15.31:0/1231526827 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f378808a020 con 0x5592ce50eb60 2024-04-25T01:57:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.522+0000 7f37a2fda740 10 client.25982 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:57:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.522+0000 7f37a2fda740 10 client.25982 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.522+0000 7f37a2fda740 20 client.25982 populate_metadata read hostname 'smithi031' 2024-04-25T01:57:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.522+0000 7f37a2fda740 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5592ce529720 0x5592ce52bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.522+0000 7f37a2fda740 4 client.25982 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.522+0000 7f37a2fda740 1 -- 172.21.15.31:0/1231526827 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592ce4801f0 con 0x5592ce529720 2024-04-25T01:57:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.522+0000 7f379e069640 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5592ce529720 0x5592ce52bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.522+0000 7f379e069640 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5592ce529720 0x5592ce52bba0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66d74060602b6d7b server_cookie=f6407ff130215f0a in_seq=0 out_seq=0 2024-04-25T01:57:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.522+0000 7f3781ffb640 10 client.25982 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.523+0000 7f3781ffb640 1 -- 172.21.15.31:0/1231526827 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5592ce4801f0 con 0x5592ce529720 2024-04-25T01:57:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.523+0000 7f3781ffb640 10 client.25982 handle_command_reply: tid=0 2024-04-25T01:57:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.523+0000 7f37a2fda740 1 client.25982 shutdown 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.523+0000 7f37a2fda740 1 -- 172.21.15.31:0/1231526827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 msgr2=0x5592ce5166b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.523+0000 7f37a2fda740 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 0x5592ce5166b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f37880ed2c0 tx=0x7f378808afc0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.523+0000 7f37a2fda740 1 -- 172.21.15.31:0/1231526827 shutdown_connections 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.524+0000 7f37a2fda740 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5592ce529720 0x5592ce52bba0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.524+0000 7f37a2fda740 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 0x5592ce516bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.524+0000 7f3781ffb640 0 client.25982 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.524+0000 7f37a2fda740 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50eb60 0x5592ce5166b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.524+0000 7f37a2fda740 1 --2- 172.21.15.31:0/1231526827 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50e240 0x5592ce516170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.524+0000 7f37a2fda740 1 -- 172.21.15.31:0/1231526827 >> 172.21.15.31:0/1231526827 conn(0x5592ce50a560 msgr2=0x5592ce510260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.524+0000 7f37a2fda740 1 -- 172.21.15.31:0/1231526827 shutdown_connections 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.524+0000 7f37a2fda740 1 -- 172.21.15.31:0/1231526827 wait complete. 2024-04-25T01:57:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.524+0000 7f37a2fda740 20 client.25982 trim_cache size 0 max 16384 2024-04-25T01:57:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f37a2fda740 1 Processor -- start 2024-04-25T01:57:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f37a2fda740 1 -- start start 2024-04-25T01:57:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 0x5592ce5188a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 0x5592ce524f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce519230 0x5592ce5152d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f37a2fda740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f37981af3e0 con 0x5592ce50e240 2024-04-25T01:57:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f37a2fda740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f37981b0fe0 con 0x5592ce519230 2024-04-25T01:57:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f37a2fda740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f37981afab0 con 0x5592ce50eb60 2024-04-25T01:57:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f379e86a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce519230 0x5592ce5152d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f379d868640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 0x5592ce524f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f379e069640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 0x5592ce5188a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f379e86a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce519230 0x5592ce5152d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33128/0 (socket says 172.21.15.31:33128) 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f379d868640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 0x5592ce524f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:43316/0 (socket says 172.21.15.31:43316) 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.540+0000 7f379e86a640 1 -- 172.21.15.31:0/1764077801 learned_addr learned my addr 172.21.15.31:0/1764077801 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.541+0000 7f379e86a640 1 -- 172.21.15.31:0/1764077801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 msgr2=0x5592ce524f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.541+0000 7f379e86a640 1 --2- 172.21.15.31:0/1764077801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 0x5592ce524f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.541+0000 7f379e86a640 1 -- 172.21.15.31:0/1764077801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 msgr2=0x5592ce5188a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.541+0000 7f379e86a640 1 --2- 172.21.15.31:0/1764077801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 0x5592ce5188a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.541+0000 7f379e86a640 1 -- 172.21.15.31:0/1764077801 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592ce51b990 con 0x5592ce519230 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.541+0000 7f379e069640 1 --2- 172.21.15.31:0/1764077801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 0x5592ce5188a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.541+0000 7f379e86a640 1 --2- 172.21.15.31:0/1764077801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce519230 0x5592ce5152d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f37900134a0 tx=0x7f3790009be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=253dc37adee2da68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.541+0000 7f3783fff640 1 -- 172.21.15.31:0/1764077801 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37900109b0 con 0x5592ce519230 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.541+0000 7f3783fff640 1 -- 172.21.15.31:0/1764077801 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3790022440 con 0x5592ce519230 2024-04-25T01:57:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.541+0000 7f3783fff640 1 -- 172.21.15.31:0/1764077801 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3790021640 con 0x5592ce519230 2024-04-25T01:57:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.542+0000 7f37a2fda740 1 -- 172.21.15.31:0/1764077801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce519230 msgr2=0x5592ce5152d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.542+0000 7f37a2fda740 1 --2- 172.21.15.31:0/1764077801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce519230 0x5592ce5152d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f37900134a0 tx=0x7f3790009be0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.542+0000 7f37a2fda740 1 -- 172.21.15.31:0/1764077801 shutdown_connections 2024-04-25T01:57:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.542+0000 7f37a2fda740 1 --2- 172.21.15.31:0/1764077801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce519230 0x5592ce5152d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.542+0000 7f37a2fda740 1 --2- 172.21.15.31:0/1764077801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 0x5592ce524f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.542+0000 7f37a2fda740 1 --2- 172.21.15.31:0/1764077801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce50e240 0x5592ce5188a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.542+0000 7f37a2fda740 1 -- 172.21.15.31:0/1764077801 >> 172.21.15.31:0/1764077801 conn(0x5592ce50a560 msgr2=0x5592ce50f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.542+0000 7f37a2fda740 1 -- 172.21.15.31:0/1764077801 shutdown_connections 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.542+0000 7f37a2fda740 1 -- 172.21.15.31:0/1764077801 wait complete. 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.542+0000 7f37a2fda740 1 -- start start 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.542+0000 7f37a2fda740 1 Processor -- start 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50e240 0x5592ce521580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 0x5592ce51c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f37a2fda740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 0x5592ce51cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f37a2fda740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f37981b0850 con 0x5592ce519230 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f37a2fda740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f37981b0180 con 0x5592ce50e240 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f37a2fda740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f37981ad3c0 con 0x5592ce50eb60 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f37a2fda740 10 client.0 _ll_register_callbacks cb 0x7f3788009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379e86a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 0x5592ce51cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379e069640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50e240 0x5592ce521580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379d868640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 0x5592ce51c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379e069640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50e240 0x5592ce521580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33140/0 (socket says 172.21.15.31:33140) 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379e069640 1 -- 172.21.15.31:0/3630321250 learned_addr learned my addr 172.21.15.31:0/3630321250 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379e86a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 0x5592ce51cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51938/0 (socket says 172.21.15.31:51938) 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379d868640 1 -- 172.21.15.31:0/3630321250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50e240 msgr2=0x5592ce521580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379d868640 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50e240 0x5592ce521580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379d868640 1 -- 172.21.15.31:0/3630321250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 msgr2=0x5592ce51cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379d868640 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 0x5592ce51cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379d868640 1 -- 172.21.15.31:0/3630321250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592ce51d4e0 con 0x5592ce50eb60 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379e069640 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50e240 0x5592ce521580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f379d868640 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 0x5592ce51c9e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f37880ed8c0 tx=0x7f37880b8330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfda6e7b7bb0984e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f3781ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f3781ffb640 1 -- 172.21.15.31:0/3630321250 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37880bb9f0 con 0x5592ce50eb60 2024-04-25T01:57:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f3781ffb640 1 -- 172.21.15.31:0/3630321250 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f37880bc390 con 0x5592ce50eb60 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.543+0000 7f37a2fda740 1 -- 172.21.15.31:0/3630321250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3790022440 con 0x5592ce50eb60 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f3781ffb640 1 -- 172.21.15.31:0/3630321250 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f37880bc790 con 0x5592ce50eb60 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f37a2fda740 10 client.25840 fetch_fsmap learned FSMap version 30 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f37a2fda740 1 -- 172.21.15.31:0/3630321250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5592ce58fa80 con 0x5592ce50eb60 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f3781ffb640 1 -- 172.21.15.31:0/3630321250 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3790022440 con 0x5592ce50eb60 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f3781ffb640 1 -- 172.21.15.31:0/3630321250 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f378808a070 con 0x5592ce50eb60 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f37a2fda740 10 client.25840 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f37a2fda740 10 client.25840 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f37a2fda740 20 client.25840 populate_metadata read hostname 'smithi031' 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f37a2fda740 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5592ce529720 0x5592ce592130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f37a2fda740 4 client.25840 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f37a2fda740 1 -- 172.21.15.31:0/3630321250 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5592ce47a540 con 0x5592ce529720 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.544+0000 7f379e069640 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5592ce529720 0x5592ce592130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.545+0000 7f379e069640 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5592ce529720 0x5592ce592130 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6983712442e5d4e1 server_cookie=45b37622fb555d36 in_seq=0 out_seq=0 2024-04-25T01:57:30.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.545+0000 7f3781ffb640 10 client.25840 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:30.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f3781ffb640 1 -- 172.21.15.31:0/3630321250 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5592ce47a540 con 0x5592ce529720 2024-04-25T01:57:30.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f3781ffb640 10 client.25840 handle_command_reply: tid=0 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 1 client.25840 shutdown 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 1 -- 172.21.15.31:0/3630321250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 msgr2=0x5592ce51c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 0x5592ce51c9e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f37880ed8c0 tx=0x7f37880b8330 comp rx=0 tx=0).stop 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 1 -- 172.21.15.31:0/3630321250 shutdown_connections 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5592ce529720 0x5592ce592130 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592ce519230 0x5592ce51cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592ce50eb60 0x5592ce51c9e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 1 --2- 172.21.15.31:0/3630321250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5592ce50e240 0x5592ce521580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f3781ffb640 0 client.25840 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 1 -- 172.21.15.31:0/3630321250 >> 172.21.15.31:0/3630321250 conn(0x5592ce50a560 msgr2=0x5592ce510260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 1 -- 172.21.15.31:0/3630321250 shutdown_connections 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 1 -- 172.21.15.31:0/3630321250 wait complete. 2024-04-25T01:57:30.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.564+0000 7f37a2fda740 20 client.25840 trim_cache size 0 max 16384 2024-04-25T01:57:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:57:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:57:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:57:30.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:57:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 -- 172.21.15.31:0/3612180039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37840a2e20 msgr2=0x7f37840a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37840a2e20 0x7f37840a52e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3790009fd0 tx=0x7f3790004ab0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 -- 172.21.15.31:0/3612180039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3798101030 msgr2=0x7f379819a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3798101030 0x7f379819a600 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3788005940 tx=0x7f37880042f0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 -- 172.21.15.31:0/3612180039 shutdown_connections 2024-04-25T01:57:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37840a2e20 0x7f37840a52e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3798106630 0x7f379819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3798101030 0x7f379819a600 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 --2- 172.21.15.31:0/3612180039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3798100670 0x7f379819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 -- 172.21.15.31:0/3612180039 >> 172.21.15.31:0/3612180039 conn(0x7f37980777c0 msgr2=0x7f3798102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 -- 172.21.15.31:0/3612180039 shutdown_connections 2024-04-25T01:57:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:30.565+0000 7f37a02cd640 1 -- 172.21.15.31:0/3612180039 wait complete. 2024-04-25T01:57:31.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 49408 2.90 MB/sec warmup 98 sec latency 12.258 ms 2024-04-25T01:57:32.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:31 smithi134 ceph-mon[33098]: pgmap v94: 97 pgs: 97 active+clean; 41 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-25T01:57:32.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:31 smithi031 ceph-mon[29204]: pgmap v94: 97 pgs: 97 active+clean; 41 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-25T01:57:32.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 49795 2.90 MB/sec warmup 99 sec latency 12.244 ms 2024-04-25T01:57:32.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:31 smithi040 ceph-mon[32418]: pgmap v94: 97 pgs: 97 active+clean; 41 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-25T01:57:33.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 50097 2.89 MB/sec warmup 100 sec latency 20.104 ms 2024-04-25T01:57:34.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:33 smithi134 ceph-mon[33098]: pgmap v95: 97 pgs: 97 active+clean; 41 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T01:57:34.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:33 smithi031 ceph-mon[29204]: pgmap v95: 97 pgs: 97 active+clean; 41 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T01:57:34.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 50447 2.86 MB/sec warmup 101 sec latency 11.990 ms 2024-04-25T01:57:34.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:33 smithi040 ceph-mon[32418]: pgmap v95: 97 pgs: 97 active+clean; 41 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T01:57:35.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 50951 2.85 MB/sec warmup 102 sec latency 12.528 ms 2024-04-25T01:57:36.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:35 smithi134 ceph-mon[33098]: pgmap v96: 97 pgs: 97 active+clean; 46 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-04-25T01:57:36.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:35 smithi031 ceph-mon[29204]: pgmap v96: 97 pgs: 97 active+clean; 46 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-04-25T01:57:36.302 INFO:tasks.workunit.client.0.smithi031.stdout: 1 51581 2.86 MB/sec warmup 103 sec latency 18.150 ms 2024-04-25T01:57:36.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:35 smithi040 ceph-mon[32418]: pgmap v96: 97 pgs: 97 active+clean; 46 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 1.7 MiB/s wr, 191 op/s 2024-04-25T01:57:37.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:36 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:57:37.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:36 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:57:37.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 52121 2.84 MB/sec warmup 104 sec latency 18.318 ms 2024-04-25T01:57:37.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:36 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:57:38.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:37 smithi134 ceph-mon[33098]: pgmap v97: 97 pgs: 97 active+clean; 46 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-04-25T01:57:38.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:37 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:57:38.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:37 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:57:38.191 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:37 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:57:38.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:37 smithi031 ceph-mon[29204]: pgmap v97: 97 pgs: 97 active+clean; 46 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-04-25T01:57:38.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:37 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:57:38.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:37 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:57:38.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:37 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:57:38.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 52854 2.87 MB/sec warmup 105 sec latency 18.506 ms 2024-04-25T01:57:38.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:37 smithi040 ceph-mon[32418]: pgmap v97: 97 pgs: 97 active+clean; 46 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-04-25T01:57:38.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:37 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:57:38.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:37 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:57:38.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:37 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:57:39.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 53247 2.89 MB/sec warmup 106 sec latency 18.520 ms 2024-04-25T01:57:40.213 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:39 smithi040 ceph-mon[32418]: pgmap v98: 97 pgs: 97 active+clean; 46 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-04-25T01:57:40.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:39 smithi031 ceph-mon[29204]: pgmap v98: 97 pgs: 97 active+clean; 46 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-04-25T01:57:40.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 53616 2.88 MB/sec warmup 107 sec latency 14.763 ms 2024-04-25T01:57:40.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:39 smithi134 ceph-mon[33098]: pgmap v98: 97 pgs: 97 active+clean; 46 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-04-25T01:57:41.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 53903 2.85 MB/sec warmup 108 sec latency 18.462 ms 2024-04-25T01:57:42.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:41 smithi031 ceph-mon[29204]: pgmap v99: 97 pgs: 97 active+clean; 47 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.8 MiB/s wr, 191 op/s 2024-04-25T01:57:42.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 54414 2.84 MB/sec warmup 109 sec latency 18.026 ms 2024-04-25T01:57:42.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:41 smithi040 ceph-mon[32418]: pgmap v99: 97 pgs: 97 active+clean; 47 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.8 MiB/s wr, 191 op/s 2024-04-25T01:57:42.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:41 smithi134 ceph-mon[33098]: pgmap v99: 97 pgs: 97 active+clean; 47 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.8 MiB/s wr, 191 op/s 2024-04-25T01:57:43.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 55011 2.85 MB/sec warmup 110 sec latency 18.352 ms 2024-04-25T01:57:43.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.661+0000 7fe117653640 1 -- 172.21.15.31:0/3165071088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe11010e020 msgr2=0x7fe110110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:43.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.661+0000 7fe117653640 1 --2- 172.21.15.31:0/3165071088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe11010e020 0x7fe110110410 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe108009ca0 tx=0x7fe10802a950 comp rx=0 tx=0).stop 2024-04-25T01:57:43.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.661+0000 7fe117653640 1 -- 172.21.15.31:0/3165071088 shutdown_connections 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.661+0000 7fe117653640 1 --2- 172.21.15.31:0/3165071088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe11010e020 0x7fe110110410 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.661+0000 7fe117653640 1 --2- 172.21.15.31:0/3165071088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe110074410 0x7fe11010dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.661+0000 7fe117653640 1 --2- 172.21.15.31:0/3165071088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe110073af0 0x7fe110073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.661+0000 7fe117653640 1 -- 172.21.15.31:0/3165071088 >> 172.21.15.31:0/3165071088 conn(0x7fe1100fd1f0 msgr2=0x7fe1100ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.661+0000 7fe117653640 1 -- 172.21.15.31:0/3165071088 shutdown_connections 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.661+0000 7fe117653640 1 -- 172.21.15.31:0/3165071088 wait complete. 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe117653640 1 Processor -- start 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe117653640 1 -- start start 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe117653640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110073af0 0x7fe11019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe117653640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe110074410 0x7fe11019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe117653640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe11010e020 0x7fe11019e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe117653640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe110113bf0 con 0x7fe110073af0 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe117653640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe110113a70 con 0x7fe11010e020 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe117653640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe110113d70 con 0x7fe110074410 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe1153ef640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110073af0 0x7fe11019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe1153ef640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110073af0 0x7fe11019a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45352/0 (socket says 172.21.15.31:45352) 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe1153ef640 1 -- 172.21.15.31:0/1374281861 learned_addr learned my addr 172.21.15.31:0/1374281861 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe115bf0640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe11010e020 0x7fe11019e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe114bee640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe110074410 0x7fe11019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe1153ef640 1 -- 172.21.15.31:0/1374281861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe110074410 msgr2=0x7fe11019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe1153ef640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe110074410 0x7fe11019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe1153ef640 1 -- 172.21.15.31:0/1374281861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe11010e020 msgr2=0x7fe11019e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe1153ef640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe11010e020 0x7fe11019e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.662+0000 7fe1153ef640 1 -- 172.21.15.31:0/1374281861 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11019f040 con 0x7fe110073af0 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.663+0000 7fe1153ef640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110073af0 0x7fe11019a050 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00e9f0 tx=0x7fe10c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.663+0000 7fe1067fc640 1 -- 172.21.15.31:0/1374281861 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c00cd80 con 0x7fe110073af0 2024-04-25T01:57:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.663+0000 7fe117653640 1 -- 172.21.15.31:0/1374281861 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11019f330 con 0x7fe110073af0 2024-04-25T01:57:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.663+0000 7fe1067fc640 1 -- 172.21.15.31:0/1374281861 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe10c0108e0 con 0x7fe110073af0 2024-04-25T01:57:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.663+0000 7fe1067fc640 1 -- 172.21.15.31:0/1374281861 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c018920 con 0x7fe110073af0 2024-04-25T01:57:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.663+0000 7fe117653640 1 -- 172.21.15.31:0/1374281861 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1101a6bc0 con 0x7fe110073af0 2024-04-25T01:57:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.665+0000 7fe1067fc640 1 -- 172.21.15.31:0/1374281861 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe10c010a90 con 0x7fe110073af0 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.666+0000 7fe1067fc640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0fc0a2da0 0x7fe0fc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.666+0000 7fe11a434740 1 Processor -- start 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.666+0000 7fe11a434740 1 -- start start 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.666+0000 7fe114bee640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0fc0a2da0 0x7fe0fc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.666+0000 7fe1067fc640 1 -- 172.21.15.31:0/1374281861 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fe10c0cf6d0 con 0x7fe110073af0 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.666+0000 7fe11a434740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4b240 0x560f25e4b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.666+0000 7fe11a434740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4bb60 0x560f25e55cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.666+0000 7fe11a434740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e58620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.666+0000 7fe11a434740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560f25dafcb0 con 0x560f25e4b240 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.666+0000 7fe11a434740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560f25d585b0 con 0x560f25e56230 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.666+0000 7fe11a434740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560f25dabc70 con 0x560f25e4bb60 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe1153ef640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4b240 0x560f25e4b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe1153ef640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4b240 0x560f25e4b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45366/0 (socket says 172.21.15.31:45366) 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe1153ef640 1 -- 172.21.15.31:0/818643191 learned_addr learned my addr 172.21.15.31:0/818643191 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe115bf0640 1 --2- 172.21.15.31:0/818643191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e58620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe114bee640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0fc0a2da0 0x7fe0fc0a5260 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe11010f360 tx=0x7fe10000b390 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe114bee640 1 --2- 172.21.15.31:0/818643191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4bb60 0x560f25e55cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe1153ef640 1 -- 172.21.15.31:0/818643191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4bb60 msgr2=0x560f25e55cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe1153ef640 1 --2- 172.21.15.31:0/818643191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4bb60 0x560f25e55cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe1153ef640 1 -- 172.21.15.31:0/818643191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 msgr2=0x560f25e58620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe1153ef640 1 --2- 172.21.15.31:0/818643191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e58620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe1153ef640 1 -- 172.21.15.31:0/818643191 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f25ac4210 con 0x560f25e4b240 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe114bee640 1 --2- 172.21.15.31:0/818643191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4bb60 0x560f25e55cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe1153ef640 1 --2- 172.21.15.31:0/818643191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4b240 0x560f25e4b620 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0986c0 tx=0x7fe10c09bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee7fb8ca4cb52e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe0fb7fe640 1 -- 172.21.15.31:0/818643191 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0cd0f0 con 0x560f25e4b240 2024-04-25T01:57:43.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe0fb7fe640 1 -- 172.21.15.31:0/818643191 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe10c098360 con 0x560f25e4b240 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.667+0000 7fe0fb7fe640 1 -- 172.21.15.31:0/818643191 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c097ca0 con 0x560f25e4b240 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.668+0000 7fe11a434740 1 -- 172.21.15.31:0/818643191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4b240 msgr2=0x560f25e4b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.668+0000 7fe11a434740 1 --2- 172.21.15.31:0/818643191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4b240 0x560f25e4b620 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0986c0 tx=0x7fe10c09bdb0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.668+0000 7fe11a434740 1 -- 172.21.15.31:0/818643191 shutdown_connections 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.668+0000 7fe11a434740 1 --2- 172.21.15.31:0/818643191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e58620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.668+0000 7fe11a434740 1 --2- 172.21.15.31:0/818643191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4bb60 0x560f25e55cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.668+0000 7fe11a434740 1 --2- 172.21.15.31:0/818643191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4b240 0x560f25e4b620 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.668+0000 7fe11a434740 1 -- 172.21.15.31:0/818643191 >> 172.21.15.31:0/818643191 conn(0x560f25e47560 msgr2=0x560f25e47940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.668+0000 7fe11a434740 1 -- 172.21.15.31:0/818643191 shutdown_connections 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.668+0000 7fe11a434740 1 -- 172.21.15.31:0/818643191 wait complete. 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.668+0000 7fe11a434740 1 -- start start 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe11a434740 1 Processor -- start 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe11a434740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e4b240 0x560f25e531a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe11a434740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e536e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe11a434740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e56230 0x560f25e53c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe11a434740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560f25e59020 con 0x560f25e4bb60 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe11a434740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560f25d6bb50 con 0x560f25e4b240 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe11a434740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560f25e591a0 con 0x560f25e56230 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe11a434740 10 client.0 _ll_register_callbacks cb 0x560f25da6ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe114bee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe1153ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e4b240 0x560f25e531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe1153ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e4b240 0x560f25e531a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60108/0 (socket says 172.21.15.31:60108) 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe114bee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45368/0 (socket says 172.21.15.31:45368) 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe1153ef640 1 -- 172.21.15.31:0/3360153050 learned_addr learned my addr 172.21.15.31:0/3360153050 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe115bf0640 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e56230 0x560f25e53c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe114bee640 1 -- 172.21.15.31:0/3360153050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e56230 msgr2=0x560f25e53c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe114bee640 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e56230 0x560f25e53c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe114bee640 1 -- 172.21.15.31:0/3360153050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e4b240 msgr2=0x560f25e531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe114bee640 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e4b240 0x560f25e531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe114bee640 1 -- 172.21.15.31:0/3360153050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f25b90620 con 0x560f25e4bb60 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.669+0000 7fe1153ef640 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e4b240 0x560f25e531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.670+0000 7fe114bee640 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e536e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe1000043b0 tx=0x7fe100004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b48df15ed71a78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.670+0000 7fe0f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.670+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/3360153050 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100027670 con 0x560f25e4bb60 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.670+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/3360153050 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fe100027810 con 0x560f25e4bb60 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.670+0000 7fe11a434740 1 -- 172.21.15.31:0/3360153050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe10c098360 con 0x560f25e4bb60 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.670+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/3360153050 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe100027a10 con 0x560f25e4bb60 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.670+0000 7fe11a434740 10 client.16401 fetch_fsmap learned FSMap version 30 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.670+0000 7fe11a434740 1 -- 172.21.15.31:0/3360153050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560f25e651c0 con 0x560f25e4bb60 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.670+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/3360153050 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe10c098360 con 0x560f25e4bb60 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.670+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/3360153050 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fe100030c80 con 0x560f25e4bb60 2024-04-25T01:57:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.671+0000 7fe11a434740 10 client.16401 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.671+0000 7fe11a434740 10 client.16401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.671+0000 7fe11a434740 20 client.16401 populate_metadata read hostname 'smithi031' 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.671+0000 7fe11a434740 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560f25e66680 0x560f25e68ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.671+0000 7fe11a434740 4 client.16401 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.671+0000 7fe11a434740 1 -- 172.21.15.31:0/3360153050 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f25dbd1f0 con 0x560f25e66680 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.671+0000 7fe1153ef640 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560f25e66680 0x560f25e68ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.671+0000 7fe0f8ff9640 10 client.16401 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.671+0000 7fe1153ef640 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560f25e66680 0x560f25e68ac0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffb18cefc47565ca server_cookie=b19d9ea5ffc1915d in_seq=0 out_seq=0 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/3360153050 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f25dbd1f0 con 0x560f25e66680 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe0f8ff9640 10 client.16401 handle_command_reply: tid=0 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 1 client.16401 shutdown 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 1 -- 172.21.15.31:0/3360153050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 msgr2=0x560f25e536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e536e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe1000043b0 tx=0x7fe100004730 comp rx=0 tx=0).stop 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 1 -- 172.21.15.31:0/3360153050 shutdown_connections 2024-04-25T01:57:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560f25e66680 0x560f25e68ac0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e56230 0x560f25e53c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e536e0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe0f8ff9640 0 client.16401 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 1 --2- 172.21.15.31:0/3360153050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e4b240 0x560f25e531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 1 -- 172.21.15.31:0/3360153050 >> 172.21.15.31:0/3360153050 conn(0x560f25e47560 msgr2=0x560f25e4d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 1 -- 172.21.15.31:0/3360153050 shutdown_connections 2024-04-25T01:57:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 1 -- 172.21.15.31:0/3360153050 wait complete. 2024-04-25T01:57:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.672+0000 7fe11a434740 20 client.16401 trim_cache size 0 max 16384 2024-04-25T01:57:43.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe11a434740 1 Processor -- start 2024-04-25T01:57:43.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe11a434740 1 -- start start 2024-04-25T01:57:43.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe11a434740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4b240 0x560f25e61ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe11a434740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e4d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe11a434740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e6e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe11a434740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101af530 con 0x560f25e4bb60 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe11a434740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101b1130 con 0x560f25e56230 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe11a434740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101afc00 con 0x560f25e4b240 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe114bee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e4d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe114bee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e4d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45376/0 (socket says 172.21.15.31:45376) 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe114bee640 1 -- 172.21.15.31:0/3609139060 learned_addr learned my addr 172.21.15.31:0/3609139060 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe1153ef640 1 --2- 172.21.15.31:0/3609139060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4b240 0x560f25e61ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.689+0000 7fe115bf0640 1 --2- 172.21.15.31:0/3609139060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e6e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.690+0000 7fe114bee640 1 -- 172.21.15.31:0/3609139060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4b240 msgr2=0x560f25e61ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.690+0000 7fe114bee640 1 --2- 172.21.15.31:0/3609139060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4b240 0x560f25e61ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.690+0000 7fe114bee640 1 -- 172.21.15.31:0/3609139060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 msgr2=0x560f25e6e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.690+0000 7fe114bee640 1 --2- 172.21.15.31:0/3609139060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e6e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.690+0000 7fe114bee640 1 -- 172.21.15.31:0/3609139060 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f25e4c700 con 0x560f25e4bb60 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.690+0000 7fe1153ef640 1 --2- 172.21.15.31:0/3609139060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4b240 0x560f25e61ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.690+0000 7fe114bee640 1 --2- 172.21.15.31:0/3609139060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e4d9a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe100003f50 tx=0x7fe100011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8e62bf5bb5793 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:43.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.690+0000 7fe0faffd640 1 -- 172.21.15.31:0/3609139060 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10003f070 con 0x560f25e4bb60 2024-04-25T01:57:43.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.690+0000 7fe0faffd640 1 -- 172.21.15.31:0/3609139060 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe100022070 con 0x560f25e4bb60 2024-04-25T01:57:43.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.690+0000 7fe0faffd640 1 -- 172.21.15.31:0/3609139060 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100036070 con 0x560f25e4bb60 2024-04-25T01:57:43.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.691+0000 7fe11a434740 1 -- 172.21.15.31:0/3609139060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 msgr2=0x560f25e4d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.691+0000 7fe11a434740 1 --2- 172.21.15.31:0/3609139060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e4d9a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe100003f50 tx=0x7fe100011ca0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.691+0000 7fe11a434740 1 -- 172.21.15.31:0/3609139060 shutdown_connections 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.691+0000 7fe11a434740 1 --2- 172.21.15.31:0/3609139060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e6e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.691+0000 7fe11a434740 1 --2- 172.21.15.31:0/3609139060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e4d9a0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.691+0000 7fe11a434740 1 --2- 172.21.15.31:0/3609139060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4b240 0x560f25e61ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.691+0000 7fe11a434740 1 -- 172.21.15.31:0/3609139060 >> 172.21.15.31:0/3609139060 conn(0x560f25e47560 msgr2=0x560f25e465c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.691+0000 7fe11a434740 1 -- 172.21.15.31:0/3609139060 shutdown_connections 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.691+0000 7fe11a434740 1 -- 172.21.15.31:0/3609139060 wait complete. 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.691+0000 7fe11a434740 1 -- start start 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.691+0000 7fe11a434740 1 Processor -- start 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe11a434740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4b240 0x560f25e50120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe11a434740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e50660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe11a434740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e50ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe11a434740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101b09a0 con 0x560f25e4bb60 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe11a434740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101b02d0 con 0x560f25e56230 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe11a434740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101ad510 con 0x560f25e4b240 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe11a434740 10 client.0 _ll_register_callbacks cb 0x7fe1000042c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe114bee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e50660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe115bf0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e50ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe114bee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e50660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45382/0 (socket says 172.21.15.31:45382) 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe114bee640 1 -- 172.21.15.31:0/500657430 learned_addr learned my addr 172.21.15.31:0/500657430 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe115bf0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e50ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60122/0 (socket says 172.21.15.31:60122) 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe1153ef640 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4b240 0x560f25e50120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe115bf0640 1 -- 172.21.15.31:0/500657430 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4b240 msgr2=0x560f25e50120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe115bf0640 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4b240 0x560f25e50120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe115bf0640 1 -- 172.21.15.31:0/500657430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 msgr2=0x560f25e50660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe115bf0640 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e50660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe115bf0640 1 -- 172.21.15.31:0/500657430 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f25ecbee0 con 0x560f25e56230 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.692+0000 7fe114bee640 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e50660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:43.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.693+0000 7fe115bf0640 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e50ba0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe108005cc0 tx=0x7fe10802f5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82f117f8b8f76930 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.693+0000 7fe0f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.693+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/500657430 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108034040 con 0x560f25e56230 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.693+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/500657430 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fe108003d80 con 0x560f25e56230 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.693+0000 7fe11a434740 1 -- 172.21.15.31:0/500657430 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe100022070 con 0x560f25e56230 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.693+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/500657430 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe108055050 con 0x560f25e56230 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.693+0000 7fe11a434740 10 client.26006 fetch_fsmap learned FSMap version 30 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.693+0000 7fe11a434740 1 -- 172.21.15.31:0/500657430 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560f25ecd1d0 con 0x560f25e56230 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.693+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/500657430 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe100022070 con 0x560f25e56230 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.693+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/500657430 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fe108030560 con 0x560f25e56230 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.694+0000 7fe11a434740 10 client.26006 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.694+0000 7fe11a434740 10 client.26006 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.694+0000 7fe11a434740 20 client.26006 populate_metadata read hostname 'smithi031' 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.694+0000 7fe11a434740 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560f25ecda00 0x560f25ecfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.694+0000 7fe11a434740 4 client.26006 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:57:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.694+0000 7fe11a434740 1 -- 172.21.15.31:0/500657430 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560f25db7540 con 0x560f25ecda00 2024-04-25T01:57:43.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.694+0000 7fe1153ef640 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560f25ecda00 0x560f25ecfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:43.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.694+0000 7fe0f8ff9640 10 client.26006 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:43.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.694+0000 7fe1153ef640 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560f25ecda00 0x560f25ecfe20 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39c5910fcc57dd2f server_cookie=26f0cc4afa2619ec in_seq=0 out_seq=0 2024-04-25T01:57:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe0f8ff9640 1 -- 172.21.15.31:0/500657430 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560f25db7540 con 0x560f25ecda00 2024-04-25T01:57:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe0f8ff9640 10 client.26006 handle_command_reply: tid=0 2024-04-25T01:57:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 1 client.26006 shutdown 2024-04-25T01:57:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 1 -- 172.21.15.31:0/500657430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 msgr2=0x560f25e50ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e50ba0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe108005cc0 tx=0x7fe10802f5d0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 1 -- 172.21.15.31:0/500657430 shutdown_connections 2024-04-25T01:57:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560f25ecda00 0x560f25ecfe20 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f25e56230 0x560f25e50ba0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f25e4bb60 0x560f25e50660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 1 --2- 172.21.15.31:0/500657430 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f25e4b240 0x560f25e50120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 1 -- 172.21.15.31:0/500657430 >> 172.21.15.31:0/500657430 conn(0x560f25e47560 msgr2=0x560f25e53160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe0f8ff9640 0 client.26006 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 1 -- 172.21.15.31:0/500657430 shutdown_connections 2024-04-25T01:57:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 1 -- 172.21.15.31:0/500657430 wait complete. 2024-04-25T01:57:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.717+0000 7fe11a434740 20 client.26006 trim_cache size 0 max 16384 2024-04-25T01:57:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:57:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:57:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:57:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:57:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 -- 172.21.15.31:0/1374281861 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0fc0a2da0 msgr2=0x7fe0fc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0fc0a2da0 0x7fe0fc0a5260 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe11010f360 tx=0x7fe10000b390 comp rx=0 tx=0).stop 2024-04-25T01:57:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 -- 172.21.15.31:0/1374281861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110073af0 msgr2=0x7fe11019a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110073af0 0x7fe11019a050 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00e9f0 tx=0x7fe10c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 -- 172.21.15.31:0/1374281861 shutdown_connections 2024-04-25T01:57:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe0fc0a2da0 0x7fe0fc0a5260 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe11010e020 0x7fe11019e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe110074410 0x7fe11019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 --2- 172.21.15.31:0/1374281861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe110073af0 0x7fe11019a050 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 -- 172.21.15.31:0/1374281861 >> 172.21.15.31:0/1374281861 conn(0x7fe1100fd1f0 msgr2=0x7fe1100fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 -- 172.21.15.31:0/1374281861 shutdown_connections 2024-04-25T01:57:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:43.718+0000 7fe117653640 1 -- 172.21.15.31:0/1374281861 wait complete. 2024-04-25T01:57:44.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:43 smithi031 ceph-mon[29204]: pgmap v100: 97 pgs: 97 active+clean; 47 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-25T01:57:44.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 55597 2.84 MB/sec warmup 111 sec latency 18.971 ms 2024-04-25T01:57:44.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:43 smithi040 ceph-mon[32418]: pgmap v100: 97 pgs: 97 active+clean; 47 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-25T01:57:44.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:43 smithi134 ceph-mon[33098]: pgmap v100: 97 pgs: 97 active+clean; 47 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-25T01:57:45.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 56167 2.86 MB/sec warmup 112 sec latency 18.817 ms 2024-04-25T01:57:46.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:45 smithi031 ceph-mon[29204]: pgmap v101: 97 pgs: 97 active+clean; 42 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 1.8 MiB/s wr, 192 op/s 2024-04-25T01:57:46.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 56610 2.85 MB/sec warmup 113 sec latency 16.747 ms 2024-04-25T01:57:46.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:45 smithi040 ceph-mon[32418]: pgmap v101: 97 pgs: 97 active+clean; 42 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 1.8 MiB/s wr, 192 op/s 2024-04-25T01:57:46.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:45 smithi134 ceph-mon[33098]: pgmap v101: 97 pgs: 97 active+clean; 42 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 1.8 MiB/s wr, 192 op/s 2024-04-25T01:57:47.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 57083 2.87 MB/sec warmup 114 sec latency 18.358 ms 2024-04-25T01:57:48.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:47 smithi031 ceph-mon[29204]: pgmap v102: 97 pgs: 97 active+clean; 42 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:57:48.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 57362 2.84 MB/sec warmup 115 sec latency 11.939 ms 2024-04-25T01:57:48.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:47 smithi040 ceph-mon[32418]: pgmap v102: 97 pgs: 97 active+clean; 42 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:57:48.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:47 smithi134 ceph-mon[33098]: pgmap v102: 97 pgs: 97 active+clean; 42 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:57:49.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 57840 2.84 MB/sec warmup 116 sec latency 17.844 ms 2024-04-25T01:57:50.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:49 smithi031 ceph-mon[29204]: pgmap v103: 97 pgs: 97 active+clean; 42 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:57:50.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 58389 2.84 MB/sec warmup 117 sec latency 19.333 ms 2024-04-25T01:57:50.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:49 smithi040 ceph-mon[32418]: pgmap v103: 97 pgs: 97 active+clean; 42 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:57:50.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:49 smithi134 ceph-mon[33098]: pgmap v103: 97 pgs: 97 active+clean; 42 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:57:51.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 59077 2.83 MB/sec warmup 118 sec latency 11.820 ms 2024-04-25T01:57:52.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:51 smithi031 ceph-mon[29204]: pgmap v104: 97 pgs: 97 active+clean; 47 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-04-25T01:57:52.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 59547 2.84 MB/sec warmup 119 sec latency 19.747 ms 2024-04-25T01:57:52.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:51 smithi040 ceph-mon[32418]: pgmap v104: 97 pgs: 97 active+clean; 47 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-04-25T01:57:52.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:51 smithi134 ceph-mon[33098]: pgmap v104: 97 pgs: 97 active+clean; 47 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-04-25T01:57:54.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:53 smithi031 ceph-mon[29204]: pgmap v105: 97 pgs: 97 active+clean; 47 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:57:54.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 60579 4.40 MB/sec execute 1 sec latency 19.540 ms 2024-04-25T01:57:54.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:53 smithi040 ceph-mon[32418]: pgmap v105: 97 pgs: 97 active+clean; 47 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:57:54.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:53 smithi134 ceph-mon[33098]: pgmap v105: 97 pgs: 97 active+clean; 47 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:57:55.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 60839 2.36 MB/sec execute 2 sec latency 18.186 ms 2024-04-25T01:57:56.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:55 smithi031 ceph-mon[29204]: pgmap v106: 97 pgs: 97 active+clean; 48 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.8 MiB/s wr, 189 op/s 2024-04-25T01:57:56.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 61266 1.84 MB/sec execute 3 sec latency 19.550 ms 2024-04-25T01:57:56.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:55 smithi040 ceph-mon[32418]: pgmap v106: 97 pgs: 97 active+clean; 48 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.8 MiB/s wr, 189 op/s 2024-04-25T01:57:56.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:55 smithi134 ceph-mon[33098]: pgmap v106: 97 pgs: 97 active+clean; 48 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.8 MiB/s wr, 189 op/s 2024-04-25T01:57:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.814+0000 7fd4a8aba640 1 -- 172.21.15.31:0/83049281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4a41029d0 msgr2=0x7fd4a4102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.814+0000 7fd4a8aba640 1 --2- 172.21.15.31:0/83049281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4a41029d0 0x7fd4a4102db0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd494009900 tx=0x7fd49402a510 comp rx=0 tx=0).stop 2024-04-25T01:57:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.814+0000 7fd4a8aba640 1 -- 172.21.15.31:0/83049281 shutdown_connections 2024-04-25T01:57:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.814+0000 7fd4a8aba640 1 --2- 172.21.15.31:0/83049281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a410bd10 0x7fd4a410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.814+0000 7fd4a8aba640 1 --2- 172.21.15.31:0/83049281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4a41032f0 0x7fd4a410b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.814+0000 7fd4a8aba640 1 --2- 172.21.15.31:0/83049281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4a41029d0 0x7fd4a4102db0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.814+0000 7fd4a8aba640 1 -- 172.21.15.31:0/83049281 >> 172.21.15.31:0/83049281 conn(0x7fd4a40777c0 msgr2=0x7fd4a4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.814+0000 7fd4a8aba640 1 -- 172.21.15.31:0/83049281 shutdown_connections 2024-04-25T01:57:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.814+0000 7fd4a8aba640 1 -- 172.21.15.31:0/83049281 wait complete. 2024-04-25T01:57:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a8aba640 1 Processor -- start 2024-04-25T01:57:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a8aba640 1 -- start start 2024-04-25T01:57:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a8aba640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a41029d0 0x7fd4a4100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a8aba640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4a41032f0 0x7fd4a4100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a8aba640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4a410bd10 0x7fd4a41010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a8aba640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41119c0 con 0x7fd4a410bd10 2024-04-25T01:57:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a8aba640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a4111840 con 0x7fd4a41029d0 2024-04-25T01:57:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a8aba640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a4111b40 con 0x7fd4a41032f0 2024-04-25T01:57:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4a410bd10 0x7fd4a41010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4a410bd10 0x7fd4a41010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48204/0 (socket says 172.21.15.31:48204) 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a2d9d640 1 -- 172.21.15.31:0/321255708 learned_addr learned my addr 172.21.15.31:0/321255708 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a41029d0 0x7fd4a4100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4a41032f0 0x7fd4a4100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a2d9d640 1 -- 172.21.15.31:0/321255708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4a41032f0 msgr2=0x7fd4a4100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a2d9d640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4a41032f0 0x7fd4a4100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a2d9d640 1 -- 172.21.15.31:0/321255708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a41029d0 msgr2=0x7fd4a4100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a2d9d640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a41029d0 0x7fd4a4100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.815+0000 7fd4a2d9d640 1 -- 172.21.15.31:0/321255708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a41a6360 con 0x7fd4a410bd10 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.816+0000 7fd4a2d9d640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4a410bd10 0x7fd4a41010a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00eb10 tx=0x7fd49c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.816+0000 7fd49b7fe640 1 -- 172.21.15.31:0/321255708 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c00ce20 con 0x7fd4a410bd10 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.816+0000 7fd49b7fe640 1 -- 172.21.15.31:0/321255708 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd49c005020 con 0x7fd4a410bd10 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.816+0000 7fd4a8aba640 1 -- 172.21.15.31:0/321255708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a41a6650 con 0x7fd4a410bd10 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.816+0000 7fd49b7fe640 1 -- 172.21.15.31:0/321255708 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c010690 con 0x7fd4a410bd10 2024-04-25T01:57:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.816+0000 7fd4a8aba640 1 -- 172.21.15.31:0/321255708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a41a6b50 con 0x7fd4a410bd10 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.818+0000 7fd49b7fe640 1 -- 172.21.15.31:0/321255708 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd49c0051d0 con 0x7fd4a410bd10 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.819+0000 7fd4ab7c8740 1 Processor -- start 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.819+0000 7fd4ab7c8740 1 -- start start 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.819+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324240 0x555d47324620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.819+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d4732ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.819+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d4732f230 0x555d47331620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.819+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555d47288cb0 con 0x555d47324b60 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.819+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d472315b0 con 0x555d47324240 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.819+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555d47284c70 con 0x555d4732f230 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.819+0000 7fd49b7fe640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4900a2da0 0x7fd4900a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324240 0x555d47324620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d4732ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324240 0x555d47324620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60880/0 (socket says 172.21.15.31:60880) 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/554008505 learned_addr learned my addr 172.21.15.31:0/554008505 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a2d9d640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4900a2da0 0x7fd4900a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/554008505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d4732f230 0x555d47331620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd49b7fe640 1 -- 172.21.15.31:0/321255708 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd49c0cf930 con 0x7fd4a410bd10 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/554008505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d4732f230 msgr2=0x555d47331620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/554008505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d4732f230 0x555d47331620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/554008505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 msgr2=0x555d4732ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/554008505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d4732ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/554008505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d46f9d210 con 0x555d47324240 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a259c640 1 --2- 172.21.15.31:0/554008505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d4732ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a2d9d640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4900a2da0 0x7fd4900a5260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0cd180 tx=0x7fd49c0bc040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/554008505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324240 0x555d47324620 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0117e0 tx=0x7fd48c00a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef8c3fe0833e203f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd498ff9640 1 -- 172.21.15.31:0/554008505 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c011e20 con 0x555d47324240 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd498ff9640 1 -- 172.21.15.31:0/554008505 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd48c00fe80 con 0x555d47324240 2024-04-25T01:57:56.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.820+0000 7fd498ff9640 1 -- 172.21.15.31:0/554008505 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c004a00 con 0x555d47324240 2024-04-25T01:57:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.821+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/554008505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324240 msgr2=0x555d47324620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.821+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/554008505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324240 0x555d47324620 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0117e0 tx=0x7fd48c00a760 comp rx=0 tx=0).stop 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.821+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/554008505 shutdown_connections 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.821+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/554008505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d4732f230 0x555d47331620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.821+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/554008505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d4732ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.821+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/554008505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324240 0x555d47324620 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.821+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/554008505 >> 172.21.15.31:0/554008505 conn(0x555d47320560 msgr2=0x555d47320940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.821+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/554008505 shutdown_connections 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.821+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/554008505 wait complete. 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4ab7c8740 1 -- start start 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4ab7c8740 1 Processor -- start 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d4732c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324b60 0x555d4732c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d4732f230 0x555d4732cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555d47332020 con 0x555d4732f230 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d47244b50 con 0x555d47324b60 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555d473321a0 con 0x555d47324240 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4ab7c8740 10 client.0 _ll_register_callbacks cb 0x555d4727fca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4a1d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d4732f230 0x555d4732cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4a259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324b60 0x555d4732c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4a1d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d4732f230 0x555d4732cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48222/0 (socket says 172.21.15.31:48222) 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4a259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324b60 0x555d4732c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60896/0 (socket says 172.21.15.31:60896) 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/207112971 learned_addr learned my addr 172.21.15.31:0/207112971 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.822+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d4732c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/207112971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 msgr2=0x555d4732c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d4732c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/207112971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324b60 msgr2=0x555d4732c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324b60 0x555d4732c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/207112971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d47069620 con 0x555d4732f230 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd4a259c640 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324b60 0x555d4732c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d4732f230 0x555d4732cca0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0056b0 tx=0x7fd48c01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96cd714b385ec88e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd489ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd489ffb640 1 -- 172.21.15.31:0/207112971 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c011e20 con 0x555d4732f230 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd489ffb640 1 -- 172.21.15.31:0/207112971 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd48c00fbe0 con 0x555d4732f230 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/207112971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555d472961f0 con 0x555d4732f230 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd489ffb640 1 -- 172.21.15.31:0/207112971 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd48c035050 con 0x555d4732f230 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd4ab7c8740 10 client.16419 fetch_fsmap learned FSMap version 30 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/207112971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555d4733e270 con 0x555d4732f230 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd489ffb640 1 -- 172.21.15.31:0/207112971 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x555d472961f0 con 0x555d4732f230 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.823+0000 7fd489ffb640 1 -- 172.21.15.31:0/207112971 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fd48c01e600 con 0x555d4732f230 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.824+0000 7fd4ab7c8740 10 client.16419 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.824+0000 7fd4ab7c8740 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.824+0000 7fd4ab7c8740 20 client.16419 populate_metadata read hostname 'smithi031' 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.824+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d4733f5c0 0x555d473419b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.824+0000 7fd4ab7c8740 4 client.16419 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.824+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/207112971 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d47290540 con 0x555d4733f5c0 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.824+0000 7fd4a259c640 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d4733f5c0 0x555d473419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.824+0000 7fd4a259c640 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d4733f5c0 0x555d473419b0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7097f5e4cec2feef server_cookie=3320b67d84d7b085 in_seq=0 out_seq=0 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.824+0000 7fd489ffb640 10 client.16419 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd489ffb640 1 -- 172.21.15.31:0/207112971 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d47290540 con 0x555d4733f5c0 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd489ffb640 10 client.16419 handle_command_reply: tid=0 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 1 client.16419 shutdown 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/207112971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d4732f230 msgr2=0x555d4732cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d4732f230 0x555d4732cca0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0056b0 tx=0x7fd48c01a040 comp rx=0 tx=0).stop 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/207112971 shutdown_connections 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d4733f5c0 0x555d473419b0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d4732f230 0x555d4732cca0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d47324b60 0x555d4732c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/207112971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d4732c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd489ffb640 0 client.16419 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/207112971 >> 172.21.15.31:0/207112971 conn(0x555d47320560 msgr2=0x555d47326360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/207112971 shutdown_connections 2024-04-25T01:57:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/207112971 wait complete. 2024-04-25T01:57:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.825+0000 7fd4ab7c8740 20 client.16419 trim_cache size 0 max 16384 2024-04-25T01:57:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.842+0000 7fd4ab7c8740 1 Processor -- start 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4ab7c8740 1 -- start start 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d4733aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47320940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d4732f230 0x555d473471c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41af3e0 con 0x555d47324b60 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41b0fe0 con 0x555d4732f230 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41afab0 con 0x555d47324240 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4a259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47320940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4a259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47320940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48224/0 (socket says 172.21.15.31:48224) 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4a259c640 1 -- 172.21.15.31:0/1346842374 learned_addr learned my addr 172.21.15.31:0/1346842374 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/1346842374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d4732f230 0x555d473471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/1346842374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d4733aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4a259c640 1 -- 172.21.15.31:0/1346842374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 msgr2=0x555d4733aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4a259c640 1 --2- 172.21.15.31:0/1346842374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d4733aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4a259c640 1 -- 172.21.15.31:0/1346842374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d4732f230 msgr2=0x555d473471c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4a259c640 1 --2- 172.21.15.31:0/1346842374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d4732f230 0x555d473471c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4a259c640 1 -- 172.21.15.31:0/1346842374 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d4732b1f0 con 0x555d47324b60 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.843+0000 7fd4a259c640 1 --2- 172.21.15.31:0/1346842374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47320940 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd494009900 tx=0x7fd494009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c3385366410ba04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd48bfff640 1 -- 172.21.15.31:0/1346842374 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49404b070 con 0x555d47324b60 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd48bfff640 1 -- 172.21.15.31:0/1346842374 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd494046dc0 con 0x555d47324b60 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd48bfff640 1 -- 172.21.15.31:0/1346842374 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd494035040 con 0x555d47324b60 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/1346842374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 msgr2=0x555d47320940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/1346842374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47320940 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd494009900 tx=0x7fd494009610 comp rx=0 tx=0).stop 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/1346842374 shutdown_connections 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/1346842374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d4732f230 0x555d473471c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/1346842374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47320940 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/1346842374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d4733aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/1346842374 >> 172.21.15.31:0/1346842374 conn(0x555d47320560 msgr2=0x555d4731f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/1346842374 shutdown_connections 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/1346842374 wait complete. 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.844+0000 7fd4ab7c8740 1 -- start start 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4ab7c8740 1 Processor -- start 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d473375e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47328ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4ab7c8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d4732f230 0x555d47328ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41b0850 con 0x555d47324b60 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41b0180 con 0x555d4732f230 2024-04-25T01:57:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4ab7c8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a41ad3c0 con 0x555d47324240 2024-04-25T01:57:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4ab7c8740 10 client.0 _ll_register_callbacks cb 0x7fd494009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a1d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d473375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47328ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a1d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d473375e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:48500/0 (socket says 172.21.15.31:48500) 2024-04-25T01:57:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/898854003 learned_addr learned my addr 172.21.15.31:0/898854003 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:57:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a259c640 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47328ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48226/0 (socket says 172.21.15.31:48226) 2024-04-25T01:57:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d4732f230 0x555d47328ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/898854003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d4732f230 msgr2=0x555d47328ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d4732f230 0x555d47328ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/898854003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 msgr2=0x555d47328ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47328ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a1d9b640 1 -- 172.21.15.31:0/898854003 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d47329560 con 0x555d47324240 2024-04-25T01:57:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.845+0000 7fd4a259c640 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47328ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd4a1d9b640 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d473375e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0056b0 tx=0x7fd48c0115b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fd748ac7e6bfbbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd489ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:57:56.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd489ffb640 1 -- 172.21.15.31:0/898854003 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c00fbe0 con 0x555d47324240 2024-04-25T01:57:56.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/898854003 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd494046dc0 con 0x555d47324240 2024-04-25T01:57:56.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd489ffb640 1 -- 172.21.15.31:0/898854003 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7fd48c005330 con 0x555d47324240 2024-04-25T01:57:56.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd489ffb640 1 -- 172.21.15.31:0/898854003 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd48c01e680 con 0x555d47324240 2024-04-25T01:57:56.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd4ab7c8740 10 client.25867 fetch_fsmap learned FSMap version 30 2024-04-25T01:57:56.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/898854003 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555d473a57e0 con 0x555d47324240 2024-04-25T01:57:56.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd489ffb640 1 -- 172.21.15.31:0/898854003 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd494046dc0 con 0x555d47324240 2024-04-25T01:57:56.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd489ffb640 1 -- 172.21.15.31:0/898854003 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fd48c0049d0 con 0x555d47324240 2024-04-25T01:57:56.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd4ab7c8740 10 client.25867 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:57:56.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd4ab7c8740 10 client.25867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:56.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd4ab7c8740 20 client.25867 populate_metadata read hostname 'smithi031' 2024-04-25T01:57:56.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d473a5f40 0x555d473a8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd4ab7c8740 4 client.25867 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:57:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.846+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/898854003 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555d4720e8e0 con 0x555d473a5f40 2024-04-25T01:57:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.847+0000 7fd4a259c640 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d473a5f40 0x555d473a8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.847+0000 7fd489ffb640 10 client.25867 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:56.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.847+0000 7fd4a259c640 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d473a5f40 0x555d473a8360 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5b48aeb0949c437 server_cookie=1c6715e87602ad7e in_seq=0 out_seq=0 2024-04-25T01:57:56.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.867+0000 7fd489ffb640 1 -- 172.21.15.31:0/898854003 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555d4720e8e0 con 0x555d473a5f40 2024-04-25T01:57:56.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd489ffb640 10 client.25867 handle_command_reply: tid=0 2024-04-25T01:57:56.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 1 client.25867 shutdown 2024-04-25T01:57:56.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/898854003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 msgr2=0x555d473375e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:56.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d473375e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0056b0 tx=0x7fd48c0115b0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/898854003 shutdown_connections 2024-04-25T01:57:56.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555d473a5f40 0x555d473a8360 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d4732f230 0x555d47328ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d47324b60 0x555d47328ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 1 --2- 172.21.15.31:0/898854003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555d47324240 0x555d473375e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd489ffb640 0 client.25867 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:57:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/898854003 >> 172.21.15.31:0/898854003 conn(0x555d47320560 msgr2=0x555d4731f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/898854003 shutdown_connections 2024-04-25T01:57:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 1 -- 172.21.15.31:0/898854003 wait complete. 2024-04-25T01:57:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4ab7c8740 20 client.25867 trim_cache size 0 max 16384 2024-04-25T01:57:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:57:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:57:56.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:57:56.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:57:56.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4a8aba640 1 -- 172.21.15.31:0/321255708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4900a2da0 msgr2=0x7fd4900a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:56.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4a8aba640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4900a2da0 0x7fd4900a5260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0cd180 tx=0x7fd49c0bc040 comp rx=0 tx=0).stop 2024-04-25T01:57:56.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4a8aba640 1 -- 172.21.15.31:0/321255708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4a410bd10 msgr2=0x7fd4a41010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:56.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.868+0000 7fd4a8aba640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4a410bd10 0x7fd4a41010a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00eb10 tx=0x7fd49c00ef10 comp rx=0 tx=0).stop 2024-04-25T01:57:56.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.869+0000 7fd4a8aba640 1 -- 172.21.15.31:0/321255708 shutdown_connections 2024-04-25T01:57:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.869+0000 7fd4a8aba640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4900a2da0 0x7fd4900a5260 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.869+0000 7fd4a8aba640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4a410bd10 0x7fd4a41010a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.869+0000 7fd4a8aba640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4a41032f0 0x7fd4a4100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.869+0000 7fd4a8aba640 1 --2- 172.21.15.31:0/321255708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4a41029d0 0x7fd4a4100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.869+0000 7fd4a8aba640 1 -- 172.21.15.31:0/321255708 >> 172.21.15.31:0/321255708 conn(0x7fd4a40777c0 msgr2=0x7fd4a40fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.869+0000 7fd4a8aba640 1 -- 172.21.15.31:0/321255708 shutdown_connections 2024-04-25T01:57:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:57:56.869+0000 7fd4a8aba640 1 -- 172.21.15.31:0/321255708 wait complete. 2024-04-25T01:57:57.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 61781 2.44 MB/sec execute 4 sec latency 18.477 ms 2024-04-25T01:57:58.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:57 smithi031 ceph-mon[29204]: pgmap v107: 97 pgs: 97 active+clean; 48 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:57:58.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 62386 2.08 MB/sec execute 5 sec latency 18.659 ms 2024-04-25T01:57:58.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:57 smithi040 ceph-mon[32418]: pgmap v107: 97 pgs: 97 active+clean; 48 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:57:58.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:57 smithi134 ceph-mon[33098]: pgmap v107: 97 pgs: 97 active+clean; 48 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:57:59.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 62905 2.12 MB/sec execute 6 sec latency 17.919 ms 2024-04-25T01:58:00.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:57:59 smithi031 ceph-mon[29204]: pgmap v108: 97 pgs: 97 active+clean; 48 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:58:00.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 63617 2.73 MB/sec execute 7 sec latency 18.519 ms 2024-04-25T01:58:00.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:57:59 smithi040 ceph-mon[32418]: pgmap v108: 97 pgs: 97 active+clean; 48 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:58:00.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:57:59 smithi134 ceph-mon[33098]: pgmap v108: 97 pgs: 97 active+clean; 48 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:58:01.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 64010 2.81 MB/sec execute 8 sec latency 23.132 ms 2024-04-25T01:58:02.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:01 smithi031 ceph-mon[29204]: pgmap v109: 97 pgs: 97 active+clean; 44 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-04-25T01:58:02.303 INFO:tasks.workunit.client.0.smithi031.stdout: 1 64292 2.65 MB/sec execute 9 sec latency 14.841 ms 2024-04-25T01:58:02.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:01 smithi040 ceph-mon[32418]: pgmap v109: 97 pgs: 97 active+clean; 44 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-04-25T01:58:02.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:01 smithi134 ceph-mon[33098]: pgmap v109: 97 pgs: 97 active+clean; 44 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-04-25T01:58:03.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 64604 2.42 MB/sec execute 10 sec latency 45.785 ms 2024-04-25T01:58:04.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:03 smithi031 ceph-mon[29204]: pgmap v110: 97 pgs: 97 active+clean; 44 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:58:04.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 65091 2.36 MB/sec execute 11 sec latency 17.758 ms 2024-04-25T01:58:04.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:03 smithi040 ceph-mon[32418]: pgmap v110: 97 pgs: 97 active+clean; 44 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:58:04.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:03 smithi134 ceph-mon[33098]: pgmap v110: 97 pgs: 97 active+clean; 44 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:58:05.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 65716 2.44 MB/sec execute 12 sec latency 14.247 ms 2024-04-25T01:58:06.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:05 smithi031 ceph-mon[29204]: pgmap v111: 97 pgs: 97 active+clean; 49 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-04-25T01:58:06.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 66266 2.37 MB/sec execute 13 sec latency 18.359 ms 2024-04-25T01:58:06.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:05 smithi040 ceph-mon[32418]: pgmap v111: 97 pgs: 97 active+clean; 49 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-04-25T01:58:06.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:05 smithi134 ceph-mon[33098]: pgmap v111: 97 pgs: 97 active+clean; 49 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-04-25T01:58:07.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 66864 2.60 MB/sec execute 14 sec latency 12.450 ms 2024-04-25T01:58:08.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:07 smithi031 ceph-mon[29204]: pgmap v112: 97 pgs: 97 active+clean; 49 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-25T01:58:08.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:07 smithi134 ceph-mon[33098]: pgmap v112: 97 pgs: 97 active+clean; 49 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-25T01:58:08.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 67386 2.75 MB/sec execute 15 sec latency 18.608 ms 2024-04-25T01:58:08.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:07 smithi040 ceph-mon[32418]: pgmap v112: 97 pgs: 97 active+clean; 49 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-25T01:58:09.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 67758 2.68 MB/sec execute 16 sec latency 14.172 ms 2024-04-25T01:58:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.968+0000 7f74b5d70640 1 -- 172.21.15.31:0/567367869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0101730 msgr2=0x7f74b0101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.968+0000 7f74b5d70640 1 --2- 172.21.15.31:0/567367869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0101730 0x7f74b0101bb0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009930 tx=0x7f74a402a420 comp rx=0 tx=0).stop 2024-04-25T01:58:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.968+0000 7f74b5d70640 1 -- 172.21.15.31:0/567367869 shutdown_connections 2024-04-25T01:58:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.968+0000 7f74b5d70640 1 --2- 172.21.15.31:0/567367869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74b01020f0 0x7f74b010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.968+0000 7f74b5d70640 1 --2- 172.21.15.31:0/567367869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0101730 0x7f74b0101bb0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.968+0000 7f74b5d70640 1 --2- 172.21.15.31:0/567367869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74b0107740 0x7f74b0107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.968+0000 7f74b5d70640 1 -- 172.21.15.31:0/567367869 >> 172.21.15.31:0/567367869 conn(0x7f74b00fd200 msgr2=0x7f74b00ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.968+0000 7f74b5d70640 1 -- 172.21.15.31:0/567367869 shutdown_connections 2024-04-25T01:58:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.968+0000 7f74b5d70640 1 -- 172.21.15.31:0/567367869 wait complete. 2024-04-25T01:58:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.968+0000 7f74b5d70640 1 Processor -- start 2024-04-25T01:58:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74b5d70640 1 -- start start 2024-04-25T01:58:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74b5d70640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0101730 0x7f74b019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74b5d70640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74b01020f0 0x7f74b019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74b5d70640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74b0107740 0x7f74b019e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74b5d70640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0111a00 con 0x7f74b0101730 2024-04-25T01:58:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74b5d70640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0111880 con 0x7f74b0107740 2024-04-25T01:58:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74b5d70640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0111b80 con 0x7f74b01020f0 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0101730 0x7f74b019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0101730 0x7f74b019a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51746/0 (socket says 172.21.15.31:51746) 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74affff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74b0107740 0x7f74b019e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74af7fe640 1 -- 172.21.15.31:0/1365767852 learned_addr learned my addr 172.21.15.31:0/1365767852 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74aeffd640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74b01020f0 0x7f74b019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74af7fe640 1 -- 172.21.15.31:0/1365767852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74b01020f0 msgr2=0x7f74b019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74af7fe640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74b01020f0 0x7f74b019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74af7fe640 1 -- 172.21.15.31:0/1365767852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74b0107740 msgr2=0x7f74b019e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74af7fe640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74b0107740 0x7f74b019e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74af7fe640 1 -- 172.21.15.31:0/1365767852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b019f110 con 0x7f74b0101730 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.969+0000 7f74affff640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74b0107740 0x7f74b019e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.970+0000 7f74af7fe640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0101730 0x7f74b019a170 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f74a000d950 tx=0x7f74a000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.970+0000 7f74acff9640 1 -- 172.21.15.31:0/1365767852 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0017070 con 0x7f74b0101730 2024-04-25T01:58:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.970+0000 7f74acff9640 1 -- 172.21.15.31:0/1365767852 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74a000de80 con 0x7f74b0101730 2024-04-25T01:58:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.970+0000 7f74b5d70640 1 -- 172.21.15.31:0/1365767852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b019f400 con 0x7f74b0101730 2024-04-25T01:58:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.970+0000 7f74acff9640 1 -- 172.21.15.31:0/1365767852 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a000f620 con 0x7f74b0101730 2024-04-25T01:58:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.970+0000 7f74b5d70640 1 -- 172.21.15.31:0/1365767852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b01a6c90 con 0x7f74b0101730 2024-04-25T01:58:09.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74b8722740 1 Processor -- start 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74b8722740 1 -- start start 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74b8722740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 0x556f8df1a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74b8722740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1ab60 0x556f8df24cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74b8722740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df27620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74b8722740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556f8de7ecb0 con 0x556f8df25230 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74b8722740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556f8de275b0 con 0x556f8df1a240 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74b8722740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556f8de7ac70 con 0x556f8df1ab60 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74aeffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df27620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74affff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 0x556f8df1a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74af7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1ab60 0x556f8df24cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74affff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 0x556f8df1a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46868/0 (socket says 172.21.15.31:46868) 2024-04-25T01:58:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74affff640 1 -- 172.21.15.31:0/2905345208 learned_addr learned my addr 172.21.15.31:0/2905345208 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.973+0000 7f74af7fe640 1 -- 172.21.15.31:0/2905345208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 msgr2=0x556f8df1a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.974+0000 7f74af7fe640 1 --2- 172.21.15.31:0/2905345208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 0x556f8df1a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.974+0000 7f74af7fe640 1 -- 172.21.15.31:0/2905345208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 msgr2=0x556f8df27620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.974+0000 7f74af7fe640 1 --2- 172.21.15.31:0/2905345208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df27620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.974+0000 7f74af7fe640 1 -- 172.21.15.31:0/2905345208 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f8db93210 con 0x556f8df1ab60 2024-04-25T01:58:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.974+0000 7f74affff640 1 --2- 172.21.15.31:0/2905345208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 0x556f8df1a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.974+0000 7f74aeffd640 1 --2- 172.21.15.31:0/2905345208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df27620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.974+0000 7f74af7fe640 1 --2- 172.21.15.31:0/2905345208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1ab60 0x556f8df24cf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f74a0021000 tx=0x7f74a0010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c494f62d8d0bcbc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.974+0000 7f7495ffb640 1 -- 172.21.15.31:0/2905345208 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0020060 con 0x556f8df1ab60 2024-04-25T01:58:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.974+0000 7f7495ffb640 1 -- 172.21.15.31:0/2905345208 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74a0029070 con 0x556f8df1ab60 2024-04-25T01:58:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.974+0000 7f7495ffb640 1 -- 172.21.15.31:0/2905345208 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0004e10 con 0x556f8df1ab60 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.975+0000 7f74b8722740 1 -- 172.21.15.31:0/2905345208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1ab60 msgr2=0x556f8df24cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.975+0000 7f74b8722740 1 --2- 172.21.15.31:0/2905345208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1ab60 0x556f8df24cf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f74a0021000 tx=0x7f74a0010830 comp rx=0 tx=0).stop 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.975+0000 7f74b8722740 1 -- 172.21.15.31:0/2905345208 shutdown_connections 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.975+0000 7f74b8722740 1 --2- 172.21.15.31:0/2905345208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df27620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.975+0000 7f74b8722740 1 --2- 172.21.15.31:0/2905345208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1ab60 0x556f8df24cf0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f74a0021000 tx=0x7f74a0010830 comp rx=0 tx=0).stop 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.975+0000 7f74b8722740 1 --2- 172.21.15.31:0/2905345208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 0x556f8df1a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.975+0000 7f74b8722740 1 -- 172.21.15.31:0/2905345208 >> 172.21.15.31:0/2905345208 conn(0x556f8df16560 msgr2=0x556f8df16940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.975+0000 7f74acff9640 1 -- 172.21.15.31:0/1365767852 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f74a0025450 con 0x7f74b0101730 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.975+0000 7f74b8722740 1 -- 172.21.15.31:0/2905345208 shutdown_connections 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.975+0000 7f74b8722740 1 -- 172.21.15.31:0/2905345208 wait complete. 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74b8722740 1 -- start start 2024-04-25T01:58:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74b8722740 1 Processor -- start 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74acff9640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f74980a2d30 0x7f74980a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74b8722740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1a240 0x556f8df22380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74b8722740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df228c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74b8722740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 0x556f8df31430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74b8722740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556f8df27fc0 con 0x556f8df25230 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74b8722740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556f8de3ab50 con 0x556f8df22e00 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74b8722740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556f8df28140 con 0x556f8df1a240 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74affff640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f74980a2d30 0x7f74980a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74b8722740 10 client.0 _ll_register_callbacks cb 0x556f8de75ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74acff9640 1 -- 172.21.15.31:0/1365767852 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f74a00d2010 con 0x7f74b0101730 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74aeffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df228c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74affff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 0x556f8df31430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74aeffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df228c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51758/0 (socket says 172.21.15.31:51758) 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74affff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 0x556f8df31430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46870/0 (socket says 172.21.15.31:46870) 2024-04-25T01:58:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.976+0000 7f74af7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1a240 0x556f8df22380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f74aeffd640 1 -- 172.21.15.31:0/4056490688 learned_addr learned my addr 172.21.15.31:0/4056490688 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f74af7fe640 1 -- 172.21.15.31:0/4056490688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 msgr2=0x556f8df31430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f74af7fe640 1 --2- 172.21.15.31:0/4056490688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 0x556f8df31430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f74af7fe640 1 -- 172.21.15.31:0/4056490688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 msgr2=0x556f8df228c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f74af7fe640 1 --2- 172.21.15.31:0/4056490688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df228c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f74af7fe640 1 -- 172.21.15.31:0/4056490688 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f8dc5f620 con 0x556f8df1a240 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f74affff640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f74980a2d30 0x7f74980a51f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f74a800f5b0 tx=0x7f74a8014870 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f74af7fe640 1 --2- 172.21.15.31:0/4056490688 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1a240 0x556f8df22380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f74a00cf8c0 tx=0x7f74a009e5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2441754a79f1af86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f748effd640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f748effd640 1 -- 172.21.15.31:0/4056490688 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0020070 con 0x556f8df1a240 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f748effd640 1 -- 172.21.15.31:0/4056490688 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f74a00046c0 con 0x556f8df1a240 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f74b8722740 1 -- 172.21.15.31:0/4056490688 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74a0029070 con 0x556f8df1a240 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f748effd640 1 -- 172.21.15.31:0/4056490688 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f74a00b6070 con 0x556f8df1a240 2024-04-25T01:58:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f74b8722740 10 client.25879 fetch_fsmap learned FSMap version 30 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.977+0000 7f74b8722740 1 -- 172.21.15.31:0/4056490688 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f8df27fc0 con 0x556f8df1a240 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.978+0000 7f748effd640 1 -- 172.21.15.31:0/4056490688 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74a0029070 con 0x556f8df1a240 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.978+0000 7f748effd640 1 -- 172.21.15.31:0/4056490688 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f74a00cfa30 con 0x556f8df1a240 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.978+0000 7f74b8722740 10 client.25879 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.978+0000 7f74b8722740 10 client.25879 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.978+0000 7f74b8722740 20 client.25879 populate_metadata read hostname 'smithi031' 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.978+0000 7f74b8722740 1 --2- 172.21.15.31:0/4056490688 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f8df35980 0x556f8df37d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.978+0000 7f74b8722740 4 client.25879 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.978+0000 7f74b8722740 1 -- 172.21.15.31:0/4056490688 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f8de8c1f0 con 0x556f8df35980 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.978+0000 7f74aeffd640 1 --2- 172.21.15.31:0/4056490688 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f8df35980 0x556f8df37d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.978+0000 7f748effd640 10 client.25879 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.978+0000 7f74aeffd640 1 --2- 172.21.15.31:0/4056490688 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f8df35980 0x556f8df37d70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=881915302d1a0705 server_cookie=41bd7b2276c947db in_seq=0 out_seq=0 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.979+0000 7f748effd640 1 -- 172.21.15.31:0/4056490688 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556f8de8c1f0 con 0x556f8df35980 2024-04-25T01:58:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.979+0000 7f748effd640 10 client.25879 handle_command_reply: tid=0 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.979+0000 7f74b8722740 1 client.25879 shutdown 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.979+0000 7f74b8722740 1 -- 172.21.15.31:0/4056490688 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1a240 msgr2=0x556f8df22380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.979+0000 7f74b8722740 1 --2- 172.21.15.31:0/4056490688 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1a240 0x556f8df22380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f74a00cf8c0 tx=0x7f74a009e5f0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.980+0000 7f74b8722740 1 -- 172.21.15.31:0/4056490688 shutdown_connections 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.980+0000 7f74b8722740 1 --2- 172.21.15.31:0/4056490688 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f8df35980 0x556f8df37d70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.980+0000 7f74b8722740 1 --2- 172.21.15.31:0/4056490688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 0x556f8df31430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.980+0000 7f748effd640 0 client.25879 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.980+0000 7f74b8722740 1 --2- 172.21.15.31:0/4056490688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df228c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.980+0000 7f74b8722740 1 --2- 172.21.15.31:0/4056490688 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1a240 0x556f8df22380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.980+0000 7f74b8722740 1 -- 172.21.15.31:0/4056490688 >> 172.21.15.31:0/4056490688 conn(0x556f8df16560 msgr2=0x556f8df1c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.980+0000 7f74b8722740 1 -- 172.21.15.31:0/4056490688 shutdown_connections 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.980+0000 7f74b8722740 1 -- 172.21.15.31:0/4056490688 wait complete. 2024-04-25T01:58:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.980+0000 7f74b8722740 20 client.25879 trim_cache size 0 max 16384 2024-04-25T01:58:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74b8722740 1 Processor -- start 2024-04-25T01:58:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74b8722740 1 -- start start 2024-04-25T01:58:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74b8722740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1a240 0x556f8df155c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74b8722740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 0x556f8df1c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74b8722740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df3d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74b8722740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01af510 con 0x556f8df25230 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74b8722740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01b1110 con 0x556f8df22e00 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74b8722740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01afbe0 con 0x556f8df1a240 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df3d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df3d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51770/0 (socket says 172.21.15.31:51770) 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74affff640 1 -- 172.21.15.31:0/2668354584 learned_addr learned my addr 172.21.15.31:0/2668354584 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74aeffd640 1 --2- 172.21.15.31:0/2668354584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 0x556f8df1c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74af7fe640 1 --2- 172.21.15.31:0/2668354584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1a240 0x556f8df155c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74affff640 1 -- 172.21.15.31:0/2668354584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1a240 msgr2=0x556f8df155c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74affff640 1 --2- 172.21.15.31:0/2668354584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1a240 0x556f8df155c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74affff640 1 -- 172.21.15.31:0/2668354584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 msgr2=0x556f8df1c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74affff640 1 --2- 172.21.15.31:0/2668354584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 0x556f8df1c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74affff640 1 -- 172.21.15.31:0/2668354584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f8df383a0 con 0x556f8df25230 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.998+0000 7f74aeffd640 1 --2- 172.21.15.31:0/2668354584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 0x556f8df1c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.999+0000 7f74affff640 1 --2- 172.21.15.31:0/2668354584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df3d5e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f74a8012210 tx=0x7f74a8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec014ffae45253c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.999+0000 7f74957fa640 1 -- 172.21.15.31:0/2668354584 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a8027070 con 0x556f8df25230 2024-04-25T01:58:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.999+0000 7f74957fa640 1 -- 172.21.15.31:0/2668354584 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74a800ee80 con 0x556f8df25230 2024-04-25T01:58:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:09.999+0000 7f74957fa640 1 -- 172.21.15.31:0/2668354584 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a8013b70 con 0x556f8df25230 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.000+0000 7f74b8722740 1 -- 172.21.15.31:0/2668354584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 msgr2=0x556f8df3d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.000+0000 7f74b8722740 1 --2- 172.21.15.31:0/2668354584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df3d5e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f74a8012210 tx=0x7f74a8020000 comp rx=0 tx=0).stop 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.000+0000 7f74b8722740 1 -- 172.21.15.31:0/2668354584 shutdown_connections 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.000+0000 7f74b8722740 1 --2- 172.21.15.31:0/2668354584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df25230 0x556f8df3d5e0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.000+0000 7f74b8722740 1 --2- 172.21.15.31:0/2668354584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df22e00 0x556f8df1c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.000+0000 7f74b8722740 1 --2- 172.21.15.31:0/2668354584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df1a240 0x556f8df155c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.000+0000 7f74b8722740 1 -- 172.21.15.31:0/2668354584 >> 172.21.15.31:0/2668354584 conn(0x556f8df16560 msgr2=0x556f8df1b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.000+0000 7f74b8722740 1 -- 172.21.15.31:0/2668354584 shutdown_connections 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.000+0000 7f74b8722740 1 -- 172.21.15.31:0/2668354584 wait complete. 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.000+0000 7f74b8722740 1 -- start start 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.000+0000 7f74b8722740 1 Processor -- start 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74b8722740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 0x556f8df1f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74b8722740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df22e00 0x556f8df1f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74b8722740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df25230 0x556f8df1faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74b8722740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01b0980 con 0x556f8df22e00 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74b8722740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01b02b0 con 0x556f8df1a240 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74b8722740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01ad4f0 con 0x556f8df25230 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74b8722740 10 client.0 _ll_register_callbacks cb 0x7f74a0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74aeffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df22e00 0x556f8df1f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74aeffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df22e00 0x556f8df1f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51772/0 (socket says 172.21.15.31:51772) 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74aeffd640 1 -- 172.21.15.31:0/1374994961 learned_addr learned my addr 172.21.15.31:0/1374994961 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74affff640 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df25230 0x556f8df1faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74af7fe640 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 0x556f8df1f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74aeffd640 1 -- 172.21.15.31:0/1374994961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df25230 msgr2=0x556f8df1faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74aeffd640 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df25230 0x556f8df1faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74aeffd640 1 -- 172.21.15.31:0/1374994961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 msgr2=0x556f8df1f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74aeffd640 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 0x556f8df1f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74aeffd640 1 -- 172.21.15.31:0/1374994961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f8df9ae50 con 0x556f8df22e00 2024-04-25T01:58:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f74aeffd640 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df22e00 0x556f8df1f560 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009900 tx=0x7f74a40043d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2caed1825fc1574 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.001+0000 7f748effd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f748effd640 1 -- 172.21.15.31:0/1374994961 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a4035040 con 0x556f8df22e00 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f748effd640 1 -- 172.21.15.31:0/1374994961 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f74a40314a0 con 0x556f8df22e00 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f74b8722740 1 -- 172.21.15.31:0/1374994961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556f8de86540 con 0x556f8df22e00 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f748effd640 1 -- 172.21.15.31:0/1374994961 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f74a4054050 con 0x556f8df22e00 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f74b8722740 10 client.16443 fetch_fsmap learned FSMap version 30 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f74b8722740 1 -- 172.21.15.31:0/1374994961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f8df9c1c0 con 0x556f8df22e00 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f748effd640 1 -- 172.21.15.31:0/1374994961 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x556f8de86540 con 0x556f8df22e00 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f748effd640 1 -- 172.21.15.31:0/1374994961 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f74a4045290 con 0x556f8df22e00 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f74b8722740 10 client.16443 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f74b8722740 10 client.16443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f74b8722740 20 client.16443 populate_metadata read hostname 'smithi031' 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f74b8722740 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f8df9ca00 0x556f8df9ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f74b8722740 4 client.16443 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:58:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f74b8722740 1 -- 172.21.15.31:0/1374994961 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556f8de048e0 con 0x556f8df9ca00 2024-04-25T01:58:10.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.002+0000 7f74af7fe640 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f8df9ca00 0x556f8df9ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:10.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.003+0000 7f74af7fe640 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f8df9ca00 0x556f8df9ee20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdf5ebb260b5dcda server_cookie=c2038affe5699c in_seq=0 out_seq=0 2024-04-25T01:58:10.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.003+0000 7f748effd640 10 client.16443 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f748effd640 1 -- 172.21.15.31:0/1374994961 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556f8de048e0 con 0x556f8df9ca00 2024-04-25T01:58:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f748effd640 10 client.16443 handle_command_reply: tid=0 2024-04-25T01:58:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 1 client.16443 shutdown 2024-04-25T01:58:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 1 -- 172.21.15.31:0/1374994961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df22e00 msgr2=0x556f8df1f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df22e00 0x556f8df1f560 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009900 tx=0x7f74a40043d0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 1 -- 172.21.15.31:0/1374994961 shutdown_connections 2024-04-25T01:58:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f8df9ca00 0x556f8df9ee20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f8df25230 0x556f8df1faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f748effd640 0 client.16443 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f8df22e00 0x556f8df1f560 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 1 --2- 172.21.15.31:0/1374994961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8df1a240 0x556f8df1f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 1 -- 172.21.15.31:0/1374994961 >> 172.21.15.31:0/1374994961 conn(0x556f8df16560 msgr2=0x556f8df15440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 1 -- 172.21.15.31:0/1374994961 shutdown_connections 2024-04-25T01:58:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 1 -- 172.21.15.31:0/1374994961 wait complete. 2024-04-25T01:58:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.025+0000 7f74b8722740 20 client.16443 trim_cache size 0 max 16384 2024-04-25T01:58:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:58:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:58:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:58:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:58:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 -- 172.21.15.31:0/1365767852 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f74980a2d30 msgr2=0x7f74980a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f74980a2d30 0x7f74980a51f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f74a800f5b0 tx=0x7f74a8014870 comp rx=0 tx=0).stop 2024-04-25T01:58:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 -- 172.21.15.31:0/1365767852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0101730 msgr2=0x7f74b019a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0101730 0x7f74b019a170 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f74a000d950 tx=0x7f74a000dd50 comp rx=0 tx=0).stop 2024-04-25T01:58:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 -- 172.21.15.31:0/1365767852 shutdown_connections 2024-04-25T01:58:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f74980a2d30 0x7f74980a51f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74b0107740 0x7f74b019e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74b01020f0 0x7f74b019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 --2- 172.21.15.31:0/1365767852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0101730 0x7f74b019a170 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 -- 172.21.15.31:0/1365767852 >> 172.21.15.31:0/1365767852 conn(0x7f74b00fd200 msgr2=0x7f74b00fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 -- 172.21.15.31:0/1365767852 shutdown_connections 2024-04-25T01:58:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:10.026+0000 7f74b5d70640 1 -- 172.21.15.31:0/1365767852 wait complete. 2024-04-25T01:58:10.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:09 smithi031 ceph-mon[29204]: pgmap v113: 97 pgs: 97 active+clean; 49 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-25T01:58:10.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 68023 2.53 MB/sec execute 17 sec latency 14.294 ms 2024-04-25T01:58:10.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:09 smithi040 ceph-mon[32418]: pgmap v113: 97 pgs: 97 active+clean; 49 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-25T01:58:10.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:09 smithi134 ceph-mon[33098]: pgmap v113: 97 pgs: 97 active+clean; 49 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-25T01:58:11.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 68513 2.50 MB/sec execute 18 sec latency 18.977 ms 2024-04-25T01:58:12.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:11 smithi031 ceph-mon[29204]: pgmap v114: 97 pgs: 97 active+clean; 53 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 1.8 MiB/s wr, 201 op/s 2024-04-25T01:58:12.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 69067 2.54 MB/sec execute 19 sec latency 19.028 ms 2024-04-25T01:58:12.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:11 smithi040 ceph-mon[32418]: pgmap v114: 97 pgs: 97 active+clean; 53 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 1.8 MiB/s wr, 201 op/s 2024-04-25T01:58:12.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:11 smithi134 ceph-mon[33098]: pgmap v114: 97 pgs: 97 active+clean; 53 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 1.8 MiB/s wr, 201 op/s 2024-04-25T01:58:13.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 69715 2.49 MB/sec execute 20 sec latency 19.444 ms 2024-04-25T01:58:14.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:13 smithi031 ceph-mon[29204]: pgmap v115: 97 pgs: 97 active+clean; 53 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-04-25T01:58:14.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 70289 2.58 MB/sec execute 21 sec latency 18.031 ms 2024-04-25T01:58:14.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:13 smithi040 ceph-mon[32418]: pgmap v115: 97 pgs: 97 active+clean; 53 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-04-25T01:58:14.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:13 smithi134 ceph-mon[33098]: pgmap v115: 97 pgs: 97 active+clean; 53 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-04-25T01:58:15.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 70761 2.61 MB/sec execute 22 sec latency 12.649 ms 2024-04-25T01:58:16.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:15 smithi031 ceph-mon[29204]: pgmap v116: 97 pgs: 97 active+clean; 54 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-04-25T01:58:16.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 71247 2.69 MB/sec execute 23 sec latency 12.887 ms 2024-04-25T01:58:16.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:15 smithi040 ceph-mon[32418]: pgmap v116: 97 pgs: 97 active+clean; 54 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-04-25T01:58:16.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:15 smithi134 ceph-mon[33098]: pgmap v116: 97 pgs: 97 active+clean; 54 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-04-25T01:58:17.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 71517 2.59 MB/sec execute 24 sec latency 18.058 ms 2024-04-25T01:58:18.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:17 smithi031 ceph-mon[29204]: pgmap v117: 97 pgs: 97 active+clean; 54 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T01:58:18.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 72004 2.56 MB/sec execute 25 sec latency 18.163 ms 2024-04-25T01:58:18.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:17 smithi040 ceph-mon[32418]: pgmap v117: 97 pgs: 97 active+clean; 54 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T01:58:18.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:17 smithi134 ceph-mon[33098]: pgmap v117: 97 pgs: 97 active+clean; 54 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T01:58:19.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 72539 2.59 MB/sec execute 26 sec latency 18.906 ms 2024-04-25T01:58:20.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:19 smithi031 ceph-mon[29204]: pgmap v118: 97 pgs: 97 active+clean; 54 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T01:58:20.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 73220 2.52 MB/sec execute 27 sec latency 17.458 ms 2024-04-25T01:58:20.411 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:19 smithi040 ceph-mon[32418]: pgmap v118: 97 pgs: 97 active+clean; 54 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T01:58:20.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:19 smithi134 ceph-mon[33098]: pgmap v118: 97 pgs: 97 active+clean; 54 MiB data, 705 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T01:58:21.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 73769 2.61 MB/sec execute 28 sec latency 18.466 ms 2024-04-25T01:58:22.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:21 smithi031 ceph-mon[29204]: pgmap v119: 97 pgs: 97 active+clean; 54 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 1.8 MiB/s wr, 192 op/s 2024-04-25T01:58:22.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 74313 2.64 MB/sec execute 29 sec latency 20.181 ms 2024-04-25T01:58:22.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:21 smithi040 ceph-mon[32418]: pgmap v119: 97 pgs: 97 active+clean; 54 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 1.8 MiB/s wr, 192 op/s 2024-04-25T01:58:22.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:21 smithi134 ceph-mon[33098]: pgmap v119: 97 pgs: 97 active+clean; 54 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 1.8 MiB/s wr, 192 op/s 2024-04-25T01:58:23.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.122+0000 7f2bbbe15640 1 -- 172.21.15.31:0/1590614660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb41020f0 msgr2=0x7f2bb410e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.122+0000 7f2bbbe15640 1 --2- 172.21.15.31:0/1590614660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb41020f0 0x7f2bb410e600 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f2bac009cd0 tx=0x7f2bac02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.122+0000 7f2bbbe15640 1 -- 172.21.15.31:0/1590614660 shutdown_connections 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.122+0000 7f2bbbe15640 1 --2- 172.21.15.31:0/1590614660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb41020f0 0x7f2bb410e600 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.122+0000 7f2bbbe15640 1 --2- 172.21.15.31:0/1590614660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bb4101730 0x7f2bb4101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.122+0000 7f2bbbe15640 1 --2- 172.21.15.31:0/1590614660 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4107740 0x7f2bb4107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.122+0000 7f2bbbe15640 1 -- 172.21.15.31:0/1590614660 >> 172.21.15.31:0/1590614660 conn(0x7f2bb40fd200 msgr2=0x7f2bb40ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.122+0000 7f2bbbe15640 1 -- 172.21.15.31:0/1590614660 shutdown_connections 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.122+0000 7f2bbbe15640 1 -- 172.21.15.31:0/1590614660 wait complete. 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bbbe15640 1 Processor -- start 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bbbe15640 1 -- start start 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bbbe15640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4101730 0x7f2bb4197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bbbe15640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bb41020f0 0x7f2bb41983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bbbe15640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb4107740 0x7f2bb419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bbbe15640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb41119c0 con 0x7f2bb4107740 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bbbe15640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4111840 con 0x7f2bb41020f0 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bbbe15640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4111b40 con 0x7f2bb4101730 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bba3b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb4107740 0x7f2bb419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bba3b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb4107740 0x7f2bb419c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57838/0 (socket says 172.21.15.31:57838) 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bba3b2640 1 -- 172.21.15.31:0/2365217046 learned_addr learned my addr 172.21.15.31:0/2365217046 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bb93b0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bb41020f0 0x7f2bb41983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.123+0000 7f2bb9bb1640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4101730 0x7f2bb4197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.124+0000 7f2bba3b2640 1 -- 172.21.15.31:0/2365217046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4101730 msgr2=0x7f2bb4197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.124+0000 7f2bba3b2640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4101730 0x7f2bb4197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.124+0000 7f2bba3b2640 1 -- 172.21.15.31:0/2365217046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bb41020f0 msgr2=0x7f2bb41983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.124+0000 7f2bba3b2640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bb41020f0 0x7f2bb41983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.124+0000 7f2bba3b2640 1 -- 172.21.15.31:0/2365217046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb419ce70 con 0x7f2bb4107740 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.124+0000 7f2bba3b2640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb4107740 0x7f2bb419c740 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2bac033040 tx=0x7f2bac003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.124+0000 7f2baaffd640 1 -- 172.21.15.31:0/2365217046 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac004040 con 0x7f2bb4107740 2024-04-25T01:58:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.124+0000 7f2baaffd640 1 -- 172.21.15.31:0/2365217046 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2bac004690 con 0x7f2bb4107740 2024-04-25T01:58:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.124+0000 7f2bbbe15640 1 -- 172.21.15.31:0/2365217046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb419d100 con 0x7f2bb4107740 2024-04-25T01:58:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.124+0000 7f2baaffd640 1 -- 172.21.15.31:0/2365217046 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac030e10 con 0x7f2bb4107740 2024-04-25T01:58:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.124+0000 7f2bbbe15640 1 -- 172.21.15.31:0/2365217046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb41a49f0 con 0x7f2bb4107740 2024-04-25T01:58:23.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.126+0000 7f2baaffd640 1 -- 172.21.15.31:0/2365217046 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2bac0305a0 con 0x7f2bb4107740 2024-04-25T01:58:23.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.127+0000 7f2bbec03740 1 Processor -- start 2024-04-25T01:58:23.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.127+0000 7f2bbec03740 1 -- start start 2024-04-25T01:58:23.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.127+0000 7f2baaffd640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2ba00a2e20 0x7f2ba00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.127+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b78620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.127+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b78b60 0x560a51b82cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.127+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b83230 0x560a51b85620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.127+0000 7f2bbec03740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560a51adccb0 con 0x560a51b78240 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.127+0000 7f2bbec03740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560a51a855b0 con 0x560a51b83230 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.127+0000 7f2bbec03740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560a51ad8c70 con 0x560a51b78b60 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb9bb1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b78620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2ba00a2e20 0x7f2ba00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb9bb1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b78620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57846/0 (socket says 172.21.15.31:57846) 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb9bb1640 1 -- 172.21.15.31:0/3499220631 learned_addr learned my addr 172.21.15.31:0/3499220631 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bba3b2640 1 --2- 172.21.15.31:0/3499220631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b83230 0x560a51b85620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2baaffd640 1 -- 172.21.15.31:0/2365217046 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f2bac02e030 con 0x7f2bb4107740 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/3499220631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b78b60 0x560a51b82cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb9bb1640 1 -- 172.21.15.31:0/3499220631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b78b60 msgr2=0x560a51b82cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb9bb1640 1 --2- 172.21.15.31:0/3499220631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b78b60 0x560a51b82cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb9bb1640 1 -- 172.21.15.31:0/3499220631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b83230 msgr2=0x560a51b85620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb9bb1640 1 --2- 172.21.15.31:0/3499220631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b83230 0x560a51b85620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb9bb1640 1 -- 172.21.15.31:0/3499220631 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a517f1210 con 0x560a51b78240 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2ba00a2e20 0x7f2ba00a52e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40ff800 tx=0x7f2ba4013880 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2bb9bb1640 1 --2- 172.21.15.31:0/3499220631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b78620 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2bb000aba0 tx=0x7f2bb000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced7cc87b80dda53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2b9ffff640 1 -- 172.21.15.31:0/3499220631 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb001d070 con 0x560a51b78240 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2b9ffff640 1 -- 172.21.15.31:0/3499220631 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2bb0018af0 con 0x560a51b78240 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.128+0000 7f2b9ffff640 1 -- 172.21.15.31:0/3499220631 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb00109b0 con 0x560a51b78240 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.129+0000 7f2bbec03740 1 -- 172.21.15.31:0/3499220631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 msgr2=0x560a51b78620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.129+0000 7f2bbec03740 1 --2- 172.21.15.31:0/3499220631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b78620 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2bb000aba0 tx=0x7f2bb000f790 comp rx=0 tx=0).stop 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.129+0000 7f2bbec03740 1 -- 172.21.15.31:0/3499220631 shutdown_connections 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.129+0000 7f2bbec03740 1 --2- 172.21.15.31:0/3499220631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b83230 0x560a51b85620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.129+0000 7f2bbec03740 1 --2- 172.21.15.31:0/3499220631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b78b60 0x560a51b82cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.129+0000 7f2bbec03740 1 --2- 172.21.15.31:0/3499220631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b78620 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.129+0000 7f2bbec03740 1 -- 172.21.15.31:0/3499220631 >> 172.21.15.31:0/3499220631 conn(0x560a51b74560 msgr2=0x560a51b74940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.129+0000 7f2bbec03740 1 -- 172.21.15.31:0/3499220631 shutdown_connections 2024-04-25T01:58:23.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.129+0000 7f2bbec03740 1 -- 172.21.15.31:0/3499220631 wait complete. 2024-04-25T01:58:23.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bbec03740 1 -- start start 2024-04-25T01:58:23.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bbec03740 1 Processor -- start 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 0x560a51b80190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b806d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b80c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bbec03740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560a51b86020 con 0x560a51b78b60 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bbec03740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560a51a98b50 con 0x560a51b78240 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bbec03740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560a51b861a0 con 0x560a51b83230 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bbec03740 10 client.0 _ll_register_callbacks cb 0x560a51ad3ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bb93b0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b806d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bb9bb1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 0x560a51b80190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bb93b0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b806d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57854/0 (socket says 172.21.15.31:57854) 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.130+0000 7f2bb93b0640 1 -- 172.21.15.31:0/1588682564 learned_addr learned my addr 172.21.15.31:0/1588682564 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:23.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bba3b2640 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b80c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bb93b0640 1 -- 172.21.15.31:0/1588682564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 msgr2=0x560a51b80c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b80c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bb93b0640 1 -- 172.21.15.31:0/1588682564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 msgr2=0x560a51b80190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 0x560a51b80190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bb93b0640 1 -- 172.21.15.31:0/1588682564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a518bd620 con 0x560a51b78b60 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bb9bb1640 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 0x560a51b80190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bba3b2640 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b80c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b806d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4004480 tx=0x7f2ba4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f24f1056003a858b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2b9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/1588682564 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4024030 con 0x560a51b78b60 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/1588682564 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f2ba400f750 con 0x560a51b78b60 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bbec03740 1 -- 172.21.15.31:0/1588682564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bb0018af0 con 0x560a51b78b60 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/1588682564 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2ba402d020 con 0x560a51b78b60 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bbec03740 10 client.16455 fetch_fsmap learned FSMap version 30 2024-04-25T01:58:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.131+0000 7f2bbec03740 1 -- 172.21.15.31:0/1588682564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560a51b92240 con 0x560a51b78b60 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.132+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/1588682564 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2bb0018af0 con 0x560a51b78b60 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.132+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/1588682564 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2ba401f070 con 0x560a51b78b60 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.132+0000 7f2bbec03740 10 client.16455 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.132+0000 7f2bbec03740 10 client.16455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.132+0000 7f2bbec03740 20 client.16455 populate_metadata read hostname 'smithi031' 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.132+0000 7f2bbec03740 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560a51b937c0 0x560a51b95c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.132+0000 7f2bbec03740 4 client.16455 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.132+0000 7f2bbec03740 1 -- 172.21.15.31:0/1588682564 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a51aea1f0 con 0x560a51b937c0 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.132+0000 7f2bb9bb1640 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560a51b937c0 0x560a51b95c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.132+0000 7f2bb9bb1640 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560a51b937c0 0x560a51b95c00 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49677f8dae31994b server_cookie=9821d44a496be4a1 in_seq=0 out_seq=0 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.132+0000 7f2b9d7fa640 10 client.16455 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/1588682564 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a51aea1f0 con 0x560a51b937c0 2024-04-25T01:58:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2b9d7fa640 10 client.16455 handle_command_reply: tid=0 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2bbec03740 1 client.16455 shutdown 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2bbec03740 1 -- 172.21.15.31:0/1588682564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 msgr2=0x560a51b806d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2bbec03740 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b806d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4004480 tx=0x7f2ba4004730 comp rx=0 tx=0).stop 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2bbec03740 1 -- 172.21.15.31:0/1588682564 shutdown_connections 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2bbec03740 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560a51b937c0 0x560a51b95c00 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2bbec03740 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b80c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2bbec03740 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b806d0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2b9d7fa640 0 client.16455 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2bbec03740 1 --2- 172.21.15.31:0/1588682564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 0x560a51b80190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2bbec03740 1 -- 172.21.15.31:0/1588682564 >> 172.21.15.31:0/1588682564 conn(0x560a51b74560 msgr2=0x560a51b7a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.133+0000 7f2bbec03740 1 -- 172.21.15.31:0/1588682564 shutdown_connections 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.134+0000 7f2bbec03740 1 -- 172.21.15.31:0/1588682564 wait complete. 2024-04-25T01:58:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.134+0000 7f2bbec03740 20 client.16455 trim_cache size 0 max 16384 2024-04-25T01:58:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.150+0000 7f2bbec03740 1 Processor -- start 2024-04-25T01:58:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.150+0000 7f2bbec03740 1 -- start start 2024-04-25T01:58:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.150+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 0x560a51b828c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.150+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b8eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.150+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b7f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bbec03740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb41ad180 con 0x560a51b78b60 2024-04-25T01:58:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bbec03740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb41aed80 con 0x560a51b78240 2024-04-25T01:58:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bbec03740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb41ad850 con 0x560a51b83230 2024-04-25T01:58:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bb9bb1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 0x560a51b828c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bb93b0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b8eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bb93b0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b8eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57858/0 (socket says 172.21.15.31:57858) 2024-04-25T01:58:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bb93b0640 1 -- 172.21.15.31:0/2320996628 learned_addr learned my addr 172.21.15.31:0/2320996628 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bba3b2640 1 --2- 172.21.15.31:0/2320996628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b7f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bb93b0640 1 -- 172.21.15.31:0/2320996628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 msgr2=0x560a51b7f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/2320996628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b7f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bb93b0640 1 -- 172.21.15.31:0/2320996628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 msgr2=0x560a51b828c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/2320996628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 0x560a51b828c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bb93b0640 1 -- 172.21.15.31:0/2320996628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a51b96280 con 0x560a51b78b60 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bb9bb1640 1 --2- 172.21.15.31:0/2320996628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 0x560a51b828c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.151+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/2320996628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b8eee0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4020c20 tx=0x7f2ba40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae5f7ecb458b139 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2b9f7fe640 1 -- 172.21.15.31:0/2320996628 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4024070 con 0x560a51b78b60 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2b9f7fe640 1 -- 172.21.15.31:0/2320996628 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ba4005020 con 0x560a51b78b60 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2b9f7fe640 1 -- 172.21.15.31:0/2320996628 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba40047a0 con 0x560a51b78b60 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2bbec03740 1 -- 172.21.15.31:0/2320996628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 msgr2=0x560a51b8eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2bbec03740 1 --2- 172.21.15.31:0/2320996628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b8eee0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4020c20 tx=0x7f2ba40046b0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2bbec03740 1 -- 172.21.15.31:0/2320996628 shutdown_connections 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2bbec03740 1 --2- 172.21.15.31:0/2320996628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b7f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2bbec03740 1 --2- 172.21.15.31:0/2320996628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78b60 0x560a51b8eee0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2bbec03740 1 --2- 172.21.15.31:0/2320996628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78240 0x560a51b828c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2bbec03740 1 -- 172.21.15.31:0/2320996628 >> 172.21.15.31:0/2320996628 conn(0x560a51b74560 msgr2=0x560a51b72bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2bbec03740 1 -- 172.21.15.31:0/2320996628 shutdown_connections 2024-04-25T01:58:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.152+0000 7f2bbec03740 1 -- 172.21.15.31:0/2320996628 wait complete. 2024-04-25T01:58:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bbec03740 1 -- start start 2024-04-25T01:58:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bbec03740 1 Processor -- start 2024-04-25T01:58:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b8b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78b60 0x560a51b8b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bbec03740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b865b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bbec03740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb41ae5f0 con 0x560a51b78240 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bbec03740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb41adf20 con 0x560a51b78b60 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bbec03740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb41ab160 con 0x560a51b83230 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bbec03740 10 client.0 _ll_register_callbacks cb 0x7f2ba4004100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bb9bb1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b8b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bb9bb1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b8b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57864/0 (socket says 172.21.15.31:57864) 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bb9bb1640 1 -- 172.21.15.31:0/2915149908 learned_addr learned my addr 172.21.15.31:0/2915149908 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78b60 0x560a51b8b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.153+0000 7f2bba3b2640 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b865b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2bb9bb1640 1 -- 172.21.15.31:0/2915149908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 msgr2=0x560a51b865b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2bb9bb1640 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b865b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2bb9bb1640 1 -- 172.21.15.31:0/2915149908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78b60 msgr2=0x560a51b8b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2bb9bb1640 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78b60 0x560a51b8b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2bb9bb1640 1 -- 172.21.15.31:0/2915149908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a51b86db0 con 0x560a51b78240 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2bb9bb1640 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b8b1b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2bb000aaf0 tx=0x7f2bb00105b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932c855053191d47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2b9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/2915149908 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb002d030 con 0x560a51b78240 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/2915149908 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f2bb0018750 con 0x560a51b78240 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2bbec03740 1 -- 172.21.15.31:0/2915149908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ba4005020 con 0x560a51b78240 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/2915149908 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2bb0036020 con 0x560a51b78240 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2bbec03740 10 client.16461 fetch_fsmap learned FSMap version 30 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2bbec03740 1 -- 172.21.15.31:0/2915149908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560a51bf9830 con 0x560a51b78240 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.154+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/2915149908 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ba4005020 con 0x560a51b78240 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.155+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/2915149908 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2bb00292e0 con 0x560a51b78240 2024-04-25T01:58:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.155+0000 7f2bbec03740 10 client.16461 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:58:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.155+0000 7f2bbec03740 10 client.16461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.155+0000 7f2bbec03740 20 client.16461 populate_metadata read hostname 'smithi031' 2024-04-25T01:58:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.155+0000 7f2bbec03740 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560a51bf9cc0 0x560a51bfc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.155+0000 7f2bbec03740 4 client.16461 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:58:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.155+0000 7f2bbec03740 1 -- 172.21.15.31:0/2915149908 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560a51ae4540 con 0x560a51bf9cc0 2024-04-25T01:58:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.155+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560a51bf9cc0 0x560a51bfc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.155+0000 7f2bb93b0640 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560a51bf9cc0 0x560a51bfc0e0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9c542b7842cf178 server_cookie=66070b2318bdd204 in_seq=0 out_seq=0 2024-04-25T01:58:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.155+0000 7f2b9d7fa640 10 client.16461 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.174+0000 7f2b9d7fa640 1 -- 172.21.15.31:0/2915149908 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560a51ae4540 con 0x560a51bf9cc0 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.174+0000 7f2b9d7fa640 10 client.16461 handle_command_reply: tid=0 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.174+0000 7f2bbec03740 1 client.16461 shutdown 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2bbec03740 1 -- 172.21.15.31:0/2915149908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 msgr2=0x560a51b8b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2bbec03740 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b8b1b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2bb000aaf0 tx=0x7f2bb00105b0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2bbec03740 1 -- 172.21.15.31:0/2915149908 shutdown_connections 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2bbec03740 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560a51bf9cc0 0x560a51bfc0e0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2bbec03740 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a51b83230 0x560a51b865b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2bbec03740 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a51b78b60 0x560a51b8b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2b9d7fa640 0 client.16461 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2bbec03740 1 --2- 172.21.15.31:0/2915149908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a51b78240 0x560a51b8b1b0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2bbec03740 1 -- 172.21.15.31:0/2915149908 >> 172.21.15.31:0/2915149908 conn(0x560a51b74560 msgr2=0x560a51b80150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2bbec03740 1 -- 172.21.15.31:0/2915149908 shutdown_connections 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2bbec03740 1 -- 172.21.15.31:0/2915149908 wait complete. 2024-04-25T01:58:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.175+0000 7f2bbec03740 20 client.16461 trim_cache size 0 max 16384 2024-04-25T01:58:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:58:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:58:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:58:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:58:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 -- 172.21.15.31:0/2365217046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2ba00a2e20 msgr2=0x7f2ba00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2ba00a2e20 0x7f2ba00a52e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40ff800 tx=0x7f2ba4013880 comp rx=0 tx=0).stop 2024-04-25T01:58:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 -- 172.21.15.31:0/2365217046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb4107740 msgr2=0x7f2bb419c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb4107740 0x7f2bb419c740 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2bac033040 tx=0x7f2bac003970 comp rx=0 tx=0).stop 2024-04-25T01:58:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 -- 172.21.15.31:0/2365217046 shutdown_connections 2024-04-25T01:58:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2ba00a2e20 0x7f2ba00a52e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb4107740 0x7f2bb419c740 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bb41020f0 0x7f2bb41983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 --2- 172.21.15.31:0/2365217046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4101730 0x7f2bb4197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 -- 172.21.15.31:0/2365217046 >> 172.21.15.31:0/2365217046 conn(0x7f2bb40fd200 msgr2=0x7f2bb40fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 -- 172.21.15.31:0/2365217046 shutdown_connections 2024-04-25T01:58:23.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:23.176+0000 7f2bbbe15640 1 -- 172.21.15.31:0/2365217046 wait complete. 2024-04-25T01:58:23.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 74779 2.70 MB/sec execute 30 sec latency 21.274 ms 2024-04-25T01:58:24.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:23 smithi031 ceph-mon[29204]: pgmap v120: 97 pgs: 97 active+clean; 54 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-04-25T01:58:24.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 75031 2.62 MB/sec execute 31 sec latency 18.514 ms 2024-04-25T01:58:24.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:23 smithi040 ceph-mon[32418]: pgmap v120: 97 pgs: 97 active+clean; 54 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-04-25T01:58:24.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:23 smithi134 ceph-mon[33098]: pgmap v120: 97 pgs: 97 active+clean; 54 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 1.1 MiB/s wr, 131 op/s 2024-04-25T01:58:25.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 75482 2.56 MB/sec execute 32 sec latency 15.567 ms 2024-04-25T01:58:26.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:25 smithi031 ceph-mon[29204]: pgmap v121: 97 pgs: 97 active+clean; 55 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-25T01:58:26.304 INFO:tasks.workunit.client.0.smithi031.stdout: 1 76016 2.62 MB/sec execute 33 sec latency 19.162 ms 2024-04-25T01:58:26.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:25 smithi040 ceph-mon[32418]: pgmap v121: 97 pgs: 97 active+clean; 55 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-25T01:58:26.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:25 smithi134 ceph-mon[33098]: pgmap v121: 97 pgs: 97 active+clean; 55 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-25T01:58:27.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 76625 2.56 MB/sec execute 34 sec latency 18.696 ms 2024-04-25T01:58:28.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 77125 2.55 MB/sec execute 35 sec latency 18.231 ms 2024-04-25T01:58:28.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:27 smithi040 ceph-mon[32418]: pgmap v122: 97 pgs: 97 active+clean; 55 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:58:28.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:27 smithi134 ceph-mon[33098]: pgmap v122: 97 pgs: 97 active+clean; 55 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:58:28.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:27 smithi031 ceph-mon[29204]: pgmap v122: 97 pgs: 97 active+clean; 55 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:58:29.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 77851 2.66 MB/sec execute 36 sec latency 14.380 ms 2024-04-25T01:58:30.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 78300 2.70 MB/sec execute 37 sec latency 31.972 ms 2024-04-25T01:58:30.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:29 smithi040 ceph-mon[32418]: pgmap v123: 97 pgs: 97 active+clean; 55 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:58:30.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:29 smithi134 ceph-mon[33098]: pgmap v123: 97 pgs: 97 active+clean; 55 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:58:30.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:29 smithi031 ceph-mon[29204]: pgmap v123: 97 pgs: 97 active+clean; 55 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:58:31.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 78536 2.64 MB/sec execute 38 sec latency 48.329 ms 2024-04-25T01:58:32.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 78915 2.59 MB/sec execute 39 sec latency 18.938 ms 2024-04-25T01:58:32.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:31 smithi040 ceph-mon[32418]: pgmap v124: 97 pgs: 97 active+clean; 56 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.8 MiB/s wr, 188 op/s 2024-04-25T01:58:32.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:31 smithi134 ceph-mon[33098]: pgmap v124: 97 pgs: 97 active+clean; 56 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.8 MiB/s wr, 188 op/s 2024-04-25T01:58:32.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:31 smithi031 ceph-mon[29204]: pgmap v124: 97 pgs: 97 active+clean; 56 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.8 MiB/s wr, 188 op/s 2024-04-25T01:58:33.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 79469 2.64 MB/sec execute 40 sec latency 20.270 ms 2024-04-25T01:58:34.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 80005 2.58 MB/sec execute 41 sec latency 18.734 ms 2024-04-25T01:58:34.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:33 smithi040 ceph-mon[32418]: pgmap v125: 97 pgs: 97 active+clean; 56 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-25T01:58:34.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:33 smithi134 ceph-mon[33098]: pgmap v125: 97 pgs: 97 active+clean; 56 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-25T01:58:34.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:33 smithi031 ceph-mon[29204]: pgmap v125: 97 pgs: 97 active+clean; 56 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-25T01:58:35.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 80573 2.57 MB/sec execute 42 sec latency 18.370 ms 2024-04-25T01:58:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.273+0000 7f05e846b640 1 -- 172.21.15.31:0/2960419447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e01032f0 msgr2=0x7f05e010b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.273+0000 7f05e846b640 1 --2- 172.21.15.31:0/2960419447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e01032f0 0x7f05e010b7d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009900 tx=0x7f05d002a510 comp rx=0 tx=0).stop 2024-04-25T01:58:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.273+0000 7f05e846b640 1 -- 172.21.15.31:0/2960419447 shutdown_connections 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.273+0000 7f05e846b640 1 --2- 172.21.15.31:0/2960419447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e010bd10 0x7f05e010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.274+0000 7f05e846b640 1 --2- 172.21.15.31:0/2960419447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e01032f0 0x7f05e010b7d0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.274+0000 7f05e846b640 1 --2- 172.21.15.31:0/2960419447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e01029d0 0x7f05e0102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.274+0000 7f05e846b640 1 -- 172.21.15.31:0/2960419447 >> 172.21.15.31:0/2960419447 conn(0x7f05e00777c0 msgr2=0x7f05e0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.274+0000 7f05e846b640 1 -- 172.21.15.31:0/2960419447 shutdown_connections 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.274+0000 7f05e846b640 1 -- 172.21.15.31:0/2960419447 wait complete. 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.274+0000 7f05e846b640 1 Processor -- start 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.274+0000 7f05e846b640 1 -- start start 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.274+0000 7f05e846b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e01029d0 0x7f05e0197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.274+0000 7f05e846b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e01032f0 0x7f05e01983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e846b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e010bd10 0x7f05e019c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e846b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f05e01119c0 con 0x7f05e01029d0 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e846b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f05e0111840 con 0x7f05e010bd10 2024-04-25T01:58:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e846b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f05e0111b40 con 0x7f05e01032f0 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e6207640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e01029d0 0x7f05e0197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e6207640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e01029d0 0x7f05e0197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35710/0 (socket says 172.21.15.31:35710) 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e6207640 1 -- 172.21.15.31:0/1227644350 learned_addr learned my addr 172.21.15.31:0/1227644350 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e5a06640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e01032f0 0x7f05e01983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e6a08640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e010bd10 0x7f05e019c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e6207640 1 -- 172.21.15.31:0/1227644350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e01032f0 msgr2=0x7f05e01983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e6207640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e01032f0 0x7f05e01983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e6207640 1 -- 172.21.15.31:0/1227644350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e010bd10 msgr2=0x7f05e019c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e6207640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e010bd10 0x7f05e019c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e6207640 1 -- 172.21.15.31:0/1227644350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e019ce40 con 0x7f05e01029d0 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e6207640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e01029d0 0x7f05e0197ea0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00e9f0 tx=0x7f05dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05d77fe640 1 -- 172.21.15.31:0/1227644350 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc00cd80 con 0x7f05e01029d0 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05d77fe640 1 -- 172.21.15.31:0/1227644350 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05dc0108e0 con 0x7f05e01029d0 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e846b640 1 -- 172.21.15.31:0/1227644350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e019d130 con 0x7f05e01029d0 2024-04-25T01:58:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05d77fe640 1 -- 172.21.15.31:0/1227644350 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc018920 con 0x7f05e01029d0 2024-04-25T01:58:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.275+0000 7f05e846b640 1 -- 172.21.15.31:0/1227644350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e01a49c0 con 0x7f05e01029d0 2024-04-25T01:58:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.278+0000 7f05d77fe640 1 -- 172.21.15.31:0/1227644350 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f05dc010a90 con 0x7f05e01029d0 2024-04-25T01:58:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.279+0000 7f05eb17b740 1 Processor -- start 2024-04-25T01:58:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.279+0000 7f05eb17b740 1 -- start start 2024-04-25T01:58:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.279+0000 7f05d77fe640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05cc0a2e20 0x7f05cc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.279+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10c99350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.279+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.279+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10caaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.279+0000 7f05eb17b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556a10bfdcb0 con 0x556a10c98f70 2024-04-25T01:58:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.279+0000 7f05eb17b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556a10ba65b0 con 0x556a10ca8640 2024-04-25T01:58:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.279+0000 7f05eb17b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556a10bf9c70 con 0x556a10c99920 2024-04-25T01:58:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.279+0000 7f05e6a08640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05cc0a2e20 0x7f05cc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05e5a06640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05d77fe640 1 -- 172.21.15.31:0/1227644350 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f05dc0cf690 con 0x7f05e01029d0 2024-04-25T01:58:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05e6a08640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10caaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05e5a06640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:40332/0 (socket says 172.21.15.31:40332) 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05e5a06640 1 -- 172.21.15.31:0/4229784154 learned_addr learned my addr 172.21.15.31:0/4229784154 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05e5a06640 1 -- 172.21.15.31:0/4229784154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 msgr2=0x556a10caaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05e5a06640 1 --2- 172.21.15.31:0/4229784154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10caaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05e5a06640 1 -- 172.21.15.31:0/4229784154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 msgr2=0x556a10c99350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05e5a06640 1 --2- 172.21.15.31:0/4229784154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10c99350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05e5a06640 1 -- 172.21.15.31:0/4229784154 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a10912210 con 0x556a10c99920 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05e6a08640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05cc0a2e20 0x7f05cc0a52e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f05d8007fd0 tx=0x7f05d8007d40 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05e5a06640 1 --2- 172.21.15.31:0/4229784154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca8100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f05d0033000 tx=0x7f05d0032cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f21da839a9abd79b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05d4bfe640 1 -- 172.21.15.31:0/4229784154 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d0035040 con 0x556a10c99920 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05d4bfe640 1 -- 172.21.15.31:0/4229784154 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05d0032e50 con 0x556a10c99920 2024-04-25T01:58:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.280+0000 7f05d4bfe640 1 -- 172.21.15.31:0/4229784154 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d00055b0 con 0x556a10c99920 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.281+0000 7f05eb17b740 1 -- 172.21.15.31:0/4229784154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 msgr2=0x556a10ca8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.281+0000 7f05eb17b740 1 --2- 172.21.15.31:0/4229784154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca8100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f05d0033000 tx=0x7f05d0032cb0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.281+0000 7f05eb17b740 1 -- 172.21.15.31:0/4229784154 shutdown_connections 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.281+0000 7f05eb17b740 1 --2- 172.21.15.31:0/4229784154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10caaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.281+0000 7f05eb17b740 1 --2- 172.21.15.31:0/4229784154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca8100 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.281+0000 7f05eb17b740 1 --2- 172.21.15.31:0/4229784154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10c99350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.281+0000 7f05eb17b740 1 -- 172.21.15.31:0/4229784154 >> 172.21.15.31:0/4229784154 conn(0x556a10c95560 msgr2=0x556a10c95940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.281+0000 7f05eb17b740 1 -- 172.21.15.31:0/4229784154 shutdown_connections 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.281+0000 7f05eb17b740 1 -- 172.21.15.31:0/4229784154 wait complete. 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05eb17b740 1 -- start start 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05eb17b740 1 Processor -- start 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10ca55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10ca6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05eb17b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556a10cab3a0 con 0x556a10c98f70 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05eb17b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556a10bb9b50 con 0x556a10ca8640 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05eb17b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556a10cab520 con 0x556a10c99920 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05eb17b740 10 client.0 _ll_register_callbacks cb 0x556a10bf4ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05e6207640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10ca55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05e6207640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10ca55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35732/0 (socket says 172.21.15.31:35732) 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05e6207640 1 -- 172.21.15.31:0/449637533 learned_addr learned my addr 172.21.15.31:0/449637533 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05e5a06640 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05e6a08640 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10ca6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05e6207640 1 -- 172.21.15.31:0/449637533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 msgr2=0x556a10ca5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05e6207640 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05e6207640 1 -- 172.21.15.31:0/449637533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 msgr2=0x556a10ca6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05e6207640 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10ca6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05e6207640 1 -- 172.21.15.31:0/449637533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a109de620 con 0x556a10c98f70 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.282+0000 7f05e5a06640 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05e6207640 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10ca55f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0a42e0 tx=0x7f05dc07ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a279a6d137f511 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05c9ffb640 1 -- 172.21.15.31:0/449637533 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc09ff50 con 0x556a10c98f70 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05c9ffb640 1 -- 172.21.15.31:0/449637533 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f05dc0cd050 con 0x556a10c98f70 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05eb17b740 1 -- 172.21.15.31:0/449637533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05d0032e50 con 0x556a10c98f70 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05c9ffb640 1 -- 172.21.15.31:0/449637533 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f05dc084030 con 0x556a10c98f70 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05eb17b740 10 client.16470 fetch_fsmap learned FSMap version 30 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05eb17b740 1 -- 172.21.15.31:0/449637533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556a10cb7690 con 0x556a10c98f70 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05c9ffb640 1 -- 172.21.15.31:0/449637533 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05d0032e50 con 0x556a10c98f70 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05c9ffb640 1 -- 172.21.15.31:0/449637533 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f05dc0979a0 con 0x556a10c98f70 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05eb17b740 10 client.16470 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05eb17b740 10 client.16470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05eb17b740 20 client.16470 populate_metadata read hostname 'smithi031' 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05eb17b740 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a10cb8ba0 0x556a10cbb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05eb17b740 4 client.16470 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.283+0000 7f05eb17b740 1 -- 172.21.15.31:0/449637533 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a10c0b1f0 con 0x556a10cb8ba0 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.284+0000 7f05e5a06640 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a10cb8ba0 0x556a10cbb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.284+0000 7f05c9ffb640 10 client.16470 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.284+0000 7f05e5a06640 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a10cb8ba0 0x556a10cbb010 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72ee8b3b4a356fd6 server_cookie=8c5bd4fa1089d9c8 in_seq=0 out_seq=0 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05c9ffb640 1 -- 172.21.15.31:0/449637533 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a10c0b1f0 con 0x556a10cb8ba0 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05c9ffb640 10 client.16470 handle_command_reply: tid=0 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 1 client.16470 shutdown 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 1 -- 172.21.15.31:0/449637533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 msgr2=0x556a10ca55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10ca55f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0a42e0 tx=0x7f05dc07ea80 comp rx=0 tx=0).stop 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 1 -- 172.21.15.31:0/449637533 shutdown_connections 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a10cb8ba0 0x556a10cbb010 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10ca6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 1 --2- 172.21.15.31:0/449637533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10ca55f0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05c9ffb640 0 client.16470 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 1 -- 172.21.15.31:0/449637533 >> 172.21.15.31:0/449637533 conn(0x556a10c95560 msgr2=0x556a10c9b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 1 -- 172.21.15.31:0/449637533 shutdown_connections 2024-04-25T01:58:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 1 -- 172.21.15.31:0/449637533 wait complete. 2024-04-25T01:58:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.285+0000 7f05eb17b740 20 client.16470 trim_cache size 0 max 16384 2024-04-25T01:58:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05eb17b740 1 Processor -- start 2024-04-25T01:58:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05eb17b740 1 -- start start 2024-04-25T01:58:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10cb42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10c93d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05eb17b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f05e01aafb0 con 0x556a10c98f70 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05eb17b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f05e01acbb0 con 0x556a10ca8640 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05eb17b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f05e01ab680 con 0x556a10c99920 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05e6207640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10cb42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05e6207640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10cb42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35738/0 (socket says 172.21.15.31:35738) 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05e6207640 1 -- 172.21.15.31:0/2513792173 learned_addr learned my addr 172.21.15.31:0/2513792173 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05e6a08640 1 --2- 172.21.15.31:0/2513792173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10c93d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05e5a06640 1 --2- 172.21.15.31:0/2513792173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05e6207640 1 -- 172.21.15.31:0/2513792173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 msgr2=0x556a10ca4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05e6207640 1 --2- 172.21.15.31:0/2513792173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05e6207640 1 -- 172.21.15.31:0/2513792173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 msgr2=0x556a10c93d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05e6207640 1 --2- 172.21.15.31:0/2513792173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10c93d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.301+0000 7f05e6207640 1 -- 172.21.15.31:0/2513792173 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a10c9a470 con 0x556a10c98f70 2024-04-25T01:58:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05e6207640 1 --2- 172.21.15.31:0/2513792173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10cb42d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f05dc09bbe0 tx=0x7f05dc00c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed6b4f988b0578 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:36.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 81364 2.67 MB/sec execute 43 sec latency 16.467 ms 2024-04-25T01:58:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05cbfff640 1 -- 172.21.15.31:0/2513792173 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc09ff50 con 0x556a10c98f70 2024-04-25T01:58:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05cbfff640 1 -- 172.21.15.31:0/2513792173 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05dc010de0 con 0x556a10c98f70 2024-04-25T01:58:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05cbfff640 1 -- 172.21.15.31:0/2513792173 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc09b5b0 con 0x556a10c98f70 2024-04-25T01:58:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05eb17b740 1 -- 172.21.15.31:0/2513792173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 msgr2=0x556a10cb42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05eb17b740 1 --2- 172.21.15.31:0/2513792173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10cb42d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f05dc09bbe0 tx=0x7f05dc00c7d0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05eb17b740 1 -- 172.21.15.31:0/2513792173 shutdown_connections 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05eb17b740 1 --2- 172.21.15.31:0/2513792173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10ca8640 0x556a10c93d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05eb17b740 1 --2- 172.21.15.31:0/2513792173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c99920 0x556a10ca4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05eb17b740 1 --2- 172.21.15.31:0/2513792173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10c98f70 0x556a10cb42d0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05eb17b740 1 -- 172.21.15.31:0/2513792173 >> 172.21.15.31:0/2513792173 conn(0x556a10c95560 msgr2=0x556a10ca5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05eb17b740 1 -- 172.21.15.31:0/2513792173 shutdown_connections 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.302+0000 7f05eb17b740 1 -- 172.21.15.31:0/2513792173 wait complete. 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05eb17b740 1 -- start start 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05eb17b740 1 Processor -- start 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c98f70 0x556a10cb0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10c99920 0x556a10ca1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05eb17b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10ca8640 0x556a10ca2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05eb17b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f05e01ac420 con 0x556a10ca8640 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05eb17b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f05e01abd50 con 0x556a10c99920 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05eb17b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f05e01a8f90 con 0x556a10c98f70 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05eb17b740 10 client.0 _ll_register_callbacks cb 0x7f05dc00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05e6a08640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10ca8640 0x556a10ca2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05e6a08640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10ca8640 0x556a10ca2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35750/0 (socket says 172.21.15.31:35750) 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05e6a08640 1 -- 172.21.15.31:0/1120514999 learned_addr learned my addr 172.21.15.31:0/1120514999 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05e5a06640 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10c99920 0x556a10ca1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05e6207640 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c98f70 0x556a10cb0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.303+0000 7f05e6a08640 1 -- 172.21.15.31:0/1120514999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c98f70 msgr2=0x556a10cb0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05e6a08640 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c98f70 0x556a10cb0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05e6a08640 1 -- 172.21.15.31:0/1120514999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10c99920 msgr2=0x556a10ca1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05e6a08640 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10c99920 0x556a10ca1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05e6a08640 1 -- 172.21.15.31:0/1120514999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a10ca2d90 con 0x556a10ca8640 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05e5a06640 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10c99920 0x556a10ca1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05e6a08640 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10ca8640 0x556a10ca2560 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f05d80072c0 tx=0x7f05d801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6153a2906e5e59b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05c9ffb640 1 -- 172.21.15.31:0/1120514999 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d800fc60 con 0x556a10ca8640 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05c9ffb640 1 -- 172.21.15.31:0/1120514999 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8440+0+0 (secure 0 0 0) 0x7f05d800fe00 con 0x556a10ca8640 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05eb17b740 1 -- 172.21.15.31:0/1120514999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556a10c05540 con 0x556a10ca8640 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05c9ffb640 1 -- 172.21.15.31:0/1120514999 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f05d8029470 con 0x556a10ca8640 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05eb17b740 10 client.16476 fetch_fsmap learned FSMap version 30 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05eb17b740 1 -- 172.21.15.31:0/1120514999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556a10d1f200 con 0x556a10ca8640 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05c9ffb640 1 -- 172.21.15.31:0/1120514999 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x556a10c05540 con 0x556a10ca8640 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.304+0000 7f05c9ffb640 1 -- 172.21.15.31:0/1120514999 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f05d8033430 con 0x556a10ca8640 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.305+0000 7f05eb17b740 10 client.16476 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.305+0000 7f05eb17b740 10 client.16476 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.305+0000 7f05eb17b740 20 client.16476 populate_metadata read hostname 'smithi031' 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.305+0000 7f05eb17b740 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a10d1fa70 0x556a10d21e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.305+0000 7f05eb17b740 4 client.16476 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:58:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.305+0000 7f05eb17b740 1 -- 172.21.15.31:0/1120514999 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556a10b838e0 con 0x556a10d1fa70 2024-04-25T01:58:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.305+0000 7f05e6207640 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a10d1fa70 0x556a10d21e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.305+0000 7f05e6207640 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a10d1fa70 0x556a10d21e90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fb64d755457e36c server_cookie=e836a56d167658d6 in_seq=0 out_seq=0 2024-04-25T01:58:36.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.305+0000 7f05c9ffb640 10 client.16476 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:36.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.326+0000 7f05c9ffb640 1 -- 172.21.15.31:0/1120514999 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556a10b838e0 con 0x556a10d1fa70 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.326+0000 7f05c9ffb640 10 client.16476 handle_command_reply: tid=0 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.326+0000 7f05eb17b740 1 client.16476 shutdown 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05eb17b740 1 -- 172.21.15.31:0/1120514999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10ca8640 msgr2=0x556a10ca2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05eb17b740 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10ca8640 0x556a10ca2560 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f05d80072c0 tx=0x7f05d801e000 comp rx=0 tx=0).stop 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05eb17b740 1 -- 172.21.15.31:0/1120514999 shutdown_connections 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05eb17b740 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a10d1fa70 0x556a10d21e90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05eb17b740 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a10ca8640 0x556a10ca2560 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05c9ffb640 0 client.16476 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05eb17b740 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a10c99920 0x556a10ca1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05eb17b740 1 --2- 172.21.15.31:0/1120514999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a10c98f70 0x556a10cb0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05eb17b740 1 -- 172.21.15.31:0/1120514999 >> 172.21.15.31:0/1120514999 conn(0x556a10c95560 msgr2=0x556a10c9a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05eb17b740 1 -- 172.21.15.31:0/1120514999 shutdown_connections 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05eb17b740 1 -- 172.21.15.31:0/1120514999 wait complete. 2024-04-25T01:58:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05eb17b740 20 client.16476 trim_cache size 0 max 16384 2024-04-25T01:58:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:58:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:58:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:58:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:58:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05e846b640 1 -- 172.21.15.31:0/1227644350 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05cc0a2e20 msgr2=0x7f05cc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05e846b640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05cc0a2e20 0x7f05cc0a52e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f05d8007fd0 tx=0x7f05d8007d40 comp rx=0 tx=0).stop 2024-04-25T01:58:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05e846b640 1 -- 172.21.15.31:0/1227644350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e01029d0 msgr2=0x7f05e0197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.327+0000 7f05e846b640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e01029d0 0x7f05e0197ea0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00e9f0 tx=0x7f05dc00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.328+0000 7f05e846b640 1 -- 172.21.15.31:0/1227644350 shutdown_connections 2024-04-25T01:58:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.328+0000 7f05e846b640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f05cc0a2e20 0x7f05cc0a52e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.328+0000 7f05e846b640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05e010bd10 0x7f05e019c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.328+0000 7f05e846b640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e01032f0 0x7f05e01983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.328+0000 7f05e846b640 1 --2- 172.21.15.31:0/1227644350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05e01029d0 0x7f05e0197ea0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.328+0000 7f05e846b640 1 -- 172.21.15.31:0/1227644350 >> 172.21.15.31:0/1227644350 conn(0x7f05e00777c0 msgr2=0x7f05e00fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.328+0000 7f05e846b640 1 -- 172.21.15.31:0/1227644350 shutdown_connections 2024-04-25T01:58:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:36.328+0000 7f05e846b640 1 -- 172.21.15.31:0/1227644350 wait complete. 2024-04-25T01:58:36.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:35 smithi040 ceph-mon[32418]: pgmap v126: 97 pgs: 97 active+clean; 56 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-04-25T01:58:36.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:35 smithi134 ceph-mon[33098]: pgmap v126: 97 pgs: 97 active+clean; 56 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-04-25T01:58:36.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:35 smithi031 ceph-mon[29204]: pgmap v126: 97 pgs: 97 active+clean; 56 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-04-25T01:58:37.009 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:58:36 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e[61957]: ignoring --setuser ceph since I am not root 2024-04-25T01:58:37.010 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:58:36 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e[61957]: ignoring --setgroup ceph since I am not root 2024-04-25T01:58:37.010 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:58:36 smithi040 ceph-mds[61961]: -- 172.21.15.40:0/83075158 <== mon.2 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56199ac33a40 con 0x56199ac06c00 2024-04-25T01:58:37.010 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 01:58:36 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e[61957]: starting mds.e at 2024-04-25T01:58:37.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 81725 2.68 MB/sec execute 44 sec latency 21.679 ms 2024-04-25T01:58:37.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:37 smithi040 ceph-mon[32418]: daemon mds.e restarted 2024-04-25T01:58:37.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:37 smithi134 ceph-mon[33098]: daemon mds.e restarted 2024-04-25T01:58:37.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:37 smithi031 ceph-mon[29204]: daemon mds.e restarted 2024-04-25T01:58:38.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 82013 2.66 MB/sec execute 45 sec latency 17.610 ms 2024-04-25T01:58:38.342 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:38 smithi134 ceph-mon[33098]: pgmap v127: 97 pgs: 97 active+clean; 56 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:58:38.342 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:38 smithi134 ceph-mon[33098]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T01:58:38.342 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:38 smithi134 ceph-mon[33098]: mds.? [v2:172.21.15.40:6834/4184412222,v1:172.21.15.40:6835/4184412222] up:boot 2024-04-25T01:58:38.343 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:38 smithi134 ceph-mon[33098]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T01:58:38.343 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:38 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:58:38.343 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:38 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:58:38.343 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:38 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:58:38.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:38 smithi031 ceph-mon[29204]: pgmap v127: 97 pgs: 97 active+clean; 56 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:58:38.392 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:38 smithi031 ceph-mon[29204]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T01:58:38.392 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:38 smithi031 ceph-mon[29204]: mds.? [v2:172.21.15.40:6834/4184412222,v1:172.21.15.40:6835/4184412222] up:boot 2024-04-25T01:58:38.392 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:38 smithi031 ceph-mon[29204]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T01:58:38.392 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:38 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:58:38.392 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:38 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:58:38.392 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:38 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:58:38.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:38 smithi040 ceph-mon[32418]: pgmap v127: 97 pgs: 97 active+clean; 56 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-25T01:58:38.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:38 smithi040 ceph-mon[32418]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T01:58:38.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:38 smithi040 ceph-mon[32418]: mds.? [v2:172.21.15.40:6834/4184412222,v1:172.21.15.40:6835/4184412222] up:boot 2024-04-25T01:58:38.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:38 smithi040 ceph-mon[32418]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T01:58:38.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:38 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:58:38.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:38 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:58:38.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:38 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:58:39.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 82303 2.60 MB/sec execute 46 sec latency 18.231 ms 2024-04-25T01:58:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:39 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:58:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:39 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:58:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:39 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:58:39.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:39 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:58:39.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:39 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:58:39.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:39 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:58:39.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:39 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:58:39.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:39 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:58:39.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:39 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:58:40.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 82811 2.59 MB/sec execute 47 sec latency 21.082 ms 2024-04-25T01:58:40.313 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:40 smithi040 ceph-mon[32418]: pgmap v129: 97 pgs: 97 active+clean; 56 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 150 op/s 2024-04-25T01:58:40.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:40 smithi134 ceph-mon[33098]: pgmap v129: 97 pgs: 97 active+clean; 56 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 150 op/s 2024-04-25T01:58:40.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:40 smithi031 ceph-mon[29204]: pgmap v129: 97 pgs: 97 active+clean; 56 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 150 op/s 2024-04-25T01:58:41.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 83413 2.60 MB/sec execute 48 sec latency 19.114 ms 2024-04-25T01:58:42.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 84028 2.58 MB/sec execute 49 sec latency 13.338 ms 2024-04-25T01:58:42.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:42 smithi040 ceph-mon[32418]: pgmap v130: 97 pgs: 97 active+clean; 57 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-04-25T01:58:42.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:42 smithi134 ceph-mon[33098]: pgmap v130: 97 pgs: 97 active+clean; 57 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-04-25T01:58:42.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:42 smithi031 ceph-mon[29204]: pgmap v130: 97 pgs: 97 active+clean; 57 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-04-25T01:58:43.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 84634 2.64 MB/sec execute 50 sec latency 13.446 ms 2024-04-25T01:58:44.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 85169 2.68 MB/sec execute 51 sec latency 13.316 ms 2024-04-25T01:58:44.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:44 smithi040 ceph-mon[32418]: pgmap v131: 97 pgs: 97 active+clean; 57 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-04-25T01:58:44.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:44 smithi134 ceph-mon[33098]: pgmap v131: 97 pgs: 97 active+clean; 57 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-04-25T01:58:44.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:44 smithi031 ceph-mon[29204]: pgmap v131: 97 pgs: 97 active+clean; 57 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 155 op/s 2024-04-25T01:58:45.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 85541 2.67 MB/sec execute 52 sec latency 14.571 ms 2024-04-25T01:58:46.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 85825 2.62 MB/sec execute 53 sec latency 11.808 ms 2024-04-25T01:58:46.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:46 smithi040 ceph-mon[32418]: pgmap v132: 97 pgs: 97 active+clean; 58 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-04-25T01:58:46.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:46 smithi134 ceph-mon[33098]: pgmap v132: 97 pgs: 97 active+clean; 58 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-04-25T01:58:46.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:46 smithi031 ceph-mon[29204]: pgmap v132: 97 pgs: 97 active+clean; 58 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-04-25T01:58:47.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 86331 2.60 MB/sec execute 54 sec latency 19.058 ms 2024-04-25T01:58:48.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 86886 2.62 MB/sec execute 55 sec latency 13.717 ms 2024-04-25T01:58:48.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:48 smithi040 ceph-mon[32418]: pgmap v133: 97 pgs: 97 active+clean; 58 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-04-25T01:58:48.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:48 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:58:48.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:48 smithi134 ceph-mon[33098]: pgmap v133: 97 pgs: 97 active+clean; 58 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-04-25T01:58:48.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:48 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:58:48.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:48 smithi031 ceph-mon[29204]: pgmap v133: 97 pgs: 97 active+clean; 58 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-04-25T01:58:48.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:48 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:58:49.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 87544 2.60 MB/sec execute 56 sec latency 13.964 ms 2024-04-25T01:58:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.424+0000 7f2703f77640 1 -- 172.21.15.31:0/2940484162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc107740 msgr2=0x7f26fc107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.424+0000 7f2703f77640 1 --2- 172.21.15.31:0/2940484162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc107740 0x7f26fc107b20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009cd0 tx=0x7f26ec02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.424+0000 7f2703f77640 1 -- 172.21.15.31:0/2940484162 shutdown_connections 2024-04-25T01:58:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.424+0000 7f2703f77640 1 --2- 172.21.15.31:0/2940484162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26fc1020f0 0x7f26fc10e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.424+0000 7f2703f77640 1 --2- 172.21.15.31:0/2940484162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26fc101730 0x7f26fc101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.424+0000 7f2703f77640 1 --2- 172.21.15.31:0/2940484162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc107740 0x7f26fc107b20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.424+0000 7f2703f77640 1 -- 172.21.15.31:0/2940484162 >> 172.21.15.31:0/2940484162 conn(0x7f26fc0fd200 msgr2=0x7f26fc0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.424+0000 7f2703f77640 1 -- 172.21.15.31:0/2940484162 shutdown_connections 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.424+0000 7f2703f77640 1 -- 172.21.15.31:0/2940484162 wait complete. 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2703f77640 1 Processor -- start 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2703f77640 1 -- start start 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2703f77640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26fc101730 0x7f26fc073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2703f77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26fc1020f0 0x7f26fc073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2703f77640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc107740 0x7f26fc0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2703f77640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1119c0 con 0x7f26fc107740 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2703f77640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc111840 con 0x7f26fc1020f0 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2703f77640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc111b40 con 0x7f26fc101730 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2702514640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc107740 0x7f26fc0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2701d13640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26fc101730 0x7f26fc073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2701512640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26fc1020f0 0x7f26fc073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2702514640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc107740 0x7f26fc0744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52586/0 (socket says 172.21.15.31:52586) 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2701d13640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26fc101730 0x7f26fc073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:57300/0 (socket says 172.21.15.31:57300) 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2701512640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26fc1020f0 0x7f26fc073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56456/0 (socket says 172.21.15.31:56456) 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2702514640 1 -- 172.21.15.31:0/699529347 learned_addr learned my addr 172.21.15.31:0/699529347 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2702514640 1 -- 172.21.15.31:0/699529347 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26fc101730 msgr2=0x7f26fc073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2702514640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26fc101730 0x7f26fc073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2702514640 1 -- 172.21.15.31:0/699529347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26fc1020f0 msgr2=0x7f26fc073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2702514640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26fc1020f0 0x7f26fc073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.425+0000 7f2702514640 1 -- 172.21.15.31:0/699529347 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc1a63c0 con 0x7f26fc107740 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.426+0000 7f2702514640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc107740 0x7f26fc0744c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f26f400e9b0 tx=0x7f26f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.426+0000 7f26faffd640 1 -- 172.21.15.31:0/699529347 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f400cd80 con 0x7f26fc107740 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.426+0000 7f2703f77640 1 -- 172.21.15.31:0/699529347 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26fc1a66b0 con 0x7f26fc107740 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.426+0000 7f26faffd640 1 -- 172.21.15.31:0/699529347 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26f40035f0 con 0x7f26fc107740 2024-04-25T01:58:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.426+0000 7f26faffd640 1 -- 172.21.15.31:0/699529347 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4010690 con 0x7f26fc107740 2024-04-25T01:58:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.426+0000 7f2703f77640 1 -- 172.21.15.31:0/699529347 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26fc1a6bb0 con 0x7f26fc107740 2024-04-25T01:58:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f26faffd640 1 -- 172.21.15.31:0/699529347 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f26f40037a0 con 0x7f26fc107740 2024-04-25T01:58:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2706d64740 1 Processor -- start 2024-04-25T01:58:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2706d64740 1 -- start start 2024-04-25T01:58:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2706d64740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6b240 0x558002a6b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2706d64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a75cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2706d64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 0x558002a78620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2706d64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5580029cfcb0 con 0x558002a76230 2024-04-25T01:58:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2706d64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5580029785b0 con 0x558002a6bb60 2024-04-25T01:58:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2706d64740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5580029cbc70 con 0x558002a6b240 2024-04-25T01:58:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2701d13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a75cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2701512640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 0x558002a78620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2701d13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a75cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56472/0 (socket says 172.21.15.31:56472) 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2701d13640 1 -- 172.21.15.31:0/687095676 learned_addr learned my addr 172.21.15.31:0/687095676 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f2701512640 1 --2- 172.21.15.31:0/687095676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6b240 0x558002a6b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.429+0000 7f26faffd640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f26e40a2da0 0x7f26e40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.430+0000 7f2701d13640 1 -- 172.21.15.31:0/687095676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6b240 msgr2=0x558002a6b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.430+0000 7f2701d13640 1 --2- 172.21.15.31:0/687095676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6b240 0x558002a6b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.430+0000 7f2701d13640 1 -- 172.21.15.31:0/687095676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 msgr2=0x558002a78620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.430+0000 7f2701d13640 1 --2- 172.21.15.31:0/687095676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 0x558002a78620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.430+0000 7f2701d13640 1 -- 172.21.15.31:0/687095676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580026e4210 con 0x558002a6bb60 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.430+0000 7f2702514640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f26e40a2da0 0x7f26e40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.430+0000 7f2701d13640 1 --2- 172.21.15.31:0/687095676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a75cf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009cd0 tx=0x7f26ec009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5fae99f6017a2ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.430+0000 7f26faffd640 1 -- 172.21.15.31:0/699529347 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f26f40cf6c0 con 0x7f26fc107740 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.430+0000 7f26e3fff640 1 -- 172.21.15.31:0/687095676 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec049070 con 0x558002a6bb60 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.430+0000 7f26e3fff640 1 -- 172.21.15.31:0/687095676 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26ec036040 con 0x558002a6bb60 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.430+0000 7f26e3fff640 1 -- 172.21.15.31:0/687095676 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec003e00 con 0x558002a6bb60 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.431+0000 7f2706d64740 1 -- 172.21.15.31:0/687095676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 msgr2=0x558002a75cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.431+0000 7f2702514640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f26e40a2da0 0x7f26e40a5260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f26f40ccf10 tx=0x7f26f401b6d0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.431+0000 7f2706d64740 1 --2- 172.21.15.31:0/687095676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a75cf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009cd0 tx=0x7f26ec009360 comp rx=0 tx=0).stop 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.431+0000 7f2706d64740 1 -- 172.21.15.31:0/687095676 shutdown_connections 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.431+0000 7f2706d64740 1 --2- 172.21.15.31:0/687095676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 0x558002a78620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.431+0000 7f2706d64740 1 --2- 172.21.15.31:0/687095676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a75cf0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.431+0000 7f2706d64740 1 --2- 172.21.15.31:0/687095676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6b240 0x558002a6b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.431+0000 7f2706d64740 1 -- 172.21.15.31:0/687095676 >> 172.21.15.31:0/687095676 conn(0x558002a67560 msgr2=0x558002a67940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.431+0000 7f2706d64740 1 -- 172.21.15.31:0/687095676 shutdown_connections 2024-04-25T01:58:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.431+0000 7f2706d64740 1 -- 172.21.15.31:0/687095676 wait complete. 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.431+0000 7f2706d64740 1 -- start start 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2706d64740 1 Processor -- start 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2706d64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a731f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2706d64740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6bb60 0x558002a73730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2706d64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a76230 0x558002a73c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2706d64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558002a78fc0 con 0x558002a6b240 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2706d64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55800298bb50 con 0x558002a76230 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2706d64740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x558002a79140 con 0x558002a6bb60 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2706d64740 10 client.0 _ll_register_callbacks cb 0x5580029c6ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2701512640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2701512640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a731f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52600/0 (socket says 172.21.15.31:52600) 2024-04-25T01:58:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2701512640 1 -- 172.21.15.31:0/4004557830 learned_addr learned my addr 172.21.15.31:0/4004557830 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2701512640 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a76230 0x558002a73c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2701d13640 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6bb60 0x558002a73730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2701512640 1 -- 172.21.15.31:0/4004557830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6bb60 msgr2=0x558002a73730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2701512640 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6bb60 0x558002a73730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2701512640 1 -- 172.21.15.31:0/4004557830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a76230 msgr2=0x558002a73c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2701512640 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a76230 0x558002a73c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.432+0000 7f2701512640 1 -- 172.21.15.31:0/4004557830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580027b0620 con 0x558002a6b240 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.433+0000 7f2701512640 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a731f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f26f000e3b0 tx=0x7f26f0009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439b51d585c73429 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.433+0000 7f26e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.433+0000 7f26e17fa640 1 -- 172.21.15.31:0/4004557830 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f0026060 con 0x558002a6b240 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.433+0000 7f26e17fa640 1 -- 172.21.15.31:0/4004557830 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f26f001f470 con 0x558002a6b240 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.433+0000 7f2706d64740 1 -- 172.21.15.31:0/4004557830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26ec036040 con 0x558002a6b240 2024-04-25T01:58:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.433+0000 7f26e17fa640 1 -- 172.21.15.31:0/4004557830 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f26f001f670 con 0x558002a6b240 2024-04-25T01:58:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.433+0000 7f2706d64740 10 client.16494 fetch_fsmap learned FSMap version 33 2024-04-25T01:58:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.433+0000 7f2706d64740 1 -- 172.21.15.31:0/4004557830 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558002a85200 con 0x558002a6b240 2024-04-25T01:58:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.433+0000 7f26e17fa640 1 -- 172.21.15.31:0/4004557830 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26ec036040 con 0x558002a6b240 2024-04-25T01:58:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.433+0000 7f26e17fa640 1 -- 172.21.15.31:0/4004557830 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f26f00206a0 con 0x558002a6b240 2024-04-25T01:58:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.434+0000 7f2706d64740 10 client.16494 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:58:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.434+0000 7f2706d64740 10 client.16494 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.434+0000 7f2706d64740 20 client.16494 populate_metadata read hostname 'smithi031' 2024-04-25T01:58:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.434+0000 7f2706d64740 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558002a866c0 0x558002a88ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.434+0000 7f2706d64740 4 client.16494 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.434+0000 7f2706d64740 1 -- 172.21.15.31:0/4004557830 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580029dd1f0 con 0x558002a866c0 2024-04-25T01:58:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.434+0000 7f2701d13640 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558002a866c0 0x558002a88ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.434+0000 7f2701d13640 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558002a866c0 0x558002a88ab0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=132d179d9e1fc80 server_cookie=71cadfd63530f2d3 in_seq=0 out_seq=0 2024-04-25T01:58:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.434+0000 7f26e17fa640 10 client.16494 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f26e17fa640 1 -- 172.21.15.31:0/4004557830 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5580029dd1f0 con 0x558002a866c0 2024-04-25T01:58:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f26e17fa640 10 client.16494 handle_command_reply: tid=0 2024-04-25T01:58:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 1 client.16494 shutdown 2024-04-25T01:58:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 1 -- 172.21.15.31:0/4004557830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 msgr2=0x558002a731f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a731f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f26f000e3b0 tx=0x7f26f0009aa0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 1 -- 172.21.15.31:0/4004557830 shutdown_connections 2024-04-25T01:58:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558002a866c0 0x558002a88ab0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a76230 0x558002a73c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6bb60 0x558002a73730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 1 --2- 172.21.15.31:0/4004557830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a731f0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f26e17fa640 0 client.16494 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 1 -- 172.21.15.31:0/4004557830 >> 172.21.15.31:0/4004557830 conn(0x558002a67560 msgr2=0x558002a6d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 1 -- 172.21.15.31:0/4004557830 shutdown_connections 2024-04-25T01:58:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 1 -- 172.21.15.31:0/4004557830 wait complete. 2024-04-25T01:58:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.435+0000 7f2706d64740 20 client.16494 trim_cache size 0 max 16384 2024-04-25T01:58:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.452+0000 7f2706d64740 1 Processor -- start 2024-04-25T01:58:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.452+0000 7f2706d64740 1 -- start start 2024-04-25T01:58:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.452+0000 7f2706d64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a81e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.452+0000 7f2706d64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a6d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.452+0000 7f2706d64740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a76230 0x558002a66440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.452+0000 7f2706d64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1af3b0 con 0x558002a6b240 2024-04-25T01:58:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.452+0000 7f2706d64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1b0ef0 con 0x558002a6bb60 2024-04-25T01:58:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.452+0000 7f2706d64740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1afa80 con 0x558002a76230 2024-04-25T01:58:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701512640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a81e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701d13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a6d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701512640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a81e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52610/0 (socket says 172.21.15.31:52610) 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701d13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a6d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:56482/0 (socket says 172.21.15.31:56482) 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701512640 1 -- 172.21.15.31:0/111053394 learned_addr learned my addr 172.21.15.31:0/111053394 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701512640 1 --2- 172.21.15.31:0/111053394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a76230 0x558002a66440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701512640 1 -- 172.21.15.31:0/111053394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a76230 msgr2=0x558002a66440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701512640 1 --2- 172.21.15.31:0/111053394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a76230 0x558002a66440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701512640 1 -- 172.21.15.31:0/111053394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 msgr2=0x558002a6d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701512640 1 --2- 172.21.15.31:0/111053394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a6d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701512640 1 -- 172.21.15.31:0/111053394 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558002a88e20 con 0x558002a6b240 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f2701512640 1 --2- 172.21.15.31:0/111053394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a81e80 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f26f0000c00 tx=0x7f26f0011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e40c12341c6c05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f26e37fe640 1 -- 172.21.15.31:0/111053394 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f0026070 con 0x558002a6b240 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f26e37fe640 1 -- 172.21.15.31:0/111053394 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26f0005d30 con 0x558002a6b240 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.453+0000 7f26e37fe640 1 -- 172.21.15.31:0/111053394 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f0005350 con 0x558002a6b240 2024-04-25T01:58:49.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.454+0000 7f2706d64740 1 -- 172.21.15.31:0/111053394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 msgr2=0x558002a81e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.454+0000 7f2706d64740 1 --2- 172.21.15.31:0/111053394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a81e80 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f26f0000c00 tx=0x7f26f0011ce0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.454+0000 7f2706d64740 1 -- 172.21.15.31:0/111053394 shutdown_connections 2024-04-25T01:58:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.454+0000 7f2706d64740 1 --2- 172.21.15.31:0/111053394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a76230 0x558002a66440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.454+0000 7f2706d64740 1 --2- 172.21.15.31:0/111053394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a6d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.454+0000 7f2706d64740 1 --2- 172.21.15.31:0/111053394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a6b240 0x558002a81e80 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.454+0000 7f2706d64740 1 -- 172.21.15.31:0/111053394 >> 172.21.15.31:0/111053394 conn(0x558002a67560 msgr2=0x558002a65d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.454+0000 7f2706d64740 1 -- 172.21.15.31:0/111053394 shutdown_connections 2024-04-25T01:58:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.454+0000 7f2706d64740 1 -- 172.21.15.31:0/111053394 wait complete. 2024-04-25T01:58:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.454+0000 7f2706d64740 1 -- start start 2024-04-25T01:58:49.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.454+0000 7f2706d64740 1 Processor -- start 2024-04-25T01:58:49.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2706d64740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6b240 0x558002a6ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2706d64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a704c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2706d64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 0x558002a70a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2706d64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1b0820 con 0x558002a76230 2024-04-25T01:58:49.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2706d64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1b0150 con 0x558002a6bb60 2024-04-25T01:58:49.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2706d64740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1ad3b0 con 0x558002a6b240 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2706d64740 10 client.0 _ll_register_callbacks cb 0x7f26f0011580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2701512640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 0x558002a70a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2701d13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2701512640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 0x558002a70a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52620/0 (socket says 172.21.15.31:52620) 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2701512640 1 -- 172.21.15.31:0/139206421 learned_addr learned my addr 172.21.15.31:0/139206421 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2701512640 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6b240 0x558002a6ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2701512640 1 -- 172.21.15.31:0/139206421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6b240 msgr2=0x558002a6ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2701512640 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6b240 0x558002a6ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2701512640 1 -- 172.21.15.31:0/139206421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 msgr2=0x558002a704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2701512640 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a704c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.455+0000 7f2701512640 1 -- 172.21.15.31:0/139206421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558002a78990 con 0x558002a76230 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f2701512640 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 0x558002a70a00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f26f0004000 tx=0x7f26f0020d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7561f47a8e6d1b8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:49.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f26e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f26e17fa640 1 -- 172.21.15.31:0/139206421 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f0026040 con 0x558002a76230 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f2706d64740 1 -- 172.21.15.31:0/139206421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26f0005d30 con 0x558002a76230 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f26e17fa640 1 -- 172.21.15.31:0/139206421 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f26f001f470 con 0x558002a76230 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f26e17fa640 1 -- 172.21.15.31:0/139206421 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f26f0009e70 con 0x558002a76230 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f2706d64740 10 client.16500 fetch_fsmap learned FSMap version 33 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f2706d64740 1 -- 172.21.15.31:0/139206421 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558002aecde0 con 0x558002a76230 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f26e17fa640 1 -- 172.21.15.31:0/139206421 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26f0005d30 con 0x558002a76230 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f26e17fa640 1 -- 172.21.15.31:0/139206421 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f26f0020680 con 0x558002a76230 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f2706d64740 10 client.16500 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f2706d64740 10 client.16500 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f2706d64740 20 client.16500 populate_metadata read hostname 'smithi031' 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f2706d64740 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558002aed390 0x558002aef780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f2706d64740 4 client.16500 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:58:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.456+0000 7f2706d64740 1 -- 172.21.15.31:0/139206421 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5580029d7540 con 0x558002aed390 2024-04-25T01:58:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.457+0000 7f2701d13640 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558002aed390 0x558002aef780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.457+0000 7f2701d13640 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558002aed390 0x558002aef780 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7860a4934028d0dc server_cookie=d1c261cb1a3e8d6 in_seq=0 out_seq=0 2024-04-25T01:58:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.457+0000 7f26e17fa640 10 client.16500 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f26e17fa640 1 -- 172.21.15.31:0/139206421 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5580029d7540 con 0x558002aed390 2024-04-25T01:58:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f26e17fa640 10 client.16500 handle_command_reply: tid=0 2024-04-25T01:58:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 1 client.16500 shutdown 2024-04-25T01:58:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 1 -- 172.21.15.31:0/139206421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 msgr2=0x558002a70a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 0x558002a70a00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f26f0004000 tx=0x7f26f0020d50 comp rx=0 tx=0).stop 2024-04-25T01:58:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 1 -- 172.21.15.31:0/139206421 shutdown_connections 2024-04-25T01:58:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558002aed390 0x558002aef780 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558002a76230 0x558002a70a00 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558002a6bb60 0x558002a704c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 1 --2- 172.21.15.31:0/139206421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558002a6b240 0x558002a6ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f26e17fa640 0 client.16500 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:58:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 1 -- 172.21.15.31:0/139206421 >> 172.21.15.31:0/139206421 conn(0x558002a67560 msgr2=0x558002a665c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 1 -- 172.21.15.31:0/139206421 shutdown_connections 2024-04-25T01:58:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 1 -- 172.21.15.31:0/139206421 wait complete. 2024-04-25T01:58:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.476+0000 7f2706d64740 20 client.16500 trim_cache size 0 max 16384 2024-04-25T01:58:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:58:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:58:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:58:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:58:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 -- 172.21.15.31:0/699529347 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f26e40a2da0 msgr2=0x7f26e40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f26e40a2da0 0x7f26e40a5260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f26f40ccf10 tx=0x7f26f401b6d0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 -- 172.21.15.31:0/699529347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc107740 msgr2=0x7f26fc0744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc107740 0x7f26fc0744c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f26f400e9b0 tx=0x7f26f400ee70 comp rx=0 tx=0).stop 2024-04-25T01:58:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 -- 172.21.15.31:0/699529347 shutdown_connections 2024-04-25T01:58:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f26e40a2da0 0x7f26e40a5260 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26fc107740 0x7f26fc0744c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26fc1020f0 0x7f26fc073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 --2- 172.21.15.31:0/699529347 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26fc101730 0x7f26fc073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 -- 172.21.15.31:0/699529347 >> 172.21.15.31:0/699529347 conn(0x7f26fc0fd200 msgr2=0x7f26fc0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 -- 172.21.15.31:0/699529347 shutdown_connections 2024-04-25T01:58:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:58:49.477+0000 7f2703f77640 1 -- 172.21.15.31:0/699529347 wait complete. 2024-04-25T01:58:50.305 INFO:tasks.workunit.client.0.smithi031.stdout: 1 88102 2.65 MB/sec execute 57 sec latency 21.982 ms 2024-04-25T01:58:50.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:50 smithi040 ceph-mon[32418]: pgmap v134: 97 pgs: 97 active+clean; 58 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-04-25T01:58:50.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:50 smithi134 ceph-mon[33098]: pgmap v134: 97 pgs: 97 active+clean; 58 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-04-25T01:58:50.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:50 smithi031 ceph-mon[29204]: pgmap v134: 97 pgs: 97 active+clean; 58 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-04-25T01:58:51.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 88547 2.64 MB/sec execute 58 sec latency 18.847 ms 2024-04-25T01:58:52.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 89022 2.67 MB/sec execute 59 sec latency 15.627 ms 2024-04-25T01:58:52.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:52 smithi040 ceph-mon[32418]: pgmap v135: 97 pgs: 97 active+clean; 58 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-04-25T01:58:52.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:52 smithi134 ceph-mon[33098]: pgmap v135: 97 pgs: 97 active+clean; 58 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-04-25T01:58:52.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:52 smithi031 ceph-mon[29204]: pgmap v135: 97 pgs: 97 active+clean; 58 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.9 MiB/s wr, 190 op/s 2024-04-25T01:58:53.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 89284 2.63 MB/sec execute 60 sec latency 18.313 ms 2024-04-25T01:58:54.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 89713 2.60 MB/sec execute 61 sec latency 19.429 ms 2024-04-25T01:58:54.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:54 smithi040 ceph-mon[32418]: pgmap v136: 97 pgs: 97 active+clean; 58 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-25T01:58:54.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:54 smithi134 ceph-mon[33098]: pgmap v136: 97 pgs: 97 active+clean; 58 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-25T01:58:54.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:54 smithi031 ceph-mon[29204]: pgmap v136: 97 pgs: 97 active+clean; 58 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-25T01:58:55.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 90263 2.63 MB/sec execute 62 sec latency 19.326 ms 2024-04-25T01:58:56.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 90929 2.60 MB/sec execute 63 sec latency 13.403 ms 2024-04-25T01:58:56.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:56 smithi040 ceph-mon[32418]: pgmap v137: 97 pgs: 97 active+clean; 63 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-25T01:58:56.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:56 smithi134 ceph-mon[33098]: pgmap v137: 97 pgs: 97 active+clean; 63 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-25T01:58:56.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:56 smithi031 ceph-mon[29204]: pgmap v137: 97 pgs: 97 active+clean; 63 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-25T01:58:57.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 91488 2.64 MB/sec execute 64 sec latency 13.291 ms 2024-04-25T01:58:58.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 92039 2.65 MB/sec execute 65 sec latency 20.880 ms 2024-04-25T01:58:58.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:58:58 smithi040 ceph-mon[32418]: pgmap v138: 97 pgs: 97 active+clean; 63 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:58:58.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:58:58 smithi134 ceph-mon[33098]: pgmap v138: 97 pgs: 97 active+clean; 63 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:58:58.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:58:58 smithi031 ceph-mon[29204]: pgmap v138: 97 pgs: 97 active+clean; 63 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:58:59.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 92499 2.68 MB/sec execute 66 sec latency 18.314 ms 2024-04-25T01:59:00.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 92743 2.64 MB/sec execute 67 sec latency 12.611 ms 2024-04-25T01:59:00.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:00 smithi040 ceph-mon[32418]: pgmap v139: 97 pgs: 97 active+clean; 63 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:59:00.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:00 smithi134 ceph-mon[33098]: pgmap v139: 97 pgs: 97 active+clean; 63 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:59:00.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:00 smithi031 ceph-mon[29204]: pgmap v139: 97 pgs: 97 active+clean; 63 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-25T01:59:01.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 93178 2.61 MB/sec execute 68 sec latency 18.765 ms 2024-04-25T01:59:02.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 93724 2.64 MB/sec execute 69 sec latency 16.614 ms 2024-04-25T01:59:02.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:02 smithi040 ceph-mon[32418]: pgmap v140: 97 pgs: 97 active+clean; 63 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 1.8 MiB/s wr, 189 op/s 2024-04-25T01:59:02.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:02 smithi134 ceph-mon[33098]: pgmap v140: 97 pgs: 97 active+clean; 63 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 1.8 MiB/s wr, 189 op/s 2024-04-25T01:59:02.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:02 smithi031 ceph-mon[29204]: pgmap v140: 97 pgs: 97 active+clean; 63 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 1.8 MiB/s wr, 189 op/s 2024-04-25T01:59:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.581+0000 7fe008bd5640 1 -- 172.21.15.31:0/2128167719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0041032f0 msgr2=0x7fe00410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.581+0000 7fe008bd5640 1 --2- 172.21.15.31:0/2128167719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0041032f0 0x7fe00410b7d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009900 tx=0x7fdff802a510 comp rx=0 tx=0).stop 2024-04-25T01:59:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.581+0000 7fe008bd5640 1 -- 172.21.15.31:0/2128167719 shutdown_connections 2024-04-25T01:59:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.581+0000 7fe008bd5640 1 --2- 172.21.15.31:0/2128167719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe00410bd10 0x7fe00410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.581+0000 7fe008bd5640 1 --2- 172.21.15.31:0/2128167719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0041032f0 0x7fe00410b7d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.581+0000 7fe008bd5640 1 --2- 172.21.15.31:0/2128167719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0041029d0 0x7fe004102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.581+0000 7fe008bd5640 1 -- 172.21.15.31:0/2128167719 >> 172.21.15.31:0/2128167719 conn(0x7fe0040777c0 msgr2=0x7fe004077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.581+0000 7fe008bd5640 1 -- 172.21.15.31:0/2128167719 shutdown_connections 2024-04-25T01:59:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.581+0000 7fe008bd5640 1 -- 172.21.15.31:0/2128167719 wait complete. 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.582+0000 7fe008bd5640 1 Processor -- start 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.582+0000 7fe008bd5640 1 -- start start 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe008bd5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0041029d0 0x7fe004197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe008bd5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0041032f0 0x7fe004198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe008bd5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe00410bd10 0x7fe00419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe008bd5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0041119c0 con 0x7fe0041032f0 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe008bd5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe004111840 con 0x7fe0041029d0 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe008bd5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe004111b40 con 0x7fe00410bd10 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe001d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0041032f0 0x7fe004198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe001d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0041032f0 0x7fe004198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45234/0 (socket says 172.21.15.31:45234) 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe001d9b640 1 -- 172.21.15.31:0/3673972326 learned_addr learned my addr 172.21.15.31:0/3673972326 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe00259c640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0041029d0 0x7fe004197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe002d9d640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe00410bd10 0x7fe00419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe001d9b640 1 -- 172.21.15.31:0/3673972326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe00410bd10 msgr2=0x7fe00419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe001d9b640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe00410bd10 0x7fe00419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe001d9b640 1 -- 172.21.15.31:0/3673972326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0041029d0 msgr2=0x7fe004197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe001d9b640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0041029d0 0x7fe004197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe001d9b640 1 -- 172.21.15.31:0/3673972326 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe00419ceb0 con 0x7fe0041032f0 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe00259c640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0041029d0 0x7fe004197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fe001d9b640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0041032f0 0x7fe004198400 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdff8005940 tx=0x7fdff80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.583+0000 7fdff37fe640 1 -- 172.21.15.31:0/3673972326 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff803f030 con 0x7fe0041032f0 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.584+0000 7fe008bd5640 1 -- 172.21.15.31:0/3673972326 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe00419d140 con 0x7fe0041032f0 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.584+0000 7fdff37fe640 1 -- 172.21.15.31:0/3673972326 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdff80044c0 con 0x7fe0041032f0 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.584+0000 7fdff37fe640 1 -- 172.21.15.31:0/3673972326 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff8035070 con 0x7fe0041032f0 2024-04-25T01:59:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.584+0000 7fe008bd5640 1 -- 172.21.15.31:0/3673972326 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0041a4a30 con 0x7fe0041032f0 2024-04-25T01:59:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.586+0000 7fdff37fe640 1 -- 172.21.15.31:0/3673972326 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fdff8042030 con 0x7fe0041032f0 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.586+0000 7fe00b9b6740 1 Processor -- start 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00b9b6740 1 -- start start 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5d5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5dfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5e0230 0x56238c5e2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56238c539cb0 con 0x56238c5d5240 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56238c4e25b0 con 0x56238c5d5b60 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56238c535c70 con 0x56238c5e0230 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fdff37fe640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfec0a2da0 0x7fdfec0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5dfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe001d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5e0230 0x56238c5e2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe002d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5d5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5dfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59022/0 (socket says 172.21.15.31:59022) 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00259c640 1 -- 172.21.15.31:0/1048088025 learned_addr learned my addr 172.21.15.31:0/1048088025 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe002d9d640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfec0a2da0 0x7fdfec0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fdff37fe640 1 -- 172.21.15.31:0/3673972326 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fdff80ef770 con 0x7fe0041032f0 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00259c640 1 -- 172.21.15.31:0/1048088025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5e0230 msgr2=0x56238c5e2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00259c640 1 --2- 172.21.15.31:0/1048088025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5e0230 0x56238c5e2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00259c640 1 -- 172.21.15.31:0/1048088025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 msgr2=0x56238c5d5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00259c640 1 --2- 172.21.15.31:0/1048088025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5d5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.587+0000 7fe00259c640 1 -- 172.21.15.31:0/1048088025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56238c24e210 con 0x56238c5d5b60 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.588+0000 7fe00259c640 1 --2- 172.21.15.31:0/1048088025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5dfcf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdff40115f0 tx=0x7fdff4011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e354fc7f33c9b508 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.588+0000 7fe002d9d640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfec0a2da0 0x7fdfec0a5260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdffc005fd0 tx=0x7fdffc009500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.588+0000 7fdff0ff9640 1 -- 172.21.15.31:0/1048088025 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff401e070 con 0x56238c5d5b60 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.588+0000 7fdff0ff9640 1 -- 172.21.15.31:0/1048088025 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdff400faf0 con 0x56238c5d5b60 2024-04-25T01:59:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.588+0000 7fdff0ff9640 1 -- 172.21.15.31:0/1048088025 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4019ac0 con 0x56238c5d5b60 2024-04-25T01:59:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.589+0000 7fe00b9b6740 1 -- 172.21.15.31:0/1048088025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 msgr2=0x56238c5dfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.589+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/1048088025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5dfcf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdff40115f0 tx=0x7fdff4011ab0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.589+0000 7fe00b9b6740 1 -- 172.21.15.31:0/1048088025 shutdown_connections 2024-04-25T01:59:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.589+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/1048088025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5e0230 0x56238c5e2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.589+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/1048088025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5dfcf0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.589+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/1048088025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5d5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.589+0000 7fe00b9b6740 1 -- 172.21.15.31:0/1048088025 >> 172.21.15.31:0/1048088025 conn(0x56238c5d1560 msgr2=0x56238c5d1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.589+0000 7fe00b9b6740 1 -- 172.21.15.31:0/1048088025 shutdown_connections 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.589+0000 7fe00b9b6740 1 -- 172.21.15.31:0/1048088025 wait complete. 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.589+0000 7fe00b9b6740 1 -- start start 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.589+0000 7fe00b9b6740 1 Processor -- start 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5d5240 0x56238c5dd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5b60 0x56238c5dd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 0x56238c5ddc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56238c5e3020 con 0x56238c5d5b60 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56238c4f5b50 con 0x56238c5e0230 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56238c5e31a0 con 0x56238c5d5240 2024-04-25T01:59:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe00b9b6740 10 client.0 _ll_register_callbacks cb 0x56238c530ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe002d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 0x56238c5ddc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe002d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 0x56238c5ddc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59024/0 (socket says 172.21.15.31:59024) 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe002d9d640 1 -- 172.21.15.31:0/660717157 learned_addr learned my addr 172.21.15.31:0/660717157 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe001d9b640 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5b60 0x56238c5dd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe00259c640 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5d5240 0x56238c5dd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe002d9d640 1 -- 172.21.15.31:0/660717157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5d5240 msgr2=0x56238c5dd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe002d9d640 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5d5240 0x56238c5dd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe002d9d640 1 -- 172.21.15.31:0/660717157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5b60 msgr2=0x56238c5dd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe002d9d640 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5b60 0x56238c5dd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe002d9d640 1 -- 172.21.15.31:0/660717157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56238c31a620 con 0x56238c5e0230 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fe002d9d640 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 0x56238c5ddc60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdffc00ff50 tx=0x7fdffc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea8a24057c60fcce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.590+0000 7fdfe9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/660717157 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdffc021e40 con 0x56238c5e0230 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/660717157 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fdffc010430 con 0x56238c5e0230 2024-04-25T01:59:02.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fe00b9b6740 1 -- 172.21.15.31:0/660717157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdff400faf0 con 0x56238c5e0230 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/660717157 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdffc010630 con 0x56238c5e0230 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fe00b9b6740 10 client.26114 fetch_fsmap learned FSMap version 33 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fe00b9b6740 1 -- 172.21.15.31:0/660717157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56238c5f36b0 con 0x56238c5e0230 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/660717157 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdffc010c80 con 0x56238c5e0230 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/660717157 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fdffc028070 con 0x56238c5e0230 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fe00b9b6740 10 client.26114 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fe00b9b6740 10 client.26114 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fe00b9b6740 20 client.26114 populate_metadata read hostname 'smithi031' 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.591+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56238c5f4bc0 0x56238c5f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.592+0000 7fe00b9b6740 4 client.26114 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.592+0000 7fe00b9b6740 1 -- 172.21.15.31:0/660717157 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdff400faf0 con 0x56238c5f4bc0 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.592+0000 7fe00259c640 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56238c5f4bc0 0x56238c5f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.592+0000 7fe00259c640 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56238c5f4bc0 0x56238c5f6fb0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d44fe0463ede5226 server_cookie=af1ab717c96cf5be in_seq=0 out_seq=0 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.592+0000 7fdfe9ffb640 10 client.26114 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:02.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/660717157 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fdff400faf0 con 0x56238c5f4bc0 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fdfe9ffb640 10 client.26114 handle_command_reply: tid=0 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 1 client.26114 shutdown 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 1 -- 172.21.15.31:0/660717157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 msgr2=0x56238c5ddc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 0x56238c5ddc60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdffc00ff50 tx=0x7fdffc004a40 comp rx=0 tx=0).stop 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 1 -- 172.21.15.31:0/660717157 shutdown_connections 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56238c5f4bc0 0x56238c5f6fb0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fdfe9ffb640 0 client.26114 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 0x56238c5ddc60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5b60 0x56238c5dd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/660717157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5d5240 0x56238c5dd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 1 -- 172.21.15.31:0/660717157 >> 172.21.15.31:0/660717157 conn(0x56238c5d1560 msgr2=0x56238c5d7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 1 -- 172.21.15.31:0/660717157 shutdown_connections 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 1 -- 172.21.15.31:0/660717157 wait complete. 2024-04-25T01:59:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.593+0000 7fe00b9b6740 20 client.26114 trim_cache size 0 max 16384 2024-04-25T01:59:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.608+0000 7fe00b9b6740 1 Processor -- start 2024-04-25T01:59:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.608+0000 7fe00b9b6740 1 -- start start 2024-04-25T01:59:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.608+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5df910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5e0230 0x56238c5d1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0041ab060 con 0x56238c5d5240 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe0041acba0 con 0x56238c5d5b60 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe0041ab730 con 0x56238c5e0230 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe00259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5df910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe001d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe00259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5df910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45260/0 (socket says 172.21.15.31:45260) 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe001d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5ebee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59028/0 (socket says 172.21.15.31:59028) 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe00259c640 1 -- 172.21.15.31:0/3685719407 learned_addr learned my addr 172.21.15.31:0/3685719407 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe002d9d640 1 --2- 172.21.15.31:0/3685719407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5e0230 0x56238c5d1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe001d9b640 1 -- 172.21.15.31:0/3685719407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5e0230 msgr2=0x56238c5d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe001d9b640 1 --2- 172.21.15.31:0/3685719407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5e0230 0x56238c5d1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe001d9b640 1 -- 172.21.15.31:0/3685719407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 msgr2=0x56238c5df910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe001d9b640 1 --2- 172.21.15.31:0/3685719407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5df910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe001d9b640 1 -- 172.21.15.31:0/3685719407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56238c5dc190 con 0x56238c5d5b60 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe00259c640 1 --2- 172.21.15.31:0/3685719407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5df910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fe001d9b640 1 --2- 172.21.15.31:0/3685719407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5ebee0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdff80eceb0 tx=0x7fdff808dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81d64aceb03b9bea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fdfebfff640 1 -- 172.21.15.31:0/3685719407 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff8049040 con 0x56238c5d5b60 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fdfebfff640 1 -- 172.21.15.31:0/3685719407 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdff8003d60 con 0x56238c5d5b60 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.609+0000 7fdfebfff640 1 -- 172.21.15.31:0/3685719407 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff80c46c0 con 0x56238c5d5b60 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.610+0000 7fe00b9b6740 1 -- 172.21.15.31:0/3685719407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 msgr2=0x56238c5ebee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.610+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/3685719407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5ebee0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdff80eceb0 tx=0x7fdff808dcf0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.610+0000 7fe00b9b6740 1 -- 172.21.15.31:0/3685719407 shutdown_connections 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.610+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/3685719407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5e0230 0x56238c5d1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.610+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/3685719407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5d5b60 0x56238c5ebee0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.610+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/3685719407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5df910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.610+0000 7fe00b9b6740 1 -- 172.21.15.31:0/3685719407 >> 172.21.15.31:0/3685719407 conn(0x56238c5d1560 msgr2=0x56238c5d7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.610+0000 7fe00b9b6740 1 -- 172.21.15.31:0/3685719407 shutdown_connections 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.610+0000 7fe00b9b6740 1 -- 172.21.15.31:0/3685719407 wait complete. 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.610+0000 7fe00b9b6740 1 -- start start 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00b9b6740 1 Processor -- start 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5e4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5d5b60 0x56238c5e4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00b9b6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 0x56238c65a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0041ac4d0 con 0x56238c5d5240 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe0041abe00 con 0x56238c5e0230 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00b9b6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe0041a9060 con 0x56238c5d5b60 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00b9b6740 10 client.0 _ll_register_callbacks cb 0x7fdffc00fdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5e4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5e4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45268/0 (socket says 172.21.15.31:45268) 2024-04-25T01:59:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00259c640 1 -- 172.21.15.31:0/2199668618 learned_addr learned my addr 172.21.15.31:0/2199668618 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe001d9b640 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5d5b60 0x56238c5e4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe002d9d640 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 0x56238c65a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00259c640 1 -- 172.21.15.31:0/2199668618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5d5b60 msgr2=0x56238c5e4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00259c640 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5d5b60 0x56238c5e4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00259c640 1 -- 172.21.15.31:0/2199668618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 msgr2=0x56238c65a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00259c640 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 0x56238c65a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe00259c640 1 -- 172.21.15.31:0/2199668618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56238c5d5990 con 0x56238c5d5240 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.611+0000 7fe002d9d640 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 0x56238c65a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fe00259c640 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5e4530 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x56238c5d6800 tx=0x7fdff4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff918fe15cb8575 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fdfe9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/2199668618 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff402f060 con 0x56238c5d5240 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/2199668618 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fdff400fe20 con 0x56238c5d5240 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fe00b9b6740 1 -- 172.21.15.31:0/2199668618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdff8003d60 con 0x56238c5d5240 2024-04-25T01:59:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/2199668618 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdff401e040 con 0x56238c5d5240 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fe00b9b6740 10 client.16518 fetch_fsmap learned FSMap version 33 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fe00b9b6740 1 -- 172.21.15.31:0/2199668618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56238c65b750 con 0x56238c5d5240 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/2199668618 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdff8003d60 con 0x56238c5d5240 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/2199668618 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fdff4005490 con 0x56238c5d5240 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fe00b9b6740 10 client.16518 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fe00b9b6740 10 client.16518 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fe00b9b6740 20 client.16518 populate_metadata read hostname 'smithi031' 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56238c65bd10 0x56238c65e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fe00b9b6740 4 client.16518 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.612+0000 7fe00b9b6740 1 -- 172.21.15.31:0/2199668618 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56238c5471f0 con 0x56238c65bd10 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.613+0000 7fe001d9b640 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56238c65bd10 0x56238c65e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.613+0000 7fdfe9ffb640 10 client.16518 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.613+0000 7fe001d9b640 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56238c65bd10 0x56238c65e130 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d3dec2343bde36a server_cookie=49ff54b60c28f11c in_seq=0 out_seq=0 2024-04-25T01:59:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fdfe9ffb640 1 -- 172.21.15.31:0/2199668618 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56238c5471f0 con 0x56238c65bd10 2024-04-25T01:59:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fdfe9ffb640 10 client.16518 handle_command_reply: tid=0 2024-04-25T01:59:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fe00b9b6740 1 client.16518 shutdown 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fe00b9b6740 1 -- 172.21.15.31:0/2199668618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 msgr2=0x56238c5e4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5e4530 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x56238c5d6800 tx=0x7fdff4005dc0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fe00b9b6740 1 -- 172.21.15.31:0/2199668618 shutdown_connections 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56238c65bd10 0x56238c65e130 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56238c5e0230 0x56238c65a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56238c5d5b60 0x56238c5e4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fe00b9b6740 1 --2- 172.21.15.31:0/2199668618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56238c5d5240 0x56238c5e4530 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fdfe9ffb640 0 client.16518 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fe00b9b6740 1 -- 172.21.15.31:0/2199668618 >> 172.21.15.31:0/2199668618 conn(0x56238c5d1560 msgr2=0x56238c5d7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fe00b9b6740 1 -- 172.21.15.31:0/2199668618 shutdown_connections 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.633+0000 7fe00b9b6740 1 -- 172.21.15.31:0/2199668618 wait complete. 2024-04-25T01:59:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe00b9b6740 20 client.16518 trim_cache size 0 max 16384 2024-04-25T01:59:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:59:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:59:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:59:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 -- 172.21.15.31:0/3673972326 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfec0a2da0 msgr2=0x7fdfec0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfec0a2da0 0x7fdfec0a5260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdffc005fd0 tx=0x7fdffc009500 comp rx=0 tx=0).stop 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 -- 172.21.15.31:0/3673972326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0041032f0 msgr2=0x7fe004198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0041032f0 0x7fe004198400 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdff8005940 tx=0x7fdff80040c0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 -- 172.21.15.31:0/3673972326 shutdown_connections 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfec0a2da0 0x7fdfec0a5260 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe00410bd10 0x7fe00419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0041032f0 0x7fe004198400 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 --2- 172.21.15.31:0/3673972326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0041029d0 0x7fe004197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 -- 172.21.15.31:0/3673972326 >> 172.21.15.31:0/3673972326 conn(0x7fe0040777c0 msgr2=0x7fe0040fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 -- 172.21.15.31:0/3673972326 shutdown_connections 2024-04-25T01:59:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:02.634+0000 7fe008bd5640 1 -- 172.21.15.31:0/3673972326 wait complete. 2024-04-25T01:59:03.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 94316 2.61 MB/sec execute 70 sec latency 25.393 ms 2024-04-25T01:59:04.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 94772 2.60 MB/sec execute 71 sec latency 19.203 ms 2024-04-25T01:59:04.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:04 smithi040 ceph-mon[32418]: pgmap v141: 97 pgs: 97 active+clean; 63 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:59:04.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:04 smithi134 ceph-mon[33098]: pgmap v141: 97 pgs: 97 active+clean; 63 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:59:04.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:04 smithi031 ceph-mon[29204]: pgmap v141: 97 pgs: 97 active+clean; 63 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-25T01:59:05.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 95553 2.66 MB/sec execute 72 sec latency 18.176 ms 2024-04-25T01:59:06.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 95934 2.67 MB/sec execute 73 sec latency 18.474 ms 2024-04-25T01:59:06.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:06 smithi040 ceph-mon[32418]: pgmap v142: 97 pgs: 97 active+clean; 61 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 1.8 MiB/s wr, 203 op/s 2024-04-25T01:59:06.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:06 smithi134 ceph-mon[33098]: pgmap v142: 97 pgs: 97 active+clean; 61 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 1.8 MiB/s wr, 203 op/s 2024-04-25T01:59:06.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:06 smithi031 ceph-mon[29204]: pgmap v142: 97 pgs: 97 active+clean; 61 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 1.8 MiB/s wr, 203 op/s 2024-04-25T01:59:07.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 96233 2.65 MB/sec execute 74 sec latency 18.103 ms 2024-04-25T01:59:08.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 96410 2.61 MB/sec execute 75 sec latency 401.724 ms 2024-04-25T01:59:08.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:08 smithi040 ceph-mon[32418]: pgmap v143: 97 pgs: 97 active+clean; 61 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-04-25T01:59:08.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:08 smithi134 ceph-mon[33098]: pgmap v143: 97 pgs: 97 active+clean; 61 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-04-25T01:59:08.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:08 smithi031 ceph-mon[29204]: pgmap v143: 97 pgs: 97 active+clean; 61 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-04-25T01:59:09.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 96410 2.58 MB/sec execute 76 sec latency 1401.772 ms 2024-04-25T01:59:10.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 96410 2.55 MB/sec execute 77 sec latency 2401.814 ms 2024-04-25T01:59:10.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:10 smithi040 ceph-mon[32418]: pgmap v144: 97 pgs: 97 active+clean; 61 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-04-25T01:59:10.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:10 smithi134 ceph-mon[33098]: pgmap v144: 97 pgs: 97 active+clean; 61 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-04-25T01:59:10.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:10 smithi031 ceph-mon[29204]: pgmap v144: 97 pgs: 97 active+clean; 61 MiB data, 841 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-04-25T01:59:11.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 96410 2.51 MB/sec execute 78 sec latency 3401.934 ms 2024-04-25T01:59:12.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 96410 2.48 MB/sec execute 79 sec latency 4401.987 ms 2024-04-25T01:59:12.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:12 smithi040 ceph-mon[32418]: pgmap v145: 97 pgs: 97 active+clean; 64 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-04-25T01:59:12.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:12 smithi134 ceph-mon[33098]: pgmap v145: 97 pgs: 97 active+clean; 64 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-04-25T01:59:12.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:12 smithi031 ceph-mon[29204]: pgmap v145: 97 pgs: 97 active+clean; 64 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-04-25T01:59:13.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 96681 2.46 MB/sec execute 80 sec latency 4842.342 ms 2024-04-25T01:59:14.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 97210 2.48 MB/sec execute 81 sec latency 19.167 ms 2024-04-25T01:59:14.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:14 smithi040 ceph-mon[32418]: pgmap v146: 97 pgs: 97 active+clean; 64 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.0 MiB/s wr, 124 op/s 2024-04-25T01:59:14.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:14 smithi134 ceph-mon[33098]: pgmap v146: 97 pgs: 97 active+clean; 64 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.0 MiB/s wr, 124 op/s 2024-04-25T01:59:14.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:14 smithi031 ceph-mon[29204]: pgmap v146: 97 pgs: 97 active+clean; 64 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.0 MiB/s wr, 124 op/s 2024-04-25T01:59:15.306 INFO:tasks.workunit.client.0.smithi031.stdout: 1 97680 2.46 MB/sec execute 82 sec latency 20.523 ms 2024-04-25T01:59:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.729+0000 7f25e7370640 1 -- 172.21.15.31:0/2564521308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e01032f0 msgr2=0x7f25e010b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.729+0000 7f25e7370640 1 --2- 172.21.15.31:0/2564521308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e01032f0 0x7f25e010b7d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009900 tx=0x7f25d002a510 comp rx=0 tx=0).stop 2024-04-25T01:59:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.729+0000 7f25e7370640 1 -- 172.21.15.31:0/2564521308 shutdown_connections 2024-04-25T01:59:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.729+0000 7f25e7370640 1 --2- 172.21.15.31:0/2564521308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e010bd10 0x7f25e010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.729+0000 7f25e7370640 1 --2- 172.21.15.31:0/2564521308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e01032f0 0x7f25e010b7d0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.729+0000 7f25e7370640 1 --2- 172.21.15.31:0/2564521308 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25e01029d0 0x7f25e0102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.729+0000 7f25e7370640 1 -- 172.21.15.31:0/2564521308 >> 172.21.15.31:0/2564521308 conn(0x7f25e00777c0 msgr2=0x7f25e0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.729+0000 7f25e7370640 1 -- 172.21.15.31:0/2564521308 shutdown_connections 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.729+0000 7f25e7370640 1 -- 172.21.15.31:0/2564521308 wait complete. 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.729+0000 7f25e7370640 1 Processor -- start 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e7370640 1 -- start start 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e7370640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e01029d0 0x7f25e0197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e7370640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25e01032f0 0x7f25e0198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e7370640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e010bd10 0x7f25e019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e7370640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f25e01119c0 con 0x7f25e010bd10 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e7370640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25e0111840 con 0x7f25e01029d0 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e7370640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f25e0111b40 con 0x7f25e01032f0 2024-04-25T01:59:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e590d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e010bd10 0x7f25e019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e590d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e010bd10 0x7f25e019c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44214/0 (socket says 172.21.15.31:44214) 2024-04-25T01:59:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e590d640 1 -- 172.21.15.31:0/4071005160 learned_addr learned my addr 172.21.15.31:0/4071005160 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e510c640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e01029d0 0x7f25e0197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e490b640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25e01032f0 0x7f25e0198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e590d640 1 -- 172.21.15.31:0/4071005160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25e01032f0 msgr2=0x7f25e0198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e590d640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25e01032f0 0x7f25e0198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e590d640 1 -- 172.21.15.31:0/4071005160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e01029d0 msgr2=0x7f25e0197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e590d640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e01029d0 0x7f25e0197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e590d640 1 -- 172.21.15.31:0/4071005160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e019ced0 con 0x7f25e010bd10 2024-04-25T01:59:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.730+0000 7f25e510c640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e01029d0 0x7f25e0197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.731+0000 7f25e590d640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e010bd10 0x7f25e019c7a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f25d800eb10 tx=0x7f25d800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.731+0000 7f25d67fc640 1 -- 172.21.15.31:0/4071005160 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d8004b20 con 0x7f25e010bd10 2024-04-25T01:59:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.731+0000 7f25d67fc640 1 -- 172.21.15.31:0/4071005160 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25d8004cc0 con 0x7f25e010bd10 2024-04-25T01:59:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.731+0000 7f25e7370640 1 -- 172.21.15.31:0/4071005160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e019d1c0 con 0x7f25e010bd10 2024-04-25T01:59:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.731+0000 7f25d67fc640 1 -- 172.21.15.31:0/4071005160 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d8010690 con 0x7f25e010bd10 2024-04-25T01:59:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.731+0000 7f25e7370640 1 -- 172.21.15.31:0/4071005160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e01a4a50 con 0x7f25e010bd10 2024-04-25T01:59:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.733+0000 7f25e9d22740 1 Processor -- start 2024-04-25T01:59:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.733+0000 7f25d67fc640 1 -- 172.21.15.31:0/4071005160 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f25d8005000 con 0x7f25e010bd10 2024-04-25T01:59:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e9d22740 1 -- start start 2024-04-25T01:59:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 0x560938643620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 0x56093864dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56093864e230 0x560938650620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e9d22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5609385a7cb0 con 0x56093864e230 2024-04-25T01:59:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e9d22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5609385505b0 con 0x560938643b60 2024-04-25T01:59:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e9d22740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5609385a3c70 con 0x560938643240 2024-04-25T01:59:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e490b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 0x560938643620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e510c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 0x56093864dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e490b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56093864e230 0x560938650620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e510c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 0x56093864dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41190/0 (socket says 172.21.15.31:41190) 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e510c640 1 -- 172.21.15.31:0/2505962705 learned_addr learned my addr 172.21.15.31:0/2505962705 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25d67fc640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f25cc0a2da0 0x7f25cc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e510c640 1 -- 172.21.15.31:0/2505962705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 msgr2=0x560938643620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e510c640 1 --2- 172.21.15.31:0/2505962705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 0x560938643620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e510c640 1 -- 172.21.15.31:0/2505962705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56093864e230 msgr2=0x560938650620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e510c640 1 --2- 172.21.15.31:0/2505962705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56093864e230 0x560938650620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e510c640 1 -- 172.21.15.31:0/2505962705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609382bc210 con 0x560938643b60 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.734+0000 7f25e490b640 1 --2- 172.21.15.31:0/2505962705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 0x560938643620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.735+0000 7f25e590d640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f25cc0a2da0 0x7f25cc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.735+0000 7f25e510c640 1 --2- 172.21.15.31:0/2505962705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 0x56093864dcf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0104d0 tx=0x7f25dc010990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19b1d50d6d6dda69 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.735+0000 7f25d67fc640 1 -- 172.21.15.31:0/4071005160 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f25d80cf910 con 0x7f25e010bd10 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.735+0000 7f25cb7fe640 1 -- 172.21.15.31:0/2505962705 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc011760 con 0x560938643b60 2024-04-25T01:59:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.735+0000 7f25cb7fe640 1 -- 172.21.15.31:0/2505962705 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25dc011db0 con 0x560938643b60 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.735+0000 7f25cb7fe640 1 -- 172.21.15.31:0/2505962705 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc01cdd0 con 0x560938643b60 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e9d22740 1 -- 172.21.15.31:0/2505962705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 msgr2=0x56093864dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e590d640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f25cc0a2da0 0x7f25cc0a5260 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f25d80049d0 tx=0x7f25d800cf70 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2505962705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 0x56093864dcf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0104d0 tx=0x7f25dc010990 comp rx=0 tx=0).stop 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e9d22740 1 -- 172.21.15.31:0/2505962705 shutdown_connections 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2505962705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56093864e230 0x560938650620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2505962705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 0x56093864dcf0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2505962705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 0x560938643620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e9d22740 1 -- 172.21.15.31:0/2505962705 >> 172.21.15.31:0/2505962705 conn(0x56093863f560 msgr2=0x56093863f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e9d22740 1 -- 172.21.15.31:0/2505962705 shutdown_connections 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e9d22740 1 -- 172.21.15.31:0/2505962705 wait complete. 2024-04-25T01:59:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e9d22740 1 -- start start 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.736+0000 7f25e9d22740 1 Processor -- start 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 0x56093864b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x56093864b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56093864e230 0x56093864bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e9d22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560938651020 con 0x560938643b60 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e9d22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560938563b50 con 0x560938643240 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e9d22740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5609386511a0 con 0x56093864e230 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e9d22740 10 client.0 _ll_register_callbacks cb 0x56093859eca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e490b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 0x56093864b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e510c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x56093864b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e490b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 0x56093864b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41206/0 (socket says 172.21.15.31:41206) 2024-04-25T01:59:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e510c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x56093864b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44232/0 (socket says 172.21.15.31:44232) 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e490b640 1 -- 172.21.15.31:0/2023005748 learned_addr learned my addr 172.21.15.31:0/2023005748 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e490b640 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56093864e230 0x56093864bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e510c640 1 -- 172.21.15.31:0/2023005748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56093864e230 msgr2=0x56093864bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e510c640 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56093864e230 0x56093864bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e510c640 1 -- 172.21.15.31:0/2023005748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 msgr2=0x56093864b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e510c640 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 0x56093864b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e510c640 1 -- 172.21.15.31:0/2023005748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560938388620 con 0x560938643b60 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e490b640 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 0x56093864b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e510c640 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x56093864b700 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x560938644800 tx=0x7f25dc005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e6572f5b33e50b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2023005748 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc02b060 con 0x560938643b60 2024-04-25T01:59:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25e9d22740 1 -- 172.21.15.31:0/2023005748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25dc011db0 con 0x560938643b60 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.737+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2023005748 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f25dc01cd40 con 0x560938643b60 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2023005748 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f25dc004040 con 0x560938643b60 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25e9d22740 10 client.16530 fetch_fsmap learned FSMap version 33 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25e9d22740 1 -- 172.21.15.31:0/2023005748 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56093865d270 con 0x560938643b60 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2023005748 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25dc011db0 con 0x560938643b60 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2023005748 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f25dc0035f0 con 0x560938643b60 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25e9d22740 10 client.16530 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25e9d22740 10 client.16530 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25e9d22740 20 client.16530 populate_metadata read hostname 'smithi031' 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56093865e680 0x560938660a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25e9d22740 4 client.16530 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25e9d22740 1 -- 172.21.15.31:0/2023005748 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609385b51f0 con 0x56093865e680 2024-04-25T01:59:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.738+0000 7f25e490b640 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56093865e680 0x560938660a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.739+0000 7f25e490b640 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56093865e680 0x560938660a70 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2517b5345a94a660 server_cookie=8dcc1e7570994b75 in_seq=0 out_seq=0 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.739+0000 7f25c8ff9640 10 client.16530 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2023005748 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5609385b51f0 con 0x56093865e680 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25c8ff9640 10 client.16530 handle_command_reply: tid=0 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 1 client.16530 shutdown 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 1 -- 172.21.15.31:0/2023005748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 msgr2=0x56093864b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x56093864b700 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x560938644800 tx=0x7f25dc005f30 comp rx=0 tx=0).stop 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 1 -- 172.21.15.31:0/2023005748 shutdown_connections 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56093865e680 0x560938660a70 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56093864e230 0x56093864bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25c8ff9640 0 client.16530 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x56093864b700 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2023005748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 0x56093864b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 1 -- 172.21.15.31:0/2023005748 >> 172.21.15.31:0/2023005748 conn(0x56093863f560 msgr2=0x560938645360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 1 -- 172.21.15.31:0/2023005748 shutdown_connections 2024-04-25T01:59:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 1 -- 172.21.15.31:0/2023005748 wait complete. 2024-04-25T01:59:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.740+0000 7f25e9d22740 20 client.16530 trim_cache size 0 max 16384 2024-04-25T01:59:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.757+0000 7f25e9d22740 1 Processor -- start 2024-04-25T01:59:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.757+0000 7f25e9d22740 1 -- start start 2024-04-25T01:59:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 0x56093864d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x560938659ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56093864e230 0x5609386459e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e9d22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f25e01ad2c0 con 0x560938643b60 2024-04-25T01:59:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e9d22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25e01aee00 con 0x560938643240 2024-04-25T01:59:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e9d22740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f25e01ad990 con 0x56093864e230 2024-04-25T01:59:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e510c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x560938659ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e510c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x560938659ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44246/0 (socket says 172.21.15.31:44246) 2024-04-25T01:59:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e510c640 1 -- 172.21.15.31:0/1521031855 learned_addr learned my addr 172.21.15.31:0/1521031855 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e490b640 1 --2- 172.21.15.31:0/1521031855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 0x56093864d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e490b640 1 --2- 172.21.15.31:0/1521031855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56093864e230 0x5609386459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e510c640 1 -- 172.21.15.31:0/1521031855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56093864e230 msgr2=0x5609386459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e510c640 1 --2- 172.21.15.31:0/1521031855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56093864e230 0x5609386459e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e510c640 1 -- 172.21.15.31:0/1521031855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 msgr2=0x56093864d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e510c640 1 --2- 172.21.15.31:0/1521031855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 0x56093864d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.758+0000 7f25e510c640 1 -- 172.21.15.31:0/1521031855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560938660de0 con 0x560938643b60 2024-04-25T01:59:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.759+0000 7f25e510c640 1 --2- 172.21.15.31:0/1521031855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x560938659ee0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00bfd0 tx=0x7f25dc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0c7930d3a08f91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.759+0000 7f25caffd640 1 -- 172.21.15.31:0/1521031855 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc02b040 con 0x560938643b60 2024-04-25T01:59:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.759+0000 7f25caffd640 1 -- 172.21.15.31:0/1521031855 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25dc00bcb0 con 0x560938643b60 2024-04-25T01:59:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.759+0000 7f25caffd640 1 -- 172.21.15.31:0/1521031855 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc005430 con 0x560938643b60 2024-04-25T01:59:15.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 -- 172.21.15.31:0/1521031855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 msgr2=0x560938659ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 --2- 172.21.15.31:0/1521031855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x560938659ee0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00bfd0 tx=0x7f25dc001e70 comp rx=0 tx=0).stop 2024-04-25T01:59:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 -- 172.21.15.31:0/1521031855 shutdown_connections 2024-04-25T01:59:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 --2- 172.21.15.31:0/1521031855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56093864e230 0x5609386459e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 --2- 172.21.15.31:0/1521031855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560938643b60 0x560938659ee0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 --2- 172.21.15.31:0/1521031855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643240 0x56093864d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 -- 172.21.15.31:0/1521031855 >> 172.21.15.31:0/1521031855 conn(0x56093863f560 msgr2=0x56093863f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 -- 172.21.15.31:0/1521031855 shutdown_connections 2024-04-25T01:59:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 -- 172.21.15.31:0/1521031855 wait complete. 2024-04-25T01:59:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 -- start start 2024-04-25T01:59:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 Processor -- start 2024-04-25T01:59:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.760+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 0x560938656650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 0x560938647a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e9d22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56093864e230 0x560938647f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e9d22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f25e01ae730 con 0x56093864e230 2024-04-25T01:59:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e9d22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25e01ae060 con 0x560938643b60 2024-04-25T01:59:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e9d22740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f25e01ab2a0 con 0x560938643240 2024-04-25T01:59:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e9d22740 10 client.0 _ll_register_callbacks cb 0x7f25d0032c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e490b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 0x560938656650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e490b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 0x560938656650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:46174/0 (socket says 172.21.15.31:46174) 2024-04-25T01:59:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e490b640 1 -- 172.21.15.31:0/2384949683 learned_addr learned my addr 172.21.15.31:0/2384949683 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e510c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 0x560938647a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e490b640 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56093864e230 0x560938647f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e490b640 1 -- 172.21.15.31:0/2384949683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 msgr2=0x560938647a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e490b640 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 0x560938647a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e490b640 1 -- 172.21.15.31:0/2384949683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56093864e230 msgr2=0x560938647f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e490b640 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56093864e230 0x560938647f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e490b640 1 -- 172.21.15.31:0/2384949683 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609386485c0 con 0x560938643240 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e510c640 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 0x560938647a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e490b640 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 0x560938656650 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f25d0000c00 tx=0x7f25d002a510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41224000daa80b0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2384949683 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0035040 con 0x560938643240 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25e9d22740 1 -- 172.21.15.31:0/2384949683 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25dc00bcb0 con 0x560938643240 2024-04-25T01:59:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.761+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2384949683 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f25d0004260 con 0x560938643240 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2384949683 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f25d005f050 con 0x560938643240 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25e9d22740 10 client.25972 fetch_fsmap learned FSMap version 33 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25e9d22740 1 -- 172.21.15.31:0/2384949683 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5609386ccfd0 con 0x560938643240 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2384949683 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25dc00bcb0 con 0x560938643240 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2384949683 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f25d0057020 con 0x560938643240 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25e9d22740 10 client.25972 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25e9d22740 10 client.25972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25e9d22740 20 client.25972 populate_metadata read hostname 'smithi031' 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56093865e680 0x5609386cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25e9d22740 4 client.25972 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25e9d22740 1 -- 172.21.15.31:0/2384949683 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5609385af540 con 0x56093865e680 2024-04-25T01:59:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.762+0000 7f25e510c640 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56093865e680 0x5609386cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.763+0000 7f25c8ff9640 10 client.25972 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.763+0000 7f25e510c640 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56093865e680 0x5609386cf790 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bedf9b11991c92d server_cookie=6a22b0b76e76b0a8 in_seq=0 out_seq=0 2024-04-25T01:59:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.779+0000 7f25c8ff9640 1 -- 172.21.15.31:0/2384949683 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5609385af540 con 0x56093865e680 2024-04-25T01:59:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.779+0000 7f25c8ff9640 10 client.25972 handle_command_reply: tid=0 2024-04-25T01:59:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.779+0000 7f25e9d22740 1 client.25972 shutdown 2024-04-25T01:59:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25e9d22740 1 -- 172.21.15.31:0/2384949683 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 msgr2=0x560938656650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 0x560938656650 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f25d0000c00 tx=0x7f25d002a510 comp rx=0 tx=0).stop 2024-04-25T01:59:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25e9d22740 1 -- 172.21.15.31:0/2384949683 shutdown_connections 2024-04-25T01:59:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56093865e680 0x5609386cf790 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56093864e230 0x560938647f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25c8ff9640 0 client.25972 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560938643b60 0x560938647a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25e9d22740 1 --2- 172.21.15.31:0/2384949683 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560938643240 0x560938656650 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25e9d22740 1 -- 172.21.15.31:0/2384949683 >> 172.21.15.31:0/2384949683 conn(0x56093863f560 msgr2=0x56093863dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25e9d22740 1 -- 172.21.15.31:0/2384949683 shutdown_connections 2024-04-25T01:59:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25e9d22740 1 -- 172.21.15.31:0/2384949683 wait complete. 2024-04-25T01:59:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.780+0000 7f25e9d22740 20 client.25972 trim_cache size 0 max 16384 2024-04-25T01:59:15.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:59:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:59:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:59:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:59:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 -- 172.21.15.31:0/4071005160 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f25cc0a2da0 msgr2=0x7f25cc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f25cc0a2da0 0x7f25cc0a5260 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f25d80049d0 tx=0x7f25d800cf70 comp rx=0 tx=0).stop 2024-04-25T01:59:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 -- 172.21.15.31:0/4071005160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e010bd10 msgr2=0x7f25e019c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e010bd10 0x7f25e019c7a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f25d800eb10 tx=0x7f25d800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 -- 172.21.15.31:0/4071005160 shutdown_connections 2024-04-25T01:59:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f25cc0a2da0 0x7f25cc0a5260 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e010bd10 0x7f25e019c7a0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25e01032f0 0x7f25e0198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 --2- 172.21.15.31:0/4071005160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e01029d0 0x7f25e0197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 -- 172.21.15.31:0/4071005160 >> 172.21.15.31:0/4071005160 conn(0x7f25e00777c0 msgr2=0x7f25e00fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 -- 172.21.15.31:0/4071005160 shutdown_connections 2024-04-25T01:59:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:15.781+0000 7f25e7370640 1 -- 172.21.15.31:0/4071005160 wait complete. 2024-04-25T01:59:16.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 98225 2.45 MB/sec execute 83 sec latency 25.081 ms 2024-04-25T01:59:16.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:16 smithi040 ceph-mon[32418]: pgmap v147: 97 pgs: 97 active+clean; 63 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.2 MiB/s wr, 156 op/s 2024-04-25T01:59:16.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:16 smithi134 ceph-mon[33098]: pgmap v147: 97 pgs: 97 active+clean; 63 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.2 MiB/s wr, 156 op/s 2024-04-25T01:59:16.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:16 smithi031 ceph-mon[29204]: pgmap v147: 97 pgs: 97 active+clean; 63 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.2 MiB/s wr, 156 op/s 2024-04-25T01:59:17.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 98836 2.48 MB/sec execute 84 sec latency 17.499 ms 2024-04-25T01:59:18.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 99370 2.51 MB/sec execute 85 sec latency 11.239 ms 2024-04-25T01:59:18.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:18 smithi040 ceph-mon[32418]: pgmap v148: 97 pgs: 97 active+clean; 63 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 704 KiB/s wr, 75 op/s 2024-04-25T01:59:18.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:18 smithi134 ceph-mon[33098]: pgmap v148: 97 pgs: 97 active+clean; 63 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 704 KiB/s wr, 75 op/s 2024-04-25T01:59:18.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:18 smithi031 ceph-mon[29204]: pgmap v148: 97 pgs: 97 active+clean; 63 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 704 KiB/s wr, 75 op/s 2024-04-25T01:59:19.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 99730 2.50 MB/sec execute 86 sec latency 18.325 ms 2024-04-25T01:59:20.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 99971 2.47 MB/sec execute 87 sec latency 19.854 ms 2024-04-25T01:59:20.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:20 smithi040 ceph-mon[32418]: pgmap v149: 97 pgs: 97 active+clean; 63 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 704 KiB/s wr, 75 op/s 2024-04-25T01:59:20.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:20 smithi134 ceph-mon[33098]: pgmap v149: 97 pgs: 97 active+clean; 63 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 704 KiB/s wr, 75 op/s 2024-04-25T01:59:20.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:20 smithi031 ceph-mon[29204]: pgmap v149: 97 pgs: 97 active+clean; 63 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 704 KiB/s wr, 75 op/s 2024-04-25T01:59:21.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 100492 2.47 MB/sec execute 88 sec latency 18.011 ms 2024-04-25T01:59:22.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 100977 2.48 MB/sec execute 89 sec latency 20.160 ms 2024-04-25T01:59:22.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:22 smithi040 ceph-mon[32418]: pgmap v150: 97 pgs: 97 active+clean; 68 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-04-25T01:59:22.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:22 smithi134 ceph-mon[33098]: pgmap v150: 97 pgs: 97 active+clean; 68 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-04-25T01:59:22.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:22 smithi031 ceph-mon[29204]: pgmap v150: 97 pgs: 97 active+clean; 68 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-04-25T01:59:23.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 101502 2.45 MB/sec execute 90 sec latency 20.485 ms 2024-04-25T01:59:24.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 101957 2.45 MB/sec execute 91 sec latency 19.385 ms 2024-04-25T01:59:24.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:24 smithi040 ceph-mon[32418]: pgmap v151: 97 pgs: 97 active+clean; 68 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 919 KiB/s wr, 91 op/s 2024-04-25T01:59:24.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:24 smithi134 ceph-mon[33098]: pgmap v151: 97 pgs: 97 active+clean; 68 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 919 KiB/s wr, 91 op/s 2024-04-25T01:59:24.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:24 smithi031 ceph-mon[29204]: pgmap v151: 97 pgs: 97 active+clean; 68 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 919 KiB/s wr, 91 op/s 2024-04-25T01:59:25.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 102664 2.49 MB/sec execute 92 sec latency 18.385 ms 2024-04-25T01:59:26.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 103106 2.52 MB/sec execute 93 sec latency 18.228 ms 2024-04-25T01:59:26.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:26 smithi040 ceph-mon[32418]: pgmap v152: 97 pgs: 97 active+clean; 69 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-04-25T01:59:26.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:26 smithi134 ceph-mon[33098]: pgmap v152: 97 pgs: 97 active+clean; 69 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-04-25T01:59:26.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:26 smithi031 ceph-mon[29204]: pgmap v152: 97 pgs: 97 active+clean; 69 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-04-25T01:59:27.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 103352 2.49 MB/sec execute 94 sec latency 18.185 ms 2024-04-25T01:59:28.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 103734 2.47 MB/sec execute 95 sec latency 18.225 ms 2024-04-25T01:59:28.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:28 smithi040 ceph-mon[32418]: pgmap v153: 97 pgs: 97 active+clean; 69 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-04-25T01:59:28.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:28 smithi134 ceph-mon[33098]: pgmap v153: 97 pgs: 97 active+clean; 69 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-04-25T01:59:28.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:28 smithi031 ceph-mon[29204]: pgmap v153: 97 pgs: 97 active+clean; 69 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-04-25T01:59:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.876+0000 7f6439a4d640 1 -- 172.21.15.31:0/2496303029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6434101760 msgr2=0x7f6434101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.876+0000 7f6439a4d640 1 --2- 172.21.15.31:0/2496303029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6434101760 0x7f6434101be0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6428009930 tx=0x7f642802a420 comp rx=0 tx=0).stop 2024-04-25T01:59:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.876+0000 7f6439a4d640 1 -- 172.21.15.31:0/2496303029 shutdown_connections 2024-04-25T01:59:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.876+0000 7f6439a4d640 1 --2- 172.21.15.31:0/2496303029 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6434102120 0x7f643410e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.876+0000 7f6439a4d640 1 --2- 172.21.15.31:0/2496303029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6434101760 0x7f6434101be0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.876+0000 7f6439a4d640 1 --2- 172.21.15.31:0/2496303029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6434107770 0x7f6434107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.876+0000 7f6439a4d640 1 -- 172.21.15.31:0/2496303029 >> 172.21.15.31:0/2496303029 conn(0x7f64340fd1f0 msgr2=0x7f64340ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.876+0000 7f6439a4d640 1 -- 172.21.15.31:0/2496303029 shutdown_connections 2024-04-25T01:59:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.877+0000 7f6439a4d640 1 -- 172.21.15.31:0/2496303029 wait complete. 2024-04-25T01:59:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.877+0000 7f6439a4d640 1 Processor -- start 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.877+0000 7f6439a4d640 1 -- start start 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.877+0000 7f6439a4d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6434101760 0x7f6434197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.877+0000 7f6439a4d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6434102120 0x7f6434198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.877+0000 7f6439a4d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6434107770 0x7f643419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.877+0000 7f6439a4d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6434111a30 con 0x7f6434102120 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f6439a4d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f64341118b0 con 0x7f6434107770 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f6439a4d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6434111bb0 con 0x7f6434101760 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f64327fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6434102120 0x7f6434198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f64327fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6434102120 0x7f6434198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46588/0 (socket says 172.21.15.31:46588) 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f64327fc640 1 -- 172.21.15.31:0/552260656 learned_addr learned my addr 172.21.15.31:0/552260656 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f64337fe640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6434107770 0x7f643419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f6432ffd640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6434101760 0x7f6434197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f64327fc640 1 -- 172.21.15.31:0/552260656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6434101760 msgr2=0x7f6434197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f64327fc640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6434101760 0x7f6434197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f64327fc640 1 -- 172.21.15.31:0/552260656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6434107770 msgr2=0x7f643419c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f64327fc640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6434107770 0x7f643419c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f64327fc640 1 -- 172.21.15.31:0/552260656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643419cec0 con 0x7f6434102120 2024-04-25T01:59:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f64327fc640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6434102120 0x7f6434198410 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6428005940 tx=0x7f64280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f6438a4b640 1 -- 172.21.15.31:0/552260656 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642803f030 con 0x7f6434102120 2024-04-25T01:59:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f6438a4b640 1 -- 172.21.15.31:0/552260656 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64280318a0 con 0x7f6434102120 2024-04-25T01:59:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f6439a4d640 1 -- 172.21.15.31:0/552260656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f643419d150 con 0x7f6434102120 2024-04-25T01:59:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.878+0000 7f6438a4b640 1 -- 172.21.15.31:0/552260656 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428035070 con 0x7f6434102120 2024-04-25T01:59:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.879+0000 7f6439a4d640 1 -- 172.21.15.31:0/552260656 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64341a4a40 con 0x7f6434102120 2024-04-25T01:59:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.881+0000 7f6438a4b640 1 -- 172.21.15.31:0/552260656 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6428042030 con 0x7f6434102120 2024-04-25T01:59:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.881+0000 7f643c836740 1 Processor -- start 2024-04-25T01:59:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.881+0000 7f643c836740 1 -- start start 2024-04-25T01:59:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f643c836740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d09912620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f643c836740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d09912b60 0x561d0991ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f643c836740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d0991d230 0x561d0991f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f643c836740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561d09876cb0 con 0x561d09912240 2024-04-25T01:59:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f643c836740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561d0981f5b0 con 0x561d0991d230 2024-04-25T01:59:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f643c836740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561d09872c70 con 0x561d09912b60 2024-04-25T01:59:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f6438a4b640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f641c0a2da0 0x7f641c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f64337fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d09912620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f64337fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d09912620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46596/0 (socket says 172.21.15.31:46596) 2024-04-25T01:59:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f64337fe640 1 -- 172.21.15.31:0/2199802131 learned_addr learned my addr 172.21.15.31:0/2199802131 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f64327fc640 1 --2- 172.21.15.31:0/2199802131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d0991d230 0x561d0991f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f6438a4b640 1 -- 172.21.15.31:0/552260656 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f64280efaf0 con 0x7f6434102120 2024-04-25T01:59:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f6432ffd640 1 --2- 172.21.15.31:0/2199802131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d09912b60 0x561d0991ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f64337fe640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f641c0a2da0 0x7f641c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f64337fe640 1 -- 172.21.15.31:0/2199802131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d09912b60 msgr2=0x561d0991ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f64337fe640 1 --2- 172.21.15.31:0/2199802131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d09912b60 0x561d0991ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f64337fe640 1 -- 172.21.15.31:0/2199802131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d0991d230 msgr2=0x561d0991f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f64337fe640 1 --2- 172.21.15.31:0/2199802131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d0991d230 0x561d0991f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f64337fe640 1 -- 172.21.15.31:0/2199802131 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d0958b210 con 0x561d09912240 2024-04-25T01:59:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.882+0000 7f64327fc640 1 --2- 172.21.15.31:0/2199802131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d0991d230 0x561d0991f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.883+0000 7f64337fe640 1 --2- 172.21.15.31:0/2199802131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d09912620 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f642c005f40 tx=0x7f642c014f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf451431c350d18d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.883+0000 7f64337fe640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f641c0a2da0 0x7f641c0a5260 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f642c012820 tx=0x7f642c026000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.883+0000 7f6419ffb640 1 -- 172.21.15.31:0/2199802131 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c020e30 con 0x561d09912240 2024-04-25T01:59:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.883+0000 7f6419ffb640 1 -- 172.21.15.31:0/2199802131 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f642c00e760 con 0x561d09912240 2024-04-25T01:59:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.883+0000 7f6419ffb640 1 -- 172.21.15.31:0/2199802131 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c005290 con 0x561d09912240 2024-04-25T01:59:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.884+0000 7f643c836740 1 -- 172.21.15.31:0/2199802131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 msgr2=0x561d09912620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.884+0000 7f643c836740 1 --2- 172.21.15.31:0/2199802131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d09912620 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f642c005f40 tx=0x7f642c014f70 comp rx=0 tx=0).stop 2024-04-25T01:59:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.884+0000 7f643c836740 1 -- 172.21.15.31:0/2199802131 shutdown_connections 2024-04-25T01:59:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.884+0000 7f643c836740 1 --2- 172.21.15.31:0/2199802131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d0991d230 0x561d0991f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.884+0000 7f643c836740 1 --2- 172.21.15.31:0/2199802131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d09912b60 0x561d0991ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.884+0000 7f643c836740 1 --2- 172.21.15.31:0/2199802131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d09912620 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.884+0000 7f643c836740 1 -- 172.21.15.31:0/2199802131 >> 172.21.15.31:0/2199802131 conn(0x561d0990e560 msgr2=0x561d0990e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.884+0000 7f643c836740 1 -- 172.21.15.31:0/2199802131 shutdown_connections 2024-04-25T01:59:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.884+0000 7f643c836740 1 -- 172.21.15.31:0/2199802131 wait complete. 2024-04-25T01:59:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.884+0000 7f643c836740 1 -- start start 2024-04-25T01:59:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f643c836740 1 Processor -- start 2024-04-25T01:59:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f643c836740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d0991a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f643c836740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 0x561d0991a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f643c836740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d0991d230 0x561d0991ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f643c836740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561d0991ff60 con 0x561d09912240 2024-04-25T01:59:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f643c836740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561d09832b50 con 0x561d09912b60 2024-04-25T01:59:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f643c836740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561d099200e0 con 0x561d0991d230 2024-04-25T01:59:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f643c836740 10 client.0 _ll_register_callbacks cb 0x561d0986dca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f6432ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d0991a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f6432ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d0991a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46612/0 (socket says 172.21.15.31:46612) 2024-04-25T01:59:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f6432ffd640 1 -- 172.21.15.31:0/699721399 learned_addr learned my addr 172.21.15.31:0/699721399 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f64327fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 0x561d0991a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f64337fe640 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d0991d230 0x561d0991ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f6432ffd640 1 -- 172.21.15.31:0/699721399 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d0991d230 msgr2=0x561d0991ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f6432ffd640 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d0991d230 0x561d0991ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f6432ffd640 1 -- 172.21.15.31:0/699721399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 msgr2=0x561d0991a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f6432ffd640 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 0x561d0991a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.885+0000 7f6432ffd640 1 -- 172.21.15.31:0/699721399 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d09657620 con 0x561d09912240 2024-04-25T01:59:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f6432ffd640 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d0991a1a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f64240126e0 tx=0x7f6424012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8591e8ffc66a6ce8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f6412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:59:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f6412ffd640 1 -- 172.21.15.31:0/699721399 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64240120d0 con 0x561d09912240 2024-04-25T01:59:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f6412ffd640 1 -- 172.21.15.31:0/699721399 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f642400f920 con 0x561d09912240 2024-04-25T01:59:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f643c836740 1 -- 172.21.15.31:0/699721399 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f642c00e760 con 0x561d09912240 2024-04-25T01:59:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f6412ffd640 1 -- 172.21.15.31:0/699721399 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6424012270 con 0x561d09912240 2024-04-25T01:59:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f643c836740 10 client.16548 fetch_fsmap learned FSMap version 33 2024-04-25T01:59:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f643c836740 1 -- 172.21.15.31:0/699721399 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561d0992c190 con 0x561d09912240 2024-04-25T01:59:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f6412ffd640 1 -- 172.21.15.31:0/699721399 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f642c00e760 con 0x561d09912240 2024-04-25T01:59:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f6412ffd640 1 -- 172.21.15.31:0/699721399 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f6424010430 con 0x561d09912240 2024-04-25T01:59:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f643c836740 10 client.16548 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:59:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f643c836740 10 client.16548 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f643c836740 20 client.16548 populate_metadata read hostname 'smithi031' 2024-04-25T01:59:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f643c836740 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561d0992d650 0x561d0992fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f643c836740 4 client.16548 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.886+0000 7f643c836740 1 -- 172.21.15.31:0/699721399 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d098841f0 con 0x561d0992d650 2024-04-25T01:59:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.887+0000 7f64327fc640 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561d0992d650 0x561d0992fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.887+0000 7f64327fc640 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561d0992d650 0x561d0992fa80 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=418aaf40e9c5f3f8 server_cookie=eb4d1719f71ebf03 in_seq=0 out_seq=0 2024-04-25T01:59:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.887+0000 7f6412ffd640 10 client.16548 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f6412ffd640 1 -- 172.21.15.31:0/699721399 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561d098841f0 con 0x561d0992d650 2024-04-25T01:59:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f6412ffd640 10 client.16548 handle_command_reply: tid=0 2024-04-25T01:59:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 1 client.16548 shutdown 2024-04-25T01:59:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 1 -- 172.21.15.31:0/699721399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 msgr2=0x561d0991a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d0991a1a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f64240126e0 tx=0x7f6424012ba0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 1 -- 172.21.15.31:0/699721399 shutdown_connections 2024-04-25T01:59:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561d0992d650 0x561d0992fa80 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d0991d230 0x561d0991ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 0x561d0991a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 1 --2- 172.21.15.31:0/699721399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d0991a1a0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 1 -- 172.21.15.31:0/699721399 >> 172.21.15.31:0/699721399 conn(0x561d0990e560 msgr2=0x561d09914360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f6412ffd640 0 client.16548 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:28.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 1 -- 172.21.15.31:0/699721399 shutdown_connections 2024-04-25T01:59:28.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 1 -- 172.21.15.31:0/699721399 wait complete. 2024-04-25T01:59:28.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.888+0000 7f643c836740 20 client.16548 trim_cache size 0 max 16384 2024-04-25T01:59:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f643c836740 1 Processor -- start 2024-04-25T01:59:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f643c836740 1 -- start start 2024-04-25T01:59:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f643c836740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912240 0x561d0991c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f643c836740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d09912b60 0x561d09928e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f643c836740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d0991d230 0x561d09935390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f643c836740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f64341ab000 con 0x561d0991d230 2024-04-25T01:59:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f643c836740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f64341acb40 con 0x561d09912240 2024-04-25T01:59:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f643c836740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f64341ab6d0 con 0x561d09912b60 2024-04-25T01:59:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f6432ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912240 0x561d0991c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f64337fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d0991d230 0x561d09935390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f6432ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912240 0x561d0991c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:43348/0 (socket says 172.21.15.31:43348) 2024-04-25T01:59:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f64327fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d09912b60 0x561d09928e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f6432ffd640 1 -- 172.21.15.31:0/84980235 learned_addr learned my addr 172.21.15.31:0/84980235 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.904+0000 7f64337fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d0991d230 0x561d09935390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46616/0 (socket says 172.21.15.31:46616) 2024-04-25T01:59:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.905+0000 7f64327fc640 1 -- 172.21.15.31:0/84980235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912240 msgr2=0x561d0991c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.905+0000 7f64327fc640 1 --2- 172.21.15.31:0/84980235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912240 0x561d0991c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.905+0000 7f64327fc640 1 -- 172.21.15.31:0/84980235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d0991d230 msgr2=0x561d09935390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.905+0000 7f64327fc640 1 --2- 172.21.15.31:0/84980235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d0991d230 0x561d09935390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.905+0000 7f64327fc640 1 -- 172.21.15.31:0/84980235 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d0990f560 con 0x561d09912b60 2024-04-25T01:59:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.905+0000 7f6432ffd640 1 --2- 172.21.15.31:0/84980235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912240 0x561d0991c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.905+0000 7f64337fe640 1 --2- 172.21.15.31:0/84980235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d0991d230 0x561d09935390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.905+0000 7f64327fc640 1 --2- 172.21.15.31:0/84980235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d09912b60 0x561d09928e20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f64280ed290 tx=0x7f64280efdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d8065cffb37a082 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.905+0000 7f64197fa640 1 -- 172.21.15.31:0/84980235 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428049040 con 0x561d09912b60 2024-04-25T01:59:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.905+0000 7f64197fa640 1 -- 172.21.15.31:0/84980235 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6428003e20 con 0x561d09912b60 2024-04-25T01:59:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.905+0000 7f64197fa640 1 -- 172.21.15.31:0/84980235 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64280c00d0 con 0x561d09912b60 2024-04-25T01:59:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.906+0000 7f643c836740 1 -- 172.21.15.31:0/84980235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d09912b60 msgr2=0x561d09928e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.906+0000 7f643c836740 1 --2- 172.21.15.31:0/84980235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d09912b60 0x561d09928e20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f64280ed290 tx=0x7f64280efdf0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.906+0000 7f643c836740 1 -- 172.21.15.31:0/84980235 shutdown_connections 2024-04-25T01:59:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.906+0000 7f643c836740 1 --2- 172.21.15.31:0/84980235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d0991d230 0x561d09935390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.906+0000 7f643c836740 1 --2- 172.21.15.31:0/84980235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d09912b60 0x561d09928e20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.906+0000 7f643c836740 1 --2- 172.21.15.31:0/84980235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912240 0x561d0991c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.906+0000 7f643c836740 1 -- 172.21.15.31:0/84980235 >> 172.21.15.31:0/84980235 conn(0x561d0990e560 msgr2=0x561d0990d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.906+0000 7f643c836740 1 -- 172.21.15.31:0/84980235 shutdown_connections 2024-04-25T01:59:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.906+0000 7f643c836740 1 -- 172.21.15.31:0/84980235 wait complete. 2024-04-25T01:59:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.906+0000 7f643c836740 1 -- start start 2024-04-25T01:59:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.906+0000 7f643c836740 1 Processor -- start 2024-04-25T01:59:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f643c836740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d09917260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f643c836740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 0x561d099177a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f643c836740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d0991d230 0x561d09992c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f643c836740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f64341ac470 con 0x561d09912240 2024-04-25T01:59:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f64327fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 0x561d099177a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f64327fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 0x561d099177a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:43358/0 (socket says 172.21.15.31:43358) 2024-04-25T01:59:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f64327fc640 1 -- 172.21.15.31:0/52646545 learned_addr learned my addr 172.21.15.31:0/52646545 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f643c836740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f64341abda0 con 0x561d09912b60 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f643c836740 1 -- 172.21.15.31:0/52646545 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f64341a8fe0 con 0x561d0991d230 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f6432ffd640 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d09917260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f643c836740 10 client.0 _ll_register_callbacks cb 0x7f6424009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f64337fe640 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d0991d230 0x561d09992c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f64327fc640 1 -- 172.21.15.31:0/52646545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d0991d230 msgr2=0x561d09992c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f64327fc640 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d0991d230 0x561d09992c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f64327fc640 1 -- 172.21.15.31:0/52646545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 msgr2=0x561d09917260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f64327fc640 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d09917260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f64327fc640 1 -- 172.21.15.31:0/52646545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d09993140 con 0x561d09912b60 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f64337fe640 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d0991d230 0x561d09992c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f64327fc640 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 0x561d099177a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6428004650 tx=0x7f64280dc0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af60e6abc80f72ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.907+0000 7f6412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f6412ffd640 1 -- 172.21.15.31:0/52646545 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428049070 con 0x561d09912b60 2024-04-25T01:59:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f6412ffd640 1 -- 172.21.15.31:0/52646545 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f64280ed510 con 0x561d09912b60 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f643c836740 1 -- 172.21.15.31:0/52646545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561d0987e540 con 0x561d09912b60 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f6412ffd640 1 -- 172.21.15.31:0/52646545 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f64280c0410 con 0x561d09912b60 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f643c836740 10 client.26156 fetch_fsmap learned FSMap version 33 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f643c836740 1 -- 172.21.15.31:0/52646545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561d099942d0 con 0x561d09912b60 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f6412ffd640 1 -- 172.21.15.31:0/52646545 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561d0987e540 con 0x561d09912b60 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f6412ffd640 1 -- 172.21.15.31:0/52646545 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f64280bc370 con 0x561d09912b60 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f643c836740 10 client.26156 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f643c836740 10 client.26156 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f643c836740 20 client.26156 populate_metadata read hostname 'smithi031' 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f643c836740 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561d09994990 0x561d09996db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f643c836740 4 client.26156 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.908+0000 7f643c836740 1 -- 172.21.15.31:0/52646545 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561d097fc8e0 con 0x561d09994990 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.909+0000 7f6432ffd640 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561d09994990 0x561d09996db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.909+0000 7f6432ffd640 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561d09994990 0x561d09996db0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd779145f3e99fc5 server_cookie=8931bafd9b121e0c in_seq=0 out_seq=0 2024-04-25T01:59:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.909+0000 7f6412ffd640 10 client.26156 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f6412ffd640 1 -- 172.21.15.31:0/52646545 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561d097fc8e0 con 0x561d09994990 2024-04-25T01:59:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f6412ffd640 10 client.26156 handle_command_reply: tid=0 2024-04-25T01:59:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 1 client.26156 shutdown 2024-04-25T01:59:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 1 -- 172.21.15.31:0/52646545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 msgr2=0x561d099177a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 0x561d099177a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6428004650 tx=0x7f64280dc0b0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 1 -- 172.21.15.31:0/52646545 shutdown_connections 2024-04-25T01:59:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561d09994990 0x561d09996db0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561d0991d230 0x561d09992c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f6412ffd640 0 client.26156 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d09912b60 0x561d099177a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 1 --2- 172.21.15.31:0/52646545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d09912240 0x561d09917260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 1 -- 172.21.15.31:0/52646545 >> 172.21.15.31:0/52646545 conn(0x561d0990e560 msgr2=0x561d0991a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 1 -- 172.21.15.31:0/52646545 shutdown_connections 2024-04-25T01:59:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 1 -- 172.21.15.31:0/52646545 wait complete. 2024-04-25T01:59:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.930+0000 7f643c836740 20 client.26156 trim_cache size 0 max 16384 2024-04-25T01:59:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:59:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:59:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:59:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:59:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 -- 172.21.15.31:0/552260656 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f641c0a2da0 msgr2=0x7f641c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f641c0a2da0 0x7f641c0a5260 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f642c012820 tx=0x7f642c026000 comp rx=0 tx=0).stop 2024-04-25T01:59:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 -- 172.21.15.31:0/552260656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6434102120 msgr2=0x7f6434198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6434102120 0x7f6434198410 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6428005940 tx=0x7f64280042f0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 -- 172.21.15.31:0/552260656 shutdown_connections 2024-04-25T01:59:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f641c0a2da0 0x7f641c0a5260 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6434107770 0x7f643419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6434102120 0x7f6434198410 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 --2- 172.21.15.31:0/552260656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6434101760 0x7f6434197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 -- 172.21.15.31:0/552260656 >> 172.21.15.31:0/552260656 conn(0x7f64340fd1f0 msgr2=0x7f64340fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 -- 172.21.15.31:0/552260656 shutdown_connections 2024-04-25T01:59:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:28.931+0000 7f6439a4d640 1 -- 172.21.15.31:0/552260656 wait complete. 2024-04-25T01:59:29.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 104293 2.49 MB/sec execute 96 sec latency 18.189 ms 2024-04-25T01:59:30.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 104742 2.47 MB/sec execute 97 sec latency 22.701 ms 2024-04-25T01:59:30.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:30 smithi040 ceph-mon[32418]: pgmap v154: 97 pgs: 97 active+clean; 69 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-04-25T01:59:30.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:30 smithi134 ceph-mon[33098]: pgmap v154: 97 pgs: 97 active+clean; 69 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-04-25T01:59:30.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:30 smithi031 ceph-mon[29204]: pgmap v154: 97 pgs: 97 active+clean; 69 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-04-25T01:59:31.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 105285 2.46 MB/sec execute 98 sec latency 21.383 ms 2024-04-25T01:59:32.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 105881 2.49 MB/sec execute 99 sec latency 19.201 ms 2024-04-25T01:59:32.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:32 smithi040 ceph-mon[32418]: pgmap v155: 97 pgs: 97 active+clean; 66 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-04-25T01:59:32.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:32 smithi134 ceph-mon[33098]: pgmap v155: 97 pgs: 97 active+clean; 66 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-04-25T01:59:32.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:32 smithi031 ceph-mon[29204]: pgmap v155: 97 pgs: 97 active+clean; 66 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-04-25T01:59:33.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 106299 2.49 MB/sec execute 100 sec latency 18.231 ms 2024-04-25T01:59:34.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 106768 2.51 MB/sec execute 101 sec latency 12.444 ms 2024-04-25T01:59:34.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:34 smithi040 ceph-mon[32418]: pgmap v156: 97 pgs: 97 active+clean; 66 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-04-25T01:59:34.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:34 smithi134 ceph-mon[33098]: pgmap v156: 97 pgs: 97 active+clean; 66 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-04-25T01:59:34.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:34 smithi031 ceph-mon[29204]: pgmap v156: 97 pgs: 97 active+clean; 66 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-04-25T01:59:35.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 107022 2.48 MB/sec execute 102 sec latency 18.478 ms 2024-04-25T01:59:36.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 107506 2.48 MB/sec execute 103 sec latency 17.377 ms 2024-04-25T01:59:36.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:36 smithi040 ceph-mon[32418]: pgmap v157: 97 pgs: 97 active+clean; 70 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-25T01:59:36.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:36 smithi134 ceph-mon[33098]: pgmap v157: 97 pgs: 97 active+clean; 70 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-25T01:59:36.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:36 smithi031 ceph-mon[29204]: pgmap v157: 97 pgs: 97 active+clean; 70 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-25T01:59:37.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 107987 2.49 MB/sec execute 104 sec latency 19.347 ms 2024-04-25T01:59:38.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 108541 2.47 MB/sec execute 105 sec latency 22.565 ms 2024-04-25T01:59:38.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:38 smithi040 ceph-mon[32418]: pgmap v158: 97 pgs: 97 active+clean; 70 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:38.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:38 smithi134 ceph-mon[33098]: pgmap v158: 97 pgs: 97 active+clean; 70 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:38.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:38 smithi031 ceph-mon[29204]: pgmap v158: 97 pgs: 97 active+clean; 70 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:39.307 INFO:tasks.workunit.client.0.smithi031.stdout: 1 109009 2.47 MB/sec execute 106 sec latency 23.167 ms 2024-04-25T01:59:39.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:39 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:59:39.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:39 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:59:39.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:39 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:59:40.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 109730 2.50 MB/sec execute 107 sec latency 56.390 ms 2024-04-25T01:59:40.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:40 smithi040 ceph-mon[32418]: pgmap v159: 97 pgs: 97 active+clean; 70 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:40.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:40 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:59:40.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:40 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:59:40.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:40 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:59:40.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:40 smithi134 ceph-mon[33098]: pgmap v159: 97 pgs: 97 active+clean; 70 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:40.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:40 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:59:40.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:40 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:59:40.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:40 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:59:40.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:40 smithi031 ceph-mon[29204]: pgmap v159: 97 pgs: 97 active+clean; 70 MiB data, 1016 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:40.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:40 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:59:40.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:40 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:59:40.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:40 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T01:59:41.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 110115 2.51 MB/sec execute 108 sec latency 18.225 ms 2024-04-25T01:59:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.032+0000 7f735b288640 1 -- 172.21.15.31:0/1509771907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7354106630 msgr2=0x7f7354106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.032+0000 7f735b288640 1 --2- 172.21.15.31:0/1509771907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7354106630 0x7f7354106a10 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7350009930 tx=0x7f735002a420 comp rx=0 tx=0).stop 2024-04-25T01:59:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.032+0000 7f735b288640 1 -- 172.21.15.31:0/1509771907 shutdown_connections 2024-04-25T01:59:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.032+0000 7f735b288640 1 --2- 172.21.15.31:0/1509771907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7354101030 0x7f735410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.032+0000 7f735b288640 1 --2- 172.21.15.31:0/1509771907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7354100670 0x7f7354100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.032+0000 7f735b288640 1 --2- 172.21.15.31:0/1509771907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7354106630 0x7f7354106a10 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.032+0000 7f735b288640 1 -- 172.21.15.31:0/1509771907 >> 172.21.15.31:0/1509771907 conn(0x7f73540777c0 msgr2=0x7f7354077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.032+0000 7f735b288640 1 -- 172.21.15.31:0/1509771907 shutdown_connections 2024-04-25T01:59:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.032+0000 7f735b288640 1 -- 172.21.15.31:0/1509771907 wait complete. 2024-04-25T01:59:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f735b288640 1 Processor -- start 2024-04-25T01:59:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f735b288640 1 -- start start 2024-04-25T01:59:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f735b288640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7354100670 0x7f73540fd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f735b288640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7354101030 0x7f73540fdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f735b288640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7354106630 0x7f73540fe080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f735b288640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f73541108b0 con 0x7f7354106630 2024-04-25T01:59:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f735b288640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7354110730 con 0x7f7354101030 2024-04-25T01:59:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f735b288640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7354110a30 con 0x7f7354100670 2024-04-25T01:59:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f7359825640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7354106630 0x7f73540fe080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f7359825640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7354106630 0x7f73540fe080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34446/0 (socket says 172.21.15.31:34446) 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f7359825640 1 -- 172.21.15.31:0/3341809149 learned_addr learned my addr 172.21.15.31:0/3341809149 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f7358823640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7354101030 0x7f73540fdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.033+0000 7f7359024640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7354100670 0x7f73540fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f7359825640 1 -- 172.21.15.31:0/3341809149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7354100670 msgr2=0x7f73540fd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f7359825640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7354100670 0x7f73540fd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f7359825640 1 -- 172.21.15.31:0/3341809149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7354101030 msgr2=0x7f73540fdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f7359825640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7354101030 0x7f73540fdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f7359825640 1 -- 172.21.15.31:0/3341809149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73541a63e0 con 0x7f7354106630 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f7359024640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7354100670 0x7f73540fd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f7359825640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7354106630 0x7f73540fe080 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f734c00efd0 tx=0x7f734c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f734a7fc640 1 -- 172.21.15.31:0/3341809149 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c003b70 con 0x7f7354106630 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f735b288640 1 -- 172.21.15.31:0/3341809149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73541a66d0 con 0x7f7354106630 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f734a7fc640 1 -- 172.21.15.31:0/3341809149 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f734c003d10 con 0x7f7354106630 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f734a7fc640 1 -- 172.21.15.31:0/3341809149 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c010980 con 0x7f7354106630 2024-04-25T01:59:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.034+0000 7f735b288640 1 -- 172.21.15.31:0/3341809149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73541a6bd0 con 0x7f7354106630 2024-04-25T01:59:42.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f734a7fc640 1 -- 172.21.15.31:0/3341809149 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f734c004020 con 0x7f7354106630 2024-04-25T01:59:42.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f735df94740 1 Processor -- start 2024-04-25T01:59:42.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f735df94740 1 -- start start 2024-04-25T01:59:42.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f735df94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c61370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f735df94740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 0x55a421c74500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f735df94740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c76e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f735df94740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a421bc5cb0 con 0x55a421c60f90 2024-04-25T01:59:42.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f735df94740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a421b6e5b0 con 0x55a421c74a40 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f735df94740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a421bc1c70 con 0x55a421c61940 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f7359024640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 0x55a421c74500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f7358823640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c61370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f7359024640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 0x55a421c74500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:56118/0 (socket says 172.21.15.31:56118) 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f7359024640 1 -- 172.21.15.31:0/2412587525 learned_addr learned my addr 172.21.15.31:0/2412587525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f7358823640 1 --2- 172.21.15.31:0/2412587525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c76e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f734a7fc640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f73400a2da0 0x7f73400a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f7359024640 1 -- 172.21.15.31:0/2412587525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 msgr2=0x55a421c76e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.038+0000 7f7359024640 1 --2- 172.21.15.31:0/2412587525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c76e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.039+0000 7f7359024640 1 -- 172.21.15.31:0/2412587525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 msgr2=0x55a421c61370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.039+0000 7f7359024640 1 --2- 172.21.15.31:0/2412587525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c61370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.039+0000 7f7359024640 1 -- 172.21.15.31:0/2412587525 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4218da210 con 0x55a421c61940 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.039+0000 7f734a7fc640 1 -- 172.21.15.31:0/3341809149 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f734c0cf850 con 0x7f7354106630 2024-04-25T01:59:42.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.039+0000 7f7359825640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f73400a2da0 0x7f73400a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.039+0000 7f7359024640 1 --2- 172.21.15.31:0/2412587525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 0x55a421c74500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f735002a940 tx=0x7f7350032c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7013162b29de24a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.039+0000 7f733f7fe640 1 -- 172.21.15.31:0/2412587525 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735003f070 con 0x55a421c61940 2024-04-25T01:59:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.039+0000 7f7359825640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f73400a2da0 0x7f73400a5260 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f734c0cd050 tx=0x7f734c06acf0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.039+0000 7f733f7fe640 1 -- 172.21.15.31:0/2412587525 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7350004030 con 0x55a421c61940 2024-04-25T01:59:42.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.039+0000 7f733f7fe640 1 -- 172.21.15.31:0/2412587525 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350035070 con 0x55a421c61940 2024-04-25T01:59:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.040+0000 7f735df94740 1 -- 172.21.15.31:0/2412587525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 msgr2=0x55a421c74500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.040+0000 7f735df94740 1 --2- 172.21.15.31:0/2412587525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 0x55a421c74500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f735002a940 tx=0x7f7350032c00 comp rx=0 tx=0).stop 2024-04-25T01:59:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.040+0000 7f735df94740 1 -- 172.21.15.31:0/2412587525 shutdown_connections 2024-04-25T01:59:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.040+0000 7f735df94740 1 --2- 172.21.15.31:0/2412587525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c76e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.040+0000 7f735df94740 1 --2- 172.21.15.31:0/2412587525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 0x55a421c74500 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.040+0000 7f735df94740 1 --2- 172.21.15.31:0/2412587525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c61370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.040+0000 7f735df94740 1 -- 172.21.15.31:0/2412587525 >> 172.21.15.31:0/2412587525 conn(0x55a421c5d560 msgr2=0x55a421c5d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.040+0000 7f735df94740 1 -- 172.21.15.31:0/2412587525 shutdown_connections 2024-04-25T01:59:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.040+0000 7f735df94740 1 -- 172.21.15.31:0/2412587525 wait complete. 2024-04-25T01:59:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.040+0000 7f735df94740 1 -- start start 2024-04-25T01:59:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.040+0000 7f735df94740 1 Processor -- start 2024-04-25T01:59:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f735df94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f735df94740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c61940 0x55a421c71f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f735df94740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c74a40 0x55a421c72440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f735df94740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a421c777a0 con 0x55a421c60f90 2024-04-25T01:59:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f735df94740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a421b81b50 con 0x55a421c61940 2024-04-25T01:59:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f735df94740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a421c77920 con 0x55a421c74a40 2024-04-25T01:59:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f7358823640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f735df94740 10 client.0 _ll_register_callbacks cb 0x55a421bbcca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f7358823640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c719c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34460/0 (socket says 172.21.15.31:34460) 2024-04-25T01:59:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f7358823640 1 -- 172.21.15.31:0/1899506874 learned_addr learned my addr 172.21.15.31:0/1899506874 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f7359024640 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c61940 0x55a421c71f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f7358823640 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c74a40 0x55a421c72440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f7358823640 1 -- 172.21.15.31:0/1899506874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c74a40 msgr2=0x55a421c72440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f7358823640 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c74a40 0x55a421c72440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f7358823640 1 -- 172.21.15.31:0/1899506874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c61940 msgr2=0x55a421c71f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f7358823640 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c61940 0x55a421c71f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f7358823640 1 -- 172.21.15.31:0/1899506874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4219a6620 con 0x55a421c60f90 2024-04-25T01:59:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f7358823640 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c719c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f73540ff090 tx=0x7f7344014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5969f97f1503f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:42.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.041+0000 7f733cff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f733cff9640 1 -- 172.21.15.31:0/1899506874 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73440128b0 con 0x55a421c60f90 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f733cff9640 1 -- 172.21.15.31:0/1899506874 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7344012a50 con 0x55a421c60f90 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f735df94740 1 -- 172.21.15.31:0/1899506874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7350004030 con 0x55a421c60f90 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f733cff9640 1 -- 172.21.15.31:0/1899506874 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7344012c50 con 0x55a421c60f90 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f735df94740 10 client.16566 fetch_fsmap learned FSMap version 33 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f735df94740 1 -- 172.21.15.31:0/1899506874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a421c83a20 con 0x55a421c60f90 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f733cff9640 1 -- 172.21.15.31:0/1899506874 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7350004030 con 0x55a421c60f90 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f733cff9640 1 -- 172.21.15.31:0/1899506874 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7344021470 con 0x55a421c60f90 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f735df94740 10 client.16566 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f735df94740 10 client.16566 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f735df94740 20 client.16566 populate_metadata read hostname 'smithi031' 2024-04-25T01:59:42.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f735df94740 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a421c84e90 0x55a421c87280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f735df94740 4 client.16566 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.042+0000 7f735df94740 1 -- 172.21.15.31:0/1899506874 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a421bd31f0 con 0x55a421c84e90 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.043+0000 7f7359024640 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a421c84e90 0x55a421c87280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.043+0000 7f7359024640 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a421c84e90 0x55a421c87280 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7673fd5b9856d1e server_cookie=aa2d5a71e66736b4 in_seq=0 out_seq=0 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.043+0000 7f733cff9640 10 client.16566 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f733cff9640 1 -- 172.21.15.31:0/1899506874 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a421bd31f0 con 0x55a421c84e90 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f733cff9640 10 client.16566 handle_command_reply: tid=0 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 1 client.16566 shutdown 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 1 -- 172.21.15.31:0/1899506874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 msgr2=0x55a421c719c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c719c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f73540ff090 tx=0x7f7344014040 comp rx=0 tx=0).stop 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 1 -- 172.21.15.31:0/1899506874 shutdown_connections 2024-04-25T01:59:42.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a421c84e90 0x55a421c87280 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c74a40 0x55a421c72440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c61940 0x55a421c71f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f733cff9640 0 client.16566 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 1 --2- 172.21.15.31:0/1899506874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c719c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 1 -- 172.21.15.31:0/1899506874 >> 172.21.15.31:0/1899506874 conn(0x55a421c5d560 msgr2=0x55a421c63360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 1 -- 172.21.15.31:0/1899506874 shutdown_connections 2024-04-25T01:59:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 1 -- 172.21.15.31:0/1899506874 wait complete. 2024-04-25T01:59:42.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.044+0000 7f735df94740 20 client.16566 trim_cache size 0 max 16384 2024-04-25T01:59:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f735df94740 1 Processor -- start 2024-04-25T01:59:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f735df94740 1 -- start start 2024-04-25T01:59:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f735df94740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c60f90 0x55a421c806a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f735df94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c61940 0x55a421c5c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f735df94740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c8cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f735df94740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f73541aaf40 con 0x55a421c61940 2024-04-25T01:59:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f735df94740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f73541aca80 con 0x55a421c74a40 2024-04-25T01:59:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f735df94740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f73541ab610 con 0x55a421c60f90 2024-04-25T01:59:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f7359024640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c61940 0x55a421c5c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f7358823640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c60f90 0x55a421c806a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f7359024640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c61940 0x55a421c5c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34462/0 (socket says 172.21.15.31:34462) 2024-04-25T01:59:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f7359024640 1 -- 172.21.15.31:0/3784776234 learned_addr learned my addr 172.21.15.31:0/3784776234 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.061+0000 7f7358823640 1 --2- 172.21.15.31:0/3784776234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c8cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.062+0000 7f7359024640 1 -- 172.21.15.31:0/3784776234 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c60f90 msgr2=0x55a421c806a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.062+0000 7f7359024640 1 --2- 172.21.15.31:0/3784776234 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c60f90 0x55a421c806a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.062+0000 7f7359024640 1 -- 172.21.15.31:0/3784776234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 msgr2=0x55a421c8cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.062+0000 7f7359024640 1 --2- 172.21.15.31:0/3784776234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c8cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.062+0000 7f7359024640 1 -- 172.21.15.31:0/3784776234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a421c6ff30 con 0x55a421c61940 2024-04-25T01:59:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.062+0000 7f7359024640 1 --2- 172.21.15.31:0/3784776234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c61940 0x55a421c5c440 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f73500051e0 tx=0x7f7350005170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c9a5d53e29a955 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.062+0000 7f733effd640 1 -- 172.21.15.31:0/3784776234 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350035040 con 0x55a421c61940 2024-04-25T01:59:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.062+0000 7f733effd640 1 -- 172.21.15.31:0/3784776234 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7350031cf0 con 0x55a421c61940 2024-04-25T01:59:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.062+0000 7f733effd640 1 -- 172.21.15.31:0/3784776234 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735003f040 con 0x55a421c61940 2024-04-25T01:59:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.063+0000 7f735df94740 1 -- 172.21.15.31:0/3784776234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c61940 msgr2=0x55a421c5c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.063+0000 7f735df94740 1 --2- 172.21.15.31:0/3784776234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c61940 0x55a421c5c440 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f73500051e0 tx=0x7f7350005170 comp rx=0 tx=0).stop 2024-04-25T01:59:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.063+0000 7f735df94740 1 -- 172.21.15.31:0/3784776234 shutdown_connections 2024-04-25T01:59:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.063+0000 7f735df94740 1 --2- 172.21.15.31:0/3784776234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c8cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.063+0000 7f735df94740 1 --2- 172.21.15.31:0/3784776234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c61940 0x55a421c5c440 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.063+0000 7f735df94740 1 --2- 172.21.15.31:0/3784776234 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c60f90 0x55a421c806a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.063+0000 7f735df94740 1 -- 172.21.15.31:0/3784776234 >> 172.21.15.31:0/3784776234 conn(0x55a421c5d560 msgr2=0x55a421c5bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.063+0000 7f735df94740 1 -- 172.21.15.31:0/3784776234 shutdown_connections 2024-04-25T01:59:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.063+0000 7f735df94740 1 -- 172.21.15.31:0/3784776234 wait complete. 2024-04-25T01:59:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.063+0000 7f735df94740 1 -- start start 2024-04-25T01:59:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.063+0000 7f735df94740 1 Processor -- start 2024-04-25T01:59:42.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f735df94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c6e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f735df94740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 0x55a421c6eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f735df94740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c6f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f735df94740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f73541ac3b0 con 0x55a421c60f90 2024-04-25T01:59:42.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f735df94740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f73541abce0 con 0x55a421c74a40 2024-04-25T01:59:42.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f735df94740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f73541a8f60 con 0x55a421c61940 2024-04-25T01:59:42.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f735df94740 10 client.0 _ll_register_callbacks cb 0x7f734400bcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:42.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7358823640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c6e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7358823640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c6e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34464/0 (socket says 172.21.15.31:34464) 2024-04-25T01:59:42.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7358823640 1 -- 172.21.15.31:0/3137169752 learned_addr learned my addr 172.21.15.31:0/3137169752 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7359024640 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 0x55a421c6eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7358823640 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c6f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7359024640 1 -- 172.21.15.31:0/3137169752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 msgr2=0x55a421c6f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7359024640 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c6f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7359024640 1 -- 172.21.15.31:0/3137169752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 msgr2=0x55a421c6e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7359024640 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c6e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7359024640 1 -- 172.21.15.31:0/3137169752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a421cea4f0 con 0x55a421c61940 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7358823640 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c6e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f7359024640 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 0x55a421c6eda0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7350000c00 tx=0x7f7350032d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f8ce50eb7f24bf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.064+0000 7f733cff9640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f733cff9640 1 -- 172.21.15.31:0/3137169752 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350035070 con 0x55a421c61940 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f733cff9640 1 -- 172.21.15.31:0/3137169752 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f73500317f0 con 0x55a421c61940 2024-04-25T01:59:42.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f735df94740 1 -- 172.21.15.31:0/3137169752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7350031cf0 con 0x55a421c61940 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f733cff9640 1 -- 172.21.15.31:0/3137169752 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f735003f040 con 0x55a421c61940 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f735df94740 10 client.26008 fetch_fsmap learned FSMap version 33 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f735df94740 1 -- 172.21.15.31:0/3137169752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a421ceb7e0 con 0x55a421c61940 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f733cff9640 1 -- 172.21.15.31:0/3137169752 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7350031cf0 con 0x55a421c61940 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f733cff9640 1 -- 172.21.15.31:0/3137169752 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f73500046c0 con 0x55a421c61940 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f735df94740 10 client.26008 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f735df94740 10 client.26008 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f735df94740 20 client.26008 populate_metadata read hostname 'smithi031' 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f735df94740 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a421ceb9b0 0x55a421cedda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f735df94740 4 client.26008 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f735df94740 1 -- 172.21.15.31:0/3137169752 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a421bcd540 con 0x55a421ceb9b0 2024-04-25T01:59:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.065+0000 7f7358823640 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a421ceb9b0 0x55a421cedda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:42.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.066+0000 7f733cff9640 10 client.26008 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:42.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.066+0000 7f7358823640 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a421ceb9b0 0x55a421cedda0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=434c87f89db4c2f6 server_cookie=64292c8105ae8f46 in_seq=0 out_seq=0 2024-04-25T01:59:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f733cff9640 1 -- 172.21.15.31:0/3137169752 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a421bcd540 con 0x55a421ceb9b0 2024-04-25T01:59:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f733cff9640 10 client.26008 handle_command_reply: tid=0 2024-04-25T01:59:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 1 client.26008 shutdown 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 1 -- 172.21.15.31:0/3137169752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 msgr2=0x55a421c6eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 0x55a421c6eda0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7350000c00 tx=0x7f7350032d80 comp rx=0 tx=0).stop 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 1 -- 172.21.15.31:0/3137169752 shutdown_connections 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a421ceb9b0 0x55a421cedda0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a421c74a40 0x55a421c6f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a421c61940 0x55a421c6eda0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f733cff9640 0 client.26008 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 1 --2- 172.21.15.31:0/3137169752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a421c60f90 0x55a421c6e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 1 -- 172.21.15.31:0/3137169752 >> 172.21.15.31:0/3137169752 conn(0x55a421c5d560 msgr2=0x55a421c5c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 1 -- 172.21.15.31:0/3137169752 shutdown_connections 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 1 -- 172.21.15.31:0/3137169752 wait complete. 2024-04-25T01:59:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.087+0000 7f735df94740 20 client.26008 trim_cache size 0 max 16384 2024-04-25T01:59:42.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:59:42.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:59:42.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:59:42.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 -- 172.21.15.31:0/3341809149 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f73400a2da0 msgr2=0x7f73400a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f73400a2da0 0x7f73400a5260 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f734c0cd050 tx=0x7f734c06acf0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 -- 172.21.15.31:0/3341809149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7354106630 msgr2=0x7f73540fe080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7354106630 0x7f73540fe080 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f734c00efd0 tx=0x7f734c00c6a0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 -- 172.21.15.31:0/3341809149 shutdown_connections 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f73400a2da0 0x7f73400a5260 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7354106630 0x7f73540fe080 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7354101030 0x7f73540fdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 --2- 172.21.15.31:0/3341809149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7354100670 0x7f73540fd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 -- 172.21.15.31:0/3341809149 >> 172.21.15.31:0/3341809149 conn(0x7f73540777c0 msgr2=0x7f73540ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 -- 172.21.15.31:0/3341809149 shutdown_connections 2024-04-25T01:59:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:42.088+0000 7f735b288640 1 -- 172.21.15.31:0/3341809149 wait complete. 2024-04-25T01:59:42.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 110400 2.50 MB/sec execute 109 sec latency 15.592 ms 2024-04-25T01:59:42.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:42 smithi040 ceph-mon[32418]: pgmap v160: 97 pgs: 97 active+clean; 70 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 1.7 MiB/s wr, 181 op/s 2024-04-25T01:59:42.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:42 smithi134 ceph-mon[33098]: pgmap v160: 97 pgs: 97 active+clean; 70 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 1.7 MiB/s wr, 181 op/s 2024-04-25T01:59:42.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:42 smithi031 ceph-mon[29204]: pgmap v160: 97 pgs: 97 active+clean; 70 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 1.7 MiB/s wr, 181 op/s 2024-04-25T01:59:43.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 110665 2.48 MB/sec execute 110 sec latency 49.568 ms 2024-04-25T01:59:44.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 111165 2.47 MB/sec execute 111 sec latency 18.114 ms 2024-04-25T01:59:44.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:44 smithi040 ceph-mon[32418]: pgmap v161: 97 pgs: 97 active+clean; 70 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-04-25T01:59:44.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:44 smithi134 ceph-mon[33098]: pgmap v161: 97 pgs: 97 active+clean; 70 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-04-25T01:59:44.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:44 smithi031 ceph-mon[29204]: pgmap v161: 97 pgs: 97 active+clean; 70 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-04-25T01:59:45.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 111623 2.48 MB/sec execute 112 sec latency 23.155 ms 2024-04-25T01:59:46.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 112181 2.46 MB/sec execute 113 sec latency 23.652 ms 2024-04-25T01:59:46.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:46 smithi040 ceph-mon[32418]: pgmap v162: 97 pgs: 97 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-04-25T01:59:46.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:46 smithi134 ceph-mon[33098]: pgmap v162: 97 pgs: 97 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-04-25T01:59:46.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:46 smithi031 ceph-mon[29204]: pgmap v162: 97 pgs: 97 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-04-25T01:59:47.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 112606 2.46 MB/sec execute 114 sec latency 18.849 ms 2024-04-25T01:59:48.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 113304 2.49 MB/sec execute 115 sec latency 18.239 ms 2024-04-25T01:59:48.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:48 smithi040 ceph-mon[32418]: pgmap v163: 97 pgs: 97 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:48.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:48 smithi134 ceph-mon[33098]: pgmap v163: 97 pgs: 97 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:48.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:48 smithi031 ceph-mon[29204]: pgmap v163: 97 pgs: 97 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:49.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 113755 2.51 MB/sec execute 116 sec latency 17.929 ms 2024-04-25T01:59:50.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 113990 2.49 MB/sec execute 117 sec latency 13.827 ms 2024-04-25T01:59:50.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:50 smithi040 ceph-mon[32418]: pgmap v164: 97 pgs: 97 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:50.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:50 smithi134 ceph-mon[33098]: pgmap v164: 97 pgs: 97 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:50.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:50 smithi031 ceph-mon[29204]: pgmap v164: 97 pgs: 97 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-25T01:59:51.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 114335 2.47 MB/sec execute 118 sec latency 20.534 ms 2024-04-25T01:59:52.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 114799 2.47 MB/sec execute 119 sec latency 19.048 ms 2024-04-25T01:59:52.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:52 smithi040 ceph-mon[32418]: pgmap v165: 97 pgs: 97 active+clean; 72 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-25T01:59:52.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:52 smithi134 ceph-mon[33098]: pgmap v165: 97 pgs: 97 active+clean; 72 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-25T01:59:52.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:52 smithi031 ceph-mon[29204]: pgmap v165: 97 pgs: 97 active+clean; 72 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-25T01:59:53.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 115320 2.47 MB/sec execute 120 sec latency 23.305 ms 2024-04-25T01:59:54.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 115907 2.47 MB/sec execute 121 sec latency 24.393 ms 2024-04-25T01:59:54.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:54 smithi040 ceph-mon[32418]: pgmap v166: 97 pgs: 97 active+clean; 72 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-04-25T01:59:54.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:54 smithi134 ceph-mon[33098]: pgmap v166: 97 pgs: 97 active+clean; 72 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-04-25T01:59:54.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:54 smithi031 ceph-mon[29204]: pgmap v166: 97 pgs: 97 active+clean; 72 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-04-25T01:59:55.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.182+0000 7fbc2e431640 1 -- 172.21.15.31:0/2578856313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc2810bdc0 msgr2=0x7fbc2810e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.182+0000 7fbc2e431640 1 --2- 172.21.15.31:0/2578856313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc2810bdc0 0x7fbc2810e1f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009ca0 tx=0x7fbc2002a950 comp rx=0 tx=0).stop 2024-04-25T01:59:55.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.182+0000 7fbc2e431640 1 -- 172.21.15.31:0/2578856313 shutdown_connections 2024-04-25T01:59:55.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.182+0000 7fbc2e431640 1 --2- 172.21.15.31:0/2578856313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc2810bdc0 0x7fbc2810e1f0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.182+0000 7fbc2e431640 1 --2- 172.21.15.31:0/2578856313 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc28103310 0x7fbc2810b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.182+0000 7fbc2e431640 1 --2- 172.21.15.31:0/2578856313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc281029f0 0x7fbc28102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.182+0000 7fbc2e431640 1 -- 172.21.15.31:0/2578856313 >> 172.21.15.31:0/2578856313 conn(0x7fbc280769c0 msgr2=0x7fbc28076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.183+0000 7fbc2e431640 1 -- 172.21.15.31:0/2578856313 shutdown_connections 2024-04-25T01:59:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.183+0000 7fbc2e431640 1 -- 172.21.15.31:0/2578856313 wait complete. 2024-04-25T01:59:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.183+0000 7fbc2e431640 1 Processor -- start 2024-04-25T01:59:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.183+0000 7fbc2e431640 1 -- start start 2024-04-25T01:59:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc2e431640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc281029f0 0x7fbc28197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc2e431640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc28103310 0x7fbc28198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc2e431640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc2810bdc0 0x7fbc2819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc2e431640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbc28111a50 con 0x7fbc28103310 2024-04-25T01:59:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc2e431640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281118d0 con 0x7fbc281029f0 2024-04-25T01:59:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc2e431640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbc28111bd0 con 0x7fbc2810bdc0 2024-04-25T01:59:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc277fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc28103310 0x7fbc28198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc277fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc28103310 0x7fbc28198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35410/0 (socket says 172.21.15.31:35410) 2024-04-25T01:59:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc277fe640 1 -- 172.21.15.31:0/2125629146 learned_addr learned my addr 172.21.15.31:0/2125629146 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc27fff640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc281029f0 0x7fbc28197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc2c9ce640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc2810bdc0 0x7fbc2819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc277fe640 1 -- 172.21.15.31:0/2125629146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc2810bdc0 msgr2=0x7fbc2819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc277fe640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc2810bdc0 0x7fbc2819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc277fe640 1 -- 172.21.15.31:0/2125629146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc281029f0 msgr2=0x7fbc28197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc277fe640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc281029f0 0x7fbc28197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.184+0000 7fbc277fe640 1 -- 172.21.15.31:0/2125629146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2819ce00 con 0x7fbc28103310 2024-04-25T01:59:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.185+0000 7fbc277fe640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc28103310 0x7fbc28198350 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00e9f0 tx=0x7fbc1c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.185+0000 7fbc257fa640 1 -- 172.21.15.31:0/2125629146 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c00cd80 con 0x7fbc28103310 2024-04-25T01:59:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.185+0000 7fbc2e431640 1 -- 172.21.15.31:0/2125629146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2819d0f0 con 0x7fbc28103310 2024-04-25T01:59:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.185+0000 7fbc257fa640 1 -- 172.21.15.31:0/2125629146 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc1c0108e0 con 0x7fbc28103310 2024-04-25T01:59:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.185+0000 7fbc257fa640 1 -- 172.21.15.31:0/2125629146 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c018920 con 0x7fbc28103310 2024-04-25T01:59:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.185+0000 7fbc2e431640 1 -- 172.21.15.31:0/2125629146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc281a4980 con 0x7fbc28103310 2024-04-25T01:59:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.187+0000 7fbc257fa640 1 -- 172.21.15.31:0/2125629146 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fbc1c010a90 con 0x7fbc28103310 2024-04-25T01:59:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.187+0000 7fbc31218740 1 Processor -- start 2024-04-25T01:59:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.187+0000 7fbc31218740 1 -- start start 2024-04-25T01:59:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.187+0000 7fbc31218740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bb8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.187+0000 7fbc31218740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 0x559531bc2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.187+0000 7fbc31218740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bc3230 0x559531bc5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc31218740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559531b1ccb0 con 0x559531bb8240 2024-04-25T01:59:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc31218740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559531ac55b0 con 0x559531bc3230 2024-04-25T01:59:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc31218740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559531b18c70 con 0x559531bb8b60 2024-04-25T01:59:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc257fa640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbc100a2da0 0x7fbc100a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc27fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 0x559531bc2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc2c9ce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bb8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc27fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 0x559531bc2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:50590/0 (socket says 172.21.15.31:50590) 2024-04-25T01:59:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc27fff640 1 -- 172.21.15.31:0/1883425097 learned_addr learned my addr 172.21.15.31:0/1883425097 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc2c9ce640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbc100a2da0 0x7fbc100a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc257fa640 1 -- 172.21.15.31:0/2125629146 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fbc1c0cf740 con 0x7fbc28103310 2024-04-25T01:59:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc277fe640 1 --2- 172.21.15.31:0/1883425097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bc3230 0x559531bc5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc27fff640 1 -- 172.21.15.31:0/1883425097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bc3230 msgr2=0x559531bc5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc27fff640 1 --2- 172.21.15.31:0/1883425097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bc3230 0x559531bc5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc27fff640 1 -- 172.21.15.31:0/1883425097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 msgr2=0x559531bb8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc27fff640 1 --2- 172.21.15.31:0/1883425097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bb8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc27fff640 1 -- 172.21.15.31:0/1883425097 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559531831210 con 0x559531bb8b60 2024-04-25T01:59:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc2c9ce640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbc100a2da0 0x7fbc100a5260 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbc20006010 tx=0x7fbc200095c0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.188+0000 7fbc27fff640 1 --2- 172.21.15.31:0/1883425097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 0x559531bc2cf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbc18009cf0 tx=0x7fbc18011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3dde578ea94ba33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc0e7fc640 1 -- 172.21.15.31:0/1883425097 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1801e070 con 0x559531bb8b60 2024-04-25T01:59:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc0e7fc640 1 -- 172.21.15.31:0/1883425097 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc18019ae0 con 0x559531bb8b60 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc0e7fc640 1 -- 172.21.15.31:0/1883425097 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1800e930 con 0x559531bb8b60 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc31218740 1 -- 172.21.15.31:0/1883425097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 msgr2=0x559531bc2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc31218740 1 --2- 172.21.15.31:0/1883425097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 0x559531bc2cf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbc18009cf0 tx=0x7fbc18011810 comp rx=0 tx=0).stop 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc31218740 1 -- 172.21.15.31:0/1883425097 shutdown_connections 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc31218740 1 --2- 172.21.15.31:0/1883425097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bc3230 0x559531bc5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc31218740 1 --2- 172.21.15.31:0/1883425097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 0x559531bc2cf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc31218740 1 --2- 172.21.15.31:0/1883425097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bb8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc31218740 1 -- 172.21.15.31:0/1883425097 >> 172.21.15.31:0/1883425097 conn(0x559531bb4560 msgr2=0x559531bb4940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc31218740 1 -- 172.21.15.31:0/1883425097 shutdown_connections 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.189+0000 7fbc31218740 1 -- 172.21.15.31:0/1883425097 wait complete. 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc31218740 1 -- start start 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc31218740 1 Processor -- start 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc31218740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bc00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc31218740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 0x559531bc0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc31218740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bc3230 0x559531bc0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc31218740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559531bc6020 con 0x559531bb8240 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc31218740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559531ad8b50 con 0x559531bb8b60 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc31218740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559531bc61a0 con 0x559531bc3230 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc31218740 10 client.0 _ll_register_callbacks cb 0x559531b13ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc27fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bc00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc27fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bc00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35426/0 (socket says 172.21.15.31:35426) 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc27fff640 1 -- 172.21.15.31:0/154760955 learned_addr learned my addr 172.21.15.31:0/154760955 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc277fe640 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 0x559531bc0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.190+0000 7fbc2c9ce640 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bc3230 0x559531bc0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc277fe640 1 -- 172.21.15.31:0/154760955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bc3230 msgr2=0x559531bc0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc277fe640 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bc3230 0x559531bc0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc277fe640 1 -- 172.21.15.31:0/154760955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 msgr2=0x559531bc00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc277fe640 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bc00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc277fe640 1 -- 172.21.15.31:0/154760955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595318fd620 con 0x559531bb8b60 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc277fe640 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 0x559531bc0610 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0ccee0 tx=0x7fbc1c09b700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b82191926734d5e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc077fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc077fe640 1 -- 172.21.15.31:0/154760955 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c0a0060 con 0x559531bb8b60 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc077fe640 1 -- 172.21.15.31:0/154760955 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fbc1c010430 con 0x559531bb8b60 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc31218740 1 -- 172.21.15.31:0/154760955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc18019ae0 con 0x559531bb8b60 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc077fe640 1 -- 172.21.15.31:0/154760955 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fbc1c0a4920 con 0x559531bb8b60 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc31218740 10 client.26186 fetch_fsmap learned FSMap version 33 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc31218740 1 -- 172.21.15.31:0/154760955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559531bd21b0 con 0x559531bb8b60 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc077fe640 1 -- 172.21.15.31:0/154760955 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc18019ae0 con 0x559531bb8b60 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc077fe640 1 -- 172.21.15.31:0/154760955 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fbc1c097a90 con 0x559531bb8b60 2024-04-25T01:59:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc31218740 10 client.26186 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc31218740 10 client.26186 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.191+0000 7fbc31218740 20 client.26186 populate_metadata read hostname 'smithi031' 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.192+0000 7fbc31218740 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559531bd3620 0x559531bd5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.192+0000 7fbc31218740 4 client.26186 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.192+0000 7fbc31218740 1 -- 172.21.15.31:0/154760955 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559531b2a1f0 con 0x559531bd3620 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.192+0000 7fbc27fff640 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559531bd3620 0x559531bd5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.192+0000 7fbc27fff640 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559531bd3620 0x559531bd5a30 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ba1f3f95caa9736 server_cookie=be7e30f3a389d31f in_seq=0 out_seq=0 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.192+0000 7fbc077fe640 10 client.26186 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc077fe640 1 -- 172.21.15.31:0/154760955 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559531b2a1f0 con 0x559531bd3620 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc077fe640 10 client.26186 handle_command_reply: tid=0 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 1 client.26186 shutdown 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 1 -- 172.21.15.31:0/154760955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 msgr2=0x559531bc0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 0x559531bc0610 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0ccee0 tx=0x7fbc1c09b700 comp rx=0 tx=0).stop 2024-04-25T01:59:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 1 -- 172.21.15.31:0/154760955 shutdown_connections 2024-04-25T01:59:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559531bd3620 0x559531bd5a30 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bc3230 0x559531bc0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 0x559531bc0610 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 1 --2- 172.21.15.31:0/154760955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bc00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc077fe640 0 client.26186 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 1 -- 172.21.15.31:0/154760955 >> 172.21.15.31:0/154760955 conn(0x559531bb4560 msgr2=0x559531bba260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 1 -- 172.21.15.31:0/154760955 shutdown_connections 2024-04-25T01:59:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 1 -- 172.21.15.31:0/154760955 wait complete. 2024-04-25T01:59:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.193+0000 7fbc31218740 20 client.26186 trim_cache size 0 max 16384 2024-04-25T01:59:55.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc31218740 1 Processor -- start 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc31218740 1 -- start start 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc31218740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bcef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc31218740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 0x559531bbf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc31218740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bc3230 0x559531bb3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc31218740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281ad1b0 con 0x559531bb8240 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc31218740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281aecf0 con 0x559531bb8b60 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc31218740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281ad880 con 0x559531bc3230 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc27fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bcef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc277fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 0x559531bbf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc27fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bcef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35428/0 (socket says 172.21.15.31:35428) 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc27fff640 1 -- 172.21.15.31:0/1416724815 learned_addr learned my addr 172.21.15.31:0/1416724815 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc2c9ce640 1 --2- 172.21.15.31:0/1416724815 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bc3230 0x559531bb3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc277fe640 1 -- 172.21.15.31:0/1416724815 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bc3230 msgr2=0x559531bb3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc277fe640 1 --2- 172.21.15.31:0/1416724815 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bc3230 0x559531bb3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc277fe640 1 -- 172.21.15.31:0/1416724815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 msgr2=0x559531bcef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc277fe640 1 --2- 172.21.15.31:0/1416724815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bcef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc277fe640 1 -- 172.21.15.31:0/1416724815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559531bb8990 con 0x559531bb8b60 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.209+0000 7fbc27fff640 1 --2- 172.21.15.31:0/1416724815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bcef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc277fe640 1 --2- 172.21.15.31:0/1416724815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 0x559531bbf230 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0ccf90 tx=0x7fbc1c09b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a70d460e14b22297 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc0dffb640 1 -- 172.21.15.31:0/1416724815 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c0a0060 con 0x559531bb8b60 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc0dffb640 1 -- 172.21.15.31:0/1416724815 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc1c018ac0 con 0x559531bb8b60 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc0dffb640 1 -- 172.21.15.31:0/1416724815 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c097f50 con 0x559531bb8b60 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc31218740 1 -- 172.21.15.31:0/1416724815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 msgr2=0x559531bbf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc31218740 1 --2- 172.21.15.31:0/1416724815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 0x559531bbf230 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0ccf90 tx=0x7fbc1c09b610 comp rx=0 tx=0).stop 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc31218740 1 -- 172.21.15.31:0/1416724815 shutdown_connections 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc31218740 1 --2- 172.21.15.31:0/1416724815 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bc3230 0x559531bb3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc31218740 1 --2- 172.21.15.31:0/1416724815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8b60 0x559531bbf230 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc31218740 1 --2- 172.21.15.31:0/1416724815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bb8240 0x559531bcef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc31218740 1 -- 172.21.15.31:0/1416724815 >> 172.21.15.31:0/1416724815 conn(0x559531bb4560 msgr2=0x559531bb2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.210+0000 7fbc31218740 1 -- 172.21.15.31:0/1416724815 shutdown_connections 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc31218740 1 -- 172.21.15.31:0/1416724815 wait complete. 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc31218740 1 -- start start 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc31218740 1 Processor -- start 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc31218740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8240 0x559531bcb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc31218740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 0x559531bbc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc31218740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bc3230 0x559531bbce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc31218740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281ae620 con 0x559531bc3230 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc31218740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281adf50 con 0x559531bb8240 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc31218740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281ab190 con 0x559531bb8b60 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc31218740 10 client.0 _ll_register_callbacks cb 0x7fbc1c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc27fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8240 0x559531bcb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc2c9ce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bc3230 0x559531bbce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc27fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8240 0x559531bcb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33898/0 (socket says 172.21.15.31:33898) 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc27fff640 1 -- 172.21.15.31:0/3840910226 learned_addr learned my addr 172.21.15.31:0/3840910226 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.211+0000 7fbc277fe640 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 0x559531bbc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc2c9ce640 1 -- 172.21.15.31:0/3840910226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 msgr2=0x559531bbc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc2c9ce640 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 0x559531bbc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc2c9ce640 1 -- 172.21.15.31:0/3840910226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8240 msgr2=0x559531bcb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc2c9ce640 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8240 0x559531bcb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc2c9ce640 1 -- 172.21.15.31:0/3840910226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559531bbd4f0 con 0x559531bc3230 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc27fff640 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8240 0x559531bcb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc2c9ce640 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bc3230 0x559531bbce80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fbc2002af40 tx=0x7fbc2002a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f66bd56a6bdde4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc077fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc077fe640 1 -- 172.21.15.31:0/3840910226 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2005a060 con 0x559531bc3230 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc077fe640 1 -- 172.21.15.31:0/3840910226 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fbc2004dcb0 con 0x559531bc3230 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc31218740 1 -- 172.21.15.31:0/3840910226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc1c018ac0 con 0x559531bc3230 2024-04-25T01:59:55.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc077fe640 1 -- 172.21.15.31:0/3840910226 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fbc20034040 con 0x559531bc3230 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc31218740 10 client.16590 fetch_fsmap learned FSMap version 33 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc31218740 1 -- 172.21.15.31:0/3840910226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559531c39a30 con 0x559531bc3230 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.212+0000 7fbc077fe640 1 -- 172.21.15.31:0/3840910226 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc1c018ac0 con 0x559531bc3230 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.213+0000 7fbc077fe640 1 -- 172.21.15.31:0/3840910226 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fbc20054030 con 0x559531bc3230 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.213+0000 7fbc31218740 10 client.16590 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.213+0000 7fbc31218740 10 client.16590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.213+0000 7fbc31218740 20 client.16590 populate_metadata read hostname 'smithi031' 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.213+0000 7fbc31218740 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559531c3a150 0x559531c3c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.213+0000 7fbc31218740 4 client.16590 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.213+0000 7fbc31218740 1 -- 172.21.15.31:0/3840910226 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559531b24540 con 0x559531c3a150 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.213+0000 7fbc27fff640 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559531c3a150 0x559531c3c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.213+0000 7fbc27fff640 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559531c3a150 0x559531c3c570 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f475dc8efa1559d server_cookie=560b2d5a9771ca84 in_seq=0 out_seq=0 2024-04-25T01:59:55.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.213+0000 7fbc077fe640 10 client.16590 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc077fe640 1 -- 172.21.15.31:0/3840910226 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559531b24540 con 0x559531c3a150 2024-04-25T01:59:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc077fe640 10 client.16590 handle_command_reply: tid=0 2024-04-25T01:59:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 1 client.16590 shutdown 2024-04-25T01:59:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 1 -- 172.21.15.31:0/3840910226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bc3230 msgr2=0x559531bbce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bc3230 0x559531bbce80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fbc2002af40 tx=0x7fbc2002a950 comp rx=0 tx=0).stop 2024-04-25T01:59:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 1 -- 172.21.15.31:0/3840910226 shutdown_connections 2024-04-25T01:59:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559531c3a150 0x559531c3c570 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559531bc3230 0x559531bbce80 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559531bb8b60 0x559531bbc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc077fe640 0 client.16590 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T01:59:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 1 --2- 172.21.15.31:0/3840910226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559531bb8240 0x559531bcb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 1 -- 172.21.15.31:0/3840910226 >> 172.21.15.31:0/3840910226 conn(0x559531bb4560 msgr2=0x559531bb35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 1 -- 172.21.15.31:0/3840910226 shutdown_connections 2024-04-25T01:59:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 1 -- 172.21.15.31:0/3840910226 wait complete. 2024-04-25T01:59:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.233+0000 7fbc31218740 20 client.16590 trim_cache size 0 max 16384 2024-04-25T01:59:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T01:59:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T01:59:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T01:59:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T01:59:55.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 -- 172.21.15.31:0/2125629146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbc100a2da0 msgr2=0x7fbc100a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbc100a2da0 0x7fbc100a5260 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbc20006010 tx=0x7fbc200095c0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 -- 172.21.15.31:0/2125629146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc28103310 msgr2=0x7fbc28198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc28103310 0x7fbc28198350 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00e9f0 tx=0x7fbc1c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 -- 172.21.15.31:0/2125629146 shutdown_connections 2024-04-25T01:59:55.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbc100a2da0 0x7fbc100a5260 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc2810bdc0 0x7fbc2819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc28103310 0x7fbc28198350 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 --2- 172.21.15.31:0/2125629146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc281029f0 0x7fbc28197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 -- 172.21.15.31:0/2125629146 >> 172.21.15.31:0/2125629146 conn(0x7fbc280769c0 msgr2=0x7fbc280fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 -- 172.21.15.31:0/2125629146 shutdown_connections 2024-04-25T01:59:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T01:59:55.234+0000 7fbc2e431640 1 -- 172.21.15.31:0/2125629146 wait complete. 2024-04-25T01:59:55.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 116506 2.49 MB/sec execute 122 sec latency 18.607 ms 2024-04-25T01:59:56.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 116916 2.49 MB/sec execute 123 sec latency 19.349 ms 2024-04-25T01:59:56.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:56 smithi040 ceph-mon[32418]: pgmap v167: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T01:59:56.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:56 smithi134 ceph-mon[33098]: pgmap v167: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T01:59:56.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:56 smithi031 ceph-mon[29204]: pgmap v167: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T01:59:57.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 117375 2.50 MB/sec execute 124 sec latency 18.473 ms 2024-04-25T01:59:58.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 117643 2.49 MB/sec execute 125 sec latency 15.243 ms 2024-04-25T01:59:58.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 01:59:58 smithi040 ceph-mon[32418]: pgmap v168: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-04-25T01:59:58.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 01:59:58 smithi134 ceph-mon[33098]: pgmap v168: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-04-25T01:59:58.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 01:59:58 smithi031 ceph-mon[29204]: pgmap v168: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-04-25T01:59:59.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 118126 2.48 MB/sec execute 126 sec latency 17.495 ms 2024-04-25T02:00:00.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 118609 2.49 MB/sec execute 127 sec latency 19.906 ms 2024-04-25T02:00:00.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:00 smithi040 ceph-mon[32418]: pgmap v169: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-04-25T02:00:00.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:00 smithi040 ceph-mon[32418]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:00:00.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:00 smithi040 ceph-mon[32418]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-25T02:00:00.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:00 smithi040 ceph-mon[32418]: mds.h(mds.2): Metadata damage detected 2024-04-25T02:00:00.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:00 smithi134 ceph-mon[33098]: pgmap v169: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-04-25T02:00:00.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:00 smithi134 ceph-mon[33098]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:00:00.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:00 smithi134 ceph-mon[33098]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-25T02:00:00.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:00 smithi134 ceph-mon[33098]: mds.h(mds.2): Metadata damage detected 2024-04-25T02:00:00.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a[29200]: 2024-04-25T02:00:00.000+0000 7fc418dd9700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:00:00.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a[29200]: 2024-04-25T02:00:00.000+0000 7fc418dd9700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-25T02:00:00.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a[29200]: 2024-04-25T02:00:00.000+0000 7fc418dd9700 -1 log_channel(cluster) log [ERR] : mds.h(mds.2): Metadata damage detected 2024-04-25T02:00:00.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:00 smithi031 ceph-mon[29204]: pgmap v169: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-04-25T02:00:00.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:00 smithi031 ceph-mon[29204]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:00:00.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:00 smithi031 ceph-mon[29204]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-25T02:00:00.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:00 smithi031 ceph-mon[29204]: mds.h(mds.2): Metadata damage detected 2024-04-25T02:00:01.308 INFO:tasks.workunit.client.0.smithi031.stdout: 1 119146 2.47 MB/sec execute 128 sec latency 26.411 ms 2024-04-25T02:00:02.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 119639 2.47 MB/sec execute 129 sec latency 19.600 ms 2024-04-25T02:00:02.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:02 smithi040 ceph-mon[32418]: pgmap v170: 97 pgs: 97 active+clean; 73 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T02:00:02.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:02 smithi134 ceph-mon[33098]: pgmap v170: 97 pgs: 97 active+clean; 73 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T02:00:02.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:02 smithi031 ceph-mon[29204]: pgmap v170: 97 pgs: 97 active+clean; 73 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T02:00:03.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 120368 2.50 MB/sec execute 130 sec latency 42.873 ms 2024-04-25T02:00:04.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 120729 2.51 MB/sec execute 131 sec latency 18.457 ms 2024-04-25T02:00:04.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:04 smithi040 ceph-mon[32418]: pgmap v171: 97 pgs: 97 active+clean; 73 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T02:00:04.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:04 smithi134 ceph-mon[33098]: pgmap v171: 97 pgs: 97 active+clean; 73 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T02:00:04.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:04 smithi031 ceph-mon[29204]: pgmap v171: 97 pgs: 97 active+clean; 73 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T02:00:05.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 121027 2.50 MB/sec execute 132 sec latency 16.386 ms 2024-04-25T02:00:06.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 121328 2.48 MB/sec execute 133 sec latency 15.831 ms 2024-04-25T02:00:06.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:06 smithi040 ceph-mon[32418]: pgmap v172: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.9 MiB/s wr, 196 op/s 2024-04-25T02:00:06.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:06 smithi134 ceph-mon[33098]: pgmap v172: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.9 MiB/s wr, 196 op/s 2024-04-25T02:00:06.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:06 smithi031 ceph-mon[29204]: pgmap v172: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.9 MiB/s wr, 196 op/s 2024-04-25T02:00:07.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 121815 2.48 MB/sec execute 134 sec latency 18.234 ms 2024-04-25T02:00:08.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 122325 2.48 MB/sec execute 135 sec latency 21.885 ms 2024-04-25T02:00:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.330+0000 7f2e563b7640 1 -- 172.21.15.31:0/4254110671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e50102850 msgr2=0x7f2e50102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.330+0000 7f2e563b7640 1 --2- 172.21.15.31:0/4254110671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e50102850 0x7f2e50102c30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009900 tx=0x7f2e4002a480 comp rx=0 tx=0).stop 2024-04-25T02:00:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.330+0000 7f2e563b7640 1 -- 172.21.15.31:0/4254110671 shutdown_connections 2024-04-25T02:00:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.330+0000 7f2e563b7640 1 --2- 172.21.15.31:0/4254110671 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5010bd70 0x7f2e5010e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.330+0000 7f2e563b7640 1 --2- 172.21.15.31:0/4254110671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e50103170 0x7f2e5010b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.330+0000 7f2e563b7640 1 --2- 172.21.15.31:0/4254110671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e50102850 0x7f2e50102c30 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.330+0000 7f2e563b7640 1 -- 172.21.15.31:0/4254110671 >> 172.21.15.31:0/4254110671 conn(0x7f2e500777c0 msgr2=0x7f2e50077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.330+0000 7f2e563b7640 1 -- 172.21.15.31:0/4254110671 shutdown_connections 2024-04-25T02:00:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.330+0000 7f2e563b7640 1 -- 172.21.15.31:0/4254110671 wait complete. 2024-04-25T02:00:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.330+0000 7f2e563b7640 1 Processor -- start 2024-04-25T02:00:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.330+0000 7f2e563b7640 1 -- start start 2024-04-25T02:00:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e563b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e50102850 0x7f2e5019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e563b7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e50103170 0x7f2e5019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e563b7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5010bd70 0x7f2e5019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e563b7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2e50111980 con 0x7f2e50102850 2024-04-25T02:00:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e563b7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2e50111800 con 0x7f2e50103170 2024-04-25T02:00:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e563b7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2e50111b00 con 0x7f2e5010bd70 2024-04-25T02:00:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e50102850 0x7f2e5019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e50102850 0x7f2e5019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45754/0 (socket says 172.21.15.31:45754) 2024-04-25T02:00:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e4ffff640 1 -- 172.21.15.31:0/2511356335 learned_addr learned my addr 172.21.15.31:0/2511356335 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e50103170 0x7f2e5019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e54954640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5010bd70 0x7f2e5019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e4ffff640 1 -- 172.21.15.31:0/2511356335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5010bd70 msgr2=0x7f2e5019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5010bd70 0x7f2e5019ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e4ffff640 1 -- 172.21.15.31:0/2511356335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e50103170 msgr2=0x7f2e5019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e50103170 0x7f2e5019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e4ffff640 1 -- 172.21.15.31:0/2511356335 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5019f1b0 con 0x7f2e50102850 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e50103170 0x7f2e5019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.331+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e50102850 0x7f2e5019a1c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2e40005940 tx=0x7f2e400040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.332+0000 7f2e4d7fa640 1 -- 172.21.15.31:0/2511356335 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4003f030 con 0x7f2e50102850 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.332+0000 7f2e563b7640 1 -- 172.21.15.31:0/2511356335 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5019f440 con 0x7f2e50102850 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.332+0000 7f2e4d7fa640 1 -- 172.21.15.31:0/2511356335 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e400044c0 con 0x7f2e50102850 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.332+0000 7f2e4d7fa640 1 -- 172.21.15.31:0/2511356335 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40035070 con 0x7f2e50102850 2024-04-25T02:00:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.332+0000 7f2e563b7640 1 -- 172.21.15.31:0/2511356335 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e501a6d30 con 0x7f2e50102850 2024-04-25T02:00:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.334+0000 7f2e4d7fa640 1 -- 172.21.15.31:0/2511356335 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2e40042030 con 0x7f2e50102850 2024-04-25T02:00:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e591a2740 1 Processor -- start 2024-04-25T02:00:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e591a2740 1 -- start start 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 0x55e325eab620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eabb60 0x55e325eb5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eb6230 0x55e325eb8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e591a2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e325e0fcb0 con 0x55e325eab240 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e591a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e325db85b0 con 0x55e325eb6230 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e591a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e325e0bc70 con 0x55e325eabb60 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e4f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eb6230 0x55e325eb8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e4d7fa640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2e380a2da0 0x7f2e380a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e4f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eb6230 0x55e325eb8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42206/0 (socket says 172.21.15.31:42206) 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e4f7fe640 1 -- 172.21.15.31:0/2997004252 learned_addr learned my addr 172.21.15.31:0/2997004252 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/2997004252 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eabb60 0x55e325eb5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e54954640 1 --2- 172.21.15.31:0/2997004252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 0x55e325eab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e4f7fe640 1 -- 172.21.15.31:0/2997004252 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eabb60 msgr2=0x55e325eb5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/2997004252 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eabb60 0x55e325eb5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e54954640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2e380a2da0 0x7f2e380a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e4f7fe640 1 -- 172.21.15.31:0/2997004252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 msgr2=0x55e325eab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/2997004252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 0x55e325eab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e4f7fe640 1 -- 172.21.15.31:0/2997004252 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e325b24210 con 0x55e325eb6230 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.335+0000 7f2e4d7fa640 1 -- 172.21.15.31:0/2511356335 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f2e400ef710 con 0x7f2e50102850 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.336+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/2997004252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eb6230 0x55e325eb8620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2e440114f0 tx=0x7f2e440119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a43e874d7977a9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.336+0000 7f2e54954640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2e380a2da0 0x7f2e380a5260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009fd0 tx=0x7f2e48004ab0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.336+0000 7f2e367fc640 1 -- 172.21.15.31:0/2997004252 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4401e070 con 0x55e325eb6230 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.336+0000 7f2e367fc640 1 -- 172.21.15.31:0/2997004252 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e440035f0 con 0x55e325eb6230 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.336+0000 7f2e367fc640 1 -- 172.21.15.31:0/2997004252 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4400f820 con 0x55e325eb6230 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 -- 172.21.15.31:0/2997004252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eb6230 msgr2=0x55e325eb8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 --2- 172.21.15.31:0/2997004252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eb6230 0x55e325eb8620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2e440114f0 tx=0x7f2e440119b0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 -- 172.21.15.31:0/2997004252 shutdown_connections 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 --2- 172.21.15.31:0/2997004252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eb6230 0x55e325eb8620 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 --2- 172.21.15.31:0/2997004252 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eabb60 0x55e325eb5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 --2- 172.21.15.31:0/2997004252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 0x55e325eab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 -- 172.21.15.31:0/2997004252 >> 172.21.15.31:0/2997004252 conn(0x55e325ea7560 msgr2=0x55e325ea7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 -- 172.21.15.31:0/2997004252 shutdown_connections 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 -- 172.21.15.31:0/2997004252 wait complete. 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 -- start start 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 Processor -- start 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eab240 0x55e325eb31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.337+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eabb60 0x55e325eb36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eb6230 0x55e325eb3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e591a2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e325eb9020 con 0x55e325eabb60 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e591a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e325dcbb50 con 0x55e325eab240 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e591a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e325eb91a0 con 0x55e325eb6230 2024-04-25T02:00:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e591a2740 10 client.0 _ll_register_callbacks cb 0x55e325e06ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e4f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eabb60 0x55e325eb36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e4f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eabb60 0x55e325eb36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45762/0 (socket says 172.21.15.31:45762) 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e4f7fe640 1 -- 172.21.15.31:0/826954483 learned_addr learned my addr 172.21.15.31:0/826954483 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eab240 0x55e325eb31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e54954640 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eb6230 0x55e325eb3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e4f7fe640 1 -- 172.21.15.31:0/826954483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eb6230 msgr2=0x55e325eb3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eb6230 0x55e325eb3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e4f7fe640 1 -- 172.21.15.31:0/826954483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eab240 msgr2=0x55e325eb31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eab240 0x55e325eb31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e4f7fe640 1 -- 172.21.15.31:0/826954483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e325bf0620 con 0x55e325eabb60 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eab240 0x55e325eb31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eabb60 0x55e325eb36e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2e44005890 tx=0x7f2e44003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f19c8b04f52a06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e2f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/826954483 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4401e040 con 0x55e325eabb60 2024-04-25T02:00:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/826954483 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f2e44016050 con 0x55e325eabb60 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e591a2740 1 -- 172.21.15.31:0/826954483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e440035f0 con 0x55e325eabb60 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.338+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/826954483 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2e44003c40 con 0x55e325eabb60 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.339+0000 7f2e591a2740 10 client.16602 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.339+0000 7f2e591a2740 1 -- 172.21.15.31:0/826954483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e325ec51f0 con 0x55e325eabb60 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.339+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/826954483 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e440035f0 con 0x55e325eabb60 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.339+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/826954483 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2e44021030 con 0x55e325eabb60 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.339+0000 7f2e591a2740 10 client.16602 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.339+0000 7f2e591a2740 10 client.16602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.339+0000 7f2e591a2740 20 client.16602 populate_metadata read hostname 'smithi031' 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.339+0000 7f2e591a2740 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e325ec6600 0x55e325ec89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.339+0000 7f2e591a2740 4 client.16602 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.339+0000 7f2e591a2740 1 -- 172.21.15.31:0/826954483 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e325e1d1f0 con 0x55e325ec6600 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.339+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e325ec6600 0x55e325ec89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.340+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e325ec6600 0x55e325ec89f0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=754d7a138ddf04b4 server_cookie=1016f399654a4907 in_seq=0 out_seq=0 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.340+0000 7f2e2f7fe640 10 client.16602 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.340+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/826954483 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e325e1d1f0 con 0x55e325ec6600 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.340+0000 7f2e2f7fe640 10 client.16602 handle_command_reply: tid=0 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.340+0000 7f2e591a2740 1 client.16602 shutdown 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.340+0000 7f2e591a2740 1 -- 172.21.15.31:0/826954483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eabb60 msgr2=0x55e325eb36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.340+0000 7f2e591a2740 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eabb60 0x55e325eb36e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2e44005890 tx=0x7f2e44003a50 comp rx=0 tx=0).stop 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.341+0000 7f2e591a2740 1 -- 172.21.15.31:0/826954483 shutdown_connections 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.341+0000 7f2e591a2740 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e325ec6600 0x55e325ec89f0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.341+0000 7f2e591a2740 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eb6230 0x55e325eb3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.341+0000 7f2e591a2740 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eabb60 0x55e325eb36e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.341+0000 7f2e2f7fe640 0 client.16602 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.341+0000 7f2e591a2740 1 --2- 172.21.15.31:0/826954483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eab240 0x55e325eb31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.341+0000 7f2e591a2740 1 -- 172.21.15.31:0/826954483 >> 172.21.15.31:0/826954483 conn(0x55e325ea7560 msgr2=0x55e325ead360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.341+0000 7f2e591a2740 1 -- 172.21.15.31:0/826954483 shutdown_connections 2024-04-25T02:00:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.341+0000 7f2e591a2740 1 -- 172.21.15.31:0/826954483 wait complete. 2024-04-25T02:00:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.341+0000 7f2e591a2740 20 client.16602 trim_cache size 0 max 16384 2024-04-25T02:00:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e591a2740 1 Processor -- start 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e591a2740 1 -- start start 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eab240 0x55e325eb58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 0x55e325ec1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eb6230 0x55e325ece1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e591a2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2e501af430 con 0x55e325eb6230 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e591a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2e501b0f70 con 0x55e325eabb60 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e591a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2e501afb00 con 0x55e325eab240 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e54954640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eb6230 0x55e325ece1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e54954640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eb6230 0x55e325ece1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45770/0 (socket says 172.21.15.31:45770) 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e54954640 1 -- 172.21.15.31:0/3891748907 learned_addr learned my addr 172.21.15.31:0/3891748907 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/3891748907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eab240 0x55e325eb58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.357+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/3891748907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 0x55e325ec1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.358+0000 7f2e54954640 1 -- 172.21.15.31:0/3891748907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eab240 msgr2=0x55e325eb58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.358+0000 7f2e54954640 1 --2- 172.21.15.31:0/3891748907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eab240 0x55e325eb58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.358+0000 7f2e54954640 1 -- 172.21.15.31:0/3891748907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 msgr2=0x55e325ec1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.358+0000 7f2e54954640 1 --2- 172.21.15.31:0/3891748907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 0x55e325ec1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.358+0000 7f2e54954640 1 -- 172.21.15.31:0/3891748907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e325eb16f0 con 0x55e325eb6230 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.358+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/3891748907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 0x55e325ec1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.358+0000 7f2e54954640 1 --2- 172.21.15.31:0/3891748907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eb6230 0x55e325ece1c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2e48013ae0 tx=0x7f2e48009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe7fdc2afedfc00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.358+0000 7f2e35ffb640 1 -- 172.21.15.31:0/3891748907 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4802f070 con 0x55e325eb6230 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.358+0000 7f2e35ffb640 1 -- 172.21.15.31:0/3891748907 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e4800fe40 con 0x55e325eb6230 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.358+0000 7f2e35ffb640 1 -- 172.21.15.31:0/3891748907 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48010cf0 con 0x55e325eb6230 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.359+0000 7f2e591a2740 1 -- 172.21.15.31:0/3891748907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eb6230 msgr2=0x55e325ece1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.359+0000 7f2e591a2740 1 --2- 172.21.15.31:0/3891748907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eb6230 0x55e325ece1c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2e48013ae0 tx=0x7f2e48009c80 comp rx=0 tx=0).stop 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.359+0000 7f2e591a2740 1 -- 172.21.15.31:0/3891748907 shutdown_connections 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.359+0000 7f2e591a2740 1 --2- 172.21.15.31:0/3891748907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eb6230 0x55e325ece1c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.359+0000 7f2e591a2740 1 --2- 172.21.15.31:0/3891748907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 0x55e325ec1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.359+0000 7f2e591a2740 1 --2- 172.21.15.31:0/3891748907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eab240 0x55e325eb58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.359+0000 7f2e591a2740 1 -- 172.21.15.31:0/3891748907 >> 172.21.15.31:0/3891748907 conn(0x55e325ea7560 msgr2=0x55e325ea7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.359+0000 7f2e591a2740 1 -- 172.21.15.31:0/3891748907 shutdown_connections 2024-04-25T02:00:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.359+0000 7f2e591a2740 1 -- 172.21.15.31:0/3891748907 wait complete. 2024-04-25T02:00:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.359+0000 7f2e591a2740 1 -- start start 2024-04-25T02:00:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e591a2740 1 Processor -- start 2024-04-25T02:00:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 0x55e325ebe5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 0x55e325eafa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e591a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eb6230 0x55e325eaff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e591a2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2e501b08a0 con 0x55e325eab240 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e591a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2e501b01d0 con 0x55e325eabb60 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e591a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2e501ad410 con 0x55e325eb6230 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e591a2740 10 client.0 _ll_register_callbacks cb 0x7f2e4401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 0x55e325ebe5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 0x55e325ebe5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45780/0 (socket says 172.21.15.31:45780) 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e4ffff640 1 -- 172.21.15.31:0/1093256945 learned_addr learned my addr 172.21.15.31:0/1093256945 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 0x55e325eafa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e54954640 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eb6230 0x55e325eaff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e4ffff640 1 -- 172.21.15.31:0/1093256945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eb6230 msgr2=0x55e325eaff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eb6230 0x55e325eaff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e4ffff640 1 -- 172.21.15.31:0/1093256945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 msgr2=0x55e325eafa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 0x55e325eafa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e4ffff640 1 -- 172.21.15.31:0/1093256945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e325eb0530 con 0x55e325eab240 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e54954640 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eb6230 0x55e325eaff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.360+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 0x55e325eafa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e4ffff640 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 0x55e325ebe5e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2e400ecf10 tx=0x7f2e400f1c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8579db9b709c61e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e2f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/1093256945 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40049040 con 0x55e325eab240 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e591a2740 1 -- 172.21.15.31:0/1093256945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e325e17540 con 0x55e325eab240 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/1093256945 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f2e400ed130 con 0x55e325eab240 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/1093256945 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2e400bbd30 con 0x55e325eab240 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e591a2740 10 client.16608 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e591a2740 1 -- 172.21.15.31:0/1093256945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e325f2c890 con 0x55e325eab240 2024-04-25T02:00:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/1093256945 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e400b83a0 con 0x55e325eab240 2024-04-25T02:00:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/1093256945 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2e400efbc0 con 0x55e325eab240 2024-04-25T02:00:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e591a2740 10 client.16608 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e591a2740 10 client.16608 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e591a2740 20 client.16608 populate_metadata read hostname 'smithi031' 2024-04-25T02:00:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e591a2740 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e325f2cfc0 0x55e325f2f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e591a2740 4 client.16608 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:00:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.361+0000 7f2e591a2740 1 -- 172.21.15.31:0/1093256945 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e325e17540 con 0x55e325f2cfc0 2024-04-25T02:00:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.362+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e325f2cfc0 0x55e325f2f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.362+0000 7f2e4f7fe640 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e325f2cfc0 0x55e325f2f3e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc7283161231367f server_cookie=be5e42c2436ba42f in_seq=0 out_seq=0 2024-04-25T02:00:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.362+0000 7f2e2f7fe640 10 client.16608 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.383+0000 7f2e2f7fe640 1 -- 172.21.15.31:0/1093256945 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e325e17540 con 0x55e325f2cfc0 2024-04-25T02:00:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.383+0000 7f2e2f7fe640 10 client.16608 handle_command_reply: tid=0 2024-04-25T02:00:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.383+0000 7f2e591a2740 1 client.16608 shutdown 2024-04-25T02:00:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e591a2740 1 -- 172.21.15.31:0/1093256945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 msgr2=0x55e325ebe5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e591a2740 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 0x55e325ebe5e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2e400ecf10 tx=0x7f2e400f1c20 comp rx=0 tx=0).stop 2024-04-25T02:00:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e591a2740 1 -- 172.21.15.31:0/1093256945 shutdown_connections 2024-04-25T02:00:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e591a2740 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e325f2cfc0 0x55e325f2f3e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e591a2740 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e325eb6230 0x55e325eaff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e591a2740 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e325eabb60 0x55e325eafa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e591a2740 1 --2- 172.21.15.31:0/1093256945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e325eab240 0x55e325ebe5e0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e591a2740 1 -- 172.21.15.31:0/1093256945 >> 172.21.15.31:0/1093256945 conn(0x55e325ea7560 msgr2=0x55e325ea5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e2f7fe640 0 client.16608 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e591a2740 1 -- 172.21.15.31:0/1093256945 shutdown_connections 2024-04-25T02:00:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e591a2740 1 -- 172.21.15.31:0/1093256945 wait complete. 2024-04-25T02:00:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.384+0000 7f2e591a2740 20 client.16608 trim_cache size 0 max 16384 2024-04-25T02:00:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:00:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:00:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:00:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:00:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 -- 172.21.15.31:0/2511356335 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2e380a2da0 msgr2=0x7f2e380a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2e380a2da0 0x7f2e380a5260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009fd0 tx=0x7f2e48004ab0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 -- 172.21.15.31:0/2511356335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e50102850 msgr2=0x7f2e5019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e50102850 0x7f2e5019a1c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2e40005940 tx=0x7f2e400040c0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 -- 172.21.15.31:0/2511356335 shutdown_connections 2024-04-25T02:00:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2e380a2da0 0x7f2e380a5260 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e5010bd70 0x7f2e5019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e50103170 0x7f2e5019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 --2- 172.21.15.31:0/2511356335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e50102850 0x7f2e5019a1c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 -- 172.21.15.31:0/2511356335 >> 172.21.15.31:0/2511356335 conn(0x7f2e500777c0 msgr2=0x7f2e50107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 -- 172.21.15.31:0/2511356335 shutdown_connections 2024-04-25T02:00:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:08.385+0000 7f2e563b7640 1 -- 172.21.15.31:0/2511356335 wait complete. 2024-04-25T02:00:08.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:08 smithi040 ceph-mon[32418]: pgmap v173: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-04-25T02:00:08.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:08 smithi134 ceph-mon[33098]: pgmap v173: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-04-25T02:00:08.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:08 smithi031 ceph-mon[29204]: pgmap v173: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-04-25T02:00:09.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 122837 2.47 MB/sec execute 136 sec latency 23.022 ms 2024-04-25T02:00:10.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 123253 2.47 MB/sec execute 137 sec latency 21.650 ms 2024-04-25T02:00:10.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:10 smithi040 ceph-mon[32418]: pgmap v174: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-04-25T02:00:10.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:10 smithi134 ceph-mon[33098]: pgmap v174: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-04-25T02:00:10.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:10 smithi031 ceph-mon[29204]: pgmap v174: 97 pgs: 97 active+clean; 75 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-04-25T02:00:11.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 123950 2.49 MB/sec execute 138 sec latency 18.513 ms 2024-04-25T02:00:12.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 124406 2.51 MB/sec execute 139 sec latency 18.991 ms 2024-04-25T02:00:12.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:12 smithi040 ceph-mon[32418]: pgmap v175: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-04-25T02:00:12.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:12 smithi134 ceph-mon[33098]: pgmap v175: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-04-25T02:00:12.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:12 smithi031 ceph-mon[29204]: pgmap v175: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-04-25T02:00:13.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 124648 2.49 MB/sec execute 140 sec latency 12.590 ms 2024-04-25T02:00:14.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 125052 2.48 MB/sec execute 141 sec latency 23.312 ms 2024-04-25T02:00:14.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:14 smithi040 ceph-mon[32418]: pgmap v176: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-04-25T02:00:14.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:14 smithi134 ceph-mon[33098]: pgmap v176: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-04-25T02:00:14.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:14 smithi031 ceph-mon[29204]: pgmap v176: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-04-25T02:00:15.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 125574 2.49 MB/sec execute 142 sec latency 19.156 ms 2024-04-25T02:00:16.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 126041 2.48 MB/sec execute 143 sec latency 24.145 ms 2024-04-25T02:00:16.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:16 smithi040 ceph-mon[32418]: pgmap v177: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-04-25T02:00:16.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:16 smithi134 ceph-mon[33098]: pgmap v177: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-04-25T02:00:16.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:16 smithi031 ceph-mon[29204]: pgmap v177: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-04-25T02:00:17.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 126579 2.47 MB/sec execute 144 sec latency 23.019 ms 2024-04-25T02:00:18.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 127176 2.49 MB/sec execute 145 sec latency 18.759 ms 2024-04-25T02:00:18.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:18 smithi040 ceph-mon[32418]: pgmap v178: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T02:00:18.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:18 smithi134 ceph-mon[33098]: pgmap v178: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T02:00:18.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:18 smithi031 ceph-mon[29204]: pgmap v178: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T02:00:19.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 127704 2.51 MB/sec execute 146 sec latency 11.921 ms 2024-04-25T02:00:20.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 128055 2.50 MB/sec execute 147 sec latency 13.498 ms 2024-04-25T02:00:20.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:20 smithi040 ceph-mon[32418]: pgmap v179: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T02:00:20.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:20 smithi134 ceph-mon[33098]: pgmap v179: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T02:00:20.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:20 smithi031 ceph-mon[29204]: pgmap v179: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T02:00:21.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 128320 2.49 MB/sec execute 148 sec latency 19.095 ms 2024-04-25T02:00:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.477+0000 7fcf61190640 1 -- 172.21.15.31:0/3987722885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5c103330 msgr2=0x7fcf5c10b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.477+0000 7fcf61190640 1 --2- 172.21.15.31:0/3987722885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5c103330 0x7fcf5c10b810 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009900 tx=0x7fcf5002a480 comp rx=0 tx=0).stop 2024-04-25T02:00:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.477+0000 7fcf61190640 1 -- 172.21.15.31:0/3987722885 shutdown_connections 2024-04-25T02:00:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.477+0000 7fcf61190640 1 --2- 172.21.15.31:0/3987722885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf5c10bd50 0x7fcf5c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.477+0000 7fcf61190640 1 --2- 172.21.15.31:0/3987722885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5c103330 0x7fcf5c10b810 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.477+0000 7fcf61190640 1 --2- 172.21.15.31:0/3987722885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf5c102a10 0x7fcf5c102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.477+0000 7fcf61190640 1 -- 172.21.15.31:0/3987722885 >> 172.21.15.31:0/3987722885 conn(0x7fcf5c0777c0 msgr2=0x7fcf5c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.477+0000 7fcf61190640 1 -- 172.21.15.31:0/3987722885 shutdown_connections 2024-04-25T02:00:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.477+0000 7fcf61190640 1 -- 172.21.15.31:0/3987722885 wait complete. 2024-04-25T02:00:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.478+0000 7fcf61190640 1 Processor -- start 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.478+0000 7fcf61190640 1 -- start start 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf61190640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf5c102a10 0x7fcf5c19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf61190640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf5c103330 0x7fcf5c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf61190640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5c10bd50 0x7fcf5c19e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf61190640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c1119b0 con 0x7fcf5c10bd50 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf61190640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c111830 con 0x7fcf5c103330 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf61190640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c111b30 con 0x7fcf5c102a10 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5b59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5c10bd50 0x7fcf5c19e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf5c103330 0x7fcf5c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf5c103330 0x7fcf5c19a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39692/0 (socket says 172.21.15.31:39692) 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5a59c640 1 -- 172.21.15.31:0/71455138 learned_addr learned my addr 172.21.15.31:0/71455138 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5b59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5c10bd50 0x7fcf5c19e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46888/0 (socket says 172.21.15.31:46888) 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5ad9d640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf5c102a10 0x7fcf5c19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5b59e640 1 -- 172.21.15.31:0/71455138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf5c102a10 msgr2=0x7fcf5c19a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5b59e640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf5c102a10 0x7fcf5c19a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5b59e640 1 -- 172.21.15.31:0/71455138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf5c103330 msgr2=0x7fcf5c19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5b59e640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf5c103330 0x7fcf5c19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5b59e640 1 -- 172.21.15.31:0/71455138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c19f0c0 con 0x7fcf5c10bd50 2024-04-25T02:00:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5a59c640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf5c103330 0x7fcf5c19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf5b59e640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5c10bd50 0x7fcf5c19e990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400ec50 tx=0x7fcf5400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf43fff640 1 -- 172.21.15.31:0/71455138 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54003b70 con 0x7fcf5c10bd50 2024-04-25T02:00:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf61190640 1 -- 172.21.15.31:0/71455138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5c19f3b0 con 0x7fcf5c10bd50 2024-04-25T02:00:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf43fff640 1 -- 172.21.15.31:0/71455138 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf54003d10 con 0x7fcf5c10bd50 2024-04-25T02:00:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.479+0000 7fcf43fff640 1 -- 172.21.15.31:0/71455138 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54010760 con 0x7fcf5c10bd50 2024-04-25T02:00:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.480+0000 7fcf61190640 1 -- 172.21.15.31:0/71455138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf5c1a6c40 con 0x7fcf5c10bd50 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.482+0000 7fcf43fff640 1 -- 172.21.15.31:0/71455138 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcf54004020 con 0x7fcf5c10bd50 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.482+0000 7fcf63f80740 1 Processor -- start 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.482+0000 7fcf63f80740 1 -- start start 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.482+0000 7fcf43fff640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcf440a2da0 0x7fcf440a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.482+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770ca3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.482+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770ca3b60 0x55d770cadcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.482+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770cae230 0x55d770cb0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.482+0000 7fcf63f80740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d770c07cb0 con 0x55d770ca3240 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.482+0000 7fcf63f80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d770bb05b0 con 0x55d770cae230 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.482+0000 7fcf63f80740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d770c03c70 con 0x55d770ca3b60 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf43fff640 1 -- 172.21.15.31:0/71455138 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fcf540cf970 con 0x7fcf5c10bd50 2024-04-25T02:00:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770ca3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5a59c640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcf440a2da0 0x7fcf440a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770ca3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46904/0 (socket says 172.21.15.31:46904) 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5ad9d640 1 -- 172.21.15.31:0/2625884490 learned_addr learned my addr 172.21.15.31:0/2625884490 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5b59e640 1 --2- 172.21.15.31:0/2625884490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770cae230 0x55d770cb0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5a59c640 1 --2- 172.21.15.31:0/2625884490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770ca3b60 0x55d770cadcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5ad9d640 1 -- 172.21.15.31:0/2625884490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770ca3b60 msgr2=0x55d770cadcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5ad9d640 1 --2- 172.21.15.31:0/2625884490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770ca3b60 0x55d770cadcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5ad9d640 1 -- 172.21.15.31:0/2625884490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770cae230 msgr2=0x55d770cb0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5ad9d640 1 --2- 172.21.15.31:0/2625884490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770cae230 0x55d770cb0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5ad9d640 1 -- 172.21.15.31:0/2625884490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d77091c210 con 0x55d770ca3240 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5a59c640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcf440a2da0 0x7fcf440a5260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcf50031440 tx=0x7fcf50005bb0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf5ad9d640 1 --2- 172.21.15.31:0/2625884490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770ca3620 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00bcd0 tx=0x7fcf4c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa582116566a8e47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf417fa640 1 -- 172.21.15.31:0/2625884490 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c005220 con 0x55d770ca3240 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.483+0000 7fcf417fa640 1 -- 172.21.15.31:0/2625884490 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf4c004c50 con 0x55d770ca3240 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.484+0000 7fcf417fa640 1 -- 172.21.15.31:0/2625884490 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c011730 con 0x55d770ca3240 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.484+0000 7fcf63f80740 1 -- 172.21.15.31:0/2625884490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 msgr2=0x55d770ca3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.484+0000 7fcf63f80740 1 --2- 172.21.15.31:0/2625884490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770ca3620 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00bcd0 tx=0x7fcf4c0107a0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.484+0000 7fcf63f80740 1 -- 172.21.15.31:0/2625884490 shutdown_connections 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.484+0000 7fcf63f80740 1 --2- 172.21.15.31:0/2625884490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770cae230 0x55d770cb0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.484+0000 7fcf63f80740 1 --2- 172.21.15.31:0/2625884490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770ca3b60 0x55d770cadcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.484+0000 7fcf63f80740 1 --2- 172.21.15.31:0/2625884490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770ca3620 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.484+0000 7fcf63f80740 1 -- 172.21.15.31:0/2625884490 >> 172.21.15.31:0/2625884490 conn(0x55d770c9f560 msgr2=0x55d770c9f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.484+0000 7fcf63f80740 1 -- 172.21.15.31:0/2625884490 shutdown_connections 2024-04-25T02:00:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf63f80740 1 -- 172.21.15.31:0/2625884490 wait complete. 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf63f80740 1 -- start start 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf63f80740 1 Processor -- start 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cab160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770ca3b60 0x55d770cab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770cae230 0x55d770cabbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf63f80740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d770cb1020 con 0x55d770ca3240 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf63f80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d770bc3b50 con 0x55d770ca3b60 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf63f80740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d770cb11a0 con 0x55d770cae230 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf63f80740 10 client.0 _ll_register_callbacks cb 0x55d770bfeca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf5ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf5ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cab160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46918/0 (socket says 172.21.15.31:46918) 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf5ad9d640 1 -- 172.21.15.31:0/1721706288 learned_addr learned my addr 172.21.15.31:0/1721706288 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf5a59c640 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770ca3b60 0x55d770cab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.485+0000 7fcf5b59e640 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770cae230 0x55d770cabbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf5ad9d640 1 -- 172.21.15.31:0/1721706288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770cae230 msgr2=0x55d770cabbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf5ad9d640 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770cae230 0x55d770cabbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf5ad9d640 1 -- 172.21.15.31:0/1721706288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770ca3b60 msgr2=0x55d770cab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf5ad9d640 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770ca3b60 0x55d770cab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf5ad9d640 1 -- 172.21.15.31:0/1721706288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7709e8620 con 0x55d770ca3240 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf5ad9d640 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cab160 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00bca0 tx=0x7fcf4c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a880c6fc9b0f7485 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf3eff5640 1 -- 172.21.15.31:0/1721706288 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c00b8c0 con 0x55d770ca3240 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf3eff5640 1 -- 172.21.15.31:0/1721706288 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fcf4c005010 con 0x55d770ca3240 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf63f80740 1 -- 172.21.15.31:0/1721706288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf4c004c50 con 0x55d770ca3240 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf3eff5640 1 -- 172.21.15.31:0/1721706288 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcf4c005210 con 0x55d770ca3240 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf63f80740 10 client.16620 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf63f80740 1 -- 172.21.15.31:0/1721706288 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d770cbd1c0 con 0x55d770ca3240 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.486+0000 7fcf3eff5640 1 -- 172.21.15.31:0/1721706288 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf4c004c50 con 0x55d770ca3240 2024-04-25T02:00:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.487+0000 7fcf3eff5640 1 -- 172.21.15.31:0/1721706288 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fcf4c01b430 con 0x55d770ca3240 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.487+0000 7fcf63f80740 10 client.16620 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.487+0000 7fcf63f80740 10 client.16620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.487+0000 7fcf63f80740 20 client.16620 populate_metadata read hostname 'smithi031' 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.487+0000 7fcf63f80740 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d770cbe620 0x55d770cc0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.487+0000 7fcf63f80740 4 client.16620 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.487+0000 7fcf63f80740 1 -- 172.21.15.31:0/1721706288 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d770c151f0 con 0x55d770cbe620 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.487+0000 7fcf5a59c640 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d770cbe620 0x55d770cc0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.487+0000 7fcf5a59c640 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d770cbe620 0x55d770cc0a10 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c3609b2ae978076 server_cookie=ae815628207c1fc0 in_seq=0 out_seq=0 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.487+0000 7fcf3eff5640 10 client.16620 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf3eff5640 1 -- 172.21.15.31:0/1721706288 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d770c151f0 con 0x55d770cbe620 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf3eff5640 10 client.16620 handle_command_reply: tid=0 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 1 client.16620 shutdown 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 1 -- 172.21.15.31:0/1721706288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 msgr2=0x55d770cab160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cab160 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00bca0 tx=0x7fcf4c0048f0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 1 -- 172.21.15.31:0/1721706288 shutdown_connections 2024-04-25T02:00:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d770cbe620 0x55d770cc0a10 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770cae230 0x55d770cabbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770ca3b60 0x55d770cab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 1 --2- 172.21.15.31:0/1721706288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cab160 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf3eff5640 0 client.16620 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 1 -- 172.21.15.31:0/1721706288 >> 172.21.15.31:0/1721706288 conn(0x55d770c9f560 msgr2=0x55d770ca5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 1 -- 172.21.15.31:0/1721706288 shutdown_connections 2024-04-25T02:00:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 1 -- 172.21.15.31:0/1721706288 wait complete. 2024-04-25T02:00:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.488+0000 7fcf63f80740 20 client.16620 trim_cache size 0 max 16384 2024-04-25T02:00:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.504+0000 7fcf63f80740 1 Processor -- start 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.504+0000 7fcf63f80740 1 -- start start 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.504+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770ca3240 0x55d770cad890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.504+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3b60 0x55d770cb9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.504+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770cae230 0x55d770caa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.504+0000 7fcf63f80740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c1af520 con 0x55d770ca3b60 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.504+0000 7fcf63f80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c1b1060 con 0x55d770ca3240 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.504+0000 7fcf63f80740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c1afbf0 con 0x55d770cae230 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3b60 0x55d770cb9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770ca3240 0x55d770cad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3b60 0x55d770cb9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46932/0 (socket says 172.21.15.31:46932) 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5a59c640 1 -- 172.21.15.31:0/1961514678 learned_addr learned my addr 172.21.15.31:0/1961514678 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5b59e640 1 --2- 172.21.15.31:0/1961514678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770cae230 0x55d770caa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5a59c640 1 -- 172.21.15.31:0/1961514678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770cae230 msgr2=0x55d770caa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5a59c640 1 --2- 172.21.15.31:0/1961514678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770cae230 0x55d770caa2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5a59c640 1 -- 172.21.15.31:0/1961514678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770ca3240 msgr2=0x55d770cad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5a59c640 1 --2- 172.21.15.31:0/1961514678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770ca3240 0x55d770cad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5a59c640 1 -- 172.21.15.31:0/1961514678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d770ca96b0 con 0x55d770ca3b60 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5ad9d640 1 --2- 172.21.15.31:0/1961514678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770ca3240 0x55d770cad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf5a59c640 1 --2- 172.21.15.31:0/1961514678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3b60 0x55d770cb9ee0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcf50031a70 tx=0x7fcf5003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f73e67cc673ef3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf40ff9640 1 -- 172.21.15.31:0/1961514678 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50045040 con 0x55d770ca3b60 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf40ff9640 1 -- 172.21.15.31:0/1961514678 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf500036b0 con 0x55d770ca3b60 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.505+0000 7fcf40ff9640 1 -- 172.21.15.31:0/1961514678 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50035040 con 0x55d770ca3b60 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.506+0000 7fcf63f80740 1 -- 172.21.15.31:0/1961514678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3b60 msgr2=0x55d770cb9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.506+0000 7fcf63f80740 1 --2- 172.21.15.31:0/1961514678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3b60 0x55d770cb9ee0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcf50031a70 tx=0x7fcf5003f000 comp rx=0 tx=0).stop 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.506+0000 7fcf63f80740 1 -- 172.21.15.31:0/1961514678 shutdown_connections 2024-04-25T02:00:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.506+0000 7fcf63f80740 1 --2- 172.21.15.31:0/1961514678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770cae230 0x55d770caa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.506+0000 7fcf63f80740 1 --2- 172.21.15.31:0/1961514678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3b60 0x55d770cb9ee0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.506+0000 7fcf63f80740 1 --2- 172.21.15.31:0/1961514678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770ca3240 0x55d770cad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.506+0000 7fcf63f80740 1 -- 172.21.15.31:0/1961514678 >> 172.21.15.31:0/1961514678 conn(0x55d770c9f560 msgr2=0x55d770c9dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.506+0000 7fcf63f80740 1 -- 172.21.15.31:0/1961514678 shutdown_connections 2024-04-25T02:00:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.506+0000 7fcf63f80740 1 -- 172.21.15.31:0/1961514678 wait complete. 2024-04-25T02:00:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.506+0000 7fcf63f80740 1 -- start start 2024-04-25T02:00:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf63f80740 1 Processor -- start 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cb67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770ca3b60 0x55d770cb1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf63f80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770cae230 0x55d770cb2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf63f80740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c1b0990 con 0x55d770ca3240 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf63f80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c1b02c0 con 0x55d770cae230 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf63f80740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c1ad500 con 0x55d770ca3b60 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf63f80740 10 client.0 _ll_register_callbacks cb 0x7fcf54004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cb67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cb67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46934/0 (socket says 172.21.15.31:46934) 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5ad9d640 1 -- 172.21.15.31:0/4173255731 learned_addr learned my addr 172.21.15.31:0/4173255731 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5b59e640 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770cae230 0x55d770cb2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5a59c640 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770ca3b60 0x55d770cb1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5b59e640 1 -- 172.21.15.31:0/4173255731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770ca3b60 msgr2=0x55d770cb1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5b59e640 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770ca3b60 0x55d770cb1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5b59e640 1 -- 172.21.15.31:0/4173255731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 msgr2=0x55d770cb67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5b59e640 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cb67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5b59e640 1 -- 172.21.15.31:0/4173255731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d770cb2890 con 0x55d770cae230 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5ad9d640 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cb67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.507+0000 7fcf5b59e640 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770cae230 0x55d770cb2090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cd110 tx=0x7fcf54009590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=451267b4e264a51f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf3eff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf3eff5640 1 -- 172.21.15.31:0/4173255731 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5409c5e0 con 0x55d770cae230 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf63f80740 1 -- 172.21.15.31:0/4173255731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf500036b0 con 0x55d770cae230 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf3eff5640 1 -- 172.21.15.31:0/4173255731 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7fcf54004b40 con 0x55d770cae230 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf3eff5640 1 -- 172.21.15.31:0/4173255731 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcf5409bee0 con 0x55d770cae230 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf63f80740 10 client.26228 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf63f80740 1 -- 172.21.15.31:0/4173255731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d770d24b40 con 0x55d770cae230 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf3eff5640 1 -- 172.21.15.31:0/4173255731 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf500036b0 con 0x55d770cae230 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf3eff5640 1 -- 172.21.15.31:0/4173255731 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fcf540bab20 con 0x55d770cae230 2024-04-25T02:00:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf63f80740 10 client.26228 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf63f80740 10 client.26228 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf63f80740 20 client.26228 populate_metadata read hostname 'smithi031' 2024-04-25T02:00:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf63f80740 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d770d251c0 0x55d770d275e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf63f80740 4 client.26228 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:00:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.508+0000 7fcf63f80740 1 -- 172.21.15.31:0/4173255731 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d770c0f540 con 0x55d770d251c0 2024-04-25T02:00:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.509+0000 7fcf5ad9d640 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d770d251c0 0x55d770d275e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.509+0000 7fcf5ad9d640 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d770d251c0 0x55d770d275e0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=412d5505899dad8b server_cookie=ddabb3ca70a60be7 in_seq=0 out_seq=0 2024-04-25T02:00:21.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.509+0000 7fcf3eff5640 10 client.26228 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf3eff5640 1 -- 172.21.15.31:0/4173255731 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d770c0f540 con 0x55d770d251c0 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf3eff5640 10 client.26228 handle_command_reply: tid=0 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 1 client.26228 shutdown 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 1 -- 172.21.15.31:0/4173255731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770cae230 msgr2=0x55d770cb2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770cae230 0x55d770cb2090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cd110 tx=0x7fcf54009590 comp rx=0 tx=0).stop 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 1 -- 172.21.15.31:0/4173255731 shutdown_connections 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d770d251c0 0x55d770d275e0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d770cae230 0x55d770cb2090 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d770ca3b60 0x55d770cb1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf3eff5640 0 client.26228 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 1 --2- 172.21.15.31:0/4173255731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d770ca3240 0x55d770cb67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 1 -- 172.21.15.31:0/4173255731 >> 172.21.15.31:0/4173255731 conn(0x55d770c9f560 msgr2=0x55d770c9dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 1 -- 172.21.15.31:0/4173255731 shutdown_connections 2024-04-25T02:00:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 1 -- 172.21.15.31:0/4173255731 wait complete. 2024-04-25T02:00:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.531+0000 7fcf63f80740 20 client.26228 trim_cache size 0 max 16384 2024-04-25T02:00:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:00:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:00:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:00:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:00:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 -- 172.21.15.31:0/71455138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcf440a2da0 msgr2=0x7fcf440a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcf440a2da0 0x7fcf440a5260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcf50031440 tx=0x7fcf50005bb0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 -- 172.21.15.31:0/71455138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5c10bd50 msgr2=0x7fcf5c19e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5c10bd50 0x7fcf5c19e990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400ec50 tx=0x7fcf5400c6a0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 -- 172.21.15.31:0/71455138 shutdown_connections 2024-04-25T02:00:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcf440a2da0 0x7fcf440a5260 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5c10bd50 0x7fcf5c19e990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf5c103330 0x7fcf5c19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 --2- 172.21.15.31:0/71455138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcf5c102a10 0x7fcf5c19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 -- 172.21.15.31:0/71455138 >> 172.21.15.31:0/71455138 conn(0x7fcf5c0777c0 msgr2=0x7fcf5c107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 -- 172.21.15.31:0/71455138 shutdown_connections 2024-04-25T02:00:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:21.532+0000 7fcf61190640 1 -- 172.21.15.31:0/71455138 wait complete. 2024-04-25T02:00:22.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 128789 2.48 MB/sec execute 149 sec latency 21.092 ms 2024-04-25T02:00:22.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:22 smithi040 ceph-mon[32418]: pgmap v180: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T02:00:22.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:22 smithi134 ceph-mon[33098]: pgmap v180: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T02:00:22.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:22 smithi031 ceph-mon[29204]: pgmap v180: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T02:00:23.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 129288 2.49 MB/sec execute 150 sec latency 24.606 ms 2024-04-25T02:00:24.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 129868 2.48 MB/sec execute 151 sec latency 20.946 ms 2024-04-25T02:00:24.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:24 smithi040 ceph-mon[32418]: pgmap v181: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-04-25T02:00:24.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:24 smithi134 ceph-mon[33098]: pgmap v181: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-04-25T02:00:24.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:24 smithi031 ceph-mon[29204]: pgmap v181: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-04-25T02:00:25.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 130318 2.47 MB/sec execute 152 sec latency 21.029 ms 2024-04-25T02:00:26.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 131029 2.50 MB/sec execute 153 sec latency 18.428 ms 2024-04-25T02:00:26.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:26 smithi040 ceph-mon[32418]: pgmap v182: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-04-25T02:00:26.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:26 smithi134 ceph-mon[33098]: pgmap v182: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-04-25T02:00:26.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:26 smithi031 ceph-mon[29204]: pgmap v182: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-04-25T02:00:27.309 INFO:tasks.workunit.client.0.smithi031.stdout: 1 131446 2.51 MB/sec execute 154 sec latency 17.968 ms 2024-04-25T02:00:28.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 131698 2.50 MB/sec execute 155 sec latency 15.966 ms 2024-04-25T02:00:28.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:28 smithi040 ceph-mon[32418]: pgmap v183: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:00:28.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:28 smithi134 ceph-mon[33098]: pgmap v183: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:00:28.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:28 smithi031 ceph-mon[29204]: pgmap v183: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:00:29.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 132034 2.48 MB/sec execute 156 sec latency 18.249 ms 2024-04-25T02:00:30.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 132524 2.48 MB/sec execute 157 sec latency 18.500 ms 2024-04-25T02:00:30.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:30 smithi040 ceph-mon[32418]: pgmap v184: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:00:30.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:30 smithi134 ceph-mon[33098]: pgmap v184: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:00:30.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:30 smithi031 ceph-mon[29204]: pgmap v184: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:00:31.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 132973 2.48 MB/sec execute 158 sec latency 27.299 ms 2024-04-25T02:00:32.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 133586 2.48 MB/sec execute 159 sec latency 22.213 ms 2024-04-25T02:00:32.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:32 smithi040 ceph-mon[32418]: pgmap v185: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:00:32.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:32 smithi134 ceph-mon[33098]: pgmap v185: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:00:32.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:32 smithi031 ceph-mon[29204]: pgmap v185: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:00:33.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 134055 2.49 MB/sec execute 160 sec latency 20.265 ms 2024-04-25T02:00:34.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 134604 2.49 MB/sec execute 161 sec latency 19.084 ms 2024-04-25T02:00:34.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:34 smithi040 ceph-mon[32418]: pgmap v186: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T02:00:34.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:34 smithi134 ceph-mon[33098]: pgmap v186: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T02:00:34.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:34 smithi031 ceph-mon[29204]: pgmap v186: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-25T02:00:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.623+0000 7f7c50ba2640 1 -- 172.21.15.31:0/24970769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c106630 msgr2=0x7f7c4c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.623+0000 7f7c50ba2640 1 --2- 172.21.15.31:0/24970769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c106630 0x7f7c4c106a10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009930 tx=0x7f7c3c02a420 comp rx=0 tx=0).stop 2024-04-25T02:00:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.623+0000 7f7c50ba2640 1 -- 172.21.15.31:0/24970769 shutdown_connections 2024-04-25T02:00:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.623+0000 7f7c50ba2640 1 --2- 172.21.15.31:0/24970769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c4c101030 0x7f7c4c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.623+0000 7f7c50ba2640 1 --2- 172.21.15.31:0/24970769 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c4c100670 0x7f7c4c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.623+0000 7f7c50ba2640 1 --2- 172.21.15.31:0/24970769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c106630 0x7f7c4c106a10 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.623+0000 7f7c50ba2640 1 -- 172.21.15.31:0/24970769 >> 172.21.15.31:0/24970769 conn(0x7f7c4c0777c0 msgr2=0x7f7c4c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.623+0000 7f7c50ba2640 1 -- 172.21.15.31:0/24970769 shutdown_connections 2024-04-25T02:00:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.623+0000 7f7c50ba2640 1 -- 172.21.15.31:0/24970769 wait complete. 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c50ba2640 1 Processor -- start 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c50ba2640 1 -- start start 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c50ba2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c4c100670 0x7f7c4c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c50ba2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c4c101030 0x7f7c4c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c50ba2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c106630 0x7f7c4c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c50ba2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1108f0 con 0x7f7c4c106630 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c50ba2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c110770 con 0x7f7c4c100670 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c50ba2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c110a70 con 0x7f7c4c101030 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c4c100670 0x7f7c4c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c4ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c106630 0x7f7c4c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c4c100670 0x7f7c4c197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46168/0 (socket says 172.21.15.31:46168) 2024-04-25T02:00:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.624+0000 7f7c4a59c640 1 -- 172.21.15.31:0/2432741545 learned_addr learned my addr 172.21.15.31:0/2432741545 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c4c101030 0x7f7c4c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c4ad9d640 1 -- 172.21.15.31:0/2432741545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c4c101030 msgr2=0x7f7c4c198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c4ad9d640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c4c101030 0x7f7c4c198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c4ad9d640 1 -- 172.21.15.31:0/2432741545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c4c100670 msgr2=0x7f7c4c197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c4ad9d640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c4c100670 0x7f7c4c197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c4ad9d640 1 -- 172.21.15.31:0/2432741545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c19ceb0 con 0x7f7c4c106630 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c4c100670 0x7f7c4c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c4ad9d640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c106630 0x7f7c4c19c780 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009fd0 tx=0x7f7c4400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c3b7fe640 1 -- 172.21.15.31:0/2432741545 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4400ce20 con 0x7f7c4c106630 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c3b7fe640 1 -- 172.21.15.31:0/2432741545 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c44005020 con 0x7f7c4c106630 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c50ba2640 1 -- 172.21.15.31:0/2432741545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4c19d1a0 con 0x7f7c4c106630 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c3b7fe640 1 -- 172.21.15.31:0/2432741545 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44010690 con 0x7f7c4c106630 2024-04-25T02:00:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.625+0000 7f7c50ba2640 1 -- 172.21.15.31:0/2432741545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4c1a4a30 con 0x7f7c4c106630 2024-04-25T02:00:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.628+0000 7f7c3b7fe640 1 -- 172.21.15.31:0/2432741545 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7c440051d0 con 0x7f7c4c106630 2024-04-25T02:00:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.628+0000 7f7c53990740 1 Processor -- start 2024-04-25T02:00:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c53990740 1 -- start start 2024-04-25T02:00:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c53990740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 0x564faeeef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c53990740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 0x564faeef9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c53990740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeefc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c53990740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564faee53cb0 con 0x564faeeef240 2024-04-25T02:00:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c53990740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564faedfc5b0 con 0x564faeeefb60 2024-04-25T02:00:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c53990740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564faee4fc70 con 0x564faeefa230 2024-04-25T02:00:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c3b7fe640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7c340a2da0 0x7f7c340a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 0x564faeef9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c49d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 0x564faeeef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 0x564faeef9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46180/0 (socket says 172.21.15.31:46180) 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c4a59c640 1 -- 172.21.15.31:0/3738910525 learned_addr learned my addr 172.21.15.31:0/3738910525 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/3738910525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeefc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c4ad9d640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7c340a2da0 0x7f7c340a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c3b7fe640 1 -- 172.21.15.31:0/2432741545 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7c440cfb70 con 0x7f7c4c106630 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c4a59c640 1 -- 172.21.15.31:0/3738910525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 msgr2=0x564faeefc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/3738910525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeefc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c4a59c640 1 -- 172.21.15.31:0/3738910525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 msgr2=0x564faeeef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/3738910525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 0x564faeeef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c4a59c640 1 -- 172.21.15.31:0/3738910525 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564faeb68210 con 0x564faeeefb60 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.629+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/3738910525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeefc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.630+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/3738910525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 0x564faeef9cf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009230 tx=0x7f7c3c032eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4844fccb314e4468 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.630+0000 7f7c4ad9d640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7c340a2da0 0x7f7c340a5260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7c4409bd80 tx=0x7f7c44091510 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.630+0000 7f7c38ff9640 1 -- 172.21.15.31:0/3738910525 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c035040 con 0x564faeeefb60 2024-04-25T02:00:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.630+0000 7f7c38ff9640 1 -- 172.21.15.31:0/3738910525 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c3c0317a0 con 0x564faeeefb60 2024-04-25T02:00:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.630+0000 7f7c38ff9640 1 -- 172.21.15.31:0/3738910525 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c0049a0 con 0x564faeeefb60 2024-04-25T02:00:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.631+0000 7f7c53990740 1 -- 172.21.15.31:0/3738910525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 msgr2=0x564faeef9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.631+0000 7f7c53990740 1 --2- 172.21.15.31:0/3738910525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 0x564faeef9cf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009230 tx=0x7f7c3c032eb0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.631+0000 7f7c53990740 1 -- 172.21.15.31:0/3738910525 shutdown_connections 2024-04-25T02:00:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.631+0000 7f7c53990740 1 --2- 172.21.15.31:0/3738910525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeefc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.631+0000 7f7c53990740 1 --2- 172.21.15.31:0/3738910525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 0x564faeef9cf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.631+0000 7f7c53990740 1 --2- 172.21.15.31:0/3738910525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 0x564faeeef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.631+0000 7f7c53990740 1 -- 172.21.15.31:0/3738910525 >> 172.21.15.31:0/3738910525 conn(0x564faeeeb560 msgr2=0x564faeeeb940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.631+0000 7f7c53990740 1 -- 172.21.15.31:0/3738910525 shutdown_connections 2024-04-25T02:00:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.631+0000 7f7c53990740 1 -- 172.21.15.31:0/3738910525 wait complete. 2024-04-25T02:00:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.631+0000 7f7c53990740 1 -- start start 2024-04-25T02:00:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c53990740 1 Processor -- start 2024-04-25T02:00:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c53990740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeef240 0x564faeef7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c53990740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c53990740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeef7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c53990740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564faeefcfc0 con 0x564faeeefb60 2024-04-25T02:00:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c53990740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564faee0fb50 con 0x564faeeef240 2024-04-25T02:00:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c53990740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564faeefd140 con 0x564faeefa230 2024-04-25T02:00:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46678/0 (socket says 172.21.15.31:46678) 2024-04-25T02:00:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c4a59c640 1 -- 172.21.15.31:0/1313805975 learned_addr learned my addr 172.21.15.31:0/1313805975 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c53990740 10 client.0 _ll_register_callbacks cb 0x564faee4aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeef240 0x564faeef7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c4a59c640 1 -- 172.21.15.31:0/1313805975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 msgr2=0x564faeef7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeef7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeef7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c4a59c640 1 -- 172.21.15.31:0/1313805975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeef240 msgr2=0x564faeef7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeef240 0x564faeef7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c4a59c640 1 -- 172.21.15.31:0/1313805975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564faec34620 con 0x564faeeefb60 2024-04-25T02:00:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.632+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeef7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.633+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef76b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c031440 tx=0x7f7c3c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcd08368878a379 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.633+0000 7f7c31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:00:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.633+0000 7f7c31ffb640 1 -- 172.21.15.31:0/1313805975 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c04e070 con 0x564faeeefb60 2024-04-25T02:00:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.633+0000 7f7c31ffb640 1 -- 172.21.15.31:0/1313805975 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7c3c003800 con 0x564faeeefb60 2024-04-25T02:00:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.633+0000 7f7c53990740 1 -- 172.21.15.31:0/1313805975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c3c0317a0 con 0x564faeeefb60 2024-04-25T02:00:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.633+0000 7f7c31ffb640 1 -- 172.21.15.31:0/1313805975 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7c3c05a050 con 0x564faeeefb60 2024-04-25T02:00:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.633+0000 7f7c53990740 10 client.16638 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.633+0000 7f7c53990740 1 -- 172.21.15.31:0/1313805975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564faef0d570 con 0x564faeeefb60 2024-04-25T02:00:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.633+0000 7f7c31ffb640 1 -- 172.21.15.31:0/1313805975 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c3c0317a0 con 0x564faeeefb60 2024-04-25T02:00:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.633+0000 7f7c31ffb640 1 -- 172.21.15.31:0/1313805975 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7c3c0464c0 con 0x564faeeefb60 2024-04-25T02:00:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.634+0000 7f7c53990740 10 client.16638 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.634+0000 7f7c53990740 10 client.16638 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.634+0000 7f7c53990740 20 client.16638 populate_metadata read hostname 'smithi031' 2024-04-25T02:00:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.634+0000 7f7c53990740 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564faef0e8c0 0x564faef10cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.634+0000 7f7c53990740 4 client.16638 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.634+0000 7f7c53990740 1 -- 172.21.15.31:0/1313805975 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564faee611f0 con 0x564faef0e8c0 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.634+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564faef0e8c0 0x564faef10cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.634+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564faef0e8c0 0x564faef10cb0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=368a34b57cecd835 server_cookie=837e2a6b122f565a in_seq=0 out_seq=0 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.634+0000 7f7c31ffb640 10 client.16638 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c31ffb640 1 -- 172.21.15.31:0/1313805975 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564faee611f0 con 0x564faef0e8c0 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c31ffb640 10 client.16638 handle_command_reply: tid=0 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 1 client.16638 shutdown 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 1 -- 172.21.15.31:0/1313805975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 msgr2=0x564faeef76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef76b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c031440 tx=0x7f7c3c0042f0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 1 -- 172.21.15.31:0/1313805975 shutdown_connections 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564faef0e8c0 0x564faef10cb0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeef7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef76b0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 1 --2- 172.21.15.31:0/1313805975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeef240 0x564faeef7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 1 -- 172.21.15.31:0/1313805975 >> 172.21.15.31:0/1313805975 conn(0x564faeeeb560 msgr2=0x564faeef1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c31ffb640 0 client.16638 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 1 -- 172.21.15.31:0/1313805975 shutdown_connections 2024-04-25T02:00:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 1 -- 172.21.15.31:0/1313805975 wait complete. 2024-04-25T02:00:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.635+0000 7f7c53990740 20 client.16638 trim_cache size 0 max 16384 2024-04-25T02:00:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.650+0000 7f7c53990740 1 Processor -- start 2024-04-25T02:00:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.650+0000 7f7c53990740 1 -- start start 2024-04-25T02:00:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c53990740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeeef240 0x564faeef98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c53990740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c53990740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeefa230 0x564faeef62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c53990740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1aafa0 con 0x564faeeefb60 2024-04-25T02:00:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c53990740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1acae0 con 0x564faeefa230 2024-04-25T02:00:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c53990740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1ab670 con 0x564faeeef240 2024-04-25T02:00:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c4a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46690/0 (socket says 172.21.15.31:46690) 2024-04-25T02:00:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c4a59c640 1 -- 172.21.15.31:0/1663910718 learned_addr learned my addr 172.21.15.31:0/1663910718 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/1663910718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeefa230 0x564faeef62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/1663910718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeeef240 0x564faeef98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c4a59c640 1 -- 172.21.15.31:0/1663910718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeeef240 msgr2=0x564faeef98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/1663910718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeeef240 0x564faeef98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c4a59c640 1 -- 172.21.15.31:0/1663910718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeefa230 msgr2=0x564faeef62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/1663910718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeefa230 0x564faeef62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c4a59c640 1 -- 172.21.15.31:0/1663910718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564faeef56c0 con 0x564faeeefb60 2024-04-25T02:00:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/1663910718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeeef240 0x564faeef98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/1663910718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef14e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c046a80 tx=0x7f7c3c0055c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcfdb3493417e2eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c33fff640 1 -- 172.21.15.31:0/1663910718 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c057070 con 0x564faeeefb60 2024-04-25T02:00:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c33fff640 1 -- 172.21.15.31:0/1663910718 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c3c046450 con 0x564faeeefb60 2024-04-25T02:00:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.651+0000 7f7c33fff640 1 -- 172.21.15.31:0/1663910718 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c04e070 con 0x564faeeefb60 2024-04-25T02:00:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.652+0000 7f7c53990740 1 -- 172.21.15.31:0/1663910718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 msgr2=0x564faeef14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.652+0000 7f7c53990740 1 --2- 172.21.15.31:0/1663910718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef14e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c046a80 tx=0x7f7c3c0055c0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.652+0000 7f7c53990740 1 -- 172.21.15.31:0/1663910718 shutdown_connections 2024-04-25T02:00:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.652+0000 7f7c53990740 1 --2- 172.21.15.31:0/1663910718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeefa230 0x564faeef62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.652+0000 7f7c53990740 1 --2- 172.21.15.31:0/1663910718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeefb60 0x564faeef14e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.652+0000 7f7c53990740 1 --2- 172.21.15.31:0/1663910718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeeef240 0x564faeef98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.652+0000 7f7c53990740 1 -- 172.21.15.31:0/1663910718 >> 172.21.15.31:0/1663910718 conn(0x564faeeeb560 msgr2=0x564faeef7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.652+0000 7f7c53990740 1 -- 172.21.15.31:0/1663910718 shutdown_connections 2024-04-25T02:00:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.652+0000 7f7c53990740 1 -- 172.21.15.31:0/1663910718 wait complete. 2024-04-25T02:00:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c53990740 1 -- start start 2024-04-25T02:00:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c53990740 1 Processor -- start 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c53990740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 0x564faef06b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c53990740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 0x564faeef3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c53990740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeef4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c53990740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1ac410 con 0x564faeeef240 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c53990740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1abd40 con 0x564faeeefb60 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c53990740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1a8fa0 con 0x564faeefa230 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c53990740 10 client.0 _ll_register_callbacks cb 0x7f7c3c040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c49d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 0x564faef06b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c49d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 0x564faef06b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46700/0 (socket says 172.21.15.31:46700) 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c49d9b640 1 -- 172.21.15.31:0/3483445804 learned_addr learned my addr 172.21.15.31:0/3483445804 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 0x564faeef3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeef4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c49d9b640 1 -- 172.21.15.31:0/3483445804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 msgr2=0x564faeef4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeef4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c49d9b640 1 -- 172.21.15.31:0/3483445804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 msgr2=0x564faeef3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 0x564faeef3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c49d9b640 1 -- 172.21.15.31:0/3483445804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564faeef4740 con 0x564faeeef240 2024-04-25T02:00:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.653+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 0x564faeef3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c49d9b640 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 0x564faef06b10 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009970 tx=0x7f7c40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6321472eaf0d0cde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:00:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c31ffb640 1 -- 172.21.15.31:0/3483445804 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40029070 con 0x564faeeef240 2024-04-25T02:00:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c31ffb640 1 -- 172.21.15.31:0/3483445804 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f7c40004250 con 0x564faeeef240 2024-04-25T02:00:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c53990740 1 -- 172.21.15.31:0/3483445804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c3c046450 con 0x564faeeef240 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c31ffb640 1 -- 172.21.15.31:0/3483445804 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7c40032050 con 0x564faeeef240 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c53990740 10 client.16644 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c53990740 1 -- 172.21.15.31:0/3483445804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564faef74aa0 con 0x564faeeef240 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c31ffb640 1 -- 172.21.15.31:0/3483445804 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c3c046450 con 0x564faeeef240 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c31ffb640 1 -- 172.21.15.31:0/3483445804 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f7c4002c020 con 0x564faeeef240 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c53990740 10 client.16644 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c53990740 10 client.16644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c53990740 20 client.16644 populate_metadata read hostname 'smithi031' 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c53990740 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564faef75220 0x564faef77640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c53990740 4 client.16644 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.654+0000 7f7c53990740 1 -- 172.21.15.31:0/3483445804 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564faee5b540 con 0x564faef75220 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.655+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564faef75220 0x564faef77640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.655+0000 7f7c4a59c640 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564faef75220 0x564faef77640 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49440dfd80d61587 server_cookie=62cd056bf90b82cf in_seq=0 out_seq=0 2024-04-25T02:00:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.655+0000 7f7c31ffb640 10 client.16644 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.680+0000 7f7c31ffb640 1 -- 172.21.15.31:0/3483445804 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564faee5b540 con 0x564faef75220 2024-04-25T02:00:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.680+0000 7f7c31ffb640 10 client.16644 handle_command_reply: tid=0 2024-04-25T02:00:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.680+0000 7f7c53990740 1 client.16644 shutdown 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.680+0000 7f7c53990740 1 -- 172.21.15.31:0/3483445804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 msgr2=0x564faef06b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.680+0000 7f7c53990740 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 0x564faef06b10 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009970 tx=0x7f7c40004730 comp rx=0 tx=0).stop 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c53990740 1 -- 172.21.15.31:0/3483445804 shutdown_connections 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c53990740 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564faef75220 0x564faef77640 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c53990740 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564faeefa230 0x564faeef4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c53990740 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564faeeefb60 0x564faeef3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c31ffb640 0 client.16644 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c53990740 1 --2- 172.21.15.31:0/3483445804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564faeeef240 0x564faef06b10 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c53990740 1 -- 172.21.15.31:0/3483445804 >> 172.21.15.31:0/3483445804 conn(0x564faeeeb560 msgr2=0x564faeee9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c53990740 1 -- 172.21.15.31:0/3483445804 shutdown_connections 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c53990740 1 -- 172.21.15.31:0/3483445804 wait complete. 2024-04-25T02:00:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c53990740 20 client.16644 trim_cache size 0 max 16384 2024-04-25T02:00:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:00:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:00:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:00:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:00:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c50ba2640 1 -- 172.21.15.31:0/2432741545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7c340a2da0 msgr2=0x7f7c340a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c50ba2640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7c340a2da0 0x7f7c340a5260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7c4409bd80 tx=0x7f7c44091510 comp rx=0 tx=0).stop 2024-04-25T02:00:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c50ba2640 1 -- 172.21.15.31:0/2432741545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c106630 msgr2=0x7f7c4c19c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.681+0000 7f7c50ba2640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c106630 0x7f7c4c19c780 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009fd0 tx=0x7f7c4400ef10 comp rx=0 tx=0).stop 2024-04-25T02:00:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.682+0000 7f7c50ba2640 1 -- 172.21.15.31:0/2432741545 shutdown_connections 2024-04-25T02:00:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.682+0000 7f7c50ba2640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7c340a2da0 0x7f7c340a5260 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.682+0000 7f7c50ba2640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c4c106630 0x7f7c4c19c780 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.682+0000 7f7c50ba2640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c4c101030 0x7f7c4c198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.682+0000 7f7c50ba2640 1 --2- 172.21.15.31:0/2432741545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c4c100670 0x7f7c4c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.682+0000 7f7c50ba2640 1 -- 172.21.15.31:0/2432741545 >> 172.21.15.31:0/2432741545 conn(0x7f7c4c0777c0 msgr2=0x7f7c4c069190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.682+0000 7f7c50ba2640 1 -- 172.21.15.31:0/2432741545 shutdown_connections 2024-04-25T02:00:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:34.682+0000 7f7c50ba2640 1 -- 172.21.15.31:0/2432741545 wait complete. 2024-04-25T02:00:35.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 135062 2.51 MB/sec execute 162 sec latency 20.004 ms 2024-04-25T02:00:36.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 135294 2.49 MB/sec execute 163 sec latency 13.761 ms 2024-04-25T02:00:36.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:36 smithi040 ceph-mon[32418]: pgmap v187: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:00:36.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:36 smithi134 ceph-mon[33098]: pgmap v187: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:00:36.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:36 smithi031 ceph-mon[29204]: pgmap v187: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:00:37.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 135694 2.48 MB/sec execute 164 sec latency 18.236 ms 2024-04-25T02:00:38.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 136211 2.49 MB/sec execute 165 sec latency 24.767 ms 2024-04-25T02:00:38.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:38 smithi040 ceph-mon[32418]: pgmap v188: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-04-25T02:00:38.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:38 smithi134 ceph-mon[33098]: pgmap v188: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-04-25T02:00:38.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:38 smithi031 ceph-mon[29204]: pgmap v188: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-04-25T02:00:39.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 136654 2.48 MB/sec execute 166 sec latency 20.448 ms 2024-04-25T02:00:40.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 137195 2.47 MB/sec execute 167 sec latency 20.244 ms 2024-04-25T02:00:40.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:40 smithi040 ceph-mon[32418]: pgmap v189: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-04-25T02:00:40.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:40 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:00:40.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:40 smithi134 ceph-mon[33098]: pgmap v189: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-04-25T02:00:40.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:40 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:00:40.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:40 smithi031 ceph-mon[29204]: pgmap v189: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-04-25T02:00:40.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:40 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:00:41.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 137791 2.49 MB/sec execute 168 sec latency 19.002 ms 2024-04-25T02:00:41.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:41 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:00:41.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:41 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:00:41.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:41 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:00:41.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:41 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:00:41.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:41 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:00:41.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:41 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:00:41.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:41 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:00:41.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:41 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:00:41.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:41 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:00:42.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 138205 2.49 MB/sec execute 169 sec latency 21.878 ms 2024-04-25T02:00:42.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:42 smithi031 ceph-mon[29204]: pgmap v190: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T02:00:42.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:42 smithi040 ceph-mon[32418]: pgmap v190: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T02:00:42.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:42 smithi134 ceph-mon[33098]: pgmap v190: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-04-25T02:00:43.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 138661 2.50 MB/sec execute 170 sec latency 19.694 ms 2024-04-25T02:00:44.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 138921 2.49 MB/sec execute 171 sec latency 14.895 ms 2024-04-25T02:00:44.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:44 smithi031 ceph-mon[29204]: pgmap v191: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-25T02:00:44.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:44 smithi040 ceph-mon[32418]: pgmap v191: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-25T02:00:44.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:44 smithi134 ceph-mon[33098]: pgmap v191: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-25T02:00:45.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 139305 2.48 MB/sec execute 172 sec latency 133.171 ms 2024-04-25T02:00:46.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 139819 2.49 MB/sec execute 173 sec latency 19.850 ms 2024-04-25T02:00:46.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:46 smithi031 ceph-mon[29204]: pgmap v192: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-25T02:00:46.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:46 smithi040 ceph-mon[32418]: pgmap v192: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-25T02:00:46.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:46 smithi134 ceph-mon[33098]: pgmap v192: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-25T02:00:47.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 140293 2.48 MB/sec execute 174 sec latency 23.169 ms 2024-04-25T02:00:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.778+0000 7f37c1ff9640 1 -- 172.21.15.31:0/3729728438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37bc1029d0 msgr2=0x7f37bc102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.778+0000 7f37c1ff9640 1 --2- 172.21.15.31:0/3729728438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37bc1029d0 0x7f37bc102db0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009ca0 tx=0x7f37ac02a9e0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.778+0000 7f37c1ff9640 1 -- 172.21.15.31:0/3729728438 shutdown_connections 2024-04-25T02:00:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.778+0000 7f37c1ff9640 1 --2- 172.21.15.31:0/3729728438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc10bd10 0x7f37bc10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.778+0000 7f37c1ff9640 1 --2- 172.21.15.31:0/3729728438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37bc1032f0 0x7f37bc10b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.778+0000 7f37c1ff9640 1 --2- 172.21.15.31:0/3729728438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37bc1029d0 0x7f37bc102db0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.778+0000 7f37c1ff9640 1 -- 172.21.15.31:0/3729728438 >> 172.21.15.31:0/3729728438 conn(0x7f37bc0777c0 msgr2=0x7f37bc077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.778+0000 7f37c1ff9640 1 -- 172.21.15.31:0/3729728438 shutdown_connections 2024-04-25T02:00:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.778+0000 7f37c1ff9640 1 -- 172.21.15.31:0/3729728438 wait complete. 2024-04-25T02:00:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37c1ff9640 1 Processor -- start 2024-04-25T02:00:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37c1ff9640 1 -- start start 2024-04-25T02:00:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37c1ff9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37bc1029d0 0x7f37bc100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37c1ff9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc1032f0 0x7f37bc100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37c1ff9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37bc10bd10 0x7f37bc1010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37c1ff9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1119c0 con 0x7f37bc10bd10 2024-04-25T02:00:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37c1ff9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc111840 con 0x7f37bc1032f0 2024-04-25T02:00:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37c1ff9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc111b40 con 0x7f37bc1029d0 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37bbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37bc10bd10 0x7f37bc1010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37baffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc1032f0 0x7f37bc100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37bbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37bc10bd10 0x7f37bc1010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42984/0 (socket says 172.21.15.31:42984) 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37baffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc1032f0 0x7f37bc100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40518/0 (socket says 172.21.15.31:40518) 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37bbfff640 1 -- 172.21.15.31:0/1778442972 learned_addr learned my addr 172.21.15.31:0/1778442972 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.779+0000 7f37bb7fe640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37bc1029d0 0x7f37bc100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37bbfff640 1 -- 172.21.15.31:0/1778442972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37bc1029d0 msgr2=0x7f37bc100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37bbfff640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37bc1029d0 0x7f37bc100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37bbfff640 1 -- 172.21.15.31:0/1778442972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc1032f0 msgr2=0x7f37bc100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37bbfff640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc1032f0 0x7f37bc100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37bbfff640 1 -- 172.21.15.31:0/1778442972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37bc1a64c0 con 0x7f37bc10bd10 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37baffd640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc1032f0 0x7f37bc100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37bbfff640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37bc10bd10 0x7f37bc1010a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f37b400e970 tx=0x7f37b400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37b8ff9640 1 -- 172.21.15.31:0/1778442972 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b400ccc0 con 0x7f37bc10bd10 2024-04-25T02:00:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37c1ff9640 1 -- 172.21.15.31:0/1778442972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37bc1a67b0 con 0x7f37bc10bd10 2024-04-25T02:00:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37b8ff9640 1 -- 172.21.15.31:0/1778442972 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37b400ce60 con 0x7f37bc10bd10 2024-04-25T02:00:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37b8ff9640 1 -- 172.21.15.31:0/1778442972 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4010430 con 0x7f37bc10bd10 2024-04-25T02:00:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.780+0000 7f37c1ff9640 1 -- 172.21.15.31:0/1778442972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37bc1a6cb0 con 0x7f37bc10bd10 2024-04-25T02:00:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.782+0000 7f37b8ff9640 1 -- 172.21.15.31:0/1778442972 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f37b4004ff0 con 0x7f37bc10bd10 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37c4de8740 1 Processor -- start 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37c4de8740 1 -- start start 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15b240 0x56371e15b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15bb60 0x56371e165cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e168620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37c4de8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56371e0bfcb0 con 0x56371e15bb60 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37c4de8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56371e0685b0 con 0x56371e166230 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37c4de8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56371e0bbc70 con 0x56371e15b240 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37bb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15bb60 0x56371e165cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37bb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15bb60 0x56371e165cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42988/0 (socket says 172.21.15.31:42988) 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37bb7fe640 1 -- 172.21.15.31:0/270905625 learned_addr learned my addr 172.21.15.31:0/270905625 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37baffd640 1 --2- 172.21.15.31:0/270905625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e168620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.783+0000 7f37b8ff9640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37a40a2da0 0x7f37a40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.784+0000 7f37baffd640 1 --2- 172.21.15.31:0/270905625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15b240 0x56371e15b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.784+0000 7f37bbfff640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37a40a2da0 0x7f37a40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.784+0000 7f37bb7fe640 1 -- 172.21.15.31:0/270905625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15b240 msgr2=0x56371e15b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.784+0000 7f37bb7fe640 1 --2- 172.21.15.31:0/270905625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15b240 0x56371e15b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.784+0000 7f37bb7fe640 1 -- 172.21.15.31:0/270905625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 msgr2=0x56371e168620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.784+0000 7f37bb7fe640 1 --2- 172.21.15.31:0/270905625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e168620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.784+0000 7f37bb7fe640 1 -- 172.21.15.31:0/270905625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56371ddd4210 con 0x56371e15bb60 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.784+0000 7f37baffd640 1 --2- 172.21.15.31:0/270905625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e168620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.784+0000 7f37b8ff9640 1 -- 172.21.15.31:0/1778442972 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f37b40cf7a0 con 0x7f37bc10bd10 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.784+0000 7f37bb7fe640 1 --2- 172.21.15.31:0/270905625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15bb60 0x56371e165cf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009430 tx=0x7f37ac005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5403f7cfd0f66ee7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.784+0000 7f37bbfff640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37a40a2da0 0x7f37a40a5260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f37b40ccff0 tx=0x7f37b401bb30 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37a1ffb640 1 -- 172.21.15.31:0/270905625 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac049070 con 0x56371e15bb60 2024-04-25T02:00:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37a1ffb640 1 -- 172.21.15.31:0/270905625 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37ac02e840 con 0x56371e15bb60 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37a1ffb640 1 -- 172.21.15.31:0/270905625 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac033040 con 0x56371e15bb60 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37c4de8740 1 -- 172.21.15.31:0/270905625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15bb60 msgr2=0x56371e165cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37c4de8740 1 --2- 172.21.15.31:0/270905625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15bb60 0x56371e165cf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009430 tx=0x7f37ac005fa0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37c4de8740 1 -- 172.21.15.31:0/270905625 shutdown_connections 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37c4de8740 1 --2- 172.21.15.31:0/270905625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e168620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37c4de8740 1 --2- 172.21.15.31:0/270905625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15bb60 0x56371e165cf0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37c4de8740 1 --2- 172.21.15.31:0/270905625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15b240 0x56371e15b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37c4de8740 1 -- 172.21.15.31:0/270905625 >> 172.21.15.31:0/270905625 conn(0x56371e157560 msgr2=0x56371e157940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37c4de8740 1 -- 172.21.15.31:0/270905625 shutdown_connections 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.785+0000 7f37c4de8740 1 -- 172.21.15.31:0/270905625 wait complete. 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.786+0000 7f37c4de8740 1 -- start start 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.786+0000 7f37c4de8740 1 Processor -- start 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.786+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e163230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.786+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e163770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.786+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e163cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.786+0000 7f37c4de8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56371e168fc0 con 0x56371e15b240 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.786+0000 7f37c4de8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56371e07bb50 con 0x56371e166230 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.786+0000 7f37c4de8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56371e169140 con 0x56371e15bb60 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.786+0000 7f37c4de8740 10 client.0 _ll_register_callbacks cb 0x56371e0b6ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37baffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e163230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37baffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e163230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42992/0 (socket says 172.21.15.31:42992) 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37baffd640 1 -- 172.21.15.31:0/4234678341 learned_addr learned my addr 172.21.15.31:0/4234678341 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37bb7fe640 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e163770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37baffd640 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e163cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37baffd640 1 -- 172.21.15.31:0/4234678341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 msgr2=0x56371e163770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37baffd640 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e163770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37baffd640 1 -- 172.21.15.31:0/4234678341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 msgr2=0x56371e163cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37baffd640 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e163cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37baffd640 1 -- 172.21.15.31:0/4234678341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56371dea0620 con 0x56371e15b240 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37baffd640 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e163230 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f37b0011500 tx=0x7f37b0013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a707408902a82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f379affd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f379affd640 1 -- 172.21.15.31:0/4234678341 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b0004220 con 0x56371e15b240 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f379affd640 1 -- 172.21.15.31:0/4234678341 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f37b00043c0 con 0x56371e15b240 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f37c4de8740 1 -- 172.21.15.31:0/4234678341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37ac02e840 con 0x56371e15b240 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.787+0000 7f379affd640 1 -- 172.21.15.31:0/4234678341 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f37b0028050 con 0x56371e15b240 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.788+0000 7f37c4de8740 10 client.16656 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.788+0000 7f37c4de8740 1 -- 172.21.15.31:0/4234678341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56371e175280 con 0x56371e15b240 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.788+0000 7f379affd640 1 -- 172.21.15.31:0/4234678341 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37ac02e840 con 0x56371e15b240 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.788+0000 7f379affd640 1 -- 172.21.15.31:0/4234678341 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f37b0003790 con 0x56371e15b240 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.788+0000 7f37c4de8740 10 client.16656 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.788+0000 7f37c4de8740 10 client.16656 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:47.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.788+0000 7f37c4de8740 20 client.16656 populate_metadata read hostname 'smithi031' 2024-04-25T02:00:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.788+0000 7f37c4de8740 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56371e176740 0x56371e178b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.788+0000 7f37c4de8740 4 client.16656 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.788+0000 7f37c4de8740 1 -- 172.21.15.31:0/4234678341 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56371e0cd1f0 con 0x56371e176740 2024-04-25T02:00:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.788+0000 7f37bb7fe640 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56371e176740 0x56371e178b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.789+0000 7f379affd640 10 client.16656 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.789+0000 7f37bb7fe640 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56371e176740 0x56371e178b30 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4877b1fdb2af7cb5 server_cookie=93c88a9cd6e8cb84 in_seq=0 out_seq=0 2024-04-25T02:00:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.789+0000 7f379affd640 1 -- 172.21.15.31:0/4234678341 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56371e0cd1f0 con 0x56371e176740 2024-04-25T02:00:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.789+0000 7f379affd640 10 client.16656 handle_command_reply: tid=0 2024-04-25T02:00:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.789+0000 7f37c4de8740 1 client.16656 shutdown 2024-04-25T02:00:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f37c4de8740 1 -- 172.21.15.31:0/4234678341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 msgr2=0x56371e163230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f37c4de8740 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e163230 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f37b0011500 tx=0x7f37b0013040 comp rx=0 tx=0).stop 2024-04-25T02:00:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f37c4de8740 1 -- 172.21.15.31:0/4234678341 shutdown_connections 2024-04-25T02:00:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f37c4de8740 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56371e176740 0x56371e178b30 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f37c4de8740 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e163cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f379affd640 0 client.16656 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f37c4de8740 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e163770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f37c4de8740 1 --2- 172.21.15.31:0/4234678341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e163230 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f37c4de8740 1 -- 172.21.15.31:0/4234678341 >> 172.21.15.31:0/4234678341 conn(0x56371e157560 msgr2=0x56371e15d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:47.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f37c4de8740 1 -- 172.21.15.31:0/4234678341 shutdown_connections 2024-04-25T02:00:47.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f37c4de8740 1 -- 172.21.15.31:0/4234678341 wait complete. 2024-04-25T02:00:47.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.790+0000 7f37c4de8740 20 client.16656 trim_cache size 0 max 16384 2024-04-25T02:00:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37c4de8740 1 Processor -- start 2024-04-25T02:00:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37c4de8740 1 -- start start 2024-04-25T02:00:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e171e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e15d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e17e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37c4de8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1af530 con 0x56371e15b240 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37c4de8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1b1070 con 0x56371e166230 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37c4de8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1afc00 con 0x56371e15bb60 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37baffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e171e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37baffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e171e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43006/0 (socket says 172.21.15.31:43006) 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37baffd640 1 -- 172.21.15.31:0/155068350 learned_addr learned my addr 172.21.15.31:0/155068350 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37bb7fe640 1 --2- 172.21.15.31:0/155068350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e15d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.806+0000 7f37baffd640 1 --2- 172.21.15.31:0/155068350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e17e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.807+0000 7f37baffd640 1 -- 172.21.15.31:0/155068350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 msgr2=0x56371e15d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.807+0000 7f37baffd640 1 --2- 172.21.15.31:0/155068350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e15d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.807+0000 7f37baffd640 1 -- 172.21.15.31:0/155068350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 msgr2=0x56371e17e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.807+0000 7f37baffd640 1 --2- 172.21.15.31:0/155068350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e17e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.807+0000 7f37baffd640 1 -- 172.21.15.31:0/155068350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56371e168620 con 0x56371e15b240 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.807+0000 7f37baffd640 1 --2- 172.21.15.31:0/155068350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e171e80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f37b000aab0 tx=0x7f37b0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6710eff675b232b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:47.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.807+0000 7f37a17fa640 1 -- 172.21.15.31:0/155068350 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b0034020 con 0x56371e15b240 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.807+0000 7f37a17fa640 1 -- 172.21.15.31:0/155068350 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37b00037f0 con 0x56371e15b240 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.807+0000 7f37a17fa640 1 -- 172.21.15.31:0/155068350 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b0004c40 con 0x56371e15b240 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 -- 172.21.15.31:0/155068350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 msgr2=0x56371e171e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 --2- 172.21.15.31:0/155068350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e171e80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f37b000aab0 tx=0x7f37b0000bc0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 -- 172.21.15.31:0/155068350 shutdown_connections 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 --2- 172.21.15.31:0/155068350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e17e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 --2- 172.21.15.31:0/155068350 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e15d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 --2- 172.21.15.31:0/155068350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e171e80 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 -- 172.21.15.31:0/155068350 >> 172.21.15.31:0/155068350 conn(0x56371e157560 msgr2=0x56371e1565c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 -- 172.21.15.31:0/155068350 shutdown_connections 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 -- 172.21.15.31:0/155068350 wait complete. 2024-04-25T02:00:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 -- start start 2024-04-25T02:00:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 Processor -- start 2024-04-25T02:00:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e16e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e169cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.808+0000 7f37c4de8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e16a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37c4de8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1b09a0 con 0x56371e15b240 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37c4de8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1b02d0 con 0x56371e166230 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37c4de8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1ad510 con 0x56371e15bb60 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37c4de8740 10 client.0 _ll_register_callbacks cb 0x7f37b0011b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37baffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e16e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37baffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e16e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43018/0 (socket says 172.21.15.31:43018) 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37baffd640 1 -- 172.21.15.31:0/3883502688 learned_addr learned my addr 172.21.15.31:0/3883502688 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37bb7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e169cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37baffd640 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e16a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37baffd640 1 -- 172.21.15.31:0/3883502688 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 msgr2=0x56371e169cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37baffd640 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e169cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37baffd640 1 -- 172.21.15.31:0/3883502688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 msgr2=0x56371e16a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37baffd640 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e16a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37baffd640 1 -- 172.21.15.31:0/3883502688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56371e1dbc90 con 0x56371e15b240 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37baffd640 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e16e8f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f37b0011530 tx=0x7f37b0010c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f2a79d51ecceb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f379affd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f379affd640 1 -- 172.21.15.31:0/3883502688 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b0003fe0 con 0x56371e15b240 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f379affd640 1 -- 172.21.15.31:0/3883502688 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f37b0004180 con 0x56371e15b240 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37c4de8740 1 -- 172.21.15.31:0/3883502688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37b00037f0 con 0x56371e15b240 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f379affd640 1 -- 172.21.15.31:0/3883502688 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f37b003e080 con 0x56371e15b240 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37c4de8740 10 client.16662 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.809+0000 7f37c4de8740 1 -- 172.21.15.31:0/3883502688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56371e1dce40 con 0x56371e15b240 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.810+0000 7f379affd640 1 -- 172.21.15.31:0/3883502688 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37b00037f0 con 0x56371e15b240 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.810+0000 7f379affd640 1 -- 172.21.15.31:0/3883502688 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f37b0046480 con 0x56371e15b240 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.810+0000 7f37c4de8740 10 client.16662 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.810+0000 7f37c4de8740 10 client.16662 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.810+0000 7f37c4de8740 20 client.16662 populate_metadata read hostname 'smithi031' 2024-04-25T02:00:47.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.810+0000 7f37c4de8740 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56371e1dd3f0 0x56371e1df7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:47.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.810+0000 7f37c4de8740 4 client.16662 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:00:47.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.810+0000 7f37c4de8740 1 -- 172.21.15.31:0/3883502688 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56371e0c7540 con 0x56371e1dd3f0 2024-04-25T02:00:47.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.810+0000 7f37bb7fe640 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56371e1dd3f0 0x56371e1df7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:47.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.810+0000 7f37bb7fe640 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56371e1dd3f0 0x56371e1df7e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd029f93fa6dabd9 server_cookie=7a87504829341463 in_seq=0 out_seq=0 2024-04-25T02:00:47.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.810+0000 7f379affd640 10 client.16662 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f379affd640 1 -- 172.21.15.31:0/3883502688 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56371e0c7540 con 0x56371e1dd3f0 2024-04-25T02:00:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f379affd640 10 client.16662 handle_command_reply: tid=0 2024-04-25T02:00:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f37c4de8740 1 client.16662 shutdown 2024-04-25T02:00:47.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f37c4de8740 1 -- 172.21.15.31:0/3883502688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 msgr2=0x56371e16e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f37c4de8740 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e16e8f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f37b0011530 tx=0x7f37b0010c90 comp rx=0 tx=0).stop 2024-04-25T02:00:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f37c4de8740 1 -- 172.21.15.31:0/3883502688 shutdown_connections 2024-04-25T02:00:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f37c4de8740 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56371e1dd3f0 0x56371e1df7e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f37c4de8740 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56371e166230 0x56371e16a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f379affd640 0 client.16662 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:00:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f37c4de8740 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56371e15bb60 0x56371e169cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f37c4de8740 1 --2- 172.21.15.31:0/3883502688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56371e15b240 0x56371e16e8f0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.833+0000 7f37c4de8740 1 -- 172.21.15.31:0/3883502688 >> 172.21.15.31:0/3883502688 conn(0x56371e157560 msgr2=0x56371e1631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c4de8740 1 -- 172.21.15.31:0/3883502688 shutdown_connections 2024-04-25T02:00:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c4de8740 1 -- 172.21.15.31:0/3883502688 wait complete. 2024-04-25T02:00:47.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c4de8740 20 client.16662 trim_cache size 0 max 16384 2024-04-25T02:00:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:00:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:00:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:00:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:00:47.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c1ff9640 1 -- 172.21.15.31:0/1778442972 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37a40a2da0 msgr2=0x7f37a40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c1ff9640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37a40a2da0 0x7f37a40a5260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f37b40ccff0 tx=0x7f37b401bb30 comp rx=0 tx=0).stop 2024-04-25T02:00:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c1ff9640 1 -- 172.21.15.31:0/1778442972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37bc10bd10 msgr2=0x7f37bc1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c1ff9640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37bc10bd10 0x7f37bc1010a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f37b400e970 tx=0x7f37b400ed70 comp rx=0 tx=0).stop 2024-04-25T02:00:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c1ff9640 1 -- 172.21.15.31:0/1778442972 shutdown_connections 2024-04-25T02:00:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c1ff9640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37a40a2da0 0x7f37a40a5260 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c1ff9640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37bc10bd10 0x7f37bc1010a0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c1ff9640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc1032f0 0x7f37bc100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c1ff9640 1 --2- 172.21.15.31:0/1778442972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37bc1029d0 0x7f37bc100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c1ff9640 1 -- 172.21.15.31:0/1778442972 >> 172.21.15.31:0/1778442972 conn(0x7f37bc0777c0 msgr2=0x7f37bc0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.834+0000 7f37c1ff9640 1 -- 172.21.15.31:0/1778442972 shutdown_connections 2024-04-25T02:00:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:00:47.835+0000 7f37c1ff9640 1 -- 172.21.15.31:0/1778442972 wait complete. 2024-04-25T02:00:48.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 140808 2.47 MB/sec execute 175 sec latency 21.663 ms 2024-04-25T02:00:48.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:48 smithi031 ceph-mon[29204]: pgmap v193: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T02:00:48.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:48 smithi040 ceph-mon[32418]: pgmap v193: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T02:00:48.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:48 smithi134 ceph-mon[33098]: pgmap v193: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T02:00:49.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 141535 2.49 MB/sec execute 176 sec latency 19.366 ms 2024-04-25T02:00:50.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 141910 2.50 MB/sec execute 177 sec latency 38.022 ms 2024-04-25T02:00:50.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:50 smithi031 ceph-mon[29204]: pgmap v194: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T02:00:50.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:50 smithi040 ceph-mon[32418]: pgmap v194: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T02:00:50.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:50 smithi134 ceph-mon[33098]: pgmap v194: 97 pgs: 97 active+clean; 87 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T02:00:51.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 142251 2.50 MB/sec execute 178 sec latency 19.445 ms 2024-04-25T02:00:52.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 142517 2.48 MB/sec execute 179 sec latency 17.502 ms 2024-04-25T02:00:52.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:52 smithi031 ceph-mon[29204]: pgmap v195: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 1.8 MiB/s wr, 182 op/s 2024-04-25T02:00:52.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:52 smithi040 ceph-mon[32418]: pgmap v195: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 1.8 MiB/s wr, 182 op/s 2024-04-25T02:00:52.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:52 smithi134 ceph-mon[33098]: pgmap v195: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 1.8 MiB/s wr, 182 op/s 2024-04-25T02:00:53.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 143019 2.48 MB/sec execute 180 sec latency 20.396 ms 2024-04-25T02:00:54.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 143527 2.48 MB/sec execute 181 sec latency 21.342 ms 2024-04-25T02:00:54.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:54 smithi031 ceph-mon[29204]: pgmap v196: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-25T02:00:54.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:54 smithi040 ceph-mon[32418]: pgmap v196: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-25T02:00:54.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:54 smithi134 ceph-mon[33098]: pgmap v196: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-25T02:00:55.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 144092 2.47 MB/sec execute 182 sec latency 21.976 ms 2024-04-25T02:00:56.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 144628 2.49 MB/sec execute 183 sec latency 18.076 ms 2024-04-25T02:00:56.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:56 smithi031 ceph-mon[29204]: pgmap v197: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.9 MiB/s wr, 180 op/s 2024-04-25T02:00:56.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:56 smithi040 ceph-mon[32418]: pgmap v197: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.9 MiB/s wr, 180 op/s 2024-04-25T02:00:56.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:56 smithi134 ceph-mon[33098]: pgmap v197: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.9 MiB/s wr, 180 op/s 2024-04-25T02:00:57.310 INFO:tasks.workunit.client.0.smithi031.stdout: 1 145225 2.49 MB/sec execute 184 sec latency 20.290 ms 2024-04-25T02:00:58.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 145683 2.50 MB/sec execute 185 sec latency 18.294 ms 2024-04-25T02:00:58.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:00:58 smithi031 ceph-mon[29204]: pgmap v198: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-04-25T02:00:58.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:00:58 smithi040 ceph-mon[32418]: pgmap v198: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-04-25T02:00:58.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:00:58 smithi134 ceph-mon[33098]: pgmap v198: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-04-25T02:00:59.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 145924 2.49 MB/sec execute 186 sec latency 13.274 ms 2024-04-25T02:01:00.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 146177 2.48 MB/sec execute 187 sec latency 279.954 ms 2024-04-25T02:01:00.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:00 smithi031 ceph-mon[29204]: pgmap v199: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-04-25T02:01:00.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:00 smithi040 ceph-mon[32418]: pgmap v199: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-04-25T02:01:00.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:00 smithi134 ceph-mon[33098]: pgmap v199: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-04-25T02:01:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.935+0000 7f244979e640 1 -- 172.21.15.31:0/2669979986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2444107740 msgr2=0x7f2444107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.935+0000 7f244979e640 1 --2- 172.21.15.31:0/2669979986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2444107740 0x7f2444107b20 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f2434009900 tx=0x7f243402a410 comp rx=0 tx=0).stop 2024-04-25T02:01:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.936+0000 7f244979e640 1 -- 172.21.15.31:0/2669979986 shutdown_connections 2024-04-25T02:01:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.936+0000 7f244979e640 1 --2- 172.21.15.31:0/2669979986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444100600 0x7f244410e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.936+0000 7f244979e640 1 --2- 172.21.15.31:0/2669979986 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24440ffc40 0x7f24441000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.936+0000 7f244979e640 1 --2- 172.21.15.31:0/2669979986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2444107740 0x7f2444107b20 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.936+0000 7f244979e640 1 -- 172.21.15.31:0/2669979986 >> 172.21.15.31:0/2669979986 conn(0x7f2444076b80 msgr2=0x7f24440fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.936+0000 7f244979e640 1 -- 172.21.15.31:0/2669979986 shutdown_connections 2024-04-25T02:01:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.936+0000 7f244979e640 1 -- 172.21.15.31:0/2669979986 wait complete. 2024-04-25T02:01:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.936+0000 7f244979e640 1 Processor -- start 2024-04-25T02:01:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.936+0000 7f244979e640 1 -- start start 2024-04-25T02:01:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f244979e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24440ffc40 0x7f2444197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f244979e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444100600 0x7f24441983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f244979e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2444107740 0x7f244419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f244979e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f24441119f0 con 0x7f2444107740 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f244979e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2444111870 con 0x7f2444100600 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f244979e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2444111b70 con 0x7f24440ffc40 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f24437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2444107740 0x7f244419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f24437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2444107740 0x7f244419c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38890/0 (socket says 172.21.15.31:38890) 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f24437fe640 1 -- 172.21.15.31:0/486539568 learned_addr learned my addr 172.21.15.31:0/486539568 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f24427fc640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444100600 0x7f24441983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f2442ffd640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24440ffc40 0x7f2444197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f24437fe640 1 -- 172.21.15.31:0/486539568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24440ffc40 msgr2=0x7f2444197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f24437fe640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24440ffc40 0x7f2444197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f24437fe640 1 -- 172.21.15.31:0/486539568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444100600 msgr2=0x7f24441983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f24437fe640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444100600 0x7f24441983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f24437fe640 1 -- 172.21.15.31:0/486539568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244419cea0 con 0x7f2444107740 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f24427fc640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444100600 0x7f24441983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f2442ffd640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24440ffc40 0x7f2444197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.937+0000 7f24437fe640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2444107740 0x7f244419c770 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f243c00eaa0 tx=0x7f243c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.938+0000 7f242bfff640 1 -- 172.21.15.31:0/486539568 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c00cdb0 con 0x7f2444107740 2024-04-25T02:01:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.938+0000 7f244979e640 1 -- 172.21.15.31:0/486539568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244419d190 con 0x7f2444107740 2024-04-25T02:01:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.938+0000 7f242bfff640 1 -- 172.21.15.31:0/486539568 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f243c005020 con 0x7f2444107740 2024-04-25T02:01:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.938+0000 7f242bfff640 1 -- 172.21.15.31:0/486539568 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c0116d0 con 0x7f2444107740 2024-04-25T02:01:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.938+0000 7f244979e640 1 -- 172.21.15.31:0/486539568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24441a4a20 con 0x7f2444107740 2024-04-25T02:01:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f242bfff640 1 -- 172.21.15.31:0/486539568 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f243c0051d0 con 0x7f2444107740 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f244c585740 1 Processor -- start 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f244c585740 1 -- start start 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f244c585740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a1937f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f244c585740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1937fa60 0x557a1938e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f244c585740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1938e640 0x557a19390a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f244c585740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557a192e3cb0 con 0x557a1937f140 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f244c585740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557a1928c5b0 con 0x557a1937fa60 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f244c585740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557a192dfc70 con 0x557a1938e640 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f24427fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a1937f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f24427fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a1937f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38900/0 (socket says 172.21.15.31:38900) 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f24427fc640 1 -- 172.21.15.31:0/3709952731 learned_addr learned my addr 172.21.15.31:0/3709952731 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f242bfff640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f242c0a2da0 0x7f242c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.941+0000 7f2442ffd640 1 --2- 172.21.15.31:0/3709952731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1937fa60 0x557a1938e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f24427fc640 1 --2- 172.21.15.31:0/3709952731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1938e640 0x557a19390a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f24437fe640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f242c0a2da0 0x7f242c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f24427fc640 1 -- 172.21.15.31:0/3709952731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1938e640 msgr2=0x557a19390a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f24427fc640 1 --2- 172.21.15.31:0/3709952731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1938e640 0x557a19390a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f24427fc640 1 -- 172.21.15.31:0/3709952731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1937fa60 msgr2=0x557a1938e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f24427fc640 1 --2- 172.21.15.31:0/3709952731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1937fa60 0x557a1938e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f24427fc640 1 -- 172.21.15.31:0/3709952731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a18ff8210 con 0x557a1937f140 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f242bfff640 1 -- 172.21.15.31:0/486539568 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f243c0cf640 con 0x7f2444107740 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f24427fc640 1 --2- 172.21.15.31:0/3709952731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a1937f520 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f243800be80 tx=0x7f243800b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7996c42ccbf980f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f24297fa640 1 -- 172.21.15.31:0/3709952731 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438019030 con 0x557a1937f140 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f24297fa640 1 -- 172.21.15.31:0/3709952731 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24380119f0 con 0x557a1937f140 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.942+0000 7f24297fa640 1 -- 172.21.15.31:0/3709952731 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243801d980 con 0x557a1937f140 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f244c585740 1 -- 172.21.15.31:0/3709952731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 msgr2=0x557a1937f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f24437fe640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f242c0a2da0 0x7f242c0a5260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f243c0a42f0 tx=0x7f243c00cf50 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f244c585740 1 --2- 172.21.15.31:0/3709952731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a1937f520 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f243800be80 tx=0x7f243800b730 comp rx=0 tx=0).stop 2024-04-25T02:01:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f244c585740 1 -- 172.21.15.31:0/3709952731 shutdown_connections 2024-04-25T02:01:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f244c585740 1 --2- 172.21.15.31:0/3709952731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1938e640 0x557a19390a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f244c585740 1 --2- 172.21.15.31:0/3709952731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1937fa60 0x557a1938e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f244c585740 1 --2- 172.21.15.31:0/3709952731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a1937f520 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f244c585740 1 -- 172.21.15.31:0/3709952731 >> 172.21.15.31:0/3709952731 conn(0x557a1937b560 msgr2=0x557a1937b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f244c585740 1 -- 172.21.15.31:0/3709952731 shutdown_connections 2024-04-25T02:01:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f244c585740 1 -- 172.21.15.31:0/3709952731 wait complete. 2024-04-25T02:01:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f244c585740 1 -- start start 2024-04-25T02:01:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.943+0000 7f244c585740 1 Processor -- start 2024-04-25T02:01:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f244c585740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1937f140 0x557a1938b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f244c585740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f244c585740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1938e640 0x557a1938bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f244c585740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557a19391400 con 0x557a1937fa60 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f244c585740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557a1929fb50 con 0x557a1938e640 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f244c585740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557a19391580 con 0x557a1937f140 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f244c585740 10 client.0 _ll_register_callbacks cb 0x557a192daca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2442ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2442ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38902/0 (socket says 172.21.15.31:38902) 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2442ffd640 1 -- 172.21.15.31:0/3416122203 learned_addr learned my addr 172.21.15.31:0/3416122203 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f24427fc640 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1937f140 0x557a1938b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f24427fc640 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1938e640 0x557a1938bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2442ffd640 1 -- 172.21.15.31:0/3416122203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1937f140 msgr2=0x557a1938b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2442ffd640 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1937f140 0x557a1938b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2442ffd640 1 -- 172.21.15.31:0/3416122203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1938e640 msgr2=0x557a1938bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2442ffd640 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1938e640 0x557a1938bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2442ffd640 1 -- 172.21.15.31:0/3416122203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a190c4620 con 0x557a1937fa60 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2442ffd640 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938bab0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2434009230 tx=0x7f2434005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b329f919449cd2d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2426ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2426ff5640 1 -- 172.21.15.31:0/3416122203 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2434035040 con 0x557a1937fa60 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f2426ff5640 1 -- 172.21.15.31:0/3416122203 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f24340048c0 con 0x557a1937fa60 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.944+0000 7f244c585740 1 -- 172.21.15.31:0/3416122203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24380119f0 con 0x557a1937fa60 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f2426ff5640 1 -- 172.21.15.31:0/3416122203 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2434005650 con 0x557a1937fa60 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f244c585740 10 client.16674 fetch_fsmap learned FSMap version 33 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f244c585740 1 -- 172.21.15.31:0/3416122203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x557a1939d5e0 con 0x557a1937fa60 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f2426ff5640 1 -- 172.21.15.31:0/3416122203 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24380119f0 con 0x557a1937fa60 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f2426ff5640 1 -- 172.21.15.31:0/3416122203 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f24340314a0 con 0x557a1937fa60 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f244c585740 10 client.16674 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f244c585740 10 client.16674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f244c585740 20 client.16674 populate_metadata read hostname 'smithi031' 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f244c585740 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a1939ea50 0x557a193a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f244c585740 4 client.16674 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f244c585740 1 -- 172.21.15.31:0/3416122203 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a192f11f0 con 0x557a1939ea50 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.945+0000 7f24427fc640 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a1939ea50 0x557a193a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.946+0000 7f2426ff5640 10 client.16674 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.946+0000 7f24427fc640 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a1939ea50 0x557a193a0e40 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c69635e445029c18 server_cookie=a8d7c2b13701948b in_seq=0 out_seq=0 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.946+0000 7f2426ff5640 1 -- 172.21.15.31:0/3416122203 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a192f11f0 con 0x557a1939ea50 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.946+0000 7f2426ff5640 10 client.16674 handle_command_reply: tid=0 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 1 client.16674 shutdown 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 1 -- 172.21.15.31:0/3416122203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 msgr2=0x557a1938bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938bab0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2434009230 tx=0x7f2434005250 comp rx=0 tx=0).stop 2024-04-25T02:01:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 1 -- 172.21.15.31:0/3416122203 shutdown_connections 2024-04-25T02:01:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a1939ea50 0x557a193a0e40 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1938e640 0x557a1938bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938bab0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 1 --2- 172.21.15.31:0/3416122203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1937f140 0x557a1938b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f2426ff5640 0 client.16674 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 1 -- 172.21.15.31:0/3416122203 >> 172.21.15.31:0/3416122203 conn(0x557a1937b560 msgr2=0x557a19381390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 1 -- 172.21.15.31:0/3416122203 shutdown_connections 2024-04-25T02:01:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 1 -- 172.21.15.31:0/3416122203 wait complete. 2024-04-25T02:01:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.947+0000 7f244c585740 20 client.16674 trim_cache size 0 max 16384 2024-04-25T02:01:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.962+0000 7f244c585740 1 Processor -- start 2024-04-25T02:01:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.962+0000 7f244c585740 1 -- start start 2024-04-25T02:01:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.962+0000 7f244c585740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1937f140 0x557a1939a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.962+0000 7f244c585740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.962+0000 7f244c585740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1938e640 0x557a193a6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.962+0000 7f244c585740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f24441ab050 con 0x557a1937fa60 2024-04-25T02:01:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.962+0000 7f244c585740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24441acb90 con 0x557a1937f140 2024-04-25T02:01:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.962+0000 7f244c585740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f24441ab720 con 0x557a1938e640 2024-04-25T02:01:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.962+0000 7f2442ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.962+0000 7f2442ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38918/0 (socket says 172.21.15.31:38918) 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.962+0000 7f2442ffd640 1 -- 172.21.15.31:0/1167722494 learned_addr learned my addr 172.21.15.31:0/1167722494 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f24427fc640 1 --2- 172.21.15.31:0/1167722494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1938e640 0x557a193a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f24427fc640 1 --2- 172.21.15.31:0/1167722494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1937f140 0x557a1939a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f2442ffd640 1 -- 172.21.15.31:0/1167722494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1938e640 msgr2=0x557a193a6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f2442ffd640 1 --2- 172.21.15.31:0/1167722494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1938e640 0x557a193a6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f2442ffd640 1 -- 172.21.15.31:0/1167722494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1937f140 msgr2=0x557a1939a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f2442ffd640 1 --2- 172.21.15.31:0/1167722494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1937f140 0x557a1939a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f2442ffd640 1 -- 172.21.15.31:0/1167722494 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a193a14a0 con 0x557a1937fa60 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f24427fc640 1 --2- 172.21.15.31:0/1167722494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1937f140 0x557a1939a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f2442ffd640 1 --2- 172.21.15.31:0/1167722494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938be10 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f243402a930 tx=0x7f24340316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed984b859cd8760 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f2428ff9640 1 -- 172.21.15.31:0/1167722494 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2434000e60 con 0x557a1937fa60 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f2428ff9640 1 -- 172.21.15.31:0/1167722494 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2434040070 con 0x557a1937fa60 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.963+0000 7f2428ff9640 1 -- 172.21.15.31:0/1167722494 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2434035070 con 0x557a1937fa60 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 -- 172.21.15.31:0/1167722494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 msgr2=0x557a1938be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 --2- 172.21.15.31:0/1167722494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938be10 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f243402a930 tx=0x7f24340316b0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 -- 172.21.15.31:0/1167722494 shutdown_connections 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 --2- 172.21.15.31:0/1167722494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1938e640 0x557a193a6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 --2- 172.21.15.31:0/1167722494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937fa60 0x557a1938be10 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 --2- 172.21.15.31:0/1167722494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1937f140 0x557a1939a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 -- 172.21.15.31:0/1167722494 >> 172.21.15.31:0/1167722494 conn(0x557a1937b560 msgr2=0x557a1937a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 -- 172.21.15.31:0/1167722494 shutdown_connections 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 -- 172.21.15.31:0/1167722494 wait complete. 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 -- start start 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 Processor -- start 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a193882b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1937fa60 0x557a193887f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1938e640 0x557a19388d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f24441ac4c0 con 0x557a1937f140 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24441abdf0 con 0x557a1938e640 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f24441a9050 con 0x557a1937fa60 2024-04-25T02:01:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.964+0000 7f244c585740 10 client.0 _ll_register_callbacks cb 0x7f2434004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f24427fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a193882b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f24427fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a193882b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38932/0 (socket says 172.21.15.31:38932) 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f24427fc640 1 -- 172.21.15.31:0/1416611458 learned_addr learned my addr 172.21.15.31:0/1416611458 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f2442ffd640 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1937fa60 0x557a193887f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f24427fc640 1 -- 172.21.15.31:0/1416611458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1937fa60 msgr2=0x557a193887f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f24427fc640 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1937fa60 0x557a193887f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f24427fc640 1 -- 172.21.15.31:0/1416611458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1938e640 msgr2=0x557a19388d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f24427fc640 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1938e640 0x557a19388d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f24427fc640 1 -- 172.21.15.31:0/1416611458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a19403fb0 con 0x557a1937f140 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f2442ffd640 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1937fa60 0x557a193887f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f24427fc640 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a193882b0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2438013fa0 tx=0x7f243800b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72bf29dafde68e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f2426ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f2426ff5640 1 -- 172.21.15.31:0/1416611458 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438019040 con 0x557a1937f140 2024-04-25T02:01:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f2426ff5640 1 -- 172.21.15.31:0/1416611458 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8440+0+0 (secure 0 0 0) 0x7f243802cae0 con 0x557a1937f140 2024-04-25T02:01:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f244c585740 1 -- 172.21.15.31:0/1416611458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2434040070 con 0x557a1937f140 2024-04-25T02:01:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f2426ff5640 1 -- 172.21.15.31:0/1416611458 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f243803a050 con 0x557a1937f140 2024-04-25T02:01:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f244c585740 10 client.16680 fetch_fsmap learned FSMap version 33 2024-04-25T02:01:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.965+0000 7f244c585740 1 -- 172.21.15.31:0/1416611458 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x557a19405230 con 0x557a1937f140 2024-04-25T02:01:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.966+0000 7f2426ff5640 1 -- 172.21.15.31:0/1416611458 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2434040070 con 0x557a1937f140 2024-04-25T02:01:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.966+0000 7f2426ff5640 1 -- 172.21.15.31:0/1416611458 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f243802cce0 con 0x557a1937f140 2024-04-25T02:01:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.966+0000 7f244c585740 10 client.16680 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:01:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.966+0000 7f244c585740 10 client.16680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.966+0000 7f244c585740 20 client.16680 populate_metadata read hostname 'smithi031' 2024-04-25T02:01:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.966+0000 7f244c585740 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a19405720 0x557a19407b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.966+0000 7f244c585740 4 client.16680 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:01:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.966+0000 7f244c585740 1 -- 172.21.15.31:0/1416611458 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a192eb540 con 0x557a19405720 2024-04-25T02:01:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.966+0000 7f2442ffd640 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a19405720 0x557a19407b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.966+0000 7f2426ff5640 10 client.16680 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.966+0000 7f2442ffd640 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a19405720 0x557a19407b40 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f941fae6d07d5fc server_cookie=2c59770219c31d5e in_seq=0 out_seq=0 2024-04-25T02:01:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f2426ff5640 1 -- 172.21.15.31:0/1416611458 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557a192eb540 con 0x557a19405720 2024-04-25T02:01:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f2426ff5640 10 client.16680 handle_command_reply: tid=0 2024-04-25T02:01:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 1 client.16680 shutdown 2024-04-25T02:01:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 1 -- 172.21.15.31:0/1416611458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 msgr2=0x557a193882b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a193882b0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2438013fa0 tx=0x7f243800b970 comp rx=0 tx=0).stop 2024-04-25T02:01:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 1 -- 172.21.15.31:0/1416611458 shutdown_connections 2024-04-25T02:01:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557a19405720 0x557a19407b40 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a1938e640 0x557a19388d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f2426ff5640 0 client.16680 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557a1937fa60 0x557a193887f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 1 --2- 172.21.15.31:0/1416611458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a1937f140 0x557a193882b0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 1 -- 172.21.15.31:0/1416611458 >> 172.21.15.31:0/1416611458 conn(0x557a1937b560 msgr2=0x557a19380a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 1 -- 172.21.15.31:0/1416611458 shutdown_connections 2024-04-25T02:01:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 1 -- 172.21.15.31:0/1416611458 wait complete. 2024-04-25T02:01:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.988+0000 7f244c585740 20 client.16680 trim_cache size 0 max 16384 2024-04-25T02:01:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:01:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:01:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:01:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:01:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 -- 172.21.15.31:0/486539568 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f242c0a2da0 msgr2=0x7f242c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f242c0a2da0 0x7f242c0a5260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f243c0a42f0 tx=0x7f243c00cf50 comp rx=0 tx=0).stop 2024-04-25T02:01:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 -- 172.21.15.31:0/486539568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2444107740 msgr2=0x7f244419c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2444107740 0x7f244419c770 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f243c00eaa0 tx=0x7f243c00ef60 comp rx=0 tx=0).stop 2024-04-25T02:01:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 -- 172.21.15.31:0/486539568 shutdown_connections 2024-04-25T02:01:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f242c0a2da0 0x7f242c0a5260 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2444107740 0x7f244419c770 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2444100600 0x7f24441983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 --2- 172.21.15.31:0/486539568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24440ffc40 0x7f2444197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 -- 172.21.15.31:0/486539568 >> 172.21.15.31:0/486539568 conn(0x7f2444076b80 msgr2=0x7f24440fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 -- 172.21.15.31:0/486539568 shutdown_connections 2024-04-25T02:01:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:00.989+0000 7f244979e640 1 -- 172.21.15.31:0/486539568 wait complete. 2024-04-25T02:01:01.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 146177 2.47 MB/sec execute 188 sec latency 1280.067 ms 2024-04-25T02:01:02.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 146177 2.45 MB/sec execute 189 sec latency 2280.190 ms 2024-04-25T02:01:02.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:02 smithi031 ceph-mon[29204]: pgmap v200: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-04-25T02:01:02.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:02 smithi040 ceph-mon[32418]: pgmap v200: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-04-25T02:01:02.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:02 smithi134 ceph-mon[33098]: pgmap v200: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-04-25T02:01:03.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 146452 2.45 MB/sec execute 190 sec latency 2684.926 ms 2024-04-25T02:01:04.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 146932 2.45 MB/sec execute 191 sec latency 19.332 ms 2024-04-25T02:01:04.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:04 smithi040 ceph-mon[32418]: pgmap v201: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:01:04.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:04 smithi134 ceph-mon[33098]: pgmap v201: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:01:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:04 smithi031 ceph-mon[29204]: pgmap v201: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:01:05.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 147420 2.44 MB/sec execute 192 sec latency 20.365 ms 2024-04-25T02:01:06.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 147928 2.44 MB/sec execute 193 sec latency 21.338 ms 2024-04-25T02:01:06.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:06 smithi040 ceph-mon[32418]: pgmap v202: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.6 MiB/s wr, 176 op/s 2024-04-25T02:01:06.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:06 smithi134 ceph-mon[33098]: pgmap v202: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.6 MiB/s wr, 176 op/s 2024-04-25T02:01:06.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:06 smithi031 ceph-mon[29204]: pgmap v202: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.6 MiB/s wr, 176 op/s 2024-04-25T02:01:07.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 148666 2.46 MB/sec execute 194 sec latency 17.516 ms 2024-04-25T02:01:08.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 149036 2.47 MB/sec execute 195 sec latency 18.850 ms 2024-04-25T02:01:08.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:08 smithi031 ceph-mon[29204]: pgmap v203: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1021 KiB/s wr, 113 op/s 2024-04-25T02:01:08.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:08 smithi031 ceph-mon[29204]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T02:01:08.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:08 smithi040 ceph-mon[32418]: pgmap v203: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1021 KiB/s wr, 113 op/s 2024-04-25T02:01:08.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:08 smithi040 ceph-mon[32418]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T02:01:08.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:08 smithi134 ceph-mon[33098]: pgmap v203: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1021 KiB/s wr, 113 op/s 2024-04-25T02:01:08.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:08 smithi134 ceph-mon[33098]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T02:01:09.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 149368 2.46 MB/sec execute 196 sec latency 42.797 ms 2024-04-25T02:01:10.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 149601 2.45 MB/sec execute 197 sec latency 60.426 ms 2024-04-25T02:01:10.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:10 smithi040 ceph-mon[32418]: pgmap v205: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.2 MiB/s wr, 135 op/s 2024-04-25T02:01:10.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:10 smithi134 ceph-mon[33098]: pgmap v205: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.2 MiB/s wr, 135 op/s 2024-04-25T02:01:10.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:10 smithi031 ceph-mon[29204]: pgmap v205: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.2 MiB/s wr, 135 op/s 2024-04-25T02:01:11.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 150091 2.45 MB/sec execute 198 sec latency 17.972 ms 2024-04-25T02:01:12.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 150559 2.45 MB/sec execute 199 sec latency 25.190 ms 2024-04-25T02:01:12.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:12 smithi040 ceph-mon[32418]: pgmap v206: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-25T02:01:12.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:12 smithi134 ceph-mon[33098]: pgmap v206: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-25T02:01:12.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:12 smithi031 ceph-mon[29204]: pgmap v206: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-25T02:01:13.311 INFO:tasks.workunit.client.0.smithi031.stdout: 1 151084 2.44 MB/sec execute 200 sec latency 25.578 ms 2024-04-25T02:01:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.088+0000 7f9a575ee640 1 -- 172.21.15.31:0/513373192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a50101750 msgr2=0x7f9a50101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.088+0000 7f9a575ee640 1 --2- 172.21.15.31:0/513373192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a50101750 0x7f9a50101bd0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009930 tx=0x7f9a4002a420 comp rx=0 tx=0).stop 2024-04-25T02:01:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.088+0000 7f9a575ee640 1 -- 172.21.15.31:0/513373192 shutdown_connections 2024-04-25T02:01:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.088+0000 7f9a575ee640 1 --2- 172.21.15.31:0/513373192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a50102110 0x7f9a5010e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.088+0000 7f9a575ee640 1 --2- 172.21.15.31:0/513373192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a50101750 0x7f9a50101bd0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.088+0000 7f9a575ee640 1 --2- 172.21.15.31:0/513373192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a50107760 0x7f9a50107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.088+0000 7f9a575ee640 1 -- 172.21.15.31:0/513373192 >> 172.21.15.31:0/513373192 conn(0x7f9a50078bd0 msgr2=0x7f9a50078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.088+0000 7f9a575ee640 1 -- 172.21.15.31:0/513373192 shutdown_connections 2024-04-25T02:01:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a575ee640 1 -- 172.21.15.31:0/513373192 wait complete. 2024-04-25T02:01:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a575ee640 1 Processor -- start 2024-04-25T02:01:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a575ee640 1 -- start start 2024-04-25T02:01:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a575ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a50101750 0x7f9a50197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a575ee640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a50102110 0x7f9a501983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a575ee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a50107760 0x7f9a5019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a575ee640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9a50111a20 con 0x7f9a50107760 2024-04-25T02:01:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a575ee640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9a501118a0 con 0x7f9a50101750 2024-04-25T02:01:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a575ee640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9a50111ba0 con 0x7f9a50102110 2024-04-25T02:01:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a55b8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a50107760 0x7f9a5019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a5538a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a50101750 0x7f9a50197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.089+0000 7f9a54b89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a50102110 0x7f9a501983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a55b8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a50107760 0x7f9a5019c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48628/0 (socket says 172.21.15.31:48628) 2024-04-25T02:01:14.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a5538a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a50101750 0x7f9a50197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55626/0 (socket says 172.21.15.31:55626) 2024-04-25T02:01:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a55b8b640 1 -- 172.21.15.31:0/3369354917 learned_addr learned my addr 172.21.15.31:0/3369354917 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a54b89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a50102110 0x7f9a501983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:39938/0 (socket says 172.21.15.31:39938) 2024-04-25T02:01:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a55b8b640 1 -- 172.21.15.31:0/3369354917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a50102110 msgr2=0x7f9a501983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a55b8b640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a50102110 0x7f9a501983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a55b8b640 1 -- 172.21.15.31:0/3369354917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a50101750 msgr2=0x7f9a50197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a55b8b640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a50101750 0x7f9a50197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a55b8b640 1 -- 172.21.15.31:0/3369354917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5019ce70 con 0x7f9a50107760 2024-04-25T02:01:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a55b8b640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a50107760 0x7f9a5019c740 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800ea60 tx=0x7f9a4800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a467fc640 1 -- 172.21.15.31:0/3369354917 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4800ce40 con 0x7f9a50107760 2024-04-25T02:01:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a575ee640 1 -- 172.21.15.31:0/3369354917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a5019d160 con 0x7f9a50107760 2024-04-25T02:01:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a467fc640 1 -- 172.21.15.31:0/3369354917 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a48005020 con 0x7f9a50107760 2024-04-25T02:01:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a575ee640 1 -- 172.21.15.31:0/3369354917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a501a49f0 con 0x7f9a50107760 2024-04-25T02:01:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.090+0000 7f9a467fc640 1 -- 172.21.15.31:0/3369354917 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a48010670 con 0x7f9a50107760 2024-04-25T02:01:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.093+0000 7f9a467fc640 1 -- 172.21.15.31:0/3369354917 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9a480051d0 con 0x7f9a50107760 2024-04-25T02:01:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.093+0000 7f9a5a3dc740 1 Processor -- start 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.093+0000 7f9a5a3dc740 1 -- start start 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.093+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfb1240 0x55822cfb1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.093+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1b60 0x55822cfbbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 0x55822cfbe620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55822cf15cb0 con 0x55822cfbc230 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55822cebe5b0 con 0x55822cfb1240 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55822cf11c70 con 0x55822cfb1b60 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a54b89640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfb1240 0x55822cfb1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a467fc640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9a3c0a2da0 0x7f9a3c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a54b89640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfb1240 0x55822cfb1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55628/0 (socket says 172.21.15.31:55628) 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a54b89640 1 -- 172.21.15.31:0/839420778 learned_addr learned my addr 172.21.15.31:0/839420778 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a54b89640 1 --2- 172.21.15.31:0/839420778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 0x55822cfbe620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a5538a640 1 --2- 172.21.15.31:0/839420778 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1b60 0x55822cfbbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a55b8b640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9a3c0a2da0 0x7f9a3c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a467fc640 1 -- 172.21.15.31:0/3369354917 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f9a480cf910 con 0x7f9a50107760 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a5538a640 1 -- 172.21.15.31:0/839420778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfb1240 msgr2=0x55822cfb1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a5538a640 1 --2- 172.21.15.31:0/839420778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfb1240 0x55822cfb1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a5538a640 1 -- 172.21.15.31:0/839420778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 msgr2=0x55822cfbe620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a5538a640 1 --2- 172.21.15.31:0/839420778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 0x55822cfbe620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a5538a640 1 -- 172.21.15.31:0/839420778 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55822cc2a210 con 0x55822cfb1b60 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a5538a640 1 --2- 172.21.15.31:0/839420778 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1b60 0x55822cfbbcf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0045b0 tx=0x7f9a4c00bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=378dddaea7735ddf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.094+0000 7f9a55b8b640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9a3c0a2da0 0x7f9a3c0a5260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9a4806ee90 tx=0x7f9a4806de10 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a3b7fe640 1 -- 172.21.15.31:0/839420778 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c003bd0 con 0x55822cfb1b60 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a3b7fe640 1 -- 172.21.15.31:0/839420778 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a4c015070 con 0x55822cfb1b60 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a3b7fe640 1 -- 172.21.15.31:0/839420778 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c004c30 con 0x55822cfb1b60 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/839420778 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1b60 msgr2=0x55822cfbbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/839420778 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1b60 0x55822cfbbcf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0045b0 tx=0x7f9a4c00bb20 comp rx=0 tx=0).stop 2024-04-25T02:01:14.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/839420778 shutdown_connections 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/839420778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 0x55822cfbe620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/839420778 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1b60 0x55822cfbbcf0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/839420778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfb1240 0x55822cfb1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/839420778 >> 172.21.15.31:0/839420778 conn(0x55822cfad560 msgr2=0x55822cfad940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/839420778 shutdown_connections 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.095+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/839420778 wait complete. 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a5a3dc740 1 -- start start 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a5a3dc740 1 Processor -- start 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 0x55822cfc6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 0x55822cfc6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfbc230 0x55822cfc0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55822cfbf020 con 0x55822cfb1b60 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55822ced1b50 con 0x55822cfbc230 2024-04-25T02:01:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55822cfbf1a0 con 0x55822cfb1240 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a5a3dc740 10 client.0 _ll_register_callbacks cb 0x55822cf0cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a54b89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 0x55822cfc6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a5538a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 0x55822cfc6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a54b89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 0x55822cfc6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:39960/0 (socket says 172.21.15.31:39960) 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a54b89640 1 -- 172.21.15.31:0/3998275765 learned_addr learned my addr 172.21.15.31:0/3998275765 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.096+0000 7f9a54b89640 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfbc230 0x55822cfc0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a54b89640 1 -- 172.21.15.31:0/3998275765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfbc230 msgr2=0x55822cfc0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a54b89640 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfbc230 0x55822cfc0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a54b89640 1 -- 172.21.15.31:0/3998275765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 msgr2=0x55822cfc6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a54b89640 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 0x55822cfc6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a54b89640 1 -- 172.21.15.31:0/3998275765 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55822ccf6620 con 0x55822cfb1240 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a54b89640 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 0x55822cfc6430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9a40005940 tx=0x7f9a40009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3ea69e1d5f3b502 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3998275765 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40035040 con 0x55822cfb1240 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3998275765 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f9a40004480 con 0x55822cfb1240 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3998275765 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a4c015070 con 0x55822cfb1240 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3998275765 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9a40005020 con 0x55822cfb1240 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a5a3dc740 10 client.26119 fetch_fsmap learned FSMap version 33 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3998275765 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55822cfcf6b0 con 0x55822cfb1240 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.097+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3998275765 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a4c015070 con 0x55822cfb1240 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.098+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3998275765 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f9a400035f0 con 0x55822cfb1240 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.098+0000 7f9a5a3dc740 10 client.26119 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.098+0000 7f9a5a3dc740 10 client.26119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.098+0000 7f9a5a3dc740 20 client.26119 populate_metadata read hostname 'smithi031' 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.098+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55822cfd0b70 0x55822cfd2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.098+0000 7f9a5a3dc740 4 client.26119 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.098+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3998275765 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55822cf231f0 con 0x55822cfd0b70 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.098+0000 7f9a5538a640 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55822cfd0b70 0x55822cfd2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.098+0000 7f9a5538a640 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55822cfd0b70 0x55822cfd2f60 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3e71391bfd32a86 server_cookie=a4b5558f67359e79 in_seq=0 out_seq=0 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.098+0000 7f9a38ff9640 10 client.26119 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3998275765 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55822cf231f0 con 0x55822cfd0b70 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a38ff9640 10 client.26119 handle_command_reply: tid=0 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a5a3dc740 1 client.26119 shutdown 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3998275765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 msgr2=0x55822cfc6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 0x55822cfc6430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9a40005940 tx=0x7f9a40009260 comp rx=0 tx=0).stop 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3998275765 shutdown_connections 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55822cfd0b70 0x55822cfd2f60 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfbc230 0x55822cfc0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a38ff9640 0 client.26119 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:14.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 0x55822cfc6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3998275765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 0x55822cfc6430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.099+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3998275765 >> 172.21.15.31:0/3998275765 conn(0x55822cfad560 msgr2=0x55822cfb3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:14.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.100+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3998275765 shutdown_connections 2024-04-25T02:01:14.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.100+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3998275765 wait complete. 2024-04-25T02:01:14.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.100+0000 7f9a5a3dc740 20 client.26119 trim_cache size 0 max 16384 2024-04-25T02:01:14.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a5a3dc740 1 Processor -- start 2024-04-25T02:01:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a5a3dc740 1 -- start start 2024-04-25T02:01:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfb1240 0x55822cfc6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1b60 0x55822cfd87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 0x55822cfd8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9a501ab0b0 con 0x55822cfbc230 2024-04-25T02:01:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9a501acbf0 con 0x55822cfb1240 2024-04-25T02:01:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9a501ab780 con 0x55822cfb1b60 2024-04-25T02:01:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a54b89640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 0x55822cfd8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a54b89640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 0x55822cfd8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48656/0 (socket says 172.21.15.31:48656) 2024-04-25T02:01:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a54b89640 1 -- 172.21.15.31:0/2151149079 learned_addr learned my addr 172.21.15.31:0/2151149079 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a54b89640 1 --2- 172.21.15.31:0/2151149079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfb1240 0x55822cfc6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a5538a640 1 --2- 172.21.15.31:0/2151149079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1b60 0x55822cfd87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a54b89640 1 -- 172.21.15.31:0/2151149079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1b60 msgr2=0x55822cfd87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a54b89640 1 --2- 172.21.15.31:0/2151149079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1b60 0x55822cfd87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a54b89640 1 -- 172.21.15.31:0/2151149079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfb1240 msgr2=0x55822cfc6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.117+0000 7f9a54b89640 1 --2- 172.21.15.31:0/2151149079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfb1240 0x55822cfc6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.118+0000 7f9a54b89640 1 -- 172.21.15.31:0/2151149079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55822cfb1990 con 0x55822cfbc230 2024-04-25T02:01:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.118+0000 7f9a54b89640 1 --2- 172.21.15.31:0/2151149079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 0x55822cfd8d20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9a40031800 tx=0x7f9a4002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfc443cb7d9bfad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.118+0000 7f9a3affd640 1 -- 172.21.15.31:0/2151149079 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40058070 con 0x55822cfbc230 2024-04-25T02:01:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.118+0000 7f9a3affd640 1 -- 172.21.15.31:0/2151149079 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a40059040 con 0x55822cfbc230 2024-04-25T02:01:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.118+0000 7f9a3affd640 1 -- 172.21.15.31:0/2151149079 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a40035040 con 0x55822cfbc230 2024-04-25T02:01:14.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.119+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/2151149079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 msgr2=0x55822cfd8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.119+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/2151149079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 0x55822cfd8d20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9a40031800 tx=0x7f9a4002a420 comp rx=0 tx=0).stop 2024-04-25T02:01:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.119+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/2151149079 shutdown_connections 2024-04-25T02:01:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.119+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/2151149079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfbc230 0x55822cfd8d20 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.119+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/2151149079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1b60 0x55822cfd87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.119+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/2151149079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfb1240 0x55822cfc6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.119+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/2151149079 >> 172.21.15.31:0/2151149079 conn(0x55822cfad560 msgr2=0x55822cfc5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.119+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/2151149079 shutdown_connections 2024-04-25T02:01:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.119+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/2151149079 wait complete. 2024-04-25T02:01:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.119+0000 7f9a5a3dc740 1 -- start start 2024-04-25T02:01:14.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.119+0000 7f9a5a3dc740 1 Processor -- start 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 0x55822cfc9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 0x55822cfca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5a3dc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfbc230 0x55822d035050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9a501ac520 con 0x55822cfb1b60 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9a501abe50 con 0x55822cfbc230 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5a3dc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9a501a90b0 con 0x55822cfb1240 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5a3dc740 10 client.0 _ll_register_callbacks cb 0x7f9a40005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5538a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 0x55822cfca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a54b89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 0x55822cfc9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a54b89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 0x55822cfc9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:39974/0 (socket says 172.21.15.31:39974) 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5538a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 0x55822cfca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48668/0 (socket says 172.21.15.31:48668) 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a54b89640 1 -- 172.21.15.31:0/3041123205 learned_addr learned my addr 172.21.15.31:0/3041123205 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:14.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a54b89640 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfbc230 0x55822d035050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5538a640 1 -- 172.21.15.31:0/3041123205 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 msgr2=0x55822cfc9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5538a640 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 0x55822cfc9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5538a640 1 -- 172.21.15.31:0/3041123205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfbc230 msgr2=0x55822d035050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5538a640 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfbc230 0x55822d035050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5538a640 1 -- 172.21.15.31:0/3041123205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55822d0355c0 con 0x55822cfb1b60 2024-04-25T02:01:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a5538a640 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 0x55822cfca080 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c004910 tx=0x7f9a4c00bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102aed5ca3df4479 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:01:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.120+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3041123205 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c0280e0 con 0x55822cfb1b60 2024-04-25T02:01:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3041123205 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f9a4c028280 con 0x55822cfb1b60 2024-04-25T02:01:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3041123205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a40059040 con 0x55822cfb1b60 2024-04-25T02:01:14.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3041123205 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9a4c037050 con 0x55822cfb1b60 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a5a3dc740 10 client.16698 fetch_fsmap learned FSMap version 33 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3041123205 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55822d0368d0 con 0x55822cfb1b60 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3041123205 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a40059040 con 0x55822cfb1b60 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3041123205 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f9a4c011900 con 0x55822cfb1b60 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a5a3dc740 10 client.16698 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a5a3dc740 10 client.16698 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a5a3dc740 20 client.16698 populate_metadata read hostname 'smithi031' 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55822d036e70 0x55822d039290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a5a3dc740 4 client.16698 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3041123205 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55822cf1d540 con 0x55822d036e70 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.121+0000 7f9a54b89640 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55822d036e70 0x55822d039290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.122+0000 7f9a54b89640 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55822d036e70 0x55822d039290 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=161bfa8db54d6f61 server_cookie=fbbda76dd1838428 in_seq=0 out_seq=0 2024-04-25T02:01:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.122+0000 7f9a38ff9640 10 client.16698 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:14.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a38ff9640 1 -- 172.21.15.31:0/3041123205 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55822cf1d540 con 0x55822d036e70 2024-04-25T02:01:14.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a38ff9640 10 client.16698 handle_command_reply: tid=0 2024-04-25T02:01:14.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a5a3dc740 1 client.16698 shutdown 2024-04-25T02:01:14.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3041123205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 msgr2=0x55822cfca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:14.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 0x55822cfca080 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c004910 tx=0x7f9a4c00bef0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3041123205 shutdown_connections 2024-04-25T02:01:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55822d036e70 0x55822d039290 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55822cfbc230 0x55822d035050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a38ff9640 0 client.16698 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55822cfb1b60 0x55822cfca080 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a5a3dc740 1 --2- 172.21.15.31:0/3041123205 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55822cfb1240 0x55822cfc9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3041123205 >> 172.21.15.31:0/3041123205 conn(0x55822cfad560 msgr2=0x55822cfabd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3041123205 shutdown_connections 2024-04-25T02:01:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.147+0000 7f9a5a3dc740 1 -- 172.21.15.31:0/3041123205 wait complete. 2024-04-25T02:01:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a5a3dc740 20 client.16698 trim_cache size 0 max 16384 2024-04-25T02:01:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:01:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:01:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:01:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:01:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 -- 172.21.15.31:0/3369354917 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9a3c0a2da0 msgr2=0x7f9a3c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9a3c0a2da0 0x7f9a3c0a5260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9a4806ee90 tx=0x7f9a4806de10 comp rx=0 tx=0).stop 2024-04-25T02:01:14.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 -- 172.21.15.31:0/3369354917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a50107760 msgr2=0x7f9a5019c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:14.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a50107760 0x7f9a5019c740 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800ea60 tx=0x7f9a4800ef20 comp rx=0 tx=0).stop 2024-04-25T02:01:14.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 -- 172.21.15.31:0/3369354917 shutdown_connections 2024-04-25T02:01:14.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9a3c0a2da0 0x7f9a3c0a5260 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a50107760 0x7f9a5019c740 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9a50102110 0x7f9a501983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 --2- 172.21.15.31:0/3369354917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a50101750 0x7f9a50197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:14.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 -- 172.21.15.31:0/3369354917 >> 172.21.15.31:0/3369354917 conn(0x7f9a50078bd0 msgr2=0x7f9a500ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:14.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 -- 172.21.15.31:0/3369354917 shutdown_connections 2024-04-25T02:01:14.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:14.148+0000 7f9a575ee640 1 -- 172.21.15.31:0/3369354917 wait complete. 2024-04-25T02:01:14.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 151566 2.44 MB/sec execute 201 sec latency 24.610 ms 2024-04-25T02:01:14.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:14 smithi040 ceph-mon[32418]: pgmap v207: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-25T02:01:14.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:14 smithi040 ceph-mon[32418]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T02:01:14.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:14 smithi134 ceph-mon[33098]: pgmap v207: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-25T02:01:14.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:14 smithi134 ceph-mon[33098]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T02:01:14.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:14 smithi031 ceph-mon[29204]: pgmap v207: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-25T02:01:14.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:14 smithi031 ceph-mon[29204]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T02:01:15.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 152283 2.46 MB/sec execute 202 sec latency 19.383 ms 2024-04-25T02:01:16.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 152666 2.47 MB/sec execute 203 sec latency 13.585 ms 2024-04-25T02:01:16.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:16 smithi040 ceph-mon[32418]: pgmap v209: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-04-25T02:01:16.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:16 smithi134 ceph-mon[33098]: pgmap v209: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-04-25T02:01:16.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:16 smithi031 ceph-mon[29204]: pgmap v209: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-04-25T02:01:17.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 152949 2.46 MB/sec execute 204 sec latency 12.996 ms 2024-04-25T02:01:18.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 153245 2.45 MB/sec execute 205 sec latency 18.573 ms 2024-04-25T02:01:18.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:18 smithi040 ceph-mon[32418]: pgmap v210: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-04-25T02:01:18.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:18 smithi134 ceph-mon[33098]: pgmap v210: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-04-25T02:01:18.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:18 smithi031 ceph-mon[29204]: pgmap v210: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.6 MiB/s wr, 146 op/s 2024-04-25T02:01:19.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 153734 2.45 MB/sec execute 206 sec latency 18.229 ms 2024-04-25T02:01:20.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 154242 2.45 MB/sec execute 207 sec latency 23.630 ms 2024-04-25T02:01:20.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:20 smithi040 ceph-mon[32418]: pgmap v211: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-04-25T02:01:20.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:20 smithi134 ceph-mon[33098]: pgmap v211: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-04-25T02:01:20.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:20 smithi031 ceph-mon[29204]: pgmap v211: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-04-25T02:01:21.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 154750 2.44 MB/sec execute 208 sec latency 25.843 ms 2024-04-25T02:01:22.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 155295 2.45 MB/sec execute 209 sec latency 18.711 ms 2024-04-25T02:01:22.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:22 smithi040 ceph-mon[32418]: pgmap v212: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-04-25T02:01:22.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:22 smithi134 ceph-mon[33098]: pgmap v212: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-04-25T02:01:22.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:22 smithi031 ceph-mon[29204]: pgmap v212: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-04-25T02:01:23.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 155865 2.46 MB/sec execute 210 sec latency 18.138 ms 2024-04-25T02:01:24.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 156319 2.47 MB/sec execute 211 sec latency 19.181 ms 2024-04-25T02:01:24.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:24 smithi040 ceph-mon[32418]: pgmap v213: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-04-25T02:01:24.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:24 smithi134 ceph-mon[33098]: pgmap v213: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-04-25T02:01:24.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:24 smithi031 ceph-mon[29204]: pgmap v213: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.5 MiB/s wr, 146 op/s 2024-04-25T02:01:25.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 156550 2.46 MB/sec execute 212 sec latency 18.953 ms 2024-04-25T02:01:26.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 156921 2.45 MB/sec execute 213 sec latency 21.028 ms 2024-04-25T02:01:26.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:26 smithi040 ceph-mon[32418]: pgmap v214: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.1 MiB/s wr, 196 op/s 2024-04-25T02:01:26.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:26 smithi134 ceph-mon[33098]: pgmap v214: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.1 MiB/s wr, 196 op/s 2024-04-25T02:01:26.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:26 smithi031 ceph-mon[29204]: pgmap v214: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.1 MiB/s wr, 196 op/s 2024-04-25T02:01:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.248+0000 7f41d2695640 1 -- 172.21.15.31:0/489745033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41cc106630 msgr2=0x7f41cc106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.248+0000 7f41d2695640 1 --2- 172.21.15.31:0/489745033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41cc106630 0x7f41cc106a10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009930 tx=0x7f41bc02a420 comp rx=0 tx=0).stop 2024-04-25T02:01:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.248+0000 7f41d2695640 1 -- 172.21.15.31:0/489745033 shutdown_connections 2024-04-25T02:01:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.248+0000 7f41d2695640 1 --2- 172.21.15.31:0/489745033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41cc101030 0x7f41cc10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.248+0000 7f41d2695640 1 --2- 172.21.15.31:0/489745033 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41cc100670 0x7f41cc100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.248+0000 7f41d2695640 1 --2- 172.21.15.31:0/489745033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41cc106630 0x7f41cc106a10 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.248+0000 7f41d2695640 1 -- 172.21.15.31:0/489745033 >> 172.21.15.31:0/489745033 conn(0x7f41cc0777c0 msgr2=0x7f41cc077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.248+0000 7f41d2695640 1 -- 172.21.15.31:0/489745033 shutdown_connections 2024-04-25T02:01:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.248+0000 7f41d2695640 1 -- 172.21.15.31:0/489745033 wait complete. 2024-04-25T02:01:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.249+0000 7f41d2695640 1 Processor -- start 2024-04-25T02:01:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.249+0000 7f41d2695640 1 -- start start 2024-04-25T02:01:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.249+0000 7f41d2695640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41cc100670 0x7f41cc19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.249+0000 7f41d2695640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41cc101030 0x7f41cc19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.249+0000 7f41d2695640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41cc106630 0x7f41cc19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.249+0000 7f41d2695640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc110900 con 0x7f41cc106630 2024-04-25T02:01:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.249+0000 7f41d2695640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc110780 con 0x7f41cc100670 2024-04-25T02:01:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.249+0000 7f41d2695640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc110a80 con 0x7f41cc101030 2024-04-25T02:01:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41d0c32640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41cc106630 0x7f41cc19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41d0c32640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41cc106630 0x7f41cc19ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55494/0 (socket says 172.21.15.31:55494) 2024-04-25T02:01:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41d0c32640 1 -- 172.21.15.31:0/304349192 learned_addr learned my addr 172.21.15.31:0/304349192 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41cbfff640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41cc100670 0x7f41cc19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41cc101030 0x7f41cc19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41cbfff640 1 -- 172.21.15.31:0/304349192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41cc101030 msgr2=0x7f41cc19a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41cbfff640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41cc101030 0x7f41cc19a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41cbfff640 1 -- 172.21.15.31:0/304349192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41cc106630 msgr2=0x7f41cc19ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41cbfff640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41cc106630 0x7f41cc19ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41cbfff640 1 -- 172.21.15.31:0/304349192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc19f140 con 0x7f41cc100670 2024-04-25T02:01:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41cc101030 0x7f41cc19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.250+0000 7f41d0c32640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41cc106630 0x7f41cc19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.251+0000 7f41cbfff640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41cc100670 0x7f41cc19a150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f41bc02a940 tx=0x7f41bc003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.251+0000 7f41c97fa640 1 -- 172.21.15.31:0/304349192 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc03f070 con 0x7f41cc100670 2024-04-25T02:01:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.251+0000 7f41d2695640 1 -- 172.21.15.31:0/304349192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41cc19f3d0 con 0x7f41cc100670 2024-04-25T02:01:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.251+0000 7f41c97fa640 1 -- 172.21.15.31:0/304349192 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41bc031b60 con 0x7f41cc100670 2024-04-25T02:01:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.251+0000 7f41c97fa640 1 -- 172.21.15.31:0/304349192 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc035030 con 0x7f41cc100670 2024-04-25T02:01:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.251+0000 7f41d2695640 1 -- 172.21.15.31:0/304349192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41cc1a6cc0 con 0x7f41cc100670 2024-04-25T02:01:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.253+0000 7f41c97fa640 1 -- 172.21.15.31:0/304349192 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f41bc042070 con 0x7f41cc100670 2024-04-25T02:01:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.253+0000 7f41d539b740 1 Processor -- start 2024-04-25T02:01:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.253+0000 7f41d539b740 1 -- start start 2024-04-25T02:01:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41d539b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 0x565187d01620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41d539b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d0bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41d539b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d0e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41d539b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x565187c65cb0 con 0x565187d01240 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41d539b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565187c0e5b0 con 0x565187d01b60 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41d539b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x565187c61c70 con 0x565187d0c230 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41c97fa640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b40a2da0 0x7f41b40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41d0c32640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 0x565187d01620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41d0c32640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 0x565187d01620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55504/0 (socket says 172.21.15.31:55504) 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41cbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d0bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41d0c32640 1 -- 172.21.15.31:0/1671180485 learned_addr learned my addr 172.21.15.31:0/1671180485 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/1671180485 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d0e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41d0c32640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b40a2da0 0x7f41b40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41c97fa640 1 -- 172.21.15.31:0/304349192 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f41bc0efa00 con 0x7f41cc100670 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41cb7fe640 1 -- 172.21.15.31:0/1671180485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 msgr2=0x565187d0bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/1671180485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d0bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41cb7fe640 1 -- 172.21.15.31:0/1671180485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 msgr2=0x565187d01620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/1671180485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 0x565187d01620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41cb7fe640 1 -- 172.21.15.31:0/1671180485 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56518797a210 con 0x565187d0c230 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/1671180485 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d0e620 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009fa0 tx=0x7f41c00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdbeb0ef6dac838d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41d0c32640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b40a2da0 0x7f41b40a5260 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f41c4007fd0 tx=0x7f41c40075e0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.254+0000 7f41b27fc640 1 -- 172.21.15.31:0/1671180485 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c0004030 con 0x565187d0c230 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.255+0000 7f41b27fc640 1 -- 172.21.15.31:0/1671180485 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41c00054d0 con 0x565187d0c230 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.255+0000 7f41b27fc640 1 -- 172.21.15.31:0/1671180485 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c0004c20 con 0x565187d0c230 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.255+0000 7f41d539b740 1 -- 172.21.15.31:0/1671180485 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 msgr2=0x565187d0e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.255+0000 7f41d539b740 1 --2- 172.21.15.31:0/1671180485 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d0e620 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009fa0 tx=0x7f41c00118d0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.255+0000 7f41d539b740 1 -- 172.21.15.31:0/1671180485 shutdown_connections 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.255+0000 7f41d539b740 1 --2- 172.21.15.31:0/1671180485 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d0e620 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.255+0000 7f41d539b740 1 --2- 172.21.15.31:0/1671180485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d0bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.255+0000 7f41d539b740 1 --2- 172.21.15.31:0/1671180485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 0x565187d01620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.255+0000 7f41d539b740 1 -- 172.21.15.31:0/1671180485 >> 172.21.15.31:0/1671180485 conn(0x565187cfd560 msgr2=0x565187cfd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.255+0000 7f41d539b740 1 -- 172.21.15.31:0/1671180485 shutdown_connections 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.255+0000 7f41d539b740 1 -- 172.21.15.31:0/1671180485 wait complete. 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.256+0000 7f41d539b740 1 -- start start 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.256+0000 7f41d539b740 1 Processor -- start 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.256+0000 7f41d539b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 0x565187d09140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.256+0000 7f41d539b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d09680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.256+0000 7f41d539b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d09bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.256+0000 7f41d539b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x565187d0f020 con 0x565187d01240 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.256+0000 7f41d539b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565187c21b50 con 0x565187d01b60 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.256+0000 7f41d539b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x565187d0f1a0 con 0x565187d0c230 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.256+0000 7f41d539b740 10 client.0 _ll_register_callbacks cb 0x565187c5cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41cbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 0x565187d09140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41cb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d09680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41cbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 0x565187d09140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55510/0 (socket says 172.21.15.31:55510) 2024-04-25T02:01:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41cbfff640 1 -- 172.21.15.31:0/651259324 learned_addr learned my addr 172.21.15.31:0/651259324 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41d0c32640 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d09bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41cb7fe640 1 -- 172.21.15.31:0/651259324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 msgr2=0x565187d09bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d09bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41cb7fe640 1 -- 172.21.15.31:0/651259324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 msgr2=0x565187d09140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 0x565187d09140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41cb7fe640 1 -- 172.21.15.31:0/651259324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565187a46620 con 0x565187d01b60 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41cbfff640 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 0x565187d09140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d09680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009fa0 tx=0x7f41c000fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90ea792e5fcc3322 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41ab7fe640 1 -- 172.21.15.31:0/651259324 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c0028020 con 0x565187d01b60 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41d539b740 1 -- 172.21.15.31:0/651259324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41c00054d0 con 0x565187d01b60 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41ab7fe640 1 -- 172.21.15.31:0/651259324 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f41c0024480 con 0x565187d01b60 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.257+0000 7f41ab7fe640 1 -- 172.21.15.31:0/651259324 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f41c0033050 con 0x565187d01b60 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.258+0000 7f41d539b740 10 client.26306 fetch_fsmap learned FSMap version 33 2024-04-25T02:01:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.258+0000 7f41d539b740 1 -- 172.21.15.31:0/651259324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x565187d1b260 con 0x565187d01b60 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.258+0000 7f41ab7fe640 1 -- 172.21.15.31:0/651259324 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41c00054d0 con 0x565187d01b60 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.258+0000 7f41ab7fe640 1 -- 172.21.15.31:0/651259324 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f41c0004920 con 0x565187d01b60 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.258+0000 7f41d539b740 10 client.26306 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.258+0000 7f41d539b740 10 client.26306 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.258+0000 7f41d539b740 20 client.26306 populate_metadata read hostname 'smithi031' 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.258+0000 7f41d539b740 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565187d1c6d0 0x565187d1eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.258+0000 7f41d539b740 4 client.26306 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.258+0000 7f41d539b740 1 -- 172.21.15.31:0/651259324 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565187c731f0 con 0x565187d1c6d0 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.258+0000 7f41cbfff640 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565187d1c6d0 0x565187d1eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41ab7fe640 10 client.26306 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41cbfff640 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565187d1c6d0 0x565187d1eac0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d651c4914b3cb2e3 server_cookie=120c85961ffab386 in_seq=0 out_seq=0 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41ab7fe640 1 -- 172.21.15.31:0/651259324 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565187c731f0 con 0x565187d1c6d0 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41ab7fe640 10 client.26306 handle_command_reply: tid=0 2024-04-25T02:01:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41d539b740 1 client.26306 shutdown 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41d539b740 1 -- 172.21.15.31:0/651259324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 msgr2=0x565187d09680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41d539b740 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d09680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009fa0 tx=0x7f41c000fc00 comp rx=0 tx=0).stop 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41d539b740 1 -- 172.21.15.31:0/651259324 shutdown_connections 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41d539b740 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565187d1c6d0 0x565187d1eac0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41d539b740 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d09bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41ab7fe640 0 client.26306 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41d539b740 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d09680 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41d539b740 1 --2- 172.21.15.31:0/651259324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01240 0x565187d09140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.259+0000 7f41d539b740 1 -- 172.21.15.31:0/651259324 >> 172.21.15.31:0/651259324 conn(0x565187cfd560 msgr2=0x565187d03260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.260+0000 7f41d539b740 1 -- 172.21.15.31:0/651259324 shutdown_connections 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.260+0000 7f41d539b740 1 -- 172.21.15.31:0/651259324 wait complete. 2024-04-25T02:01:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.260+0000 7f41d539b740 20 client.26306 trim_cache size 0 max 16384 2024-04-25T02:01:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d539b740 1 Processor -- start 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d539b740 1 -- start start 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d539b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d01240 0x565187d0b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d539b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d17f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d539b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d0c230 0x565187d082a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d539b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc1af420 con 0x565187d0c230 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d539b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc1b0f60 con 0x565187d01b60 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d539b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc1afaf0 con 0x565187d01240 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d0c32640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d0c230 0x565187d082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41cb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d17f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d0c32640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d0c230 0x565187d082a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55518/0 (socket says 172.21.15.31:55518) 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d0c32640 1 -- 172.21.15.31:0/3065866874 learned_addr learned my addr 172.21.15.31:0/3065866874 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41cbfff640 1 --2- 172.21.15.31:0/3065866874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d01240 0x565187d0b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d0c32640 1 -- 172.21.15.31:0/3065866874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d01240 msgr2=0x565187d0b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.277+0000 7f41d0c32640 1 --2- 172.21.15.31:0/3065866874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d01240 0x565187d0b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.278+0000 7f41d0c32640 1 -- 172.21.15.31:0/3065866874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 msgr2=0x565187d17f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.278+0000 7f41d0c32640 1 --2- 172.21.15.31:0/3065866874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d17f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.278+0000 7f41d0c32640 1 -- 172.21.15.31:0/3065866874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565187d01990 con 0x565187d0c230 2024-04-25T02:01:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.278+0000 7f41d0c32640 1 --2- 172.21.15.31:0/3065866874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d0c230 0x565187d082a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f41c400f560 tx=0x7f41c4007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8af4b1c76583d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.278+0000 7f41b1ffb640 1 -- 172.21.15.31:0/3065866874 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4026070 con 0x565187d0c230 2024-04-25T02:01:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.278+0000 7f41b1ffb640 1 -- 172.21.15.31:0/3065866874 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41c4004d20 con 0x565187d0c230 2024-04-25T02:01:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.278+0000 7f41b1ffb640 1 -- 172.21.15.31:0/3065866874 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4005590 con 0x565187d0c230 2024-04-25T02:01:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.279+0000 7f41d539b740 1 -- 172.21.15.31:0/3065866874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d0c230 msgr2=0x565187d082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.279+0000 7f41d539b740 1 --2- 172.21.15.31:0/3065866874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d0c230 0x565187d082a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f41c400f560 tx=0x7f41c4007df0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.279+0000 7f41d539b740 1 -- 172.21.15.31:0/3065866874 shutdown_connections 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.279+0000 7f41d539b740 1 --2- 172.21.15.31:0/3065866874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d0c230 0x565187d082a0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.279+0000 7f41d539b740 1 --2- 172.21.15.31:0/3065866874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01b60 0x565187d17f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.279+0000 7f41d539b740 1 --2- 172.21.15.31:0/3065866874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d01240 0x565187d0b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.279+0000 7f41d539b740 1 -- 172.21.15.31:0/3065866874 >> 172.21.15.31:0/3065866874 conn(0x565187cfd560 msgr2=0x565187d029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.279+0000 7f41d539b740 1 -- 172.21.15.31:0/3065866874 shutdown_connections 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.279+0000 7f41d539b740 1 -- 172.21.15.31:0/3065866874 wait complete. 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.279+0000 7f41d539b740 1 -- start start 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.279+0000 7f41d539b740 1 Processor -- start 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41d539b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01240 0x565187d14640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41d539b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01b60 0x565187d057a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41d539b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d05ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41d539b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc1b0890 con 0x565187d01b60 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41d539b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc1b01c0 con 0x565187d01240 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41d539b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc1ad420 con 0x565187d0c230 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41d539b740 10 client.0 _ll_register_callbacks cb 0x7f41c4011700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01b60 0x565187d057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41cbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01240 0x565187d14640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01b60 0x565187d057a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55520/0 (socket says 172.21.15.31:55520) 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41cb7fe640 1 -- 172.21.15.31:0/2342919186 learned_addr learned my addr 172.21.15.31:0/2342919186 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41d0c32640 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d05ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41cb7fe640 1 -- 172.21.15.31:0/2342919186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 msgr2=0x565187d05ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d05ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41cb7fe640 1 -- 172.21.15.31:0/2342919186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01240 msgr2=0x565187d14640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01240 0x565187d14640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41cb7fe640 1 -- 172.21.15.31:0/2342919186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565187d06220 con 0x565187d01b60 2024-04-25T02:01:27.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41cb7fe640 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01b60 0x565187d057a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x565187d089a0 tx=0x7f41c000f8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70cfd1d8f4b1293 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:27.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:01:27.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41ab7fe640 1 -- 172.21.15.31:0/2342919186 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c003d060 con 0x565187d01b60 2024-04-25T02:01:27.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41d539b740 1 -- 172.21.15.31:0/2342919186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41c4004d20 con 0x565187d01b60 2024-04-25T02:01:27.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.280+0000 7f41ab7fe640 1 -- 172.21.15.31:0/2342919186 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f41c00048b0 con 0x565187d01b60 2024-04-25T02:01:27.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41ab7fe640 1 -- 172.21.15.31:0/2342919186 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f41c0028040 con 0x565187d01b60 2024-04-25T02:01:27.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41d539b740 10 client.16716 fetch_fsmap learned FSMap version 33 2024-04-25T02:01:27.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41d539b740 1 -- 172.21.15.31:0/2342919186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x565187d827d0 con 0x565187d01b60 2024-04-25T02:01:27.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41ab7fe640 1 -- 172.21.15.31:0/2342919186 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41c4004d20 con 0x565187d01b60 2024-04-25T02:01:27.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41ab7fe640 1 -- 172.21.15.31:0/2342919186 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f41c0024720 con 0x565187d01b60 2024-04-25T02:01:27.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41d539b740 10 client.16716 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:01:27.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41d539b740 10 client.16716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:27.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41d539b740 20 client.16716 populate_metadata read hostname 'smithi031' 2024-04-25T02:01:27.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41d539b740 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565187d82f80 0x565187d853a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:27.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41d539b740 4 client.16716 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:01:27.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41d539b740 1 -- 172.21.15.31:0/2342919186 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565187c6d540 con 0x565187d82f80 2024-04-25T02:01:27.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.281+0000 7f41cbfff640 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565187d82f80 0x565187d853a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:27.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.282+0000 7f41cbfff640 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565187d82f80 0x565187d853a0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15efb356800f72a8 server_cookie=f297250277a4c532 in_seq=0 out_seq=0 2024-04-25T02:01:27.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.282+0000 7f41ab7fe640 10 client.16716 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:27.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 157381 2.45 MB/sec execute 214 sec latency 18.394 ms 2024-04-25T02:01:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41ab7fe640 1 -- 172.21.15.31:0/2342919186 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565187c6d540 con 0x565187d82f80 2024-04-25T02:01:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41ab7fe640 10 client.16716 handle_command_reply: tid=0 2024-04-25T02:01:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 1 client.16716 shutdown 2024-04-25T02:01:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 1 -- 172.21.15.31:0/2342919186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01b60 msgr2=0x565187d057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01b60 0x565187d057a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x565187d089a0 tx=0x7f41c000f8b0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 1 -- 172.21.15.31:0/2342919186 shutdown_connections 2024-04-25T02:01:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565187d82f80 0x565187d853a0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565187d0c230 0x565187d05ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565187d01b60 0x565187d057a0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 1 --2- 172.21.15.31:0/2342919186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565187d01240 0x565187d14640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41ab7fe640 0 client.16716 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 1 -- 172.21.15.31:0/2342919186 >> 172.21.15.31:0/2342919186 conn(0x565187cfd560 msgr2=0x565187d03220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 1 -- 172.21.15.31:0/2342919186 shutdown_connections 2024-04-25T02:01:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 1 -- 172.21.15.31:0/2342919186 wait complete. 2024-04-25T02:01:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.310+0000 7f41d539b740 20 client.16716 trim_cache size 0 max 16384 2024-04-25T02:01:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:01:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:01:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:01:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:01:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 -- 172.21.15.31:0/304349192 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b40a2da0 msgr2=0x7f41b40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b40a2da0 0x7f41b40a5260 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f41c4007fd0 tx=0x7f41c40075e0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 -- 172.21.15.31:0/304349192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41cc100670 msgr2=0x7f41cc19a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41cc100670 0x7f41cc19a150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f41bc02a940 tx=0x7f41bc003c10 comp rx=0 tx=0).stop 2024-04-25T02:01:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 -- 172.21.15.31:0/304349192 shutdown_connections 2024-04-25T02:01:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b40a2da0 0x7f41b40a5260 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41cc106630 0x7f41cc19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41cc101030 0x7f41cc19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 --2- 172.21.15.31:0/304349192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41cc100670 0x7f41cc19a150 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 -- 172.21.15.31:0/304349192 >> 172.21.15.31:0/304349192 conn(0x7f41cc0777c0 msgr2=0x7f41cc068df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 -- 172.21.15.31:0/304349192 shutdown_connections 2024-04-25T02:01:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:27.311+0000 7f41d2695640 1 -- 172.21.15.31:0/304349192 wait complete. 2024-04-25T02:01:28.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 157871 2.45 MB/sec execute 215 sec latency 25.614 ms 2024-04-25T02:01:28.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:28 smithi040 ceph-mon[32418]: pgmap v215: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-04-25T02:01:28.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:28 smithi134 ceph-mon[33098]: pgmap v215: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-04-25T02:01:28.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:28 smithi031 ceph-mon[29204]: pgmap v215: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-04-25T02:01:29.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 158429 2.44 MB/sec execute 216 sec latency 22.872 ms 2024-04-25T02:01:30.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 158988 2.46 MB/sec execute 217 sec latency 18.651 ms 2024-04-25T02:01:30.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:30 smithi040 ceph-mon[32418]: pgmap v216: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-04-25T02:01:30.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:30 smithi134 ceph-mon[33098]: pgmap v216: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-04-25T02:01:30.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:30 smithi031 ceph-mon[29204]: pgmap v216: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-04-25T02:01:31.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 159449 2.46 MB/sec execute 218 sec latency 17.546 ms 2024-04-25T02:01:32.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 159917 2.47 MB/sec execute 219 sec latency 18.406 ms 2024-04-25T02:01:32.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:32 smithi040 ceph-mon[32418]: pgmap v217: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:01:32.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:32 smithi134 ceph-mon[33098]: pgmap v217: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:01:32.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:32 smithi031 ceph-mon[29204]: pgmap v217: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:01:33.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 160186 2.46 MB/sec execute 220 sec latency 18.676 ms 2024-04-25T02:01:34.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 160604 2.45 MB/sec execute 221 sec latency 21.324 ms 2024-04-25T02:01:34.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:34 smithi040 ceph-mon[32418]: pgmap v218: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-04-25T02:01:34.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:34 smithi134 ceph-mon[33098]: pgmap v218: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-04-25T02:01:34.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:34 smithi031 ceph-mon[29204]: pgmap v218: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-04-25T02:01:35.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 161127 2.46 MB/sec execute 222 sec latency 20.101 ms 2024-04-25T02:01:36.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 161642 2.45 MB/sec execute 223 sec latency 21.571 ms 2024-04-25T02:01:36.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:36 smithi040 ceph-mon[32418]: pgmap v219: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.9 MiB/s wr, 180 op/s 2024-04-25T02:01:36.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:36 smithi134 ceph-mon[33098]: pgmap v219: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.9 MiB/s wr, 180 op/s 2024-04-25T02:01:36.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:36 smithi031 ceph-mon[29204]: pgmap v219: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.9 MiB/s wr, 180 op/s 2024-04-25T02:01:37.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 162134 2.45 MB/sec execute 224 sec latency 22.257 ms 2024-04-25T02:01:38.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 162915 2.46 MB/sec execute 225 sec latency 17.639 ms 2024-04-25T02:01:38.542 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:38 smithi031 ceph-mon[29204]: pgmap v220: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-04-25T02:01:38.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:38 smithi040 ceph-mon[32418]: pgmap v220: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-04-25T02:01:38.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:38 smithi134 ceph-mon[33098]: pgmap v220: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-04-25T02:01:39.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 163301 2.47 MB/sec execute 226 sec latency 19.112 ms 2024-04-25T02:01:40.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 163581 2.46 MB/sec execute 227 sec latency 75.736 ms 2024-04-25T02:01:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.406+0000 7f004b941640 1 -- 172.21.15.31:0/1573638903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0044102a10 msgr2=0x7f0044102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.406+0000 7f004b941640 1 --2- 172.21.15.31:0/1573638903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0044102a10 0x7f0044102df0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0040009860 tx=0x7f004002a3d0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.407+0000 7f004b941640 1 -- 172.21.15.31:0/1573638903 shutdown_connections 2024-04-25T02:01:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.407+0000 7f004b941640 1 --2- 172.21.15.31:0/1573638903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f004410bd50 0x7f004410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.407+0000 7f004b941640 1 --2- 172.21.15.31:0/1573638903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0044103330 0x7f004410b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.407+0000 7f004b941640 1 --2- 172.21.15.31:0/1573638903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0044102a10 0x7f0044102df0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.407+0000 7f004b941640 1 -- 172.21.15.31:0/1573638903 >> 172.21.15.31:0/1573638903 conn(0x7f00440777c0 msgr2=0x7f0044077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.407+0000 7f004b941640 1 -- 172.21.15.31:0/1573638903 shutdown_connections 2024-04-25T02:01:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.407+0000 7f004b941640 1 -- 172.21.15.31:0/1573638903 wait complete. 2024-04-25T02:01:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.407+0000 7f004b941640 1 Processor -- start 2024-04-25T02:01:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.407+0000 7f004b941640 1 -- start start 2024-04-25T02:01:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f004b941640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0044102a10 0x7f004419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f004b941640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0044103330 0x7f004419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f004b941640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f004410bd50 0x7f004419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f004b941640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f00441119b0 con 0x7f0044102a10 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f004b941640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0044111830 con 0x7f0044103330 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f004b941640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0044111b30 con 0x7f004410bd50 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f00496dd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0044102a10 0x7f004419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f0048edc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0044103330 0x7f004419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f00496dd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0044102a10 0x7f004419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38904/0 (socket says 172.21.15.31:38904) 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f00496dd640 1 -- 172.21.15.31:0/1688613020 learned_addr learned my addr 172.21.15.31:0/1688613020 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f0049ede640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f004410bd50 0x7f004419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f0048edc640 1 -- 172.21.15.31:0/1688613020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f004410bd50 msgr2=0x7f004419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f0048edc640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f004410bd50 0x7f004419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f0048edc640 1 -- 172.21.15.31:0/1688613020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0044102a10 msgr2=0x7f004419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f0048edc640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0044102a10 0x7f004419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f0048edc640 1 -- 172.21.15.31:0/1688613020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004419f080 con 0x7f0044103330 2024-04-25T02:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f00496dd640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0044102a10 0x7f004419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.408+0000 7f0048edc640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0044103330 0x7f004419a620 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f003400e9f0 tx=0x7f003400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.409+0000 7f003a7fc640 1 -- 172.21.15.31:0/1688613020 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003400ccc0 con 0x7f0044103330 2024-04-25T02:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.409+0000 7f004b941640 1 -- 172.21.15.31:0/1688613020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f004419f370 con 0x7f0044103330 2024-04-25T02:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.409+0000 7f003a7fc640 1 -- 172.21.15.31:0/1688613020 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0034005020 con 0x7f0044103330 2024-04-25T02:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.409+0000 7f003a7fc640 1 -- 172.21.15.31:0/1688613020 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0034004a00 con 0x7f0044103330 2024-04-25T02:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.409+0000 7f004b941640 1 -- 172.21.15.31:0/1688613020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00441a6c00 con 0x7f0044103330 2024-04-25T02:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f004e645740 1 Processor -- start 2024-04-25T02:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f004e645740 1 -- start start 2024-04-25T02:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f003a7fc640 1 -- 172.21.15.31:0/1688613020 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0034004ba0 con 0x7f0044103330 2024-04-25T02:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f004e645740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 0x5560b7e09620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f004e645740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09b60 0x5560b7e13cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f004e645740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e16620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f004e645740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5560b7d6dcb0 con 0x5560b7e14230 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f004e645740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5560b7d165b0 con 0x5560b7e09240 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f004e645740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5560b7d69c70 con 0x5560b7e09b60 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f0048edc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e16620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f0049ede640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 0x5560b7e09620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f0048edc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e16620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38908/0 (socket says 172.21.15.31:38908) 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f0049ede640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 0x5560b7e09620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45568/0 (socket says 172.21.15.31:45568) 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.412+0000 7f0048edc640 1 -- 172.21.15.31:0/4293281581 learned_addr learned my addr 172.21.15.31:0/4293281581 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f00496dd640 1 --2- 172.21.15.31:0/4293281581 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09b60 0x5560b7e13cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f0048edc640 1 -- 172.21.15.31:0/4293281581 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09b60 msgr2=0x5560b7e13cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f0048edc640 1 --2- 172.21.15.31:0/4293281581 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09b60 0x5560b7e13cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f0048edc640 1 -- 172.21.15.31:0/4293281581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 msgr2=0x5560b7e09620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f0048edc640 1 --2- 172.21.15.31:0/4293281581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 0x5560b7e09620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f0048edc640 1 -- 172.21.15.31:0/4293281581 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560b7a82210 con 0x5560b7e14230 2024-04-25T02:01:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f003a7fc640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f00300a2da0 0x7f00300a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f00496dd640 1 --2- 172.21.15.31:0/4293281581 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09b60 0x5560b7e13cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f0049ede640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f00300a2da0 0x7f00300a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f003a7fc640 1 -- 172.21.15.31:0/1688613020 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f00340cf390 con 0x7f0044103330 2024-04-25T02:01:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f0048edc640 1 --2- 172.21.15.31:0/4293281581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e16620 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f00340cca70 tx=0x7f00340d5640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340b1219b5774233 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f002f7fe640 1 -- 172.21.15.31:0/4293281581 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003401c030 con 0x5560b7e14230 2024-04-25T02:01:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f002f7fe640 1 -- 172.21.15.31:0/4293281581 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f003409bfd0 con 0x5560b7e14230 2024-04-25T02:01:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.413+0000 7f002f7fe640 1 -- 172.21.15.31:0/4293281581 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003409b410 con 0x5560b7e14230 2024-04-25T02:01:40.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.414+0000 7f004e645740 1 -- 172.21.15.31:0/4293281581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 msgr2=0x5560b7e16620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:40.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.414+0000 7f0049ede640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f00300a2da0 0x7f00300a5260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f003c005fd0 tx=0x7f003c010040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.414+0000 7f004e645740 1 --2- 172.21.15.31:0/4293281581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e16620 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f00340cca70 tx=0x7f00340d5640 comp rx=0 tx=0).stop 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.414+0000 7f004e645740 1 -- 172.21.15.31:0/4293281581 shutdown_connections 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.414+0000 7f004e645740 1 --2- 172.21.15.31:0/4293281581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e16620 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.414+0000 7f004e645740 1 --2- 172.21.15.31:0/4293281581 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09b60 0x5560b7e13cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.414+0000 7f004e645740 1 --2- 172.21.15.31:0/4293281581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 0x5560b7e09620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.414+0000 7f004e645740 1 -- 172.21.15.31:0/4293281581 >> 172.21.15.31:0/4293281581 conn(0x5560b7e05560 msgr2=0x5560b7e05940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.414+0000 7f004e645740 1 -- 172.21.15.31:0/4293281581 shutdown_connections 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.414+0000 7f004e645740 1 -- 172.21.15.31:0/4293281581 wait complete. 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.414+0000 7f004e645740 1 -- start start 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f004e645740 1 Processor -- start 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f004e645740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 0x5560b7e110e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f004e645740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 0x5560b7e11620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f004e645740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e11b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f004e645740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5560b7e17020 con 0x5560b7e14230 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f004e645740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5560b7d29b50 con 0x5560b7e09b60 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f004e645740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5560b7e171a0 con 0x5560b7e09240 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f004e645740 10 client.0 _ll_register_callbacks cb 0x5560b7d64ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f0049ede640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e11b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f0048edc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 0x5560b7e11620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f0049ede640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e11b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38920/0 (socket says 172.21.15.31:38920) 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f00496dd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 0x5560b7e110e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.415+0000 7f0049ede640 1 -- 172.21.15.31:0/420523458 learned_addr learned my addr 172.21.15.31:0/420523458 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f0048edc640 1 -- 172.21.15.31:0/420523458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 msgr2=0x5560b7e110e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f0048edc640 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 0x5560b7e110e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f0048edc640 1 -- 172.21.15.31:0/420523458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 msgr2=0x5560b7e11b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f0048edc640 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e11b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f0048edc640 1 -- 172.21.15.31:0/420523458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560b7b4e620 con 0x5560b7e09b60 2024-04-25T02:01:40.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f0049ede640 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e11b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f00496dd640 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 0x5560b7e110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f0048edc640 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 0x5560b7e11620 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0034099b80 tx=0x7f00340096e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=421858849056ae32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f002cff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f002cff9640 1 -- 172.21.15.31:0/420523458 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003401c040 con 0x5560b7e09b60 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f002cff9640 1 -- 172.21.15.31:0/420523458 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f00340051d0 con 0x5560b7e09b60 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f004e645740 1 -- 172.21.15.31:0/420523458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f003409bfd0 con 0x5560b7e09b60 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f002cff9640 1 -- 172.21.15.31:0/420523458 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f003409b230 con 0x5560b7e09b60 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f004e645740 10 client.26324 fetch_fsmap learned FSMap version 33 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f004e645740 1 -- 172.21.15.31:0/420523458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5560b7e17020 con 0x5560b7e09b60 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.416+0000 7f002cff9640 1 -- 172.21.15.31:0/420523458 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f003409bfd0 con 0x5560b7e09b60 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.417+0000 7f002cff9640 1 -- 172.21.15.31:0/420523458 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0034097f40 con 0x5560b7e09b60 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.417+0000 7f004e645740 10 client.26324 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.417+0000 7f004e645740 10 client.26324 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.417+0000 7f004e645740 20 client.26324 populate_metadata read hostname 'smithi031' 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.417+0000 7f004e645740 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560b7e24270 0x5560b7e26660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.417+0000 7f004e645740 4 client.26324 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.417+0000 7f004e645740 1 -- 172.21.15.31:0/420523458 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560b7d7b1f0 con 0x5560b7e24270 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.417+0000 7f00496dd640 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560b7e24270 0x5560b7e26660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.417+0000 7f00496dd640 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560b7e24270 0x5560b7e26660 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b8a469c381adca4 server_cookie=74000b608dd76aa0 in_seq=0 out_seq=0 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.417+0000 7f002cff9640 10 client.26324 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.418+0000 7f002cff9640 1 -- 172.21.15.31:0/420523458 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560b7d7b1f0 con 0x5560b7e24270 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.418+0000 7f002cff9640 10 client.26324 handle_command_reply: tid=0 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.418+0000 7f004e645740 1 client.26324 shutdown 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.418+0000 7f004e645740 1 -- 172.21.15.31:0/420523458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 msgr2=0x5560b7e11620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.418+0000 7f004e645740 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 0x5560b7e11620 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0034099b80 tx=0x7f00340096e0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.419+0000 7f004e645740 1 -- 172.21.15.31:0/420523458 shutdown_connections 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.419+0000 7f004e645740 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560b7e24270 0x5560b7e26660 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.419+0000 7f004e645740 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e11b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.419+0000 7f004e645740 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 0x5560b7e11620 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.419+0000 7f004e645740 1 --2- 172.21.15.31:0/420523458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 0x5560b7e110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.419+0000 7f002cff9640 0 client.26324 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.419+0000 7f004e645740 1 -- 172.21.15.31:0/420523458 >> 172.21.15.31:0/420523458 conn(0x5560b7e05560 msgr2=0x5560b7e0b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.419+0000 7f004e645740 1 -- 172.21.15.31:0/420523458 shutdown_connections 2024-04-25T02:01:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.419+0000 7f004e645740 1 -- 172.21.15.31:0/420523458 wait complete. 2024-04-25T02:01:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.419+0000 7f004e645740 20 client.26324 trim_cache size 0 max 16384 2024-04-25T02:01:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f004e645740 1 Processor -- start 2024-04-25T02:01:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f004e645740 1 -- start start 2024-04-25T02:01:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f004e645740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 0x5560b7e1ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f004e645740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 0x5560b7e10240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f004e645740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e0bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f004e645740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f00441af4d0 con 0x5560b7e14230 2024-04-25T02:01:40.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f004e645740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00441b1010 con 0x5560b7e09b60 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f004e645740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f00441afba0 con 0x5560b7e09240 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f0049ede640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e0bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f00496dd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 0x5560b7e1ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f0048edc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 0x5560b7e10240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f0049ede640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e0bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38922/0 (socket says 172.21.15.31:38922) 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.437+0000 7f0049ede640 1 -- 172.21.15.31:0/2469155955 learned_addr learned my addr 172.21.15.31:0/2469155955 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.438+0000 7f0049ede640 1 -- 172.21.15.31:0/2469155955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 msgr2=0x5560b7e1ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.438+0000 7f0049ede640 1 --2- 172.21.15.31:0/2469155955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 0x5560b7e1ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.438+0000 7f0049ede640 1 -- 172.21.15.31:0/2469155955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 msgr2=0x5560b7e10240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.438+0000 7f0049ede640 1 --2- 172.21.15.31:0/2469155955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 0x5560b7e10240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.438+0000 7f0049ede640 1 -- 172.21.15.31:0/2469155955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560b7e0bff0 con 0x5560b7e14230 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.438+0000 7f0048edc640 1 --2- 172.21.15.31:0/2469155955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 0x5560b7e10240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.438+0000 7f00496dd640 1 --2- 172.21.15.31:0/2469155955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 0x5560b7e1ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.438+0000 7f0049ede640 1 --2- 172.21.15.31:0/2469155955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e0bab0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f003c004390 tx=0x7f003c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc377e894dcf487 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.438+0000 7f002effd640 1 -- 172.21.15.31:0/2469155955 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c00de10 con 0x5560b7e14230 2024-04-25T02:01:40.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.438+0000 7f002effd640 1 -- 172.21.15.31:0/2469155955 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f003c02b070 con 0x5560b7e14230 2024-04-25T02:01:40.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.438+0000 7f002effd640 1 -- 172.21.15.31:0/2469155955 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c018680 con 0x5560b7e14230 2024-04-25T02:01:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.439+0000 7f004e645740 1 -- 172.21.15.31:0/2469155955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 msgr2=0x5560b7e0bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.439+0000 7f004e645740 1 --2- 172.21.15.31:0/2469155955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e0bab0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f003c004390 tx=0x7f003c016000 comp rx=0 tx=0).stop 2024-04-25T02:01:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.439+0000 7f004e645740 1 -- 172.21.15.31:0/2469155955 shutdown_connections 2024-04-25T02:01:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.439+0000 7f004e645740 1 --2- 172.21.15.31:0/2469155955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e14230 0x5560b7e0bab0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.439+0000 7f004e645740 1 --2- 172.21.15.31:0/2469155955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09b60 0x5560b7e10240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.439+0000 7f004e645740 1 --2- 172.21.15.31:0/2469155955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e09240 0x5560b7e1ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.439+0000 7f004e645740 1 -- 172.21.15.31:0/2469155955 >> 172.21.15.31:0/2469155955 conn(0x5560b7e05560 msgr2=0x5560b7e03d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.439+0000 7f004e645740 1 -- 172.21.15.31:0/2469155955 shutdown_connections 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.439+0000 7f004e645740 1 -- 172.21.15.31:0/2469155955 wait complete. 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f004e645740 1 -- start start 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f004e645740 1 Processor -- start 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f004e645740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 0x5560b7e1c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f004e645740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e09b60 0x5560b7e17a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f004e645740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e14230 0x5560b7e17f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f004e645740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f00441b0940 con 0x5560b7e09b60 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f004e645740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00441b0270 con 0x5560b7e09240 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f004e645740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f00441ad4d0 con 0x5560b7e14230 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f004e645740 10 client.0 _ll_register_callbacks cb 0x7f003c009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f0048edc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e09b60 0x5560b7e17a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f00496dd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 0x5560b7e1c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f0049ede640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e14230 0x5560b7e17f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f0048edc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e09b60 0x5560b7e17a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38926/0 (socket says 172.21.15.31:38926) 2024-04-25T02:01:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.440+0000 7f0048edc640 1 -- 172.21.15.31:0/749655309 learned_addr learned my addr 172.21.15.31:0/749655309 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f0048edc640 1 -- 172.21.15.31:0/749655309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e14230 msgr2=0x5560b7e17f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f0048edc640 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e14230 0x5560b7e17f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f0048edc640 1 -- 172.21.15.31:0/749655309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 msgr2=0x5560b7e1c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f0048edc640 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 0x5560b7e1c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f0048edc640 1 -- 172.21.15.31:0/749655309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560b7e18480 con 0x5560b7e09b60 2024-04-25T02:01:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f00496dd640 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 0x5560b7e1c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f0049ede640 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e14230 0x5560b7e17f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f0048edc640 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e09b60 0x5560b7e17a00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f003409b970 tx=0x7f003400c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c52971e933601ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f002cff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:01:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f002cff9640 1 -- 172.21.15.31:0/749655309 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003401c040 con 0x5560b7e09b60 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f002cff9640 1 -- 172.21.15.31:0/749655309 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f00340a40b0 con 0x5560b7e09b60 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f004e645740 1 -- 172.21.15.31:0/749655309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f003c02b070 con 0x5560b7e09b60 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f002cff9640 1 -- 172.21.15.31:0/749655309 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f003409b410 con 0x5560b7e09b60 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f004e645740 10 client.16734 fetch_fsmap learned FSMap version 33 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.441+0000 7f004e645740 1 -- 172.21.15.31:0/749655309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5560b7e8a330 con 0x5560b7e09b60 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.442+0000 7f002cff9640 1 -- 172.21.15.31:0/749655309 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f003c02b070 con 0x5560b7e09b60 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.442+0000 7f002cff9640 1 -- 172.21.15.31:0/749655309 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f003409bb30 con 0x5560b7e09b60 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.442+0000 7f004e645740 10 client.16734 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.442+0000 7f004e645740 10 client.16734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.442+0000 7f004e645740 20 client.16734 populate_metadata read hostname 'smithi031' 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.442+0000 7f004e645740 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560b7e8a810 0x5560b7e8cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.442+0000 7f004e645740 4 client.16734 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.442+0000 7f004e645740 1 -- 172.21.15.31:0/749655309 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5560b7d75540 con 0x5560b7e8a810 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.442+0000 7f00496dd640 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560b7e8a810 0x5560b7e8cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.443+0000 7f002cff9640 10 client.16734 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.443+0000 7f00496dd640 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560b7e8a810 0x5560b7e8cc00 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efe3882e4678f125 server_cookie=b6be8f7f568681ff in_seq=0 out_seq=0 2024-04-25T02:01:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f002cff9640 1 -- 172.21.15.31:0/749655309 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5560b7d75540 con 0x5560b7e8a810 2024-04-25T02:01:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f002cff9640 10 client.16734 handle_command_reply: tid=0 2024-04-25T02:01:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f004e645740 1 client.16734 shutdown 2024-04-25T02:01:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f004e645740 1 -- 172.21.15.31:0/749655309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e09b60 msgr2=0x5560b7e17a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f004e645740 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e09b60 0x5560b7e17a00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f003409b970 tx=0x7f003400c7d0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f004e645740 1 -- 172.21.15.31:0/749655309 shutdown_connections 2024-04-25T02:01:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f004e645740 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560b7e8a810 0x5560b7e8cc00 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f004e645740 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5560b7e14230 0x5560b7e17f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f004e645740 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5560b7e09b60 0x5560b7e17a00 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f002cff9640 0 client.16734 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f004e645740 1 --2- 172.21.15.31:0/749655309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560b7e09240 0x5560b7e1c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f004e645740 1 -- 172.21.15.31:0/749655309 >> 172.21.15.31:0/749655309 conn(0x5560b7e05560 msgr2=0x5560b7e045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f004e645740 1 -- 172.21.15.31:0/749655309 shutdown_connections 2024-04-25T02:01:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.463+0000 7f004e645740 1 -- 172.21.15.31:0/749655309 wait complete. 2024-04-25T02:01:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.464+0000 7f004e645740 20 client.16734 trim_cache size 0 max 16384 2024-04-25T02:01:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:01:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:01:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:01:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:01:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.464+0000 7f004b941640 1 -- 172.21.15.31:0/1688613020 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f00300a2da0 msgr2=0x7f00300a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.464+0000 7f004b941640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f00300a2da0 0x7f00300a5260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f003c005fd0 tx=0x7f003c010040 comp rx=0 tx=0).stop 2024-04-25T02:01:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.464+0000 7f004b941640 1 -- 172.21.15.31:0/1688613020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0044103330 msgr2=0x7f004419a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.464+0000 7f004b941640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0044103330 0x7f004419a620 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f003400e9f0 tx=0x7f003400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.464+0000 7f004b941640 1 -- 172.21.15.31:0/1688613020 shutdown_connections 2024-04-25T02:01:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.464+0000 7f004b941640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f00300a2da0 0x7f00300a5260 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.464+0000 7f004b941640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f004410bd50 0x7f004419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.464+0000 7f004b941640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0044103330 0x7f004419a620 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.464+0000 7f004b941640 1 --2- 172.21.15.31:0/1688613020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0044102a10 0x7f004419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.464+0000 7f004b941640 1 -- 172.21.15.31:0/1688613020 >> 172.21.15.31:0/1688613020 conn(0x7f00440777c0 msgr2=0x7f0044107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.465+0000 7f004b941640 1 -- 172.21.15.31:0/1688613020 shutdown_connections 2024-04-25T02:01:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:40.465+0000 7f004b941640 1 -- 172.21.15.31:0/1688613020 wait complete. 2024-04-25T02:01:40.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:40 smithi040 ceph-mon[32418]: pgmap v221: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-04-25T02:01:40.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:40 smithi134 ceph-mon[33098]: pgmap v221: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-04-25T02:01:40.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:40 smithi031 ceph-mon[29204]: pgmap v221: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.3 MiB/s wr, 123 op/s 2024-04-25T02:01:41.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 163850 2.45 MB/sec execute 228 sec latency 17.860 ms 2024-04-25T02:01:41.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:41 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:01:41.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:41 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:01:41.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:41 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:01:41.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:41 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:01:41.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:41 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:01:41.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:41 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:01:41.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:41 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:01:41.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:41 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:01:41.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:41 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:01:41.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:41 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:01:41.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:41 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:01:41.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:41 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:01:42.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 164358 2.45 MB/sec execute 229 sec latency 17.509 ms 2024-04-25T02:01:42.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:42 smithi040 ceph-mon[32418]: pgmap v222: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:01:42.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:42 smithi134 ceph-mon[33098]: pgmap v222: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:01:42.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:42 smithi031 ceph-mon[29204]: pgmap v222: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:01:43.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 164842 2.45 MB/sec execute 230 sec latency 22.317 ms 2024-04-25T02:01:43.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:43 smithi040 ceph-mon[32418]: pgmap v223: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-25T02:01:43.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:43 smithi134 ceph-mon[33098]: pgmap v223: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-25T02:01:43.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:43 smithi031 ceph-mon[29204]: pgmap v223: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-25T02:01:44.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 165432 2.45 MB/sec execute 231 sec latency 23.421 ms 2024-04-25T02:01:45.312 INFO:tasks.workunit.client.0.smithi031.stdout: 1 165960 2.46 MB/sec execute 232 sec latency 12.713 ms 2024-04-25T02:01:46.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:45 smithi040 ceph-mon[32418]: pgmap v224: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-04-25T02:01:46.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:45 smithi134 ceph-mon[33098]: pgmap v224: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-04-25T02:01:46.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:45 smithi031 ceph-mon[29204]: pgmap v224: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.9 MiB/s wr, 182 op/s 2024-04-25T02:01:46.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 166510 2.46 MB/sec execute 233 sec latency 18.354 ms 2024-04-25T02:01:47.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 166976 2.47 MB/sec execute 234 sec latency 18.216 ms 2024-04-25T02:01:48.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:47 smithi040 ceph-mon[32418]: pgmap v225: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:01:48.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:47 smithi134 ceph-mon[33098]: pgmap v225: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:01:48.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:47 smithi031 ceph-mon[29204]: pgmap v225: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:01:48.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 167233 2.46 MB/sec execute 235 sec latency 18.052 ms 2024-04-25T02:01:49.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 167653 2.45 MB/sec execute 236 sec latency 19.994 ms 2024-04-25T02:01:50.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:49 smithi040 ceph-mon[32418]: pgmap v226: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:01:50.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:49 smithi134 ceph-mon[33098]: pgmap v226: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:01:50.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:49 smithi031 ceph-mon[29204]: pgmap v226: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:01:50.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 168142 2.46 MB/sec execute 237 sec latency 83.513 ms 2024-04-25T02:01:51.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 168605 2.45 MB/sec execute 238 sec latency 21.991 ms 2024-04-25T02:01:52.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:51 smithi040 ceph-mon[32418]: pgmap v227: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:01:52.192 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:51 smithi134 ceph-mon[33098]: pgmap v227: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:01:52.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:51 smithi031 ceph-mon[29204]: pgmap v227: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:01:52.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 169134 2.45 MB/sec execute 239 sec latency 25.377 ms 2024-04-25T02:01:53.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 169703 2.46 MB/sec execute 240 sec latency 18.928 ms 2024-04-25T02:01:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.562+0000 7f5c3164d640 1 -- 172.21.15.31:0/224889456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c2c10e020 msgr2=0x7f5c2c110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.562+0000 7f5c3164d640 1 --2- 172.21.15.31:0/224889456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c2c10e020 0x7f5c2c110410 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009900 tx=0x7f5c2402a480 comp rx=0 tx=0).stop 2024-04-25T02:01:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.562+0000 7f5c3164d640 1 -- 172.21.15.31:0/224889456 shutdown_connections 2024-04-25T02:01:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.562+0000 7f5c3164d640 1 --2- 172.21.15.31:0/224889456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c2c10e020 0x7f5c2c110410 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.562+0000 7f5c3164d640 1 --2- 172.21.15.31:0/224889456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c2c074410 0x7f5c2c10dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.562+0000 7f5c3164d640 1 --2- 172.21.15.31:0/224889456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c2c073af0 0x7f5c2c073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.562+0000 7f5c3164d640 1 -- 172.21.15.31:0/224889456 >> 172.21.15.31:0/224889456 conn(0x7f5c2c0fd1f0 msgr2=0x7f5c2c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:53.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.562+0000 7f5c3164d640 1 -- 172.21.15.31:0/224889456 shutdown_connections 2024-04-25T02:01:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.562+0000 7f5c3164d640 1 -- 172.21.15.31:0/224889456 wait complete. 2024-04-25T02:01:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.562+0000 7f5c3164d640 1 Processor -- start 2024-04-25T02:01:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c3164d640 1 -- start start 2024-04-25T02:01:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c3164d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c2c073af0 0x7f5c2c19a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c3164d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c2c074410 0x7f5c2c19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c3164d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c2c10e020 0x7f5c2c19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c3164d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c113c40 con 0x7f5c2c10e020 2024-04-25T02:01:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c3164d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c113ac0 con 0x7f5c2c074410 2024-04-25T02:01:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c3164d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c113dc0 con 0x7f5c2c073af0 2024-04-25T02:01:53.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c2c10e020 0x7f5c2c19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c2c073af0 0x7f5c2c19a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c2c074410 0x7f5c2c19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c2c073af0 0x7f5c2c19a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:40940/0 (socket says 172.21.15.31:40940) 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2affd640 1 -- 172.21.15.31:0/4012730426 learned_addr learned my addr 172.21.15.31:0/4012730426 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2b7fe640 1 -- 172.21.15.31:0/4012730426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c2c073af0 msgr2=0x7f5c2c19a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2b7fe640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c2c073af0 0x7f5c2c19a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2b7fe640 1 -- 172.21.15.31:0/4012730426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c2c074410 msgr2=0x7f5c2c19a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2b7fe640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c2c074410 0x7f5c2c19a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2b7fe640 1 -- 172.21.15.31:0/4012730426 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2c19f090 con 0x7f5c2c10e020 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c2c074410 0x7f5c2c19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2affd640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c2c073af0 0x7f5c2c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.563+0000 7f5c2b7fe640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c2c10e020 0x7f5c2c19e960 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f5c2402a9a0 tx=0x7f5c24003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:53.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.564+0000 7f5c23fff640 1 -- 172.21.15.31:0/4012730426 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2403f070 con 0x7f5c2c10e020 2024-04-25T02:01:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.564+0000 7f5c3164d640 1 -- 172.21.15.31:0/4012730426 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c2c19f320 con 0x7f5c2c10e020 2024-04-25T02:01:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.564+0000 7f5c23fff640 1 -- 172.21.15.31:0/4012730426 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c24031b50 con 0x7f5c2c10e020 2024-04-25T02:01:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.564+0000 7f5c23fff640 1 -- 172.21.15.31:0/4012730426 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24035030 con 0x7f5c2c10e020 2024-04-25T02:01:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.564+0000 7f5c3164d640 1 -- 172.21.15.31:0/4012730426 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c2c1a6c10 con 0x7f5c2c10e020 2024-04-25T02:01:53.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.566+0000 7f5c23fff640 1 -- 172.21.15.31:0/4012730426 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5c24042070 con 0x7f5c2c10e020 2024-04-25T02:01:53.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.566+0000 7f5c34439740 1 Processor -- start 2024-04-25T02:01:53.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.566+0000 7f5c34439740 1 -- start start 2024-04-25T02:01:53.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c34439740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 0x556a5de7d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c34439740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7db60 0x556a5de87cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c34439740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de88230 0x556a5de8a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c34439740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556a5dde1cb0 con 0x556a5de88230 2024-04-25T02:01:53.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c34439740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556a5dd8a5b0 con 0x556a5de7db60 2024-04-25T02:01:53.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c34439740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556a5ddddc70 con 0x556a5de7d240 2024-04-25T02:01:53.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2a7fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 0x556a5de7d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7db60 0x556a5de87cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c23fff640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5c180a2da0 0x7f5c180a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2a7fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 0x556a5de7d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:40942/0 (socket says 172.21.15.31:40942) 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2a7fc640 1 -- 172.21.15.31:0/3561709032 learned_addr learned my addr 172.21.15.31:0/3561709032 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7db60 0x556a5de87cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55068/0 (socket says 172.21.15.31:55068) 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/3561709032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de88230 0x556a5de8a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2b7fe640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5c180a2da0 0x7f5c180a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2a7fc640 1 -- 172.21.15.31:0/3561709032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7db60 msgr2=0x556a5de87cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/3561709032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7db60 0x556a5de87cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2a7fc640 1 -- 172.21.15.31:0/3561709032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de88230 msgr2=0x556a5de8a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/3561709032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de88230 0x556a5de8a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2a7fc640 1 -- 172.21.15.31:0/3561709032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a5daf6210 con 0x556a5de7d240 2024-04-25T02:01:53.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c23fff640 1 -- 172.21.15.31:0/4012730426 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f5c240ef9a0 con 0x7f5c2c10e020 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.567+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/3561709032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 0x556a5de7d620 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5c14005f60 tx=0x7f5c14005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d60ab12257bf4784 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.568+0000 7f5c2b7fe640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5c180a2da0 0x7f5c180a5260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c240bbc30 tx=0x7f5c2402a480 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.568+0000 7f5c217fa640 1 -- 172.21.15.31:0/3561709032 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c14004030 con 0x556a5de7d240 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.568+0000 7f5c217fa640 1 -- 172.21.15.31:0/3561709032 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c1400f880 con 0x556a5de7d240 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.568+0000 7f5c217fa640 1 -- 172.21.15.31:0/3561709032 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c14003a60 con 0x556a5de7d240 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.568+0000 7f5c34439740 1 -- 172.21.15.31:0/3561709032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 msgr2=0x556a5de7d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.568+0000 7f5c34439740 1 --2- 172.21.15.31:0/3561709032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 0x556a5de7d620 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5c14005f60 tx=0x7f5c14005850 comp rx=0 tx=0).stop 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 -- 172.21.15.31:0/3561709032 shutdown_connections 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 --2- 172.21.15.31:0/3561709032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de88230 0x556a5de8a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 --2- 172.21.15.31:0/3561709032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7db60 0x556a5de87cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 --2- 172.21.15.31:0/3561709032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 0x556a5de7d620 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 -- 172.21.15.31:0/3561709032 >> 172.21.15.31:0/3561709032 conn(0x556a5de79560 msgr2=0x556a5de79940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 -- 172.21.15.31:0/3561709032 shutdown_connections 2024-04-25T02:01:53.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 -- 172.21.15.31:0/3561709032 wait complete. 2024-04-25T02:01:53.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 -- start start 2024-04-25T02:01:53.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 Processor -- start 2024-04-25T02:01:53.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7d240 0x556a5de851e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de85720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de88230 0x556a5de85c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556a5de8b020 con 0x556a5de7db60 2024-04-25T02:01:53.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556a5dd9db50 con 0x556a5de7d240 2024-04-25T02:01:53.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556a5de8b1a0 con 0x556a5de88230 2024-04-25T02:01:53.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c34439740 10 client.0 _ll_register_callbacks cb 0x556a5ddd8ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:53.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de85720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de85720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32772/0 (socket says 172.21.15.31:32772) 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c2a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7d240 0x556a5de851e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c2affd640 1 -- 172.21.15.31:0/1432153891 learned_addr learned my addr 172.21.15.31:0/1432153891 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.569+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de88230 0x556a5de85c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c2affd640 1 -- 172.21.15.31:0/1432153891 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de88230 msgr2=0x556a5de85c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c2affd640 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de88230 0x556a5de85c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c2affd640 1 -- 172.21.15.31:0/1432153891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7d240 msgr2=0x556a5de851e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c2affd640 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7d240 0x556a5de851e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c2affd640 1 -- 172.21.15.31:0/1432153891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a5dbc2620 con 0x556a5de7db60 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7d240 0x556a5de851e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c2affd640 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de85720 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00ff50 tx=0x7f5c1c0128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=522bb2689cee6697 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:53.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c127fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c127fc640 1 -- 172.21.15.31:0/1432153891 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c010430 con 0x556a5de7db60 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c127fc640 1 -- 172.21.15.31:0/1432153891 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f5c1c0105d0 con 0x556a5de7db60 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c34439740 1 -- 172.21.15.31:0/1432153891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c1400f880 con 0x556a5de7db60 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c127fc640 1 -- 172.21.15.31:0/1432153891 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5c1c0107d0 con 0x556a5de7db60 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c34439740 10 client.16746 fetch_fsmap learned FSMap version 33 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c34439740 1 -- 172.21.15.31:0/1432153891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556a5de972b0 con 0x556a5de7db60 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.570+0000 7f5c127fc640 1 -- 172.21.15.31:0/1432153891 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c1400f880 con 0x556a5de7db60 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.571+0000 7f5c127fc640 1 -- 172.21.15.31:0/1432153891 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f5c1c023070 con 0x556a5de7db60 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.571+0000 7f5c34439740 10 client.16746 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.571+0000 7f5c34439740 10 client.16746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.571+0000 7f5c34439740 20 client.16746 populate_metadata read hostname 'smithi031' 2024-04-25T02:01:53.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.571+0000 7f5c34439740 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a5de98770 0x556a5de9abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.571+0000 7f5c34439740 4 client.16746 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.571+0000 7f5c34439740 1 -- 172.21.15.31:0/1432153891 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a5ddef1f0 con 0x556a5de98770 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.571+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a5de98770 0x556a5de9abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.571+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a5de98770 0x556a5de9abc0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a36f42ef2d86831 server_cookie=ce3987630cec2fb in_seq=0 out_seq=0 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.571+0000 7f5c127fc640 10 client.16746 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c127fc640 1 -- 172.21.15.31:0/1432153891 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a5ddef1f0 con 0x556a5de98770 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c127fc640 10 client.16746 handle_command_reply: tid=0 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 1 client.16746 shutdown 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 1 -- 172.21.15.31:0/1432153891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 msgr2=0x556a5de85720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de85720 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00ff50 tx=0x7f5c1c0128e0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 1 -- 172.21.15.31:0/1432153891 shutdown_connections 2024-04-25T02:01:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a5de98770 0x556a5de9abc0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de88230 0x556a5de85c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de85720 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 1 --2- 172.21.15.31:0/1432153891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7d240 0x556a5de851e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 1 -- 172.21.15.31:0/1432153891 >> 172.21.15.31:0/1432153891 conn(0x556a5de79560 msgr2=0x556a5de7f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:53.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c127fc640 0 client.16746 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:53.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 1 -- 172.21.15.31:0/1432153891 shutdown_connections 2024-04-25T02:01:53.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 1 -- 172.21.15.31:0/1432153891 wait complete. 2024-04-25T02:01:53.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.572+0000 7f5c34439740 20 client.16746 trim_cache size 0 max 16384 2024-04-25T02:01:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.589+0000 7f5c34439740 1 Processor -- start 2024-04-25T02:01:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.589+0000 7f5c34439740 1 -- start start 2024-04-25T02:01:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c34439740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 0x556a5de87910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c34439740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de93f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c34439740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de88230 0x556a5dea04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c34439740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1af470 con 0x556a5de7db60 2024-04-25T02:01:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c34439740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1b0fb0 con 0x556a5de88230 2024-04-25T02:01:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c34439740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1afb40 con 0x556a5de7d240 2024-04-25T02:01:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de93f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de93f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32774/0 (socket says 172.21.15.31:32774) 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2affd640 1 -- 172.21.15.31:0/1848028869 learned_addr learned my addr 172.21.15.31:0/1848028869 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/1848028869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de88230 0x556a5dea04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/1848028869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 0x556a5de87910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2affd640 1 -- 172.21.15.31:0/1848028869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 msgr2=0x556a5de87910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2affd640 1 --2- 172.21.15.31:0/1848028869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 0x556a5de87910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2affd640 1 -- 172.21.15.31:0/1848028869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de88230 msgr2=0x556a5dea04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2affd640 1 --2- 172.21.15.31:0/1848028869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de88230 0x556a5dea04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2affd640 1 -- 172.21.15.31:0/1848028869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a5de83730 con 0x556a5de7db60 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/1848028869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 0x556a5de87910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c2affd640 1 --2- 172.21.15.31:0/1848028869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de93f10 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00eba0 tx=0x7f5c1c00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8f09b7e7534e5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.590+0000 7f5c20ff9640 1 -- 172.21.15.31:0/1848028869 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c029020 con 0x556a5de7db60 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.591+0000 7f5c20ff9640 1 -- 172.21.15.31:0/1848028869 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c1c00e920 con 0x556a5de7db60 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.591+0000 7f5c20ff9640 1 -- 172.21.15.31:0/1848028869 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c003d90 con 0x556a5de7db60 2024-04-25T02:01:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.591+0000 7f5c34439740 1 -- 172.21.15.31:0/1848028869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 msgr2=0x556a5de93f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:53.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.591+0000 7f5c34439740 1 --2- 172.21.15.31:0/1848028869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de93f10 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00eba0 tx=0x7f5c1c00f3c0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.591+0000 7f5c34439740 1 -- 172.21.15.31:0/1848028869 shutdown_connections 2024-04-25T02:01:53.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.591+0000 7f5c34439740 1 --2- 172.21.15.31:0/1848028869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de88230 0x556a5dea04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.591+0000 7f5c34439740 1 --2- 172.21.15.31:0/1848028869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de93f10 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.591+0000 7f5c34439740 1 --2- 172.21.15.31:0/1848028869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de7d240 0x556a5de87910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.591+0000 7f5c34439740 1 -- 172.21.15.31:0/1848028869 >> 172.21.15.31:0/1848028869 conn(0x556a5de79560 msgr2=0x556a5de785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:53.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.591+0000 7f5c34439740 1 -- 172.21.15.31:0/1848028869 shutdown_connections 2024-04-25T02:01:53.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.591+0000 7f5c34439740 1 -- 172.21.15.31:0/1848028869 wait complete. 2024-04-25T02:01:53.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c34439740 1 -- start start 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c34439740 1 Processor -- start 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c34439740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7d240 0x556a5de81da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c34439740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de822e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c34439740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de88230 0x556a5de82820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c34439740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1b08e0 con 0x556a5de7db60 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c34439740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1b0210 con 0x556a5de7d240 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c34439740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1ad470 con 0x556a5de88230 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c34439740 10 client.0 _ll_register_callbacks cb 0x7f5c1c00f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de822e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de822e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32780/0 (socket says 172.21.15.31:32780) 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c2affd640 1 -- 172.21.15.31:0/3548476262 learned_addr learned my addr 172.21.15.31:0/3548476262 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7d240 0x556a5de81da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c2affd640 1 -- 172.21.15.31:0/3548476262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de88230 msgr2=0x556a5de82820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c2affd640 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de88230 0x556a5de82820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c2affd640 1 -- 172.21.15.31:0/3548476262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7d240 msgr2=0x556a5de81da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c2affd640 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7d240 0x556a5de81da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.592+0000 7f5c2affd640 1 -- 172.21.15.31:0/3548476262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a5defdd40 con 0x556a5de7db60 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c2affd640 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de822e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c009690 tx=0x7f5c1c00fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51977c691a750ba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c127fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c127fc640 1 -- 172.21.15.31:0/3548476262 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c029070 con 0x556a5de7db60 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c127fc640 1 -- 172.21.15.31:0/3548476262 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8484+0+0 (secure 0 0 0) 0x7f5c1c00e680 con 0x556a5de7db60 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c34439740 1 -- 172.21.15.31:0/3548476262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c1c00e920 con 0x556a5de7db60 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c127fc640 1 -- 172.21.15.31:0/3548476262 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5c1c003b10 con 0x556a5de7db60 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c34439740 10 client.16752 fetch_fsmap learned FSMap version 33 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c34439740 1 -- 172.21.15.31:0/3548476262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556a5defeec0 con 0x556a5de7db60 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c127fc640 1 -- 172.21.15.31:0/3548476262 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c1c00e920 con 0x556a5de7db60 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c127fc640 1 -- 172.21.15.31:0/3548476262 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f5c1c023070 con 0x556a5de7db60 2024-04-25T02:01:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c34439740 10 client.16752 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:01:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c34439740 10 client.16752 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c34439740 20 client.16752 populate_metadata read hostname 'smithi031' 2024-04-25T02:01:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c34439740 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a5deff4d0 0x556a5df018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c34439740 4 client.16752 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:01:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.593+0000 7f5c34439740 1 -- 172.21.15.31:0/3548476262 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556a5dde9540 con 0x556a5deff4d0 2024-04-25T02:01:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.594+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a5deff4d0 0x556a5df018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.594+0000 7f5c2a7fc640 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a5deff4d0 0x556a5df018f0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=163dfe0326e74e55 server_cookie=4dce8b94af22acd5 in_seq=0 out_seq=0 2024-04-25T02:01:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.594+0000 7f5c127fc640 10 client.16752 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:53.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c127fc640 1 -- 172.21.15.31:0/3548476262 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556a5dde9540 con 0x556a5deff4d0 2024-04-25T02:01:53.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c127fc640 10 client.16752 handle_command_reply: tid=0 2024-04-25T02:01:53.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 1 client.16752 shutdown 2024-04-25T02:01:53.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 1 -- 172.21.15.31:0/3548476262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 msgr2=0x556a5de822e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:53.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de822e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c009690 tx=0x7f5c1c00fef0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 1 -- 172.21.15.31:0/3548476262 shutdown_connections 2024-04-25T02:01:53.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556a5deff4d0 0x556a5df018f0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a5de88230 0x556a5de82820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556a5de7db60 0x556a5de822e0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 1 --2- 172.21.15.31:0/3548476262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a5de7d240 0x556a5de81da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 1 -- 172.21.15.31:0/3548476262 >> 172.21.15.31:0/3548476262 conn(0x556a5de79560 msgr2=0x556a5de85110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:53.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c127fc640 0 client.16752 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:01:53.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 1 -- 172.21.15.31:0/3548476262 shutdown_connections 2024-04-25T02:01:53.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 1 -- 172.21.15.31:0/3548476262 wait complete. 2024-04-25T02:01:53.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.615+0000 7f5c34439740 20 client.16752 trim_cache size 0 max 16384 2024-04-25T02:01:53.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:01:53.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:01:53.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:01:53.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 -- 172.21.15.31:0/4012730426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5c180a2da0 msgr2=0x7f5c180a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5c180a2da0 0x7f5c180a5260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c240bbc30 tx=0x7f5c2402a480 comp rx=0 tx=0).stop 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 -- 172.21.15.31:0/4012730426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c2c10e020 msgr2=0x7f5c2c19e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c2c10e020 0x7f5c2c19e960 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f5c2402a9a0 tx=0x7f5c24003b90 comp rx=0 tx=0).stop 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 -- 172.21.15.31:0/4012730426 shutdown_connections 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5c180a2da0 0x7f5c180a5260 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c2c10e020 0x7f5c2c19e960 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c2c074410 0x7f5c2c19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 --2- 172.21.15.31:0/4012730426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c2c073af0 0x7f5c2c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 -- 172.21.15.31:0/4012730426 >> 172.21.15.31:0/4012730426 conn(0x7f5c2c0fd1f0 msgr2=0x7f5c2c0fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 -- 172.21.15.31:0/4012730426 shutdown_connections 2024-04-25T02:01:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:01:53.616+0000 7f5c3164d640 1 -- 172.21.15.31:0/4012730426 wait complete. 2024-04-25T02:01:54.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:53 smithi040 ceph-mon[32418]: pgmap v228: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-04-25T02:01:54.192 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:53 smithi134 ceph-mon[33098]: pgmap v228: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-04-25T02:01:54.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:53 smithi031 ceph-mon[29204]: pgmap v228: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-04-25T02:01:54.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 170119 2.46 MB/sec execute 241 sec latency 18.902 ms 2024-04-25T02:01:55.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 170577 2.47 MB/sec execute 242 sec latency 19.104 ms 2024-04-25T02:01:56.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:55 smithi040 ceph-mon[32418]: pgmap v229: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:01:56.192 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:55 smithi134 ceph-mon[33098]: pgmap v229: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:01:56.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:55 smithi031 ceph-mon[29204]: pgmap v229: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:01:56.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 170843 2.46 MB/sec execute 243 sec latency 16.513 ms 2024-04-25T02:01:57.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 171253 2.45 MB/sec execute 244 sec latency 20.944 ms 2024-04-25T02:01:58.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:57 smithi040 ceph-mon[32418]: pgmap v230: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:01:58.192 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:57 smithi134 ceph-mon[33098]: pgmap v230: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:01:58.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:57 smithi031 ceph-mon[29204]: pgmap v230: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:01:58.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 171772 2.46 MB/sec execute 245 sec latency 20.782 ms 2024-04-25T02:01:59.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 172276 2.45 MB/sec execute 246 sec latency 25.128 ms 2024-04-25T02:02:00.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:01:59 smithi040 ceph-mon[32418]: pgmap v231: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:02:00.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:01:59 smithi134 ceph-mon[33098]: pgmap v231: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:02:00.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:01:59 smithi031 ceph-mon[29204]: pgmap v231: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:02:00.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 172753 2.45 MB/sec execute 247 sec latency 23.428 ms 2024-04-25T02:02:01.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 173482 2.46 MB/sec execute 248 sec latency 14.523 ms 2024-04-25T02:02:02.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:01 smithi040 ceph-mon[32418]: pgmap v232: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-04-25T02:02:02.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:01 smithi134 ceph-mon[33098]: pgmap v232: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-04-25T02:02:02.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:01 smithi031 ceph-mon[29204]: pgmap v232: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-04-25T02:02:02.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 173893 2.47 MB/sec execute 249 sec latency 27.405 ms 2024-04-25T02:02:03.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:02 smithi040 ceph-mon[32418]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T02:02:03.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:02 smithi134 ceph-mon[33098]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T02:02:03.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:02 smithi031 ceph-mon[29204]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T02:02:03.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 174195 2.47 MB/sec execute 250 sec latency 18.760 ms 2024-04-25T02:02:04.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:03 smithi040 ceph-mon[32418]: pgmap v234: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-25T02:02:04.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:03 smithi134 ceph-mon[33098]: pgmap v234: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-25T02:02:04.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:03 smithi031 ceph-mon[29204]: pgmap v234: 97 pgs: 97 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-25T02:02:04.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 174439 2.46 MB/sec execute 251 sec latency 17.719 ms 2024-04-25T02:02:05.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 174902 2.45 MB/sec execute 252 sec latency 19.293 ms 2024-04-25T02:02:06.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:05 smithi040 ceph-mon[32418]: pgmap v235: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-25T02:02:06.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:05 smithi134 ceph-mon[33098]: pgmap v235: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-25T02:02:06.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:05 smithi031 ceph-mon[29204]: pgmap v235: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-25T02:02:06.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 175396 2.46 MB/sec execute 253 sec latency 25.033 ms 2024-04-25T02:02:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.715+0000 7ff658f34640 1 -- 172.21.15.31:0/1948697368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6541032f0 msgr2=0x7ff65410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.715+0000 7ff658f34640 1 --2- 172.21.15.31:0/1948697368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6541032f0 0x7ff65410b7d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009900 tx=0x7ff63c02a510 comp rx=0 tx=0).stop 2024-04-25T02:02:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.715+0000 7ff658f34640 1 -- 172.21.15.31:0/1948697368 shutdown_connections 2024-04-25T02:02:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.715+0000 7ff658f34640 1 --2- 172.21.15.31:0/1948697368 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff65410bd10 0x7ff65410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.715+0000 7ff658f34640 1 --2- 172.21.15.31:0/1948697368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6541032f0 0x7ff65410b7d0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.715+0000 7ff658f34640 1 --2- 172.21.15.31:0/1948697368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6541029d0 0x7ff654102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.715+0000 7ff658f34640 1 -- 172.21.15.31:0/1948697368 >> 172.21.15.31:0/1948697368 conn(0x7ff6540777c0 msgr2=0x7ff654077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.715+0000 7ff658f34640 1 -- 172.21.15.31:0/1948697368 shutdown_connections 2024-04-25T02:02:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.716+0000 7ff658f34640 1 -- 172.21.15.31:0/1948697368 wait complete. 2024-04-25T02:02:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.716+0000 7ff658f34640 1 Processor -- start 2024-04-25T02:02:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.716+0000 7ff658f34640 1 -- start start 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff658f34640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6541029d0 0x7ff654100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff658f34640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6541032f0 0x7ff654100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff658f34640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff65410bd10 0x7ff6541010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff658f34640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff6541119c0 con 0x7ff6541032f0 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff658f34640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff654111840 con 0x7ff6541029d0 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff658f34640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff654111b40 con 0x7ff65410bd10 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff64bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6541032f0 0x7ff654100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff64bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6541032f0 0x7ff654100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60130/0 (socket says 172.21.15.31:60130) 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff652d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff65410bd10 0x7ff6541010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff64bfff640 1 -- 172.21.15.31:0/63949331 learned_addr learned my addr 172.21.15.31:0/63949331 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff65259c640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6541029d0 0x7ff654100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff64bfff640 1 -- 172.21.15.31:0/63949331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff65410bd10 msgr2=0x7ff6541010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff64bfff640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff65410bd10 0x7ff6541010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff64bfff640 1 -- 172.21.15.31:0/63949331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6541029d0 msgr2=0x7ff654100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff64bfff640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6541029d0 0x7ff654100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff64bfff640 1 -- 172.21.15.31:0/63949331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6541a63c0 con 0x7ff6541032f0 2024-04-25T02:02:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff65259c640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6541029d0 0x7ff654100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff64bfff640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6541032f0 0x7ff654100b60 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff63c005940 tx=0x7ff63c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.717+0000 7ff64b7fe640 1 -- 172.21.15.31:0/63949331 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c03f030 con 0x7ff6541032f0 2024-04-25T02:02:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.718+0000 7ff658f34640 1 -- 172.21.15.31:0/63949331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6541a6650 con 0x7ff6541032f0 2024-04-25T02:02:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.718+0000 7ff64b7fe640 1 -- 172.21.15.31:0/63949331 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff63c0044c0 con 0x7ff6541032f0 2024-04-25T02:02:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.718+0000 7ff64b7fe640 1 -- 172.21.15.31:0/63949331 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c035070 con 0x7ff6541032f0 2024-04-25T02:02:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.718+0000 7ff658f34640 1 -- 172.21.15.31:0/63949331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6541a6b40 con 0x7ff6541032f0 2024-04-25T02:02:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.720+0000 7ff64b7fe640 1 -- 172.21.15.31:0/63949331 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff63c042030 con 0x7ff6541032f0 2024-04-25T02:02:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff65bc40740 1 Processor -- start 2024-04-25T02:02:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff65bc40740 1 -- start start 2024-04-25T02:02:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x559895ff1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ffbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895ffe620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff65bc40740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559895f55cb0 con 0x559895ffc230 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff65bc40740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559895efe5b0 con 0x559895ff1240 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff65bc40740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559895f51c70 con 0x559895ff1b60 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff64bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895ffe620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff652d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x559895ff1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff65259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ffbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff652d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x559895ff1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55236/0 (socket says 172.21.15.31:55236) 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff64b7fe640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff6400a2da0 0x7ff6400a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff64bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895ffe620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60142/0 (socket says 172.21.15.31:60142) 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff652d9d640 1 -- 172.21.15.31:0/2735774508 learned_addr learned my addr 172.21.15.31:0/2735774508 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.721+0000 7ff65259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ffbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36988/0 (socket says 172.21.15.31:36988) 2024-04-25T02:02:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff652d9d640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff6400a2da0 0x7ff6400a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff65259c640 1 -- 172.21.15.31:0/2735774508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 msgr2=0x559895ff1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff65259c640 1 --2- 172.21.15.31:0/2735774508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x559895ff1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff65259c640 1 -- 172.21.15.31:0/2735774508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 msgr2=0x559895ffe620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff65259c640 1 --2- 172.21.15.31:0/2735774508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895ffe620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff65259c640 1 -- 172.21.15.31:0/2735774508 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559895c6a210 con 0x559895ff1b60 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff64b7fe640 1 -- 172.21.15.31:0/63949331 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8528+0+0 (secure 0 0 0) 0x7ff63c0ef830 con 0x7ff6541032f0 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff65259c640 1 --2- 172.21.15.31:0/2735774508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ffbcf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff644009c30 tx=0x7ff6440117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6de94395a304c2b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff652d9d640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff6400a2da0 0x7ff6400a5260 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0049d0 tx=0x7ff64c004ab0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff648ff9640 1 -- 172.21.15.31:0/2735774508 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644012450 con 0x559895ff1b60 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff648ff9640 1 -- 172.21.15.31:0/2735774508 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff644012aa0 con 0x559895ff1b60 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.722+0000 7ff648ff9640 1 -- 172.21.15.31:0/2735774508 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64400faf0 con 0x559895ff1b60 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.723+0000 7ff65bc40740 1 -- 172.21.15.31:0/2735774508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 msgr2=0x559895ffbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.723+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2735774508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ffbcf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff644009c30 tx=0x7ff6440117b0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.723+0000 7ff65bc40740 1 -- 172.21.15.31:0/2735774508 shutdown_connections 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.723+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2735774508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895ffe620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.723+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2735774508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ffbcf0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.723+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2735774508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x559895ff1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.723+0000 7ff65bc40740 1 -- 172.21.15.31:0/2735774508 >> 172.21.15.31:0/2735774508 conn(0x559895fed560 msgr2=0x559895fed940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.723+0000 7ff65bc40740 1 -- 172.21.15.31:0/2735774508 shutdown_connections 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.723+0000 7ff65bc40740 1 -- 172.21.15.31:0/2735774508 wait complete. 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.723+0000 7ff65bc40740 1 -- start start 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.723+0000 7ff65bc40740 1 Processor -- start 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x559895ff91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ff9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895ff9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65bc40740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559895fff020 con 0x559895ffc230 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65bc40740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559895f11b50 con 0x559895ff1240 2024-04-25T02:02:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65bc40740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559895fff1a0 con 0x559895ff1b60 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65bc40740 10 client.0 _ll_register_callbacks cb 0x559895f4cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff652d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895ff9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x559895ff91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff652d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895ff9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60146/0 (socket says 172.21.15.31:60146) 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x559895ff91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55248/0 (socket says 172.21.15.31:55248) 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff64bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ff9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff652d9d640 1 -- 172.21.15.31:0/2702908748 learned_addr learned my addr 172.21.15.31:0/2702908748 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65259c640 1 -- 172.21.15.31:0/2702908748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 msgr2=0x559895ff9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65259c640 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ff9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65259c640 1 -- 172.21.15.31:0/2702908748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 msgr2=0x559895ff9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65259c640 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895ff9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65259c640 1 -- 172.21.15.31:0/2702908748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559895d36620 con 0x559895ff1240 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff652d9d640 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895ff9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff65259c640 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x559895ff91e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff644005830 tx=0x7ff644012e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f6d2850d1317914 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff639ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.724+0000 7ff639ffb640 1 -- 172.21.15.31:0/2702908748 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644029020 con 0x559895ff1240 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff639ffb640 1 -- 172.21.15.31:0/2702908748 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8528+0+0 (secure 0 0 0) 0x7ff644012660 con 0x559895ff1240 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff65bc40740 1 -- 172.21.15.31:0/2702908748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff644012aa0 con 0x559895ff1240 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff639ffb640 1 -- 172.21.15.31:0/2702908748 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff644034050 con 0x559895ff1240 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff65bc40740 10 client.26360 fetch_fsmap learned FSMap version 33 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff65bc40740 1 -- 172.21.15.31:0/2702908748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55989600b240 con 0x559895ff1240 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff639ffb640 1 -- 172.21.15.31:0/2702908748 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff644012aa0 con 0x559895ff1240 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff639ffb640 1 -- 172.21.15.31:0/2702908748 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff64400fa30 con 0x559895ff1240 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff65bc40740 10 client.26360 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff65bc40740 10 client.26360 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff65bc40740 20 client.26360 populate_metadata read hostname 'smithi031' 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55989600c6a0 0x55989600ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff65bc40740 4 client.26360 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff65bc40740 1 -- 172.21.15.31:0/2702908748 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559895f631f0 con 0x55989600c6a0 2024-04-25T02:02:06.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.725+0000 7ff64bfff640 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55989600c6a0 0x55989600ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.726+0000 7ff639ffb640 10 client.26360 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.726+0000 7ff64bfff640 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55989600c6a0 0x55989600ea90 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1639dbca0857f2 server_cookie=ed3d7f998c16345b in_seq=0 out_seq=0 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.726+0000 7ff639ffb640 1 -- 172.21.15.31:0/2702908748 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559895f631f0 con 0x55989600c6a0 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.726+0000 7ff639ffb640 10 client.26360 handle_command_reply: tid=0 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.726+0000 7ff65bc40740 1 client.26360 shutdown 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff65bc40740 1 -- 172.21.15.31:0/2702908748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 msgr2=0x559895ff91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x559895ff91e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff644005830 tx=0x7ff644012e80 comp rx=0 tx=0).stop 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff65bc40740 1 -- 172.21.15.31:0/2702908748 shutdown_connections 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55989600c6a0 0x55989600ea90 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895ff9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff639ffb640 0 client.26360 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ff9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2702908748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x559895ff91e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff65bc40740 1 -- 172.21.15.31:0/2702908748 >> 172.21.15.31:0/2702908748 conn(0x559895fed560 msgr2=0x559895ff3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff65bc40740 1 -- 172.21.15.31:0/2702908748 shutdown_connections 2024-04-25T02:02:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff65bc40740 1 -- 172.21.15.31:0/2702908748 wait complete. 2024-04-25T02:02:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.727+0000 7ff65bc40740 20 client.26360 trim_cache size 0 max 16384 2024-04-25T02:02:06.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.742+0000 7ff65bc40740 1 Processor -- start 2024-04-25T02:02:06.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.742+0000 7ff65bc40740 1 -- start start 2024-04-25T02:02:06.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ff1240 0x559895ffb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ff8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ffc230 0x559895ff3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65bc40740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff6541af5e0 con 0x559895ff1240 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65bc40740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff6541b1120 con 0x559895ffc230 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65bc40740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff6541afcb0 con 0x559895ff1b60 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ff1240 0x559895ffb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ff1240 0x559895ffb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60150/0 (socket says 172.21.15.31:60150) 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65259c640 1 -- 172.21.15.31:0/2837841003 learned_addr learned my addr 172.21.15.31:0/2837841003 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff652d9d640 1 --2- 172.21.15.31:0/2837841003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ffc230 0x559895ff3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff64bfff640 1 --2- 172.21.15.31:0/2837841003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ff8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65259c640 1 -- 172.21.15.31:0/2837841003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 msgr2=0x559895ff8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65259c640 1 --2- 172.21.15.31:0/2837841003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ff8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65259c640 1 -- 172.21.15.31:0/2837841003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ffc230 msgr2=0x559895ff3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65259c640 1 --2- 172.21.15.31:0/2837841003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ffc230 0x559895ff3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65259c640 1 -- 172.21.15.31:0/2837841003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559895ffa140 con 0x559895ff1240 2024-04-25T02:02:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff65259c640 1 --2- 172.21.15.31:0/2837841003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ff1240 0x559895ffb910 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff644005f10 tx=0x7ff644005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51020617f3b564a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff63bfff640 1 -- 172.21.15.31:0/2837841003 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64403d070 con 0x559895ff1240 2024-04-25T02:02:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff63bfff640 1 -- 172.21.15.31:0/2837841003 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6440092d0 con 0x559895ff1240 2024-04-25T02:02:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.743+0000 7ff63bfff640 1 -- 172.21.15.31:0/2837841003 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644029070 con 0x559895ff1240 2024-04-25T02:02:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.744+0000 7ff65bc40740 1 -- 172.21.15.31:0/2837841003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ff1240 msgr2=0x559895ffb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.744+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2837841003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ff1240 0x559895ffb910 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff644005f10 tx=0x7ff644005f40 comp rx=0 tx=0).stop 2024-04-25T02:02:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.744+0000 7ff65bc40740 1 -- 172.21.15.31:0/2837841003 shutdown_connections 2024-04-25T02:02:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.744+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2837841003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ffc230 0x559895ff3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.744+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2837841003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895ff8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.744+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2837841003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ff1240 0x559895ffb910 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.744+0000 7ff65bc40740 1 -- 172.21.15.31:0/2837841003 >> 172.21.15.31:0/2837841003 conn(0x559895fed560 msgr2=0x559895ff9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.744+0000 7ff65bc40740 1 -- 172.21.15.31:0/2837841003 shutdown_connections 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.744+0000 7ff65bc40740 1 -- 172.21.15.31:0/2837841003 wait complete. 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.744+0000 7ff65bc40740 1 -- start start 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff65bc40740 1 Processor -- start 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x5598960045e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895fff950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff65bc40740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895fffe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff65bc40740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff6541b0a50 con 0x559895ffc230 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff65bc40740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff6541b0380 con 0x559895ff1240 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff65bc40740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff6541ad5c0 con 0x559895ff1b60 2024-04-25T02:02:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff65bc40740 10 client.0 _ll_register_callbacks cb 0x7ff63c0bbae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff652d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895fffe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff652d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895fffe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60162/0 (socket says 172.21.15.31:60162) 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff65259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x5598960045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff652d9d640 1 -- 172.21.15.31:0/2455164988 learned_addr learned my addr 172.21.15.31:0/2455164988 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff64bfff640 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895fff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff652d9d640 1 -- 172.21.15.31:0/2455164988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 msgr2=0x559895fff950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff652d9d640 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895fff950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff652d9d640 1 -- 172.21.15.31:0/2455164988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 msgr2=0x5598960045e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff652d9d640 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x5598960045e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff652d9d640 1 -- 172.21.15.31:0/2455164988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598960004d0 con 0x559895ffc230 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff652d9d640 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895fffe90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0136c0 tx=0x7ff64c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bff249d0789cb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.745+0000 7ff639ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:02:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff639ffb640 1 -- 172.21.15.31:0/2455164988 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c004b20 con 0x559895ffc230 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff639ffb640 1 -- 172.21.15.31:0/2455164988 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8528+0+0 (secure 0 0 0) 0x7ff64c004cc0 con 0x559895ffc230 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff65bc40740 1 -- 172.21.15.31:0/2455164988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559895f5d540 con 0x559895ffc230 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff639ffb640 1 -- 172.21.15.31:0/2455164988 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff64c010700 con 0x559895ffc230 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff65bc40740 10 client.16770 fetch_fsmap learned FSMap version 33 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff65bc40740 1 -- 172.21.15.31:0/2455164988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5598960726e0 con 0x559895ffc230 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff639ffb640 1 -- 172.21.15.31:0/2455164988 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x559895f5d540 con 0x559895ffc230 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff639ffb640 1 -- 172.21.15.31:0/2455164988 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7ff64c01f040 con 0x559895ffc230 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff65bc40740 10 client.16770 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff65bc40740 10 client.16770 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff65bc40740 20 client.16770 populate_metadata read hostname 'smithi031' 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559896072ee0 0x559896075300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff65bc40740 4 client.16770 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:02:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff65bc40740 1 -- 172.21.15.31:0/2455164988 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559895edb8e0 con 0x559896072ee0 2024-04-25T02:02:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.746+0000 7ff65259c640 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559896072ee0 0x559896075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.747+0000 7ff65259c640 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559896072ee0 0x559896075300 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7013eccc17f0922d server_cookie=38cb8fc7956b63b9 in_seq=0 out_seq=0 2024-04-25T02:02:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.747+0000 7ff639ffb640 10 client.16770 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:06.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.772+0000 7ff639ffb640 1 -- 172.21.15.31:0/2455164988 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559895edb8e0 con 0x559896072ee0 2024-04-25T02:02:06.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.772+0000 7ff639ffb640 10 client.16770 handle_command_reply: tid=0 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 1 client.16770 shutdown 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 1 -- 172.21.15.31:0/2455164988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 msgr2=0x559895fffe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895fffe90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0136c0 tx=0x7ff64c004a40 comp rx=0 tx=0).stop 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 1 -- 172.21.15.31:0/2455164988 shutdown_connections 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559896072ee0 0x559896075300 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559895ffc230 0x559895fffe90 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559895ff1b60 0x559895fff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 1 --2- 172.21.15.31:0/2455164988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559895ff1240 0x5598960045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 1 -- 172.21.15.31:0/2455164988 >> 172.21.15.31:0/2455164988 conn(0x559895fed560 msgr2=0x559895ff7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff639ffb640 0 client.16770 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 1 -- 172.21.15.31:0/2455164988 shutdown_connections 2024-04-25T02:02:06.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 1 -- 172.21.15.31:0/2455164988 wait complete. 2024-04-25T02:02:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff65bc40740 20 client.16770 trim_cache size 0 max 16384 2024-04-25T02:02:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:02:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:02:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:02:06.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:02:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff658f34640 1 -- 172.21.15.31:0/63949331 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff6400a2da0 msgr2=0x7ff6400a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.773+0000 7ff658f34640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff6400a2da0 0x7ff6400a5260 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0049d0 tx=0x7ff64c004ab0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.774+0000 7ff658f34640 1 -- 172.21.15.31:0/63949331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6541032f0 msgr2=0x7ff654100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.774+0000 7ff658f34640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6541032f0 0x7ff654100b60 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff63c005940 tx=0x7ff63c0040c0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.774+0000 7ff658f34640 1 -- 172.21.15.31:0/63949331 shutdown_connections 2024-04-25T02:02:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.774+0000 7ff658f34640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff6400a2da0 0x7ff6400a5260 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.774+0000 7ff658f34640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff65410bd10 0x7ff6541010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.774+0000 7ff658f34640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6541032f0 0x7ff654100b60 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.774+0000 7ff658f34640 1 --2- 172.21.15.31:0/63949331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6541029d0 0x7ff654100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.774+0000 7ff658f34640 1 -- 172.21.15.31:0/63949331 >> 172.21.15.31:0/63949331 conn(0x7ff6540777c0 msgr2=0x7ff6540fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.774+0000 7ff658f34640 1 -- 172.21.15.31:0/63949331 shutdown_connections 2024-04-25T02:02:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:06.774+0000 7ff658f34640 1 -- 172.21.15.31:0/63949331 wait complete. 2024-04-25T02:02:07.313 INFO:tasks.workunit.client.0.smithi031.stdout: 1 175913 2.45 MB/sec execute 254 sec latency 21.287 ms 2024-04-25T02:02:08.043 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:07 smithi031 ceph-mon[29204]: pgmap v236: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-25T02:02:08.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:07 smithi040 ceph-mon[32418]: pgmap v236: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-25T02:02:08.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:07 smithi134 ceph-mon[33098]: pgmap v236: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-25T02:02:08.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 176388 2.45 MB/sec execute 255 sec latency 28.985 ms 2024-04-25T02:02:09.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 177101 2.47 MB/sec execute 256 sec latency 18.619 ms 2024-04-25T02:02:10.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:09 smithi040 ceph-mon[32418]: pgmap v237: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-25T02:02:10.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:09 smithi040 ceph-mon[32418]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T02:02:10.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:09 smithi134 ceph-mon[33098]: pgmap v237: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-25T02:02:10.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:09 smithi134 ceph-mon[33098]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T02:02:10.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:09 smithi031 ceph-mon[29204]: pgmap v237: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-04-25T02:02:10.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:09 smithi031 ceph-mon[29204]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T02:02:10.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 177477 2.47 MB/sec execute 257 sec latency 14.175 ms 2024-04-25T02:02:11.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 177758 2.46 MB/sec execute 258 sec latency 13.194 ms 2024-04-25T02:02:12.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:11 smithi040 ceph-mon[32418]: pgmap v239: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:12.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:11 smithi134 ceph-mon[33098]: pgmap v239: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:12.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:11 smithi031 ceph-mon[29204]: pgmap v239: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:12.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 178061 2.46 MB/sec execute 259 sec latency 18.832 ms 2024-04-25T02:02:13.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 178564 2.45 MB/sec execute 260 sec latency 17.837 ms 2024-04-25T02:02:14.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:13 smithi040 ceph-mon[32418]: pgmap v240: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T02:02:14.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:13 smithi134 ceph-mon[33098]: pgmap v240: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T02:02:14.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:13 smithi031 ceph-mon[29204]: pgmap v240: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T02:02:14.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 179027 2.46 MB/sec execute 261 sec latency 24.410 ms 2024-04-25T02:02:15.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 179616 2.45 MB/sec execute 262 sec latency 23.004 ms 2024-04-25T02:02:16.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:15 smithi040 ceph-mon[32418]: pgmap v241: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T02:02:16.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:15 smithi134 ceph-mon[33098]: pgmap v241: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T02:02:16.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:15 smithi031 ceph-mon[29204]: pgmap v241: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T02:02:16.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 180142 2.46 MB/sec execute 263 sec latency 11.495 ms 2024-04-25T02:02:17.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 180693 2.46 MB/sec execute 264 sec latency 18.853 ms 2024-04-25T02:02:18.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:17 smithi040 ceph-mon[32418]: pgmap v242: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T02:02:18.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:17 smithi134 ceph-mon[33098]: pgmap v242: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T02:02:18.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:17 smithi031 ceph-mon[29204]: pgmap v242: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T02:02:18.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 181166 2.47 MB/sec execute 265 sec latency 14.079 ms 2024-04-25T02:02:19.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 181414 2.46 MB/sec execute 266 sec latency 13.578 ms 2024-04-25T02:02:19.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.870+0000 7f5096276640 1 -- 172.21.15.31:0/4240590965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5090101030 msgr2=0x7f509010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:19.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.870+0000 7f5096276640 1 --2- 172.21.15.31:0/4240590965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5090101030 0x7f509010d4f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f5088009cd0 tx=0x7f508802a8e0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.870+0000 7f5096276640 1 -- 172.21.15.31:0/4240590965 shutdown_connections 2024-04-25T02:02:19.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.870+0000 7f5096276640 1 --2- 172.21.15.31:0/4240590965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5090101030 0x7f509010d4f0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.870+0000 7f5096276640 1 --2- 172.21.15.31:0/4240590965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5090100670 0x7f5090100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.870+0000 7f5096276640 1 --2- 172.21.15.31:0/4240590965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090106630 0x7f5090106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.870+0000 7f5096276640 1 -- 172.21.15.31:0/4240590965 >> 172.21.15.31:0/4240590965 conn(0x7f50900777c0 msgr2=0x7f5090077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:19.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.870+0000 7f5096276640 1 -- 172.21.15.31:0/4240590965 shutdown_connections 2024-04-25T02:02:19.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.870+0000 7f5096276640 1 -- 172.21.15.31:0/4240590965 wait complete. 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f5096276640 1 Processor -- start 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f5096276640 1 -- start start 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f5096276640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5090100670 0x7f5090197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f5096276640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090101030 0x7f5090198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f5096276640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5090106630 0x7f509019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f5096276640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f50901108f0 con 0x7f5090106630 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f5096276640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5090110770 con 0x7f5090100670 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f5096276640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5090110a70 con 0x7f5090101030 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f5094813640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5090106630 0x7f509019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f508ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5090100670 0x7f5090197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f508ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5090100670 0x7f5090197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:42302/0 (socket says 172.21.15.31:42302) 2024-04-25T02:02:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f508ffff640 1 -- 172.21.15.31:0/775948762 learned_addr learned my addr 172.21.15.31:0/775948762 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f5094813640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5090106630 0x7f509019c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50574/0 (socket says 172.21.15.31:50574) 2024-04-25T02:02:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.871+0000 7f508f7fe640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090101030 0x7f5090198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f508ffff640 1 -- 172.21.15.31:0/775948762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090101030 msgr2=0x7f5090198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f508ffff640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090101030 0x7f5090198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f508ffff640 1 -- 172.21.15.31:0/775948762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5090106630 msgr2=0x7f509019c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f508ffff640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5090106630 0x7f509019c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f508ffff640 1 -- 172.21.15.31:0/775948762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509019cdf0 con 0x7f5090100670 2024-04-25T02:02:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f5094813640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5090106630 0x7f509019c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f508ffff640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5090100670 0x7f5090197e50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f508000e9f0 tx=0x7f508000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f508d7fa640 1 -- 172.21.15.31:0/775948762 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508000cd80 con 0x7f5090100670 2024-04-25T02:02:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f5096276640 1 -- 172.21.15.31:0/775948762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f509019d0e0 con 0x7f5090100670 2024-04-25T02:02:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f508d7fa640 1 -- 172.21.15.31:0/775948762 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50800108e0 con 0x7f5090100670 2024-04-25T02:02:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f508d7fa640 1 -- 172.21.15.31:0/775948762 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080018920 con 0x7f5090100670 2024-04-25T02:02:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.872+0000 7f5096276640 1 -- 172.21.15.31:0/775948762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50901a4970 con 0x7f5090100670 2024-04-25T02:02:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.874+0000 7f508d7fa640 1 -- 172.21.15.31:0/775948762 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5080010a90 con 0x7f5090100670 2024-04-25T02:02:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f5098f85740 1 Processor -- start 2024-04-25T02:02:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f5098f85740 1 -- start start 2024-04-25T02:02:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f5098f85740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 0x55fe06f5f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f5098f85740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f69cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f5098f85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f6a230 0x55fe06f6c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f5098f85740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fe06ec3cb0 con 0x55fe06f5fb60 2024-04-25T02:02:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f5098f85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fe06e6c5b0 con 0x55fe06f6a230 2024-04-25T02:02:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f5098f85740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fe06ebfc70 con 0x55fe06f5f240 2024-04-25T02:02:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f508ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f69cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f508f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f6a230 0x55fe06f6c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f508ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f69cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50582/0 (socket says 172.21.15.31:50582) 2024-04-25T02:02:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f508ffff640 1 -- 172.21.15.31:0/3142754546 learned_addr learned my addr 172.21.15.31:0/3142754546 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f508d7fa640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f50780a2da0 0x7f50780a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.875+0000 7f5094813640 1 --2- 172.21.15.31:0/3142754546 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 0x55fe06f5f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f5094813640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f50780a2da0 0x7f50780a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f508f7fe640 1 -- 172.21.15.31:0/3142754546 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 msgr2=0x55fe06f5f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f508f7fe640 1 --2- 172.21.15.31:0/3142754546 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 0x55fe06f5f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f508f7fe640 1 -- 172.21.15.31:0/3142754546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 msgr2=0x55fe06f69cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f508f7fe640 1 --2- 172.21.15.31:0/3142754546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f69cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f508f7fe640 1 -- 172.21.15.31:0/3142754546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe06bd8210 con 0x55fe06f6a230 2024-04-25T02:02:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f508d7fa640 1 -- 172.21.15.31:0/775948762 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7f50800cf6c0 con 0x7f5090100670 2024-04-25T02:02:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f5094813640 1 --2- 172.21.15.31:0/3142754546 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 0x55fe06f5f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f508f7fe640 1 --2- 172.21.15.31:0/3142754546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f6a230 0x55fe06f6c620 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5084009bf0 tx=0x7f5084011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df239b83b8e2f588 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f5094813640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f50780a2da0 0x7f50780a5260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5088009430 tx=0x7f5088004a60 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f50767fc640 1 -- 172.21.15.31:0/3142754546 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084004050 con 0x55fe06f6a230 2024-04-25T02:02:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f50767fc640 1 -- 172.21.15.31:0/3142754546 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50840054d0 con 0x55fe06f6a230 2024-04-25T02:02:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.876+0000 7f50767fc640 1 -- 172.21.15.31:0/3142754546 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084004cd0 con 0x55fe06f6a230 2024-04-25T02:02:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.877+0000 7f5098f85740 1 -- 172.21.15.31:0/3142754546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f6a230 msgr2=0x55fe06f6c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.877+0000 7f5098f85740 1 --2- 172.21.15.31:0/3142754546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f6a230 0x55fe06f6c620 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5084009bf0 tx=0x7f5084011810 comp rx=0 tx=0).stop 2024-04-25T02:02:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.877+0000 7f5098f85740 1 -- 172.21.15.31:0/3142754546 shutdown_connections 2024-04-25T02:02:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.877+0000 7f5098f85740 1 --2- 172.21.15.31:0/3142754546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f6a230 0x55fe06f6c620 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.877+0000 7f5098f85740 1 --2- 172.21.15.31:0/3142754546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f69cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.877+0000 7f5098f85740 1 --2- 172.21.15.31:0/3142754546 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 0x55fe06f5f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.877+0000 7f5098f85740 1 -- 172.21.15.31:0/3142754546 >> 172.21.15.31:0/3142754546 conn(0x55fe06f5b560 msgr2=0x55fe06f5b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.877+0000 7f5098f85740 1 -- 172.21.15.31:0/3142754546 shutdown_connections 2024-04-25T02:02:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.877+0000 7f5098f85740 1 -- 172.21.15.31:0/3142754546 wait complete. 2024-04-25T02:02:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.877+0000 7f5098f85740 1 -- start start 2024-04-25T02:02:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.877+0000 7f5098f85740 1 Processor -- start 2024-04-25T02:02:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f5098f85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f5f240 0x55fe06f67180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f5098f85740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f676c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f5098f85740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f6a230 0x55fe06f67c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f5098f85740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fe06f6d020 con 0x55fe06f5fb60 2024-04-25T02:02:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f5098f85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fe06e7fb50 con 0x55fe06f5f240 2024-04-25T02:02:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f5098f85740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fe06f6d1a0 con 0x55fe06f6a230 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f508f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f676c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f5098f85740 10 client.0 _ll_register_callbacks cb 0x55fe06ebaca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f508f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f676c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50584/0 (socket says 172.21.15.31:50584) 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f508ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f5f240 0x55fe06f67180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f508f7fe640 1 -- 172.21.15.31:0/93242468 learned_addr learned my addr 172.21.15.31:0/93242468 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f5094813640 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f6a230 0x55fe06f67c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f508ffff640 1 -- 172.21.15.31:0/93242468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f6a230 msgr2=0x55fe06f67c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f508ffff640 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f6a230 0x55fe06f67c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f508ffff640 1 -- 172.21.15.31:0/93242468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 msgr2=0x55fe06f676c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f508ffff640 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f676c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f508ffff640 1 -- 172.21.15.31:0/93242468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe06ca4620 con 0x55fe06f5f240 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f508f7fe640 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f676c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f508ffff640 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f5f240 0x55fe06f67180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f508000c6a0 tx=0x7f508009bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8e5f8bae0c50607 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.878+0000 7f506f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f506f7fe640 1 -- 172.21.15.31:0/93242468 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50800cd2c0 con 0x55fe06f5f240 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f506f7fe640 1 -- 172.21.15.31:0/93242468 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7f5080010430 con 0x55fe06f5f240 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f5098f85740 1 -- 172.21.15.31:0/93242468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50840054d0 con 0x55fe06f5f240 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f506f7fe640 1 -- 172.21.15.31:0/93242468 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5080010de0 con 0x55fe06f5f240 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f5098f85740 10 client.26378 fetch_fsmap learned FSMap version 33 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f5098f85740 1 -- 172.21.15.31:0/93242468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fe06f792c0 con 0x55fe06f5f240 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f506f7fe640 1 -- 172.21.15.31:0/93242468 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50840054d0 con 0x55fe06f5f240 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f506f7fe640 1 -- 172.21.15.31:0/93242468 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f508009b270 con 0x55fe06f5f240 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f5098f85740 10 client.26378 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f5098f85740 10 client.26378 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f5098f85740 20 client.26378 populate_metadata read hostname 'smithi031' 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f5098f85740 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fe06f7a730 0x55fe06f7cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f5098f85740 4 client.26378 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.879+0000 7f5098f85740 1 -- 172.21.15.31:0/93242468 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe06ed11f0 con 0x55fe06f7a730 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.880+0000 7f508f7fe640 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fe06f7a730 0x55fe06f7cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.880+0000 7f506f7fe640 10 client.26378 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.880+0000 7f508f7fe640 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fe06f7a730 0x55fe06f7cb20 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b855ea193fa38225 server_cookie=4c654281e64f148a in_seq=0 out_seq=0 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f506f7fe640 1 -- 172.21.15.31:0/93242468 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe06ed11f0 con 0x55fe06f7a730 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f506f7fe640 10 client.26378 handle_command_reply: tid=0 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 1 client.26378 shutdown 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 1 -- 172.21.15.31:0/93242468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f5f240 msgr2=0x55fe06f67180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f5f240 0x55fe06f67180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f508000c6a0 tx=0x7f508009bbb0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 1 -- 172.21.15.31:0/93242468 shutdown_connections 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fe06f7a730 0x55fe06f7cb20 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f6a230 0x55fe06f67c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f676c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 1 --2- 172.21.15.31:0/93242468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f5f240 0x55fe06f67180 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f506f7fe640 0 client.26378 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 1 -- 172.21.15.31:0/93242468 >> 172.21.15.31:0/93242468 conn(0x55fe06f5b560 msgr2=0x55fe06f61260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 1 -- 172.21.15.31:0/93242468 shutdown_connections 2024-04-25T02:02:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 1 -- 172.21.15.31:0/93242468 wait complete. 2024-04-25T02:02:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.881+0000 7f5098f85740 20 client.26378 trim_cache size 0 max 16384 2024-04-25T02:02:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.898+0000 7f5098f85740 1 Processor -- start 2024-04-25T02:02:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.898+0000 7f5098f85740 1 -- start start 2024-04-25T02:02:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f5098f85740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5f240 0x55fe06f698b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f5098f85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f5fb60 0x55fe06f75f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f5098f85740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f6a230 0x55fe06f662e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f5098f85740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f50901aafe0 con 0x55fe06f5f240 2024-04-25T02:02:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f5098f85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f50901acb20 con 0x55fe06f5fb60 2024-04-25T02:02:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f5098f85740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f50901ab6b0 con 0x55fe06f6a230 2024-04-25T02:02:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f508ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5f240 0x55fe06f698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f508ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5f240 0x55fe06f698b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50594/0 (socket says 172.21.15.31:50594) 2024-04-25T02:02:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f508ffff640 1 -- 172.21.15.31:0/3359071226 learned_addr learned my addr 172.21.15.31:0/3359071226 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f508f7fe640 1 --2- 172.21.15.31:0/3359071226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f5fb60 0x55fe06f75f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f5094813640 1 --2- 172.21.15.31:0/3359071226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f6a230 0x55fe06f662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f508ffff640 1 -- 172.21.15.31:0/3359071226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f6a230 msgr2=0x55fe06f662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f508ffff640 1 --2- 172.21.15.31:0/3359071226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f6a230 0x55fe06f662e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f508ffff640 1 -- 172.21.15.31:0/3359071226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f5fb60 msgr2=0x55fe06f75f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f508ffff640 1 --2- 172.21.15.31:0/3359071226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f5fb60 0x55fe06f75f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f508ffff640 1 -- 172.21.15.31:0/3359071226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe06f6c960 con 0x55fe06f5f240 2024-04-25T02:02:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.899+0000 7f508ffff640 1 --2- 172.21.15.31:0/3359071226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5f240 0x55fe06f698b0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f508009bc40 tx=0x7f508000c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7bc1e83a0221bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.900+0000 7f5075ffb640 1 -- 172.21.15.31:0/3359071226 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50800cd2c0 con 0x55fe06f5f240 2024-04-25T02:02:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.900+0000 7f5075ffb640 1 -- 172.21.15.31:0/3359071226 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f508009bcb0 con 0x55fe06f5f240 2024-04-25T02:02:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.900+0000 7f5075ffb640 1 -- 172.21.15.31:0/3359071226 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508009b610 con 0x55fe06f5f240 2024-04-25T02:02:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.900+0000 7f5098f85740 1 -- 172.21.15.31:0/3359071226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5f240 msgr2=0x55fe06f698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.900+0000 7f5098f85740 1 --2- 172.21.15.31:0/3359071226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5f240 0x55fe06f698b0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f508009bc40 tx=0x7f508000c7d0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.900+0000 7f5098f85740 1 -- 172.21.15.31:0/3359071226 shutdown_connections 2024-04-25T02:02:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.900+0000 7f5098f85740 1 --2- 172.21.15.31:0/3359071226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f6a230 0x55fe06f662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.900+0000 7f5098f85740 1 --2- 172.21.15.31:0/3359071226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f5fb60 0x55fe06f75f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.900+0000 7f5098f85740 1 --2- 172.21.15.31:0/3359071226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5f240 0x55fe06f698b0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.900+0000 7f5098f85740 1 -- 172.21.15.31:0/3359071226 >> 172.21.15.31:0/3359071226 conn(0x55fe06f5b560 msgr2=0x55fe06f59d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.900+0000 7f5098f85740 1 -- 172.21.15.31:0/3359071226 shutdown_connections 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.901+0000 7f5098f85740 1 -- 172.21.15.31:0/3359071226 wait complete. 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.901+0000 7f5098f85740 1 -- start start 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.901+0000 7f5098f85740 1 Processor -- start 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.901+0000 7f5098f85740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 0x55fe06f6d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.901+0000 7f5098f85740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f6db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.901+0000 7f5098f85740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f6a230 0x55fe06fdec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.901+0000 7f5098f85740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f50901ac450 con 0x55fe06f5fb60 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.901+0000 7f5098f85740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f50901abd80 con 0x55fe06f6a230 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.901+0000 7f5098f85740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f50901a9000 con 0x55fe06f5f240 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.901+0000 7f5098f85740 10 client.0 _ll_register_callbacks cb 0x7f50800d5240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.901+0000 7f508f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f6db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f508f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f6db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50600/0 (socket says 172.21.15.31:50600) 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f508f7fe640 1 -- 172.21.15.31:0/2751342857 learned_addr learned my addr 172.21.15.31:0/2751342857 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f5094813640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f6a230 0x55fe06fdec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f508ffff640 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 0x55fe06f6d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f508f7fe640 1 -- 172.21.15.31:0/2751342857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 msgr2=0x55fe06f6d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f508f7fe640 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 0x55fe06f6d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f508f7fe640 1 -- 172.21.15.31:0/2751342857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f6a230 msgr2=0x55fe06fdec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f508f7fe640 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f6a230 0x55fe06fdec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f508f7fe640 1 -- 172.21.15.31:0/2751342857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe06fdf200 con 0x55fe06f5fb60 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f508ffff640 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 0x55fe06f6d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f508f7fe640 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f6db10 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f5084004590 tx=0x7f5084005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0bef3a2ede1eaab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f506f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f506f7fe640 1 -- 172.21.15.31:0/2751342857 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084034060 con 0x55fe06f5fb60 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f506f7fe640 1 -- 172.21.15.31:0/2751342857 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7f508402edd0 con 0x55fe06f5fb60 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f5098f85740 1 -- 172.21.15.31:0/2751342857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f508009bcb0 con 0x55fe06f5fb60 2024-04-25T02:02:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f506f7fe640 1 -- 172.21.15.31:0/2751342857 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5084011d10 con 0x55fe06f5fb60 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f5098f85740 10 client.16788 fetch_fsmap learned FSMap version 33 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.902+0000 7f5098f85740 1 -- 172.21.15.31:0/2751342857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fe06fe0520 con 0x55fe06f5fb60 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.903+0000 7f506f7fe640 1 -- 172.21.15.31:0/2751342857 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f508009bcb0 con 0x55fe06f5fb60 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.903+0000 7f506f7fe640 1 -- 172.21.15.31:0/2751342857 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f50840039d0 con 0x55fe06f5fb60 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.903+0000 7f5098f85740 10 client.16788 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.903+0000 7f5098f85740 10 client.16788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.903+0000 7f5098f85740 20 client.16788 populate_metadata read hostname 'smithi031' 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.903+0000 7f5098f85740 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fe06fe0b40 0x55fe06fe2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.903+0000 7f5098f85740 4 client.16788 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.903+0000 7f5098f85740 1 -- 172.21.15.31:0/2751342857 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fe06ecb540 con 0x55fe06fe0b40 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.903+0000 7f508ffff640 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fe06fe0b40 0x55fe06fe2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.903+0000 7f508ffff640 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fe06fe0b40 0x55fe06fe2f60 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1851b107d5349ee9 server_cookie=109ba3b571c385a7 in_seq=0 out_seq=0 2024-04-25T02:02:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.903+0000 7f506f7fe640 10 client.16788 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.923+0000 7f506f7fe640 1 -- 172.21.15.31:0/2751342857 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fe06ecb540 con 0x55fe06fe0b40 2024-04-25T02:02:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.923+0000 7f506f7fe640 10 client.16788 handle_command_reply: tid=0 2024-04-25T02:02:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.923+0000 7f5098f85740 1 client.16788 shutdown 2024-04-25T02:02:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5098f85740 1 -- 172.21.15.31:0/2751342857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 msgr2=0x55fe06f6db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5098f85740 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f6db10 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f5084004590 tx=0x7f5084005d50 comp rx=0 tx=0).stop 2024-04-25T02:02:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5098f85740 1 -- 172.21.15.31:0/2751342857 shutdown_connections 2024-04-25T02:02:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5098f85740 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fe06fe0b40 0x55fe06fe2f60 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5098f85740 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fe06f6a230 0x55fe06fdec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5098f85740 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fe06f5fb60 0x55fe06f6db10 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f506f7fe640 0 client.16788 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5098f85740 1 --2- 172.21.15.31:0/2751342857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe06f5f240 0x55fe06f6d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5098f85740 1 -- 172.21.15.31:0/2751342857 >> 172.21.15.31:0/2751342857 conn(0x55fe06f5b560 msgr2=0x55fe06f5a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5098f85740 1 -- 172.21.15.31:0/2751342857 shutdown_connections 2024-04-25T02:02:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5098f85740 1 -- 172.21.15.31:0/2751342857 wait complete. 2024-04-25T02:02:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5098f85740 20 client.16788 trim_cache size 0 max 16384 2024-04-25T02:02:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:02:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:02:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:02:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:02:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5096276640 1 -- 172.21.15.31:0/775948762 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f50780a2da0 msgr2=0x7f50780a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5096276640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f50780a2da0 0x7f50780a5260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5088009430 tx=0x7f5088004a60 comp rx=0 tx=0).stop 2024-04-25T02:02:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5096276640 1 -- 172.21.15.31:0/775948762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5090100670 msgr2=0x7f5090197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.924+0000 7f5096276640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5090100670 0x7f5090197e50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f508000e9f0 tx=0x7f508000eeb0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.925+0000 7f5096276640 1 -- 172.21.15.31:0/775948762 shutdown_connections 2024-04-25T02:02:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.925+0000 7f5096276640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f50780a2da0 0x7f50780a5260 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.925+0000 7f5096276640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5090106630 0x7f509019c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.925+0000 7f5096276640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5090101030 0x7f5090198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.925+0000 7f5096276640 1 --2- 172.21.15.31:0/775948762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5090100670 0x7f5090197e50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.925+0000 7f5096276640 1 -- 172.21.15.31:0/775948762 >> 172.21.15.31:0/775948762 conn(0x7f50900777c0 msgr2=0x7f50900ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.925+0000 7f5096276640 1 -- 172.21.15.31:0/775948762 shutdown_connections 2024-04-25T02:02:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:19.925+0000 7f5096276640 1 -- 172.21.15.31:0/775948762 wait complete. 2024-04-25T02:02:20.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:19 smithi040 ceph-mon[32418]: pgmap v243: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T02:02:20.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:19 smithi134 ceph-mon[33098]: pgmap v243: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T02:02:20.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:19 smithi031 ceph-mon[29204]: pgmap v243: 97 pgs: 97 active+clean; 102 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T02:02:20.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 181797 2.46 MB/sec execute 267 sec latency 72.213 ms 2024-04-25T02:02:21.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 182316 2.46 MB/sec execute 268 sec latency 18.518 ms 2024-04-25T02:02:22.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:21 smithi040 ceph-mon[32418]: pgmap v244: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:22.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:21 smithi134 ceph-mon[33098]: pgmap v244: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:22.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:21 smithi031 ceph-mon[29204]: pgmap v244: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:22.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 182767 2.45 MB/sec execute 269 sec latency 25.730 ms 2024-04-25T02:02:23.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 183282 2.45 MB/sec execute 270 sec latency 25.977 ms 2024-04-25T02:02:24.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:23 smithi040 ceph-mon[32418]: pgmap v245: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:24.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:23 smithi134 ceph-mon[33098]: pgmap v245: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:24.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:23 smithi031 ceph-mon[29204]: pgmap v245: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:24.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 183880 2.46 MB/sec execute 271 sec latency 18.615 ms 2024-04-25T02:02:25.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 184301 2.46 MB/sec execute 272 sec latency 18.754 ms 2024-04-25T02:02:26.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:25 smithi040 ceph-mon[32418]: pgmap v246: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:02:26.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:25 smithi134 ceph-mon[33098]: pgmap v246: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:02:26.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:25 smithi031 ceph-mon[29204]: pgmap v246: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:02:26.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 184746 2.47 MB/sec execute 273 sec latency 46.219 ms 2024-04-25T02:02:27.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 185011 2.46 MB/sec execute 274 sec latency 12.100 ms 2024-04-25T02:02:28.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:27 smithi040 ceph-mon[32418]: pgmap v247: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:02:28.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:27 smithi134 ceph-mon[33098]: pgmap v247: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:02:28.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:27 smithi031 ceph-mon[29204]: pgmap v247: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:02:28.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 185453 2.45 MB/sec execute 275 sec latency 16.562 ms 2024-04-25T02:02:29.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 185949 2.46 MB/sec execute 276 sec latency 23.704 ms 2024-04-25T02:02:30.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:29 smithi040 ceph-mon[32418]: pgmap v248: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:02:30.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:29 smithi134 ceph-mon[33098]: pgmap v248: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:02:30.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:29 smithi031 ceph-mon[29204]: pgmap v248: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.4 MiB/s wr, 118 op/s 2024-04-25T02:02:30.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 186374 2.45 MB/sec execute 277 sec latency 81.513 ms 2024-04-25T02:02:31.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 186898 2.45 MB/sec execute 278 sec latency 24.146 ms 2024-04-25T02:02:32.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:31 smithi040 ceph-mon[32418]: pgmap v249: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:32.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:31 smithi134 ceph-mon[33098]: pgmap v249: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:32.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:31 smithi031 ceph-mon[29204]: pgmap v249: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:32.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 187649 2.46 MB/sec execute 279 sec latency 11.986 ms 2024-04-25T02:02:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.022+0000 7f247774f640 1 -- 172.21.15.31:0/576427761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f247010e120 msgr2=0x7f2470110510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.022+0000 7f247774f640 1 --2- 172.21.15.31:0/576427761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f247010e120 0x7f2470110510 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2468009ca0 tx=0x7f246802a8c0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.022+0000 7f247774f640 1 -- 172.21.15.31:0/576427761 shutdown_connections 2024-04-25T02:02:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.022+0000 7f247774f640 1 --2- 172.21.15.31:0/576427761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f247010e120 0x7f2470110510 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.022+0000 7f247774f640 1 --2- 172.21.15.31:0/576427761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2470100e90 0x7f247010dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.022+0000 7f247774f640 1 --2- 172.21.15.31:0/576427761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2470100570 0x7f2470100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.022+0000 7f247774f640 1 -- 172.21.15.31:0/576427761 >> 172.21.15.31:0/576427761 conn(0x7f24700777c0 msgr2=0x7f2470077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.023+0000 7f247774f640 1 -- 172.21.15.31:0/576427761 shutdown_connections 2024-04-25T02:02:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.023+0000 7f247774f640 1 -- 172.21.15.31:0/576427761 wait complete. 2024-04-25T02:02:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.023+0000 7f247774f640 1 Processor -- start 2024-04-25T02:02:33.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.023+0000 7f247774f640 1 -- start start 2024-04-25T02:02:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.023+0000 7f247774f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2470100570 0x7f247019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.023+0000 7f247774f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2470100e90 0x7f247019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f247774f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f247010e120 0x7f247019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f247774f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2470113be0 con 0x7f2470100e90 2024-04-25T02:02:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f247774f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2470113a60 con 0x7f2470100570 2024-04-25T02:02:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f247774f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2470113d60 con 0x7f247010e120 2024-04-25T02:02:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f2474cea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2470100e90 0x7f247019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f24754eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2470100570 0x7f247019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f2474cea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2470100e90 0x7f247019a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57226/0 (socket says 172.21.15.31:57226) 2024-04-25T02:02:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f2474cea640 1 -- 172.21.15.31:0/3352986416 learned_addr learned my addr 172.21.15.31:0/3352986416 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f24754eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2470100570 0x7f247019a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52508/0 (socket says 172.21.15.31:52508) 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f2475cec640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f247010e120 0x7f247019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f2474cea640 1 -- 172.21.15.31:0/3352986416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f247010e120 msgr2=0x7f247019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f2474cea640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f247010e120 0x7f247019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f2474cea640 1 -- 172.21.15.31:0/3352986416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2470100570 msgr2=0x7f247019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f2474cea640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2470100570 0x7f247019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f2474cea640 1 -- 172.21.15.31:0/3352986416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247019f0a0 con 0x7f2470100e90 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f24754eb640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2470100570 0x7f247019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.024+0000 7f2474cea640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2470100e90 0x7f247019a640 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f246000e9f0 tx=0x7f246000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.025+0000 7f24667fc640 1 -- 172.21.15.31:0/3352986416 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246000cd80 con 0x7f2470100e90 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.025+0000 7f247774f640 1 -- 172.21.15.31:0/3352986416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247019f390 con 0x7f2470100e90 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.025+0000 7f24667fc640 1 -- 172.21.15.31:0/3352986416 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24600108e0 con 0x7f2470100e90 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.025+0000 7f24667fc640 1 -- 172.21.15.31:0/3352986416 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2460018920 con 0x7f2470100e90 2024-04-25T02:02:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.025+0000 7f247774f640 1 -- 172.21.15.31:0/3352986416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24701a6c20 con 0x7f2470100e90 2024-04-25T02:02:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.027+0000 7f24667fc640 1 -- 172.21.15.31:0/3352986416 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2460010a90 con 0x7f2470100e90 2024-04-25T02:02:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f247a456740 1 Processor -- start 2024-04-25T02:02:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f247a456740 1 -- start start 2024-04-25T02:02:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f247a456740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a0f8240 0x56239a0f8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f247a456740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f247a456740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a103230 0x56239a105620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f247a456740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56239a05ccb0 con 0x56239a103230 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f247a456740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56239a0055b0 con 0x56239a0f8b60 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f247a456740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56239a058c70 con 0x56239a0f8240 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f24667fc640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f245c0a2da0 0x7f245c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f2474cea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a103230 0x56239a105620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f24754eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f24754eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52524/0 (socket says 172.21.15.31:52524) 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f24754eb640 1 -- 172.21.15.31:0/127850072 learned_addr learned my addr 172.21.15.31:0/127850072 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f2475cec640 1 --2- 172.21.15.31:0/127850072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a0f8240 0x56239a0f8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f2475cec640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f245c0a2da0 0x7f245c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.028+0000 7f24667fc640 1 -- 172.21.15.31:0/3352986416 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7f24600cf6c0 con 0x7f2470100e90 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.029+0000 7f24754eb640 1 -- 172.21.15.31:0/127850072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a0f8240 msgr2=0x56239a0f8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.029+0000 7f24754eb640 1 --2- 172.21.15.31:0/127850072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a0f8240 0x56239a0f8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.029+0000 7f24754eb640 1 -- 172.21.15.31:0/127850072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a103230 msgr2=0x56239a105620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.029+0000 7f24754eb640 1 --2- 172.21.15.31:0/127850072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a103230 0x56239a105620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.029+0000 7f24754eb640 1 -- 172.21.15.31:0/127850072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562399d71210 con 0x56239a0f8b60 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.029+0000 7f24754eb640 1 --2- 172.21.15.31:0/127850072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a102cf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f246c0115f0 tx=0x7f246c011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40f72b32428e5bb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.029+0000 7f2475cec640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f245c0a2da0 0x7f245c0a5260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2468005e30 tx=0x7f2468004a60 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.029+0000 7f245b7fe640 1 -- 172.21.15.31:0/127850072 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c01e070 con 0x56239a0f8b60 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.029+0000 7f245b7fe640 1 -- 172.21.15.31:0/127850072 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f246c00faf0 con 0x56239a0f8b60 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.029+0000 7f245b7fe640 1 -- 172.21.15.31:0/127850072 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c019bc0 con 0x56239a0f8b60 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.030+0000 7f247a456740 1 -- 172.21.15.31:0/127850072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 msgr2=0x56239a102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.030+0000 7f247a456740 1 --2- 172.21.15.31:0/127850072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a102cf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f246c0115f0 tx=0x7f246c011ab0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.030+0000 7f247a456740 1 -- 172.21.15.31:0/127850072 shutdown_connections 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.030+0000 7f247a456740 1 --2- 172.21.15.31:0/127850072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a103230 0x56239a105620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.030+0000 7f247a456740 1 --2- 172.21.15.31:0/127850072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a102cf0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.030+0000 7f247a456740 1 --2- 172.21.15.31:0/127850072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a0f8240 0x56239a0f8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.030+0000 7f247a456740 1 -- 172.21.15.31:0/127850072 >> 172.21.15.31:0/127850072 conn(0x56239a0f4560 msgr2=0x56239a0f4940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:33.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.030+0000 7f247a456740 1 -- 172.21.15.31:0/127850072 shutdown_connections 2024-04-25T02:02:33.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.030+0000 7f247a456740 1 -- 172.21.15.31:0/127850072 wait complete. 2024-04-25T02:02:33.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.030+0000 7f247a456740 1 -- start start 2024-04-25T02:02:33.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.030+0000 7f247a456740 1 Processor -- start 2024-04-25T02:02:33.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f247a456740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a0f8240 0x56239a1001b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f247a456740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a1006f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f247a456740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a103230 0x56239a100c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f247a456740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56239a106020 con 0x56239a103230 2024-04-25T02:02:33.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f247a456740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56239a018b50 con 0x56239a0f8b60 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f247a456740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56239a1061a0 con 0x56239a0f8240 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f247a456740 10 client.0 _ll_register_callbacks cb 0x56239a053ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2474cea640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a1006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2475cec640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a103230 0x56239a100c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2474cea640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a1006f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52534/0 (socket says 172.21.15.31:52534) 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2474cea640 1 -- 172.21.15.31:0/194589667 learned_addr learned my addr 172.21.15.31:0/194589667 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f24754eb640 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a0f8240 0x56239a1001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2474cea640 1 -- 172.21.15.31:0/194589667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a0f8240 msgr2=0x56239a1001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2474cea640 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a0f8240 0x56239a1001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2474cea640 1 -- 172.21.15.31:0/194589667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a103230 msgr2=0x56239a100c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2474cea640 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a103230 0x56239a100c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2474cea640 1 -- 172.21.15.31:0/194589667 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562399e3d620 con 0x56239a0f8b60 2024-04-25T02:02:33.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2475cec640 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a103230 0x56239a100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2474cea640 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a1006f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2460098680 tx=0x7f2460010f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=113a475fcd5f1513 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.031+0000 7f2458ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f2458ff9640 1 -- 172.21.15.31:0/194589667 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24600b2030 con 0x56239a0f8b60 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f2458ff9640 1 -- 172.21.15.31:0/194589667 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7f24600b4e00 con 0x56239a0f8b60 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f247a456740 1 -- 172.21.15.31:0/194589667 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f246c00faf0 con 0x56239a0f8b60 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f2458ff9640 1 -- 172.21.15.31:0/194589667 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2460026040 con 0x56239a0f8b60 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f247a456740 10 client.26396 fetch_fsmap learned FSMap version 33 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f247a456740 1 -- 172.21.15.31:0/194589667 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56239a112200 con 0x56239a0f8b60 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f2458ff9640 1 -- 172.21.15.31:0/194589667 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f246c00faf0 con 0x56239a0f8b60 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f2458ff9640 1 -- 172.21.15.31:0/194589667 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f2460097b90 con 0x56239a0f8b60 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f247a456740 10 client.26396 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f247a456740 10 client.26396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f247a456740 20 client.26396 populate_metadata read hostname 'smithi031' 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f247a456740 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56239a113710 0x56239a115b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f247a456740 4 client.26396 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f247a456740 1 -- 172.21.15.31:0/194589667 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56239a06a1f0 con 0x56239a113710 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.032+0000 7f24754eb640 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56239a113710 0x56239a115b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.033+0000 7f24754eb640 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56239a113710 0x56239a115b00 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd5594ae85d51557 server_cookie=b9929cfa2ab85b74 in_seq=0 out_seq=0 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.033+0000 7f2458ff9640 10 client.26396 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f2458ff9640 1 -- 172.21.15.31:0/194589667 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56239a06a1f0 con 0x56239a113710 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f2458ff9640 10 client.26396 handle_command_reply: tid=0 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 1 client.26396 shutdown 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 1 -- 172.21.15.31:0/194589667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 msgr2=0x56239a1006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a1006f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2460098680 tx=0x7f2460010f20 comp rx=0 tx=0).stop 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 1 -- 172.21.15.31:0/194589667 shutdown_connections 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56239a113710 0x56239a115b00 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a103230 0x56239a100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f2458ff9640 0 client.26396 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:33.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8b60 0x56239a1006f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 1 --2- 172.21.15.31:0/194589667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a0f8240 0x56239a1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 1 -- 172.21.15.31:0/194589667 >> 172.21.15.31:0/194589667 conn(0x56239a0f4560 msgr2=0x56239a0fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:33.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 1 -- 172.21.15.31:0/194589667 shutdown_connections 2024-04-25T02:02:33.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 1 -- 172.21.15.31:0/194589667 wait complete. 2024-04-25T02:02:33.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.034+0000 7f247a456740 20 client.26396 trim_cache size 0 max 16384 2024-04-25T02:02:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.049+0000 7f247a456740 1 Processor -- start 2024-04-25T02:02:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.049+0000 7f247a456740 1 -- start start 2024-04-25T02:02:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.049+0000 7f247a456740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.049+0000 7f247a456740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 0x56239a0fa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.049+0000 7f247a456740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a103230 0x56239a0f3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.049+0000 7f247a456740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f24701af430 con 0x56239a0f8b60 2024-04-25T02:02:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.049+0000 7f247a456740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24701b0f70 con 0x56239a0f8240 2024-04-25T02:02:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.049+0000 7f247a456740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f24701afb00 con 0x56239a103230 2024-04-25T02:02:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f24754eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f2474cea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 0x56239a0fa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f24754eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52538/0 (socket says 172.21.15.31:52538) 2024-04-25T02:02:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f24754eb640 1 -- 172.21.15.31:0/3260199260 learned_addr learned my addr 172.21.15.31:0/3260199260 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f2475cec640 1 --2- 172.21.15.31:0/3260199260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a103230 0x56239a0f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f24754eb640 1 -- 172.21.15.31:0/3260199260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a103230 msgr2=0x56239a0f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f24754eb640 1 --2- 172.21.15.31:0/3260199260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a103230 0x56239a0f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f24754eb640 1 -- 172.21.15.31:0/3260199260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 msgr2=0x56239a0fa9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f24754eb640 1 --2- 172.21.15.31:0/3260199260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 0x56239a0fa9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f24754eb640 1 -- 172.21.15.31:0/3260199260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56239a105990 con 0x56239a0f8240 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f2474cea640 1 --2- 172.21.15.31:0/3260199260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 0x56239a0fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f24754eb640 1 --2- 172.21.15.31:0/3260199260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10eee0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f246c019770 tx=0x7f246c0096e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7720089853aa4805 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f245affd640 1 -- 172.21.15.31:0/3260199260 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c02f070 con 0x56239a0f8240 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f245affd640 1 -- 172.21.15.31:0/3260199260 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f246c0054f0 con 0x56239a0f8240 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.050+0000 7f245affd640 1 -- 172.21.15.31:0/3260199260 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c01e040 con 0x56239a0f8240 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.051+0000 7f247a456740 1 -- 172.21.15.31:0/3260199260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 msgr2=0x56239a10eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.051+0000 7f247a456740 1 --2- 172.21.15.31:0/3260199260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10eee0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f246c019770 tx=0x7f246c0096e0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.051+0000 7f247a456740 1 -- 172.21.15.31:0/3260199260 shutdown_connections 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.051+0000 7f247a456740 1 --2- 172.21.15.31:0/3260199260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a103230 0x56239a0f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.051+0000 7f247a456740 1 --2- 172.21.15.31:0/3260199260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 0x56239a0fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.051+0000 7f247a456740 1 --2- 172.21.15.31:0/3260199260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10eee0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.051+0000 7f247a456740 1 -- 172.21.15.31:0/3260199260 >> 172.21.15.31:0/3260199260 conn(0x56239a0f4560 msgr2=0x56239a0f2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.051+0000 7f247a456740 1 -- 172.21.15.31:0/3260199260 shutdown_connections 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.051+0000 7f247a456740 1 -- 172.21.15.31:0/3260199260 wait complete. 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.051+0000 7f247a456740 1 -- start start 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.051+0000 7f247a456740 1 Processor -- start 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f247a456740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f247a456740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 0x56239a0fcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f247a456740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a103230 0x56239a0fd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f247a456740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f24701b08a0 con 0x56239a0f8b60 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f247a456740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24701b01d0 con 0x56239a0f8240 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f247a456740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f24701ad430 con 0x56239a103230 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f247a456740 10 client.0 _ll_register_callbacks cb 0x7f246009ba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f2474cea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 0x56239a0fcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f24754eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f2474cea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 0x56239a0fcd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57272/0 (socket says 172.21.15.31:57272) 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f2474cea640 1 -- 172.21.15.31:0/3685631017 learned_addr learned my addr 172.21.15.31:0/3685631017 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f24754eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52542/0 (socket says 172.21.15.31:52542) 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f2475cec640 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a103230 0x56239a0fd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f2474cea640 1 -- 172.21.15.31:0/3685631017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a103230 msgr2=0x56239a0fd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f2474cea640 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a103230 0x56239a0fd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f2474cea640 1 -- 172.21.15.31:0/3685631017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 msgr2=0x56239a10b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f2474cea640 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f2474cea640 1 -- 172.21.15.31:0/3685631017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56239a178c80 con 0x56239a0f8b60 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f2475cec640 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a103230 0x56239a0fd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f24754eb640 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.052+0000 7f2474cea640 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 0x56239a0fcd70 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2460098110 tx=0x7f2460010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6431bb38695a488 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f2458ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:02:33.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f2458ff9640 1 -- 172.21.15.31:0/3685631017 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24600b2050 con 0x56239a0f8b60 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f2458ff9640 1 -- 172.21.15.31:0/3685631017 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7f246009b100 con 0x56239a0f8b60 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f247a456740 1 -- 172.21.15.31:0/3685631017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f246c0054f0 con 0x56239a0f8b60 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f2458ff9640 1 -- 172.21.15.31:0/3685631017 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f246009b300 con 0x56239a0f8b60 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f247a456740 10 client.16806 fetch_fsmap learned FSMap version 33 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f247a456740 1 -- 172.21.15.31:0/3685631017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56239a179e00 con 0x56239a0f8b60 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f2458ff9640 1 -- 172.21.15.31:0/3685631017 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f246c0054f0 con 0x56239a0f8b60 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f2458ff9640 1 -- 172.21.15.31:0/3685631017 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f24600ac070 con 0x56239a0f8b60 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f247a456740 10 client.16806 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f247a456740 10 client.16806 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f247a456740 20 client.16806 populate_metadata read hostname 'smithi031' 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f247a456740 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56239a113710 0x56239a17c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f247a456740 4 client.16806 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f247a456740 1 -- 172.21.15.31:0/3685631017 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56239a064540 con 0x56239a113710 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.053+0000 7f24754eb640 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56239a113710 0x56239a17c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:33.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.054+0000 7f24754eb640 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56239a113710 0x56239a17c400 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a79bab299b986f9c server_cookie=c4e7d1a80b4eed07 in_seq=0 out_seq=0 2024-04-25T02:02:33.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.054+0000 7f2458ff9640 10 client.16806 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f2458ff9640 1 -- 172.21.15.31:0/3685631017 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56239a064540 con 0x56239a113710 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f2458ff9640 10 client.16806 handle_command_reply: tid=0 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f247a456740 1 client.16806 shutdown 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f247a456740 1 -- 172.21.15.31:0/3685631017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 msgr2=0x56239a0fcd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f247a456740 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 0x56239a0fcd70 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2460098110 tx=0x7f2460010660 comp rx=0 tx=0).stop 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f247a456740 1 -- 172.21.15.31:0/3685631017 shutdown_connections 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f247a456740 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56239a113710 0x56239a17c400 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f247a456740 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56239a103230 0x56239a0fd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f247a456740 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56239a0f8b60 0x56239a0fcd70 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f2458ff9640 0 client.16806 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f247a456740 1 --2- 172.21.15.31:0/3685631017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56239a0f8240 0x56239a10b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.077+0000 7f247a456740 1 -- 172.21.15.31:0/3685631017 >> 172.21.15.31:0/3685631017 conn(0x56239a0f4560 msgr2=0x56239a0f35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247a456740 1 -- 172.21.15.31:0/3685631017 shutdown_connections 2024-04-25T02:02:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247a456740 1 -- 172.21.15.31:0/3685631017 wait complete. 2024-04-25T02:02:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247a456740 20 client.16806 trim_cache size 0 max 16384 2024-04-25T02:02:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:02:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:02:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:02:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 -- 172.21.15.31:0/3352986416 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f245c0a2da0 msgr2=0x7f245c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f245c0a2da0 0x7f245c0a5260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2468005e30 tx=0x7f2468004a60 comp rx=0 tx=0).stop 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 -- 172.21.15.31:0/3352986416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2470100e90 msgr2=0x7f247019a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2470100e90 0x7f247019a640 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f246000e9f0 tx=0x7f246000eeb0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 -- 172.21.15.31:0/3352986416 shutdown_connections 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f245c0a2da0 0x7f245c0a5260 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f247010e120 0x7f247019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2470100e90 0x7f247019a640 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 --2- 172.21.15.31:0/3352986416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2470100570 0x7f247019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 -- 172.21.15.31:0/3352986416 >> 172.21.15.31:0/3352986416 conn(0x7f24700777c0 msgr2=0x7f247010f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 -- 172.21.15.31:0/3352986416 shutdown_connections 2024-04-25T02:02:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:33.078+0000 7f247774f640 1 -- 172.21.15.31:0/3352986416 wait complete. 2024-04-25T02:02:33.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 188028 2.47 MB/sec execute 280 sec latency 24.498 ms 2024-04-25T02:02:34.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:33 smithi040 ceph-mon[32418]: pgmap v250: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T02:02:34.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:33 smithi134 ceph-mon[33098]: pgmap v250: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T02:02:34.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:33 smithi031 ceph-mon[29204]: pgmap v250: 97 pgs: 97 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T02:02:34.314 INFO:tasks.workunit.client.0.smithi031.stdout: 1 188377 2.47 MB/sec execute 281 sec latency 16.161 ms 2024-04-25T02:02:35.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 188625 2.46 MB/sec execute 282 sec latency 19.838 ms 2024-04-25T02:02:36.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:35 smithi040 ceph-mon[32418]: pgmap v251: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:02:36.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:35 smithi134 ceph-mon[33098]: pgmap v251: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:02:36.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:35 smithi031 ceph-mon[29204]: pgmap v251: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-04-25T02:02:36.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 189149 2.46 MB/sec execute 283 sec latency 12.380 ms 2024-04-25T02:02:37.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 189646 2.46 MB/sec execute 284 sec latency 22.315 ms 2024-04-25T02:02:38.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:37 smithi040 ceph-mon[32418]: pgmap v252: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:38.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:37 smithi134 ceph-mon[33098]: pgmap v252: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:38.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:37 smithi031 ceph-mon[29204]: pgmap v252: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:38.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 190184 2.45 MB/sec execute 285 sec latency 25.627 ms 2024-04-25T02:02:39.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 190612 2.45 MB/sec execute 286 sec latency 19.249 ms 2024-04-25T02:02:40.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:39 smithi040 ceph-mon[32418]: pgmap v253: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:40.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:39 smithi134 ceph-mon[33098]: pgmap v253: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:40.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:39 smithi031 ceph-mon[29204]: pgmap v253: 97 pgs: 97 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:40.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 191312 2.47 MB/sec execute 287 sec latency 21.173 ms 2024-04-25T02:02:41.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 191764 2.47 MB/sec execute 288 sec latency 13.290 ms 2024-04-25T02:02:42.163 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:41 smithi040 ceph-mon[32418]: pgmap v254: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.9 MiB/s wr, 177 op/s 2024-04-25T02:02:42.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:41 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:02:42.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:41 smithi134 ceph-mon[33098]: pgmap v254: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.9 MiB/s wr, 177 op/s 2024-04-25T02:02:42.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:41 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:02:42.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:41 smithi031 ceph-mon[29204]: pgmap v254: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.9 MiB/s wr, 177 op/s 2024-04-25T02:02:42.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:41 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:02:42.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 191976 2.46 MB/sec execute 289 sec latency 14.956 ms 2024-04-25T02:02:43.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:42 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:02:43.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:42 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:02:43.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:42 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:02:43.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:42 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:02:43.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:42 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:02:43.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:42 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:02:43.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:42 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:02:43.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:42 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:02:43.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:42 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:02:43.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 192331 2.46 MB/sec execute 290 sec latency 18.168 ms 2024-04-25T02:02:44.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 192810 2.45 MB/sec execute 291 sec latency 12.802 ms 2024-04-25T02:02:44.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:44 smithi040 ceph-mon[32418]: pgmap v255: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-25T02:02:44.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:44 smithi134 ceph-mon[33098]: pgmap v255: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-25T02:02:44.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:44 smithi031 ceph-mon[29204]: pgmap v255: 97 pgs: 97 active+clean; 109 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-25T02:02:45.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 193323 2.46 MB/sec execute 292 sec latency 22.974 ms 2024-04-25T02:02:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.178+0000 7f8909b9e640 1 -- 172.21.15.31:0/230679631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8904102120 msgr2=0x7f890410e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.178+0000 7f8909b9e640 1 --2- 172.21.15.31:0/230679631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8904102120 0x7f890410e630 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f88fc009cd0 tx=0x7f88fc02a8e0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.178+0000 7f8909b9e640 1 -- 172.21.15.31:0/230679631 shutdown_connections 2024-04-25T02:02:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.178+0000 7f8909b9e640 1 --2- 172.21.15.31:0/230679631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8904102120 0x7f890410e630 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.178+0000 7f8909b9e640 1 --2- 172.21.15.31:0/230679631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8904101760 0x7f8904101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.178+0000 7f8909b9e640 1 --2- 172.21.15.31:0/230679631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8904107770 0x7f8904107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.178+0000 7f8909b9e640 1 -- 172.21.15.31:0/230679631 >> 172.21.15.31:0/230679631 conn(0x7f89040fd1f0 msgr2=0x7f89040ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.178+0000 7f8909b9e640 1 -- 172.21.15.31:0/230679631 shutdown_connections 2024-04-25T02:02:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.178+0000 7f8909b9e640 1 -- 172.21.15.31:0/230679631 wait complete. 2024-04-25T02:02:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8909b9e640 1 Processor -- start 2024-04-25T02:02:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8909b9e640 1 -- start start 2024-04-25T02:02:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8909b9e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8904101760 0x7f8904073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8909b9e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8904102120 0x7f8904073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8909b9e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8904107770 0x7f89040744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8909b9e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8904111a80 con 0x7f8904107770 2024-04-25T02:02:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8909b9e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8904111900 con 0x7f8904102120 2024-04-25T02:02:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8909b9e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8904111c00 con 0x7f8904101760 2024-04-25T02:02:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8903fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8904107770 0x7f89040744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8903fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8904107770 0x7f89040744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35936/0 (socket says 172.21.15.31:35936) 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8903fff640 1 -- 172.21.15.31:0/2373266813 learned_addr learned my addr 172.21.15.31:0/2373266813 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f8902ffd640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8904102120 0x7f8904073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.179+0000 7f89037fe640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8904101760 0x7f8904073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8903fff640 1 -- 172.21.15.31:0/2373266813 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8904101760 msgr2=0x7f8904073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8903fff640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8904101760 0x7f8904073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8903fff640 1 -- 172.21.15.31:0/2373266813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8904102120 msgr2=0x7f8904073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8903fff640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8904102120 0x7f8904073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8903fff640 1 -- 172.21.15.31:0/2373266813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89041a6300 con 0x7f8904107770 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8902ffd640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8904102120 0x7f8904073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f89037fe640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8904101760 0x7f8904073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8903fff640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8904107770 0x7f89040744c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f88fc005e30 tx=0x7f88fc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8900ff9640 1 -- 172.21.15.31:0/2373266813 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88fc004240 con 0x7f8904107770 2024-04-25T02:02:46.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8909b9e640 1 -- 172.21.15.31:0/2373266813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89041a6590 con 0x7f8904107770 2024-04-25T02:02:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8900ff9640 1 -- 172.21.15.31:0/2373266813 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88fc031d00 con 0x7f8904107770 2024-04-25T02:02:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8900ff9640 1 -- 172.21.15.31:0/2373266813 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88fc030e30 con 0x7f8904107770 2024-04-25T02:02:46.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.180+0000 7f8909b9e640 1 -- 172.21.15.31:0/2373266813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89041a6b30 con 0x7f8904107770 2024-04-25T02:02:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.182+0000 7f8900ff9640 1 -- 172.21.15.31:0/2373266813 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f88fc0043e0 con 0x7f8904107770 2024-04-25T02:02:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f890c988740 1 Processor -- start 2024-04-25T02:02:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f890c988740 1 -- start start 2024-04-25T02:02:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f890c988740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b57620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f890c988740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b61cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f890c988740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b64620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f890c988740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562837abbcb0 con 0x562837b57b60 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f890c988740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562837a645b0 con 0x562837b62230 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f890c988740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562837ab7c70 con 0x562837b57240 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f89037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b61cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f89037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b61cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35940/0 (socket says 172.21.15.31:35940) 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f8902ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b64620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f89037fe640 1 -- 172.21.15.31:0/2249868238 learned_addr learned my addr 172.21.15.31:0/2249868238 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f8900ff9640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f88ec0a2dd0 0x7f88ec0a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f8902ffd640 1 --2- 172.21.15.31:0/2249868238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b57620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f8903fff640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f88ec0a2dd0 0x7f88ec0a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f8902ffd640 1 -- 172.21.15.31:0/2249868238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 msgr2=0x562837b57620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f8902ffd640 1 --2- 172.21.15.31:0/2249868238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b57620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f8902ffd640 1 -- 172.21.15.31:0/2249868238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 msgr2=0x562837b61cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f8902ffd640 1 --2- 172.21.15.31:0/2249868238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b61cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f8902ffd640 1 -- 172.21.15.31:0/2249868238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628377d0210 con 0x562837b62230 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.183+0000 7f89037fe640 1 --2- 172.21.15.31:0/2249868238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b61cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.184+0000 7f8900ff9640 1 -- 172.21.15.31:0/2373266813 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7f88fc02e030 con 0x7f8904107770 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.184+0000 7f8902ffd640 1 --2- 172.21.15.31:0/2249868238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b64620 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f88f8005d00 tx=0x7f88f800bb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=546ff9fe354cdc47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.184+0000 7f8903fff640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f88ec0a2dd0 0x7f88ec0a5290 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f88fc0edf50 tx=0x7f88fc0b7040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.184+0000 7f88e9ffb640 1 -- 172.21.15.31:0/2249868238 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8004020 con 0x562837b62230 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.184+0000 7f88e9ffb640 1 -- 172.21.15.31:0/2249868238 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88f80054d0 con 0x562837b62230 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.184+0000 7f88e9ffb640 1 -- 172.21.15.31:0/2249868238 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8010c30 con 0x562837b62230 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 -- 172.21.15.31:0/2249868238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 msgr2=0x562837b64620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 --2- 172.21.15.31:0/2249868238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b64620 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f88f8005d00 tx=0x7f88f800bb30 comp rx=0 tx=0).stop 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 -- 172.21.15.31:0/2249868238 shutdown_connections 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 --2- 172.21.15.31:0/2249868238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b64620 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 --2- 172.21.15.31:0/2249868238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b61cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 --2- 172.21.15.31:0/2249868238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b57620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 -- 172.21.15.31:0/2249868238 >> 172.21.15.31:0/2249868238 conn(0x562837b53560 msgr2=0x562837b53940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 -- 172.21.15.31:0/2249868238 shutdown_connections 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 -- 172.21.15.31:0/2249868238 wait complete. 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 -- start start 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 Processor -- start 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.185+0000 7f890c988740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b5f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f890c988740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b5f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f890c988740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b5fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f890c988740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562837b65020 con 0x562837b57b60 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f890c988740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562837a77b50 con 0x562837b62230 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f890c988740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562837b651a0 con 0x562837b57240 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f890c988740 10 client.0 _ll_register_callbacks cb 0x562837ab2ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f89037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b5f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f89037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b5f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35952/0 (socket says 172.21.15.31:35952) 2024-04-25T02:02:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f89037fe640 1 -- 172.21.15.31:0/2870605320 learned_addr learned my addr 172.21.15.31:0/2870605320 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f8902ffd640 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b5fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f8902ffd640 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b5f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f89037fe640 1 -- 172.21.15.31:0/2870605320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 msgr2=0x562837b5f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f89037fe640 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b5f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f89037fe640 1 -- 172.21.15.31:0/2870605320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 msgr2=0x562837b5fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f89037fe640 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b5fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f89037fe640 1 -- 172.21.15.31:0/2870605320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56283789c620 con 0x562837b57b60 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f89037fe640 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b5f6c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f88f4013780 tx=0x7f88f4013c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd1264a2510a50e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f88e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f88e2ffd640 1 -- 172.21.15.31:0/2870605320 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f4011d50 con 0x562837b57b60 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f88e2ffd640 1 -- 172.21.15.31:0/2870605320 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7f88f401b3f0 con 0x562837b57b60 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.186+0000 7f890c988740 1 -- 172.21.15.31:0/2870605320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88f80054d0 con 0x562837b57b60 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f88e2ffd640 1 -- 172.21.15.31:0/2870605320 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f88f401b5f0 con 0x562837b57b60 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f890c988740 10 client.16818 fetch_fsmap learned FSMap version 33 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f890c988740 1 -- 172.21.15.31:0/2870605320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562837b71290 con 0x562837b57b60 2024-04-25T02:02:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f88e2ffd640 1 -- 172.21.15.31:0/2870605320 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88f80054d0 con 0x562837b57b60 2024-04-25T02:02:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f88e2ffd640 1 -- 172.21.15.31:0/2870605320 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f88f4022020 con 0x562837b57b60 2024-04-25T02:02:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f890c988740 10 client.16818 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:02:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f890c988740 10 client.16818 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f890c988740 20 client.16818 populate_metadata read hostname 'smithi031' 2024-04-25T02:02:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f890c988740 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562837b727f0 0x562837b74be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f890c988740 4 client.16818 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f890c988740 1 -- 172.21.15.31:0/2870605320 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562837ac91f0 con 0x562837b727f0 2024-04-25T02:02:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.187+0000 7f8902ffd640 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562837b727f0 0x562837b74be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.188+0000 7f8902ffd640 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562837b727f0 0x562837b74be0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53d981e60a599b17 server_cookie=40d46238eba896ed in_seq=0 out_seq=0 2024-04-25T02:02:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.188+0000 7f88e2ffd640 10 client.16818 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f88e2ffd640 1 -- 172.21.15.31:0/2870605320 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562837ac91f0 con 0x562837b727f0 2024-04-25T02:02:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f88e2ffd640 10 client.16818 handle_command_reply: tid=0 2024-04-25T02:02:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 1 client.16818 shutdown 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 1 -- 172.21.15.31:0/2870605320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 msgr2=0x562837b5f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b5f6c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f88f4013780 tx=0x7f88f4013c40 comp rx=0 tx=0).stop 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 1 -- 172.21.15.31:0/2870605320 shutdown_connections 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562837b727f0 0x562837b74be0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b5fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57b60 0x562837b5f6c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 1 --2- 172.21.15.31:0/2870605320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b5f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 1 -- 172.21.15.31:0/2870605320 >> 172.21.15.31:0/2870605320 conn(0x562837b53560 msgr2=0x562837b59360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f88e2ffd640 0 client.16818 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 1 -- 172.21.15.31:0/2870605320 shutdown_connections 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 1 -- 172.21.15.31:0/2870605320 wait complete. 2024-04-25T02:02:46.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.189+0000 7f890c988740 20 client.16818 trim_cache size 0 max 16384 2024-04-25T02:02:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.204+0000 7f890c988740 1 Processor -- start 2024-04-25T02:02:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.204+0000 7f890c988740 1 -- start start 2024-04-25T02:02:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.204+0000 7f890c988740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57240 0x562837b618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.204+0000 7f890c988740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57b60 0x562837b599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.204+0000 7f890c988740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b5fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.204+0000 7f890c988740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f89041af3a0 con 0x562837b57240 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.204+0000 7f890c988740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f89041b0ee0 con 0x562837b62230 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.204+0000 7f890c988740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f89041afa70 con 0x562837b57b60 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.204+0000 7f8902ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57240 0x562837b618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.204+0000 7f8902ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57240 0x562837b618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35966/0 (socket says 172.21.15.31:35966) 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f8902ffd640 1 -- 172.21.15.31:0/3058315667 learned_addr learned my addr 172.21.15.31:0/3058315667 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f8902ffd640 1 --2- 172.21.15.31:0/3058315667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b5fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f89037fe640 1 --2- 172.21.15.31:0/3058315667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57b60 0x562837b599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f8902ffd640 1 -- 172.21.15.31:0/3058315667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57b60 msgr2=0x562837b599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f8902ffd640 1 --2- 172.21.15.31:0/3058315667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57b60 0x562837b599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f8902ffd640 1 -- 172.21.15.31:0/3058315667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 msgr2=0x562837b5fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f8902ffd640 1 --2- 172.21.15.31:0/3058315667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b5fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f8902ffd640 1 -- 172.21.15.31:0/3058315667 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562837b57620 con 0x562837b57240 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f8902ffd640 1 --2- 172.21.15.31:0/3058315667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57240 0x562837b618b0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f88f80109c0 tx=0x7f88f8010910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd89fc12e627df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f88e97fa640 1 -- 172.21.15.31:0/3058315667 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8010d80 con 0x562837b57240 2024-04-25T02:02:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f88e97fa640 1 -- 172.21.15.31:0/3058315667 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88f8031910 con 0x562837b57240 2024-04-25T02:02:46.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.205+0000 7f88e97fa640 1 -- 172.21.15.31:0/3058315667 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f801fc20 con 0x562837b57240 2024-04-25T02:02:46.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.206+0000 7f890c988740 1 -- 172.21.15.31:0/3058315667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57240 msgr2=0x562837b618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:46.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.206+0000 7f890c988740 1 --2- 172.21.15.31:0/3058315667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57240 0x562837b618b0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f88f80109c0 tx=0x7f88f8010910 comp rx=0 tx=0).stop 2024-04-25T02:02:46.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.206+0000 7f890c988740 1 -- 172.21.15.31:0/3058315667 shutdown_connections 2024-04-25T02:02:46.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.206+0000 7f890c988740 1 --2- 172.21.15.31:0/3058315667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b62230 0x562837b5fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.206+0000 7f890c988740 1 --2- 172.21.15.31:0/3058315667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57b60 0x562837b599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.206+0000 7f890c988740 1 --2- 172.21.15.31:0/3058315667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b57240 0x562837b618b0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.206+0000 7f890c988740 1 -- 172.21.15.31:0/3058315667 >> 172.21.15.31:0/3058315667 conn(0x562837b53560 msgr2=0x562837b525c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:46.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.206+0000 7f890c988740 1 -- 172.21.15.31:0/3058315667 shutdown_connections 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.206+0000 7f890c988740 1 -- 172.21.15.31:0/3058315667 wait complete. 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.206+0000 7f890c988740 1 -- start start 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f890c988740 1 Processor -- start 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f890c988740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b6a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f890c988740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b57b60 0x562837b5ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f890c988740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b62230 0x562837b5bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f890c988740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f89041b0810 con 0x562837b62230 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f890c988740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f89041b0140 con 0x562837b57b60 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f890c988740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f89041ad3a0 con 0x562837b57240 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f890c988740 10 client.0 _ll_register_callbacks cb 0x7f88f4009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f8902ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b62230 0x562837b5bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f89037fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b57b60 0x562837b5ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f89037fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b57b60 0x562837b5ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:34778/0 (socket says 172.21.15.31:34778) 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f89037fe640 1 -- 172.21.15.31:0/1296163404 learned_addr learned my addr 172.21.15.31:0/1296163404 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:46.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f8902ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b62230 0x562837b5bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35980/0 (socket says 172.21.15.31:35980) 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f8902ffd640 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b6a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f89037fe640 1 -- 172.21.15.31:0/1296163404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 msgr2=0x562837b6a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f89037fe640 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b6a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f89037fe640 1 -- 172.21.15.31:0/1296163404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b62230 msgr2=0x562837b5bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f89037fe640 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b62230 0x562837b5bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f89037fe640 1 -- 172.21.15.31:0/1296163404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562837b5c6d0 con 0x562837b57b60 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.207+0000 7f8902ffd640 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b62230 0x562837b5bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.208+0000 7f89037fe640 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b57b60 0x562837b5ba50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x562837b5e9e0 tx=0x7f88f4011d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cea1201a2e099e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.208+0000 7f88e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.208+0000 7f88e2ffd640 1 -- 172.21.15.31:0/1296163404 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f4010d20 con 0x562837b57b60 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.208+0000 7f890c988740 1 -- 172.21.15.31:0/1296163404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88f8031910 con 0x562837b57b60 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.208+0000 7f88e2ffd640 1 -- 172.21.15.31:0/1296163404 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7f88f4013020 con 0x562837b57b60 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.208+0000 7f88e2ffd640 1 -- 172.21.15.31:0/1296163404 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f88f4034050 con 0x562837b57b60 2024-04-25T02:02:46.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.208+0000 7f890c988740 10 client.26420 fetch_fsmap learned FSMap version 33 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.208+0000 7f890c988740 1 -- 172.21.15.31:0/1296163404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562837bd8cf0 con 0x562837b57b60 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.208+0000 7f88e2ffd640 1 -- 172.21.15.31:0/1296163404 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88f8031910 con 0x562837b57b60 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.209+0000 7f88e2ffd640 1 -- 172.21.15.31:0/1296163404 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f88f401a030 con 0x562837b57b60 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.209+0000 7f890c988740 10 client.26420 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.209+0000 7f890c988740 10 client.26420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.209+0000 7f890c988740 20 client.26420 populate_metadata read hostname 'smithi031' 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.209+0000 7f890c988740 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562837bd94a0 0x562837bdb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.209+0000 7f890c988740 4 client.26420 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.209+0000 7f890c988740 1 -- 172.21.15.31:0/1296163404 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562837ac3540 con 0x562837bd94a0 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.209+0000 7f8902ffd640 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562837bd94a0 0x562837bdb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.209+0000 7f8902ffd640 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562837bd94a0 0x562837bdb8c0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bac815ef6112b283 server_cookie=f7ce359ba70e236b in_seq=0 out_seq=0 2024-04-25T02:02:46.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.209+0000 7f88e2ffd640 10 client.26420 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.236+0000 7f88e2ffd640 1 -- 172.21.15.31:0/1296163404 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562837ac3540 con 0x562837bd94a0 2024-04-25T02:02:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.236+0000 7f88e2ffd640 10 client.26420 handle_command_reply: tid=0 2024-04-25T02:02:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.236+0000 7f890c988740 1 client.26420 shutdown 2024-04-25T02:02:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.236+0000 7f890c988740 1 -- 172.21.15.31:0/1296163404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b57b60 msgr2=0x562837b5ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.236+0000 7f890c988740 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b57b60 0x562837b5ba50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x562837b5e9e0 tx=0x7f88f4011d80 comp rx=0 tx=0).stop 2024-04-25T02:02:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f890c988740 1 -- 172.21.15.31:0/1296163404 shutdown_connections 2024-04-25T02:02:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f890c988740 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562837bd94a0 0x562837bdb8c0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f890c988740 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562837b62230 0x562837b5bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f890c988740 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562837b57b60 0x562837b5ba50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f890c988740 1 --2- 172.21.15.31:0/1296163404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562837b57240 0x562837b6a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f88e2ffd640 0 client.26420 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f890c988740 1 -- 172.21.15.31:0/1296163404 >> 172.21.15.31:0/1296163404 conn(0x562837b53560 msgr2=0x562837b6ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f890c988740 1 -- 172.21.15.31:0/1296163404 shutdown_connections 2024-04-25T02:02:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f890c988740 1 -- 172.21.15.31:0/1296163404 wait complete. 2024-04-25T02:02:46.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f890c988740 20 client.26420 trim_cache size 0 max 16384 2024-04-25T02:02:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:02:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:02:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:02:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:02:46.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f8909b9e640 1 -- 172.21.15.31:0/2373266813 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f88ec0a2dd0 msgr2=0x7f88ec0a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f8909b9e640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f88ec0a2dd0 0x7f88ec0a5290 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f88fc0edf50 tx=0x7f88fc0b7040 comp rx=0 tx=0).stop 2024-04-25T02:02:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f8909b9e640 1 -- 172.21.15.31:0/2373266813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8904107770 msgr2=0x7f89040744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f8909b9e640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8904107770 0x7f89040744c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f88fc005e30 tx=0x7f88fc003ba0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f8909b9e640 1 -- 172.21.15.31:0/2373266813 shutdown_connections 2024-04-25T02:02:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f8909b9e640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f88ec0a2dd0 0x7f88ec0a5290 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f8909b9e640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8904107770 0x7f89040744c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f8909b9e640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8904102120 0x7f8904073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f8909b9e640 1 --2- 172.21.15.31:0/2373266813 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8904101760 0x7f8904073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.237+0000 7f8909b9e640 1 -- 172.21.15.31:0/2373266813 >> 172.21.15.31:0/2373266813 conn(0x7f89040fd1f0 msgr2=0x7f89040fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.238+0000 7f8909b9e640 1 -- 172.21.15.31:0/2373266813 shutdown_connections 2024-04-25T02:02:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:46.238+0000 7f8909b9e640 1 -- 172.21.15.31:0/2373266813 wait complete. 2024-04-25T02:02:46.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 193907 2.45 MB/sec execute 293 sec latency 25.391 ms 2024-04-25T02:02:46.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:46 smithi040 ceph-mon[32418]: pgmap v256: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:46.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:46 smithi134 ceph-mon[33098]: pgmap v256: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:46.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:46 smithi031 ceph-mon[29204]: pgmap v256: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 1.9 MiB/s wr, 179 op/s 2024-04-25T02:02:47.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 194471 2.47 MB/sec execute 294 sec latency 19.725 ms 2024-04-25T02:02:48.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 194905 2.46 MB/sec execute 295 sec latency 18.476 ms 2024-04-25T02:02:48.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:48 smithi040 ceph-mon[32418]: pgmap v257: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:02:48.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:48 smithi134 ceph-mon[33098]: pgmap v257: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:02:48.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:48 smithi031 ceph-mon[29204]: pgmap v257: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:02:49.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 195371 2.47 MB/sec execute 296 sec latency 14.825 ms 2024-04-25T02:02:50.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 195633 2.46 MB/sec execute 297 sec latency 14.736 ms 2024-04-25T02:02:50.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:50 smithi040 ceph-mon[32418]: pgmap v258: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:02:50.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:50 smithi134 ceph-mon[33098]: pgmap v258: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:02:50.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:50 smithi031 ceph-mon[29204]: pgmap v258: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-04-25T02:02:51.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 196038 2.46 MB/sec execute 298 sec latency 18.172 ms 2024-04-25T02:02:52.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 196568 2.46 MB/sec execute 299 sec latency 23.189 ms 2024-04-25T02:02:52.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:52 smithi040 ceph-mon[32418]: pgmap v259: 97 pgs: 97 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-04-25T02:02:52.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:52 smithi134 ceph-mon[33098]: pgmap v259: 97 pgs: 97 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-04-25T02:02:52.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:52 smithi031 ceph-mon[29204]: pgmap v259: 97 pgs: 97 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-04-25T02:02:53.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 197064 2.46 MB/sec execute 300 sec latency 22.114 ms 2024-04-25T02:02:54.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 197563 2.46 MB/sec execute 301 sec latency 22.782 ms 2024-04-25T02:02:54.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:54 smithi040 ceph-mon[32418]: pgmap v260: 97 pgs: 97 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:54.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:54 smithi134 ceph-mon[33098]: pgmap v260: 97 pgs: 97 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:54.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:54 smithi031 ceph-mon[29204]: pgmap v260: 97 pgs: 97 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-25T02:02:55.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 198299 2.47 MB/sec execute 302 sec latency 18.931 ms 2024-04-25T02:02:56.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 198728 2.47 MB/sec execute 303 sec latency 11.806 ms 2024-04-25T02:02:56.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:56 smithi040 ceph-mon[32418]: pgmap v261: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-25T02:02:56.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:56 smithi134 ceph-mon[33098]: pgmap v261: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-25T02:02:56.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:56 smithi031 ceph-mon[29204]: pgmap v261: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-25T02:02:57.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 199015 2.47 MB/sec execute 304 sec latency 95.404 ms 2024-04-25T02:02:58.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 199299 2.46 MB/sec execute 305 sec latency 18.051 ms 2024-04-25T02:02:58.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:02:58 smithi040 ceph-mon[32418]: pgmap v262: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-04-25T02:02:58.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:02:58 smithi134 ceph-mon[33098]: pgmap v262: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-04-25T02:02:58.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:02:58 smithi031 ceph-mon[29204]: pgmap v262: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-04-25T02:02:59.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 199816 2.46 MB/sec execute 306 sec latency 11.502 ms 2024-04-25T02:02:59.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.336+0000 7fb268909640 1 -- 172.21.15.31:0/3486088800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb264107730 msgr2=0x7fb264107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:59.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.336+0000 7fb268909640 1 --2- 172.21.15.31:0/3486088800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb264107730 0x7fb264107b10 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb254009930 tx=0x7fb25402a420 comp rx=0 tx=0).stop 2024-04-25T02:02:59.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.336+0000 7fb268909640 1 -- 172.21.15.31:0/3486088800 shutdown_connections 2024-04-25T02:02:59.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.336+0000 7fb268909640 1 --2- 172.21.15.31:0/3486088800 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2641020e0 0x7fb26410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.336+0000 7fb268909640 1 --2- 172.21.15.31:0/3486088800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb264101720 0x7fb264101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.336+0000 7fb268909640 1 --2- 172.21.15.31:0/3486088800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb264107730 0x7fb264107b10 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.336+0000 7fb268909640 1 -- 172.21.15.31:0/3486088800 >> 172.21.15.31:0/3486088800 conn(0x7fb2640fd1f0 msgr2=0x7fb2640ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:59.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.336+0000 7fb268909640 1 -- 172.21.15.31:0/3486088800 shutdown_connections 2024-04-25T02:02:59.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.336+0000 7fb268909640 1 -- 172.21.15.31:0/3486088800 wait complete. 2024-04-25T02:02:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb268909640 1 Processor -- start 2024-04-25T02:02:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb268909640 1 -- start start 2024-04-25T02:02:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb268909640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb264101720 0x7fb264073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb268909640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2641020e0 0x7fb264073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb268909640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb264107730 0x7fb2640744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb268909640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2641119f0 con 0x7fb2641020e0 2024-04-25T02:02:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb268909640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb264111870 con 0x7fb264101720 2024-04-25T02:02:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb268909640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb264111b70 con 0x7fb264107730 2024-04-25T02:02:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb25bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2641020e0 0x7fb264073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb25bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2641020e0 0x7fb264073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57044/0 (socket says 172.21.15.31:57044) 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb25bfff640 1 -- 172.21.15.31:0/4036526605 learned_addr learned my addr 172.21.15.31:0/4036526605 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb26259c640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb264101720 0x7fb264073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb262d9d640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb264107730 0x7fb2640744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb25bfff640 1 -- 172.21.15.31:0/4036526605 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb264107730 msgr2=0x7fb2640744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb25bfff640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb264107730 0x7fb2640744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb25bfff640 1 -- 172.21.15.31:0/4036526605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb264101720 msgr2=0x7fb264073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb25bfff640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb264101720 0x7fb264073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.337+0000 7fb25bfff640 1 -- 172.21.15.31:0/4036526605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2641a63b0 con 0x7fb2641020e0 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.338+0000 7fb25bfff640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2641020e0 0x7fb264073f80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00e9f0 tx=0x7fb24c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.338+0000 7fb25b7fe640 1 -- 172.21.15.31:0/4036526605 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c00cd80 con 0x7fb2641020e0 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.338+0000 7fb25b7fe640 1 -- 172.21.15.31:0/4036526605 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb24c0108e0 con 0x7fb2641020e0 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.338+0000 7fb268909640 1 -- 172.21.15.31:0/4036526605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2641a66a0 con 0x7fb2641020e0 2024-04-25T02:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.338+0000 7fb25b7fe640 1 -- 172.21.15.31:0/4036526605 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c018920 con 0x7fb2641020e0 2024-04-25T02:02:59.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.338+0000 7fb268909640 1 -- 172.21.15.31:0/4036526605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2641a6c40 con 0x7fb2641020e0 2024-04-25T02:02:59.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.340+0000 7fb25b7fe640 1 -- 172.21.15.31:0/4036526605 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb24c010a90 con 0x7fb2641020e0 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.340+0000 7fb26b615740 1 Processor -- start 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26b615740 1 -- start start 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb25b7fe640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2500a2e20 0x7fb2500a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26b615740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f05620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26b615740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 0x559299f0fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26b615740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f10230 0x559299f12620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26b615740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559299e69cb0 con 0x559299f05240 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26b615740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559299e125b0 con 0x559299f05b60 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26b615740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559299e65c70 con 0x559299f10230 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f05620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb262d9d640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2500a2e20 0x7fb2500a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb25bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 0x559299f0fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f05620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57052/0 (socket says 172.21.15.31:57052) 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26259c640 1 -- 172.21.15.31:0/2774059414 learned_addr learned my addr 172.21.15.31:0/2774059414 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb25b7fe640 1 -- 172.21.15.31:0/4036526605 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7fb24c0cf6b0 con 0x7fb2641020e0 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb262d9d640 1 --2- 172.21.15.31:0/2774059414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f10230 0x559299f12620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26259c640 1 -- 172.21.15.31:0/2774059414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f10230 msgr2=0x559299f12620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26259c640 1 --2- 172.21.15.31:0/2774059414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f10230 0x559299f12620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26259c640 1 -- 172.21.15.31:0/2774059414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 msgr2=0x559299f0fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26259c640 1 --2- 172.21.15.31:0/2774059414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 0x559299f0fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb26259c640 1 -- 172.21.15.31:0/2774059414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559299b7e210 con 0x559299f05240 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.341+0000 7fb25bfff640 1 --2- 172.21.15.31:0/2774059414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 0x559299f0fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.342+0000 7fb26259c640 1 --2- 172.21.15.31:0/2774059414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f05620 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb25402a940 tx=0x7fb254005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4c555f1bc0eddd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.342+0000 7fb262d9d640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2500a2e20 0x7fb2500a52e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x559299f026f0 tx=0x7fb25c009b70 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.342+0000 7fb258ff9640 1 -- 172.21.15.31:0/2774059414 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254040070 con 0x559299f05240 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.342+0000 7fb258ff9640 1 -- 172.21.15.31:0/2774059414 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb254004030 con 0x559299f05240 2024-04-25T02:02:59.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.342+0000 7fb258ff9640 1 -- 172.21.15.31:0/2774059414 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254035040 con 0x559299f05240 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 -- 172.21.15.31:0/2774059414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 msgr2=0x559299f05620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 --2- 172.21.15.31:0/2774059414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f05620 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb25402a940 tx=0x7fb254005be0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 -- 172.21.15.31:0/2774059414 shutdown_connections 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 --2- 172.21.15.31:0/2774059414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f10230 0x559299f12620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 --2- 172.21.15.31:0/2774059414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 0x559299f0fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 --2- 172.21.15.31:0/2774059414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f05620 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 -- 172.21.15.31:0/2774059414 >> 172.21.15.31:0/2774059414 conn(0x559299f01560 msgr2=0x559299f01940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 -- 172.21.15.31:0/2774059414 shutdown_connections 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 -- 172.21.15.31:0/2774059414 wait complete. 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 -- start start 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 Processor -- start 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f0d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 0x559299f0d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f10230 0x559299f0db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559299f12fc0 con 0x559299f05240 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559299e25b50 con 0x559299f05b60 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.343+0000 7fb26b615740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559299f13140 con 0x559299f10230 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb26259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f0d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb26259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f0d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57068/0 (socket says 172.21.15.31:57068) 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb26259c640 1 -- 172.21.15.31:0/3827123034 learned_addr learned my addr 172.21.15.31:0/3827123034 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb26b615740 10 client.0 _ll_register_callbacks cb 0x559299e60ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb262d9d640 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f10230 0x559299f0db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb25bfff640 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 0x559299f0d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb26259c640 1 -- 172.21.15.31:0/3827123034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f10230 msgr2=0x559299f0db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb26259c640 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f10230 0x559299f0db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb26259c640 1 -- 172.21.15.31:0/3827123034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 msgr2=0x559299f0d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb26259c640 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 0x559299f0d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb26259c640 1 -- 172.21.15.31:0/3827123034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559299c4a620 con 0x559299f05240 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb25bfff640 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 0x559299f0d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:59.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb26259c640 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f0d0f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb25402aae0 tx=0x7fb25402ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb70d2ba6a7526de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb249ffb640 1 -- 172.21.15.31:0/3827123034 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254040030 con 0x559299f05240 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb26b615740 1 -- 172.21.15.31:0/3827123034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb254004030 con 0x559299f05240 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.344+0000 7fb249ffb640 1 -- 172.21.15.31:0/3827123034 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7fb254005230 con 0x559299f05240 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb249ffb640 1 -- 172.21.15.31:0/3827123034 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb254035070 con 0x559299f05240 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb26b615740 10 client.16836 fetch_fsmap learned FSMap version 33 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb26b615740 1 -- 172.21.15.31:0/3827123034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559299f1f0d0 con 0x559299f05240 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb249ffb640 1 -- 172.21.15.31:0/3827123034 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb254004030 con 0x559299f05240 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb249ffb640 1 -- 172.21.15.31:0/3827123034 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb254055020 con 0x559299f05240 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb26b615740 10 client.16836 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb26b615740 10 client.16836 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb26b615740 20 client.16836 populate_metadata read hostname 'smithi031' 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb26b615740 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559299f20420 0x559299f22810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb26b615740 4 client.16836 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:59.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb26b615740 1 -- 172.21.15.31:0/3827123034 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559299e771f0 con 0x559299f20420 2024-04-25T02:02:59.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.345+0000 7fb25bfff640 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559299f20420 0x559299f22810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.346+0000 7fb25bfff640 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559299f20420 0x559299f22810 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b226f13f6ccbbdc server_cookie=7b84b78e1e4cc1a3 in_seq=0 out_seq=0 2024-04-25T02:02:59.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.346+0000 7fb249ffb640 10 client.16836 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:59.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb249ffb640 1 -- 172.21.15.31:0/3827123034 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559299e771f0 con 0x559299f20420 2024-04-25T02:02:59.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb249ffb640 10 client.16836 handle_command_reply: tid=0 2024-04-25T02:02:59.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 1 client.16836 shutdown 2024-04-25T02:02:59.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 1 -- 172.21.15.31:0/3827123034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 msgr2=0x559299f0d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:59.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f0d0f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb25402aae0 tx=0x7fb25402ab10 comp rx=0 tx=0).stop 2024-04-25T02:02:59.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 1 -- 172.21.15.31:0/3827123034 shutdown_connections 2024-04-25T02:02:59.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559299f20420 0x559299f22810 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f10230 0x559299f0db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f05b60 0x559299f0d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb249ffb640 0 client.16836 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:59.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 1 --2- 172.21.15.31:0/3827123034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05240 0x559299f0d0f0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 1 -- 172.21.15.31:0/3827123034 >> 172.21.15.31:0/3827123034 conn(0x559299f01560 msgr2=0x559299f07260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:59.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 1 -- 172.21.15.31:0/3827123034 shutdown_connections 2024-04-25T02:02:59.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 1 -- 172.21.15.31:0/3827123034 wait complete. 2024-04-25T02:02:59.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.347+0000 7fb26b615740 20 client.16836 trim_cache size 0 max 16384 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.364+0000 7fb26b615740 1 Processor -- start 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.364+0000 7fb26b615740 1 -- start start 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.364+0000 7fb26b615740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 0x559299f1be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.364+0000 7fb26b615740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f074e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.364+0000 7fb26b615740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f10230 0x559299f00440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.364+0000 7fb26b615740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2641af3c0 con 0x559299f05b60 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.364+0000 7fb26b615740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2641b0f00 con 0x559299f10230 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.364+0000 7fb26b615740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb2641afa90 con 0x559299f05240 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb25bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f074e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb26259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 0x559299f1be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb25bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f074e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57078/0 (socket says 172.21.15.31:57078) 2024-04-25T02:02:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb26259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 0x559299f1be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:53898/0 (socket says 172.21.15.31:53898) 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb25bfff640 1 -- 172.21.15.31:0/350430289 learned_addr learned my addr 172.21.15.31:0/350430289 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb262d9d640 1 --2- 172.21.15.31:0/350430289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f10230 0x559299f00440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb26259c640 1 -- 172.21.15.31:0/350430289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f10230 msgr2=0x559299f00440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb26259c640 1 --2- 172.21.15.31:0/350430289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f10230 0x559299f00440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb26259c640 1 -- 172.21.15.31:0/350430289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 msgr2=0x559299f074e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb26259c640 1 --2- 172.21.15.31:0/350430289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f074e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb26259c640 1 -- 172.21.15.31:0/350430289 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559299f0b640 con 0x559299f05240 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb25bfff640 1 --2- 172.21.15.31:0/350430289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f074e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb26259c640 1 --2- 172.21.15.31:0/350430289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 0x559299f1be80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb2540092a0 tx=0x7fb254005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f31c959166d07c11 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb24bfff640 1 -- 172.21.15.31:0/350430289 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254040040 con 0x559299f05240 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb24bfff640 1 -- 172.21.15.31:0/350430289 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb25405c070 con 0x559299f05240 2024-04-25T02:02:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.365+0000 7fb24bfff640 1 -- 172.21.15.31:0/350430289 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2540051a0 con 0x559299f05240 2024-04-25T02:02:59.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.366+0000 7fb26b615740 1 -- 172.21.15.31:0/350430289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 msgr2=0x559299f1be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:59.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.366+0000 7fb26b615740 1 --2- 172.21.15.31:0/350430289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 0x559299f1be80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb2540092a0 tx=0x7fb254005fd0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.366+0000 7fb26b615740 1 -- 172.21.15.31:0/350430289 shutdown_connections 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.366+0000 7fb26b615740 1 --2- 172.21.15.31:0/350430289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f10230 0x559299f00440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.366+0000 7fb26b615740 1 --2- 172.21.15.31:0/350430289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f074e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.366+0000 7fb26b615740 1 --2- 172.21.15.31:0/350430289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 0x559299f1be80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.366+0000 7fb26b615740 1 -- 172.21.15.31:0/350430289 >> 172.21.15.31:0/350430289 conn(0x559299f01560 msgr2=0x559299effd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.366+0000 7fb26b615740 1 -- 172.21.15.31:0/350430289 shutdown_connections 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.366+0000 7fb26b615740 1 -- 172.21.15.31:0/350430289 wait complete. 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.366+0000 7fb26b615740 1 -- start start 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.366+0000 7fb26b615740 1 Processor -- start 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb26b615740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 0x559299f14430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb26b615740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f14970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb26b615740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f10230 0x559299f856e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb26b615740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2641b0830 con 0x559299f05b60 2024-04-25T02:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb26b615740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2641b0160 con 0x559299f10230 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb26b615740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb2641ad3c0 con 0x559299f05240 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb26b615740 10 client.0 _ll_register_callbacks cb 0x7fb24c009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb25bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f14970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb25bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f14970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57082/0 (socket says 172.21.15.31:57082) 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb25bfff640 1 -- 172.21.15.31:0/1344959805 learned_addr learned my addr 172.21.15.31:0/1344959805 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb26259c640 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 0x559299f14430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb262d9d640 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f10230 0x559299f856e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb25bfff640 1 -- 172.21.15.31:0/1344959805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 msgr2=0x559299f14430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb25bfff640 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 0x559299f14430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb25bfff640 1 -- 172.21.15.31:0/1344959805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f10230 msgr2=0x559299f856e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb25bfff640 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f10230 0x559299f856e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb25bfff640 1 -- 172.21.15.31:0/1344959805 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559299f85c20 con 0x559299f05b60 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb25bfff640 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f14970 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0cced0 tx=0x7fb24c09bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8032646657f38686 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb249ffb640 1 -- 172.21.15.31:0/1344959805 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c099fa0 con 0x559299f05b60 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb249ffb640 1 -- 172.21.15.31:0/1344959805 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8484+0+0 (secure 0 0 0) 0x7fb24c0cd0a0 con 0x559299f05b60 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.367+0000 7fb26b615740 1 -- 172.21.15.31:0/1344959805 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb25405c070 con 0x559299f05b60 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb249ffb640 1 -- 172.21.15.31:0/1344959805 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb24c098310 con 0x559299f05b60 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb26b615740 10 client.16842 fetch_fsmap learned FSMap version 33 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb26b615740 1 -- 172.21.15.31:0/1344959805 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559299f86e90 con 0x559299f05b60 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb249ffb640 1 -- 172.21.15.31:0/1344959805 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb25405c070 con 0x559299f05b60 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb249ffb640 1 -- 172.21.15.31:0/1344959805 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fb24c0a4460 con 0x559299f05b60 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb26b615740 10 client.16842 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb26b615740 10 client.16842 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb26b615740 20 client.16842 populate_metadata read hostname 'smithi031' 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb26b615740 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559299f875f0 0x559299f89a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb26b615740 4 client.16842 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb26b615740 1 -- 172.21.15.31:0/1344959805 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559299e71540 con 0x559299f875f0 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.368+0000 7fb26259c640 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559299f875f0 0x559299f89a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.369+0000 7fb249ffb640 10 client.16842 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.369+0000 7fb26259c640 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559299f875f0 0x559299f89a10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccf7c287082cb2c2 server_cookie=d4d0ce19ddd64c4d in_seq=0 out_seq=0 2024-04-25T02:02:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.387+0000 7fb249ffb640 1 -- 172.21.15.31:0/1344959805 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559299e71540 con 0x559299f875f0 2024-04-25T02:02:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.387+0000 7fb249ffb640 10 client.16842 handle_command_reply: tid=0 2024-04-25T02:02:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.387+0000 7fb26b615740 1 client.16842 shutdown 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb26b615740 1 -- 172.21.15.31:0/1344959805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 msgr2=0x559299f14970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb26b615740 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f14970 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0cced0 tx=0x7fb24c09bdf0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb26b615740 1 -- 172.21.15.31:0/1344959805 shutdown_connections 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb26b615740 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559299f875f0 0x559299f89a10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb26b615740 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559299f10230 0x559299f856e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb26b615740 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559299f05b60 0x559299f14970 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb249ffb640 0 client.16842 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb26b615740 1 --2- 172.21.15.31:0/1344959805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559299f05240 0x559299f14430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb26b615740 1 -- 172.21.15.31:0/1344959805 >> 172.21.15.31:0/1344959805 conn(0x559299f01560 msgr2=0x559299f005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb26b615740 1 -- 172.21.15.31:0/1344959805 shutdown_connections 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb26b615740 1 -- 172.21.15.31:0/1344959805 wait complete. 2024-04-25T02:02:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb26b615740 20 client.16842 trim_cache size 0 max 16384 2024-04-25T02:02:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:02:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:02:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:02:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:02:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb268909640 1 -- 172.21.15.31:0/4036526605 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2500a2e20 msgr2=0x7fb2500a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb268909640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2500a2e20 0x7fb2500a52e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x559299f026f0 tx=0x7fb25c009b70 comp rx=0 tx=0).stop 2024-04-25T02:02:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb268909640 1 -- 172.21.15.31:0/4036526605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2641020e0 msgr2=0x7fb264073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.388+0000 7fb268909640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2641020e0 0x7fb264073f80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00e9f0 tx=0x7fb24c00eeb0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.389+0000 7fb268909640 1 -- 172.21.15.31:0/4036526605 shutdown_connections 2024-04-25T02:02:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.389+0000 7fb268909640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2500a2e20 0x7fb2500a52e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.389+0000 7fb268909640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb264107730 0x7fb2640744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.389+0000 7fb268909640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2641020e0 0x7fb264073f80 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.389+0000 7fb268909640 1 --2- 172.21.15.31:0/4036526605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb264101720 0x7fb264073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.389+0000 7fb268909640 1 -- 172.21.15.31:0/4036526605 >> 172.21.15.31:0/4036526605 conn(0x7fb2640fd1f0 msgr2=0x7fb2640fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.389+0000 7fb268909640 1 -- 172.21.15.31:0/4036526605 shutdown_connections 2024-04-25T02:02:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:02:59.389+0000 7fb268909640 1 -- 172.21.15.31:0/4036526605 wait complete. 2024-04-25T02:03:00.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 200288 2.46 MB/sec execute 307 sec latency 24.050 ms 2024-04-25T02:03:00.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:00 smithi040 ceph-mon[32418]: pgmap v263: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-04-25T02:03:00.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:00 smithi134 ceph-mon[33098]: pgmap v263: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-04-25T02:03:00.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:00 smithi031 ceph-mon[29204]: pgmap v263: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-04-25T02:03:01.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 200825 2.46 MB/sec execute 308 sec latency 22.494 ms 2024-04-25T02:03:02.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 201256 2.46 MB/sec execute 309 sec latency 23.152 ms 2024-04-25T02:03:02.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:02 smithi040 ceph-mon[32418]: pgmap v264: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-04-25T02:03:02.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:02 smithi134 ceph-mon[33098]: pgmap v264: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-04-25T02:03:02.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:02 smithi031 ceph-mon[29204]: pgmap v264: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-04-25T02:03:03.315 INFO:tasks.workunit.client.0.smithi031.stdout: 1 201952 2.47 MB/sec execute 310 sec latency 13.813 ms 2024-04-25T02:03:03.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:03 smithi040 ceph-mon[32418]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T02:03:03.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:03 smithi134 ceph-mon[33098]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T02:03:03.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:03 smithi031 ceph-mon[29204]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T02:03:04.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 202407 2.47 MB/sec execute 311 sec latency 16.170 ms 2024-04-25T02:03:04.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:04 smithi040 ceph-mon[32418]: pgmap v266: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-04-25T02:03:04.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:04 smithi134 ceph-mon[33098]: pgmap v266: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-04-25T02:03:04.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:04 smithi031 ceph-mon[29204]: pgmap v266: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-04-25T02:03:05.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 202649 2.47 MB/sec execute 312 sec latency 15.410 ms 2024-04-25T02:03:06.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 203053 2.46 MB/sec execute 313 sec latency 19.153 ms 2024-04-25T02:03:06.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:06 smithi040 ceph-mon[32418]: pgmap v267: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-25T02:03:06.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:06 smithi134 ceph-mon[33098]: pgmap v267: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-25T02:03:06.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:06 smithi031 ceph-mon[29204]: pgmap v267: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-25T02:03:07.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 203584 2.47 MB/sec execute 314 sec latency 18.077 ms 2024-04-25T02:03:08.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 204047 2.46 MB/sec execute 315 sec latency 21.509 ms 2024-04-25T02:03:08.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:08 smithi040 ceph-mon[32418]: pgmap v268: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-25T02:03:08.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:08 smithi134 ceph-mon[33098]: pgmap v268: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-25T02:03:08.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:08 smithi031 ceph-mon[29204]: pgmap v268: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-25T02:03:09.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 204572 2.46 MB/sec execute 316 sec latency 24.446 ms 2024-04-25T02:03:10.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:09 smithi040 ceph-mon[32418]: pgmap v269: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-25T02:03:10.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:09 smithi040 ceph-mon[32418]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T02:03:10.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:09 smithi134 ceph-mon[33098]: pgmap v269: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-25T02:03:10.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:09 smithi134 ceph-mon[33098]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T02:03:10.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:09 smithi031 ceph-mon[29204]: pgmap v269: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-04-25T02:03:10.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:09 smithi031 ceph-mon[29204]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T02:03:10.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205167 2.47 MB/sec execute 317 sec latency 17.773 ms 2024-04-25T02:03:11.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205577 2.47 MB/sec execute 318 sec latency 21.695 ms 2024-04-25T02:03:12.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:11 smithi040 ceph-mon[32418]: pgmap v271: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-04-25T02:03:12.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:11 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:12.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:11 smithi134 ceph-mon[33098]: pgmap v271: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-04-25T02:03:12.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:11 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:12.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:11 smithi031 ceph-mon[29204]: pgmap v271: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-04-25T02:03:12.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:11 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:12.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205802 2.47 MB/sec execute 319 sec latency 681.759 ms 2024-04-25T02:03:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.491+0000 7f9c733d5640 1 -- 172.21.15.31:0/354455307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c6c100760 msgr2=0x7f9c6c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.491+0000 7f9c733d5640 1 --2- 172.21.15.31:0/354455307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c6c100760 0x7f9c6c100b40 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9c68009930 tx=0x7f9c6802a4b0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.491+0000 7f9c733d5640 1 -- 172.21.15.31:0/354455307 shutdown_connections 2024-04-25T02:03:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.491+0000 7f9c733d5640 1 --2- 172.21.15.31:0/354455307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9c6c105b60 0x7f9c6c105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.491+0000 7f9c733d5640 1 --2- 172.21.15.31:0/354455307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c6c101080 0x7f9c6c105540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.491+0000 7f9c733d5640 1 --2- 172.21.15.31:0/354455307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c6c100760 0x7f9c6c100b40 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.491+0000 7f9c733d5640 1 -- 172.21.15.31:0/354455307 >> 172.21.15.31:0/354455307 conn(0x7f9c6c0777c0 msgr2=0x7f9c6c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.491+0000 7f9c733d5640 1 -- 172.21.15.31:0/354455307 shutdown_connections 2024-04-25T02:03:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.491+0000 7f9c733d5640 1 -- 172.21.15.31:0/354455307 wait complete. 2024-04-25T02:03:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c733d5640 1 Processor -- start 2024-04-25T02:03:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c733d5640 1 -- start start 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c733d5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9c6c100760 0x7f9c6c19e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c733d5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c6c101080 0x7f9c6c19e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c733d5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c6c105b60 0x7f9c6c1a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c733d5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c117fb0 con 0x7f9c6c101080 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c733d5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c117e30 con 0x7f9c6c105b60 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c733d5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c118130 con 0x7f9c6c100760 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c70970640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c6c101080 0x7f9c6c19e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c70970640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c6c101080 0x7f9c6c19e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52692/0 (socket says 172.21.15.31:52692) 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c70970640 1 -- 172.21.15.31:0/3802116021 learned_addr learned my addr 172.21.15.31:0/3802116021 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c71972640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c6c105b60 0x7f9c6c1a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c70970640 1 -- 172.21.15.31:0/3802116021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9c6c100760 msgr2=0x7f9c6c19e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c71171640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9c6c100760 0x7f9c6c19e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c70970640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9c6c100760 0x7f9c6c19e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c70970640 1 -- 172.21.15.31:0/3802116021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c6c105b60 msgr2=0x7f9c6c1a2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c70970640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c6c105b60 0x7f9c6c1a2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c70970640 1 -- 172.21.15.31:0/3802116021 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c6c1a3440 con 0x7f9c6c101080 2024-04-25T02:03:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.492+0000 7f9c71972640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c6c105b60 0x7f9c6c1a2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:03:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.493+0000 7f9c70970640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c6c101080 0x7f9c6c19e990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00d950 tx=0x7f9c5c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.493+0000 7f9c627fc640 1 -- 172.21.15.31:0/3802116021 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c017030 con 0x7f9c6c101080 2024-04-25T02:03:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.493+0000 7f9c627fc640 1 -- 172.21.15.31:0/3802116021 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c5c004020 con 0x7f9c6c101080 2024-04-25T02:03:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.493+0000 7f9c733d5640 1 -- 172.21.15.31:0/3802116021 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c6c1a3730 con 0x7f9c6c101080 2024-04-25T02:03:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.493+0000 7f9c627fc640 1 -- 172.21.15.31:0/3802116021 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c0047a0 con 0x7f9c6c101080 2024-04-25T02:03:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.493+0000 7f9c733d5640 1 -- 172.21.15.31:0/3802116021 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c6c1aafc0 con 0x7f9c6c101080 2024-04-25T02:03:12.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.495+0000 7f9c627fc640 1 -- 172.21.15.31:0/3802116021 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9c5c0041d0 con 0x7f9c6c101080 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c761c4740 1 Processor -- start 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c761c4740 1 -- start start 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890f240 0x55fa5890f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890fb60 0x55fa58919cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5891a230 0x55fa5891c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c761c4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fa58873cb0 con 0x55fa5890fb60 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c761c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fa5881c5b0 con 0x55fa5890f240 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c761c4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fa5886fc70 con 0x55fa5891a230 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c627fc640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9c580a2da0 0x7f9c580a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c71171640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890fb60 0x55fa58919cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c71171640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890fb60 0x55fa58919cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52708/0 (socket says 172.21.15.31:52708) 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c71171640 1 -- 172.21.15.31:0/3823033486 learned_addr learned my addr 172.21.15.31:0/3823033486 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c71972640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9c580a2da0 0x7f9c580a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c71972640 1 --2- 172.21.15.31:0/3823033486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890f240 0x55fa5890f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c627fc640 1 -- 172.21.15.31:0/3802116021 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f9c5c0cefc0 con 0x7f9c6c101080 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.496+0000 7f9c70970640 1 --2- 172.21.15.31:0/3823033486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5891a230 0x55fa5891c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.497+0000 7f9c71171640 1 -- 172.21.15.31:0/3823033486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5891a230 msgr2=0x55fa5891c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.497+0000 7f9c71171640 1 --2- 172.21.15.31:0/3823033486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5891a230 0x55fa5891c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.497+0000 7f9c71171640 1 -- 172.21.15.31:0/3823033486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890f240 msgr2=0x55fa5890f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.497+0000 7f9c71171640 1 --2- 172.21.15.31:0/3823033486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890f240 0x55fa5890f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.497+0000 7f9c71171640 1 -- 172.21.15.31:0/3823033486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa58588210 con 0x55fa5890fb60 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.497+0000 7f9c71972640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9c580a2da0 0x7f9c580a5260 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9c64007fd0 tx=0x7f9c64007d00 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.497+0000 7f9c71171640 1 --2- 172.21.15.31:0/3823033486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890fb60 0x55fa58919cf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f9c68006010 tx=0x7f9c68032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9660d322840a1544 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.497+0000 7f9c577fe640 1 -- 172.21.15.31:0/3823033486 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6803f070 con 0x55fa5890fb60 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.497+0000 7f9c577fe640 1 -- 172.21.15.31:0/3823033486 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c680316a0 con 0x55fa5890fb60 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.497+0000 7f9c577fe640 1 -- 172.21.15.31:0/3823033486 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c68035070 con 0x55fa5890fb60 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.498+0000 7f9c761c4740 1 -- 172.21.15.31:0/3823033486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890fb60 msgr2=0x55fa58919cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.498+0000 7f9c761c4740 1 --2- 172.21.15.31:0/3823033486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890fb60 0x55fa58919cf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f9c68006010 tx=0x7f9c68032c30 comp rx=0 tx=0).stop 2024-04-25T02:03:12.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.498+0000 7f9c761c4740 1 -- 172.21.15.31:0/3823033486 shutdown_connections 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.498+0000 7f9c761c4740 1 --2- 172.21.15.31:0/3823033486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5891a230 0x55fa5891c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.498+0000 7f9c761c4740 1 --2- 172.21.15.31:0/3823033486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890fb60 0x55fa58919cf0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.498+0000 7f9c761c4740 1 --2- 172.21.15.31:0/3823033486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890f240 0x55fa5890f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.498+0000 7f9c761c4740 1 -- 172.21.15.31:0/3823033486 >> 172.21.15.31:0/3823033486 conn(0x55fa5890b560 msgr2=0x55fa5890b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.498+0000 7f9c761c4740 1 -- 172.21.15.31:0/3823033486 shutdown_connections 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.498+0000 7f9c761c4740 1 -- 172.21.15.31:0/3823033486 wait complete. 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.498+0000 7f9c761c4740 1 -- start start 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.498+0000 7f9c761c4740 1 Processor -- start 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890f240 0x55fa58917170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890fb60 0x55fa589176b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5891a230 0x55fa58917bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c761c4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fa5891cfc0 con 0x55fa5890f240 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c761c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fa5882fb50 con 0x55fa5891a230 2024-04-25T02:03:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c761c4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fa5891d140 con 0x55fa5890fb60 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c761c4740 10 client.0 _ll_register_callbacks cb 0x55fa5886aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c71171640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890f240 0x55fa58917170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c71171640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890f240 0x55fa58917170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52712/0 (socket says 172.21.15.31:52712) 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c71171640 1 -- 172.21.15.31:0/898356352 learned_addr learned my addr 172.21.15.31:0/898356352 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c70970640 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890fb60 0x55fa589176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c71972640 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5891a230 0x55fa58917bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c71171640 1 -- 172.21.15.31:0/898356352 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890fb60 msgr2=0x55fa589176b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c71171640 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890fb60 0x55fa589176b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c71171640 1 -- 172.21.15.31:0/898356352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5891a230 msgr2=0x55fa58917bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c71171640 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5891a230 0x55fa58917bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c71171640 1 -- 172.21.15.31:0/898356352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa58654620 con 0x55fa5890f240 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c70970640 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890fb60 0x55fa589176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c71171640 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890f240 0x55fa58917170 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9c68031520 tx=0x7f9c68031550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a691f88458601a34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.499+0000 7f9c54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:03:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c54ff9640 1 -- 172.21.15.31:0/898356352 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c68035030 con 0x55fa5890f240 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c761c4740 1 -- 172.21.15.31:0/898356352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c680316a0 con 0x55fa5890f240 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c54ff9640 1 -- 172.21.15.31:0/898356352 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f9c68004230 con 0x55fa5890f240 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c54ff9640 1 -- 172.21.15.31:0/898356352 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9c68055050 con 0x55fa5890f240 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c761c4740 10 client.16854 fetch_fsmap learned FSMap version 34 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c761c4740 1 -- 172.21.15.31:0/898356352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55fa58929180 con 0x55fa5890f240 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c54ff9640 1 -- 172.21.15.31:0/898356352 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c680316a0 con 0x55fa5890f240 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c54ff9640 1 -- 172.21.15.31:0/898356352 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f9c6803b4f0 con 0x55fa5890f240 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c761c4740 10 client.16854 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c761c4740 10 client.16854 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c761c4740 20 client.16854 populate_metadata read hostname 'smithi031' 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c761c4740 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa5892a4d0 0x55fa5892c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c761c4740 4 client.16854 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.500+0000 7f9c761c4740 1 -- 172.21.15.31:0/898356352 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa588811f0 con 0x55fa5892a4d0 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.501+0000 7f9c70970640 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa5892a4d0 0x55fa5892c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.501+0000 7f9c54ff9640 10 client.16854 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:12.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.501+0000 7f9c70970640 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa5892a4d0 0x55fa5892c8c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d772014f9dc7222 server_cookie=87288c70d19a8d7 in_seq=0 out_seq=0 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c54ff9640 1 -- 172.21.15.31:0/898356352 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa588811f0 con 0x55fa5892a4d0 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c54ff9640 10 client.16854 handle_command_reply: tid=0 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 1 client.16854 shutdown 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 1 -- 172.21.15.31:0/898356352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890f240 msgr2=0x55fa58917170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890f240 0x55fa58917170 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9c68031520 tx=0x7f9c68031550 comp rx=0 tx=0).stop 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 1 -- 172.21.15.31:0/898356352 shutdown_connections 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa5892a4d0 0x55fa5892c8c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5891a230 0x55fa58917bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c54ff9640 0 client.16854 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890fb60 0x55fa589176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 1 --2- 172.21.15.31:0/898356352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5890f240 0x55fa58917170 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 1 -- 172.21.15.31:0/898356352 >> 172.21.15.31:0/898356352 conn(0x55fa5890b560 msgr2=0x55fa58911260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 1 -- 172.21.15.31:0/898356352 shutdown_connections 2024-04-25T02:03:12.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 1 -- 172.21.15.31:0/898356352 wait complete. 2024-04-25T02:03:12.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.502+0000 7f9c761c4740 20 client.16854 trim_cache size 0 max 16384 2024-04-25T02:03:12.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c761c4740 1 Processor -- start 2024-04-25T02:03:12.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c761c4740 1 -- start start 2024-04-25T02:03:12.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890f240 0x55fa589198a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890fb60 0x55fa58925e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589162d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c761c4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c1b3820 con 0x55fa5891a230 2024-04-25T02:03:12.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c761c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c1b5360 con 0x55fa5890f240 2024-04-25T02:03:12.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c761c4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c1b3ef0 con 0x55fa5890fb60 2024-04-25T02:03:12.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c71972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589162d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c71972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589162d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52724/0 (socket says 172.21.15.31:52724) 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c71972640 1 -- 172.21.15.31:0/2765323374 learned_addr learned my addr 172.21.15.31:0/2765323374 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c71171640 1 --2- 172.21.15.31:0/2765323374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890f240 0x55fa589198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c70970640 1 --2- 172.21.15.31:0/2765323374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890fb60 0x55fa58925e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c71972640 1 -- 172.21.15.31:0/2765323374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890fb60 msgr2=0x55fa58925e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c71972640 1 --2- 172.21.15.31:0/2765323374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890fb60 0x55fa58925e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c71972640 1 -- 172.21.15.31:0/2765323374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890f240 msgr2=0x55fa589198a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c71972640 1 --2- 172.21.15.31:0/2765323374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890f240 0x55fa589198a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c71972640 1 -- 172.21.15.31:0/2765323374 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa5890f990 con 0x55fa5891a230 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.518+0000 7f9c71972640 1 --2- 172.21.15.31:0/2765323374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589162d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9c64007840 tx=0x7f9c64004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edfffc4eca20edb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c56ffd640 1 -- 172.21.15.31:0/2765323374 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c64024070 con 0x55fa5891a230 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c56ffd640 1 -- 172.21.15.31:0/2765323374 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c64004d20 con 0x55fa5891a230 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c56ffd640 1 -- 172.21.15.31:0/2765323374 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c640054c0 con 0x55fa5891a230 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c761c4740 1 -- 172.21.15.31:0/2765323374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 msgr2=0x55fa589162d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c761c4740 1 --2- 172.21.15.31:0/2765323374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589162d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9c64007840 tx=0x7f9c64004ab0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c761c4740 1 -- 172.21.15.31:0/2765323374 shutdown_connections 2024-04-25T02:03:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c761c4740 1 --2- 172.21.15.31:0/2765323374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589162d0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c761c4740 1 --2- 172.21.15.31:0/2765323374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890fb60 0x55fa58925e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c761c4740 1 --2- 172.21.15.31:0/2765323374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890f240 0x55fa589198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c761c4740 1 -- 172.21.15.31:0/2765323374 >> 172.21.15.31:0/2765323374 conn(0x55fa5890b560 msgr2=0x55fa5890b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c761c4740 1 -- 172.21.15.31:0/2765323374 shutdown_connections 2024-04-25T02:03:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.519+0000 7f9c761c4740 1 -- 172.21.15.31:0/2765323374 wait complete. 2024-04-25T02:03:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c761c4740 1 -- start start 2024-04-25T02:03:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c761c4740 1 Processor -- start 2024-04-25T02:03:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890f240 0x55fa58922390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890fb60 0x55fa589228d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c761c4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589139a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c761c4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c1b4c90 con 0x55fa5891a230 2024-04-25T02:03:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c761c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c1b45c0 con 0x55fa5890fb60 2024-04-25T02:03:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c761c4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c1b1820 con 0x55fa5890f240 2024-04-25T02:03:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c761c4740 10 client.0 _ll_register_callbacks cb 0x7f9c64013640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c71972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c71972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52726/0 (socket says 172.21.15.31:52726) 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c71972640 1 -- 172.21.15.31:0/920568576 learned_addr learned my addr 172.21.15.31:0/920568576 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c70970640 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890fb60 0x55fa589228d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c71171640 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890f240 0x55fa58922390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.520+0000 7f9c71972640 1 -- 172.21.15.31:0/920568576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890f240 msgr2=0x55fa58922390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c71972640 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890f240 0x55fa58922390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c71972640 1 -- 172.21.15.31:0/920568576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890fb60 msgr2=0x55fa589228d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c71972640 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890fb60 0x55fa589228d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c71972640 1 -- 172.21.15.31:0/920568576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa589141a0 con 0x55fa5891a230 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c70970640 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890fb60 0x55fa589228d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c71171640 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890f240 0x55fa58922390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c71972640 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589139a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9c64011bd0 tx=0x7f9c64011c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab0cd267018cd51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:03:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c54ff9640 1 -- 172.21.15.31:0/920568576 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c64024040 con 0x55fa5891a230 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c761c4740 1 -- 172.21.15.31:0/920568576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c64004d20 con 0x55fa5891a230 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c54ff9640 1 -- 172.21.15.31:0/920568576 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f9c640053d0 con 0x55fa5891a230 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c54ff9640 1 -- 172.21.15.31:0/920568576 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9c64011e20 con 0x55fa5891a230 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c761c4740 10 client.16860 fetch_fsmap learned FSMap version 34 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c761c4740 1 -- 172.21.15.31:0/920568576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55fa58990410 con 0x55fa5891a230 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c54ff9640 1 -- 172.21.15.31:0/920568576 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c64004d20 con 0x55fa5891a230 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.521+0000 7f9c54ff9640 1 -- 172.21.15.31:0/920568576 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f9c64039020 con 0x55fa5891a230 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.522+0000 7f9c761c4740 10 client.16860 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.522+0000 7f9c761c4740 10 client.16860 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.522+0000 7f9c761c4740 20 client.16860 populate_metadata read hostname 'smithi031' 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.522+0000 7f9c761c4740 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa58990b50 0x55fa58992f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.522+0000 7f9c761c4740 4 client.16860 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.522+0000 7f9c761c4740 1 -- 172.21.15.31:0/920568576 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fa5887b540 con 0x55fa58990b50 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.522+0000 7f9c71171640 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa58990b50 0x55fa58992f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.522+0000 7f9c54ff9640 10 client.16860 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:12.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.522+0000 7f9c71171640 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa58990b50 0x55fa58992f70 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=594d8010f8f11a1d server_cookie=da9fb266ea77b784 in_seq=0 out_seq=0 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c54ff9640 1 -- 172.21.15.31:0/920568576 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fa5887b540 con 0x55fa58990b50 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c54ff9640 10 client.16860 handle_command_reply: tid=0 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 1 client.16860 shutdown 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 1 -- 172.21.15.31:0/920568576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 msgr2=0x55fa589139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589139a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9c64011bd0 tx=0x7f9c64011c00 comp rx=0 tx=0).stop 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 1 -- 172.21.15.31:0/920568576 shutdown_connections 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fa58990b50 0x55fa58992f70 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa5891a230 0x55fa589139a0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fa5890fb60 0x55fa589228d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c54ff9640 0 client.16860 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 1 --2- 172.21.15.31:0/920568576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa5890f240 0x55fa58922390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 1 -- 172.21.15.31:0/920568576 >> 172.21.15.31:0/920568576 conn(0x55fa5890b560 msgr2=0x55fa58917110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 1 -- 172.21.15.31:0/920568576 shutdown_connections 2024-04-25T02:03:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 1 -- 172.21.15.31:0/920568576 wait complete. 2024-04-25T02:03:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.557+0000 7f9c761c4740 20 client.16860 trim_cache size 0 max 16384 2024-04-25T02:03:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:03:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:03:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:03:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:03:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 -- 172.21.15.31:0/3802116021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9c580a2da0 msgr2=0x7f9c580a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9c580a2da0 0x7f9c580a5260 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9c64007fd0 tx=0x7f9c64007d00 comp rx=0 tx=0).stop 2024-04-25T02:03:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 -- 172.21.15.31:0/3802116021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c6c101080 msgr2=0x7f9c6c19e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c6c101080 0x7f9c6c19e990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00d950 tx=0x7f9c5c00de10 comp rx=0 tx=0).stop 2024-04-25T02:03:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 -- 172.21.15.31:0/3802116021 shutdown_connections 2024-04-25T02:03:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f9c580a2da0 0x7f9c580a5260 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c6c105b60 0x7f9c6c1a2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c6c101080 0x7f9c6c19e990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 --2- 172.21.15.31:0/3802116021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9c6c100760 0x7f9c6c19e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 -- 172.21.15.31:0/3802116021 >> 172.21.15.31:0/3802116021 conn(0x7f9c6c0777c0 msgr2=0x7f9c6c103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 -- 172.21.15.31:0/3802116021 shutdown_connections 2024-04-25T02:03:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:12.558+0000 7f9c733d5640 1 -- 172.21.15.31:0/3802116021 wait complete. 2024-04-25T02:03:13.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205802 2.46 MB/sec execute 320 sec latency 1681.793 ms 2024-04-25T02:03:14.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:13 smithi040 ceph-mon[32418]: pgmap v272: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-25T02:03:14.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:13 smithi134 ceph-mon[33098]: pgmap v272: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-25T02:03:14.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:13 smithi031 ceph-mon[29204]: pgmap v272: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-25T02:03:14.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205802 2.45 MB/sec execute 321 sec latency 2681.886 ms 2024-04-25T02:03:15.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205869 2.45 MB/sec execute 322 sec latency 3409.877 ms 2024-04-25T02:03:16.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:15 smithi040 ceph-mon[32418]: pgmap v273: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:03:16.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:15 smithi134 ceph-mon[33098]: pgmap v273: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:03:16.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:15 smithi031 ceph-mon[29204]: pgmap v273: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:03:16.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205869 2.44 MB/sec execute 323 sec latency 1029.187 ms 2024-04-25T02:03:17.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205869 2.43 MB/sec execute 324 sec latency 2029.297 ms 2024-04-25T02:03:18.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:17 smithi040 ceph-mon[32418]: pgmap v274: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:03:18.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:17 smithi134 ceph-mon[33098]: pgmap v274: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:03:18.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:17 smithi031 ceph-mon[29204]: pgmap v274: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:03:18.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205869 2.42 MB/sec execute 325 sec latency 3029.388 ms 2024-04-25T02:03:19.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205869 2.42 MB/sec execute 326 sec latency 4029.494 ms 2024-04-25T02:03:20.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:19 smithi040 ceph-mon[32418]: pgmap v275: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:03:20.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:19 smithi134 ceph-mon[33098]: pgmap v275: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:03:20.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:19 smithi031 ceph-mon[29204]: pgmap v275: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:03:20.316 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205963 2.41 MB/sec execute 327 sec latency 4755.077 ms 2024-04-25T02:03:21.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 206196 2.40 MB/sec execute 328 sec latency 18.971 ms 2024-04-25T02:03:22.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:21 smithi040 ceph-mon[32418]: pgmap v276: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 972 KiB/s wr, 104 op/s 2024-04-25T02:03:22.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:21 smithi134 ceph-mon[33098]: pgmap v276: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 972 KiB/s wr, 104 op/s 2024-04-25T02:03:22.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:21 smithi031 ceph-mon[29204]: pgmap v276: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 972 KiB/s wr, 104 op/s 2024-04-25T02:03:22.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 206597 2.40 MB/sec execute 329 sec latency 19.245 ms 2024-04-25T02:03:23.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 207133 2.41 MB/sec execute 330 sec latency 23.241 ms 2024-04-25T02:03:24.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:23 smithi040 ceph-mon[32418]: pgmap v277: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 378 KiB/s wr, 41 op/s 2024-04-25T02:03:24.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:23 smithi134 ceph-mon[33098]: pgmap v277: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 378 KiB/s wr, 41 op/s 2024-04-25T02:03:24.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:23 smithi031 ceph-mon[29204]: pgmap v277: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 378 KiB/s wr, 41 op/s 2024-04-25T02:03:24.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 207602 2.40 MB/sec execute 331 sec latency 19.755 ms 2024-04-25T02:03:25.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 208148 2.40 MB/sec execute 332 sec latency 20.240 ms 2024-04-25T02:03:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.651+0000 7f215606b640 1 -- 172.21.15.31:0/1435764679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2150101030 msgr2=0x7f215010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.651+0000 7f215606b640 1 --2- 172.21.15.31:0/1435764679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2150101030 0x7f215010d4f0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f2148009cd0 tx=0x7f214802a8e0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.651+0000 7f215606b640 1 -- 172.21.15.31:0/1435764679 shutdown_connections 2024-04-25T02:03:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.651+0000 7f215606b640 1 --2- 172.21.15.31:0/1435764679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2150101030 0x7f215010d4f0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.651+0000 7f215606b640 1 --2- 172.21.15.31:0/1435764679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2150100670 0x7f2150100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.651+0000 7f215606b640 1 --2- 172.21.15.31:0/1435764679 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2150106630 0x7f2150106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.651+0000 7f215606b640 1 -- 172.21.15.31:0/1435764679 >> 172.21.15.31:0/1435764679 conn(0x7f21500777c0 msgr2=0x7f2150077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.651+0000 7f215606b640 1 -- 172.21.15.31:0/1435764679 shutdown_connections 2024-04-25T02:03:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.651+0000 7f215606b640 1 -- 172.21.15.31:0/1435764679 wait complete. 2024-04-25T02:03:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.652+0000 7f215606b640 1 Processor -- start 2024-04-25T02:03:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.652+0000 7f215606b640 1 -- start start 2024-04-25T02:03:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.652+0000 7f215606b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2150100670 0x7f2150197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.652+0000 7f215606b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2150101030 0x7f21501984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.652+0000 7f215606b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2150106630 0x7f215019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.652+0000 7f215606b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f21501108f0 con 0x7f2150101030 2024-04-25T02:03:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.652+0000 7f215606b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2150110770 con 0x7f2150100670 2024-04-25T02:03:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.652+0000 7f215606b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2150110a70 con 0x7f2150106630 2024-04-25T02:03:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.652+0000 7f214f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2150100670 0x7f2150197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.652+0000 7f214f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2150100670 0x7f2150197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46402/0 (socket says 172.21.15.31:46402) 2024-04-25T02:03:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.652+0000 7f214f7fe640 1 -- 172.21.15.31:0/250049003 learned_addr learned my addr 172.21.15.31:0/250049003 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214ffff640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2150106630 0x7f215019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214effd640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2150101030 0x7f21501984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214f7fe640 1 -- 172.21.15.31:0/250049003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2150106630 msgr2=0x7f215019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214f7fe640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2150106630 0x7f215019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214f7fe640 1 -- 172.21.15.31:0/250049003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2150101030 msgr2=0x7f21501984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214f7fe640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2150101030 0x7f21501984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214f7fe640 1 -- 172.21.15.31:0/250049003 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215019cf50 con 0x7f2150100670 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214effd640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2150101030 0x7f21501984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214f7fe640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2150100670 0x7f2150197f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f214000e9f0 tx=0x7f214000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214cff9640 1 -- 172.21.15.31:0/250049003 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214000cd80 con 0x7f2150100670 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214cff9640 1 -- 172.21.15.31:0/250049003 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21400108e0 con 0x7f2150100670 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f215606b640 1 -- 172.21.15.31:0/250049003 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215019d240 con 0x7f2150100670 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f214cff9640 1 -- 172.21.15.31:0/250049003 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140018920 con 0x7f2150100670 2024-04-25T02:03:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.653+0000 7f215606b640 1 -- 172.21.15.31:0/250049003 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21501a4ad0 con 0x7f2150100670 2024-04-25T02:03:25.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.656+0000 7f214cff9640 1 -- 172.21.15.31:0/250049003 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2140010a90 con 0x7f2150100670 2024-04-25T02:03:25.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f2158d79740 1 Processor -- start 2024-04-25T02:03:25.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f2158d79740 1 -- start start 2024-04-25T02:03:25.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f2158d79740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 0x5593973cb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f2158d79740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973cb920 0x5593973da100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f2158d79740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973dca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f2158d79740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55939732fcb0 con 0x5593973caf70 2024-04-25T02:03:25.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f2158d79740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5593972d85b0 con 0x5593973cb920 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f2158d79740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55939732bc70 con 0x5593973da640 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 0x5593973cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 0x5593973cb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41464/0 (socket says 172.21.15.31:41464) 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214ffff640 1 -- 172.21.15.31:0/3040358275 learned_addr learned my addr 172.21.15.31:0/3040358275 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214cff9640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21380a2da0 0x7f21380a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214f7fe640 1 --2- 172.21.15.31:0/3040358275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973cb920 0x5593973da100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214effd640 1 --2- 172.21.15.31:0/3040358275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973dca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214ffff640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21380a2da0 0x7f21380a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214effd640 1 -- 172.21.15.31:0/3040358275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973cb920 msgr2=0x5593973da100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214effd640 1 --2- 172.21.15.31:0/3040358275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973cb920 0x5593973da100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214effd640 1 -- 172.21.15.31:0/3040358275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 msgr2=0x5593973cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214effd640 1 --2- 172.21.15.31:0/3040358275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 0x5593973cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214cff9640 1 -- 172.21.15.31:0/250049003 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f21400cf770 con 0x7f2150100670 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.657+0000 7f214effd640 1 -- 172.21.15.31:0/3040358275 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559397044210 con 0x5593973da640 2024-04-25T02:03:25.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.658+0000 7f214effd640 1 --2- 172.21.15.31:0/3040358275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973dca30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2144009b90 tx=0x7f214400e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35c5aeb7338d2200 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.658+0000 7f214ffff640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21380a2da0 0x7f21380a5260 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2148005e10 tx=0x7f214802a8e0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.658+0000 7f2135ffb640 1 -- 172.21.15.31:0/3040358275 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2144016070 con 0x5593973da640 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.658+0000 7f2135ffb640 1 -- 172.21.15.31:0/3040358275 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21440054d0 con 0x5593973da640 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.658+0000 7f2135ffb640 1 -- 172.21.15.31:0/3040358275 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2144011470 con 0x5593973da640 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.659+0000 7f2158d79740 1 -- 172.21.15.31:0/3040358275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 msgr2=0x5593973dca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.659+0000 7f2158d79740 1 --2- 172.21.15.31:0/3040358275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973dca30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2144009b90 tx=0x7f214400e6c0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.659+0000 7f2158d79740 1 -- 172.21.15.31:0/3040358275 shutdown_connections 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.659+0000 7f2158d79740 1 --2- 172.21.15.31:0/3040358275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973dca30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.659+0000 7f2158d79740 1 --2- 172.21.15.31:0/3040358275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973cb920 0x5593973da100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.659+0000 7f2158d79740 1 --2- 172.21.15.31:0/3040358275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 0x5593973cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.659+0000 7f2158d79740 1 -- 172.21.15.31:0/3040358275 >> 172.21.15.31:0/3040358275 conn(0x5593973c7560 msgr2=0x5593973c7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.659+0000 7f2158d79740 1 -- 172.21.15.31:0/3040358275 shutdown_connections 2024-04-25T02:03:25.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.659+0000 7f2158d79740 1 -- 172.21.15.31:0/3040358275 wait complete. 2024-04-25T02:03:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.659+0000 7f2158d79740 1 -- start start 2024-04-25T02:03:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.659+0000 7f2158d79740 1 Processor -- start 2024-04-25T02:03:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f2158d79740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 0x5593973d7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f2158d79740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973cb920 0x5593973d7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f2158d79740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973d7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f2158d79740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5593973dd400 con 0x5593973caf70 2024-04-25T02:03:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f2158d79740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5593972ebb50 con 0x5593973cb920 2024-04-25T02:03:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f2158d79740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5593973dd580 con 0x5593973da640 2024-04-25T02:03:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 0x5593973d7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 0x5593973d7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41480/0 (socket says 172.21.15.31:41480) 2024-04-25T02:03:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214f7fe640 1 -- 172.21.15.31:0/1365006327 learned_addr learned my addr 172.21.15.31:0/1365006327 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214effd640 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973cb920 0x5593973d7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214ffff640 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973d7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f2158d79740 10 client.0 _ll_register_callbacks cb 0x559397326ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214f7fe640 1 -- 172.21.15.31:0/1365006327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 msgr2=0x5593973d7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214f7fe640 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973d7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214f7fe640 1 -- 172.21.15.31:0/1365006327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973cb920 msgr2=0x5593973d7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214f7fe640 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973cb920 0x5593973d7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214f7fe640 1 -- 172.21.15.31:0/1365006327 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559397110620 con 0x5593973caf70 2024-04-25T02:03:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214effd640 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973cb920 0x5593973d7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f214f7fe640 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 0x5593973d7520 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f21400ccf90 tx=0x7f214009a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968c18f535473d6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f212effd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:03:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.660+0000 7f212effd640 1 -- 172.21.15.31:0/1365006327 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21400a0060 con 0x5593973caf70 2024-04-25T02:03:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f212effd640 1 -- 172.21.15.31:0/1365006327 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f2140010430 con 0x5593973caf70 2024-04-25T02:03:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f2158d79740 1 -- 172.21.15.31:0/1365006327 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21440054d0 con 0x5593973caf70 2024-04-25T02:03:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f212effd640 1 -- 172.21.15.31:0/1365006327 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f21400a4920 con 0x5593973caf70 2024-04-25T02:03:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f2158d79740 10 client.16869 fetch_fsmap learned FSMap version 34 2024-04-25T02:03:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f2158d79740 1 -- 172.21.15.31:0/1365006327 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5593973e9600 con 0x5593973caf70 2024-04-25T02:03:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f212effd640 1 -- 172.21.15.31:0/1365006327 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21440054d0 con 0x5593973caf70 2024-04-25T02:03:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f212effd640 1 -- 172.21.15.31:0/1365006327 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f2140018ac0 con 0x5593973caf70 2024-04-25T02:03:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f2158d79740 10 client.16869 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:03:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f2158d79740 10 client.16869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f2158d79740 20 client.16869 populate_metadata read hostname 'smithi031' 2024-04-25T02:03:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f2158d79740 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5593973eaac0 0x5593973eceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f2158d79740 4 client.16869 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:03:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f2158d79740 1 -- 172.21.15.31:0/1365006327 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55939733d1f0 con 0x5593973eaac0 2024-04-25T02:03:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.661+0000 7f214effd640 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5593973eaac0 0x5593973eceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.662+0000 7f212effd640 10 client.16869 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.662+0000 7f214effd640 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5593973eaac0 0x5593973eceb0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb52bc81dc64c719 server_cookie=da44ae803d2aeb5 in_seq=0 out_seq=0 2024-04-25T02:03:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f212effd640 1 -- 172.21.15.31:0/1365006327 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55939733d1f0 con 0x5593973eaac0 2024-04-25T02:03:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f212effd640 10 client.16869 handle_command_reply: tid=0 2024-04-25T02:03:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 1 client.16869 shutdown 2024-04-25T02:03:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 1 -- 172.21.15.31:0/1365006327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 msgr2=0x5593973d7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 0x5593973d7520 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f21400ccf90 tx=0x7f214009a1c0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 1 -- 172.21.15.31:0/1365006327 shutdown_connections 2024-04-25T02:03:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5593973eaac0 0x5593973eceb0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973d7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f212effd640 0 client.16869 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973cb920 0x5593973d7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 1 --2- 172.21.15.31:0/1365006327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973caf70 0x5593973d7520 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 1 -- 172.21.15.31:0/1365006327 >> 172.21.15.31:0/1365006327 conn(0x5593973c7560 msgr2=0x5593973cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 1 -- 172.21.15.31:0/1365006327 shutdown_connections 2024-04-25T02:03:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 1 -- 172.21.15.31:0/1365006327 wait complete. 2024-04-25T02:03:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.663+0000 7f2158d79740 20 client.16869 trim_cache size 0 max 16384 2024-04-25T02:03:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.681+0000 7f2158d79740 1 Processor -- start 2024-04-25T02:03:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.681+0000 7f2158d79740 1 -- start start 2024-04-25T02:03:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f2158d79740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 0x5593973d9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f2158d79740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973cb920 0x5593973e6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f2158d79740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973da640 0x5593973f2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f2158d79740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f21501ab0c0 con 0x5593973da640 2024-04-25T02:03:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f2158d79740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21501acc00 con 0x5593973caf70 2024-04-25T02:03:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f2158d79740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f21501ab790 con 0x5593973cb920 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 0x5593973d9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 0x5593973d9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46424/0 (socket says 172.21.15.31:46424) 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973da640 0x5593973f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214f7fe640 1 -- 172.21.15.31:0/1622399341 learned_addr learned my addr 172.21.15.31:0/1622399341 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214effd640 1 --2- 172.21.15.31:0/1622399341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973cb920 0x5593973e6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214f7fe640 1 -- 172.21.15.31:0/1622399341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973cb920 msgr2=0x5593973e6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214f7fe640 1 --2- 172.21.15.31:0/1622399341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973cb920 0x5593973e6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214f7fe640 1 -- 172.21.15.31:0/1622399341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973da640 msgr2=0x5593973f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214f7fe640 1 --2- 172.21.15.31:0/1622399341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973da640 0x5593973f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214f7fe640 1 -- 172.21.15.31:0/1622399341 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593973cb6c0 con 0x5593973caf70 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214ffff640 1 --2- 172.21.15.31:0/1622399341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973da640 0x5593973f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.682+0000 7f214f7fe640 1 --2- 172.21.15.31:0/1622399341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 0x5593973d9c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f21400097b0 tx=0x7f2140000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed20b380476a01b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f21357fa640 1 -- 172.21.15.31:0/1622399341 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214009bcb0 con 0x5593973caf70 2024-04-25T02:03:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f21357fa640 1 -- 172.21.15.31:0/1622399341 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2140018ac0 con 0x5593973caf70 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f21357fa640 1 -- 172.21.15.31:0/1622399341 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140010690 con 0x5593973caf70 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f2158d79740 1 -- 172.21.15.31:0/1622399341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 msgr2=0x5593973d9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f2158d79740 1 --2- 172.21.15.31:0/1622399341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 0x5593973d9c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f21400097b0 tx=0x7f2140000bc0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f2158d79740 1 -- 172.21.15.31:0/1622399341 shutdown_connections 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f2158d79740 1 --2- 172.21.15.31:0/1622399341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973da640 0x5593973f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f2158d79740 1 --2- 172.21.15.31:0/1622399341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973cb920 0x5593973e6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f2158d79740 1 --2- 172.21.15.31:0/1622399341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 0x5593973d9c50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f2158d79740 1 -- 172.21.15.31:0/1622399341 >> 172.21.15.31:0/1622399341 conn(0x5593973c7560 msgr2=0x5593973c65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f2158d79740 1 -- 172.21.15.31:0/1622399341 shutdown_connections 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f2158d79740 1 -- 172.21.15.31:0/1622399341 wait complete. 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.683+0000 7f2158d79740 1 -- start start 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f2158d79740 1 Processor -- start 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f2158d79740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 0x5593973e2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f2158d79740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973cb920 0x5593973ddfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f2158d79740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973de540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f2158d79740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f21501ac530 con 0x5593973cb920 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f2158d79740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21501abe60 con 0x5593973caf70 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f2158d79740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f21501a90c0 con 0x5593973da640 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f2158d79740 10 client.0 _ll_register_callbacks cb 0x7f214000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f214effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973cb920 0x5593973ddfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f214f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 0x5593973e2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f214effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973cb920 0x5593973ddfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41502/0 (socket says 172.21.15.31:41502) 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f214effd640 1 -- 172.21.15.31:0/1017168601 learned_addr learned my addr 172.21.15.31:0/1017168601 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.684+0000 7f214ffff640 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f214effd640 1 -- 172.21.15.31:0/1017168601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 msgr2=0x5593973de540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f214effd640 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973de540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f214effd640 1 -- 172.21.15.31:0/1017168601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 msgr2=0x5593973e2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f214effd640 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 0x5593973e2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f214effd640 1 -- 172.21.15.31:0/1017168601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593973ded70 con 0x5593973cb920 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f214f7fe640 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 0x5593973e2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f214effd640 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973cb920 0x5593973ddfe0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f21440057c0 tx=0x7f2144005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6749b5ba9386910 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f212effd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f212effd640 1 -- 172.21.15.31:0/1017168601 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2144016040 con 0x5593973cb920 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f212effd640 1 -- 172.21.15.31:0/1017168601 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f2144004220 con 0x5593973cb920 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f2158d79740 1 -- 172.21.15.31:0/1017168601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2140018ac0 con 0x5593973cb920 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f212effd640 1 -- 172.21.15.31:0/1017168601 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2144038050 con 0x5593973cb920 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f2158d79740 10 client.16875 fetch_fsmap learned FSMap version 34 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f2158d79740 1 -- 172.21.15.31:0/1017168601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559397451080 con 0x5593973cb920 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.685+0000 7f212effd640 1 -- 172.21.15.31:0/1017168601 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2140018ac0 con 0x5593973cb920 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.686+0000 7f212effd640 1 -- 172.21.15.31:0/1017168601 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f214400f040 con 0x5593973cb920 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.686+0000 7f2158d79740 10 client.16875 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.686+0000 7f2158d79740 10 client.16875 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.686+0000 7f2158d79740 20 client.16875 populate_metadata read hostname 'smithi031' 2024-04-25T02:03:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.686+0000 7f2158d79740 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559397451520 0x559397453910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.686+0000 7f2158d79740 4 client.16875 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:03:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.686+0000 7f2158d79740 1 -- 172.21.15.31:0/1017168601 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559397337540 con 0x559397451520 2024-04-25T02:03:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.686+0000 7f214f7fe640 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559397451520 0x559397453910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.686+0000 7f214f7fe640 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559397451520 0x559397453910 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=734b7868f27047dc server_cookie=bebf51ca0a2b14e3 in_seq=0 out_seq=0 2024-04-25T02:03:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.686+0000 7f212effd640 10 client.16875 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f212effd640 1 -- 172.21.15.31:0/1017168601 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559397337540 con 0x559397451520 2024-04-25T02:03:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f212effd640 10 client.16875 handle_command_reply: tid=0 2024-04-25T02:03:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f2158d79740 1 client.16875 shutdown 2024-04-25T02:03:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f2158d79740 1 -- 172.21.15.31:0/1017168601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973cb920 msgr2=0x5593973ddfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f2158d79740 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973cb920 0x5593973ddfe0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f21440057c0 tx=0x7f2144005850 comp rx=0 tx=0).stop 2024-04-25T02:03:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f2158d79740 1 -- 172.21.15.31:0/1017168601 shutdown_connections 2024-04-25T02:03:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f2158d79740 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559397451520 0x559397453910 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f2158d79740 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593973da640 0x5593973de540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f2158d79740 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593973cb920 0x5593973ddfe0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f212effd640 0 client.16875 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f2158d79740 1 --2- 172.21.15.31:0/1017168601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593973caf70 0x5593973e2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.709+0000 7f2158d79740 1 -- 172.21.15.31:0/1017168601 >> 172.21.15.31:0/1017168601 conn(0x5593973c7560 msgr2=0x5593973d74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f2158d79740 1 -- 172.21.15.31:0/1017168601 shutdown_connections 2024-04-25T02:03:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f2158d79740 1 -- 172.21.15.31:0/1017168601 wait complete. 2024-04-25T02:03:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f2158d79740 20 client.16875 trim_cache size 0 max 16384 2024-04-25T02:03:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:03:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:03:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:03:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:03:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 -- 172.21.15.31:0/250049003 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21380a2da0 msgr2=0x7f21380a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21380a2da0 0x7f21380a5260 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2148005e10 tx=0x7f214802a8e0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 -- 172.21.15.31:0/250049003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2150100670 msgr2=0x7f2150197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2150100670 0x7f2150197f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f214000e9f0 tx=0x7f214000eeb0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 -- 172.21.15.31:0/250049003 shutdown_connections 2024-04-25T02:03:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f21380a2da0 0x7f21380a5260 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2150106630 0x7f215019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2150101030 0x7f21501984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 --2- 172.21.15.31:0/250049003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2150100670 0x7f2150197f60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 -- 172.21.15.31:0/250049003 >> 172.21.15.31:0/250049003 conn(0x7f21500777c0 msgr2=0x7f2150069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 -- 172.21.15.31:0/250049003 shutdown_connections 2024-04-25T02:03:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:25.710+0000 7f215606b640 1 -- 172.21.15.31:0/250049003 wait complete. 2024-04-25T02:03:26.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:25 smithi040 ceph-mon[32418]: pgmap v278: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 866 KiB/s wr, 99 op/s 2024-04-25T02:03:26.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:25 smithi134 ceph-mon[33098]: pgmap v278: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 866 KiB/s wr, 99 op/s 2024-04-25T02:03:26.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:25 smithi031 ceph-mon[29204]: pgmap v278: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 866 KiB/s wr, 99 op/s 2024-04-25T02:03:26.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 208738 2.41 MB/sec execute 333 sec latency 19.151 ms 2024-04-25T02:03:27.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209218 2.41 MB/sec execute 334 sec latency 20.624 ms 2024-04-25T02:03:28.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:27 smithi040 ceph-mon[32418]: pgmap v279: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 523 KiB/s wr, 67 op/s 2024-04-25T02:03:28.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:27 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:28.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:27 smithi134 ceph-mon[33098]: pgmap v279: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 523 KiB/s wr, 67 op/s 2024-04-25T02:03:28.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:27 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:28.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:27 smithi031 ceph-mon[29204]: pgmap v279: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 523 KiB/s wr, 67 op/s 2024-04-25T02:03:28.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:27 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:28.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209346 2.41 MB/sec execute 335 sec latency 731.067 ms 2024-04-25T02:03:29.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:28 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:29.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:28 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:29.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:28 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:29.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209346 2.40 MB/sec execute 336 sec latency 1731.140 ms 2024-04-25T02:03:30.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:29 smithi040 ceph-mon[32418]: pgmap v280: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 523 KiB/s wr, 67 op/s 2024-04-25T02:03:30.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:29 smithi134 ceph-mon[33098]: pgmap v280: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 523 KiB/s wr, 67 op/s 2024-04-25T02:03:30.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:29 smithi031 ceph-mon[29204]: pgmap v280: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 523 KiB/s wr, 67 op/s 2024-04-25T02:03:30.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209413 2.39 MB/sec execute 337 sec latency 2455.777 ms 2024-04-25T02:03:31.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209413 2.39 MB/sec execute 338 sec latency 1025.101 ms 2024-04-25T02:03:32.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:31 smithi040 ceph-mon[32418]: pgmap v281: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:03:32.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:31 smithi134 ceph-mon[33098]: pgmap v281: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:03:32.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:31 smithi031 ceph-mon[29204]: pgmap v281: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:03:32.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209413 2.38 MB/sec execute 339 sec latency 2025.163 ms 2024-04-25T02:03:33.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:32 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:33.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:32 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:33.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:32 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:33.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209413 2.37 MB/sec execute 340 sec latency 3025.284 ms 2024-04-25T02:03:34.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:33 smithi040 ceph-mon[32418]: pgmap v282: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1009 KiB/s wr, 97 op/s 2024-04-25T02:03:34.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:33 smithi134 ceph-mon[33098]: pgmap v282: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1009 KiB/s wr, 97 op/s 2024-04-25T02:03:34.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:33 smithi031 ceph-mon[29204]: pgmap v282: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1009 KiB/s wr, 97 op/s 2024-04-25T02:03:34.317 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209413 2.37 MB/sec execute 341 sec latency 4025.329 ms 2024-04-25T02:03:35.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209425 2.36 MB/sec execute 342 sec latency 4747.825 ms 2024-04-25T02:03:36.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:35 smithi040 ceph-mon[32418]: pgmap v283: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-04-25T02:03:36.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:35 smithi134 ceph-mon[33098]: pgmap v283: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-04-25T02:03:36.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:35 smithi031 ceph-mon[29204]: pgmap v283: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-04-25T02:03:36.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209425 2.35 MB/sec execute 343 sec latency 1223.842 ms 2024-04-25T02:03:37.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209694 2.35 MB/sec execute 344 sec latency 1235.191 ms 2024-04-25T02:03:38.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:37 smithi040 ceph-mon[32418]: pgmap v284: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 559 KiB/s wr, 51 op/s 2024-04-25T02:03:38.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:37 smithi134 ceph-mon[33098]: pgmap v284: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 559 KiB/s wr, 51 op/s 2024-04-25T02:03:38.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:37 smithi031 ceph-mon[29204]: pgmap v284: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 559 KiB/s wr, 51 op/s 2024-04-25T02:03:38.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 210036 2.34 MB/sec execute 345 sec latency 17.984 ms 2024-04-25T02:03:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.807+0000 7fa822c91640 1 -- 172.21.15.31:0/3512510774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa81c1032f0 msgr2=0x7fa81c10b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.807+0000 7fa822c91640 1 --2- 172.21.15.31:0/3512510774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa81c1032f0 0x7fa81c10b7d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009860 tx=0x7fa80c02a460 comp rx=0 tx=0).stop 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.807+0000 7fa822c91640 1 -- 172.21.15.31:0/3512510774 shutdown_connections 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.807+0000 7fa822c91640 1 --2- 172.21.15.31:0/3512510774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa81c10bd10 0x7fa81c10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.807+0000 7fa822c91640 1 --2- 172.21.15.31:0/3512510774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa81c1032f0 0x7fa81c10b7d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.807+0000 7fa822c91640 1 --2- 172.21.15.31:0/3512510774 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c1029d0 0x7fa81c102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.807+0000 7fa822c91640 1 -- 172.21.15.31:0/3512510774 >> 172.21.15.31:0/3512510774 conn(0x7fa81c0777c0 msgr2=0x7fa81c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.807+0000 7fa822c91640 1 -- 172.21.15.31:0/3512510774 shutdown_connections 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.807+0000 7fa822c91640 1 -- 172.21.15.31:0/3512510774 wait complete. 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa822c91640 1 Processor -- start 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa822c91640 1 -- start start 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa822c91640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c1029d0 0x7fa81c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa822c91640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa81c1032f0 0x7fa81c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa822c91640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa81c10bd10 0x7fa81c19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa822c91640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1119c0 con 0x7fa81c10bd10 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa822c91640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c111840 con 0x7fa81c1032f0 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa822c91640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c111b40 con 0x7fa81c1029d0 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa82122e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa81c10bd10 0x7fa81c19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa82122e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa81c10bd10 0x7fa81c19ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57322/0 (socket says 172.21.15.31:57322) 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa81c1032f0 0x7fa81c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa82122e640 1 -- 172.21.15.31:0/1648656937 learned_addr learned my addr 172.21.15.31:0/1648656937 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa820a2d640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c1029d0 0x7fa81c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa82122e640 1 -- 172.21.15.31:0/1648656937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c1029d0 msgr2=0x7fa81c19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa82122e640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c1029d0 0x7fa81c19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa82122e640 1 -- 172.21.15.31:0/1648656937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa81c1032f0 msgr2=0x7fa81c19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa82122e640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa81c1032f0 0x7fa81c19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa82122e640 1 -- 172.21.15.31:0/1648656937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c19f190 con 0x7fa81c10bd10 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa820a2d640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c1029d0 0x7fa81c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:03:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.808+0000 7fa81bfff640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa81c1032f0 0x7fa81c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.809+0000 7fa82122e640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa81c10bd10 0x7fa81c19ea60 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fa81400efc0 tx=0x7fa81400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.809+0000 7fa819ffb640 1 -- 172.21.15.31:0/1648656937 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814003970 con 0x7fa81c10bd10 2024-04-25T02:03:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.809+0000 7fa822c91640 1 -- 172.21.15.31:0/1648656937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81c19f480 con 0x7fa81c10bd10 2024-04-25T02:03:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.809+0000 7fa819ffb640 1 -- 172.21.15.31:0/1648656937 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa814004020 con 0x7fa81c10bd10 2024-04-25T02:03:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.809+0000 7fa819ffb640 1 -- 172.21.15.31:0/1648656937 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81400b940 con 0x7fa81c10bd10 2024-04-25T02:03:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.809+0000 7fa822c91640 1 -- 172.21.15.31:0/1648656937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81c1a6d10 con 0x7fa81c10bd10 2024-04-25T02:03:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.812+0000 7fa819ffb640 1 -- 172.21.15.31:0/1648656937 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa81401d020 con 0x7fa81c10bd10 2024-04-25T02:03:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.812+0000 7fa825995740 1 Processor -- start 2024-04-25T02:03:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.812+0000 7fa825995740 1 -- start start 2024-04-25T02:03:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.812+0000 7fa825995740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x556610184620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.812+0000 7fa825995740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 0x55661018ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.812+0000 7fa825995740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55661018f230 0x556610191620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.812+0000 7fa825995740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5566100e8cb0 con 0x556610184240 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.812+0000 7fa825995740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5566100915b0 con 0x556610184b60 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.812+0000 7fa825995740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5566100e4c70 con 0x55661018f230 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa819ffb640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8040a2da0 0x7fa8040a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x556610184620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa820a2d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 0x55661018ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x556610184620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57338/0 (socket says 172.21.15.31:57338) 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa820a2d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 0x55661018ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39078/0 (socket says 172.21.15.31:39078) 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa81bfff640 1 -- 172.21.15.31:0/1509447906 learned_addr learned my addr 172.21.15.31:0/1509447906 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa82122e640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8040a2da0 0x7fa8040a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa819ffb640 1 -- 172.21.15.31:0/1648656937 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7fa8140cf2f0 con 0x7fa81c10bd10 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa81bfff640 1 --2- 172.21.15.31:0/1509447906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55661018f230 0x556610191620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa820a2d640 1 -- 172.21.15.31:0/1509447906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55661018f230 msgr2=0x556610191620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa820a2d640 1 --2- 172.21.15.31:0/1509447906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55661018f230 0x556610191620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa820a2d640 1 -- 172.21.15.31:0/1509447906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 msgr2=0x556610184620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa820a2d640 1 --2- 172.21.15.31:0/1509447906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x556610184620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa820a2d640 1 -- 172.21.15.31:0/1509447906 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55660fdfd210 con 0x556610184b60 2024-04-25T02:03:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa81bfff640 1 --2- 172.21.15.31:0/1509447906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55661018f230 0x556610191620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:03:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa820a2d640 1 --2- 172.21.15.31:0/1509447906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 0x55661018ecf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa810009c30 tx=0x7fa8100117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f635adf4238bf337 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa82122e640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8040a2da0 0x7fa8040a5260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa8140cd060 tx=0x7fa8140cf5f0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa802ffd640 1 -- 172.21.15.31:0/1509447906 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810005020 con 0x556610184b60 2024-04-25T02:03:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa802ffd640 1 -- 172.21.15.31:0/1509447906 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8100047a0 con 0x556610184b60 2024-04-25T02:03:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.813+0000 7fa802ffd640 1 -- 172.21.15.31:0/1509447906 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810012740 con 0x556610184b60 2024-04-25T02:03:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.814+0000 7fa825995740 1 -- 172.21.15.31:0/1509447906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 msgr2=0x55661018ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.814+0000 7fa825995740 1 --2- 172.21.15.31:0/1509447906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 0x55661018ecf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa810009c30 tx=0x7fa8100117b0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.814+0000 7fa825995740 1 -- 172.21.15.31:0/1509447906 shutdown_connections 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.814+0000 7fa825995740 1 --2- 172.21.15.31:0/1509447906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55661018f230 0x556610191620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.814+0000 7fa825995740 1 --2- 172.21.15.31:0/1509447906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 0x55661018ecf0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.814+0000 7fa825995740 1 --2- 172.21.15.31:0/1509447906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x556610184620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.814+0000 7fa825995740 1 -- 172.21.15.31:0/1509447906 >> 172.21.15.31:0/1509447906 conn(0x556610180560 msgr2=0x556610180940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.814+0000 7fa825995740 1 -- 172.21.15.31:0/1509447906 shutdown_connections 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.814+0000 7fa825995740 1 -- 172.21.15.31:0/1509447906 wait complete. 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa825995740 1 -- start start 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa825995740 1 Processor -- start 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa825995740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x55661018c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa825995740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 0x55661018c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa825995740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55661018f230 0x55661018cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa825995740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556610192020 con 0x556610184240 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa825995740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5566100a4b50 con 0x556610184b60 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa825995740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5566101921a0 con 0x55661018f230 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa825995740 10 client.0 _ll_register_callbacks cb 0x5566100dfca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x55661018c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x55661018c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57342/0 (socket says 172.21.15.31:57342) 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa81bfff640 1 -- 172.21.15.31:0/1800745763 learned_addr learned my addr 172.21.15.31:0/1800745763 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.815+0000 7fa820a2d640 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 0x55661018c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa81bfff640 1 -- 172.21.15.31:0/1800745763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55661018f230 msgr2=0x55661018cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa81bfff640 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55661018f230 0x55661018cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa81bfff640 1 -- 172.21.15.31:0/1800745763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 msgr2=0x55661018c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa81bfff640 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 0x55661018c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa81bfff640 1 -- 172.21.15.31:0/1800745763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55660fec9620 con 0x556610184240 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa820a2d640 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 0x55661018c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa81bfff640 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x55661018c150 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa80c031000 tx=0x7fa80c030830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1abaad2140b89ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa7fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa7fbfff640 1 -- 172.21.15.31:0/1800745763 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c0046c0 con 0x556610184240 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa7fbfff640 1 -- 172.21.15.31:0/1800745763 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7fa80c004860 con 0x556610184240 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa825995740 1 -- 172.21.15.31:0/1800745763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8100047a0 con 0x556610184240 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa7fbfff640 1 -- 172.21.15.31:0/1800745763 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa80c004a60 con 0x556610184240 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa825995740 10 client.16887 fetch_fsmap learned FSMap version 38 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.816+0000 7fa825995740 1 -- 172.21.15.31:0/1800745763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55661019e1a0 con 0x556610184240 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.817+0000 7fa7fbfff640 1 -- 172.21.15.31:0/1800745763 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8100047a0 con 0x556610184240 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.817+0000 7fa7fbfff640 1 -- 172.21.15.31:0/1800745763 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fa80c050020 con 0x556610184240 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.817+0000 7fa825995740 10 client.16887 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.817+0000 7fa825995740 10 client.16887 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.817+0000 7fa825995740 20 client.16887 populate_metadata read hostname 'smithi031' 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.817+0000 7fa825995740 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55661019f660 0x5566101a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.817+0000 7fa825995740 4 client.16887 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.817+0000 7fa825995740 1 -- 172.21.15.31:0/1800745763 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566100f61f0 con 0x55661019f660 2024-04-25T02:03:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.817+0000 7fa820a2d640 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55661019f660 0x5566101a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa7fbfff640 10 client.16887 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa820a2d640 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55661019f660 0x5566101a1a50 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44b483313b514c3c server_cookie=75e1ca7e9de5053a in_seq=0 out_seq=0 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa7fbfff640 1 -- 172.21.15.31:0/1800745763 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5566100f61f0 con 0x55661019f660 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa7fbfff640 10 client.16887 handle_command_reply: tid=0 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa825995740 1 client.16887 shutdown 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa825995740 1 -- 172.21.15.31:0/1800745763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 msgr2=0x55661018c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa825995740 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x55661018c150 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa80c031000 tx=0x7fa80c030830 comp rx=0 tx=0).stop 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa825995740 1 -- 172.21.15.31:0/1800745763 shutdown_connections 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa825995740 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55661019f660 0x5566101a1a50 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa7fbfff640 0 client.16887 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa825995740 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55661018f230 0x55661018cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa825995740 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184b60 0x55661018c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa825995740 1 --2- 172.21.15.31:0/1800745763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x55661018c150 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.818+0000 7fa825995740 1 -- 172.21.15.31:0/1800745763 >> 172.21.15.31:0/1800745763 conn(0x556610180560 msgr2=0x556610186260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.819+0000 7fa825995740 1 -- 172.21.15.31:0/1800745763 shutdown_connections 2024-04-25T02:03:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.819+0000 7fa825995740 1 -- 172.21.15.31:0/1800745763 wait complete. 2024-04-25T02:03:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.819+0000 7fa825995740 20 client.16887 trim_cache size 0 max 16384 2024-04-25T02:03:38.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa825995740 1 Processor -- start 2024-04-25T02:03:38.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa825995740 1 -- start start 2024-04-25T02:03:38.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa825995740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x55661018e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa825995740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 0x55661019af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa825995740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55661018f230 0x55661018ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa825995740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1af3d0 con 0x556610184240 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa825995740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1b1030 con 0x55661018f230 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa825995740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1afaa0 con 0x556610184b60 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x55661018e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa820a2d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 0x55661019af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x55661018e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57356/0 (socket says 172.21.15.31:57356) 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa820a2d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 0x55661019af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:33220/0 (socket says 172.21.15.31:33220) 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa81bfff640 1 -- 172.21.15.31:0/4003943402 learned_addr learned my addr 172.21.15.31:0/4003943402 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa81bfff640 1 --2- 172.21.15.31:0/4003943402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55661018f230 0x55661018ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa820a2d640 1 -- 172.21.15.31:0/4003943402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55661018f230 msgr2=0x55661018ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa820a2d640 1 --2- 172.21.15.31:0/4003943402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55661018f230 0x55661018ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa820a2d640 1 -- 172.21.15.31:0/4003943402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 msgr2=0x55661018e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa820a2d640 1 --2- 172.21.15.31:0/4003943402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x55661018e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa820a2d640 1 -- 172.21.15.31:0/4003943402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556610184990 con 0x556610184b60 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.835+0000 7fa81bfff640 1 --2- 172.21.15.31:0/4003943402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55661018f230 0x55661018ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.836+0000 7fa820a2d640 1 --2- 172.21.15.31:0/4003943402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 0x55661019af10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa810011cb0 tx=0x7fa810005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0fa300b2eea46e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.836+0000 7fa8027fc640 1 -- 172.21.15.31:0/4003943402 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810004770 con 0x556610184b60 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.836+0000 7fa8027fc640 1 -- 172.21.15.31:0/4003943402 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa810009d20 con 0x556610184b60 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.836+0000 7fa8027fc640 1 -- 172.21.15.31:0/4003943402 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810012770 con 0x556610184b60 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.836+0000 7fa825995740 1 -- 172.21.15.31:0/4003943402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 msgr2=0x55661019af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.836+0000 7fa825995740 1 --2- 172.21.15.31:0/4003943402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 0x55661019af10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa810011cb0 tx=0x7fa810005f90 comp rx=0 tx=0).stop 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 -- 172.21.15.31:0/4003943402 shutdown_connections 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 --2- 172.21.15.31:0/4003943402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55661018f230 0x55661018ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 --2- 172.21.15.31:0/4003943402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 0x55661019af10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 --2- 172.21.15.31:0/4003943402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556610184240 0x55661018e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 -- 172.21.15.31:0/4003943402 >> 172.21.15.31:0/4003943402 conn(0x556610180560 msgr2=0x55661017f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 -- 172.21.15.31:0/4003943402 shutdown_connections 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 -- 172.21.15.31:0/4003943402 wait complete. 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 -- start start 2024-04-25T02:03:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 Processor -- start 2024-04-25T02:03:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184240 0x556610197420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 0x556610197960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55661018f230 0x556610188480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1b0960 con 0x55661018f230 2024-04-25T02:03:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1b01d0 con 0x556610184240 2024-04-25T02:03:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1ad3d0 con 0x556610184b60 2024-04-25T02:03:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.837+0000 7fa825995740 10 client.0 _ll_register_callbacks cb 0x7fa80c0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184240 0x556610197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55661018f230 0x556610188480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55661018f230 0x556610188480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57368/0 (socket says 172.21.15.31:57368) 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa820a2d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 0x556610197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa81bfff640 1 -- 172.21.15.31:0/4262341905 learned_addr learned my addr 172.21.15.31:0/4262341905 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa820a2d640 1 -- 172.21.15.31:0/4262341905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184240 msgr2=0x556610197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa820a2d640 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184240 0x556610197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa820a2d640 1 -- 172.21.15.31:0/4262341905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55661018f230 msgr2=0x556610188480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa820a2d640 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55661018f230 0x556610188480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa820a2d640 1 -- 172.21.15.31:0/4262341905 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556610188b20 con 0x556610184b60 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa81bfff640 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55661018f230 0x556610188480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa820a2d640 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 0x556610197960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55661019bbe0 tx=0x7fa8100117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7863f879f536cb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa7fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa7fbfff640 1 -- 172.21.15.31:0/4262341905 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810039060 con 0x556610184b60 2024-04-25T02:03:38.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa825995740 1 -- 172.21.15.31:0/4262341905 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa810009d20 con 0x556610184b60 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.838+0000 7fa7fbfff640 1 -- 172.21.15.31:0/4262341905 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7fa8100125f0 con 0x556610184b60 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa7fbfff640 1 -- 172.21.15.31:0/4262341905 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa810012850 con 0x556610184b60 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa825995740 10 client.26293 fetch_fsmap learned FSMap version 38 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa825995740 1 -- 172.21.15.31:0/4262341905 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556610205490 con 0x556610184b60 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa7fbfff640 1 -- 172.21.15.31:0/4262341905 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa810009d20 con 0x556610184b60 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa7fbfff640 1 -- 172.21.15.31:0/4262341905 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fa810005370 con 0x556610184b60 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa825995740 10 client.26293 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa825995740 10 client.26293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa825995740 20 client.26293 populate_metadata read hostname 'smithi031' 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa825995740 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55661019f660 0x556610207b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa825995740 4 client.26293 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa825995740 1 -- 172.21.15.31:0/4262341905 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5566100f0540 con 0x55661019f660 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.839+0000 7fa81bfff640 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55661019f660 0x556610207b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.840+0000 7fa81bfff640 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55661019f660 0x556610207b20 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69ab4fa3354722c0 server_cookie=1bbcd33824ab55f0 in_seq=0 out_seq=0 2024-04-25T02:03:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.840+0000 7fa7fbfff640 10 client.26293 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:38.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.868+0000 7fa7fbfff640 1 -- 172.21.15.31:0/4262341905 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5566100f0540 con 0x55661019f660 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.868+0000 7fa7fbfff640 10 client.26293 handle_command_reply: tid=0 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.868+0000 7fa825995740 1 client.26293 shutdown 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.868+0000 7fa825995740 1 -- 172.21.15.31:0/4262341905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 msgr2=0x556610197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.868+0000 7fa825995740 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 0x556610197960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55661019bbe0 tx=0x7fa8100117b0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.868+0000 7fa825995740 1 -- 172.21.15.31:0/4262341905 shutdown_connections 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa825995740 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55661019f660 0x556610207b20 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa825995740 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55661018f230 0x556610188480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa825995740 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556610184b60 0x556610197960 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa7fbfff640 0 client.26293 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa825995740 1 --2- 172.21.15.31:0/4262341905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556610184240 0x556610197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa825995740 1 -- 172.21.15.31:0/4262341905 >> 172.21.15.31:0/4262341905 conn(0x556610180560 msgr2=0x556610185990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa825995740 1 -- 172.21.15.31:0/4262341905 shutdown_connections 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa825995740 1 -- 172.21.15.31:0/4262341905 wait complete. 2024-04-25T02:03:38.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa825995740 20 client.26293 trim_cache size 0 max 16384 2024-04-25T02:03:38.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:03:38.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:03:38.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:03:38.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa822c91640 1 -- 172.21.15.31:0/1648656937 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8040a2da0 msgr2=0x7fa8040a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa822c91640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8040a2da0 0x7fa8040a5260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa8140cd060 tx=0x7fa8140cf5f0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa822c91640 1 -- 172.21.15.31:0/1648656937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa81c10bd10 msgr2=0x7fa81c19ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.869+0000 7fa822c91640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa81c10bd10 0x7fa81c19ea60 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fa81400efc0 tx=0x7fa81400c800 comp rx=0 tx=0).stop 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.870+0000 7fa822c91640 1 -- 172.21.15.31:0/1648656937 shutdown_connections 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.870+0000 7fa822c91640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8040a2da0 0x7fa8040a5260 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.870+0000 7fa822c91640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa81c10bd10 0x7fa81c19ea60 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.870+0000 7fa822c91640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa81c1032f0 0x7fa81c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.870+0000 7fa822c91640 1 --2- 172.21.15.31:0/1648656937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c1029d0 0x7fa81c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.870+0000 7fa822c91640 1 -- 172.21.15.31:0/1648656937 >> 172.21.15.31:0/1648656937 conn(0x7fa81c0777c0 msgr2=0x7fa81c107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.870+0000 7fa822c91640 1 -- 172.21.15.31:0/1648656937 shutdown_connections 2024-04-25T02:03:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:38.870+0000 7fa822c91640 1 -- 172.21.15.31:0/1648656937 wait complete. 2024-04-25T02:03:39.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:38 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:39.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:38 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:39.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:38 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:03:39.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 210505 2.34 MB/sec execute 346 sec latency 18.994 ms 2024-04-25T02:03:40.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:39 smithi040 ceph-mon[32418]: pgmap v285: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 559 KiB/s wr, 51 op/s 2024-04-25T02:03:40.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:39 smithi134 ceph-mon[33098]: pgmap v285: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 559 KiB/s wr, 51 op/s 2024-04-25T02:03:40.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:39 smithi031 ceph-mon[29204]: pgmap v285: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 559 KiB/s wr, 51 op/s 2024-04-25T02:03:40.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 210973 2.34 MB/sec execute 347 sec latency 25.343 ms 2024-04-25T02:03:41.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 211538 2.34 MB/sec execute 348 sec latency 23.517 ms 2024-04-25T02:03:42.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:41 smithi134 ceph-mon[33098]: pgmap v286: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1021 KiB/s wr, 102 op/s 2024-04-25T02:03:42.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:41 smithi031 ceph-mon[29204]: pgmap v286: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1021 KiB/s wr, 102 op/s 2024-04-25T02:03:42.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 212051 2.35 MB/sec execute 349 sec latency 18.074 ms 2024-04-25T02:03:42.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:41 smithi040 ceph-mon[32418]: pgmap v286: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1021 KiB/s wr, 102 op/s 2024-04-25T02:03:43.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:42 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:03:43.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:42 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:03:43.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:42 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:03:43.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:42 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:03:43.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:42 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:03:43.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:42 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:03:43.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:42 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:03:43.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:42 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:03:43.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 212600 2.35 MB/sec execute 350 sec latency 17.669 ms 2024-04-25T02:03:43.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:42 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:03:43.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:42 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:03:43.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:42 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:03:43.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:42 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:03:44.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:43 smithi134 ceph-mon[33098]: pgmap v287: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 501 KiB/s wr, 62 op/s 2024-04-25T02:03:44.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:43 smithi031 ceph-mon[29204]: pgmap v287: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 501 KiB/s wr, 62 op/s 2024-04-25T02:03:44.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 212892 2.35 MB/sec execute 351 sec latency 461.906 ms 2024-04-25T02:03:44.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:43 smithi040 ceph-mon[32418]: pgmap v287: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 501 KiB/s wr, 62 op/s 2024-04-25T02:03:45.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 212959 2.35 MB/sec execute 352 sec latency 1188.371 ms 2024-04-25T02:03:46.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:45 smithi134 ceph-mon[33098]: pgmap v288: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-04-25T02:03:46.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:45 smithi031 ceph-mon[29204]: pgmap v288: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-04-25T02:03:46.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 212959 2.34 MB/sec execute 353 sec latency 1031.646 ms 2024-04-25T02:03:46.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:45 smithi040 ceph-mon[32418]: pgmap v288: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-04-25T02:03:47.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 212959 2.33 MB/sec execute 354 sec latency 2031.749 ms 2024-04-25T02:03:48.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:47 smithi134 ceph-mon[33098]: pgmap v289: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1000 KiB/s wr, 95 op/s 2024-04-25T02:03:48.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:47 smithi031 ceph-mon[29204]: pgmap v289: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1000 KiB/s wr, 95 op/s 2024-04-25T02:03:48.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 212959 2.33 MB/sec execute 355 sec latency 3031.801 ms 2024-04-25T02:03:48.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:47 smithi040 ceph-mon[32418]: pgmap v289: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1000 KiB/s wr, 95 op/s 2024-04-25T02:03:49.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 212959 2.32 MB/sec execute 356 sec latency 4031.863 ms 2024-04-25T02:03:50.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:49 smithi134 ceph-mon[33098]: pgmap v290: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1000 KiB/s wr, 95 op/s 2024-04-25T02:03:50.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:49 smithi031 ceph-mon[29204]: pgmap v290: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1000 KiB/s wr, 95 op/s 2024-04-25T02:03:50.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 213055 2.32 MB/sec execute 357 sec latency 4754.717 ms 2024-04-25T02:03:50.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:49 smithi040 ceph-mon[32418]: pgmap v290: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1000 KiB/s wr, 95 op/s 2024-04-25T02:03:51.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 213298 2.31 MB/sec execute 358 sec latency 17.091 ms 2024-04-25T02:03:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.968+0000 7ff1399fb640 1 -- 172.21.15.31:0/2190939010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff134107770 msgr2=0x7ff134107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.968+0000 7ff1399fb640 1 --2- 172.21.15.31:0/2190939010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff134107770 0x7ff134107b50 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7ff128009930 tx=0x7ff12802a420 comp rx=0 tx=0).stop 2024-04-25T02:03:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.968+0000 7ff1399fb640 1 -- 172.21.15.31:0/2190939010 shutdown_connections 2024-04-25T02:03:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.968+0000 7ff1399fb640 1 --2- 172.21.15.31:0/2190939010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff134102120 0x7ff13410e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.968+0000 7ff1399fb640 1 --2- 172.21.15.31:0/2190939010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff134101760 0x7ff134101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.968+0000 7ff1399fb640 1 --2- 172.21.15.31:0/2190939010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff134107770 0x7ff134107b50 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.968+0000 7ff1399fb640 1 -- 172.21.15.31:0/2190939010 >> 172.21.15.31:0/2190939010 conn(0x7ff1340fd1f0 msgr2=0x7ff1340ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.968+0000 7ff1399fb640 1 -- 172.21.15.31:0/2190939010 shutdown_connections 2024-04-25T02:03:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.968+0000 7ff1399fb640 1 -- 172.21.15.31:0/2190939010 wait complete. 2024-04-25T02:03:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1399fb640 1 Processor -- start 2024-04-25T02:03:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1399fb640 1 -- start start 2024-04-25T02:03:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1399fb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff134101760 0x7ff134073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1399fb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff134102120 0x7ff134073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1399fb640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff134107770 0x7ff1340744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1399fb640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff134111a40 con 0x7ff134102120 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1399fb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341118c0 con 0x7ff134101760 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1399fb640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff134111bc0 con 0x7ff134107770 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1327fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff134102120 0x7ff134073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1327fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff134102120 0x7ff134073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47090/0 (socket says 172.21.15.31:47090) 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1327fc640 1 -- 172.21.15.31:0/2634183535 learned_addr learned my addr 172.21.15.31:0/2634183535 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff1337fe640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff134107770 0x7ff1340744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.969+0000 7ff132ffd640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff134101760 0x7ff134073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1327fc640 1 -- 172.21.15.31:0/2634183535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff134107770 msgr2=0x7ff1340744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1327fc640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff134107770 0x7ff1340744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1327fc640 1 -- 172.21.15.31:0/2634183535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff134101760 msgr2=0x7ff134073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1327fc640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff134101760 0x7ff134073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1327fc640 1 -- 172.21.15.31:0/2634183535 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1341a6400 con 0x7ff134102120 2024-04-25T02:03:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1337fe640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff134107770 0x7ff1340744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1327fc640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff134102120 0x7ff134073f80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7ff12400e9f0 tx=0x7ff12400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1389f9640 1 -- 172.21.15.31:0/2634183535 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12400cd80 con 0x7ff134102120 2024-04-25T02:03:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1399fb640 1 -- 172.21.15.31:0/2634183535 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1341a66f0 con 0x7ff134102120 2024-04-25T02:03:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1389f9640 1 -- 172.21.15.31:0/2634183535 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1240108e0 con 0x7ff134102120 2024-04-25T02:03:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1389f9640 1 -- 172.21.15.31:0/2634183535 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124018920 con 0x7ff134102120 2024-04-25T02:03:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.970+0000 7ff1399fb640 1 -- 172.21.15.31:0/2634183535 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1341a6c40 con 0x7ff134102120 2024-04-25T02:03:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.972+0000 7ff1389f9640 1 -- 172.21.15.31:0/2634183535 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff124010a90 con 0x7ff134102120 2024-04-25T02:03:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff13c7e0740 1 Processor -- start 2024-04-25T02:03:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff13c7e0740 1 -- start start 2024-04-25T02:03:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff1389f9640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff11c0a2da0 0x7ff11c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d18620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff1337fe640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff11c0a2da0 0x7ff11c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d18b60 0x563918d22cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d23230 0x563918d25620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563918c7ccb0 con 0x563918d18240 2024-04-25T02:03:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563918c255b0 con 0x563918d23230 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563918c78c70 con 0x563918d18b60 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff1389f9640 1 -- 172.21.15.31:0/2634183535 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7ff1240cf700 con 0x7ff134102120 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff132ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d18620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff132ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d18620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47102/0 (socket says 172.21.15.31:47102) 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff132ffd640 1 -- 172.21.15.31:0/2364808826 learned_addr learned my addr 172.21.15.31:0/2364808826 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.973+0000 7ff1327fc640 1 --2- 172.21.15.31:0/2364808826 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d18b60 0x563918d22cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.974+0000 7ff1337fe640 1 --2- 172.21.15.31:0/2364808826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d23230 0x563918d25620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.974+0000 7ff132ffd640 1 -- 172.21.15.31:0/2364808826 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d18b60 msgr2=0x563918d22cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.974+0000 7ff132ffd640 1 --2- 172.21.15.31:0/2364808826 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d18b60 0x563918d22cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.974+0000 7ff132ffd640 1 -- 172.21.15.31:0/2364808826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d23230 msgr2=0x563918d25620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.974+0000 7ff132ffd640 1 --2- 172.21.15.31:0/2364808826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d23230 0x563918d25620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.974+0000 7ff132ffd640 1 -- 172.21.15.31:0/2364808826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563918991210 con 0x563918d18240 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.974+0000 7ff1337fe640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff11c0a2da0 0x7ff11c0a5260 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009fd0 tx=0x7ff12c011f60 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.974+0000 7ff132ffd640 1 --2- 172.21.15.31:0/2364808826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d18620 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff128006010 tx=0x7ff128004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a7c7d3bbc3034a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.974+0000 7ff119ffb640 1 -- 172.21.15.31:0/2364808826 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128035040 con 0x563918d18240 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.974+0000 7ff119ffb640 1 -- 172.21.15.31:0/2364808826 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1280054f0 con 0x563918d18240 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.974+0000 7ff119ffb640 1 -- 172.21.15.31:0/2364808826 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1280035f0 con 0x563918d18240 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.975+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2364808826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 msgr2=0x563918d18620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.975+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/2364808826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d18620 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff128006010 tx=0x7ff128004040 comp rx=0 tx=0).stop 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.975+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2364808826 shutdown_connections 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.975+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/2364808826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d23230 0x563918d25620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.975+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/2364808826 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d18b60 0x563918d22cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.975+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/2364808826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d18620 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.975+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2364808826 >> 172.21.15.31:0/2364808826 conn(0x563918d14560 msgr2=0x563918d14940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.975+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2364808826 shutdown_connections 2024-04-25T02:03:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.975+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2364808826 wait complete. 2024-04-25T02:03:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.976+0000 7ff13c7e0740 1 -- start start 2024-04-25T02:03:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.976+0000 7ff13c7e0740 1 Processor -- start 2024-04-25T02:03:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.976+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d201c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.976+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d20700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.976+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d20c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.976+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563918d25fc0 con 0x563918d18240 2024-04-25T02:03:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.976+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563918c38b50 con 0x563918d18b60 2024-04-25T02:03:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.976+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563918d26140 con 0x563918d23230 2024-04-25T02:03:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.976+0000 7ff13c7e0740 10 client.0 _ll_register_callbacks cb 0x563918c73ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff132ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d201c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff1327fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d20700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff132ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d201c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47116/0 (socket says 172.21.15.31:47116) 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff132ffd640 1 -- 172.21.15.31:0/3457772975 learned_addr learned my addr 172.21.15.31:0/3457772975 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff1327fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d20700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:53136/0 (socket says 172.21.15.31:53136) 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff1337fe640 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d20c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff1327fc640 1 -- 172.21.15.31:0/3457772975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 msgr2=0x563918d20c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff1327fc640 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d20c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff1327fc640 1 -- 172.21.15.31:0/3457772975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 msgr2=0x563918d201c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff1327fc640 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d201c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff1327fc640 1 -- 172.21.15.31:0/3457772975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563918a5d620 con 0x563918d18b60 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff1327fc640 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d20700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff1240cce70 tx=0x7ff12400e940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba11c0aedebe0ba3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff112ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff112ffd640 1 -- 172.21.15.31:0/3457772975 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124010630 con 0x563918d18b60 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff13c7e0740 1 -- 172.21.15.31:0/3457772975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1280054f0 con 0x563918d18b60 2024-04-25T02:03:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.977+0000 7ff112ffd640 1 -- 172.21.15.31:0/3457772975 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7ff12409b140 con 0x563918d18b60 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff112ffd640 1 -- 172.21.15.31:0/3457772975 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff1240a48b0 con 0x563918d18b60 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff13c7e0740 10 client.26495 fetch_fsmap learned FSMap version 38 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff13c7e0740 1 -- 172.21.15.31:0/3457772975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x563918d32210 con 0x563918d18b60 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff112ffd640 1 -- 172.21.15.31:0/3457772975 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1280054f0 con 0x563918d18b60 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff112ffd640 1 -- 172.21.15.31:0/3457772975 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7ff124018ac0 con 0x563918d18b60 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff13c7e0740 10 client.26495 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff13c7e0740 10 client.26495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff13c7e0740 20 client.26495 populate_metadata read hostname 'smithi031' 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563918d336d0 0x563918d35b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff13c7e0740 4 client.26495 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff13c7e0740 1 -- 172.21.15.31:0/3457772975 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563918c8a1f0 con 0x563918d336d0 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.978+0000 7ff132ffd640 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563918d336d0 0x563918d35b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.979+0000 7ff112ffd640 10 client.26495 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.979+0000 7ff132ffd640 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563918d336d0 0x563918d35b60 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2b2f25b99c29b9f server_cookie=13adc3261b1730d2 in_seq=0 out_seq=0 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.979+0000 7ff112ffd640 1 -- 172.21.15.31:0/3457772975 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563918c8a1f0 con 0x563918d336d0 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.979+0000 7ff112ffd640 10 client.26495 handle_command_reply: tid=0 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 1 client.26495 shutdown 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 1 -- 172.21.15.31:0/3457772975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 msgr2=0x563918d20700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d20700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff1240cce70 tx=0x7ff12400e940 comp rx=0 tx=0).stop 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 1 -- 172.21.15.31:0/3457772975 shutdown_connections 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563918d336d0 0x563918d35b60 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d20c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d20700 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff112ffd640 0 client.26495 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/3457772975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d201c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 1 -- 172.21.15.31:0/3457772975 >> 172.21.15.31:0/3457772975 conn(0x563918d14560 msgr2=0x563918d1a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 1 -- 172.21.15.31:0/3457772975 shutdown_connections 2024-04-25T02:03:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 1 -- 172.21.15.31:0/3457772975 wait complete. 2024-04-25T02:03:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:51.980+0000 7ff13c7e0740 20 client.26495 trim_cache size 0 max 16384 2024-04-25T02:03:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff13c7e0740 1 Processor -- start 2024-04-25T02:03:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff13c7e0740 1 -- start start 2024-04-25T02:03:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d228f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d2ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d13440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341af510 con 0x563918d18240 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341b1170 con 0x563918d18b60 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341afbe0 con 0x563918d23230 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff132ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d228f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff1327fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d2ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff1327fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d2ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:53140/0 (socket says 172.21.15.31:53140) 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff1327fc640 1 -- 172.21.15.31:0/1781836513 learned_addr learned my addr 172.21.15.31:0/1781836513 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff132ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d228f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47124/0 (socket says 172.21.15.31:47124) 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff1337fe640 1 --2- 172.21.15.31:0/1781836513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d13440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff1327fc640 1 -- 172.21.15.31:0/1781836513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 msgr2=0x563918d13440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff1327fc640 1 --2- 172.21.15.31:0/1781836513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d13440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff1327fc640 1 -- 172.21.15.31:0/1781836513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 msgr2=0x563918d228f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff1327fc640 1 --2- 172.21.15.31:0/1781836513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d228f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.001+0000 7ff1327fc640 1 -- 172.21.15.31:0/1781836513 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563918d18990 con 0x563918d18b60 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.002+0000 7ff1327fc640 1 --2- 172.21.15.31:0/1781836513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d2ee80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff1240cd470 tx=0x7ff124009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dd35c59075c0609 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.002+0000 7ff1197fa640 1 -- 172.21.15.31:0/1781836513 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124010de0 con 0x563918d18b60 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.002+0000 7ff1197fa640 1 -- 172.21.15.31:0/1781836513 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff12409b140 con 0x563918d18b60 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.002+0000 7ff1197fa640 1 -- 172.21.15.31:0/1781836513 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1240a4df0 con 0x563918d18b60 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.002+0000 7ff13c7e0740 1 -- 172.21.15.31:0/1781836513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 msgr2=0x563918d2ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.002+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/1781836513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d2ee80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff1240cd470 tx=0x7ff124009700 comp rx=0 tx=0).stop 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 -- 172.21.15.31:0/1781836513 shutdown_connections 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/1781836513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d13440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/1781836513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18b60 0x563918d2ee80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/1781836513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18240 0x563918d228f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 -- 172.21.15.31:0/1781836513 >> 172.21.15.31:0/1781836513 conn(0x563918d14560 msgr2=0x563918d12d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 -- 172.21.15.31:0/1781836513 shutdown_connections 2024-04-25T02:03:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 -- 172.21.15.31:0/1781836513 wait complete. 2024-04-25T02:03:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 -- start start 2024-04-25T02:03:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 Processor -- start 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18240 0x563918d2b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18b60 0x563918d2b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d28360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341b0aa0 con 0x563918d18b60 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341b0310 con 0x563918d18240 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341ad510 con 0x563918d23230 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.003+0000 7ff13c7e0740 10 client.0 _ll_register_callbacks cb 0x7ff12409b830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff1327fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18b60 0x563918d2b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff132ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18240 0x563918d2b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff132ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18240 0x563918d2b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:53148/0 (socket says 172.21.15.31:53148) 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff1327fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18b60 0x563918d2b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47134/0 (socket says 172.21.15.31:47134) 2024-04-25T02:03:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff1337fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d28360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff1327fc640 1 -- 172.21.15.31:0/2657346837 learned_addr learned my addr 172.21.15.31:0/2657346837 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff1327fc640 1 -- 172.21.15.31:0/2657346837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 msgr2=0x563918d28360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff1327fc640 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d28360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff1327fc640 1 -- 172.21.15.31:0/2657346837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18240 msgr2=0x563918d2b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff1327fc640 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18240 0x563918d2b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff1327fc640 1 -- 172.21.15.31:0/2657346837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563918d289a0 con 0x563918d18b60 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff132ffd640 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18240 0x563918d2b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff1327fc640 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18b60 0x563918d2b8d0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ff124098150 tx=0x7ff12409be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63982479a4d0746 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff112ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff112ffd640 1 -- 172.21.15.31:0/2657346837 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12409bac0 con 0x563918d18b60 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2657346837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff12409b140 con 0x563918d18b60 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff112ffd640 1 -- 172.21.15.31:0/2657346837 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7ff124010430 con 0x563918d18b60 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff112ffd640 1 -- 172.21.15.31:0/2657346837 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff124010de0 con 0x563918d18b60 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff13c7e0740 10 client.16911 fetch_fsmap learned FSMap version 38 2024-04-25T02:03:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.004+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2657346837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x563918d289a0 con 0x563918d18b60 2024-04-25T02:03:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.005+0000 7ff112ffd640 1 -- 172.21.15.31:0/2657346837 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff12409b140 con 0x563918d18b60 2024-04-25T02:03:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.005+0000 7ff112ffd640 1 -- 172.21.15.31:0/2657346837 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7ff12408d030 con 0x563918d18b60 2024-04-25T02:03:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.005+0000 7ff13c7e0740 10 client.16911 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:03:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.005+0000 7ff13c7e0740 10 client.16911 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.005+0000 7ff13c7e0740 20 client.16911 populate_metadata read hostname 'smithi031' 2024-04-25T02:03:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.005+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563918d9a990 0x563918d9cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.005+0000 7ff13c7e0740 4 client.16911 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:03:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.005+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2657346837 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563918c84540 con 0x563918d9a990 2024-04-25T02:03:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.005+0000 7ff132ffd640 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563918d9a990 0x563918d9cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.005+0000 7ff132ffd640 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563918d9a990 0x563918d9cdb0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d670075e502ae944 server_cookie=eccc58f8049c64b8 in_seq=0 out_seq=0 2024-04-25T02:03:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.005+0000 7ff112ffd640 10 client.16911 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff112ffd640 1 -- 172.21.15.31:0/2657346837 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563918c84540 con 0x563918d9a990 2024-04-25T02:03:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff112ffd640 10 client.16911 handle_command_reply: tid=0 2024-04-25T02:03:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 1 client.16911 shutdown 2024-04-25T02:03:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2657346837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18b60 msgr2=0x563918d2b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18b60 0x563918d2b8d0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ff124098150 tx=0x7ff12409be70 comp rx=0 tx=0).stop 2024-04-25T02:03:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2657346837 shutdown_connections 2024-04-25T02:03:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563918d9a990 0x563918d9cdb0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563918d23230 0x563918d28360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563918d18b60 0x563918d2b8d0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 1 --2- 172.21.15.31:0/2657346837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563918d18240 0x563918d2b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2657346837 >> 172.21.15.31:0/2657346837 conn(0x563918d14560 msgr2=0x563918d135c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff112ffd640 0 client.16911 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:03:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2657346837 shutdown_connections 2024-04-25T02:03:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 1 -- 172.21.15.31:0/2657346837 wait complete. 2024-04-25T02:03:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.034+0000 7ff13c7e0740 20 client.16911 trim_cache size 0 max 16384 2024-04-25T02:03:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:03:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:03:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:03:52.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:03:52.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 -- 172.21.15.31:0/2634183535 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff11c0a2da0 msgr2=0x7ff11c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:52.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff11c0a2da0 0x7ff11c0a5260 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009fd0 tx=0x7ff12c011f60 comp rx=0 tx=0).stop 2024-04-25T02:03:52.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 -- 172.21.15.31:0/2634183535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff134102120 msgr2=0x7ff134073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:52.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff134102120 0x7ff134073f80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7ff12400e9f0 tx=0x7ff12400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 -- 172.21.15.31:0/2634183535 shutdown_connections 2024-04-25T02:03:52.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff11c0a2da0 0x7ff11c0a5260 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff134107770 0x7ff1340744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff134102120 0x7ff134073f80 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 --2- 172.21.15.31:0/2634183535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff134101760 0x7ff134073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:52.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 -- 172.21.15.31:0/2634183535 >> 172.21.15.31:0/2634183535 conn(0x7ff1340fd1f0 msgr2=0x7ff1340fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:52.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 -- 172.21.15.31:0/2634183535 shutdown_connections 2024-04-25T02:03:52.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:03:52.035+0000 7ff1399fb640 1 -- 172.21.15.31:0/2634183535 wait complete. 2024-04-25T02:03:52.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:51 smithi134 ceph-mon[33098]: pgmap v291: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:03:52.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:51 smithi031 ceph-mon[29204]: pgmap v291: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:03:52.318 INFO:tasks.workunit.client.0.smithi031.stdout: 1 213727 2.31 MB/sec execute 359 sec latency 28.196 ms 2024-04-25T02:03:52.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:51 smithi040 ceph-mon[32418]: pgmap v291: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:03:53.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 214270 2.31 MB/sec execute 360 sec latency 16.893 ms 2024-04-25T02:03:54.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:53 smithi134 ceph-mon[33098]: pgmap v292: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 577 KiB/s wr, 56 op/s 2024-04-25T02:03:54.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:53 smithi031 ceph-mon[29204]: pgmap v292: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 577 KiB/s wr, 56 op/s 2024-04-25T02:03:54.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 214769 2.31 MB/sec execute 361 sec latency 23.311 ms 2024-04-25T02:03:54.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:53 smithi040 ceph-mon[32418]: pgmap v292: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 577 KiB/s wr, 56 op/s 2024-04-25T02:03:55.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 215314 2.30 MB/sec execute 362 sec latency 21.562 ms 2024-04-25T02:03:56.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:55 smithi134 ceph-mon[33098]: pgmap v293: 97 pgs: 97 active+clean; 116 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-25T02:03:56.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:55 smithi031 ceph-mon[29204]: pgmap v293: 97 pgs: 97 active+clean; 116 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-25T02:03:56.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216045 2.31 MB/sec execute 363 sec latency 18.377 ms 2024-04-25T02:03:56.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:55 smithi040 ceph-mon[32418]: pgmap v293: 97 pgs: 97 active+clean; 116 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-04-25T02:03:57.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216427 2.32 MB/sec execute 364 sec latency 18.729 ms 2024-04-25T02:03:58.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:57 smithi134 ceph-mon[33098]: pgmap v294: 97 pgs: 97 active+clean; 116 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 544 KiB/s wr, 74 op/s 2024-04-25T02:03:58.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:57 smithi031 ceph-mon[29204]: pgmap v294: 97 pgs: 97 active+clean; 116 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 544 KiB/s wr, 74 op/s 2024-04-25T02:03:58.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216491 2.31 MB/sec execute 365 sec latency 911.464 ms 2024-04-25T02:03:58.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:57 smithi040 ceph-mon[32418]: pgmap v294: 97 pgs: 97 active+clean; 116 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 544 KiB/s wr, 74 op/s 2024-04-25T02:03:59.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216491 2.31 MB/sec execute 366 sec latency 1911.518 ms 2024-04-25T02:04:00.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:03:59 smithi031 ceph-mon[29204]: pgmap v295: 97 pgs: 97 active+clean; 116 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 544 KiB/s wr, 74 op/s 2024-04-25T02:04:00.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216558 2.30 MB/sec execute 367 sec latency 2637.723 ms 2024-04-25T02:04:00.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:03:59 smithi040 ceph-mon[32418]: pgmap v295: 97 pgs: 97 active+clean; 116 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 544 KiB/s wr, 74 op/s 2024-04-25T02:04:00.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:03:59 smithi134 ceph-mon[33098]: pgmap v295: 97 pgs: 97 active+clean; 116 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 544 KiB/s wr, 74 op/s 2024-04-25T02:04:01.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216558 2.30 MB/sec execute 368 sec latency 1041.353 ms 2024-04-25T02:04:02.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:01 smithi031 ceph-mon[29204]: pgmap v296: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T02:04:02.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216558 2.29 MB/sec execute 369 sec latency 2041.447 ms 2024-04-25T02:04:02.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:01 smithi040 ceph-mon[32418]: pgmap v296: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T02:04:02.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:01 smithi134 ceph-mon[33098]: pgmap v296: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T02:04:03.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216558 2.28 MB/sec execute 370 sec latency 3041.495 ms 2024-04-25T02:04:04.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:03 smithi031 ceph-mon[29204]: pgmap v297: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 998 KiB/s wr, 97 op/s 2024-04-25T02:04:04.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216558 2.28 MB/sec execute 371 sec latency 4041.533 ms 2024-04-25T02:04:04.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:03 smithi040 ceph-mon[32418]: pgmap v297: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 998 KiB/s wr, 97 op/s 2024-04-25T02:04:04.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:03 smithi134 ceph-mon[33098]: pgmap v297: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 998 KiB/s wr, 97 op/s 2024-04-25T02:04:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.153+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/3789412422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b68101020 msgr2=0x7f4b6810d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.153+0000 7f4b6f6bd640 1 --2- 172.21.15.31:0/3789412422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b68101020 0x7f4b6810d530 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009cd0 tx=0x7f4b6002a8e0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.153+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/3789412422 shutdown_connections 2024-04-25T02:04:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.153+0000 7f4b6f6bd640 1 --2- 172.21.15.31:0/3789412422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b68101020 0x7f4b6810d530 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.153+0000 7f4b6f6bd640 1 --2- 172.21.15.31:0/3789412422 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b68100660 0x7f4b68100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.153+0000 7f4b6f6bd640 1 --2- 172.21.15.31:0/3789412422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b68106670 0x7f4b68106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.153+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/3789412422 >> 172.21.15.31:0/3789412422 conn(0x7f4b680769c0 msgr2=0x7f4b68076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.153+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/3789412422 shutdown_connections 2024-04-25T02:04:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.153+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/3789412422 wait complete. 2024-04-25T02:04:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6f6bd640 1 Processor -- start 2024-04-25T02:04:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6f6bd640 1 -- start start 2024-04-25T02:04:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6f6bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b68100660 0x7f4b68197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6f6bd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b68101020 0x7f4b68198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6f6bd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b68106670 0x7f4b6819c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6f6bd640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4b681108f0 con 0x7f4b68101020 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6f6bd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b68110770 con 0x7f4b68100660 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6f6bd640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4b68110a70 con 0x7f4b68106670 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6cc58640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b68101020 0x7f4b68198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6cc58640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b68101020 0x7f4b68198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57110/0 (socket says 172.21.15.31:57110) 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6cc58640 1 -- 172.21.15.31:0/2169664243 learned_addr learned my addr 172.21.15.31:0/2169664243 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.154+0000 7f4b6d459640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b68100660 0x7f4b68197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b68106670 0x7f4b6819c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b6cc58640 1 -- 172.21.15.31:0/2169664243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b68106670 msgr2=0x7f4b6819c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b6cc58640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b68106670 0x7f4b6819c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b6cc58640 1 -- 172.21.15.31:0/2169664243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b68100660 msgr2=0x7f4b68197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:05.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b6cc58640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b68100660 0x7f4b68197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b6cc58640 1 -- 172.21.15.31:0/2169664243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6819cdf0 con 0x7f4b68101020 2024-04-25T02:04:05.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b6cc58640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b68101020 0x7f4b68198390 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400d950 tx=0x7f4b6400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:05.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b5a7fc640 1 -- 172.21.15.31:0/2169664243 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64017070 con 0x7f4b68101020 2024-04-25T02:04:05.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/2169664243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6819d0e0 con 0x7f4b68101020 2024-04-25T02:04:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b5a7fc640 1 -- 172.21.15.31:0/2169664243 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b64009c30 con 0x7f4b68101020 2024-04-25T02:04:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b5a7fc640 1 -- 172.21.15.31:0/2169664243 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6400f620 con 0x7f4b68101020 2024-04-25T02:04:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.155+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/2169664243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b681a4970 con 0x7f4b68101020 2024-04-25T02:04:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.158+0000 7f4b5a7fc640 1 -- 172.21.15.31:0/2169664243 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4b640109a0 con 0x7f4b68101020 2024-04-25T02:04:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.158+0000 7f4b723ce740 1 Processor -- start 2024-04-25T02:04:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.158+0000 7f4b723ce740 1 -- start start 2024-04-25T02:04:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.158+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76d6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.158+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76e0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.158+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76e1230 0x556dd76e3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.158+0000 7f4b723ce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556dd763acb0 con 0x556dd76d6b60 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.158+0000 7f4b723ce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556dd75e35b0 con 0x556dd76d6240 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.158+0000 7f4b723ce740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556dd7636c70 con 0x556dd76e1230 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.158+0000 7f4b5a7fc640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4b540a2da0 0x7f4b540a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.158+0000 7f4b6d459640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76e0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6d459640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76e0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57126/0 (socket says 172.21.15.31:57126) 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6d459640 1 -- 172.21.15.31:0/377411664 learned_addr learned my addr 172.21.15.31:0/377411664 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/377411664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76d6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4b540a2da0 0x7f4b540a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6cc58640 1 --2- 172.21.15.31:0/377411664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76e1230 0x556dd76e3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b5a7fc640 1 -- 172.21.15.31:0/2169664243 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f4b640cefd0 con 0x7f4b68101020 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6d459640 1 -- 172.21.15.31:0/377411664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76e1230 msgr2=0x556dd76e3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6d459640 1 --2- 172.21.15.31:0/377411664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76e1230 0x556dd76e3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6d459640 1 -- 172.21.15.31:0/377411664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 msgr2=0x556dd76d6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6d459640 1 --2- 172.21.15.31:0/377411664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76d6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6d459640 1 -- 172.21.15.31:0/377411664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dd734f210 con 0x556dd76d6b60 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/377411664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76d6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6d459640 1 --2- 172.21.15.31:0/377411664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76e0cf0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00bc90 tx=0x7f4b5c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6bef39509efb99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4b540a2da0 0x7f4b540a5260 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009430 tx=0x7f4b60005da0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b537fe640 1 -- 172.21.15.31:0/377411664 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c01c480 con 0x556dd76d6b60 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b537fe640 1 -- 172.21.15.31:0/377411664 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b5c01cad0 con 0x556dd76d6b60 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.159+0000 7f4b537fe640 1 -- 172.21.15.31:0/377411664 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c011ae0 con 0x556dd76d6b60 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.160+0000 7f4b723ce740 1 -- 172.21.15.31:0/377411664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 msgr2=0x556dd76e0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.160+0000 7f4b723ce740 1 --2- 172.21.15.31:0/377411664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76e0cf0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00bc90 tx=0x7f4b5c0107a0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.160+0000 7f4b723ce740 1 -- 172.21.15.31:0/377411664 shutdown_connections 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.160+0000 7f4b723ce740 1 --2- 172.21.15.31:0/377411664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76e1230 0x556dd76e3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.160+0000 7f4b723ce740 1 --2- 172.21.15.31:0/377411664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76e0cf0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.160+0000 7f4b723ce740 1 --2- 172.21.15.31:0/377411664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76d6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.160+0000 7f4b723ce740 1 -- 172.21.15.31:0/377411664 >> 172.21.15.31:0/377411664 conn(0x556dd76d2560 msgr2=0x556dd76d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.160+0000 7f4b723ce740 1 -- 172.21.15.31:0/377411664 shutdown_connections 2024-04-25T02:04:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.160+0000 7f4b723ce740 1 -- 172.21.15.31:0/377411664 wait complete. 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b723ce740 1 -- start start 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b723ce740 1 Processor -- start 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6240 0x556dd76de1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76de6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76e1230 0x556dd76dec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b723ce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556dd76e4020 con 0x556dd76d6b60 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b723ce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556dd75f6b50 con 0x556dd76e1230 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b723ce740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556dd76e41a0 con 0x556dd76d6240 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b723ce740 10 client.0 _ll_register_callbacks cb 0x556dd7631ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b6cc58640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76de6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b6cc58640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76de6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57142/0 (socket says 172.21.15.31:57142) 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b6cc58640 1 -- 172.21.15.31:0/1721218249 learned_addr learned my addr 172.21.15.31:0/1721218249 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76e1230 0x556dd76dec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.161+0000 7f4b6d459640 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6240 0x556dd76de1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b6cc58640 1 -- 172.21.15.31:0/1721218249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6240 msgr2=0x556dd76de1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b6cc58640 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6240 0x556dd76de1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b6cc58640 1 -- 172.21.15.31:0/1721218249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76e1230 msgr2=0x556dd76dec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b6cc58640 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76e1230 0x556dd76dec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b6cc58640 1 -- 172.21.15.31:0/1721218249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dd741b620 con 0x556dd76d6b60 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b6cc58640 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76de6f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4b6409b550 tx=0x7f4b6401e770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e640aca4270d91b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1721218249 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64020040 con 0x556dd76d6b60 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1721218249 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f4b6406a1b0 con 0x556dd76d6b60 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b723ce740 1 -- 172.21.15.31:0/1721218249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b5c01cad0 con 0x556dd76d6b60 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1721218249 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f4b64014030 con 0x556dd76d6b60 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b723ce740 10 client.16923 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b723ce740 1 -- 172.21.15.31:0/1721218249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556dd76f0200 con 0x556dd76d6b60 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.162+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1721218249 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b5c01cad0 con 0x556dd76d6b60 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.163+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1721218249 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f4b640a3cf0 con 0x556dd76d6b60 2024-04-25T02:04:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.163+0000 7f4b723ce740 10 client.16923 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.163+0000 7f4b723ce740 10 client.16923 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.163+0000 7f4b723ce740 20 client.16923 populate_metadata read hostname 'smithi031' 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.163+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556dd76f1c80 0x556dd76f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.163+0000 7f4b723ce740 4 client.16923 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.163+0000 7f4b723ce740 1 -- 172.21.15.31:0/1721218249 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556dd76481f0 con 0x556dd76f1c80 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.163+0000 7f4b6d459640 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556dd76f1c80 0x556dd76f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.163+0000 7f4b6d459640 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556dd76f1c80 0x556dd76f40a0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e85549f24d81f667 server_cookie=65254f89547dc898 in_seq=0 out_seq=0 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.163+0000 7f4b50ff9640 10 client.16923 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1721218249 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556dd76481f0 con 0x556dd76f1c80 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b50ff9640 10 client.16923 handle_command_reply: tid=0 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 1 client.16923 shutdown 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 1 -- 172.21.15.31:0/1721218249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 msgr2=0x556dd76de6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76de6f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4b6409b550 tx=0x7f4b6401e770 comp rx=0 tx=0).stop 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 1 -- 172.21.15.31:0/1721218249 shutdown_connections 2024-04-25T02:04:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556dd76f1c80 0x556dd76f40a0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76e1230 0x556dd76dec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76d6b60 0x556dd76de6f0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1721218249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6240 0x556dd76de1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b50ff9640 0 client.16923 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 1 -- 172.21.15.31:0/1721218249 >> 172.21.15.31:0/1721218249 conn(0x556dd76d2560 msgr2=0x556dd76d8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 1 -- 172.21.15.31:0/1721218249 shutdown_connections 2024-04-25T02:04:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 1 -- 172.21.15.31:0/1721218249 wait complete. 2024-04-25T02:04:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.164+0000 7f4b723ce740 20 client.16923 trim_cache size 0 max 16384 2024-04-25T02:04:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b723ce740 1 Processor -- start 2024-04-25T02:04:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b723ce740 1 -- start start 2024-04-25T02:04:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76ed920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6b60 0x556dd76ecee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd76d80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b723ce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4b681ad2b0 con 0x556dd76e1230 2024-04-25T02:04:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b723ce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b681aef10 con 0x556dd76d6240 2024-04-25T02:04:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b723ce740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4b681ad980 con 0x556dd76d6b60 2024-04-25T02:04:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b6dc5a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd76d80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b6dc5a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd76d80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57158/0 (socket says 172.21.15.31:57158) 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b6dc5a640 1 -- 172.21.15.31:0/968541213 learned_addr learned my addr 172.21.15.31:0/968541213 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b6d459640 1 --2- 172.21.15.31:0/968541213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76ed920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b6cc58640 1 --2- 172.21.15.31:0/968541213 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6b60 0x556dd76ecee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b6dc5a640 1 -- 172.21.15.31:0/968541213 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6b60 msgr2=0x556dd76ecee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/968541213 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6b60 0x556dd76ecee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b6dc5a640 1 -- 172.21.15.31:0/968541213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 msgr2=0x556dd76ed920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/968541213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76ed920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.180+0000 7f4b6dc5a640 1 -- 172.21.15.31:0/968541213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dd76cc850 con 0x556dd76e1230 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/968541213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd76d80e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009cd0 tx=0x7f4b6003d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb8eae841abb188 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b52ffd640 1 -- 172.21.15.31:0/968541213 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6005a070 con 0x556dd76e1230 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b52ffd640 1 -- 172.21.15.31:0/968541213 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b60036070 con 0x556dd76e1230 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b52ffd640 1 -- 172.21.15.31:0/968541213 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60044040 con 0x556dd76e1230 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b723ce740 1 -- 172.21.15.31:0/968541213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 msgr2=0x556dd76d80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b723ce740 1 --2- 172.21.15.31:0/968541213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd76d80e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009cd0 tx=0x7f4b6003d070 comp rx=0 tx=0).stop 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b723ce740 1 -- 172.21.15.31:0/968541213 shutdown_connections 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b723ce740 1 --2- 172.21.15.31:0/968541213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd76d80e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b723ce740 1 --2- 172.21.15.31:0/968541213 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6b60 0x556dd76ecee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b723ce740 1 --2- 172.21.15.31:0/968541213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76ed920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b723ce740 1 -- 172.21.15.31:0/968541213 >> 172.21.15.31:0/968541213 conn(0x556dd76d2560 msgr2=0x556dd76d79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b723ce740 1 -- 172.21.15.31:0/968541213 shutdown_connections 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.181+0000 7f4b723ce740 1 -- 172.21.15.31:0/968541213 wait complete. 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b723ce740 1 -- start start 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b723ce740 1 Processor -- start 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76e5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6b60 0x556dd76e5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b723ce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd7757220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b723ce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4b681ae840 con 0x556dd76e1230 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b723ce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b681ae0b0 con 0x556dd76d6240 2024-04-25T02:04:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b723ce740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4b681ab2b0 con 0x556dd76d6b60 2024-04-25T02:04:05.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b723ce740 10 client.0 _ll_register_callbacks cb 0x7f4b6400df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:05.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b6dc5a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd7757220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b6dc5a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd7757220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57164/0 (socket says 172.21.15.31:57164) 2024-04-25T02:04:05.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b6d459640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76e5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b6dc5a640 1 -- 172.21.15.31:0/1711163635 learned_addr learned my addr 172.21.15.31:0/1711163635 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:05.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.182+0000 7f4b6cc58640 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6b60 0x556dd76e5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b6dc5a640 1 -- 172.21.15.31:0/1711163635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6b60 msgr2=0x556dd76e5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6b60 0x556dd76e5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b6dc5a640 1 -- 172.21.15.31:0/1711163635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 msgr2=0x556dd76e5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76e5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b6dc5a640 1 -- 172.21.15.31:0/1711163635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dd7757810 con 0x556dd76e1230 2024-04-25T02:04:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b6d459640 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76e5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:04:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b6dc5a640 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd7757220 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4b6002ae00 tx=0x7f4b60030980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6290a8de043c1785 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:04:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1711163635 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6005a070 con 0x556dd76e1230 2024-04-25T02:04:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b723ce740 1 -- 172.21.15.31:0/1711163635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b60036070 con 0x556dd76e1230 2024-04-25T02:04:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1711163635 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8484+0+0 (secure 0 0 0) 0x7f4b60030af0 con 0x556dd76e1230 2024-04-25T02:04:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1711163635 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f4b60044040 con 0x556dd76e1230 2024-04-25T02:04:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b723ce740 10 client.16929 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b723ce740 1 -- 172.21.15.31:0/1711163635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556dd7758b00 con 0x556dd76e1230 2024-04-25T02:04:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.183+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1711163635 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b60036070 con 0x556dd76e1230 2024-04-25T02:04:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.184+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1711163635 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f4b60054070 con 0x556dd76e1230 2024-04-25T02:04:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.184+0000 7f4b723ce740 10 client.16929 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:05.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.184+0000 7f4b723ce740 10 client.16929 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:05.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.184+0000 7f4b723ce740 20 client.16929 populate_metadata read hostname 'smithi031' 2024-04-25T02:04:05.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.184+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556dd77591a0 0x556dd775b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:05.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.184+0000 7f4b723ce740 4 client.16929 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:04:05.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.184+0000 7f4b723ce740 1 -- 172.21.15.31:0/1711163635 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556dd7642540 con 0x556dd77591a0 2024-04-25T02:04:05.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.184+0000 7f4b6d459640 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556dd77591a0 0x556dd775b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:05.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.184+0000 7f4b6d459640 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556dd77591a0 0x556dd775b5c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd2beed449ece839 server_cookie=c737c6cfa0df0351 in_seq=0 out_seq=0 2024-04-25T02:04:05.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:05.184+0000 7f4b50ff9640 10 client.16929 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:05.320 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.27 MB/sec execute 372 sec latency 4763.947 ms 2024-04-25T02:04:06.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:05 smithi031 ceph-mon[29204]: pgmap v298: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T02:04:06.319 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.27 MB/sec execute 373 sec latency 1211.725 ms 2024-04-25T02:04:06.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:05 smithi040 ceph-mon[32418]: pgmap v298: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T02:04:06.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:05 smithi134 ceph-mon[33098]: pgmap v298: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T02:04:07.320 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.26 MB/sec execute 374 sec latency 2211.777 ms 2024-04-25T02:04:08.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:07 smithi031 ceph-mon[29204]: pgmap v299: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 531 KiB/s wr, 47 op/s 2024-04-25T02:04:08.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:07 smithi031 ceph-mon[29204]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T02:04:08.320 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.25 MB/sec execute 375 sec latency 3211.859 ms 2024-04-25T02:04:08.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:07 smithi040 ceph-mon[32418]: pgmap v299: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 531 KiB/s wr, 47 op/s 2024-04-25T02:04:08.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:07 smithi040 ceph-mon[32418]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T02:04:08.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:07 smithi134 ceph-mon[33098]: pgmap v299: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 531 KiB/s wr, 47 op/s 2024-04-25T02:04:08.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:07 smithi134 ceph-mon[33098]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T02:04:09.320 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.25 MB/sec execute 376 sec latency 4211.903 ms 2024-04-25T02:04:10.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:09 smithi031 ceph-mon[29204]: pgmap v301: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 637 KiB/s wr, 57 op/s 2024-04-25T02:04:10.246 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:09 smithi134 ceph-mon[33098]: pgmap v301: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 637 KiB/s wr, 57 op/s 2024-04-25T02:04:10.320 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.24 MB/sec execute 377 sec latency 5211.996 ms 2024-04-25T02:04:10.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:09 smithi040 ceph-mon[32418]: pgmap v301: 97 pgs: 97 active+clean; 120 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 637 KiB/s wr, 57 op/s 2024-04-25T02:04:11.320 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.24 MB/sec execute 378 sec latency 6212.058 ms 2024-04-25T02:04:12.216 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:11 smithi040 ceph-mon[32418]: pgmap v302: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 146 KiB/s wr, 48 op/s 2024-04-25T02:04:12.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:11 smithi031 ceph-mon[29204]: pgmap v302: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 146 KiB/s wr, 48 op/s 2024-04-25T02:04:12.320 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.23 MB/sec execute 379 sec latency 7212.141 ms 2024-04-25T02:04:12.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:11 smithi134 ceph-mon[33098]: pgmap v302: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 146 KiB/s wr, 48 op/s 2024-04-25T02:04:13.320 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.22 MB/sec execute 380 sec latency 8212.193 ms 2024-04-25T02:04:14.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:13 smithi031 ceph-mon[29204]: pgmap v303: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 146 KiB/s wr, 48 op/s 2024-04-25T02:04:14.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:13 smithi031 ceph-mon[29204]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T02:04:14.320 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.22 MB/sec execute 381 sec latency 9212.299 ms 2024-04-25T02:04:14.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:13 smithi040 ceph-mon[32418]: pgmap v303: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 146 KiB/s wr, 48 op/s 2024-04-25T02:04:14.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:13 smithi040 ceph-mon[32418]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T02:04:14.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:13 smithi134 ceph-mon[33098]: pgmap v303: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 146 KiB/s wr, 48 op/s 2024-04-25T02:04:14.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:13 smithi134 ceph-mon[33098]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T02:04:15.320 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.21 MB/sec execute 382 sec latency 10212.417 ms 2024-04-25T02:04:16.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:15 smithi031 ceph-mon[29204]: pgmap v305: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 139 KiB/s wr, 50 op/s 2024-04-25T02:04:16.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.21 MB/sec execute 383 sec latency 11212.519 ms 2024-04-25T02:04:16.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:15 smithi040 ceph-mon[32418]: pgmap v305: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 139 KiB/s wr, 50 op/s 2024-04-25T02:04:16.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:15 smithi134 ceph-mon[33098]: pgmap v305: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 139 KiB/s wr, 50 op/s 2024-04-25T02:04:17.320 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.20 MB/sec execute 384 sec latency 12212.611 ms 2024-04-25T02:04:18.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:17 smithi031 ceph-mon[29204]: pgmap v306: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 114 KiB/s wr, 41 op/s 2024-04-25T02:04:18.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.19 MB/sec execute 385 sec latency 13212.670 ms 2024-04-25T02:04:18.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:17 smithi040 ceph-mon[32418]: pgmap v306: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 114 KiB/s wr, 41 op/s 2024-04-25T02:04:18.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:17 smithi134 ceph-mon[33098]: pgmap v306: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 114 KiB/s wr, 41 op/s 2024-04-25T02:04:19.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216570 2.19 MB/sec execute 386 sec latency 14212.769 ms 2024-04-25T02:04:20.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b50ff9640 1 -- 172.21.15.31:0/1711163635 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556dd7642540 con 0x556dd77591a0 2024-04-25T02:04:20.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b50ff9640 10 client.16929 handle_command_reply: tid=0 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 1 client.16929 shutdown 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 1 -- 172.21.15.31:0/1711163635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 msgr2=0x556dd7757220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd7757220 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4b6002ae00 tx=0x7f4b60030980 comp rx=0 tx=0).stop 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 1 -- 172.21.15.31:0/1711163635 shutdown_connections 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556dd77591a0 0x556dd775b5c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556dd76e1230 0x556dd7757220 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556dd76d6b60 0x556dd76e5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b50ff9640 0 client.16929 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 1 --2- 172.21.15.31:0/1711163635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556dd76d6240 0x556dd76e5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 1 -- 172.21.15.31:0/1711163635 >> 172.21.15.31:0/1711163635 conn(0x556dd76d2560 msgr2=0x556dd76d8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 1 -- 172.21.15.31:0/1711163635 shutdown_connections 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 1 -- 172.21.15.31:0/1711163635 wait complete. 2024-04-25T02:04:20.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.044+0000 7f4b723ce740 20 client.16929 trim_cache size 0 max 16384 2024-04-25T02:04:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:04:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:04:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:04:20.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/2169664243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4b540a2da0 msgr2=0x7f4b540a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4b540a2da0 0x7f4b540a5260 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009430 tx=0x7f4b60005da0 comp rx=0 tx=0).stop 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/2169664243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b68101020 msgr2=0x7f4b68198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b68101020 0x7f4b68198390 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400d950 tx=0x7f4b6400de10 comp rx=0 tx=0).stop 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/2169664243 shutdown_connections 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4b540a2da0 0x7f4b540a5260 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b68106670 0x7f4b6819c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b68101020 0x7f4b68198390 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 --2- 172.21.15.31:0/2169664243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b68100660 0x7f4b68197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/2169664243 >> 172.21.15.31:0/2169664243 conn(0x7f4b680769c0 msgr2=0x7f4b6806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/2169664243 shutdown_connections 2024-04-25T02:04:20.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:20.045+0000 7f4b6f6bd640 1 -- 172.21.15.31:0/2169664243 wait complete. 2024-04-25T02:04:20.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:19 smithi031 ceph-mon[29204]: pgmap v307: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 111 KiB/s wr, 40 op/s 2024-04-25T02:04:20.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216659 2.19 MB/sec execute 387 sec latency 14955.483 ms 2024-04-25T02:04:20.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:19 smithi040 ceph-mon[32418]: pgmap v307: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 111 KiB/s wr, 40 op/s 2024-04-25T02:04:20.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:19 smithi134 ceph-mon[33098]: pgmap v307: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 111 KiB/s wr, 40 op/s 2024-04-25T02:04:21.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216897 2.18 MB/sec execute 388 sec latency 12.877 ms 2024-04-25T02:04:22.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:21 smithi031 ceph-mon[29204]: pgmap v308: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-25T02:04:22.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 217256 2.18 MB/sec execute 389 sec latency 19.582 ms 2024-04-25T02:04:22.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:21 smithi040 ceph-mon[32418]: pgmap v308: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-25T02:04:22.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:21 smithi134 ceph-mon[33098]: pgmap v308: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-25T02:04:23.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 217805 2.18 MB/sec execute 390 sec latency 18.828 ms 2024-04-25T02:04:24.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:23 smithi031 ceph-mon[29204]: pgmap v309: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-25T02:04:24.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 218232 2.18 MB/sec execute 391 sec latency 21.927 ms 2024-04-25T02:04:24.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:23 smithi040 ceph-mon[32418]: pgmap v309: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-25T02:04:24.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:23 smithi134 ceph-mon[33098]: pgmap v309: 97 pgs: 97 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-25T02:04:25.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 218791 2.18 MB/sec execute 392 sec latency 23.275 ms 2024-04-25T02:04:26.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:25 smithi031 ceph-mon[29204]: pgmap v310: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 557 KiB/s wr, 73 op/s 2024-04-25T02:04:26.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 219386 2.18 MB/sec execute 393 sec latency 19.679 ms 2024-04-25T02:04:26.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:25 smithi040 ceph-mon[32418]: pgmap v310: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 557 KiB/s wr, 73 op/s 2024-04-25T02:04:26.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:25 smithi134 ceph-mon[33098]: pgmap v310: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 557 KiB/s wr, 73 op/s 2024-04-25T02:04:27.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 219809 2.18 MB/sec execute 394 sec latency 14.415 ms 2024-04-25T02:04:28.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:27 smithi031 ceph-mon[29204]: pgmap v311: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 495 KiB/s wr, 60 op/s 2024-04-25T02:04:28.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220037 2.19 MB/sec execute 395 sec latency 689.128 ms 2024-04-25T02:04:28.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:27 smithi040 ceph-mon[32418]: pgmap v311: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 495 KiB/s wr, 60 op/s 2024-04-25T02:04:28.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:27 smithi134 ceph-mon[33098]: pgmap v311: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 495 KiB/s wr, 60 op/s 2024-04-25T02:04:29.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220037 2.18 MB/sec execute 396 sec latency 1689.176 ms 2024-04-25T02:04:30.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:29 smithi031 ceph-mon[29204]: pgmap v312: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 495 KiB/s wr, 60 op/s 2024-04-25T02:04:30.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220104 2.18 MB/sec execute 397 sec latency 2413.438 ms 2024-04-25T02:04:30.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:29 smithi040 ceph-mon[32418]: pgmap v312: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 495 KiB/s wr, 60 op/s 2024-04-25T02:04:30.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:29 smithi134 ceph-mon[33098]: pgmap v312: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 495 KiB/s wr, 60 op/s 2024-04-25T02:04:31.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220104 2.17 MB/sec execute 398 sec latency 1038.926 ms 2024-04-25T02:04:32.321 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220104 2.17 MB/sec execute 399 sec latency 2039.029 ms 2024-04-25T02:04:32.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:31 smithi040 ceph-mon[32418]: pgmap v313: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 993 KiB/s wr, 96 op/s 2024-04-25T02:04:32.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:31 smithi134 ceph-mon[33098]: pgmap v313: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 993 KiB/s wr, 96 op/s 2024-04-25T02:04:32.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:31 smithi031 ceph-mon[29204]: pgmap v313: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 993 KiB/s wr, 96 op/s 2024-04-25T02:04:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.148+0000 7efc4227d640 1 -- 172.21.15.31:0/365808909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc3c107750 msgr2=0x7efc3c107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.148+0000 7efc4227d640 1 --2- 172.21.15.31:0/365808909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc3c107750 0x7efc3c107b30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009930 tx=0x7efc2c02a420 comp rx=0 tx=0).stop 2024-04-25T02:04:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.148+0000 7efc4227d640 1 -- 172.21.15.31:0/365808909 shutdown_connections 2024-04-25T02:04:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.148+0000 7efc4227d640 1 --2- 172.21.15.31:0/365808909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc3c102100 0x7efc3c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.148+0000 7efc4227d640 1 --2- 172.21.15.31:0/365808909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc3c101740 0x7efc3c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.148+0000 7efc4227d640 1 --2- 172.21.15.31:0/365808909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc3c107750 0x7efc3c107b30 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.148+0000 7efc4227d640 1 -- 172.21.15.31:0/365808909 >> 172.21.15.31:0/365808909 conn(0x7efc3c0fd210 msgr2=0x7efc3c0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.148+0000 7efc4227d640 1 -- 172.21.15.31:0/365808909 shutdown_connections 2024-04-25T02:04:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.148+0000 7efc4227d640 1 -- 172.21.15.31:0/365808909 wait complete. 2024-04-25T02:04:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.148+0000 7efc4227d640 1 Processor -- start 2024-04-25T02:04:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.148+0000 7efc4227d640 1 -- start start 2024-04-25T02:04:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc4227d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc3c101740 0x7efc3c073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc4227d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc3c102100 0x7efc3c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc4227d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc3c107750 0x7efc3c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc4227d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c111a60 con 0x7efc3c102100 2024-04-25T02:04:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc4227d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c1118e0 con 0x7efc3c101740 2024-04-25T02:04:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc4227d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c111be0 con 0x7efc3c107750 2024-04-25T02:04:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc3b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc3c102100 0x7efc3c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc3b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc3c102100 0x7efc3c073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37788/0 (socket says 172.21.15.31:37788) 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc3b7fe640 1 -- 172.21.15.31:0/4275007235 learned_addr learned my addr 172.21.15.31:0/4275007235 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc4081a640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc3c107750 0x7efc3c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc3bfff640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc3c101740 0x7efc3c073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc3b7fe640 1 -- 172.21.15.31:0/4275007235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc3c107750 msgr2=0x7efc3c0744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc3b7fe640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc3c107750 0x7efc3c0744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc3b7fe640 1 -- 172.21.15.31:0/4275007235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc3c101740 msgr2=0x7efc3c073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc3b7fe640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc3c101740 0x7efc3c073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc3b7fe640 1 -- 172.21.15.31:0/4275007235 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3c1a63c0 con 0x7efc3c102100 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc3bfff640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc3c101740 0x7efc3c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc4081a640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc3c107750 0x7efc3c0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc3b7fe640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc3c102100 0x7efc3c073f80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7efc3000d950 tx=0x7efc3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.149+0000 7efc397fa640 1 -- 172.21.15.31:0/4275007235 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc30017030 con 0x7efc3c102100 2024-04-25T02:04:33.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.150+0000 7efc397fa640 1 -- 172.21.15.31:0/4275007235 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc30004020 con 0x7efc3c102100 2024-04-25T02:04:33.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.150+0000 7efc4227d640 1 -- 172.21.15.31:0/4275007235 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc3c1a66b0 con 0x7efc3c102100 2024-04-25T02:04:33.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.150+0000 7efc397fa640 1 -- 172.21.15.31:0/4275007235 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc300047a0 con 0x7efc3c102100 2024-04-25T02:04:33.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.150+0000 7efc4227d640 1 -- 172.21.15.31:0/4275007235 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc3c1a6c50 con 0x7efc3c102100 2024-04-25T02:04:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.152+0000 7efc397fa640 1 -- 172.21.15.31:0/4275007235 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efc300041d0 con 0x7efc3c102100 2024-04-25T02:04:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.152+0000 7efc44f8d740 1 Processor -- start 2024-04-25T02:04:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.152+0000 7efc44f8d740 1 -- start start 2024-04-25T02:04:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce6e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce78cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce79230 0x56387ce7b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc44f8d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56387cdd2cb0 con 0x56387ce6e240 2024-04-25T02:04:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc44f8d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56387cd7b5b0 con 0x56387ce6eb60 2024-04-25T02:04:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc44f8d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56387cdcec70 con 0x56387ce79230 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc4081a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce6e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc397fa640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efc240a2e20 0x7efc240a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc4081a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce6e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37804/0 (socket says 172.21.15.31:37804) 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc4081a640 1 -- 172.21.15.31:0/3042120980 learned_addr learned my addr 172.21.15.31:0/3042120980 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc3bfff640 1 --2- 172.21.15.31:0/3042120980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce78cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc3b7fe640 1 --2- 172.21.15.31:0/3042120980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce79230 0x56387ce7b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc4081a640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efc240a2e20 0x7efc240a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc397fa640 1 -- 172.21.15.31:0/4275007235 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7efc300ceed0 con 0x7efc3c102100 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc4081a640 1 -- 172.21.15.31:0/3042120980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce79230 msgr2=0x56387ce7b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc4081a640 1 --2- 172.21.15.31:0/3042120980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce79230 0x56387ce7b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc4081a640 1 -- 172.21.15.31:0/3042120980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 msgr2=0x56387ce78cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc4081a640 1 --2- 172.21.15.31:0/3042120980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce78cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc4081a640 1 -- 172.21.15.31:0/3042120980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56387cae7210 con 0x56387ce6e240 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.153+0000 7efc4081a640 1 --2- 172.21.15.31:0/3042120980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce6e620 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7efc34012920 tx=0x7efc34012de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36044c9bd072b3cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.154+0000 7efc227fc640 1 -- 172.21.15.31:0/3042120980 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34022070 con 0x56387ce6e240 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.154+0000 7efc227fc640 1 -- 172.21.15.31:0/3042120980 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc34003b70 con 0x56387ce6e240 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.154+0000 7efc227fc640 1 -- 172.21.15.31:0/3042120980 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34005280 con 0x56387ce6e240 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.154+0000 7efc44f8d740 1 -- 172.21.15.31:0/3042120980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 msgr2=0x56387ce6e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.154+0000 7efc4081a640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efc240a2e20 0x7efc240a52e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efc34020000 tx=0x7efc3401f000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.154+0000 7efc44f8d740 1 --2- 172.21.15.31:0/3042120980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce6e620 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7efc34012920 tx=0x7efc34012de0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.154+0000 7efc44f8d740 1 -- 172.21.15.31:0/3042120980 shutdown_connections 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.154+0000 7efc44f8d740 1 --2- 172.21.15.31:0/3042120980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce79230 0x56387ce7b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.154+0000 7efc44f8d740 1 --2- 172.21.15.31:0/3042120980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce78cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.154+0000 7efc44f8d740 1 --2- 172.21.15.31:0/3042120980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce6e620 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.154+0000 7efc44f8d740 1 -- 172.21.15.31:0/3042120980 >> 172.21.15.31:0/3042120980 conn(0x56387ce6a560 msgr2=0x56387ce6a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.155+0000 7efc44f8d740 1 -- 172.21.15.31:0/3042120980 shutdown_connections 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.155+0000 7efc44f8d740 1 -- 172.21.15.31:0/3042120980 wait complete. 2024-04-25T02:04:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.155+0000 7efc44f8d740 1 -- start start 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.155+0000 7efc44f8d740 1 Processor -- start 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.155+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce6e240 0x56387ce76170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.155+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce766b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.155+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce79230 0x56387ce76bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.155+0000 7efc44f8d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56387ce7c020 con 0x56387ce79230 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.155+0000 7efc44f8d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56387cd8eb50 con 0x56387ce6eb60 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.155+0000 7efc44f8d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56387ce7c1a0 con 0x56387ce6e240 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.155+0000 7efc44f8d740 10 client.0 _ll_register_callbacks cb 0x56387cdc9ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc4081a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce79230 0x56387ce76bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc4081a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce79230 0x56387ce76bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37810/0 (socket says 172.21.15.31:37810) 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc4081a640 1 -- 172.21.15.31:0/1488457451 learned_addr learned my addr 172.21.15.31:0/1488457451 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc3b7fe640 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce766b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc3bfff640 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce6e240 0x56387ce76170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc4081a640 1 -- 172.21.15.31:0/1488457451 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce6e240 msgr2=0x56387ce76170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc4081a640 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce6e240 0x56387ce76170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc4081a640 1 -- 172.21.15.31:0/1488457451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 msgr2=0x56387ce766b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc4081a640 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce766b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc4081a640 1 -- 172.21.15.31:0/1488457451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56387cbb3620 con 0x56387ce79230 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc4081a640 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce79230 0x56387ce76bf0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7efc34003f50 tx=0x7efc34003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa121422e98a76c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc1b7fe640 1 -- 172.21.15.31:0/1488457451 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34022050 con 0x56387ce79230 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc1b7fe640 1 -- 172.21.15.31:0/1488457451 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7efc34003970 con 0x56387ce79230 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc44f8d740 1 -- 172.21.15.31:0/1488457451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc34003b70 con 0x56387ce79230 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc1b7fe640 1 -- 172.21.15.31:0/1488457451 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7efc34012460 con 0x56387ce79230 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.156+0000 7efc44f8d740 10 client.16941 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.157+0000 7efc44f8d740 1 -- 172.21.15.31:0/1488457451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56387ce881e0 con 0x56387ce79230 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.157+0000 7efc1b7fe640 1 -- 172.21.15.31:0/1488457451 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc34003b70 con 0x56387ce79230 2024-04-25T02:04:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.157+0000 7efc1b7fe640 1 -- 172.21.15.31:0/1488457451 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7efc34039070 con 0x56387ce79230 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.157+0000 7efc44f8d740 10 client.16941 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.157+0000 7efc44f8d740 10 client.16941 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.157+0000 7efc44f8d740 20 client.16941 populate_metadata read hostname 'smithi031' 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.157+0000 7efc44f8d740 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56387ce89640 0x56387ce8ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.157+0000 7efc44f8d740 4 client.16941 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.157+0000 7efc44f8d740 1 -- 172.21.15.31:0/1488457451 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56387cde01f0 con 0x56387ce89640 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.157+0000 7efc3bfff640 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56387ce89640 0x56387ce8ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.158+0000 7efc1b7fe640 10 client.16941 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.158+0000 7efc3bfff640 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56387ce89640 0x56387ce8ba50 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37675ec76962384c server_cookie=4d8403c0f63579 in_seq=0 out_seq=0 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.158+0000 7efc1b7fe640 1 -- 172.21.15.31:0/1488457451 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56387cde01f0 con 0x56387ce89640 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.158+0000 7efc1b7fe640 10 client.16941 handle_command_reply: tid=0 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.158+0000 7efc44f8d740 1 client.16941 shutdown 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc44f8d740 1 -- 172.21.15.31:0/1488457451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce79230 msgr2=0x56387ce76bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc44f8d740 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce79230 0x56387ce76bf0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7efc34003f50 tx=0x7efc34003f80 comp rx=0 tx=0).stop 2024-04-25T02:04:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc44f8d740 1 -- 172.21.15.31:0/1488457451 shutdown_connections 2024-04-25T02:04:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc44f8d740 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56387ce89640 0x56387ce8ba50 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc44f8d740 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce79230 0x56387ce76bf0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc44f8d740 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce766b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc1b7fe640 0 client.16941 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc44f8d740 1 --2- 172.21.15.31:0/1488457451 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce6e240 0x56387ce76170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc44f8d740 1 -- 172.21.15.31:0/1488457451 >> 172.21.15.31:0/1488457451 conn(0x56387ce6a560 msgr2=0x56387ce70360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc44f8d740 1 -- 172.21.15.31:0/1488457451 shutdown_connections 2024-04-25T02:04:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc44f8d740 1 -- 172.21.15.31:0/1488457451 wait complete. 2024-04-25T02:04:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.159+0000 7efc44f8d740 20 client.16941 trim_cache size 0 max 16384 2024-04-25T02:04:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc44f8d740 1 Processor -- start 2024-04-25T02:04:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc44f8d740 1 -- start start 2024-04-25T02:04:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce84ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce6eb60 0x56387ce70ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce79230 0x56387ce91360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc44f8d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c1af5d0 con 0x56387ce6e240 2024-04-25T02:04:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc44f8d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c1b1230 con 0x56387ce79230 2024-04-25T02:04:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc44f8d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c1afca0 con 0x56387ce6eb60 2024-04-25T02:04:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc3bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce84ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc3bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce84ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37824/0 (socket says 172.21.15.31:37824) 2024-04-25T02:04:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc3bfff640 1 -- 172.21.15.31:0/2797786690 learned_addr learned my addr 172.21.15.31:0/2797786690 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc4081a640 1 --2- 172.21.15.31:0/2797786690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce79230 0x56387ce91360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc3b7fe640 1 --2- 172.21.15.31:0/2797786690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce6eb60 0x56387ce70ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc3bfff640 1 -- 172.21.15.31:0/2797786690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce6eb60 msgr2=0x56387ce70ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc3bfff640 1 --2- 172.21.15.31:0/2797786690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce6eb60 0x56387ce70ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc3bfff640 1 -- 172.21.15.31:0/2797786690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce79230 msgr2=0x56387ce91360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc3bfff640 1 --2- 172.21.15.31:0/2797786690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce79230 0x56387ce91360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.174+0000 7efc3bfff640 1 -- 172.21.15.31:0/2797786690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56387ce746c0 con 0x56387ce6e240 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.175+0000 7efc3b7fe640 1 --2- 172.21.15.31:0/2797786690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce6eb60 0x56387ce70ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.175+0000 7efc3bfff640 1 --2- 172.21.15.31:0/2797786690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce84ee0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7efc2c02a940 tx=0x7efc2c005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d5deac16c694c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.175+0000 7efc21ffb640 1 -- 172.21.15.31:0/2797786690 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c035040 con 0x56387ce6e240 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.175+0000 7efc21ffb640 1 -- 172.21.15.31:0/2797786690 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc2c0054f0 con 0x56387ce6e240 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.175+0000 7efc21ffb640 1 -- 172.21.15.31:0/2797786690 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c043820 con 0x56387ce6e240 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.175+0000 7efc44f8d740 1 -- 172.21.15.31:0/2797786690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 msgr2=0x56387ce84ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.175+0000 7efc44f8d740 1 --2- 172.21.15.31:0/2797786690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce84ee0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7efc2c02a940 tx=0x7efc2c005b50 comp rx=0 tx=0).stop 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.175+0000 7efc44f8d740 1 -- 172.21.15.31:0/2797786690 shutdown_connections 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.175+0000 7efc44f8d740 1 --2- 172.21.15.31:0/2797786690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce79230 0x56387ce91360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 --2- 172.21.15.31:0/2797786690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce6eb60 0x56387ce70ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 --2- 172.21.15.31:0/2797786690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce84ee0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 -- 172.21.15.31:0/2797786690 >> 172.21.15.31:0/2797786690 conn(0x56387ce6a560 msgr2=0x56387ce695c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 -- 172.21.15.31:0/2797786690 shutdown_connections 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 -- 172.21.15.31:0/2797786690 wait complete. 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 -- start start 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 Processor -- start 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce81910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce72c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce79230 0x56387ce73380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c1b0b60 con 0x56387ce6e240 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c1b03d0 con 0x56387ce6eb60 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c1ad5d0 con 0x56387ce79230 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc44f8d740 10 client.0 _ll_register_callbacks cb 0x7efc2c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.176+0000 7efc3bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce81910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc3bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce81910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37826/0 (socket says 172.21.15.31:37826) 2024-04-25T02:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc3bfff640 1 -- 172.21.15.31:0/88451184 learned_addr learned my addr 172.21.15.31:0/88451184 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc3b7fe640 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce72c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc3bfff640 1 -- 172.21.15.31:0/88451184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce79230 msgr2=0x56387ce73380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc3bfff640 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce79230 0x56387ce73380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc3bfff640 1 -- 172.21.15.31:0/88451184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 msgr2=0x56387ce72c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc3bfff640 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce72c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc3bfff640 1 -- 172.21.15.31:0/88451184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56387ceeebd0 con 0x56387ce6e240 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc3bfff640 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce81910 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7efc2c043440 tx=0x7efc2c043340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966e89ae8c6a7dca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc1b7fe640 1 -- 172.21.15.31:0/88451184 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c050070 con 0x56387ce6e240 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc44f8d740 1 -- 172.21.15.31:0/88451184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc2c0054f0 con 0x56387ce6e240 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc1b7fe640 1 -- 172.21.15.31:0/88451184 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7efc2c0438f0 con 0x56387ce6e240 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc1b7fe640 1 -- 172.21.15.31:0/88451184 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7efc2c035030 con 0x56387ce6e240 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc44f8d740 10 client.16947 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc44f8d740 1 -- 172.21.15.31:0/88451184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56387ceefc30 con 0x56387ce6e240 2024-04-25T02:04:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.177+0000 7efc1b7fe640 1 -- 172.21.15.31:0/88451184 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc2c0054f0 con 0x56387ce6e240 2024-04-25T02:04:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.178+0000 7efc1b7fe640 1 -- 172.21.15.31:0/88451184 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7efc2c0317d0 con 0x56387ce6e240 2024-04-25T02:04:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.178+0000 7efc44f8d740 10 client.16947 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.178+0000 7efc44f8d740 10 client.16947 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.178+0000 7efc44f8d740 20 client.16947 populate_metadata read hostname 'smithi031' 2024-04-25T02:04:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.178+0000 7efc44f8d740 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56387cef0480 0x56387cef28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.178+0000 7efc44f8d740 4 client.16947 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:04:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.178+0000 7efc44f8d740 1 -- 172.21.15.31:0/88451184 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56387cdda540 con 0x56387cef0480 2024-04-25T02:04:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.178+0000 7efc3b7fe640 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56387cef0480 0x56387cef28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.178+0000 7efc3b7fe640 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56387cef0480 0x56387cef28a0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c04bbc79d759fd5b server_cookie=e5a9323e77a23f47 in_seq=0 out_seq=0 2024-04-25T02:04:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:33.178+0000 7efc1b7fe640 10 client.16947 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:33.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220104 2.16 MB/sec execute 400 sec latency 3039.130 ms 2024-04-25T02:04:34.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220104 2.15 MB/sec execute 401 sec latency 4039.231 ms 2024-04-25T02:04:34.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:33 smithi040 ceph-mon[32418]: pgmap v314: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 993 KiB/s wr, 91 op/s 2024-04-25T02:04:34.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:33 smithi134 ceph-mon[33098]: pgmap v314: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 993 KiB/s wr, 91 op/s 2024-04-25T02:04:34.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:33 smithi031 ceph-mon[29204]: pgmap v314: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 993 KiB/s wr, 91 op/s 2024-04-25T02:04:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc1b7fe640 1 -- 172.21.15.31:0/88451184 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56387cdda540 con 0x56387cef0480 2024-04-25T02:04:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc1b7fe640 10 client.16947 handle_command_reply: tid=0 2024-04-25T02:04:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc44f8d740 1 client.16947 shutdown 2024-04-25T02:04:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc44f8d740 1 -- 172.21.15.31:0/88451184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 msgr2=0x56387ce81910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc44f8d740 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce81910 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7efc2c043440 tx=0x7efc2c043340 comp rx=0 tx=0).stop 2024-04-25T02:04:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc44f8d740 1 -- 172.21.15.31:0/88451184 shutdown_connections 2024-04-25T02:04:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc44f8d740 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56387cef0480 0x56387cef28a0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc44f8d740 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56387ce79230 0x56387ce73380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc44f8d740 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56387ce6eb60 0x56387ce72c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc44f8d740 1 --2- 172.21.15.31:0/88451184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56387ce6e240 0x56387ce81910 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc1b7fe640 0 client.16947 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.048+0000 7efc44f8d740 1 -- 172.21.15.31:0/88451184 >> 172.21.15.31:0/88451184 conn(0x56387ce6a560 msgr2=0x56387ce76110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.049+0000 7efc44f8d740 1 -- 172.21.15.31:0/88451184 shutdown_connections 2024-04-25T02:04:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.049+0000 7efc44f8d740 1 -- 172.21.15.31:0/88451184 wait complete. 2024-04-25T02:04:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.049+0000 7efc44f8d740 20 client.16947 trim_cache size 0 max 16384 2024-04-25T02:04:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:04:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:04:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:04:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:04:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 -- 172.21.15.31:0/4275007235 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efc240a2e20 msgr2=0x7efc240a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efc240a2e20 0x7efc240a52e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efc34020000 tx=0x7efc3401f000 comp rx=0 tx=0).stop 2024-04-25T02:04:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 -- 172.21.15.31:0/4275007235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc3c102100 msgr2=0x7efc3c073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc3c102100 0x7efc3c073f80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7efc3000d950 tx=0x7efc3000de10 comp rx=0 tx=0).stop 2024-04-25T02:04:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 -- 172.21.15.31:0/4275007235 shutdown_connections 2024-04-25T02:04:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efc240a2e20 0x7efc240a52e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc3c107750 0x7efc3c0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc3c102100 0x7efc3c073f80 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 --2- 172.21.15.31:0/4275007235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc3c101740 0x7efc3c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 -- 172.21.15.31:0/4275007235 >> 172.21.15.31:0/4275007235 conn(0x7efc3c0fd210 msgr2=0x7efc3c0fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 -- 172.21.15.31:0/4275007235 shutdown_connections 2024-04-25T02:04:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:35.050+0000 7efc4227d640 1 -- 172.21.15.31:0/4275007235 wait complete. 2024-04-25T02:04:35.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220116 2.15 MB/sec execute 402 sec latency 4757.771 ms 2024-04-25T02:04:36.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220116 2.14 MB/sec execute 403 sec latency 1215.184 ms 2024-04-25T02:04:36.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:35 smithi040 ceph-mon[32418]: pgmap v315: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:04:36.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:35 smithi134 ceph-mon[33098]: pgmap v315: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:04:36.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:35 smithi031 ceph-mon[29204]: pgmap v315: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:04:37.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220116 2.14 MB/sec execute 404 sec latency 2215.277 ms 2024-04-25T02:04:38.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:37 smithi031 ceph-mon[29204]: pgmap v316: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 558 KiB/s wr, 51 op/s 2024-04-25T02:04:38.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220282 2.14 MB/sec execute 405 sec latency 2609.342 ms 2024-04-25T02:04:38.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:37 smithi040 ceph-mon[32418]: pgmap v316: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 558 KiB/s wr, 51 op/s 2024-04-25T02:04:38.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:37 smithi134 ceph-mon[33098]: pgmap v316: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 558 KiB/s wr, 51 op/s 2024-04-25T02:04:39.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220555 2.13 MB/sec execute 406 sec latency 14.909 ms 2024-04-25T02:04:40.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 221008 2.13 MB/sec execute 407 sec latency 18.610 ms 2024-04-25T02:04:40.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:40 smithi040 ceph-mon[32418]: pgmap v317: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 558 KiB/s wr, 51 op/s 2024-04-25T02:04:40.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:40 smithi134 ceph-mon[33098]: pgmap v317: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 558 KiB/s wr, 51 op/s 2024-04-25T02:04:40.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:40 smithi031 ceph-mon[29204]: pgmap v317: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 558 KiB/s wr, 51 op/s 2024-04-25T02:04:41.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 221494 2.13 MB/sec execute 408 sec latency 21.081 ms 2024-04-25T02:04:42.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 222022 2.13 MB/sec execute 409 sec latency 21.990 ms 2024-04-25T02:04:42.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:42 smithi040 ceph-mon[32418]: pgmap v318: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 824 KiB/s wr, 80 op/s 2024-04-25T02:04:42.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:42 smithi134 ceph-mon[33098]: pgmap v318: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 824 KiB/s wr, 80 op/s 2024-04-25T02:04:42.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:42 smithi031 ceph-mon[29204]: pgmap v318: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 824 KiB/s wr, 80 op/s 2024-04-25T02:04:43.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 222514 2.13 MB/sec execute 410 sec latency 23.153 ms 2024-04-25T02:04:44.134 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:44 smithi134 ceph-mon[33098]: pgmap v319: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 326 KiB/s wr, 45 op/s 2024-04-25T02:04:44.134 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:44 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:04:44.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:44 smithi031 ceph-mon[29204]: pgmap v319: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 326 KiB/s wr, 45 op/s 2024-04-25T02:04:44.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:44 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:04:44.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223208 2.14 MB/sec execute 411 sec latency 17.562 ms 2024-04-25T02:04:44.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:44 smithi040 ceph-mon[32418]: pgmap v319: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 326 KiB/s wr, 45 op/s 2024-04-25T02:04:44.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:44 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:04:45.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223496 2.14 MB/sec execute 412 sec latency 18.641 ms 2024-04-25T02:04:45.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:45.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:45.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: pgmap v320: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 937 KiB/s wr, 100 op/s 2024-04-25T02:04:45.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:45.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:45.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:45.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:04:45.994 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:04:45.995 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: pgmap v320: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 937 KiB/s wr, 100 op/s 2024-04-25T02:04:46.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:04:46.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: pgmap v320: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 937 KiB/s wr, 100 op/s 2024-04-25T02:04:46.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:04:46.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:04:46.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:04:46.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:04:46.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223582 2.14 MB/sec execute 413 sec latency 826.146 ms 2024-04-25T02:04:46.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:46 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi134 to 176.2M 2024-04-25T02:04:46.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:46 smithi031 ceph-mon[29204]: Unable to set osd_memory_target on smithi134 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T02:04:46.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:46 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-25T02:04:46.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:46 smithi031 ceph-mon[29204]: Unable to set osd_memory_target on smithi040 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T02:04:46.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:46 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T02:04:46.993 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:46 smithi031 ceph-mon[29204]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T02:04:47.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:46 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi134 to 176.2M 2024-04-25T02:04:47.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:46 smithi040 ceph-mon[32418]: Unable to set osd_memory_target on smithi134 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T02:04:47.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:46 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-25T02:04:47.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:46 smithi040 ceph-mon[32418]: Unable to set osd_memory_target on smithi040 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T02:04:47.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:46 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T02:04:47.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:46 smithi040 ceph-mon[32418]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T02:04:47.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:46 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi134 to 176.2M 2024-04-25T02:04:47.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:46 smithi134 ceph-mon[33098]: Unable to set osd_memory_target on smithi134 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T02:04:47.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:46 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-25T02:04:47.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:46 smithi134 ceph-mon[33098]: Unable to set osd_memory_target on smithi040 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T02:04:47.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:46 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T02:04:47.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:46 smithi134 ceph-mon[33098]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T02:04:47.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223582 2.13 MB/sec execute 414 sec latency 1826.274 ms 2024-04-25T02:04:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.152+0000 7f6d0fde2640 1 -- 172.21.15.31:0/3518401312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08106670 msgr2=0x7f6d08106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.152+0000 7f6d0fde2640 1 --2- 172.21.15.31:0/3518401312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08106670 0x7f6d08106a50 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009930 tx=0x7f6d0402a420 comp rx=0 tx=0).stop 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.152+0000 7f6d0fde2640 1 -- 172.21.15.31:0/3518401312 shutdown_connections 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.152+0000 7f6d0fde2640 1 --2- 172.21.15.31:0/3518401312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08101020 0x7f6d0810d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.152+0000 7f6d0fde2640 1 --2- 172.21.15.31:0/3518401312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d08100660 0x7f6d08100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.152+0000 7f6d0fde2640 1 --2- 172.21.15.31:0/3518401312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08106670 0x7f6d08106a50 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.152+0000 7f6d0fde2640 1 -- 172.21.15.31:0/3518401312 >> 172.21.15.31:0/3518401312 conn(0x7f6d080769c0 msgr2=0x7f6d08076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.152+0000 7f6d0fde2640 1 -- 172.21.15.31:0/3518401312 shutdown_connections 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.152+0000 7f6d0fde2640 1 -- 172.21.15.31:0/3518401312 wait complete. 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0fde2640 1 Processor -- start 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0fde2640 1 -- start start 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0fde2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d08100660 0x7f6d08197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0fde2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08101020 0x7f6d081984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0fde2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08106670 0x7f6d0819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0fde2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6d08110930 con 0x7f6d08106670 2024-04-25T02:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0fde2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d081107b0 con 0x7f6d08101020 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0fde2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6d08110ab0 con 0x7f6d08100660 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0e37f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08106670 0x7f6d0819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0e37f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08106670 0x7f6d0819c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54482/0 (socket says 172.21.15.31:54482) 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0e37f640 1 -- 172.21.15.31:0/1976425949 learned_addr learned my addr 172.21.15.31:0/1976425949 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08101020 0x7f6d081984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.153+0000 7f6d0db7e640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d08100660 0x7f6d08197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.154+0000 7f6d0e37f640 1 -- 172.21.15.31:0/1976425949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d08100660 msgr2=0x7f6d08197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.154+0000 7f6d0e37f640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d08100660 0x7f6d08197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.154+0000 7f6d0e37f640 1 -- 172.21.15.31:0/1976425949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08101020 msgr2=0x7f6d081984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.154+0000 7f6d0e37f640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08101020 0x7f6d081984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.154+0000 7f6d0e37f640 1 -- 172.21.15.31:0/1976425949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0819cf70 con 0x7f6d08106670 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.154+0000 7f6d0e37f640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08106670 0x7f6d0819c840 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000eaa0 tx=0x7f6d0000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.154+0000 7f6cfeffd640 1 -- 172.21.15.31:0/1976425949 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d00004d20 con 0x7f6d08106670 2024-04-25T02:04:48.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.154+0000 7f6cfeffd640 1 -- 172.21.15.31:0/1976425949 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d00005020 con 0x7f6d08106670 2024-04-25T02:04:48.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.154+0000 7f6d0fde2640 1 -- 172.21.15.31:0/1976425949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0819d260 con 0x7f6d08106670 2024-04-25T02:04:48.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.154+0000 7f6cfeffd640 1 -- 172.21.15.31:0/1976425949 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d000116b0 con 0x7f6d08106670 2024-04-25T02:04:48.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.154+0000 7f6d0fde2640 1 -- 172.21.15.31:0/1976425949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d081a4af0 con 0x7f6d08106670 2024-04-25T02:04:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.156+0000 7f6cfeffd640 1 -- 172.21.15.31:0/1976425949 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6d000051d0 con 0x7f6d08106670 2024-04-25T02:04:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.157+0000 7f6d12bc5740 1 Processor -- start 2024-04-25T02:04:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.157+0000 7f6d12bc5740 1 -- start start 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.157+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57240 0x5609d7b57620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.157+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b57b60 0x5609d7b61cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.157+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b62230 0x5609d7b64620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.157+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5609d7abbcb0 con 0x5609d7b62230 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.157+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5609d7a645b0 con 0x5609d7b57240 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.157+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5609d7ab7c70 con 0x5609d7b57b60 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.157+0000 7f6cfeffd640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6cf40a2da0 0x7f6cf40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0d37d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b62230 0x5609d7b64620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0e37f640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6cf40a2da0 0x7f6cf40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0d37d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b62230 0x5609d7b64620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54486/0 (socket says 172.21.15.31:54486) 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0d37d640 1 -- 172.21.15.31:0/1915653286 learned_addr learned my addr 172.21.15.31:0/1915653286 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/1915653286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57240 0x5609d7b57620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0db7e640 1 --2- 172.21.15.31:0/1915653286 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b57b60 0x5609d7b61cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6cfeffd640 1 -- 172.21.15.31:0/1976425949 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f6d000cf6d0 con 0x7f6d08106670 2024-04-25T02:04:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0d37d640 1 -- 172.21.15.31:0/1915653286 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b57b60 msgr2=0x5609d7b61cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/1915653286 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b57b60 0x5609d7b61cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0d37d640 1 -- 172.21.15.31:0/1915653286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57240 msgr2=0x5609d7b57620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/1915653286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57240 0x5609d7b57620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0d37d640 1 -- 172.21.15.31:0/1915653286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609d77d0210 con 0x5609d7b62230 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0db7e640 1 --2- 172.21.15.31:0/1915653286 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b57b60 0x5609d7b61cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0e37f640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6cf40a2da0 0x7f6cf40a5260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d00098010 tx=0x7f6d00092040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/1915653286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b62230 0x5609d7b64620 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006aa0 tx=0x7f6cf8009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0fe5eda942b744 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6cf3fff640 1 -- 172.21.15.31:0/1915653286 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8015760 con 0x5609d7b62230 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6cf3fff640 1 -- 172.21.15.31:0/1915653286 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6cf801d070 con 0x5609d7b62230 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.158+0000 7f6cf3fff640 1 -- 172.21.15.31:0/1915653286 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8018b60 con 0x5609d7b62230 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.159+0000 7f6d12bc5740 1 -- 172.21.15.31:0/1915653286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b62230 msgr2=0x5609d7b64620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.159+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/1915653286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b62230 0x5609d7b64620 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006aa0 tx=0x7f6cf8009510 comp rx=0 tx=0).stop 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.159+0000 7f6d12bc5740 1 -- 172.21.15.31:0/1915653286 shutdown_connections 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.159+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/1915653286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b62230 0x5609d7b64620 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.159+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/1915653286 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b57b60 0x5609d7b61cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.159+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/1915653286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57240 0x5609d7b57620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.159+0000 7f6d12bc5740 1 -- 172.21.15.31:0/1915653286 >> 172.21.15.31:0/1915653286 conn(0x5609d7b53560 msgr2=0x5609d7b53940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.159+0000 7f6d12bc5740 1 -- 172.21.15.31:0/1915653286 shutdown_connections 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.159+0000 7f6d12bc5740 1 -- 172.21.15.31:0/1915653286 wait complete. 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d12bc5740 1 -- start start 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d12bc5740 1 Processor -- start 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b5f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 0x5609d7b5fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5609d7b64fc0 con 0x5609d7b57240 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5609d7a77b50 con 0x5609d7b57b60 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5609d7b65140 con 0x5609d7b62230 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d12bc5740 10 client.0 _ll_register_callbacks cb 0x5609d7ab2ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d0d37d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d0d37d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54496/0 (socket says 172.21.15.31:54496) 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.160+0000 7f6d0d37d640 1 -- 172.21.15.31:0/199215362 learned_addr learned my addr 172.21.15.31:0/199215362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6d0db7e640 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b5f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6d0d37d640 1 -- 172.21.15.31:0/199215362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 msgr2=0x5609d7b5fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 0x5609d7b5fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6d0d37d640 1 -- 172.21.15.31:0/199215362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 msgr2=0x5609d7b5f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b5f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6d0d37d640 1 -- 172.21.15.31:0/199215362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609d789c620 con 0x5609d7b57240 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6d0db7e640 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b5f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5f150 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009b80 tx=0x7f6cf8009130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911070f9d892f69c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6cf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6cf17fa640 1 -- 172.21.15.31:0/199215362 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8026070 con 0x5609d7b57240 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6cf17fa640 1 -- 172.21.15.31:0/199215362 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f6cf8015730 con 0x5609d7b57240 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6d12bc5740 1 -- 172.21.15.31:0/199215362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cf801d070 con 0x5609d7b57240 2024-04-25T02:04:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6cf17fa640 1 -- 172.21.15.31:0/199215362 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6cf8015930 con 0x5609d7b57240 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6d12bc5740 10 client.16959 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6d12bc5740 1 -- 172.21.15.31:0/199215362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5609d7b71150 con 0x5609d7b57240 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.161+0000 7f6cf17fa640 1 -- 172.21.15.31:0/199215362 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6cf801d070 con 0x5609d7b57240 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.162+0000 7f6cf17fa640 1 -- 172.21.15.31:0/199215362 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f6cf80228e0 con 0x5609d7b57240 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.162+0000 7f6d12bc5740 10 client.16959 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.162+0000 7f6d12bc5740 10 client.16959 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.162+0000 7f6d12bc5740 20 client.16959 populate_metadata read hostname 'smithi031' 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.162+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5609d7b72560 0x5609d7b74950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.162+0000 7f6d12bc5740 4 client.16959 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.162+0000 7f6d12bc5740 1 -- 172.21.15.31:0/199215362 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609d7ac91f0 con 0x5609d7b72560 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.162+0000 7f6d0db7e640 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5609d7b72560 0x5609d7b74950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.162+0000 7f6d0db7e640 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5609d7b72560 0x5609d7b74950 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddab09dd1980a665 server_cookie=f655bf5795b6e49b in_seq=0 out_seq=0 2024-04-25T02:04:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.162+0000 7f6cf17fa640 10 client.16959 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:48.174 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:47 smithi040 ceph-mon[32418]: pgmap v321: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 877 KiB/s wr, 84 op/s 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6cf17fa640 1 -- 172.21.15.31:0/199215362 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5609d7ac91f0 con 0x5609d7b72560 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6cf17fa640 10 client.16959 handle_command_reply: tid=0 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6d12bc5740 1 client.16959 shutdown 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6d12bc5740 1 -- 172.21.15.31:0/199215362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 msgr2=0x5609d7b5f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5f150 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009b80 tx=0x7f6cf8009130 comp rx=0 tx=0).stop 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6d12bc5740 1 -- 172.21.15.31:0/199215362 shutdown_connections 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5609d7b72560 0x5609d7b74950 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 0x5609d7b5fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b5f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/199215362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5f150 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6cf17fa640 0 client.16959 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.163+0000 7f6d12bc5740 1 -- 172.21.15.31:0/199215362 >> 172.21.15.31:0/199215362 conn(0x5609d7b53560 msgr2=0x5609d7b59360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.164+0000 7f6d12bc5740 1 -- 172.21.15.31:0/199215362 shutdown_connections 2024-04-25T02:04:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.164+0000 7f6d12bc5740 1 -- 172.21.15.31:0/199215362 wait complete. 2024-04-25T02:04:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.164+0000 7f6d12bc5740 20 client.16959 trim_cache size 0 max 16384 2024-04-25T02:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.179+0000 7f6d12bc5740 1 Processor -- start 2024-04-25T02:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.179+0000 7f6d12bc5740 1 -- start start 2024-04-25T02:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.179+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b6de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.179+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b51d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.179+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 0x5609d7b59a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.179+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6d081ad2c0 con 0x5609d7b57240 2024-04-25T02:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.179+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d081aef20 con 0x5609d7b57b60 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.179+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6d081ad990 con 0x5609d7b62230 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.179+0000 7f6d0d37d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b6de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6d0d37d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b6de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54504/0 (socket says 172.21.15.31:54504) 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6d0d37d640 1 -- 172.21.15.31:0/2698481241 learned_addr learned my addr 172.21.15.31:0/2698481241 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6d0db7e640 1 --2- 172.21.15.31:0/2698481241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b51d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6d0d37d640 1 -- 172.21.15.31:0/2698481241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 msgr2=0x5609d7b59a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/2698481241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 0x5609d7b59a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6d0d37d640 1 -- 172.21.15.31:0/2698481241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 msgr2=0x5609d7b51d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/2698481241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b51d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6d0d37d640 1 -- 172.21.15.31:0/2698481241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609d7b58700 con 0x5609d7b57240 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/2698481241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b6de80 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800b040 tx=0x7f6cf8017680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c482750f92ff8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6cf37fe640 1 -- 172.21.15.31:0/2698481241 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8026040 con 0x5609d7b57240 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6cf37fe640 1 -- 172.21.15.31:0/2698481241 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6cf801d040 con 0x5609d7b57240 2024-04-25T02:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.180+0000 7f6cf37fe640 1 -- 172.21.15.31:0/2698481241 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8018890 con 0x5609d7b57240 2024-04-25T02:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.181+0000 7f6d12bc5740 1 -- 172.21.15.31:0/2698481241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 msgr2=0x5609d7b6de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.181+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/2698481241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b6de80 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800b040 tx=0x7f6cf8017680 comp rx=0 tx=0).stop 2024-04-25T02:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.181+0000 7f6d12bc5740 1 -- 172.21.15.31:0/2698481241 shutdown_connections 2024-04-25T02:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.181+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/2698481241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 0x5609d7b59a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.181+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/2698481241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b51d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.181+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/2698481241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b6de80 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.181+0000 7f6d12bc5740 1 -- 172.21.15.31:0/2698481241 >> 172.21.15.31:0/2698481241 conn(0x5609d7b53560 msgr2=0x5609d7b5e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.181+0000 7f6d12bc5740 1 -- 172.21.15.31:0/2698481241 shutdown_connections 2024-04-25T02:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.181+0000 7f6d12bc5740 1 -- 172.21.15.31:0/2698481241 wait complete. 2024-04-25T02:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.181+0000 7f6d12bc5740 1 -- start start 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.181+0000 7f6d12bc5740 1 Processor -- start 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b5c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d12bc5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 0x5609d7b5c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6d081ae850 con 0x5609d7b57240 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d081ae0c0 con 0x5609d7b57b60 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d12bc5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6d081ab2c0 con 0x5609d7b62230 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d12bc5740 10 client.0 _ll_register_callbacks cb 0x7f6cf8009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d0d37d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d0d37d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54510/0 (socket says 172.21.15.31:54510) 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d0d37d640 1 -- 172.21.15.31:0/545076142 learned_addr learned my addr 172.21.15.31:0/545076142 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 0x5609d7b5c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d0db7e640 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b5c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d0d37d640 1 -- 172.21.15.31:0/545076142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 msgr2=0x5609d7b5c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 0x5609d7b5c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d0d37d640 1 -- 172.21.15.31:0/545076142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 msgr2=0x5609d7b5c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b5c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d0d37d640 1 -- 172.21.15.31:0/545076142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609d7bd7950 con 0x5609d7b57240 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6d0d37d640 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5be80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x5609d7b673e0 tx=0x7f6cf8017510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661434b958d3996d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.182+0000 7f6cf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6cf17fa640 1 -- 172.21.15.31:0/545076142 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8046060 con 0x5609d7b57240 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6cf17fa640 1 -- 172.21.15.31:0/545076142 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f6cf8015ce0 con 0x5609d7b57240 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6d12bc5740 1 -- 172.21.15.31:0/545076142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cf801d040 con 0x5609d7b57240 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6cf17fa640 1 -- 172.21.15.31:0/545076142 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6cf803b040 con 0x5609d7b57240 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6d12bc5740 10 client.16965 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6d12bc5740 1 -- 172.21.15.31:0/545076142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5609d7bd8c60 con 0x5609d7b57240 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6cf17fa640 1 -- 172.21.15.31:0/545076142 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6cf801d040 con 0x5609d7b57240 2024-04-25T02:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6cf17fa640 1 -- 172.21.15.31:0/545076142 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f6cf8020900 con 0x5609d7b57240 2024-04-25T02:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6d12bc5740 10 client.16965 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6d12bc5740 10 client.16965 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6d12bc5740 20 client.16965 populate_metadata read hostname 'smithi031' 2024-04-25T02:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5609d7bd9350 0x5609d7bdb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6d12bc5740 4 client.16965 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.183+0000 7f6d12bc5740 1 -- 172.21.15.31:0/545076142 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5609d7ac3540 con 0x5609d7bd9350 2024-04-25T02:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.184+0000 7f6d0db7e640 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5609d7bd9350 0x5609d7bdb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.184+0000 7f6d0db7e640 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5609d7bd9350 0x5609d7bdb770 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=595f6508a64831e0 server_cookie=5ee6c3ad84d8023e in_seq=0 out_seq=0 2024-04-25T02:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:48.184+0000 7f6cf17fa640 10 client.16965 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:48.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:47 smithi134 ceph-mon[33098]: pgmap v321: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 877 KiB/s wr, 84 op/s 2024-04-25T02:04:48.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:47 smithi031 ceph-mon[29204]: pgmap v321: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 877 KiB/s wr, 84 op/s 2024-04-25T02:04:48.322 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223582 2.13 MB/sec execute 415 sec latency 2826.379 ms 2024-04-25T02:04:49.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223582 2.12 MB/sec execute 416 sec latency 3826.475 ms 2024-04-25T02:04:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6cf17fa640 1 -- 172.21.15.31:0/545076142 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5609d7ac3540 con 0x5609d7bd9350 2024-04-25T02:04:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6cf17fa640 10 client.16965 handle_command_reply: tid=0 2024-04-25T02:04:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6d12bc5740 1 client.16965 shutdown 2024-04-25T02:04:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6d12bc5740 1 -- 172.21.15.31:0/545076142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 msgr2=0x5609d7b5be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5be80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x5609d7b673e0 tx=0x7f6cf8017510 comp rx=0 tx=0).stop 2024-04-25T02:04:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6d12bc5740 1 -- 172.21.15.31:0/545076142 shutdown_connections 2024-04-25T02:04:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5609d7bd9350 0x5609d7bdb770 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5609d7b62230 0x5609d7b5c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5609d7b57b60 0x5609d7b5c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6cf17fa640 0 client.16965 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:04:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6d12bc5740 1 --2- 172.21.15.31:0/545076142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609d7b57240 0x5609d7b5be80 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6d12bc5740 1 -- 172.21.15.31:0/545076142 >> 172.21.15.31:0/545076142 conn(0x5609d7b53560 msgr2=0x5609d7b5d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6d12bc5740 1 -- 172.21.15.31:0/545076142 shutdown_connections 2024-04-25T02:04:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.050+0000 7f6d12bc5740 1 -- 172.21.15.31:0/545076142 wait complete. 2024-04-25T02:04:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.051+0000 7f6d12bc5740 20 client.16965 trim_cache size 0 max 16384 2024-04-25T02:04:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:04:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:04:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:04:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:04:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 -- 172.21.15.31:0/1976425949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6cf40a2da0 msgr2=0x7f6cf40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6cf40a2da0 0x7f6cf40a5260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d00098010 tx=0x7f6d00092040 comp rx=0 tx=0).stop 2024-04-25T02:04:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 -- 172.21.15.31:0/1976425949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08106670 msgr2=0x7f6d0819c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08106670 0x7f6d0819c840 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000eaa0 tx=0x7f6d0000eea0 comp rx=0 tx=0).stop 2024-04-25T02:04:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 -- 172.21.15.31:0/1976425949 shutdown_connections 2024-04-25T02:04:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6cf40a2da0 0x7f6cf40a5260 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d08106670 0x7f6d0819c840 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d08101020 0x7f6d081984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 --2- 172.21.15.31:0/1976425949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d08100660 0x7f6d08197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 -- 172.21.15.31:0/1976425949 >> 172.21.15.31:0/1976425949 conn(0x7f6d080769c0 msgr2=0x7f6d0806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 -- 172.21.15.31:0/1976425949 shutdown_connections 2024-04-25T02:04:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:04:50.052+0000 7f6d0fde2640 1 -- 172.21.15.31:0/1976425949 wait complete. 2024-04-25T02:04:50.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:49 smithi040 ceph-mon[32418]: pgmap v322: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 877 KiB/s wr, 84 op/s 2024-04-25T02:04:50.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:49 smithi134 ceph-mon[33098]: pgmap v322: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 877 KiB/s wr, 84 op/s 2024-04-25T02:04:50.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:49 smithi031 ceph-mon[29204]: pgmap v322: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 877 KiB/s wr, 84 op/s 2024-04-25T02:04:50.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223594 2.12 MB/sec execute 417 sec latency 4551.952 ms 2024-04-25T02:04:51.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223594 2.11 MB/sec execute 418 sec latency 1216.266 ms 2024-04-25T02:04:52.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:51 smithi040 ceph-mon[32418]: pgmap v323: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1021 KiB/s wr, 105 op/s 2024-04-25T02:04:52.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:51 smithi134 ceph-mon[33098]: pgmap v323: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1021 KiB/s wr, 105 op/s 2024-04-25T02:04:52.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:51 smithi031 ceph-mon[29204]: pgmap v323: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1021 KiB/s wr, 105 op/s 2024-04-25T02:04:52.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223649 2.11 MB/sec execute 419 sec latency 1228.676 ms 2024-04-25T02:04:53.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223649 2.10 MB/sec execute 420 sec latency 1784.521 ms 2024-04-25T02:04:54.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:53 smithi040 ceph-mon[32418]: pgmap v324: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 755 KiB/s wr, 75 op/s 2024-04-25T02:04:54.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:53 smithi134 ceph-mon[33098]: pgmap v324: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 755 KiB/s wr, 75 op/s 2024-04-25T02:04:54.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:53 smithi031 ceph-mon[29204]: pgmap v324: 97 pgs: 97 active+clean; 125 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 755 KiB/s wr, 75 op/s 2024-04-25T02:04:54.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223649 2.10 MB/sec execute 421 sec latency 2784.632 ms 2024-04-25T02:04:55.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223744 2.09 MB/sec execute 422 sec latency 3506.269 ms 2024-04-25T02:04:56.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:55 smithi040 ceph-mon[32418]: pgmap v325: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 789 KiB/s wr, 86 op/s 2024-04-25T02:04:56.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:55 smithi134 ceph-mon[33098]: pgmap v325: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 789 KiB/s wr, 86 op/s 2024-04-25T02:04:56.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:55 smithi031 ceph-mon[29204]: pgmap v325: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 789 KiB/s wr, 86 op/s 2024-04-25T02:04:56.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223974 2.09 MB/sec execute 423 sec latency 15.044 ms 2024-04-25T02:04:57.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 224304 2.09 MB/sec execute 424 sec latency 20.012 ms 2024-04-25T02:04:58.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:57 smithi040 ceph-mon[32418]: pgmap v326: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 178 KiB/s wr, 31 op/s 2024-04-25T02:04:58.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:57 smithi134 ceph-mon[33098]: pgmap v326: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 178 KiB/s wr, 31 op/s 2024-04-25T02:04:58.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:57 smithi031 ceph-mon[29204]: pgmap v326: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 178 KiB/s wr, 31 op/s 2024-04-25T02:04:58.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 224740 2.09 MB/sec execute 425 sec latency 19.568 ms 2024-04-25T02:04:59.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:58 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:04:59.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:58 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:04:59.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:58 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:04:59.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 225217 2.09 MB/sec execute 426 sec latency 23.074 ms 2024-04-25T02:05:00.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:04:59 smithi040 ceph-mon[32418]: pgmap v327: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 178 KiB/s wr, 31 op/s 2024-04-25T02:05:00.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:04:59 smithi134 ceph-mon[33098]: pgmap v327: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 178 KiB/s wr, 31 op/s 2024-04-25T02:05:00.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:04:59 smithi031 ceph-mon[29204]: pgmap v327: 97 pgs: 97 active+clean; 121 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 178 KiB/s wr, 31 op/s 2024-04-25T02:05:00.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 225762 2.08 MB/sec execute 427 sec latency 25.609 ms 2024-04-25T02:05:01.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 226264 2.09 MB/sec execute 428 sec latency 18.253 ms 2024-04-25T02:05:02.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:01 smithi040 ceph-mon[32418]: pgmap v328: 97 pgs: 97 active+clean; 122 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 656 KiB/s wr, 86 op/s 2024-04-25T02:05:02.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:01 smithi134 ceph-mon[33098]: pgmap v328: 97 pgs: 97 active+clean; 122 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 656 KiB/s wr, 86 op/s 2024-04-25T02:05:02.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:01 smithi031 ceph-mon[29204]: pgmap v328: 97 pgs: 97 active+clean; 122 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 656 KiB/s wr, 86 op/s 2024-04-25T02:05:02.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 226828 2.10 MB/sec execute 429 sec latency 14.296 ms 2024-04-25T02:05:03.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.147+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2804780904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb81029f0 msgr2=0x7f8cb8102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:03.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.147+0000 7f8cbdbe9640 1 --2- 172.21.15.31:0/2804780904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb81029f0 0x7f8cb8102dd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009900 tx=0x7f8ca802a480 comp rx=0 tx=0).stop 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.147+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2804780904 shutdown_connections 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.147+0000 7f8cbdbe9640 1 --2- 172.21.15.31:0/2804780904 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb810bdc0 0x7f8cb810e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.147+0000 7f8cbdbe9640 1 --2- 172.21.15.31:0/2804780904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cb8103310 0x7f8cb810b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.147+0000 7f8cbdbe9640 1 --2- 172.21.15.31:0/2804780904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb81029f0 0x7f8cb8102dd0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.147+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2804780904 >> 172.21.15.31:0/2804780904 conn(0x7f8cb80769c0 msgr2=0x7f8cb8076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.147+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2804780904 shutdown_connections 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.147+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2804780904 wait complete. 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cbdbe9640 1 Processor -- start 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cbdbe9640 1 -- start start 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cbdbe9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb81029f0 0x7f8cb8197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cbdbe9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb8103310 0x7f8cb8198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cbdbe9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cb810bdc0 0x7f8cb819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cbdbe9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb8111a00 con 0x7f8cb8103310 2024-04-25T02:05:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cbdbe9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb8111880 con 0x7f8cb810bdc0 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cbdbe9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb8111b80 con 0x7f8cb81029f0 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cb6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb8103310 0x7f8cb8198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cb6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb8103310 0x7f8cb8198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54052/0 (socket says 172.21.15.31:54052) 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cb77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb81029f0 0x7f8cb8197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cb6ffd640 1 -- 172.21.15.31:0/2404676226 learned_addr learned my addr 172.21.15.31:0/2404676226 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.148+0000 7f8cb7fff640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cb810bdc0 0x7f8cb819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.149+0000 7f8cb6ffd640 1 -- 172.21.15.31:0/2404676226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb81029f0 msgr2=0x7f8cb8197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.149+0000 7f8cb6ffd640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb81029f0 0x7f8cb8197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.149+0000 7f8cb6ffd640 1 -- 172.21.15.31:0/2404676226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cb810bdc0 msgr2=0x7f8cb819c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.149+0000 7f8cb6ffd640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cb810bdc0 0x7f8cb819c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.149+0000 7f8cb6ffd640 1 -- 172.21.15.31:0/2404676226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb819ce30 con 0x7f8cb8103310 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.149+0000 7f8cb6ffd640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb8103310 0x7f8cb8198380 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00c890 tx=0x7f8cac00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.149+0000 7f8cb4ff9640 1 -- 172.21.15.31:0/2404676226 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac004220 con 0x7f8cb8103310 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.149+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2404676226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb819d120 con 0x7f8cb8103310 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.149+0000 7f8cb4ff9640 1 -- 172.21.15.31:0/2404676226 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cac0043c0 con 0x7f8cb8103310 2024-04-25T02:05:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.149+0000 7f8cb4ff9640 1 -- 172.21.15.31:0/2404676226 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac004a00 con 0x7f8cb8103310 2024-04-25T02:05:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.149+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2404676226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb81a49b0 con 0x7f8cb8103310 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.152+0000 7f8cb4ff9640 1 -- 172.21.15.31:0/2404676226 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8cac004ba0 con 0x7f8cb8103310 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.152+0000 7f8cc09cc740 1 Processor -- start 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.152+0000 7f8cc09cc740 1 -- start start 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb4ff9640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f8ca00a2da0 0x7f8ca00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fdd240 0x55fd08fdd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 0x55fd08fe7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fe8230 0x55fd08fea620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fd08f41cb0 con 0x55fd08fddb60 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fd08eea5b0 con 0x55fd08fe8230 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fd08f3dc70 con 0x55fd08fdd240 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb7fff640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f8ca00a2da0 0x7f8ca00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb4ff9640 1 -- 172.21.15.31:0/2404676226 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f8cac0cf040 con 0x7f8cb8103310 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 0x55fd08fe7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fdd240 0x55fd08fdd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 0x55fd08fe7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54068/0 (socket says 172.21.15.31:54068) 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb6ffd640 1 -- 172.21.15.31:0/3593819146 learned_addr learned my addr 172.21.15.31:0/3593819146 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb7fff640 1 --2- 172.21.15.31:0/3593819146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fe8230 0x55fd08fea620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb77fe640 1 -- 172.21.15.31:0/3593819146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fe8230 msgr2=0x55fd08fea620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb77fe640 1 --2- 172.21.15.31:0/3593819146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fe8230 0x55fd08fea620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb77fe640 1 -- 172.21.15.31:0/3593819146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 msgr2=0x55fd08fe7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb77fe640 1 --2- 172.21.15.31:0/3593819146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 0x55fd08fe7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb77fe640 1 -- 172.21.15.31:0/3593819146 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd08c56210 con 0x55fd08fdd240 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb7fff640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f8ca00a2da0 0x7f8ca00a5260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8cb806b050 tx=0x7f8cb0009e60 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.153+0000 7f8cb77fe640 1 --2- 172.21.15.31:0/3593819146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fdd240 0x55fd08fdd620 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009230 tx=0x7f8ca8032e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b850bddf98830b5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.154+0000 7f8c9dffb640 1 -- 172.21.15.31:0/3593819146 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8040070 con 0x55fd08fdd240 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.154+0000 7f8c9dffb640 1 -- 172.21.15.31:0/3593819146 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ca80316a0 con 0x55fd08fdd240 2024-04-25T02:05:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.154+0000 7f8c9dffb640 1 -- 172.21.15.31:0/3593819146 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8035040 con 0x55fd08fdd240 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.155+0000 7f8cc09cc740 1 -- 172.21.15.31:0/3593819146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fdd240 msgr2=0x55fd08fdd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.155+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/3593819146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fdd240 0x55fd08fdd620 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009230 tx=0x7f8ca8032e40 comp rx=0 tx=0).stop 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.155+0000 7f8cc09cc740 1 -- 172.21.15.31:0/3593819146 shutdown_connections 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.155+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/3593819146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fe8230 0x55fd08fea620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.155+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/3593819146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 0x55fd08fe7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.155+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/3593819146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fdd240 0x55fd08fdd620 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.155+0000 7f8cc09cc740 1 -- 172.21.15.31:0/3593819146 >> 172.21.15.31:0/3593819146 conn(0x55fd08fd9560 msgr2=0x55fd08fd9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.155+0000 7f8cc09cc740 1 -- 172.21.15.31:0/3593819146 shutdown_connections 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.155+0000 7f8cc09cc740 1 -- 172.21.15.31:0/3593819146 wait complete. 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.155+0000 7f8cc09cc740 1 -- start start 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cc09cc740 1 Processor -- start 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fdd240 0x55fd08fe5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fddb60 0x55fd08fe5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fe8230 0x55fd08fe5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fd08fef3d0 con 0x55fd08fdd240 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fd08efdb50 con 0x55fd08fddb60 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fd08fef550 con 0x55fd08fe8230 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cc09cc740 10 client.0 _ll_register_callbacks cb 0x55fd08f38ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cb77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fdd240 0x55fd08fe5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cb77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fdd240 0x55fd08fe5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54078/0 (socket says 172.21.15.31:54078) 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cb77fe640 1 -- 172.21.15.31:0/933665564 learned_addr learned my addr 172.21.15.31:0/933665564 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cb7fff640 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fe8230 0x55fd08fe5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.156+0000 7f8cb6ffd640 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fddb60 0x55fd08fe5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8cb77fe640 1 -- 172.21.15.31:0/933665564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fe8230 msgr2=0x55fd08fe5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8cb77fe640 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fe8230 0x55fd08fe5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8cb77fe640 1 -- 172.21.15.31:0/933665564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fddb60 msgr2=0x55fd08fe5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8cb77fe640 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fddb60 0x55fd08fe5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8cb77fe640 1 -- 172.21.15.31:0/933665564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd08d22620 con 0x55fd08fdd240 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8cb7fff640 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fe8230 0x55fd08fe5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8cb77fe640 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fdd240 0x55fd08fe5140 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80055e0 tx=0x7f8ca8031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6af5a07c9ef161 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8c96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8c96ffd640 1 -- 172.21.15.31:0/933665564 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca804d060 con 0x55fd08fdd240 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8c96ffd640 1 -- 172.21.15.31:0/933665564 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f8ca8031d00 con 0x55fd08fdd240 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8cc09cc740 1 -- 172.21.15.31:0/933665564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ca80316a0 con 0x55fd08fdd240 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8c96ffd640 1 -- 172.21.15.31:0/933665564 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8ca8040070 con 0x55fd08fdd240 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8cc09cc740 10 client.16977 fetch_fsmap learned FSMap version 39 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.157+0000 7f8cc09cc740 1 -- 172.21.15.31:0/933665564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55fd08ffb350 con 0x55fd08fdd240 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.158+0000 7f8c96ffd640 1 -- 172.21.15.31:0/933665564 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ca80316a0 con 0x55fd08fdd240 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.158+0000 7f8c96ffd640 1 -- 172.21.15.31:0/933665564 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 39) ==== 8390+0+0 (secure 0 0 0) 0x7f8ca8003cd0 con 0x55fd08fdd240 2024-04-25T02:05:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.158+0000 7f8cc09cc740 10 client.16977 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T02:05:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.158+0000 7f8cc09cc740 10 client.16977 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.158+0000 7f8cc09cc740 20 client.16977 populate_metadata read hostname 'smithi031' 2024-04-25T02:05:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.158+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd08ffc6a0 0x55fd08ffea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.158+0000 7f8cc09cc740 4 client.16977 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:05:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.158+0000 7f8cc09cc740 1 -- 172.21.15.31:0/933665564 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd08f4f1f0 con 0x55fd08ffc6a0 2024-04-25T02:05:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.158+0000 7f8cb6ffd640 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd08ffc6a0 0x55fd08ffea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.159+0000 7f8cb6ffd640 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd08ffc6a0 0x55fd08ffea90 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6edd40a9443735d server_cookie=279b6105f590c9a9 in_seq=0 out_seq=0 2024-04-25T02:05:03.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.159+0000 7f8c96ffd640 10 client.16977 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:03.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.159+0000 7f8c96ffd640 1 -- 172.21.15.31:0/933665564 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd08f4f1f0 con 0x55fd08ffc6a0 2024-04-25T02:05:03.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.159+0000 7f8c96ffd640 10 client.16977 handle_command_reply: tid=0 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.159+0000 7f8cc09cc740 1 client.16977 shutdown 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.159+0000 7f8cc09cc740 1 -- 172.21.15.31:0/933665564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fdd240 msgr2=0x55fd08fe5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.159+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fdd240 0x55fd08fe5140 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80055e0 tx=0x7f8ca8031400 comp rx=0 tx=0).stop 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.159+0000 7f8cc09cc740 1 -- 172.21.15.31:0/933665564 shutdown_connections 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.159+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd08ffc6a0 0x55fd08ffea90 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.159+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fe8230 0x55fd08fe5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.160+0000 7f8c96ffd640 0 client.16977 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.160+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fddb60 0x55fd08fe5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.160+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/933665564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fdd240 0x55fd08fe5140 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.160+0000 7f8cc09cc740 1 -- 172.21.15.31:0/933665564 >> 172.21.15.31:0/933665564 conn(0x55fd08fd9560 msgr2=0x55fd08fdf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.160+0000 7f8cc09cc740 1 -- 172.21.15.31:0/933665564 shutdown_connections 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.160+0000 7f8cc09cc740 1 -- 172.21.15.31:0/933665564 wait complete. 2024-04-25T02:05:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.160+0000 7f8cc09cc740 20 client.16977 trim_cache size 0 max 16384 2024-04-25T02:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.174+0000 7f8cc09cc740 1 Processor -- start 2024-04-25T02:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.174+0000 7f8cc09cc740 1 -- start start 2024-04-25T02:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.174+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fdd240 0x55fd08fd8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.174+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 0x55fd08ff8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.174+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fe8230 0x55fd08fe4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.174+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb81ab0c0 con 0x55fd08fddb60 2024-04-25T02:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb81acd80 con 0x55fd08fdd240 2024-04-25T02:05:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb81ab790 con 0x55fd08fe8230 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cb6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 0x55fd08ff8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cb6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 0x55fd08ff8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54088/0 (socket says 172.21.15.31:54088) 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cb6ffd640 1 -- 172.21.15.31:0/3190641505 learned_addr learned my addr 172.21.15.31:0/3190641505 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cb77fe640 1 --2- 172.21.15.31:0/3190641505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fdd240 0x55fd08fd8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cb6ffd640 1 -- 172.21.15.31:0/3190641505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fe8230 msgr2=0x55fd08fe4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cb6ffd640 1 --2- 172.21.15.31:0/3190641505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fe8230 0x55fd08fe4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cb6ffd640 1 -- 172.21.15.31:0/3190641505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fdd240 msgr2=0x55fd08fd8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cb6ffd640 1 --2- 172.21.15.31:0/3190641505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fdd240 0x55fd08fd8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cb6ffd640 1 -- 172.21.15.31:0/3190641505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd08fdd950 con 0x55fd08fddb60 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cb77fe640 1 --2- 172.21.15.31:0/3190641505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fdd240 0x55fd08fd8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8cb6ffd640 1 --2- 172.21.15.31:0/3190641505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 0x55fd08ff8290 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8cac005fd0 tx=0x7f8cac01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28948721bc15e11b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8c9d7fa640 1 -- 172.21.15.31:0/3190641505 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac01d040 con 0x55fd08fddb60 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8c9d7fa640 1 -- 172.21.15.31:0/3190641505 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cac09f930 con 0x55fd08fddb60 2024-04-25T02:05:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.175+0000 7f8c9d7fa640 1 -- 172.21.15.31:0/3190641505 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac0999c0 con 0x55fd08fddb60 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.176+0000 7f8cc09cc740 1 -- 172.21.15.31:0/3190641505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 msgr2=0x55fd08ff8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.176+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/3190641505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 0x55fd08ff8290 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8cac005fd0 tx=0x7f8cac01a040 comp rx=0 tx=0).stop 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.176+0000 7f8cc09cc740 1 -- 172.21.15.31:0/3190641505 shutdown_connections 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.176+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/3190641505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fe8230 0x55fd08fe4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.176+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/3190641505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fddb60 0x55fd08ff8290 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.176+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/3190641505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fdd240 0x55fd08fd8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.176+0000 7f8cc09cc740 1 -- 172.21.15.31:0/3190641505 >> 172.21.15.31:0/3190641505 conn(0x55fd08fd9560 msgr2=0x55fd08fdf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.176+0000 7f8cc09cc740 1 -- 172.21.15.31:0/3190641505 shutdown_connections 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.176+0000 7f8cc09cc740 1 -- 172.21.15.31:0/3190641505 wait complete. 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.176+0000 7f8cc09cc740 1 -- start start 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.176+0000 7f8cc09cc740 1 Processor -- start 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fdd240 0x55fd08ff0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fddb60 0x55fd08ff0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cc09cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fe8230 0x55fd08ff0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb81ac6b0 con 0x55fd08fe8230 2024-04-25T02:05:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb81abf20 con 0x55fd08fddb60 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cc09cc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb81a90a0 con 0x55fd08fdd240 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cc09cc740 10 client.0 _ll_register_callbacks cb 0x7f8cac005760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fe8230 0x55fd08ff0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fe8230 0x55fd08ff0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54094/0 (socket says 172.21.15.31:54094) 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fdd240 0x55fd08ff0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb7fff640 1 -- 172.21.15.31:0/1995695228 learned_addr learned my addr 172.21.15.31:0/1995695228 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb6ffd640 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fddb60 0x55fd08ff0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb7fff640 1 -- 172.21.15.31:0/1995695228 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fdd240 msgr2=0x55fd08ff0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb7fff640 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fdd240 0x55fd08ff0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb7fff640 1 -- 172.21.15.31:0/1995695228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fddb60 msgr2=0x55fd08ff0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb7fff640 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fddb60 0x55fd08ff0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb7fff640 1 -- 172.21.15.31:0/1995695228 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd09061a60 con 0x55fd08fe8230 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb6ffd640 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fddb60 0x55fd08ff0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cb7fff640 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fe8230 0x55fd08ff0cd0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0005fd0 tx=0x7f8cb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a7461053351671 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8c96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:05:03.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8c96ffd640 1 -- 172.21.15.31:0/1995695228 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0009920 con 0x55fd08fe8230 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8c96ffd640 1 -- 172.21.15.31:0/1995695228 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f8cb00228f0 con 0x55fd08fe8230 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.177+0000 7f8cc09cc740 1 -- 172.21.15.31:0/1995695228 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cac09f930 con 0x55fd08fe8230 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8c96ffd640 1 -- 172.21.15.31:0/1995695228 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8cb0015270 con 0x55fd08fe8230 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8cc09cc740 10 client.16983 fetch_fsmap learned FSMap version 39 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8cc09cc740 1 -- 172.21.15.31:0/1995695228 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55fd09062ca0 con 0x55fd08fe8230 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8c96ffd640 1 -- 172.21.15.31:0/1995695228 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cac09f930 con 0x55fd08fe8230 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8c96ffd640 1 -- 172.21.15.31:0/1995695228 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 39) ==== 8390+0+0 (secure 0 0 0) 0x7f8cb0029020 con 0x55fd08fe8230 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8cc09cc740 10 client.16983 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8cc09cc740 10 client.16983 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8cc09cc740 20 client.16983 populate_metadata read hostname 'smithi031' 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd09063510 0x55fd09065930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8cc09cc740 4 client.16983 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8cc09cc740 1 -- 172.21.15.31:0/1995695228 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fd08f49540 con 0x55fd09063510 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.178+0000 7f8cb77fe640 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd09063510 0x55fd09065930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.179+0000 7f8c96ffd640 10 client.16983 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.179+0000 7f8cb77fe640 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd09063510 0x55fd09065930 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a12880444f65dab server_cookie=51d451524cfe13e2 in_seq=0 out_seq=0 2024-04-25T02:05:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8c96ffd640 1 -- 172.21.15.31:0/1995695228 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fd08f49540 con 0x55fd09063510 2024-04-25T02:05:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8c96ffd640 10 client.16983 handle_command_reply: tid=0 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 1 client.16983 shutdown 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 1 -- 172.21.15.31:0/1995695228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fe8230 msgr2=0x55fd08ff0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fe8230 0x55fd08ff0cd0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0005fd0 tx=0x7f8cb0004a40 comp rx=0 tx=0).stop 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 1 -- 172.21.15.31:0/1995695228 shutdown_connections 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55fd09063510 0x55fd09065930 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fd08fe8230 0x55fd08ff0cd0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd08fddb60 0x55fd08ff0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8c96ffd640 0 client.16983 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 1 --2- 172.21.15.31:0/1995695228 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd08fdd240 0x55fd08ff0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 1 -- 172.21.15.31:0/1995695228 >> 172.21.15.31:0/1995695228 conn(0x55fd08fd9560 msgr2=0x55fd08fe5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 1 -- 172.21.15.31:0/1995695228 shutdown_connections 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 1 -- 172.21.15.31:0/1995695228 wait complete. 2024-04-25T02:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.208+0000 7f8cc09cc740 20 client.16983 trim_cache size 0 max 16384 2024-04-25T02:05:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:05:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:05:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:05:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2404676226 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f8ca00a2da0 msgr2=0x7f8ca00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f8ca00a2da0 0x7f8ca00a5260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8cb806b050 tx=0x7f8cb0009e60 comp rx=0 tx=0).stop 2024-04-25T02:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2404676226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb8103310 msgr2=0x7f8cb8198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb8103310 0x7f8cb8198380 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00c890 tx=0x7f8cac00cd50 comp rx=0 tx=0).stop 2024-04-25T02:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2404676226 shutdown_connections 2024-04-25T02:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f8ca00a2da0 0x7f8ca00a5260 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cb810bdc0 0x7f8cb819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8cb8103310 0x7f8cb8198380 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 --2- 172.21.15.31:0/2404676226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb81029f0 0x7f8cb8197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2404676226 >> 172.21.15.31:0/2404676226 conn(0x7f8cb80769c0 msgr2=0x7f8cb80fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2404676226 shutdown_connections 2024-04-25T02:05:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:03.209+0000 7f8cbdbe9640 1 -- 172.21.15.31:0/2404676226 wait complete. 2024-04-25T02:05:03.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 227128 2.10 MB/sec execute 430 sec latency 351.518 ms 2024-04-25T02:05:04.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:03 smithi040 ceph-mon[32418]: pgmap v329: 97 pgs: 97 active+clean; 122 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 512 KiB/s wr, 65 op/s 2024-04-25T02:05:04.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:03 smithi134 ceph-mon[33098]: pgmap v329: 97 pgs: 97 active+clean; 122 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 512 KiB/s wr, 65 op/s 2024-04-25T02:05:04.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:03 smithi031 ceph-mon[29204]: pgmap v329: 97 pgs: 97 active+clean; 122 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 512 KiB/s wr, 65 op/s 2024-04-25T02:05:04.323 INFO:tasks.workunit.client.0.smithi031.stdout: 1 227128 2.09 MB/sec execute 431 sec latency 1351.554 ms 2024-04-25T02:05:05.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 227195 2.09 MB/sec execute 432 sec latency 2077.369 ms 2024-04-25T02:05:06.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:05 smithi040 ceph-mon[32418]: pgmap v330: 97 pgs: 97 active+clean; 126 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.0 MiB/s wr, 123 op/s 2024-04-25T02:05:06.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:05 smithi134 ceph-mon[33098]: pgmap v330: 97 pgs: 97 active+clean; 126 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.0 MiB/s wr, 123 op/s 2024-04-25T02:05:06.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:05 smithi031 ceph-mon[29204]: pgmap v330: 97 pgs: 97 active+clean; 126 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.0 MiB/s wr, 123 op/s 2024-04-25T02:05:06.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 227195 2.08 MB/sec execute 433 sec latency 1028.828 ms 2024-04-25T02:05:07.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 227195 2.08 MB/sec execute 434 sec latency 2028.885 ms 2024-04-25T02:05:08.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:07 smithi040 ceph-mon[32418]: pgmap v331: 97 pgs: 97 active+clean; 126 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-04-25T02:05:08.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:07 smithi134 ceph-mon[33098]: pgmap v331: 97 pgs: 97 active+clean; 126 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-04-25T02:05:08.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:07 smithi031 ceph-mon[29204]: pgmap v331: 97 pgs: 97 active+clean; 126 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-04-25T02:05:08.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 227195 2.07 MB/sec execute 435 sec latency 3028.947 ms 2024-04-25T02:05:09.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 227195 2.07 MB/sec execute 436 sec latency 4029.021 ms 2024-04-25T02:05:10.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:09 smithi040 ceph-mon[32418]: pgmap v332: 97 pgs: 97 active+clean; 126 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-04-25T02:05:10.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:09 smithi134 ceph-mon[33098]: pgmap v332: 97 pgs: 97 active+clean; 126 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-04-25T02:05:10.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:09 smithi031 ceph-mon[29204]: pgmap v332: 97 pgs: 97 active+clean; 126 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-04-25T02:05:10.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 227286 2.07 MB/sec execute 437 sec latency 4751.897 ms 2024-04-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 227517 2.06 MB/sec execute 438 sec latency 18.553 ms 2024-04-25T02:05:12.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:11 smithi040 ceph-mon[32418]: pgmap v333: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-04-25T02:05:12.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:11 smithi134 ceph-mon[33098]: pgmap v333: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-04-25T02:05:12.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:11 smithi031 ceph-mon[29204]: pgmap v333: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-04-25T02:05:12.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 227871 2.06 MB/sec execute 439 sec latency 18.031 ms 2024-04-25T02:05:13.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 228336 2.06 MB/sec execute 440 sec latency 15.042 ms 2024-04-25T02:05:14.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:13 smithi040 ceph-mon[32418]: pgmap v334: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 600 KiB/s wr, 68 op/s 2024-04-25T02:05:14.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:13 smithi134 ceph-mon[33098]: pgmap v334: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 600 KiB/s wr, 68 op/s 2024-04-25T02:05:14.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:13 smithi031 ceph-mon[29204]: pgmap v334: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 600 KiB/s wr, 68 op/s 2024-04-25T02:05:14.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 228834 2.06 MB/sec execute 441 sec latency 27.154 ms 2024-04-25T02:05:15.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 229391 2.06 MB/sec execute 442 sec latency 22.806 ms 2024-04-25T02:05:16.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:15 smithi040 ceph-mon[32418]: pgmap v335: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T02:05:16.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:15 smithi134 ceph-mon[33098]: pgmap v335: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T02:05:16.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:15 smithi031 ceph-mon[29204]: pgmap v335: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-25T02:05:16.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.303+0000 7fd4cf031640 1 -- 172.21.15.31:0/1881630325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c8106630 msgr2=0x7fd4c8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:16.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.303+0000 7fd4cf031640 1 --2- 172.21.15.31:0/1881630325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c8106630 0x7fd4c8106a10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0009cd0 tx=0x7fd4c002a8e0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.303+0000 7fd4cf031640 1 -- 172.21.15.31:0/1881630325 shutdown_connections 2024-04-25T02:05:16.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.303+0000 7fd4cf031640 1 --2- 172.21.15.31:0/1881630325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4c8101030 0x7fd4c810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.303+0000 7fd4cf031640 1 --2- 172.21.15.31:0/1881630325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c8100670 0x7fd4c8100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.303+0000 7fd4cf031640 1 --2- 172.21.15.31:0/1881630325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c8106630 0x7fd4c8106a10 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.303+0000 7fd4cf031640 1 -- 172.21.15.31:0/1881630325 >> 172.21.15.31:0/1881630325 conn(0x7fd4c80777c0 msgr2=0x7fd4c8077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:16.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.303+0000 7fd4cf031640 1 -- 172.21.15.31:0/1881630325 shutdown_connections 2024-04-25T02:05:16.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.303+0000 7fd4cf031640 1 -- 172.21.15.31:0/1881630325 wait complete. 2024-04-25T02:05:16.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cf031640 1 Processor -- start 2024-04-25T02:05:16.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cf031640 1 -- start start 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cf031640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c8100670 0x7fd4c8197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cf031640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4c8101030 0x7fd4c81983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cf031640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c8106630 0x7fd4c819c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cf031640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c81119f0 con 0x7fd4c8106630 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cf031640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c8111870 con 0x7fd4c8100670 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cf031640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c8111b70 con 0x7fd4c8101030 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cd5ce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c8106630 0x7fd4c819c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cd5ce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c8106630 0x7fd4c819c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57118/0 (socket says 172.21.15.31:57118) 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cd5ce640 1 -- 172.21.15.31:0/3184525887 learned_addr learned my addr 172.21.15.31:0/3184525887 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c8100670 0x7fd4c8197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4c8101030 0x7fd4c81983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cd5ce640 1 -- 172.21.15.31:0/3184525887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4c8101030 msgr2=0x7fd4c81983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cd5ce640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4c8101030 0x7fd4c81983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cd5ce640 1 -- 172.21.15.31:0/3184525887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c8100670 msgr2=0x7fd4c8197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cd5ce640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c8100670 0x7fd4c8197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4cd5ce640 1 -- 172.21.15.31:0/3184525887 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c819ce80 con 0x7fd4c8106630 2024-04-25T02:05:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.304+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4c8101030 0x7fd4c81983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:05:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.305+0000 7fd4cd5ce640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c8106630 0x7fd4c819c750 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800e970 tx=0x7fd4b800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.305+0000 7fd4c5ffb640 1 -- 172.21.15.31:0/3184525887 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b800ccc0 con 0x7fd4c8106630 2024-04-25T02:05:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.305+0000 7fd4c5ffb640 1 -- 172.21.15.31:0/3184525887 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4b800ce60 con 0x7fd4c8106630 2024-04-25T02:05:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.305+0000 7fd4cf031640 1 -- 172.21.15.31:0/3184525887 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c819d170 con 0x7fd4c8106630 2024-04-25T02:05:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.305+0000 7fd4c5ffb640 1 -- 172.21.15.31:0/3184525887 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8010430 con 0x7fd4c8106630 2024-04-25T02:05:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.305+0000 7fd4cf031640 1 -- 172.21.15.31:0/3184525887 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c81a4a00 con 0x7fd4c8106630 2024-04-25T02:05:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.307+0000 7fd4c5ffb640 1 -- 172.21.15.31:0/3184525887 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd4b8004f90 con 0x7fd4c8106630 2024-04-25T02:05:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.307+0000 7fd4d1e1b740 1 Processor -- start 2024-04-25T02:05:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.307+0000 7fd4d1e1b740 1 -- start start 2024-04-25T02:05:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4ba120 0x560ffe4ba500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4baa40 0x560ffe4c9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4c9640 0x560ffe4cba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560ffe41ecb0 con 0x560ffe4ba120 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560ffe3c75b0 con 0x560ffe4c9640 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560ffe41ac70 con 0x560ffe4baa40 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4c7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4ba120 0x560ffe4ba500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4c7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4ba120 0x560ffe4ba500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57126/0 (socket says 172.21.15.31:57126) 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4c7fff640 1 -- 172.21.15.31:0/1836366711 learned_addr learned my addr 172.21.15.31:0/1836366711 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/1836366711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4baa40 0x560ffe4c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4c5ffb640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4b40a2da0 0x7fd4b40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/1836366711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4c9640 0x560ffe4cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4ccdcd640 1 -- 172.21.15.31:0/1836366711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4c9640 msgr2=0x560ffe4cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/1836366711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4c9640 0x560ffe4cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4cd5ce640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4b40a2da0 0x7fd4b40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4ccdcd640 1 -- 172.21.15.31:0/1836366711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4ba120 msgr2=0x560ffe4ba500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/1836366711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4ba120 0x560ffe4ba500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4ccdcd640 1 -- 172.21.15.31:0/1836366711 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ffe133210 con 0x560ffe4baa40 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/1836366711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4ba120 0x560ffe4ba500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4c5ffb640 1 -- 172.21.15.31:0/3184525887 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7fd4b80cf860 con 0x7fd4c8106630 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/1836366711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4baa40 0x560ffe4c9100 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0033040 tx=0x7fd4c0009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e01dec8718b32c0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4cd5ce640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4b40a2da0 0x7fd4b40a5260 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd4b809bcd0 tx=0x7fd4b806dd00 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.308+0000 7fd4beffd640 1 -- 172.21.15.31:0/1836366711 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0049070 con 0x560ffe4baa40 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.309+0000 7fd4beffd640 1 -- 172.21.15.31:0/1836366711 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4c0036040 con 0x560ffe4baa40 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.309+0000 7fd4beffd640 1 -- 172.21.15.31:0/1836366711 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0004040 con 0x560ffe4baa40 2024-04-25T02:05:16.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.309+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/1836366711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4baa40 msgr2=0x560ffe4c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.309+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/1836366711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4baa40 0x560ffe4c9100 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0033040 tx=0x7fd4c0009360 comp rx=0 tx=0).stop 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.309+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/1836366711 shutdown_connections 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.309+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/1836366711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4c9640 0x560ffe4cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.309+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/1836366711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4baa40 0x560ffe4c9100 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.309+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/1836366711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4ba120 0x560ffe4ba500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.309+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/1836366711 >> 172.21.15.31:0/1836366711 conn(0x560ffe4b6560 msgr2=0x560ffe4b6940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.309+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/1836366711 shutdown_connections 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.309+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/1836366711 wait complete. 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4d1e1b740 1 -- start start 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4d1e1b740 1 Processor -- start 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4ba120 0x560ffe4c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4c9640 0x560ffe4c7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560ffe4cc420 con 0x560ffe4baa40 2024-04-25T02:05:16.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560ffe3dab50 con 0x560ffe4ba120 2024-04-25T02:05:16.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560ffe4cc5a0 con 0x560ffe4c9640 2024-04-25T02:05:16.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4d1e1b740 10 client.0 _ll_register_callbacks cb 0x560ffe415ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:16.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4ccdcd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4c7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4ba120 0x560ffe4c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4ccdcd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57130/0 (socket says 172.21.15.31:57130) 2024-04-25T02:05:16.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4ccdcd640 1 -- 172.21.15.31:0/3913097086 learned_addr learned my addr 172.21.15.31:0/3913097086 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:16.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4c9640 0x560ffe4c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4ccdcd640 1 -- 172.21.15.31:0/3913097086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4c9640 msgr2=0x560ffe4c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:16.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4c9640 0x560ffe4c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.310+0000 7fd4ccdcd640 1 -- 172.21.15.31:0/3913097086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4ba120 msgr2=0x560ffe4c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4ba120 0x560ffe4c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4ccdcd640 1 -- 172.21.15.31:0/3913097086 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ffe1ff620 con 0x560ffe4baa40 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c6b00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00317f0 tx=0x7fd4c0031b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2dfefa18c653e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4a7fff640 1 -- 172.21.15.31:0/3913097086 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0052060 con 0x560ffe4baa40 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4a7fff640 1 -- 172.21.15.31:0/3913097086 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7fd4c0041040 con 0x560ffe4baa40 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/3913097086 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4c0036040 con 0x560ffe4baa40 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4a7fff640 1 -- 172.21.15.31:0/3913097086 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd4c0049070 con 0x560ffe4baa40 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4d1e1b740 10 client.16995 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/3913097086 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x560ffe4cc420 con 0x560ffe4baa40 2024-04-25T02:05:16.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4a7fff640 1 -- 172.21.15.31:0/3913097086 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4c0036040 con 0x560ffe4baa40 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.311+0000 7fd4a7fff640 1 -- 172.21.15.31:0/3913097086 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fd4c0030070 con 0x560ffe4baa40 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.312+0000 7fd4d1e1b740 10 client.16995 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.312+0000 7fd4d1e1b740 10 client.16995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.312+0000 7fd4d1e1b740 20 client.16995 populate_metadata read hostname 'smithi031' 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.312+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560ffe4d97f0 0x560ffe4dbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.312+0000 7fd4d1e1b740 4 client.16995 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.312+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/3913097086 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ffe42c1f0 con 0x560ffe4d97f0 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.312+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560ffe4d97f0 0x560ffe4dbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.312+0000 7fd4a7fff640 10 client.16995 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.312+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560ffe4d97f0 0x560ffe4dbbe0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d67c54131ef972fd server_cookie=d4d9dfa6047002ce in_seq=0 out_seq=0 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4a7fff640 1 -- 172.21.15.31:0/3913097086 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560ffe42c1f0 con 0x560ffe4d97f0 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4a7fff640 10 client.16995 handle_command_reply: tid=0 2024-04-25T02:05:16.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 1 client.16995 shutdown 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/3913097086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 msgr2=0x560ffe4c6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c6b00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00317f0 tx=0x7fd4c0031b00 comp rx=0 tx=0).stop 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/3913097086 shutdown_connections 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560ffe4d97f0 0x560ffe4dbbe0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4a7fff640 0 client.16995 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4c9640 0x560ffe4c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c6b00 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/3913097086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4ba120 0x560ffe4c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/3913097086 >> 172.21.15.31:0/3913097086 conn(0x560ffe4b6560 msgr2=0x560ffe4bc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/3913097086 shutdown_connections 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/3913097086 wait complete. 2024-04-25T02:05:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.313+0000 7fd4d1e1b740 20 client.16995 trim_cache size 0 max 16384 2024-04-25T02:05:16.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 229934 2.06 MB/sec execute 443 sec latency 18.514 ms 2024-04-25T02:05:16.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4d1e1b740 1 Processor -- start 2024-04-25T02:05:16.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4d1e1b740 1 -- start start 2024-04-25T02:05:16.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4ba120 0x560ffe4c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4baa40 0x560ffe4d5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4c9640 0x560ffe4d5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c81ab020 con 0x560ffe4c9640 2024-04-25T02:05:16.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c81acce0 con 0x560ffe4ba120 2024-04-25T02:05:16.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c81ab6f0 con 0x560ffe4baa40 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4c7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4c9640 0x560ffe4d5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4c7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4c9640 0x560ffe4d5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57142/0 (socket says 172.21.15.31:57142) 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4c7fff640 1 -- 172.21.15.31:0/864647937 learned_addr learned my addr 172.21.15.31:0/864647937 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/864647937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4baa40 0x560ffe4d5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/864647937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4ba120 0x560ffe4c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4c7fff640 1 -- 172.21.15.31:0/864647937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4baa40 msgr2=0x560ffe4d5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/864647937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4baa40 0x560ffe4d5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4c7fff640 1 -- 172.21.15.31:0/864647937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4ba120 msgr2=0x560ffe4c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/864647937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4ba120 0x560ffe4c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.330+0000 7fd4c7fff640 1 -- 172.21.15.31:0/864647937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ffe4c4b10 con 0x560ffe4c9640 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.331+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/864647937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4c9640 0x560ffe4d5350 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00127c0 tx=0x7fd4b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1011a3822bc51bb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.331+0000 7fd4be7fc640 1 -- 172.21.15.31:0/864647937 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b001d040 con 0x560ffe4c9640 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.331+0000 7fd4be7fc640 1 -- 172.21.15.31:0/864647937 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4b0025930 con 0x560ffe4c9640 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.331+0000 7fd4be7fc640 1 -- 172.21.15.31:0/864647937 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0013660 con 0x560ffe4c9640 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.331+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/864647937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4c9640 msgr2=0x560ffe4d5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.331+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/864647937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4c9640 0x560ffe4d5350 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00127c0 tx=0x7fd4b00046c0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.331+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/864647937 shutdown_connections 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.331+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/864647937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4c9640 0x560ffe4d5350 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.331+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/864647937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4baa40 0x560ffe4d5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/864647937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4ba120 0x560ffe4c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/864647937 >> 172.21.15.31:0/864647937 conn(0x560ffe4b6560 msgr2=0x560ffe4b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/864647937 shutdown_connections 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/864647937 wait complete. 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 -- start start 2024-04-25T02:05:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 Processor -- start 2024-04-25T02:05:16.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4ba120 0x560ffe4c3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4c9640 0x560ffe4c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c81ac610 con 0x560ffe4baa40 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c81abe80 con 0x560ffe4c9640 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c81a9020 con 0x560ffe4ba120 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.332+0000 7fd4d1e1b740 10 client.0 _ll_register_callbacks cb 0x7fd4c0047070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4ccdcd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4ccdcd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57158/0 (socket says 172.21.15.31:57158) 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4ccdcd640 1 -- 172.21.15.31:0/2007287766 learned_addr learned my addr 172.21.15.31:0/2007287766 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4ba120 0x560ffe4c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4c9640 0x560ffe4c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4ccdcd640 1 -- 172.21.15.31:0/2007287766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4ba120 msgr2=0x560ffe4c3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4ba120 0x560ffe4c3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4ccdcd640 1 -- 172.21.15.31:0/2007287766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4c9640 msgr2=0x560ffe4c3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4c9640 0x560ffe4c3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4ccdcd640 1 -- 172.21.15.31:0/2007287766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ffe53ed00 con 0x560ffe4baa40 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4ccdcd640 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c3680 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x560ffe4c5e20 tx=0x7fd4c0003ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65cf08605f233c6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4a7fff640 1 -- 172.21.15.31:0/2007287766 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0040060 con 0x560ffe4baa40 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4a7fff640 1 -- 172.21.15.31:0/2007287766 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7fd4c0044910 con 0x560ffe4baa40 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/2007287766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4b0025930 con 0x560ffe4baa40 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4a7fff640 1 -- 172.21.15.31:0/2007287766 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd4c0052040 con 0x560ffe4baa40 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4d1e1b740 10 client.17001 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.333+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/2007287766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x560ffe53fed0 con 0x560ffe4baa40 2024-04-25T02:05:16.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.334+0000 7fd4a7fff640 1 -- 172.21.15.31:0/2007287766 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4b0025930 con 0x560ffe4baa40 2024-04-25T02:05:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.334+0000 7fd4a7fff640 1 -- 172.21.15.31:0/2007287766 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fd4c00042e0 con 0x560ffe4baa40 2024-04-25T02:05:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.334+0000 7fd4d1e1b740 10 client.17001 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.334+0000 7fd4d1e1b740 10 client.17001 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.334+0000 7fd4d1e1b740 20 client.17001 populate_metadata read hostname 'smithi031' 2024-04-25T02:05:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.334+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560ffe4d97f0 0x560ffe542720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.334+0000 7fd4d1e1b740 4 client.17001 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:05:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.334+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/2007287766 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560ffe426540 con 0x560ffe4d97f0 2024-04-25T02:05:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.334+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560ffe4d97f0 0x560ffe542720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.334+0000 7fd4c7fff640 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560ffe4d97f0 0x560ffe542720 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=516f28416bbc8190 server_cookie=3307529e7f65b4f5 in_seq=0 out_seq=0 2024-04-25T02:05:16.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.334+0000 7fd4a7fff640 10 client.17001 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:16.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4a7fff640 1 -- 172.21.15.31:0/2007287766 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560ffe426540 con 0x560ffe4d97f0 2024-04-25T02:05:16.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4a7fff640 10 client.17001 handle_command_reply: tid=0 2024-04-25T02:05:16.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4d1e1b740 1 client.17001 shutdown 2024-04-25T02:05:16.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/2007287766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 msgr2=0x560ffe4c3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:16.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c3680 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x560ffe4c5e20 tx=0x7fd4c0003ef0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/2007287766 shutdown_connections 2024-04-25T02:05:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x560ffe4d97f0 0x560ffe542720 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ffe4c9640 0x560ffe4c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ffe4baa40 0x560ffe4c3680 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4a7fff640 0 client.17001 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4d1e1b740 1 --2- 172.21.15.31:0/2007287766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560ffe4ba120 0x560ffe4c3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.361+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/2007287766 >> 172.21.15.31:0/2007287766 conn(0x560ffe4b6560 msgr2=0x560ffe4b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/2007287766 shutdown_connections 2024-04-25T02:05:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4d1e1b740 1 -- 172.21.15.31:0/2007287766 wait complete. 2024-04-25T02:05:16.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4d1e1b740 20 client.17001 trim_cache size 0 max 16384 2024-04-25T02:05:16.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:05:16.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:05:16.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:05:16.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:05:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4cf031640 1 -- 172.21.15.31:0/3184525887 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4b40a2da0 msgr2=0x7fd4b40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4cf031640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4b40a2da0 0x7fd4b40a5260 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd4b809bcd0 tx=0x7fd4b806dd00 comp rx=0 tx=0).stop 2024-04-25T02:05:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4cf031640 1 -- 172.21.15.31:0/3184525887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c8106630 msgr2=0x7fd4c819c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4cf031640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c8106630 0x7fd4c819c750 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800e970 tx=0x7fd4b800ed70 comp rx=0 tx=0).stop 2024-04-25T02:05:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4cf031640 1 -- 172.21.15.31:0/3184525887 shutdown_connections 2024-04-25T02:05:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4cf031640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd4b40a2da0 0x7fd4b40a5260 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4cf031640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4c8106630 0x7fd4c819c750 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4cf031640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4c8101030 0x7fd4c81983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4cf031640 1 --2- 172.21.15.31:0/3184525887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c8100670 0x7fd4c8197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4cf031640 1 -- 172.21.15.31:0/3184525887 >> 172.21.15.31:0/3184525887 conn(0x7fd4c80777c0 msgr2=0x7fd4c8103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:16.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.362+0000 7fd4cf031640 1 -- 172.21.15.31:0/3184525887 shutdown_connections 2024-04-25T02:05:16.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:16.363+0000 7fd4cf031640 1 -- 172.21.15.31:0/3184525887 wait complete. 2024-04-25T02:05:17.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:16 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:05:17.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:16 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:05:17.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:16 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:05:17.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 230385 2.07 MB/sec execute 444 sec latency 19.685 ms 2024-04-25T02:05:18.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:17 smithi040 ceph-mon[32418]: pgmap v336: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 545 KiB/s wr, 67 op/s 2024-04-25T02:05:18.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:17 smithi134 ceph-mon[33098]: pgmap v336: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 545 KiB/s wr, 67 op/s 2024-04-25T02:05:18.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:17 smithi031 ceph-mon[29204]: pgmap v336: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 545 KiB/s wr, 67 op/s 2024-04-25T02:05:18.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 230672 2.07 MB/sec execute 445 sec latency 478.721 ms 2024-04-25T02:05:19.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 230672 2.07 MB/sec execute 446 sec latency 1478.786 ms 2024-04-25T02:05:20.164 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:19 smithi040 ceph-mon[32418]: pgmap v337: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 545 KiB/s wr, 67 op/s 2024-04-25T02:05:20.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:19 smithi134 ceph-mon[33098]: pgmap v337: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 545 KiB/s wr, 67 op/s 2024-04-25T02:05:20.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:19 smithi031 ceph-mon[29204]: pgmap v337: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 545 KiB/s wr, 67 op/s 2024-04-25T02:05:20.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 230739 2.06 MB/sec execute 447 sec latency 2205.196 ms 2024-04-25T02:05:21.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 230739 2.06 MB/sec execute 448 sec latency 1012.077 ms 2024-04-25T02:05:22.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:21 smithi040 ceph-mon[32418]: pgmap v338: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:05:22.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:21 smithi134 ceph-mon[33098]: pgmap v338: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:05:22.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:21 smithi031 ceph-mon[29204]: pgmap v338: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:05:22.324 INFO:tasks.workunit.client.0.smithi031.stdout: 1 230739 2.05 MB/sec execute 449 sec latency 2012.200 ms 2024-04-25T02:05:23.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 230739 2.05 MB/sec execute 450 sec latency 3012.333 ms 2024-04-25T02:05:24.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:23 smithi040 ceph-mon[32418]: pgmap v339: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-04-25T02:05:24.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:23 smithi134 ceph-mon[33098]: pgmap v339: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-04-25T02:05:24.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:23 smithi031 ceph-mon[29204]: pgmap v339: 97 pgs: 97 active+clean; 127 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-04-25T02:05:24.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 230739 2.04 MB/sec execute 451 sec latency 4012.440 ms 2024-04-25T02:05:25.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 230833 2.04 MB/sec execute 452 sec latency 4733.571 ms 2024-04-25T02:05:26.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:25 smithi134 ceph-mon[33098]: pgmap v340: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:05:26.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:25 smithi031 ceph-mon[29204]: pgmap v340: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:05:26.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 231066 2.04 MB/sec execute 453 sec latency 13.971 ms 2024-04-25T02:05:26.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:25 smithi040 ceph-mon[32418]: pgmap v340: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:05:27.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 231429 2.03 MB/sec execute 454 sec latency 20.768 ms 2024-04-25T02:05:28.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:27 smithi134 ceph-mon[33098]: pgmap v341: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 565 KiB/s wr, 50 op/s 2024-04-25T02:05:28.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:27 smithi031 ceph-mon[29204]: pgmap v341: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 565 KiB/s wr, 50 op/s 2024-04-25T02:05:28.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 231844 2.03 MB/sec execute 455 sec latency 19.936 ms 2024-04-25T02:05:28.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:27 smithi040 ceph-mon[32418]: pgmap v341: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 565 KiB/s wr, 50 op/s 2024-04-25T02:05:29.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 232280 2.03 MB/sec execute 456 sec latency 27.697 ms 2024-04-25T02:05:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.456+0000 7f1ead07d640 1 -- 172.21.15.31:0/158970562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8106630 msgr2=0x7f1ea8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.456+0000 7f1ead07d640 1 --2- 172.21.15.31:0/158970562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8106630 0x7f1ea8106a10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1e98009930 tx=0x7f1e9802a420 comp rx=0 tx=0).stop 2024-04-25T02:05:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.456+0000 7f1ead07d640 1 -- 172.21.15.31:0/158970562 shutdown_connections 2024-04-25T02:05:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.456+0000 7f1ead07d640 1 --2- 172.21.15.31:0/158970562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8101030 0x7f1ea810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.456+0000 7f1ead07d640 1 --2- 172.21.15.31:0/158970562 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8100670 0x7f1ea8100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.456+0000 7f1ead07d640 1 --2- 172.21.15.31:0/158970562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8106630 0x7f1ea8106a10 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.456+0000 7f1ead07d640 1 -- 172.21.15.31:0/158970562 >> 172.21.15.31:0/158970562 conn(0x7f1ea80777c0 msgr2=0x7f1ea8077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.456+0000 7f1ead07d640 1 -- 172.21.15.31:0/158970562 shutdown_connections 2024-04-25T02:05:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.456+0000 7f1ead07d640 1 -- 172.21.15.31:0/158970562 wait complete. 2024-04-25T02:05:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.456+0000 7f1ead07d640 1 Processor -- start 2024-04-25T02:05:29.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ead07d640 1 -- start start 2024-04-25T02:05:29.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ead07d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8100670 0x7f1ea8197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ead07d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8101030 0x7f1ea8198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ead07d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8106630 0x7f1ea819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ead07d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81108f0 con 0x7f1ea8100670 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ead07d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea8110770 con 0x7f1ea8106630 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ead07d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea8110a70 con 0x7f1ea8101030 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ea6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8100670 0x7f1ea8197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ea6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8100670 0x7f1ea8197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53034/0 (socket says 172.21.15.31:53034) 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ea6d9d640 1 -- 172.21.15.31:0/2546708539 learned_addr learned my addr 172.21.15.31:0/2546708539 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ea659c640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8101030 0x7f1ea8198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ea759e640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8106630 0x7f1ea819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ea6d9d640 1 -- 172.21.15.31:0/2546708539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8101030 msgr2=0x7f1ea8198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ea6d9d640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8101030 0x7f1ea8198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ea6d9d640 1 -- 172.21.15.31:0/2546708539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8106630 msgr2=0x7f1ea819c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ea6d9d640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8106630 0x7f1ea819c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ea6d9d640 1 -- 172.21.15.31:0/2546708539 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea819ceb0 con 0x7f1ea8100670 2024-04-25T02:05:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.457+0000 7f1ea759e640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8106630 0x7f1ea819c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.458+0000 7f1ea6d9d640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8100670 0x7f1ea8197ec0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1e98005940 tx=0x7f1e980040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.458+0000 7f1e8ffff640 1 -- 172.21.15.31:0/2546708539 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9803f030 con 0x7f1ea8100670 2024-04-25T02:05:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.458+0000 7f1ead07d640 1 -- 172.21.15.31:0/2546708539 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea819d140 con 0x7f1ea8100670 2024-04-25T02:05:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.458+0000 7f1e8ffff640 1 -- 172.21.15.31:0/2546708539 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e980044c0 con 0x7f1ea8100670 2024-04-25T02:05:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.458+0000 7f1e8ffff640 1 -- 172.21.15.31:0/2546708539 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98035070 con 0x7f1ea8100670 2024-04-25T02:05:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.458+0000 7f1ead07d640 1 -- 172.21.15.31:0/2546708539 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea81a4a30 con 0x7f1ea8100670 2024-04-25T02:05:29.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.460+0000 7f1eafd8c740 1 Processor -- start 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.460+0000 7f1e8ffff640 1 -- 172.21.15.31:0/2546708539 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1e98042030 con 0x7f1ea8100670 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.460+0000 7f1eafd8c740 1 -- start start 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835240 0x5596a1835620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835b60 0x5596a183fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1840230 0x5596a1842620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5596a1799cb0 con 0x5596a1835240 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5596a17425b0 con 0x5596a1840230 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5596a1795c70 con 0x5596a1835b60 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835240 0x5596a1835620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea6d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835b60 0x5596a183fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835240 0x5596a1835620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53048/0 (socket says 172.21.15.31:53048) 2024-04-25T02:05:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1840230 0x5596a1842620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea759e640 1 -- 172.21.15.31:0/1896824003 learned_addr learned my addr 172.21.15.31:0/1896824003 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea659c640 1 -- 172.21.15.31:0/1896824003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835b60 msgr2=0x5596a183fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea659c640 1 --2- 172.21.15.31:0/1896824003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835b60 0x5596a183fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea659c640 1 -- 172.21.15.31:0/1896824003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835240 msgr2=0x5596a1835620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea659c640 1 --2- 172.21.15.31:0/1896824003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835240 0x5596a1835620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea659c640 1 -- 172.21.15.31:0/1896824003 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596a14ae210 con 0x5596a1840230 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1e8ffff640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e900a2da0 0x7f1e900a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1e8ffff640 1 -- 172.21.15.31:0/2546708539 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f1e980ef6c0 con 0x7f1ea8100670 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea759e640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e900a2da0 0x7f1e900a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.461+0000 7f1ea659c640 1 --2- 172.21.15.31:0/1896824003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1840230 0x5596a1842620 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009bd0 tx=0x7f1e9c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7401192418306fa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.462+0000 7f1e8d7fa640 1 -- 172.21.15.31:0/1896824003 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c004230 con 0x5596a1840230 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.462+0000 7f1e8d7fa640 1 -- 172.21.15.31:0/1896824003 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e9c009e40 con 0x5596a1840230 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.462+0000 7f1e8d7fa640 1 -- 172.21.15.31:0/1896824003 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c004ca0 con 0x5596a1840230 2024-04-25T02:05:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.462+0000 7f1eafd8c740 1 -- 172.21.15.31:0/1896824003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1840230 msgr2=0x5596a1842620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.462+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/1896824003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1840230 0x5596a1842620 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009bd0 tx=0x7f1e9c011810 comp rx=0 tx=0).stop 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.462+0000 7f1ea759e640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e900a2da0 0x7f1e900a5260 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0009fd0 tx=0x7f1ea0009500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.462+0000 7f1eafd8c740 1 -- 172.21.15.31:0/1896824003 shutdown_connections 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.462+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/1896824003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1840230 0x5596a1842620 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.462+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/1896824003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835b60 0x5596a183fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.462+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/1896824003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835240 0x5596a1835620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.462+0000 7f1eafd8c740 1 -- 172.21.15.31:0/1896824003 >> 172.21.15.31:0/1896824003 conn(0x5596a1831560 msgr2=0x5596a1831940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1eafd8c740 1 -- 172.21.15.31:0/1896824003 shutdown_connections 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1eafd8c740 1 -- 172.21.15.31:0/1896824003 wait complete. 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1eafd8c740 1 -- start start 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1eafd8c740 1 Processor -- start 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835240 0x5596a183d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 0x5596a183d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 0x5596a183dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5596a1843020 con 0x5596a1835b60 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5596a1755b50 con 0x5596a1835240 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5596a18431a0 con 0x5596a1840230 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1eafd8c740 10 client.0 _ll_register_callbacks cb 0x5596a1790ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1ea659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 0x5596a183d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1ea759e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 0x5596a183dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1ea6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835240 0x5596a183d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1ea759e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 0x5596a183dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:34808/0 (socket says 172.21.15.31:34808) 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1ea759e640 1 -- 172.21.15.31:0/3532289380 learned_addr learned my addr 172.21.15.31:0/3532289380 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.463+0000 7f1ea659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 0x5596a183d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53058/0 (socket says 172.21.15.31:53058) 2024-04-25T02:05:29.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1ea6d9d640 1 -- 172.21.15.31:0/3532289380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 msgr2=0x5596a183dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1ea6d9d640 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 0x5596a183dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1ea6d9d640 1 -- 172.21.15.31:0/3532289380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 msgr2=0x5596a183d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1ea6d9d640 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 0x5596a183d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1ea6d9d640 1 -- 172.21.15.31:0/3532289380 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596a157a620 con 0x5596a1835240 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1ea759e640 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 0x5596a183dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1ea6d9d640 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835240 0x5596a183d130 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1e980ece30 tx=0x7f1e98005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de0db34d32b9f12a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1e8aff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1e8aff5640 1 -- 172.21.15.31:0/3532289380 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98049070 con 0x5596a1835240 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1e8aff5640 1 -- 172.21.15.31:0/3532289380 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f1e980b8320 con 0x5596a1835240 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1eafd8c740 1 -- 172.21.15.31:0/3532289380 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5596a17a71f0 con 0x5596a1835240 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1e8aff5640 1 -- 172.21.15.31:0/3532289380 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1e980bba80 con 0x5596a1835240 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1eafd8c740 10 client.26591 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.464+0000 7f1eafd8c740 1 -- 172.21.15.31:0/3532289380 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5596a184f190 con 0x5596a1835240 2024-04-25T02:05:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.465+0000 7f1e8aff5640 1 -- 172.21.15.31:0/3532289380 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5596a17a71f0 con 0x5596a1835240 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.465+0000 7f1e8aff5640 1 -- 172.21.15.31:0/3532289380 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f1e980c43e0 con 0x5596a1835240 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.465+0000 7f1eafd8c740 10 client.26591 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.465+0000 7f1eafd8c740 10 client.26591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.465+0000 7f1eafd8c740 20 client.26591 populate_metadata read hostname 'smithi031' 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.465+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5596a18507c0 0x5596a1852c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.465+0000 7f1eafd8c740 4 client.26591 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.465+0000 7f1eafd8c740 1 -- 172.21.15.31:0/3532289380 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596a17a1540 con 0x5596a18507c0 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.465+0000 7f1ea659c640 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5596a18507c0 0x5596a1852c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.465+0000 7f1e8aff5640 10 client.26591 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.465+0000 7f1ea659c640 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5596a18507c0 0x5596a1852c80 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e02a1e97d9d8165d server_cookie=334b6a8a6b526373 in_seq=0 out_seq=0 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1e8aff5640 1 -- 172.21.15.31:0/3532289380 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596a17a1540 con 0x5596a18507c0 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1e8aff5640 10 client.26591 handle_command_reply: tid=0 2024-04-25T02:05:29.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 1 client.26591 shutdown 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 1 -- 172.21.15.31:0/3532289380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835240 msgr2=0x5596a183d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835240 0x5596a183d130 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1e980ece30 tx=0x7f1e98005c90 comp rx=0 tx=0).stop 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 1 -- 172.21.15.31:0/3532289380 shutdown_connections 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5596a18507c0 0x5596a1852c80 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 0x5596a183dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1e8aff5640 0 client.26591 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 0x5596a183d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/3532289380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835240 0x5596a183d130 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 1 -- 172.21.15.31:0/3532289380 >> 172.21.15.31:0/3532289380 conn(0x5596a1831560 msgr2=0x5596a1837360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 1 -- 172.21.15.31:0/3532289380 shutdown_connections 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 1 -- 172.21.15.31:0/3532289380 wait complete. 2024-04-25T02:05:29.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.466+0000 7f1eafd8c740 20 client.26591 trim_cache size 0 max 16384 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.481+0000 7f1eafd8c740 1 Processor -- start 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.481+0000 7f1eafd8c740 1 -- start start 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835240 0x5596a184bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 0x5596a183f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 0x5596a1858550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81ad210 con 0x5596a1835b60 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81aeed0 con 0x5596a1835240 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81ad8e0 con 0x5596a1840230 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 0x5596a183f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 0x5596a183f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53062/0 (socket says 172.21.15.31:53062) 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835240 0x5596a184bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea659c640 1 -- 172.21.15.31:0/2133682551 learned_addr learned my addr 172.21.15.31:0/2133682551 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:29.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea759e640 1 --2- 172.21.15.31:0/2133682551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 0x5596a1858550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea659c640 1 -- 172.21.15.31:0/2133682551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 msgr2=0x5596a1858550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:29.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea659c640 1 --2- 172.21.15.31:0/2133682551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 0x5596a1858550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea659c640 1 -- 172.21.15.31:0/2133682551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835240 msgr2=0x5596a184bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:29.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea659c640 1 --2- 172.21.15.31:0/2133682551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835240 0x5596a184bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea659c640 1 -- 172.21.15.31:0/2133682551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596a183b680 con 0x5596a1835b60 2024-04-25T02:05:29.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea759e640 1 --2- 172.21.15.31:0/2133682551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 0x5596a1858550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:05:29.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1ea659c640 1 --2- 172.21.15.31:0/2133682551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 0x5596a183f7f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009710 tx=0x7f1e9c01b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7c4e3021f2d203 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:29.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.482+0000 7f1e8cff9640 1 -- 172.21.15.31:0/2133682551 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c01e040 con 0x5596a1835b60 2024-04-25T02:05:29.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.483+0000 7f1e8cff9640 1 -- 172.21.15.31:0/2133682551 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e9c02a070 con 0x5596a1835b60 2024-04-25T02:05:29.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.483+0000 7f1e8cff9640 1 -- 172.21.15.31:0/2133682551 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c004750 con 0x5596a1835b60 2024-04-25T02:05:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.483+0000 7f1eafd8c740 1 -- 172.21.15.31:0/2133682551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 msgr2=0x5596a183f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.483+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/2133682551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 0x5596a183f7f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009710 tx=0x7f1e9c01b040 comp rx=0 tx=0).stop 2024-04-25T02:05:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.483+0000 7f1eafd8c740 1 -- 172.21.15.31:0/2133682551 shutdown_connections 2024-04-25T02:05:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.483+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/2133682551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1840230 0x5596a1858550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.483+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/2133682551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1835b60 0x5596a183f7f0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.483+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/2133682551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835240 0x5596a184bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.483+0000 7f1eafd8c740 1 -- 172.21.15.31:0/2133682551 >> 172.21.15.31:0/2133682551 conn(0x5596a1831560 msgr2=0x5596a18305c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.483+0000 7f1eafd8c740 1 -- 172.21.15.31:0/2133682551 shutdown_connections 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.483+0000 7f1eafd8c740 1 -- 172.21.15.31:0/2133682551 wait complete. 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1eafd8c740 1 -- start start 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1eafd8c740 1 Processor -- start 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835240 0x5596a18483f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835b60 0x5596a1848930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1eafd8c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1840230 0x5596a1839920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81ae800 con 0x5596a1840230 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81ae070 con 0x5596a1835b60 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1eafd8c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81ab210 con 0x5596a1835240 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1eafd8c740 10 client.0 _ll_register_callbacks cb 0x7f1e980f5240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1ea759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1840230 0x5596a1839920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1ea6d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835240 0x5596a18483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1ea6d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835240 0x5596a18483f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:34832/0 (socket says 172.21.15.31:34832) 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1ea6d9d640 1 -- 172.21.15.31:0/211778125 learned_addr learned my addr 172.21.15.31:0/211778125 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1ea659c640 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835b60 0x5596a1848930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1ea759e640 1 -- 172.21.15.31:0/211778125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835240 msgr2=0x5596a18483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1ea759e640 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835240 0x5596a18483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1ea759e640 1 -- 172.21.15.31:0/211778125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835b60 msgr2=0x5596a1848930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1ea759e640 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835b60 0x5596a1848930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1ea759e640 1 -- 172.21.15.31:0/211778125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596a183a150 con 0x5596a1840230 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.484+0000 7f1ea6d9d640 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835240 0x5596a18483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1ea759e640 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1840230 0x5596a1839920 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x5596a1843240 tx=0x7f1ea0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4ab7b05f9c802 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1e8aff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1e8aff5640 1 -- 172.21.15.31:0/211778125 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea002e060 con 0x5596a1840230 2024-04-25T02:05:29.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1eafd8c740 1 -- 172.21.15.31:0/211778125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e9c02a070 con 0x5596a1840230 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1e8aff5640 1 -- 172.21.15.31:0/211778125 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f1ea001f630 con 0x5596a1840230 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1e8aff5640 1 -- 172.21.15.31:0/211778125 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1ea0011e20 con 0x5596a1840230 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1eafd8c740 10 client.17019 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1eafd8c740 1 -- 172.21.15.31:0/211778125 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5596a18b6950 con 0x5596a1840230 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1e8aff5640 1 -- 172.21.15.31:0/211778125 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e9c02a070 con 0x5596a1840230 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1e8aff5640 1 -- 172.21.15.31:0/211778125 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f1ea0028070 con 0x5596a1840230 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1eafd8c740 10 client.17019 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1eafd8c740 10 client.17019 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1eafd8c740 20 client.17019 populate_metadata read hostname 'smithi031' 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5596a18b71c0 0x5596a18b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1eafd8c740 4 client.17019 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.485+0000 7f1eafd8c740 1 -- 172.21.15.31:0/211778125 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5596a171f8e0 con 0x5596a18b71c0 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.486+0000 7f1ea6d9d640 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5596a18b71c0 0x5596a18b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.486+0000 7f1ea6d9d640 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5596a18b71c0 0x5596a18b95e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26a8a8927b045ccd server_cookie=c6fdb538d915a86f in_seq=0 out_seq=0 2024-04-25T02:05:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.486+0000 7f1e8aff5640 10 client.17019 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1e8aff5640 1 -- 172.21.15.31:0/211778125 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5596a171f8e0 con 0x5596a18b71c0 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1e8aff5640 10 client.17019 handle_command_reply: tid=0 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 1 client.17019 shutdown 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 1 -- 172.21.15.31:0/211778125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1840230 msgr2=0x5596a1839920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1840230 0x5596a1839920 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x5596a1843240 tx=0x7f1ea0004ab0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 1 -- 172.21.15.31:0/211778125 shutdown_connections 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5596a18b71c0 0x5596a18b95e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596a1840230 0x5596a1839920 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5596a1835b60 0x5596a1848930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1e8aff5640 0 client.17019 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 1 --2- 172.21.15.31:0/211778125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5596a1835240 0x5596a18483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 1 -- 172.21.15.31:0/211778125 >> 172.21.15.31:0/211778125 conn(0x5596a1831560 msgr2=0x5596a183d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 1 -- 172.21.15.31:0/211778125 shutdown_connections 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 1 -- 172.21.15.31:0/211778125 wait complete. 2024-04-25T02:05:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.510+0000 7f1eafd8c740 20 client.17019 trim_cache size 0 max 16384 2024-04-25T02:05:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:05:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:05:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:05:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 -- 172.21.15.31:0/2546708539 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e900a2da0 msgr2=0x7f1e900a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e900a2da0 0x7f1e900a5260 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0009fd0 tx=0x7f1ea0009500 comp rx=0 tx=0).stop 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 -- 172.21.15.31:0/2546708539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8100670 msgr2=0x7f1ea8197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8100670 0x7f1ea8197ec0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1e98005940 tx=0x7f1e980040c0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 -- 172.21.15.31:0/2546708539 shutdown_connections 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e900a2da0 0x7f1e900a5260 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8106630 0x7f1ea819c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8101030 0x7f1ea8198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 --2- 172.21.15.31:0/2546708539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8100670 0x7f1ea8197ec0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 -- 172.21.15.31:0/2546708539 >> 172.21.15.31:0/2546708539 conn(0x7f1ea80777c0 msgr2=0x7f1ea8069190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 -- 172.21.15.31:0/2546708539 shutdown_connections 2024-04-25T02:05:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:29.511+0000 7f1ead07d640 1 -- 172.21.15.31:0/2546708539 wait complete. 2024-04-25T02:05:30.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:29 smithi134 ceph-mon[33098]: pgmap v342: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 565 KiB/s wr, 50 op/s 2024-04-25T02:05:30.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:29 smithi031 ceph-mon[29204]: pgmap v342: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 565 KiB/s wr, 50 op/s 2024-04-25T02:05:30.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 232838 2.03 MB/sec execute 457 sec latency 23.959 ms 2024-04-25T02:05:30.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:29 smithi040 ceph-mon[32418]: pgmap v342: 97 pgs: 97 active+clean; 128 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 565 KiB/s wr, 50 op/s 2024-04-25T02:05:31.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 233322 2.04 MB/sec execute 458 sec latency 38.127 ms 2024-04-25T02:05:32.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:31 smithi134 ceph-mon[33098]: pgmap v343: 97 pgs: 97 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.1 MiB/s wr, 111 op/s 2024-04-25T02:05:32.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:31 smithi031 ceph-mon[29204]: pgmap v343: 97 pgs: 97 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.1 MiB/s wr, 111 op/s 2024-04-25T02:05:32.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 233921 2.04 MB/sec execute 459 sec latency 18.945 ms 2024-04-25T02:05:32.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:31 smithi040 ceph-mon[32418]: pgmap v343: 97 pgs: 97 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.1 MiB/s wr, 111 op/s 2024-04-25T02:05:33.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234220 2.04 MB/sec execute 460 sec latency 290.793 ms 2024-04-25T02:05:34.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:33 smithi134 ceph-mon[33098]: pgmap v344: 97 pgs: 97 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 558 KiB/s wr, 70 op/s 2024-04-25T02:05:34.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:33 smithi031 ceph-mon[29204]: pgmap v344: 97 pgs: 97 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 558 KiB/s wr, 70 op/s 2024-04-25T02:05:34.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234220 2.04 MB/sec execute 461 sec latency 1290.892 ms 2024-04-25T02:05:34.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:33 smithi040 ceph-mon[32418]: pgmap v344: 97 pgs: 97 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 558 KiB/s wr, 70 op/s 2024-04-25T02:05:35.325 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234287 2.04 MB/sec execute 462 sec latency 2010.618 ms 2024-04-25T02:05:36.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:35 smithi134 ceph-mon[33098]: pgmap v345: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-04-25T02:05:36.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:35 smithi031 ceph-mon[29204]: pgmap v345: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-04-25T02:05:36.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234287 2.03 MB/sec execute 463 sec latency 1023.492 ms 2024-04-25T02:05:36.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:35 smithi040 ceph-mon[32418]: pgmap v345: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-04-25T02:05:37.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234287 2.03 MB/sec execute 464 sec latency 2023.605 ms 2024-04-25T02:05:38.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:37 smithi134 ceph-mon[33098]: pgmap v346: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-25T02:05:38.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:37 smithi031 ceph-mon[29204]: pgmap v346: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-25T02:05:38.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234287 2.02 MB/sec execute 465 sec latency 3023.731 ms 2024-04-25T02:05:38.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:37 smithi040 ceph-mon[32418]: pgmap v346: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-25T02:05:39.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234287 2.02 MB/sec execute 466 sec latency 4023.817 ms 2024-04-25T02:05:40.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:39 smithi134 ceph-mon[33098]: pgmap v347: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-25T02:05:40.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:39 smithi031 ceph-mon[29204]: pgmap v347: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-25T02:05:40.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234299 2.01 MB/sec execute 467 sec latency 4741.851 ms 2024-04-25T02:05:40.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:39 smithi040 ceph-mon[32418]: pgmap v347: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-25T02:05:41.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234299 2.01 MB/sec execute 468 sec latency 1222.866 ms 2024-04-25T02:05:42.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:41 smithi134 ceph-mon[33098]: pgmap v348: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T02:05:42.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:41 smithi031 ceph-mon[29204]: pgmap v348: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T02:05:42.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234556 2.01 MB/sec execute 469 sec latency 1233.915 ms 2024-04-25T02:05:42.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:41 smithi040 ceph-mon[32418]: pgmap v348: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T02:05:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.607+0000 7f66b3ff9640 1 -- 172.21.15.31:0/2096556920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac101030 msgr2=0x7f66ac10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.607+0000 7f66b3ff9640 1 --2- 172.21.15.31:0/2096556920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac101030 0x7f66ac10d4f0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009930 tx=0x7f66a402a420 comp rx=0 tx=0).stop 2024-04-25T02:05:42.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.608+0000 7f66b3ff9640 1 -- 172.21.15.31:0/2096556920 shutdown_connections 2024-04-25T02:05:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.608+0000 7f66b3ff9640 1 --2- 172.21.15.31:0/2096556920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac101030 0x7f66ac10d4f0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.608+0000 7f66b3ff9640 1 --2- 172.21.15.31:0/2096556920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66ac100670 0x7f66ac100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.608+0000 7f66b3ff9640 1 --2- 172.21.15.31:0/2096556920 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac106630 0x7f66ac106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.608+0000 7f66b3ff9640 1 -- 172.21.15.31:0/2096556920 >> 172.21.15.31:0/2096556920 conn(0x7f66ac0777c0 msgr2=0x7f66ac077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.608+0000 7f66b3ff9640 1 -- 172.21.15.31:0/2096556920 shutdown_connections 2024-04-25T02:05:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.608+0000 7f66b3ff9640 1 -- 172.21.15.31:0/2096556920 wait complete. 2024-04-25T02:05:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.608+0000 7f66b3ff9640 1 Processor -- start 2024-04-25T02:05:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.608+0000 7f66b3ff9640 1 -- start start 2024-04-25T02:05:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.608+0000 7f66b3ff9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac100670 0x7f66ac196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b3ff9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac101030 0x7f66ac1972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b3ff9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66ac106630 0x7f66ac19b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b3ff9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1108f0 con 0x7f66ac100670 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b3ff9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac110770 con 0x7f66ac106630 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b3ff9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac110a70 con 0x7f66ac101030 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b1d95640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac100670 0x7f66ac196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b1d95640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac100670 0x7f66ac196d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50944/0 (socket says 172.21.15.31:50944) 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b1d95640 1 -- 172.21.15.31:0/1323699506 learned_addr learned my addr 172.21.15.31:0/1323699506 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b2596640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66ac106630 0x7f66ac19b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b1594640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac101030 0x7f66ac1972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b1d95640 1 -- 172.21.15.31:0/1323699506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac101030 msgr2=0x7f66ac1972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b1d95640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac101030 0x7f66ac1972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b1d95640 1 -- 172.21.15.31:0/1323699506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66ac106630 msgr2=0x7f66ac19b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b1d95640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66ac106630 0x7f66ac19b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b1d95640 1 -- 172.21.15.31:0/1323699506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ac19bd80 con 0x7f66ac100670 2024-04-25T02:05:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b1d95640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac100670 0x7f66ac196d90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f66a800eb90 tx=0x7f66a800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66a2ffd640 1 -- 172.21.15.31:0/1323699506 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8004d20 con 0x7f66ac100670 2024-04-25T02:05:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66a2ffd640 1 -- 172.21.15.31:0/1323699506 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a800ee90 con 0x7f66ac100670 2024-04-25T02:05:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b3ff9640 1 -- 172.21.15.31:0/1323699506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66ac19c070 con 0x7f66ac100670 2024-04-25T02:05:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66a2ffd640 1 -- 172.21.15.31:0/1323699506 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8010670 con 0x7f66ac100670 2024-04-25T02:05:42.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.609+0000 7f66b3ff9640 1 -- 172.21.15.31:0/1323699506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66ac1a3900 con 0x7f66ac100670 2024-04-25T02:05:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.612+0000 7f66a2ffd640 1 -- 172.21.15.31:0/1323699506 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f66a8005020 con 0x7f66ac100670 2024-04-25T02:05:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.612+0000 7f66b6de9740 1 Processor -- start 2024-04-25T02:05:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.612+0000 7f66b6de9740 1 -- start start 2024-04-25T02:05:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa297620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa297b60 0x55c0aa2a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa2a2230 0x55c0aa2a4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b6de9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa1fbcb0 con 0x55c0aa297240 2024-04-25T02:05:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b6de9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa1a45b0 con 0x55c0aa297b60 2024-04-25T02:05:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b6de9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa1f7c70 con 0x55c0aa2a2230 2024-04-25T02:05:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66a2ffd640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66980a2e20 0x7f66980a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b2596640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa297620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b2596640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa297620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50952/0 (socket says 172.21.15.31:50952) 2024-04-25T02:05:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b2596640 1 -- 172.21.15.31:0/112922851 learned_addr learned my addr 172.21.15.31:0/112922851 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b1d95640 1 --2- 172.21.15.31:0/112922851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa297b60 0x55c0aa2a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b2596640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66980a2e20 0x7f66980a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b1594640 1 --2- 172.21.15.31:0/112922851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa2a2230 0x55c0aa2a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66a2ffd640 1 -- 172.21.15.31:0/1323699506 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f66a80cf990 con 0x7f66ac100670 2024-04-25T02:05:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b2596640 1 -- 172.21.15.31:0/112922851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa2a2230 msgr2=0x55c0aa2a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b2596640 1 --2- 172.21.15.31:0/112922851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa2a2230 0x55c0aa2a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b2596640 1 -- 172.21.15.31:0/112922851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa297b60 msgr2=0x55c0aa2a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b2596640 1 --2- 172.21.15.31:0/112922851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa297b60 0x55c0aa2a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b2596640 1 -- 172.21.15.31:0/112922851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0a9f10210 con 0x55c0aa297240 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.613+0000 7f66b1d95640 1 --2- 172.21.15.31:0/112922851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa297b60 0x55c0aa2a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.614+0000 7f66b2596640 1 --2- 172.21.15.31:0/112922851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa297620 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f66a4005c40 tx=0x7f66a40318c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05a37e344b0a669 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.614+0000 7f66b2596640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66980a2e20 0x7f66980a52e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f66a40096c0 tx=0x7f66a4035000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.614+0000 7f6697fff640 1 -- 172.21.15.31:0/112922851 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4035070 con 0x55c0aa297240 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.614+0000 7f6697fff640 1 -- 172.21.15.31:0/112922851 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a40046c0 con 0x55c0aa297240 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.614+0000 7f6697fff640 1 -- 172.21.15.31:0/112922851 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4003910 con 0x55c0aa297240 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.615+0000 7f66b6de9740 1 -- 172.21.15.31:0/112922851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 msgr2=0x55c0aa297620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.615+0000 7f66b6de9740 1 --2- 172.21.15.31:0/112922851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa297620 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f66a4005c40 tx=0x7f66a40318c0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.615+0000 7f66b6de9740 1 -- 172.21.15.31:0/112922851 shutdown_connections 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.615+0000 7f66b6de9740 1 --2- 172.21.15.31:0/112922851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa2a2230 0x55c0aa2a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.615+0000 7f66b6de9740 1 --2- 172.21.15.31:0/112922851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa297b60 0x55c0aa2a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.615+0000 7f66b6de9740 1 --2- 172.21.15.31:0/112922851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa297620 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.615+0000 7f66b6de9740 1 -- 172.21.15.31:0/112922851 >> 172.21.15.31:0/112922851 conn(0x55c0aa293560 msgr2=0x55c0aa293940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.615+0000 7f66b6de9740 1 -- 172.21.15.31:0/112922851 shutdown_connections 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.615+0000 7f66b6de9740 1 -- 172.21.15.31:0/112922851 wait complete. 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.615+0000 7f66b6de9740 1 -- start start 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.615+0000 7f66b6de9740 1 Processor -- start 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa29f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa29fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b6de9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa2a4fc0 con 0x55c0aa297240 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b6de9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa1b7b50 con 0x55c0aa2a2230 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b6de9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c0aa2a5140 con 0x55c0aa297b60 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b6de9740 10 client.0 _ll_register_callbacks cb 0x55c0aa1f2ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b1d95640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa29f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b1594640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b1d95640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa29f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50968/0 (socket says 172.21.15.31:50968) 2024-04-25T02:05:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b2596640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa29fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b1d95640 1 -- 172.21.15.31:0/3654643703 learned_addr learned my addr 172.21.15.31:0/3654643703 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b1594640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:54440/0 (socket says 172.21.15.31:54440) 2024-04-25T02:05:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b1d95640 1 -- 172.21.15.31:0/3654643703 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 msgr2=0x55c0aa29f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b1d95640 1 --2- 172.21.15.31:0/3654643703 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b1d95640 1 -- 172.21.15.31:0/3654643703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 msgr2=0x55c0aa29fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b1d95640 1 --2- 172.21.15.31:0/3654643703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa29fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b1d95640 1 -- 172.21.15.31:0/3654643703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0a9fdc620 con 0x55c0aa297240 2024-04-25T02:05:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b1d95640 1 --2- 172.21.15.31:0/3654643703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa29f1b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f66a80058e0 tx=0x7f66a8091360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d70b7cef0cb06d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66957fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66957fa640 1 -- 172.21.15.31:0/3654643703 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a809c5d0 con 0x55c0aa297240 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66957fa640 1 -- 172.21.15.31:0/3654643703 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f66a8005560 con 0x55c0aa297240 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.616+0000 7f66b6de9740 1 -- 172.21.15.31:0/3654643703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a40046c0 con 0x55c0aa297240 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66957fa640 1 -- 172.21.15.31:0/3654643703 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f66a80a4b40 con 0x55c0aa297240 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66b6de9740 10 client.17031 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66b6de9740 1 -- 172.21.15.31:0/3654643703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c0aa2b1200 con 0x55c0aa297240 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66957fa640 1 -- 172.21.15.31:0/3654643703 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a40046c0 con 0x55c0aa297240 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66957fa640 1 -- 172.21.15.31:0/3654643703 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f66a80cd3f0 con 0x55c0aa297240 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66b6de9740 10 client.17031 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66b6de9740 10 client.17031 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66b6de9740 20 client.17031 populate_metadata read hostname 'smithi031' 2024-04-25T02:05:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3654643703 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c0aa2b2760 0x55c0aa2b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66b6de9740 4 client.17031 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66b6de9740 1 -- 172.21.15.31:0/3654643703 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0aa2091f0 con 0x55c0aa2b2760 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.617+0000 7f66b1594640 1 --2- 172.21.15.31:0/3654643703 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c0aa2b2760 0x55c0aa2b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.618+0000 7f66957fa640 10 client.17031 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.618+0000 7f66b1594640 1 --2- 172.21.15.31:0/3654643703 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c0aa2b2760 0x55c0aa2b4be0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cc6d9029288a9a2 server_cookie=1826467f9123f9c7 in_seq=0 out_seq=0 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.618+0000 7f66957fa640 1 -- 172.21.15.31:0/3654643703 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0aa2091f0 con 0x55c0aa2b2760 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.618+0000 7f66957fa640 10 client.17031 handle_command_reply: tid=0 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.618+0000 7f66b6de9740 1 client.17031 shutdown 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66b6de9740 1 -- 172.21.15.31:0/3654643703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 msgr2=0x55c0aa29f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3654643703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa29f1b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f66a80058e0 tx=0x7f66a8091360 comp rx=0 tx=0).stop 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66b6de9740 1 -- 172.21.15.31:0/3654643703 shutdown_connections 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3654643703 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c0aa2b2760 0x55c0aa2b4be0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3654643703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa29fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3654643703 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3654643703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa29f1b0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66957fa640 0 client.17031 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66b6de9740 1 -- 172.21.15.31:0/3654643703 >> 172.21.15.31:0/3654643703 conn(0x55c0aa293560 msgr2=0x55c0aa299240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66b6de9740 1 -- 172.21.15.31:0/3654643703 shutdown_connections 2024-04-25T02:05:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66b6de9740 1 -- 172.21.15.31:0/3654643703 wait complete. 2024-04-25T02:05:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.619+0000 7f66b6de9740 20 client.17031 trim_cache size 0 max 16384 2024-04-25T02:05:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b6de9740 1 Processor -- start 2024-04-25T02:05:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b6de9740 1 -- start start 2024-04-25T02:05:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2ade80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa292440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b6de9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1ab130 con 0x55c0aa297240 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b6de9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1acdf0 con 0x55c0aa2a2230 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b6de9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1ab800 con 0x55c0aa297b60 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b1d95640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b1d95640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2ade80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32818/0 (socket says 172.21.15.31:32818) 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b1d95640 1 -- 172.21.15.31:0/2437157312 learned_addr learned my addr 172.21.15.31:0/2437157312 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b2596640 1 --2- 172.21.15.31:0/2437157312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa292440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b1594640 1 --2- 172.21.15.31:0/2437157312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b1d95640 1 -- 172.21.15.31:0/2437157312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 msgr2=0x55c0aa29e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b1d95640 1 --2- 172.21.15.31:0/2437157312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b1d95640 1 -- 172.21.15.31:0/2437157312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 msgr2=0x55c0aa292440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b1d95640 1 --2- 172.21.15.31:0/2437157312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa292440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b1d95640 1 -- 172.21.15.31:0/2437157312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0aa2b5260 con 0x55c0aa297240 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.635+0000 7f66b2596640 1 --2- 172.21.15.31:0/2437157312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa292440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.636+0000 7f66b1d95640 1 --2- 172.21.15.31:0/2437157312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2ade80 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f66a809be60 tx=0x7f66a8009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cb9c489fd7af2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.636+0000 7f66977fe640 1 -- 172.21.15.31:0/2437157312 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a809bed0 con 0x55c0aa297240 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.636+0000 7f66977fe640 1 -- 172.21.15.31:0/2437157312 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a80a0280 con 0x55c0aa297240 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.636+0000 7f66977fe640 1 -- 172.21.15.31:0/2437157312 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a80a46b0 con 0x55c0aa297240 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.636+0000 7f66b6de9740 1 -- 172.21.15.31:0/2437157312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 msgr2=0x55c0aa2ade80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.636+0000 7f66b6de9740 1 --2- 172.21.15.31:0/2437157312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2ade80 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f66a809be60 tx=0x7f66a8009a40 comp rx=0 tx=0).stop 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 -- 172.21.15.31:0/2437157312 shutdown_connections 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 --2- 172.21.15.31:0/2437157312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa292440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 --2- 172.21.15.31:0/2437157312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 --2- 172.21.15.31:0/2437157312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2ade80 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 -- 172.21.15.31:0/2437157312 >> 172.21.15.31:0/2437157312 conn(0x55c0aa293560 msgr2=0x55c0aa291d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 -- 172.21.15.31:0/2437157312 shutdown_connections 2024-04-25T02:05:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 -- 172.21.15.31:0/2437157312 wait complete. 2024-04-25T02:05:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 -- start start 2024-04-25T02:05:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 Processor -- start 2024-04-25T02:05:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2aa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa29c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1ac720 con 0x55c0aa297240 2024-04-25T02:05:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1abf90 con 0x55c0aa2a2230 2024-04-25T02:05:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1a9130 con 0x55c0aa297b60 2024-04-25T02:05:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.637+0000 7f66b6de9740 10 client.0 _ll_register_callbacks cb 0x7f66a8007eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b1d95640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b1d95640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2aa740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32820/0 (socket says 172.21.15.31:32820) 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b1d95640 1 -- 172.21.15.31:0/3140760294 learned_addr learned my addr 172.21.15.31:0/3140760294 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b2596640 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa29c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b1594640 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b1d95640 1 -- 172.21.15.31:0/3140760294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 msgr2=0x55c0aa29bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b1d95640 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b1d95640 1 -- 172.21.15.31:0/3140760294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 msgr2=0x55c0aa29c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b1d95640 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa29c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b1d95640 1 -- 172.21.15.31:0/3140760294 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0aa29c8c0 con 0x55c0aa297240 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b2596640 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa29c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b1d95640 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2aa740 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f66a800cba0 tx=0x7f66a80093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2424ef93d9466382 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66957fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:05:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66957fa640 1 -- 172.21.15.31:0/3140760294 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a809bed0 con 0x55c0aa297240 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66957fa640 1 -- 172.21.15.31:0/3140760294 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f66a80a46b0 con 0x55c0aa297240 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b6de9740 1 -- 172.21.15.31:0/3140760294 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a80a0280 con 0x55c0aa297240 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66957fa640 1 -- 172.21.15.31:0/3140760294 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f66a80a48b0 con 0x55c0aa297240 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b6de9740 10 client.17037 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.638+0000 7f66b6de9740 1 -- 172.21.15.31:0/3140760294 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c0aa318d90 con 0x55c0aa297240 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.639+0000 7f66957fa640 1 -- 172.21.15.31:0/3140760294 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a80a0280 con 0x55c0aa297240 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.639+0000 7f66957fa640 1 -- 172.21.15.31:0/3140760294 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f66a8097c20 con 0x55c0aa297240 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.639+0000 7f66b6de9740 10 client.17037 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.639+0000 7f66b6de9740 10 client.17037 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.639+0000 7f66b6de9740 20 client.17037 populate_metadata read hostname 'smithi031' 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.639+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c0aa319200 0x55c0aa31b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.639+0000 7f66b6de9740 4 client.17037 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:05:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.639+0000 7f66b6de9740 1 -- 172.21.15.31:0/3140760294 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c0aa203540 con 0x55c0aa319200 2024-04-25T02:05:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.639+0000 7f66b1594640 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c0aa319200 0x55c0aa31b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.639+0000 7f66b1594640 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c0aa319200 0x55c0aa31b620 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46964544cce3624d server_cookie=a78f1eb02c15a7f0 in_seq=0 out_seq=0 2024-04-25T02:05:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.639+0000 7f66957fa640 10 client.17037 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.667+0000 7f66957fa640 1 -- 172.21.15.31:0/3140760294 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c0aa203540 con 0x55c0aa319200 2024-04-25T02:05:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.667+0000 7f66957fa640 10 client.17037 handle_command_reply: tid=0 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.667+0000 7f66b6de9740 1 client.17037 shutdown 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.667+0000 7f66b6de9740 1 -- 172.21.15.31:0/3140760294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 msgr2=0x55c0aa2aa740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.667+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2aa740 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f66a800cba0 tx=0x7f66a80093a0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.667+0000 7f66b6de9740 1 -- 172.21.15.31:0/3140760294 shutdown_connections 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.667+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c0aa319200 0x55c0aa31b620 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.667+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c0aa2a2230 0x55c0aa29c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c0aa297b60 0x55c0aa29bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b6de9740 1 --2- 172.21.15.31:0/3140760294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0aa297240 0x55c0aa2aa740 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66957fa640 0 client.17037 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b6de9740 1 -- 172.21.15.31:0/3140760294 >> 172.21.15.31:0/3140760294 conn(0x55c0aa293560 msgr2=0x55c0aa2925c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b6de9740 1 -- 172.21.15.31:0/3140760294 shutdown_connections 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b6de9740 1 -- 172.21.15.31:0/3140760294 wait complete. 2024-04-25T02:05:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b6de9740 20 client.17037 trim_cache size 0 max 16384 2024-04-25T02:05:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:05:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:05:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:05:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:05:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b3ff9640 1 -- 172.21.15.31:0/1323699506 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66980a2e20 msgr2=0x7f66980a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b3ff9640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66980a2e20 0x7f66980a52e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f66a40096c0 tx=0x7f66a4035000 comp rx=0 tx=0).stop 2024-04-25T02:05:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b3ff9640 1 -- 172.21.15.31:0/1323699506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac100670 msgr2=0x7f66ac196d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b3ff9640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac100670 0x7f66ac196d90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f66a800eb90 tx=0x7f66a800c6a0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b3ff9640 1 -- 172.21.15.31:0/1323699506 shutdown_connections 2024-04-25T02:05:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b3ff9640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f66980a2e20 0x7f66980a52e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b3ff9640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66ac106630 0x7f66ac19b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b3ff9640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac101030 0x7f66ac1972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b3ff9640 1 --2- 172.21.15.31:0/1323699506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac100670 0x7f66ac196d90 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b3ff9640 1 -- 172.21.15.31:0/1323699506 >> 172.21.15.31:0/1323699506 conn(0x7f66ac0777c0 msgr2=0x7f66ac1031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.668+0000 7f66b3ff9640 1 -- 172.21.15.31:0/1323699506 shutdown_connections 2024-04-25T02:05:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:42.669+0000 7f66b3ff9640 1 -- 172.21.15.31:0/1323699506 wait complete. 2024-04-25T02:05:43.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234865 2.00 MB/sec execute 470 sec latency 21.663 ms 2024-04-25T02:05:44.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:43 smithi134 ceph-mon[33098]: pgmap v349: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 552 KiB/s wr, 49 op/s 2024-04-25T02:05:44.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:43 smithi040 ceph-mon[32418]: pgmap v349: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 552 KiB/s wr, 49 op/s 2024-04-25T02:05:44.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:43 smithi031 ceph-mon[29204]: pgmap v349: 97 pgs: 97 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 552 KiB/s wr, 49 op/s 2024-04-25T02:05:44.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 235358 2.00 MB/sec execute 471 sec latency 13.350 ms 2024-04-25T02:05:45.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 235815 2.01 MB/sec execute 472 sec latency 26.811 ms 2024-04-25T02:05:46.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:45 smithi134 ceph-mon[33098]: pgmap v350: 97 pgs: 97 active+clean; 129 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-04-25T02:05:46.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:45 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:05:46.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:45 smithi031 ceph-mon[29204]: pgmap v350: 97 pgs: 97 active+clean; 129 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-04-25T02:05:46.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:45 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:05:46.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 236347 2.00 MB/sec execute 473 sec latency 23.218 ms 2024-04-25T02:05:46.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:45 smithi040 ceph-mon[32418]: pgmap v350: 97 pgs: 97 active+clean; 129 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-04-25T02:05:46.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:45 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:05:47.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 236756 2.00 MB/sec execute 474 sec latency 16.303 ms 2024-04-25T02:05:47.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:47 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:05:47.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:47 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:05:47.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:47 smithi040 ceph-mon[32418]: pgmap v351: 97 pgs: 97 active+clean; 129 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-04-25T02:05:47.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:47 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:05:47.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:47 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:05:47.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:47 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:05:47.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:47 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:05:47.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:47 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:05:47.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:47 smithi134 ceph-mon[33098]: pgmap v351: 97 pgs: 97 active+clean; 129 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-04-25T02:05:47.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:47 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:05:47.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:47 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:05:47.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:47 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:05:47.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:47 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:05:47.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:47 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:05:47.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:47 smithi031 ceph-mon[29204]: pgmap v351: 97 pgs: 97 active+clean; 129 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-04-25T02:05:47.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:47 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:05:47.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:47 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:05:47.768 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:47 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:05:48.326 INFO:tasks.workunit.client.0.smithi031.stdout: 1 237461 2.01 MB/sec execute 475 sec latency 13.643 ms 2024-04-25T02:05:49.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 237764 2.02 MB/sec execute 476 sec latency 337.672 ms 2024-04-25T02:05:50.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:49 smithi040 ceph-mon[32418]: pgmap v352: 97 pgs: 97 active+clean; 129 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-04-25T02:05:50.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:49 smithi134 ceph-mon[33098]: pgmap v352: 97 pgs: 97 active+clean; 129 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-04-25T02:05:50.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:49 smithi031 ceph-mon[29204]: pgmap v352: 97 pgs: 97 active+clean; 129 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 514 KiB/s wr, 62 op/s 2024-04-25T02:05:50.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 237831 2.01 MB/sec execute 477 sec latency 1059.792 ms 2024-04-25T02:05:51.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 237831 2.01 MB/sec execute 478 sec latency 1035.480 ms 2024-04-25T02:05:52.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:51 smithi040 ceph-mon[32418]: pgmap v353: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:05:52.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:51 smithi134 ceph-mon[33098]: pgmap v353: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:05:52.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:51 smithi031 ceph-mon[29204]: pgmap v353: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:05:52.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 237831 2.00 MB/sec execute 479 sec latency 2035.574 ms 2024-04-25T02:05:53.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 237831 2.00 MB/sec execute 480 sec latency 3035.628 ms 2024-04-25T02:05:54.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:53 smithi040 ceph-mon[32418]: pgmap v354: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-25T02:05:54.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:53 smithi134 ceph-mon[33098]: pgmap v354: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-25T02:05:54.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:53 smithi031 ceph-mon[29204]: pgmap v354: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-25T02:05:54.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 237831 1.99 MB/sec execute 481 sec latency 4035.699 ms 2024-04-25T02:05:55.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 237926 1.99 MB/sec execute 482 sec latency 4752.278 ms 2024-04-25T02:05:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.765+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/1092045945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb41029d0 msgr2=0x7f1fb4102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.765+0000 7f1fbc4ee640 1 --2- 172.21.15.31:0/1092045945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb41029d0 0x7f1fb4102db0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0009900 tx=0x7f1fb002a510 comp rx=0 tx=0).stop 2024-04-25T02:05:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.765+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/1092045945 shutdown_connections 2024-04-25T02:05:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.765+0000 7f1fbc4ee640 1 --2- 172.21.15.31:0/1092045945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fb410bd10 0x7f1fb410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.765+0000 7f1fbc4ee640 1 --2- 172.21.15.31:0/1092045945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fb41032f0 0x7f1fb410b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.765+0000 7f1fbc4ee640 1 --2- 172.21.15.31:0/1092045945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb41029d0 0x7f1fb4102db0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.765+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/1092045945 >> 172.21.15.31:0/1092045945 conn(0x7f1fb40777c0 msgr2=0x7f1fb4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.765+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/1092045945 shutdown_connections 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.765+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/1092045945 wait complete. 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fbc4ee640 1 Processor -- start 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fbc4ee640 1 -- start start 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fbc4ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fb41029d0 0x7f1fb4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fbc4ee640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fb41032f0 0x7f1fb4198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fbc4ee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb410bd10 0x7f1fb419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fbc4ee640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41119c0 con 0x7f1fb410bd10 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fbc4ee640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb4111840 con 0x7f1fb41029d0 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fbc4ee640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb4111b40 con 0x7f1fb41032f0 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fbaa8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb410bd10 0x7f1fb419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fbaa8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb410bd10 0x7f1fb419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35930/0 (socket says 172.21.15.31:35930) 2024-04-25T02:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fbaa8b640 1 -- 172.21.15.31:0/2841242495 learned_addr learned my addr 172.21.15.31:0/2841242495 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fba28a640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fb41029d0 0x7f1fb4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.766+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fb41032f0 0x7f1fb4198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fbaa8b640 1 -- 172.21.15.31:0/2841242495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fb41032f0 msgr2=0x7f1fb4198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fbaa8b640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fb41032f0 0x7f1fb4198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fbaa8b640 1 -- 172.21.15.31:0/2841242495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fb41029d0 msgr2=0x7f1fb4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fbaa8b640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fb41029d0 0x7f1fb4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fbaa8b640 1 -- 172.21.15.31:0/2841242495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb419cfb0 con 0x7f1fb410bd10 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fba28a640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fb41029d0 0x7f1fb4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fbaa8b640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb410bd10 0x7f1fb419c880 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00eb10 tx=0x7f1fac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fab7fe640 1 -- 172.21.15.31:0/2841242495 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac004b20 con 0x7f1fb410bd10 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fab7fe640 1 -- 172.21.15.31:0/2841242495 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1fac004cc0 con 0x7f1fb410bd10 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/2841242495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb419d2a0 con 0x7f1fb410bd10 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fab7fe640 1 -- 172.21.15.31:0/2841242495 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac010690 con 0x7f1fb410bd10 2024-04-25T02:05:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.767+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/2841242495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb41a4b30 con 0x7f1fb410bd10 2024-04-25T02:05:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.769+0000 7f1fab7fe640 1 -- 172.21.15.31:0/2841242495 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1fac005020 con 0x7f1fb410bd10 2024-04-25T02:05:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.770+0000 7f1fbf1fb740 1 Processor -- start 2024-04-25T02:05:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.770+0000 7f1fbf1fb740 1 -- start start 2024-04-25T02:05:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.770+0000 7f1fab7fe640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1fa00a2da0 0x7f1fa00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.770+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a239351500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239351a40 0x55a239360100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 0x55a239362a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a2392b5cb0 con 0x55a239360640 2024-04-25T02:05:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a23925e5b0 con 0x55a239351120 2024-04-25T02:05:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a2392b1c70 con 0x55a239351a40 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1fa00a2da0 0x7f1fa00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fba28a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a239351500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fba28a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a239351500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54114/0 (socket says 172.21.15.31:54114) 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fba28a640 1 -- 172.21.15.31:0/3332586864 learned_addr learned my addr 172.21.15.31:0/3332586864 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fbaa8b640 1 --2- 172.21.15.31:0/3332586864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 0x55a239362a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fab7fe640 1 -- 172.21.15.31:0/2841242495 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f1fac0cf940 con 0x7f1fb410bd10 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/3332586864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239351a40 0x55a239360100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fba28a640 1 -- 172.21.15.31:0/3332586864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239351a40 msgr2=0x55a239360100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fba28a640 1 --2- 172.21.15.31:0/3332586864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239351a40 0x55a239360100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fba28a640 1 -- 172.21.15.31:0/3332586864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 msgr2=0x55a239362a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fba28a640 1 --2- 172.21.15.31:0/3332586864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 0x55a239362a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fba28a640 1 -- 172.21.15.31:0/3332586864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a238fca210 con 0x55a239351120 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1fa00a2da0 0x7f1fa00a5260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4005d00 tx=0x7f1fa400be90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fba28a640 1 --2- 172.21.15.31:0/3332586864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a239351500 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0009900 tx=0x7f1fb0005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6f204a53a58d130 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.771+0000 7f1fa8bfe640 1 -- 172.21.15.31:0/3332586864 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0040070 con 0x55a239351120 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.772+0000 7f1fa8bfe640 1 -- 172.21.15.31:0/3332586864 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1fb00318e0 con 0x55a239351120 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.772+0000 7f1fa8bfe640 1 -- 172.21.15.31:0/3332586864 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0035040 con 0x55a239351120 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.772+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/3332586864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 msgr2=0x55a239351500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.772+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/3332586864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a239351500 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0009900 tx=0x7f1fb0005e40 comp rx=0 tx=0).stop 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.772+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/3332586864 shutdown_connections 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.772+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/3332586864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 0x55a239362a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.772+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/3332586864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239351a40 0x55a239360100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.772+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/3332586864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a239351500 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.772+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/3332586864 >> 172.21.15.31:0/3332586864 conn(0x55a23934d560 msgr2=0x55a23934d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.772+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/3332586864 shutdown_connections 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.772+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/3332586864 wait complete. 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fbf1fb740 1 -- start start 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fbf1fb740 1 Processor -- start 2024-04-25T02:05:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a23935d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351a40 0x55a23935da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239360640 0x55a23935dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a239363420 con 0x55a239351a40 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a239271b50 con 0x55a239351120 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a2393635a0 con 0x55a239360640 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fbf1fb740 10 client.0 _ll_register_callbacks cb 0x55a2392acca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fb9a89640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351a40 0x55a23935da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fba28a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a23935d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fb9a89640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351a40 0x55a23935da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35952/0 (socket says 172.21.15.31:35952) 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fba28a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a23935d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:54116/0 (socket says 172.21.15.31:54116) 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.773+0000 7f1fb9a89640 1 -- 172.21.15.31:0/2662866492 learned_addr learned my addr 172.21.15.31:0/2662866492 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1fba28a640 1 -- 172.21.15.31:0/2662866492 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239360640 msgr2=0x55a23935dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1fba28a640 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239360640 0x55a23935dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1fba28a640 1 -- 172.21.15.31:0/2662866492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351a40 msgr2=0x55a23935da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1fba28a640 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351a40 0x55a23935da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1fba28a640 1 -- 172.21.15.31:0/2662866492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a239096620 con 0x55a239351120 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351a40 0x55a23935da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1fba28a640 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a23935d550 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0009900 tx=0x7f1fb0031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0f37987dd94ce13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1f9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1f9dffb640 1 -- 172.21.15.31:0/2662866492 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0035070 con 0x55a239351120 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1f9dffb640 1 -- 172.21.15.31:0/2662866492 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f1fb00035f0 con 0x55a239351120 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2662866492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a2392c31f0 con 0x55a239351120 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1f9dffb640 1 -- 172.21.15.31:0/2662866492 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1fb0055050 con 0x55a239351120 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1fbf1fb740 10 client.26627 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2662866492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a23936f5f0 con 0x55a239351120 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.774+0000 7f1f9dffb640 1 -- 172.21.15.31:0/2662866492 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a2392c31f0 con 0x55a239351120 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.775+0000 7f1f9dffb640 1 -- 172.21.15.31:0/2662866492 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f1fb003c7e0 con 0x55a239351120 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.775+0000 7f1fbf1fb740 10 client.26627 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.775+0000 7f1fbf1fb740 10 client.26627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.775+0000 7f1fbf1fb740 20 client.26627 populate_metadata read hostname 'smithi031' 2024-04-25T02:05:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.775+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a239370940 0x55a239372d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.775+0000 7f1fbf1fb740 4 client.26627 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:05:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.775+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2662866492 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2392bd540 con 0x55a239370940 2024-04-25T02:05:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.775+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a239370940 0x55a239372d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.775+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a239370940 0x55a239372d30 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d68028019587c172 server_cookie=e239be77712bad95 in_seq=0 out_seq=0 2024-04-25T02:05:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.775+0000 7f1f9dffb640 10 client.26627 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1f9dffb640 1 -- 172.21.15.31:0/2662866492 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2392bd540 con 0x55a239370940 2024-04-25T02:05:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1f9dffb640 10 client.26627 handle_command_reply: tid=0 2024-04-25T02:05:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 1 client.26627 shutdown 2024-04-25T02:05:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2662866492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 msgr2=0x55a23935d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a23935d550 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0009900 tx=0x7f1fb0031400 comp rx=0 tx=0).stop 2024-04-25T02:05:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2662866492 shutdown_connections 2024-04-25T02:05:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a239370940 0x55a239372d30 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239360640 0x55a23935dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351a40 0x55a23935da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/2662866492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a23935d550 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1f9dffb640 0 client.26627 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2662866492 >> 172.21.15.31:0/2662866492 conn(0x55a23934d560 msgr2=0x55a239353310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2662866492 shutdown_connections 2024-04-25T02:05:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2662866492 wait complete. 2024-04-25T02:05:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.777+0000 7f1fbf1fb740 20 client.26627 trim_cache size 0 max 16384 2024-04-25T02:05:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fbf1fb740 1 Processor -- start 2024-04-25T02:05:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fbf1fb740 1 -- start start 2024-04-25T02:05:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351120 0x55a23935fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351a40 0x55a23936c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239360640 0x55a23935c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41ab0e0 con 0x55a239351120 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41acda0 con 0x55a239351a40 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41ab7b0 con 0x55a239360640 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fba28a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351120 0x55a23935fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fba28a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351120 0x55a23935fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35968/0 (socket says 172.21.15.31:35968) 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fba28a640 1 -- 172.21.15.31:0/2397420544 learned_addr learned my addr 172.21.15.31:0/2397420544 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/2397420544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351a40 0x55a23936c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.794+0000 7f1fbaa8b640 1 --2- 172.21.15.31:0/2397420544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239360640 0x55a23935c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.795+0000 7f1fb9a89640 1 -- 172.21.15.31:0/2397420544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239360640 msgr2=0x55a23935c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.795+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/2397420544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239360640 0x55a23935c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.795+0000 7f1fb9a89640 1 -- 172.21.15.31:0/2397420544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351120 msgr2=0x55a23935fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.795+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/2397420544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351120 0x55a23935fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.795+0000 7f1fb9a89640 1 -- 172.21.15.31:0/2397420544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a239351870 con 0x55a239351a40 2024-04-25T02:05:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.795+0000 7f1fba28a640 1 --2- 172.21.15.31:0/2397420544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351120 0x55a23935fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.795+0000 7f1fbaa8b640 1 --2- 172.21.15.31:0/2397420544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239360640 0x55a23935c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:05:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.795+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/2397420544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351a40 0x55a23936c320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55a239373420 tx=0x7f1fa401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b53537469cd17a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.795+0000 7f1f9ffff640 1 -- 172.21.15.31:0/2397420544 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4014aa0 con 0x55a239351a40 2024-04-25T02:05:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.795+0000 7f1f9ffff640 1 -- 172.21.15.31:0/2397420544 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1fa40054d0 con 0x55a239351a40 2024-04-25T02:05:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.795+0000 7f1f9ffff640 1 -- 172.21.15.31:0/2397420544 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa402a7b0 con 0x55a239351a40 2024-04-25T02:05:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2397420544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351a40 msgr2=0x55a23936c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/2397420544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351a40 0x55a23936c320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55a239373420 tx=0x7f1fa401d000 comp rx=0 tx=0).stop 2024-04-25T02:05:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2397420544 shutdown_connections 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/2397420544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239360640 0x55a23935c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/2397420544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351a40 0x55a23936c320 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/2397420544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239351120 0x55a23935fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2397420544 >> 172.21.15.31:0/2397420544 conn(0x55a23934d560 msgr2=0x55a23934d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2397420544 shutdown_connections 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/2397420544 wait complete. 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 -- start start 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 Processor -- start 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a23935b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239351a40 0x55a239368830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 0x55a2393597c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41ac6d0 con 0x55a239360640 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41abf40 con 0x55a239351120 2024-04-25T02:05:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb41a90e0 con 0x55a239351a40 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.796+0000 7f1fbf1fb740 10 client.0 _ll_register_callbacks cb 0x7f1fac004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fbaa8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 0x55a2393597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fba28a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a23935b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fbaa8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 0x55a2393597c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35970/0 (socket says 172.21.15.31:35970) 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fbaa8b640 1 -- 172.21.15.31:0/160849093 learned_addr learned my addr 172.21.15.31:0/160849093 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fb9a89640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239351a40 0x55a239368830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fba28a640 1 -- 172.21.15.31:0/160849093 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239351a40 msgr2=0x55a239368830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fba28a640 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239351a40 0x55a239368830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fba28a640 1 -- 172.21.15.31:0/160849093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 msgr2=0x55a2393597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fba28a640 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 0x55a2393597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fba28a640 1 -- 172.21.15.31:0/160849093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a239359d00 con 0x55a239351120 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fbaa8b640 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 0x55a2393597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fba28a640 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a23935b1d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0009800 tx=0x7f1fb0005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9794661cc159bd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1f9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:05:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1f9dffb640 1 -- 172.21.15.31:0/160849093 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb005d070 con 0x55a239351120 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1f9dffb640 1 -- 172.21.15.31:0/160849093 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8484+0+0 (secure 0 0 0) 0x7f1fb003c440 con 0x55a239351120 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/160849093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fa40054d0 con 0x55a239351120 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1f9dffb640 1 -- 172.21.15.31:0/160849093 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1fb0035050 con 0x55a239351120 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fbf1fb740 10 client.26633 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.797+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/160849093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a2393d6440 con 0x55a239351120 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.798+0000 7f1f9dffb640 1 -- 172.21.15.31:0/160849093 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1fa40054d0 con 0x55a239351120 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.798+0000 7f1f9dffb640 1 -- 172.21.15.31:0/160849093 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f1fb0059430 con 0x55a239351120 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.798+0000 7f1fbf1fb740 10 client.26633 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.798+0000 7f1fbf1fb740 10 client.26633 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.798+0000 7f1fbf1fb740 20 client.26633 populate_metadata read hostname 'smithi031' 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.798+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a2393d69f0 0x55a2393d8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.798+0000 7f1fbf1fb740 4 client.26633 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.798+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/160849093 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a23923b8e0 con 0x55a2393d69f0 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.798+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a2393d69f0 0x55a2393d8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.798+0000 7f1f9dffb640 10 client.26633 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.798+0000 7f1fb9a89640 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a2393d69f0 0x55a2393d8e10 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f877fa1c37aeb2be server_cookie=d4dd8c6be6e9460e in_seq=0 out_seq=0 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1f9dffb640 1 -- 172.21.15.31:0/160849093 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a23923b8e0 con 0x55a2393d69f0 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1f9dffb640 10 client.26633 handle_command_reply: tid=0 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 1 client.26633 shutdown 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/160849093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 msgr2=0x55a23935b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a23935b1d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0009800 tx=0x7f1fb0005e00 comp rx=0 tx=0).stop 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/160849093 shutdown_connections 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a2393d69f0 0x55a2393d8e10 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a239360640 0x55a2393597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a239351a40 0x55a239368830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 1 --2- 172.21.15.31:0/160849093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a239351120 0x55a23935b1d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1f9dffb640 0 client.26633 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:05:55.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/160849093 >> 172.21.15.31:0/160849093 conn(0x55a23934d560 msgr2=0x55a23934bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/160849093 shutdown_connections 2024-04-25T02:05:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 1 -- 172.21.15.31:0/160849093 wait complete. 2024-04-25T02:05:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.824+0000 7f1fbf1fb740 20 client.26633 trim_cache size 0 max 16384 2024-04-25T02:05:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:05:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:05:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:05:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:05:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/2841242495 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1fa00a2da0 msgr2=0x7f1fa00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1fa00a2da0 0x7f1fa00a5260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4005d00 tx=0x7f1fa400be90 comp rx=0 tx=0).stop 2024-04-25T02:05:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/2841242495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb410bd10 msgr2=0x7f1fb419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb410bd10 0x7f1fb419c880 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00eb10 tx=0x7f1fac00c2d0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/2841242495 shutdown_connections 2024-04-25T02:05:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1fa00a2da0 0x7f1fa00a5260 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb410bd10 0x7f1fb419c880 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fb41032f0 0x7f1fb4198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 --2- 172.21.15.31:0/2841242495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fb41029d0 0x7f1fb4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/2841242495 >> 172.21.15.31:0/2841242495 conn(0x7f1fb40777c0 msgr2=0x7f1fb40feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/2841242495 shutdown_connections 2024-04-25T02:05:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:05:55.825+0000 7f1fbc4ee640 1 -- 172.21.15.31:0/2841242495 wait complete. 2024-04-25T02:05:56.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:55 smithi040 ceph-mon[32418]: pgmap v355: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-04-25T02:05:56.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:55 smithi134 ceph-mon[33098]: pgmap v355: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-04-25T02:05:56.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:55 smithi031 ceph-mon[29204]: pgmap v355: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-04-25T02:05:56.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 238165 1.99 MB/sec execute 483 sec latency 18.474 ms 2024-04-25T02:05:57.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 238587 1.99 MB/sec execute 484 sec latency 18.969 ms 2024-04-25T02:05:58.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:57 smithi040 ceph-mon[32418]: pgmap v356: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 596 KiB/s wr, 57 op/s 2024-04-25T02:05:58.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:57 smithi134 ceph-mon[33098]: pgmap v356: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 596 KiB/s wr, 57 op/s 2024-04-25T02:05:58.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:57 smithi031 ceph-mon[29204]: pgmap v356: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 596 KiB/s wr, 57 op/s 2024-04-25T02:05:58.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 239102 1.99 MB/sec execute 485 sec latency 19.417 ms 2024-04-25T02:05:59.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 239561 1.99 MB/sec execute 486 sec latency 23.035 ms 2024-04-25T02:06:00.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:05:59 smithi040 ceph-mon[32418]: pgmap v357: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 596 KiB/s wr, 57 op/s 2024-04-25T02:06:00.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:05:59 smithi134 ceph-mon[33098]: pgmap v357: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 596 KiB/s wr, 57 op/s 2024-04-25T02:06:00.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:05:59 smithi031 ceph-mon[29204]: pgmap v357: 97 pgs: 97 active+clean; 133 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 596 KiB/s wr, 57 op/s 2024-04-25T02:06:00.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 240080 1.99 MB/sec execute 487 sec latency 24.084 ms 2024-04-25T02:06:01.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 240666 1.99 MB/sec execute 488 sec latency 19.118 ms 2024-04-25T02:06:02.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:01 smithi040 ceph-mon[32418]: pgmap v358: 97 pgs: 97 active+clean; 130 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-25T02:06:02.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:01 smithi134 ceph-mon[33098]: pgmap v358: 97 pgs: 97 active+clean; 130 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-25T02:06:02.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:01 smithi031 ceph-mon[29204]: pgmap v358: 97 pgs: 97 active+clean; 130 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-25T02:06:02.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 241082 1.99 MB/sec execute 489 sec latency 21.501 ms 2024-04-25T02:06:03.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:02 smithi040 ceph-mon[32418]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T02:06:03.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:02 smithi134 ceph-mon[33098]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T02:06:03.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:02 smithi031 ceph-mon[29204]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T02:06:03.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 241310 2.00 MB/sec execute 490 sec latency 664.119 ms 2024-04-25T02:06:04.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:03 smithi040 ceph-mon[32418]: pgmap v360: 97 pgs: 97 active+clean; 130 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 694 KiB/s wr, 89 op/s 2024-04-25T02:06:04.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:03 smithi134 ceph-mon[33098]: pgmap v360: 97 pgs: 97 active+clean; 130 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 694 KiB/s wr, 89 op/s 2024-04-25T02:06:04.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:03 smithi031 ceph-mon[29204]: pgmap v360: 97 pgs: 97 active+clean; 130 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 694 KiB/s wr, 89 op/s 2024-04-25T02:06:04.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 241310 1.99 MB/sec execute 491 sec latency 1664.153 ms 2024-04-25T02:06:05.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 241377 1.99 MB/sec execute 492 sec latency 2383.552 ms 2024-04-25T02:06:06.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:05 smithi040 ceph-mon[32418]: pgmap v361: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:06:06.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:05 smithi134 ceph-mon[33098]: pgmap v361: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:06:06.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:05 smithi031 ceph-mon[29204]: pgmap v361: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:06:06.327 INFO:tasks.workunit.client.0.smithi031.stdout: 1 241377 1.99 MB/sec execute 493 sec latency 1043.262 ms 2024-04-25T02:06:07.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 241377 1.98 MB/sec execute 494 sec latency 2043.377 ms 2024-04-25T02:06:08.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:07 smithi040 ceph-mon[32418]: pgmap v362: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:06:08.251 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:07 smithi031 ceph-mon[29204]: pgmap v362: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:06:08.251 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:07 smithi134 ceph-mon[33098]: pgmap v362: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:06:08.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 241377 1.98 MB/sec execute 495 sec latency 3043.434 ms 2024-04-25T02:06:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.927+0000 7fb274fa2640 1 -- 172.21.15.31:0/2699435156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270107750 msgr2=0x7fb270107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.927+0000 7fb274fa2640 1 --2- 172.21.15.31:0/2699435156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270107750 0x7fb270107b30 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fb260009930 tx=0x7fb26002a420 comp rx=0 tx=0).stop 2024-04-25T02:06:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.927+0000 7fb274fa2640 1 -- 172.21.15.31:0/2699435156 shutdown_connections 2024-04-25T02:06:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.927+0000 7fb274fa2640 1 --2- 172.21.15.31:0/2699435156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb270102100 0x7fb27010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.927+0000 7fb274fa2640 1 --2- 172.21.15.31:0/2699435156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270101740 0x7fb270101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.927+0000 7fb274fa2640 1 --2- 172.21.15.31:0/2699435156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270107750 0x7fb270107b30 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.927+0000 7fb274fa2640 1 -- 172.21.15.31:0/2699435156 >> 172.21.15.31:0/2699435156 conn(0x7fb2700fd210 msgr2=0x7fb2700ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.928+0000 7fb274fa2640 1 -- 172.21.15.31:0/2699435156 shutdown_connections 2024-04-25T02:06:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.928+0000 7fb274fa2640 1 -- 172.21.15.31:0/2699435156 wait complete. 2024-04-25T02:06:08.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.928+0000 7fb274fa2640 1 Processor -- start 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.928+0000 7fb274fa2640 1 -- start start 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb274fa2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270101740 0x7fb270197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb274fa2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270102100 0x7fb2701983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb274fa2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb270107750 0x7fb27019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb274fa2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb270111a60 con 0x7fb270101740 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb274fa2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701118e0 con 0x7fb270102100 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb274fa2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb270111be0 con 0x7fb270107750 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb26e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270101740 0x7fb270197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb26e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270101740 0x7fb270197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54316/0 (socket says 172.21.15.31:54316) 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb26e59c640 1 -- 172.21.15.31:0/3183260621 learned_addr learned my addr 172.21.15.31:0/3183260621 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb26ed9d640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb270107750 0x7fb27019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb26dd9b640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270102100 0x7fb2701983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb26e59c640 1 -- 172.21.15.31:0/3183260621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb270107750 msgr2=0x7fb27019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb26e59c640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb270107750 0x7fb27019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb26e59c640 1 -- 172.21.15.31:0/3183260621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270102100 msgr2=0x7fb2701983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb26e59c640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270102100 0x7fb2701983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb26e59c640 1 -- 172.21.15.31:0/3183260621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27019ce90 con 0x7fb270101740 2024-04-25T02:06:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb26e59c640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270101740 0x7fb270197ea0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb260005ab0 tx=0x7fb260003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.929+0000 7fb25f7fe640 1 -- 172.21.15.31:0/3183260621 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26003f070 con 0x7fb270101740 2024-04-25T02:06:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.930+0000 7fb25f7fe640 1 -- 172.21.15.31:0/3183260621 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb260031d60 con 0x7fb270101740 2024-04-25T02:06:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.930+0000 7fb274fa2640 1 -- 172.21.15.31:0/3183260621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27019d120 con 0x7fb270101740 2024-04-25T02:06:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.930+0000 7fb25f7fe640 1 -- 172.21.15.31:0/3183260621 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260035030 con 0x7fb270101740 2024-04-25T02:06:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.930+0000 7fb274fa2640 1 -- 172.21.15.31:0/3183260621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2701a4a10 con 0x7fb270101740 2024-04-25T02:06:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.932+0000 7fb25f7fe640 1 -- 172.21.15.31:0/3183260621 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb260042070 con 0x7fb270101740 2024-04-25T02:06:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.932+0000 7fb277d90740 1 Processor -- start 2024-04-25T02:06:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.932+0000 7fb277d90740 1 -- start start 2024-04-25T02:06:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.932+0000 7fb277d90740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5dfe520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.932+0000 7fb277d90740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 0x5653c5e0d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb277d90740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5e0d640 0x5653c5e0fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb277d90740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5653c5d62cb0 con 0x5653c5dfe140 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb277d90740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5653c5d0b5b0 con 0x5653c5dfea60 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb277d90740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5653c5d5ec70 con 0x5653c5e0d640 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb25f7fe640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2580a2da0 0x7fb2580a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5dfe520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 0x5653c5e0d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 0x5653c5e0d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47800/0 (socket says 172.21.15.31:47800) 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26e59c640 1 -- 172.21.15.31:0/3505824652 learned_addr learned my addr 172.21.15.31:0/3505824652 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26dd9b640 1 --2- 172.21.15.31:0/3505824652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5e0d640 0x5653c5e0fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26ed9d640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2580a2da0 0x7fb2580a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb25f7fe640 1 -- 172.21.15.31:0/3183260621 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7fb2600efc40 con 0x7fb270101740 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26e59c640 1 -- 172.21.15.31:0/3505824652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5e0d640 msgr2=0x5653c5e0fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26e59c640 1 --2- 172.21.15.31:0/3505824652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5e0d640 0x5653c5e0fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26e59c640 1 -- 172.21.15.31:0/3505824652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 msgr2=0x5653c5dfe520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26e59c640 1 --2- 172.21.15.31:0/3505824652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5dfe520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26e59c640 1 -- 172.21.15.31:0/3505824652 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c5a77210 con 0x5653c5dfea60 2024-04-25T02:06:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26dd9b640 1 --2- 172.21.15.31:0/3505824652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5e0d640 0x5653c5e0fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26e59c640 1 --2- 172.21.15.31:0/3505824652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 0x5653c5e0d100 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb2600b85e0 tx=0x7fb2600bbdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0b855aee924140c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.933+0000 7fb26ed9d640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2580a2da0 0x7fb2580a5260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb268009fd0 tx=0x7fb268004ab0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.934+0000 7fb25cff9640 1 -- 172.21.15.31:0/3505824652 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2600bc880 con 0x5653c5dfea60 2024-04-25T02:06:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.934+0000 7fb25cff9640 1 -- 172.21.15.31:0/3505824652 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2600bc180 con 0x5653c5dfea60 2024-04-25T02:06:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.934+0000 7fb25cff9640 1 -- 172.21.15.31:0/3505824652 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26003b580 con 0x5653c5dfea60 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.934+0000 7fb277d90740 1 -- 172.21.15.31:0/3505824652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 msgr2=0x5653c5e0d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.934+0000 7fb277d90740 1 --2- 172.21.15.31:0/3505824652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 0x5653c5e0d100 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb2600b85e0 tx=0x7fb2600bbdf0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.934+0000 7fb277d90740 1 -- 172.21.15.31:0/3505824652 shutdown_connections 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.934+0000 7fb277d90740 1 --2- 172.21.15.31:0/3505824652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5e0d640 0x5653c5e0fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.934+0000 7fb277d90740 1 --2- 172.21.15.31:0/3505824652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 0x5653c5e0d100 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.934+0000 7fb277d90740 1 --2- 172.21.15.31:0/3505824652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5dfe520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.934+0000 7fb277d90740 1 -- 172.21.15.31:0/3505824652 >> 172.21.15.31:0/3505824652 conn(0x5653c5dfa560 msgr2=0x5653c5dfa940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb277d90740 1 -- 172.21.15.31:0/3505824652 shutdown_connections 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb277d90740 1 -- 172.21.15.31:0/3505824652 wait complete. 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb277d90740 1 -- start start 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb277d90740 1 Processor -- start 2024-04-25T02:06:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb277d90740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb277d90740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5dfea60 0x5653c5e0aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb277d90740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5e0d640 0x5653c5e0af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb277d90740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5653c5e101c0 con 0x5653c5dfe140 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb277d90740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5653c5d1eb50 con 0x5653c5e0d640 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb277d90740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5653c5e10340 con 0x5653c5dfea60 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb277d90740 10 client.0 _ll_register_callbacks cb 0x5653c5d59ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb26e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb26e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54332/0 (socket says 172.21.15.31:54332) 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb26e59c640 1 -- 172.21.15.31:0/372038111 learned_addr learned my addr 172.21.15.31:0/372038111 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.935+0000 7fb26ed9d640 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5e0d640 0x5653c5e0af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb26dd9b640 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5dfea60 0x5653c5e0aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb26e59c640 1 -- 172.21.15.31:0/372038111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5dfea60 msgr2=0x5653c5e0aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb26e59c640 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5dfea60 0x5653c5e0aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb26e59c640 1 -- 172.21.15.31:0/372038111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5e0d640 msgr2=0x5653c5e0af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb26e59c640 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5e0d640 0x5653c5e0af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb26e59c640 1 -- 172.21.15.31:0/372038111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c5b43620 con 0x5653c5dfe140 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb26e59c640 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0a510 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb2600ed9b0 tx=0x7fb26002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e12b46ec3d25cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb255ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb255ffb640 1 -- 172.21.15.31:0/372038111 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260049040 con 0x5653c5dfe140 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb255ffb640 1 -- 172.21.15.31:0/372038111 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7fb2600ba480 con 0x5653c5dfe140 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb277d90740 1 -- 172.21.15.31:0/372038111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2600bc180 con 0x5653c5dfe140 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb255ffb640 1 -- 172.21.15.31:0/372038111 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb2600bc880 con 0x5653c5dfe140 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb277d90740 10 client.17067 fetch_fsmap learned FSMap version 40 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb277d90740 1 -- 172.21.15.31:0/372038111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5653c5e101c0 con 0x5653c5dfe140 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.936+0000 7fb255ffb640 1 -- 172.21.15.31:0/372038111 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2600bc180 con 0x5653c5dfe140 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.937+0000 7fb255ffb640 1 -- 172.21.15.31:0/372038111 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fb26003b580 con 0x5653c5dfe140 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.937+0000 7fb277d90740 10 client.17067 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:06:08.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.937+0000 7fb277d90740 10 client.17067 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.937+0000 7fb277d90740 20 client.17067 populate_metadata read hostname 'smithi031' 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.937+0000 7fb277d90740 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5653c5e1d630 0x5653c5e1fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.937+0000 7fb277d90740 4 client.17067 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.937+0000 7fb277d90740 1 -- 172.21.15.31:0/372038111 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653c5d701f0 con 0x5653c5e1d630 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.937+0000 7fb26dd9b640 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5653c5e1d630 0x5653c5e1fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.937+0000 7fb255ffb640 10 client.17067 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.937+0000 7fb26dd9b640 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5653c5e1d630 0x5653c5e1fa70 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a266edd59d81310 server_cookie=efc69548edcf4a30 in_seq=0 out_seq=0 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb255ffb640 1 -- 172.21.15.31:0/372038111 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653c5d701f0 con 0x5653c5e1d630 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb255ffb640 10 client.17067 handle_command_reply: tid=0 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb277d90740 1 client.17067 shutdown 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb277d90740 1 -- 172.21.15.31:0/372038111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 msgr2=0x5653c5e0a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb277d90740 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0a510 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb2600ed9b0 tx=0x7fb26002a420 comp rx=0 tx=0).stop 2024-04-25T02:06:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb277d90740 1 -- 172.21.15.31:0/372038111 shutdown_connections 2024-04-25T02:06:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb277d90740 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5653c5e1d630 0x5653c5e1fa70 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb277d90740 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5e0d640 0x5653c5e0af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb277d90740 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5dfea60 0x5653c5e0aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb277d90740 1 --2- 172.21.15.31:0/372038111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0a510 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb255ffb640 0 client.17067 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb277d90740 1 -- 172.21.15.31:0/372038111 >> 172.21.15.31:0/372038111 conn(0x5653c5dfa560 msgr2=0x5653c5e002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb277d90740 1 -- 172.21.15.31:0/372038111 shutdown_connections 2024-04-25T02:06:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.938+0000 7fb277d90740 1 -- 172.21.15.31:0/372038111 wait complete. 2024-04-25T02:06:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.939+0000 7fb277d90740 20 client.17067 trim_cache size 0 max 16384 2024-04-25T02:06:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb277d90740 1 Processor -- start 2024-04-25T02:06:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb277d90740 1 -- start start 2024-04-25T02:06:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb277d90740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb277d90740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 0x5653c5e00510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb277d90740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5e0d640 0x5653c5e09670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb277d90740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701ab020 con 0x5653c5dfe140 2024-04-25T02:06:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb277d90740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701acce0 con 0x5653c5dfea60 2024-04-25T02:06:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb277d90740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701ab6f0 con 0x5653c5e0d640 2024-04-25T02:06:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb26e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb26e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54338/0 (socket says 172.21.15.31:54338) 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb26e59c640 1 -- 172.21.15.31:0/1950167167 learned_addr learned my addr 172.21.15.31:0/1950167167 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb26dd9b640 1 --2- 172.21.15.31:0/1950167167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 0x5653c5e00510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb26ed9d640 1 --2- 172.21.15.31:0/1950167167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5e0d640 0x5653c5e09670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb26e59c640 1 -- 172.21.15.31:0/1950167167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5e0d640 msgr2=0x5653c5e09670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb26e59c640 1 --2- 172.21.15.31:0/1950167167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5e0d640 0x5653c5e09670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb26e59c640 1 -- 172.21.15.31:0/1950167167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 msgr2=0x5653c5e00510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb26e59c640 1 --2- 172.21.15.31:0/1950167167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 0x5653c5e00510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb26e59c640 1 -- 172.21.15.31:0/1950167167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c5dfe890 con 0x5653c5dfe140 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.955+0000 7fb26dd9b640 1 --2- 172.21.15.31:0/1950167167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 0x5653c5e00510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb26e59c640 1 --2- 172.21.15.31:0/1950167167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0cc40 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb260000c00 tx=0x7fb260009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c8fa7dc4474ad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb257fff640 1 -- 172.21.15.31:0/1950167167 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260049070 con 0x5653c5dfe140 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb257fff640 1 -- 172.21.15.31:0/1950167167 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2600bc180 con 0x5653c5dfe140 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb257fff640 1 -- 172.21.15.31:0/1950167167 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2600c4df0 con 0x5653c5dfe140 2024-04-25T02:06:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb277d90740 1 -- 172.21.15.31:0/1950167167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 msgr2=0x5653c5e0cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb277d90740 1 --2- 172.21.15.31:0/1950167167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0cc40 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb260000c00 tx=0x7fb260009260 comp rx=0 tx=0).stop 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb277d90740 1 -- 172.21.15.31:0/1950167167 shutdown_connections 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb277d90740 1 --2- 172.21.15.31:0/1950167167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5e0d640 0x5653c5e09670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb277d90740 1 --2- 172.21.15.31:0/1950167167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfea60 0x5653c5e00510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb277d90740 1 --2- 172.21.15.31:0/1950167167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5dfe140 0x5653c5e0cc40 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb277d90740 1 -- 172.21.15.31:0/1950167167 >> 172.21.15.31:0/1950167167 conn(0x5653c5dfa560 msgr2=0x5653c5dfa940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.956+0000 7fb277d90740 1 -- 172.21.15.31:0/1950167167 shutdown_connections 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb277d90740 1 -- 172.21.15.31:0/1950167167 wait complete. 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb277d90740 1 -- start start 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb277d90740 1 Processor -- start 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb277d90740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfe140 0x5653c5e06920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb277d90740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5dfea60 0x5653c5e06e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb277d90740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5e0d640 0x5653c5e073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb277d90740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701ac610 con 0x5653c5e0d640 2024-04-25T02:06:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb277d90740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701abe80 con 0x5653c5dfe140 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb277d90740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701a9020 con 0x5653c5dfea60 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb277d90740 10 client.0 _ll_register_callbacks cb 0x7fb268009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb26ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5e0d640 0x5653c5e073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb26ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5e0d640 0x5653c5e073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54350/0 (socket says 172.21.15.31:54350) 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb26ed9d640 1 -- 172.21.15.31:0/2622357436 learned_addr learned my addr 172.21.15.31:0/2622357436 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb26dd9b640 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5dfea60 0x5653c5e06e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.957+0000 7fb26e59c640 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfe140 0x5653c5e06920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb26ed9d640 1 -- 172.21.15.31:0/2622357436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5dfea60 msgr2=0x5653c5e06e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb26ed9d640 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5dfea60 0x5653c5e06e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb26ed9d640 1 -- 172.21.15.31:0/2622357436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfe140 msgr2=0x5653c5e06920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb26ed9d640 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfe140 0x5653c5e06920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb26ed9d640 1 -- 172.21.15.31:0/2622357436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c5e147b0 con 0x5653c5e0d640 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb26ed9d640 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5e0d640 0x5653c5e073a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb26800ef40 tx=0x7fb268009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c94e28f132fb8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb255ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb255ffb640 1 -- 172.21.15.31:0/2622357436 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2680115f0 con 0x5653c5e0d640 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb255ffb640 1 -- 172.21.15.31:0/2622357436 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7fb268011790 con 0x5653c5e0d640 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb277d90740 1 -- 172.21.15.31:0/2622357436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2600bc180 con 0x5653c5e0d640 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb255ffb640 1 -- 172.21.15.31:0/2622357436 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb268038050 con 0x5653c5e0d640 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb277d90740 10 client.17073 fetch_fsmap learned FSMap version 40 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb277d90740 1 -- 172.21.15.31:0/2622357436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5653c5e15ac0 con 0x5653c5e0d640 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.958+0000 7fb255ffb640 1 -- 172.21.15.31:0/2622357436 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2600bc180 con 0x5653c5e0d640 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.959+0000 7fb255ffb640 1 -- 172.21.15.31:0/2622357436 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fb26800f040 con 0x5653c5e0d640 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.959+0000 7fb277d90740 10 client.17073 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.959+0000 7fb277d90740 10 client.17073 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.959+0000 7fb277d90740 20 client.17073 populate_metadata read hostname 'smithi031' 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.959+0000 7fb277d90740 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5653c5e83f30 0x5653c5e86350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.959+0000 7fb277d90740 4 client.17073 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:06:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.959+0000 7fb277d90740 1 -- 172.21.15.31:0/2622357436 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5653c5d6a540 con 0x5653c5e83f30 2024-04-25T02:06:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.959+0000 7fb26e59c640 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5653c5e83f30 0x5653c5e86350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.959+0000 7fb26e59c640 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5653c5e83f30 0x5653c5e86350 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eca3c76eb668f64f server_cookie=4befcf372a5c5f63 in_seq=0 out_seq=0 2024-04-25T02:06:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.959+0000 7fb255ffb640 10 client.17073 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb255ffb640 1 -- 172.21.15.31:0/2622357436 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5653c5d6a540 con 0x5653c5e83f30 2024-04-25T02:06:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb255ffb640 10 client.17073 handle_command_reply: tid=0 2024-04-25T02:06:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb277d90740 1 client.17073 shutdown 2024-04-25T02:06:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb277d90740 1 -- 172.21.15.31:0/2622357436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5e0d640 msgr2=0x5653c5e073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb277d90740 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5e0d640 0x5653c5e073a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb26800ef40 tx=0x7fb268009600 comp rx=0 tx=0).stop 2024-04-25T02:06:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb277d90740 1 -- 172.21.15.31:0/2622357436 shutdown_connections 2024-04-25T02:06:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb277d90740 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5653c5e83f30 0x5653c5e86350 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb277d90740 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653c5e0d640 0x5653c5e073a0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb277d90740 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5653c5dfea60 0x5653c5e06e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb255ffb640 0 client.17073 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb277d90740 1 --2- 172.21.15.31:0/2622357436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653c5dfe140 0x5653c5e06920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.990+0000 7fb277d90740 1 -- 172.21.15.31:0/2622357436 >> 172.21.15.31:0/2622357436 conn(0x5653c5dfa560 msgr2=0x5653c5df8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb277d90740 1 -- 172.21.15.31:0/2622357436 shutdown_connections 2024-04-25T02:06:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb277d90740 1 -- 172.21.15.31:0/2622357436 wait complete. 2024-04-25T02:06:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb277d90740 20 client.17073 trim_cache size 0 max 16384 2024-04-25T02:06:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:06:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:06:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:06:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:06:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 -- 172.21.15.31:0/3183260621 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2580a2da0 msgr2=0x7fb2580a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2580a2da0 0x7fb2580a5260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb268009fd0 tx=0x7fb268004ab0 comp rx=0 tx=0).stop 2024-04-25T02:06:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 -- 172.21.15.31:0/3183260621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270101740 msgr2=0x7fb270197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270101740 0x7fb270197ea0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb260005ab0 tx=0x7fb260003cd0 comp rx=0 tx=0).stop 2024-04-25T02:06:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 -- 172.21.15.31:0/3183260621 shutdown_connections 2024-04-25T02:06:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb2580a2da0 0x7fb2580a5260 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb270107750 0x7fb27019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:09.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270102100 0x7fb2701983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:09.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 --2- 172.21.15.31:0/3183260621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270101740 0x7fb270197ea0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:09.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 -- 172.21.15.31:0/3183260621 >> 172.21.15.31:0/3183260621 conn(0x7fb2700fd210 msgr2=0x7fb2700fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:09.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 -- 172.21.15.31:0/3183260621 shutdown_connections 2024-04-25T02:06:09.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:08.991+0000 7fb274fa2640 1 -- 172.21.15.31:0/3183260621 wait complete. 2024-04-25T02:06:09.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 241377 1.97 MB/sec execute 496 sec latency 4043.514 ms 2024-04-25T02:06:10.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:09 smithi040 ceph-mon[32418]: pgmap v363: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:06:10.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:09 smithi040 ceph-mon[32418]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T02:06:10.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:09 smithi134 ceph-mon[33098]: pgmap v363: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:06:10.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:09 smithi134 ceph-mon[33098]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T02:06:10.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:09 smithi031 ceph-mon[29204]: pgmap v363: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:06:10.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:09 smithi031 ceph-mon[29204]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T02:06:10.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 241469 1.97 MB/sec execute 497 sec latency 4765.759 ms 2024-04-25T02:06:11.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 241697 1.97 MB/sec execute 498 sec latency 44.943 ms 2024-04-25T02:06:12.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:11 smithi040 ceph-mon[32418]: pgmap v365: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 851 KiB/s wr, 92 op/s 2024-04-25T02:06:12.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:11 smithi134 ceph-mon[33098]: pgmap v365: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 851 KiB/s wr, 92 op/s 2024-04-25T02:06:12.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:11 smithi031 ceph-mon[29204]: pgmap v365: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 851 KiB/s wr, 92 op/s 2024-04-25T02:06:12.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 242072 1.96 MB/sec execute 499 sec latency 16.981 ms 2024-04-25T02:06:13.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 242623 1.97 MB/sec execute 500 sec latency 13.457 ms 2024-04-25T02:06:14.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:13 smithi040 ceph-mon[32418]: pgmap v366: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 760 KiB/s wr, 82 op/s 2024-04-25T02:06:14.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:13 smithi134 ceph-mon[33098]: pgmap v366: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 760 KiB/s wr, 82 op/s 2024-04-25T02:06:14.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:13 smithi031 ceph-mon[29204]: pgmap v366: 97 pgs: 97 active+clean; 134 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 760 KiB/s wr, 82 op/s 2024-04-25T02:06:14.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 243053 1.97 MB/sec execute 501 sec latency 22.213 ms 2024-04-25T02:06:15.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 243594 1.97 MB/sec execute 502 sec latency 22.138 ms 2024-04-25T02:06:16.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:15 smithi040 ceph-mon[32418]: pgmap v367: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 686 KiB/s wr, 85 op/s 2024-04-25T02:06:16.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:15 smithi134 ceph-mon[33098]: pgmap v367: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 686 KiB/s wr, 85 op/s 2024-04-25T02:06:16.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:15 smithi031 ceph-mon[29204]: pgmap v367: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 686 KiB/s wr, 85 op/s 2024-04-25T02:06:16.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 244161 1.97 MB/sec execute 503 sec latency 18.265 ms 2024-04-25T02:06:17.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 244608 1.97 MB/sec execute 504 sec latency 14.323 ms 2024-04-25T02:06:18.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:17 smithi040 ceph-mon[32418]: pgmap v368: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 686 KiB/s wr, 85 op/s 2024-04-25T02:06:18.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:17 smithi134 ceph-mon[33098]: pgmap v368: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 686 KiB/s wr, 85 op/s 2024-04-25T02:06:18.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:17 smithi031 ceph-mon[29204]: pgmap v368: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 686 KiB/s wr, 85 op/s 2024-04-25T02:06:18.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 244855 1.98 MB/sec execute 505 sec latency 614.996 ms 2024-04-25T02:06:19.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 244855 1.97 MB/sec execute 506 sec latency 1615.079 ms 2024-04-25T02:06:20.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:19 smithi040 ceph-mon[32418]: pgmap v369: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 686 KiB/s wr, 85 op/s 2024-04-25T02:06:20.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:19 smithi134 ceph-mon[33098]: pgmap v369: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 686 KiB/s wr, 85 op/s 2024-04-25T02:06:20.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:19 smithi031 ceph-mon[29204]: pgmap v369: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 686 KiB/s wr, 85 op/s 2024-04-25T02:06:20.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 244922 1.97 MB/sec execute 507 sec latency 2337.222 ms 2024-04-25T02:06:21.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 244922 1.96 MB/sec execute 508 sec latency 1031.537 ms 2024-04-25T02:06:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.091+0000 7f59a2d1b640 1 -- 172.21.15.31:0/927071004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f599c107740 msgr2=0x7f599c107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.091+0000 7f59a2d1b640 1 --2- 172.21.15.31:0/927071004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f599c107740 0x7f599c107b20 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f5998009900 tx=0x7f599802a410 comp rx=0 tx=0).stop 2024-04-25T02:06:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.092+0000 7f59a2d1b640 1 -- 172.21.15.31:0/927071004 shutdown_connections 2024-04-25T02:06:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.092+0000 7f59a2d1b640 1 --2- 172.21.15.31:0/927071004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f599c100600 0x7f599c10e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.092+0000 7f59a2d1b640 1 --2- 172.21.15.31:0/927071004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f599c0ffc40 0x7f599c1000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.092+0000 7f59a2d1b640 1 --2- 172.21.15.31:0/927071004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f599c107740 0x7f599c107b20 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.092+0000 7f59a2d1b640 1 -- 172.21.15.31:0/927071004 >> 172.21.15.31:0/927071004 conn(0x7f599c076b80 msgr2=0x7f599c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.092+0000 7f59a2d1b640 1 -- 172.21.15.31:0/927071004 shutdown_connections 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.092+0000 7f59a2d1b640 1 -- 172.21.15.31:0/927071004 wait complete. 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.092+0000 7f59a2d1b640 1 Processor -- start 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.092+0000 7f59a2d1b640 1 -- start start 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.092+0000 7f59a2d1b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f599c0ffc40 0x7f599c100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f59a2d1b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f599c100600 0x7f599c101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f59a2d1b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f599c107740 0x7f599c1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f59a2d1b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f599c111a30 con 0x7f599c100600 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f59a2d1b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f599c1118b0 con 0x7f599c107740 2024-04-25T02:06:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f59a2d1b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f599c111bb0 con 0x7f599c0ffc40 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5993fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f599c100600 0x7f599c101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5993fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f599c100600 0x7f599c101360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40592/0 (socket says 172.21.15.31:40592) 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5993fff640 1 -- 172.21.15.31:0/2499418199 learned_addr learned my addr 172.21.15.31:0/2499418199 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f599c0ffc40 0x7f599c100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f59a12b8640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f599c107740 0x7f599c1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5993fff640 1 -- 172.21.15.31:0/2499418199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f599c0ffc40 msgr2=0x7f599c100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5993fff640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f599c0ffc40 0x7f599c100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5993fff640 1 -- 172.21.15.31:0/2499418199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f599c107740 msgr2=0x7f599c1018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5993fff640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f599c107740 0x7f599c1018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5993fff640 1 -- 172.21.15.31:0/2499418199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599c1a6390 con 0x7f599c100600 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5993fff640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f599c100600 0x7f599c101360 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f598c00e9f0 tx=0x7f598c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5991ffb640 1 -- 172.21.15.31:0/2499418199 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c00cd80 con 0x7f599c100600 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f59a2d1b640 1 -- 172.21.15.31:0/2499418199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f599c1a6680 con 0x7f599c100600 2024-04-25T02:06:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5991ffb640 1 -- 172.21.15.31:0/2499418199 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f598c0108e0 con 0x7f599c100600 2024-04-25T02:06:22.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.093+0000 7f5991ffb640 1 -- 172.21.15.31:0/2499418199 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c018920 con 0x7f599c100600 2024-04-25T02:06:22.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.094+0000 7f59a2d1b640 1 -- 172.21.15.31:0/2499418199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f599c1a6c20 con 0x7f599c100600 2024-04-25T02:06:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.096+0000 7f5991ffb640 1 -- 172.21.15.31:0/2499418199 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f598c010a90 con 0x7f599c100600 2024-04-25T02:06:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.096+0000 7f59a5a2e740 1 Processor -- start 2024-04-25T02:06:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.096+0000 7f59a5a2e740 1 -- start start 2024-04-25T02:06:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.096+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5240 0x5612760e5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760e5b60 0x5612760efcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760f0230 0x5612760f2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561276049cb0 con 0x5612760f0230 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561275ff25b0 con 0x5612760e5b60 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561276045c70 con 0x5612760e5240 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f5991ffb640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f59840a2da0 0x7f59840a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f5993fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760f0230 0x5612760f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f5993fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760f0230 0x5612760f2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40594/0 (socket says 172.21.15.31:40594) 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f5993fff640 1 -- 172.21.15.31:0/3130558670 learned_addr learned my addr 172.21.15.31:0/3130558670 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f59a12b8640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f59840a2da0 0x7f59840a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/3130558670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760e5b60 0x5612760efcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f59a12b8640 1 --2- 172.21.15.31:0/3130558670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5240 0x5612760e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f5991ffb640 1 -- 172.21.15.31:0/2499418199 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7f598c0cf7b0 con 0x7f599c100600 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f5993fff640 1 -- 172.21.15.31:0/3130558670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5240 msgr2=0x5612760e5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f5993fff640 1 --2- 172.21.15.31:0/3130558670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5240 0x5612760e5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f5993fff640 1 -- 172.21.15.31:0/3130558670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760e5b60 msgr2=0x5612760efcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f5993fff640 1 --2- 172.21.15.31:0/3130558670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760e5b60 0x5612760efcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f5993fff640 1 -- 172.21.15.31:0/3130558670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561275d5e210 con 0x5612760f0230 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/3130558670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760e5b60 0x5612760efcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f59a12b8640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f59840a2da0 0x7f59840a5260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5994009fd0 tx=0x7f5994009d20 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.097+0000 7f5993fff640 1 --2- 172.21.15.31:0/3130558670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760f0230 0x5612760f2620 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f598c0ccf80 tx=0x7f598c09bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aeb0a2f8e29aaae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f5982ffd640 1 -- 172.21.15.31:0/3130558670 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c09bcf0 con 0x5612760f0230 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f5982ffd640 1 -- 172.21.15.31:0/3130558670 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f598c0a00a0 con 0x5612760f0230 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f5982ffd640 1 -- 172.21.15.31:0/3130558670 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c0cd300 con 0x5612760f0230 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f59a5a2e740 1 -- 172.21.15.31:0/3130558670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760f0230 msgr2=0x5612760f2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/3130558670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760f0230 0x5612760f2620 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f598c0ccf80 tx=0x7f598c09bbf0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f59a5a2e740 1 -- 172.21.15.31:0/3130558670 shutdown_connections 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/3130558670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760f0230 0x5612760f2620 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/3130558670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760e5b60 0x5612760efcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/3130558670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5240 0x5612760e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f59a5a2e740 1 -- 172.21.15.31:0/3130558670 >> 172.21.15.31:0/3130558670 conn(0x5612760e1560 msgr2=0x5612760e1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f59a5a2e740 1 -- 172.21.15.31:0/3130558670 shutdown_connections 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.098+0000 7f59a5a2e740 1 -- 172.21.15.31:0/3130558670 wait complete. 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.099+0000 7f59a5a2e740 1 -- start start 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.099+0000 7f59a5a2e740 1 Processor -- start 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.099+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760ed210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.099+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5b60 0x5612760ed750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.099+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760edc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.099+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5612760f2f60 con 0x5612760e5240 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.099+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561276005b50 con 0x5612760f0230 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.099+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5612760f30e0 con 0x5612760e5b60 2024-04-25T02:06:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.099+0000 7f59a5a2e740 10 client.0 _ll_register_callbacks cb 0x561276040ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.099+0000 7f59a0ab7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760ed210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a0ab7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760ed210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40608/0 (socket says 172.21.15.31:40608) 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a0ab7640 1 -- 172.21.15.31:0/347301514 learned_addr learned my addr 172.21.15.31:0/347301514 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a12b8640 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760edc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f5993fff640 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5b60 0x5612760ed750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a0ab7640 1 -- 172.21.15.31:0/347301514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5b60 msgr2=0x5612760ed750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5b60 0x5612760ed750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a0ab7640 1 -- 172.21.15.31:0/347301514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 msgr2=0x5612760edc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760edc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a0ab7640 1 -- 172.21.15.31:0/347301514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561275e2a620 con 0x5612760e5240 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760ed210 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f5998009900 tx=0x7f5998004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46c648d9ecc3fa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f597bfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f597bfff640 1 -- 172.21.15.31:0/347301514 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998035040 con 0x5612760e5240 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f597bfff640 1 -- 172.21.15.31:0/347301514 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7f5998005510 con 0x5612760e5240 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a5a2e740 1 -- 172.21.15.31:0/347301514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f598c0a00a0 con 0x5612760e5240 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f597bfff640 1 -- 172.21.15.31:0/347301514 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5998004770 con 0x5612760e5240 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a5a2e740 10 client.17085 fetch_fsmap learned FSMap version 40 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.100+0000 7f59a5a2e740 1 -- 172.21.15.31:0/347301514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5612760ff200 con 0x5612760e5240 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.101+0000 7f597bfff640 1 -- 172.21.15.31:0/347301514 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f598c0a00a0 con 0x5612760e5240 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.101+0000 7f597bfff640 1 -- 172.21.15.31:0/347301514 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f5998003b90 con 0x5612760e5240 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.101+0000 7f59a5a2e740 10 client.17085 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.101+0000 7f59a5a2e740 10 client.17085 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.101+0000 7f59a5a2e740 20 client.17085 populate_metadata read hostname 'smithi031' 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.101+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561276100710 0x561276102b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.101+0000 7f59a5a2e740 4 client.17085 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.101+0000 7f59a5a2e740 1 -- 172.21.15.31:0/347301514 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612760571f0 con 0x561276100710 2024-04-25T02:06:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.101+0000 7f5993fff640 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561276100710 0x561276102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f5993fff640 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561276100710 0x561276102b60 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b997e1885e767bb server_cookie=b8a0c61ba3dc62ab in_seq=0 out_seq=0 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f597bfff640 10 client.17085 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f597bfff640 1 -- 172.21.15.31:0/347301514 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5612760571f0 con 0x561276100710 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f597bfff640 10 client.17085 handle_command_reply: tid=0 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f59a5a2e740 1 client.17085 shutdown 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f59a5a2e740 1 -- 172.21.15.31:0/347301514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 msgr2=0x5612760ed210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760ed210 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f5998009900 tx=0x7f5998004440 comp rx=0 tx=0).stop 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f59a5a2e740 1 -- 172.21.15.31:0/347301514 shutdown_connections 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561276100710 0x561276102b60 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760edc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5b60 0x5612760ed750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f597bfff640 0 client.17085 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/347301514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760ed210 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.102+0000 7f59a5a2e740 1 -- 172.21.15.31:0/347301514 >> 172.21.15.31:0/347301514 conn(0x5612760e1560 msgr2=0x5612760e7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.103+0000 7f59a5a2e740 1 -- 172.21.15.31:0/347301514 shutdown_connections 2024-04-25T02:06:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.103+0000 7f59a5a2e740 1 -- 172.21.15.31:0/347301514 wait complete. 2024-04-25T02:06:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.103+0000 7f59a5a2e740 20 client.17085 trim_cache size 0 max 16384 2024-04-25T02:06:22.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a5a2e740 1 Processor -- start 2024-04-25T02:06:22.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a5a2e740 1 -- start start 2024-04-25T02:06:22.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760fbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5b60 0x5612760ec370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760e79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f599c1af420 con 0x5612760e5240 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f599c1b10e0 con 0x5612760f0230 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f599c1afaf0 con 0x5612760e5b60 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a0ab7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760fbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a0ab7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760fbe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40612/0 (socket says 172.21.15.31:40612) 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a0ab7640 1 -- 172.21.15.31:0/3975337790 learned_addr learned my addr 172.21.15.31:0/3975337790 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a0ab7640 1 -- 172.21.15.31:0/3975337790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5b60 msgr2=0x5612760ec370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f5993fff640 1 --2- 172.21.15.31:0/3975337790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5b60 0x5612760ec370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a12b8640 1 --2- 172.21.15.31:0/3975337790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760e79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/3975337790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5b60 0x5612760ec370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a0ab7640 1 -- 172.21.15.31:0/3975337790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 msgr2=0x5612760e79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/3975337790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760e79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f59a0ab7640 1 -- 172.21.15.31:0/3975337790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612760e5620 con 0x5612760e5240 2024-04-25T02:06:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.118+0000 7f5993fff640 1 --2- 172.21.15.31:0/3975337790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5b60 0x5612760ec370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/3975337790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760fbe20 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5998005ab0 tx=0x7f599802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3cd81dca30939d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59827fc640 1 -- 172.21.15.31:0/3975337790 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998053030 con 0x5612760e5240 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59827fc640 1 -- 172.21.15.31:0/3975337790 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5998041070 con 0x5612760e5240 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59827fc640 1 -- 172.21.15.31:0/3975337790 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998035070 con 0x5612760e5240 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59a5a2e740 1 -- 172.21.15.31:0/3975337790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 msgr2=0x5612760fbe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/3975337790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760fbe20 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5998005ab0 tx=0x7f599802a410 comp rx=0 tx=0).stop 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59a5a2e740 1 -- 172.21.15.31:0/3975337790 shutdown_connections 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/3975337790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760e79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/3975337790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5b60 0x5612760ec370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/3975337790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5240 0x5612760fbe20 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59a5a2e740 1 -- 172.21.15.31:0/3975337790 >> 172.21.15.31:0/3975337790 conn(0x5612760e1560 msgr2=0x5612760dfd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.119+0000 7f59a5a2e740 1 -- 172.21.15.31:0/3975337790 shutdown_connections 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f59a5a2e740 1 -- 172.21.15.31:0/3975337790 wait complete. 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f59a5a2e740 1 -- start start 2024-04-25T02:06:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f59a5a2e740 1 Processor -- start 2024-04-25T02:06:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5240 0x5612760f8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5b60 0x5612760e9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f59a5a2e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760e9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f599c1b0a10 con 0x5612760e5b60 2024-04-25T02:06:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f599c1b0280 con 0x5612760f0230 2024-04-25T02:06:22.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f59a5a2e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f599c1ad420 con 0x5612760e5240 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f59a5a2e740 10 client.0 _ll_register_callbacks cb 0x7f5998009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f5993fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5b60 0x5612760e9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f5993fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5b60 0x5612760e9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40614/0 (socket says 172.21.15.31:40614) 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.120+0000 7f5993fff640 1 -- 172.21.15.31:0/1293212002 learned_addr learned my addr 172.21.15.31:0/1293212002 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5240 0x5612760f8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f59a12b8640 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760e9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f5993fff640 1 -- 172.21.15.31:0/1293212002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5240 msgr2=0x5612760f8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f5993fff640 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5240 0x5612760f8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f5993fff640 1 -- 172.21.15.31:0/1293212002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 msgr2=0x5612760e9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f5993fff640 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760e9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f5993fff640 1 -- 172.21.15.31:0/1293212002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612760ea600 con 0x5612760e5b60 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f5993fff640 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5b60 0x5612760e9a80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f598c0986c0 tx=0x7f598c098000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0846977686abf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f597bfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f597bfff640 1 -- 172.21.15.31:0/1293212002 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c0a00a0 con 0x5612760e5b60 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f597bfff640 1 -- 172.21.15.31:0/1293212002 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7f598c0cd1a0 con 0x5612760e5b60 2024-04-25T02:06:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f59a5a2e740 1 -- 172.21.15.31:0/1293212002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5998041070 con 0x5612760e5b60 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f597bfff640 1 -- 172.21.15.31:0/1293212002 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f598c098360 con 0x5612760e5b60 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f59a5a2e740 10 client.17091 fetch_fsmap learned FSMap version 40 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f59a5a2e740 1 -- 172.21.15.31:0/1293212002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x561276166af0 con 0x5612760e5b60 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.121+0000 7f597bfff640 1 -- 172.21.15.31:0/1293212002 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5998041070 con 0x5612760e5b60 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.122+0000 7f597bfff640 1 -- 172.21.15.31:0/1293212002 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f598c0a4bb0 con 0x5612760e5b60 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.122+0000 7f59a5a2e740 10 client.17091 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.122+0000 7f59a5a2e740 10 client.17091 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.122+0000 7f59a5a2e740 20 client.17091 populate_metadata read hostname 'smithi031' 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.122+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561276166e30 0x561276169220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.122+0000 7f59a5a2e740 4 client.17091 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.122+0000 7f59a5a2e740 1 -- 172.21.15.31:0/1293212002 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561276051540 con 0x561276166e30 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.122+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561276166e30 0x561276169220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.122+0000 7f59a0ab7640 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561276166e30 0x561276169220 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb581574ba74e6c6 server_cookie=643e84def8123488 in_seq=0 out_seq=0 2024-04-25T02:06:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.122+0000 7f597bfff640 10 client.17091 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.155+0000 7f597bfff640 1 -- 172.21.15.31:0/1293212002 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561276051540 con 0x561276166e30 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.155+0000 7f597bfff640 10 client.17091 handle_command_reply: tid=0 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 1 client.17091 shutdown 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 1 -- 172.21.15.31:0/1293212002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5b60 msgr2=0x5612760e9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5b60 0x5612760e9a80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f598c0986c0 tx=0x7f598c098000 comp rx=0 tx=0).stop 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 1 -- 172.21.15.31:0/1293212002 shutdown_connections 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561276166e30 0x561276169220 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5612760f0230 0x5612760e9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5612760e5b60 0x5612760e9a80 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 1 --2- 172.21.15.31:0/1293212002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5612760e5240 0x5612760f8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f597bfff640 0 client.17091 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 1 -- 172.21.15.31:0/1293212002 >> 172.21.15.31:0/1293212002 conn(0x5612760e1560 msgr2=0x5612760dfd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 1 -- 172.21.15.31:0/1293212002 shutdown_connections 2024-04-25T02:06:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 1 -- 172.21.15.31:0/1293212002 wait complete. 2024-04-25T02:06:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.156+0000 7f59a5a2e740 20 client.17091 trim_cache size 0 max 16384 2024-04-25T02:06:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:06:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:06:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:06:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:06:22.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 -- 172.21.15.31:0/2499418199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f59840a2da0 msgr2=0x7f59840a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:22.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f59840a2da0 0x7f59840a5260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5994009fd0 tx=0x7f5994009d20 comp rx=0 tx=0).stop 2024-04-25T02:06:22.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 -- 172.21.15.31:0/2499418199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f599c100600 msgr2=0x7f599c101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:22.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f599c100600 0x7f599c101360 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f598c00e9f0 tx=0x7f598c00eeb0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 -- 172.21.15.31:0/2499418199 shutdown_connections 2024-04-25T02:06:22.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f59840a2da0 0x7f59840a5260 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f599c107740 0x7f599c1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f599c100600 0x7f599c101360 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 --2- 172.21.15.31:0/2499418199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f599c0ffc40 0x7f599c100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:22.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 -- 172.21.15.31:0/2499418199 >> 172.21.15.31:0/2499418199 conn(0x7f599c076b80 msgr2=0x7f599c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:22.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 -- 172.21.15.31:0/2499418199 shutdown_connections 2024-04-25T02:06:22.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:22.157+0000 7f59a2d1b640 1 -- 172.21.15.31:0/2499418199 wait complete. 2024-04-25T02:06:22.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:21 smithi040 ceph-mon[32418]: pgmap v370: 97 pgs: 97 active+clean; 139 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-25T02:06:22.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:21 smithi134 ceph-mon[33098]: pgmap v370: 97 pgs: 97 active+clean; 139 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-25T02:06:22.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:21 smithi031 ceph-mon[29204]: pgmap v370: 97 pgs: 97 active+clean; 139 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-25T02:06:22.328 INFO:tasks.workunit.client.0.smithi031.stdout: 1 244922 1.96 MB/sec execute 509 sec latency 2031.638 ms 2024-04-25T02:06:23.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 244922 1.96 MB/sec execute 510 sec latency 3031.671 ms 2024-04-25T02:06:24.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:23 smithi040 ceph-mon[32418]: pgmap v371: 97 pgs: 97 active+clean; 139 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-04-25T02:06:24.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:23 smithi134 ceph-mon[33098]: pgmap v371: 97 pgs: 97 active+clean; 139 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-04-25T02:06:24.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:23 smithi031 ceph-mon[29204]: pgmap v371: 97 pgs: 97 active+clean; 139 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-04-25T02:06:24.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 244922 1.95 MB/sec execute 511 sec latency 4031.736 ms 2024-04-25T02:06:25.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 245018 1.95 MB/sec execute 512 sec latency 4747.423 ms 2024-04-25T02:06:26.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:25 smithi040 ceph-mon[32418]: pgmap v372: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:26.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:25 smithi134 ceph-mon[33098]: pgmap v372: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:26.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:25 smithi031 ceph-mon[29204]: pgmap v372: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:26.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 245247 1.95 MB/sec execute 513 sec latency 14.510 ms 2024-04-25T02:06:27.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 245652 1.94 MB/sec execute 514 sec latency 17.246 ms 2024-04-25T02:06:28.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:27 smithi040 ceph-mon[32418]: pgmap v373: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 574 KiB/s wr, 52 op/s 2024-04-25T02:06:28.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:27 smithi134 ceph-mon[33098]: pgmap v373: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 574 KiB/s wr, 52 op/s 2024-04-25T02:06:28.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:27 smithi031 ceph-mon[29204]: pgmap v373: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 574 KiB/s wr, 52 op/s 2024-04-25T02:06:28.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 246180 1.95 MB/sec execute 515 sec latency 18.583 ms 2024-04-25T02:06:29.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 246618 1.95 MB/sec execute 516 sec latency 23.869 ms 2024-04-25T02:06:30.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:29 smithi040 ceph-mon[32418]: pgmap v374: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 574 KiB/s wr, 52 op/s 2024-04-25T02:06:30.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:29 smithi134 ceph-mon[33098]: pgmap v374: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 574 KiB/s wr, 52 op/s 2024-04-25T02:06:30.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:29 smithi031 ceph-mon[29204]: pgmap v374: 97 pgs: 97 active+clean; 135 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 574 KiB/s wr, 52 op/s 2024-04-25T02:06:30.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 247149 1.95 MB/sec execute 517 sec latency 23.500 ms 2024-04-25T02:06:31.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 247702 1.95 MB/sec execute 518 sec latency 19.447 ms 2024-04-25T02:06:32.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:31 smithi040 ceph-mon[32418]: pgmap v375: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:32.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:31 smithi134 ceph-mon[33098]: pgmap v375: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:32.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:31 smithi031 ceph-mon[29204]: pgmap v375: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:32.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 248123 1.95 MB/sec execute 519 sec latency 19.553 ms 2024-04-25T02:06:33.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 248402 1.96 MB/sec execute 520 sec latency 530.804 ms 2024-04-25T02:06:34.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:33 smithi040 ceph-mon[32418]: pgmap v376: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 560 KiB/s wr, 71 op/s 2024-04-25T02:06:34.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:33 smithi134 ceph-mon[33098]: pgmap v376: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 560 KiB/s wr, 71 op/s 2024-04-25T02:06:34.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:33 smithi031 ceph-mon[29204]: pgmap v376: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 560 KiB/s wr, 71 op/s 2024-04-25T02:06:34.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 248402 1.95 MB/sec execute 521 sec latency 1530.858 ms 2024-04-25T02:06:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.251+0000 7fe569064640 1 -- 172.21.15.31:0/736489837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5641032f0 msgr2=0x7fe56410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.251+0000 7fe569064640 1 --2- 172.21.15.31:0/736489837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5641032f0 0x7fe56410b7d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fe558009900 tx=0x7fe55802a510 comp rx=0 tx=0).stop 2024-04-25T02:06:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.251+0000 7fe569064640 1 -- 172.21.15.31:0/736489837 shutdown_connections 2024-04-25T02:06:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.251+0000 7fe569064640 1 --2- 172.21.15.31:0/736489837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe56410bd10 0x7fe56410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.251+0000 7fe569064640 1 --2- 172.21.15.31:0/736489837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5641032f0 0x7fe56410b7d0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.251+0000 7fe569064640 1 --2- 172.21.15.31:0/736489837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5641029d0 0x7fe564102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.251+0000 7fe569064640 1 -- 172.21.15.31:0/736489837 >> 172.21.15.31:0/736489837 conn(0x7fe5640777c0 msgr2=0x7fe564077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.251+0000 7fe569064640 1 -- 172.21.15.31:0/736489837 shutdown_connections 2024-04-25T02:06:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.251+0000 7fe569064640 1 -- 172.21.15.31:0/736489837 wait complete. 2024-04-25T02:06:35.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe569064640 1 Processor -- start 2024-04-25T02:06:35.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe569064640 1 -- start start 2024-04-25T02:06:35.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe569064640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5641029d0 0x7fe564100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe569064640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5641032f0 0x7fe564100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe569064640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe56410bd10 0x7fe5641010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe569064640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe564111a00 con 0x7fe5641029d0 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe569064640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe564111880 con 0x7fe56410bd10 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe569064640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe564111b80 con 0x7fe5641032f0 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe562d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5641029d0 0x7fe564100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe562d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5641029d0 0x7fe564100620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51798/0 (socket says 172.21.15.31:51798) 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe562d9d640 1 -- 172.21.15.31:0/3869176907 learned_addr learned my addr 172.21.15.31:0/3869176907 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe56359e640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe56410bd10 0x7fe5641010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.252+0000 7fe56259c640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5641032f0 0x7fe564100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe562d9d640 1 -- 172.21.15.31:0/3869176907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5641032f0 msgr2=0x7fe564100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe562d9d640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5641032f0 0x7fe564100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe562d9d640 1 -- 172.21.15.31:0/3869176907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe56410bd10 msgr2=0x7fe5641010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:35.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe562d9d640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe56410bd10 0x7fe5641010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe562d9d640 1 -- 172.21.15.31:0/3869176907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5641a63e0 con 0x7fe5641029d0 2024-04-25T02:06:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe56259c640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5641032f0 0x7fe564100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:06:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe562d9d640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5641029d0 0x7fe564100620 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe55400e9f0 tx=0x7fe55400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe54bfff640 1 -- 172.21.15.31:0/3869176907 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55400cd80 con 0x7fe5641029d0 2024-04-25T02:06:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe54bfff640 1 -- 172.21.15.31:0/3869176907 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5540108e0 con 0x7fe5641029d0 2024-04-25T02:06:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe569064640 1 -- 172.21.15.31:0/3869176907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5641a66d0 con 0x7fe5641029d0 2024-04-25T02:06:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe54bfff640 1 -- 172.21.15.31:0/3869176907 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe554018920 con 0x7fe5641029d0 2024-04-25T02:06:35.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.253+0000 7fe569064640 1 -- 172.21.15.31:0/3869176907 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5641a6bf0 con 0x7fe5641029d0 2024-04-25T02:06:35.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.255+0000 7fe54bfff640 1 -- 172.21.15.31:0/3869176907 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe554010a90 con 0x7fe5641029d0 2024-04-25T02:06:35.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56bd74740 1 Processor -- start 2024-04-25T02:06:35.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56bd74740 1 -- start start 2024-04-25T02:06:35.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe54bfff640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe54c0a2dd0 0x7fe54c0a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318240 0x55c14c318620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 0x55c14c322cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c325620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56bd74740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c14c27ccb0 con 0x55c14c323230 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56bd74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c14c2255b0 con 0x55c14c318b60 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56bd74740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c14c278c70 con 0x55c14c318240 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56359e640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe54c0a2dd0 0x7fe54c0a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe54bfff640 1 -- 172.21.15.31:0/3869176907 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7fe5540cf7b0 con 0x7fe5641029d0 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe562d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318240 0x55c14c318620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c325620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 0x55c14c322cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 0x55c14c322cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:37498/0 (socket says 172.21.15.31:37498) 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c325620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51806/0 (socket says 172.21.15.31:51806) 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.256+0000 7fe56259c640 1 -- 172.21.15.31:0/2568635009 learned_addr learned my addr 172.21.15.31:0/2568635009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.257+0000 7fe56359e640 1 -- 172.21.15.31:0/2568635009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318240 msgr2=0x55c14c318620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.257+0000 7fe56359e640 1 --2- 172.21.15.31:0/2568635009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318240 0x55c14c318620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.257+0000 7fe56359e640 1 -- 172.21.15.31:0/2568635009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 msgr2=0x55c14c322cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.257+0000 7fe56359e640 1 --2- 172.21.15.31:0/2568635009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 0x55c14c322cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.257+0000 7fe56359e640 1 -- 172.21.15.31:0/2568635009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c14bf91210 con 0x55c14c323230 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.257+0000 7fe56259c640 1 --2- 172.21.15.31:0/2568635009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 0x55c14c322cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.257+0000 7fe56359e640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe54c0a2dd0 0x7fe54c0a5290 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009fd0 tx=0x7fe55c009ec0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.257+0000 7fe56359e640 1 --2- 172.21.15.31:0/2568635009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c325620 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe55c008b20 tx=0x7fe55c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d22049f543dd7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.257+0000 7fe5497fa640 1 -- 172.21.15.31:0/2568635009 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c020070 con 0x55c14c323230 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.257+0000 7fe5497fa640 1 -- 172.21.15.31:0/2568635009 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe55c019be0 con 0x55c14c323230 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.257+0000 7fe5497fa640 1 -- 172.21.15.31:0/2568635009 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c014040 con 0x55c14c323230 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.258+0000 7fe56bd74740 1 -- 172.21.15.31:0/2568635009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 msgr2=0x55c14c325620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.258+0000 7fe56bd74740 1 --2- 172.21.15.31:0/2568635009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c325620 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe55c008b20 tx=0x7fe55c01a000 comp rx=0 tx=0).stop 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.258+0000 7fe56bd74740 1 -- 172.21.15.31:0/2568635009 shutdown_connections 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.258+0000 7fe56bd74740 1 --2- 172.21.15.31:0/2568635009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c325620 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.258+0000 7fe56bd74740 1 --2- 172.21.15.31:0/2568635009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 0x55c14c322cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.258+0000 7fe56bd74740 1 --2- 172.21.15.31:0/2568635009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318240 0x55c14c318620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.258+0000 7fe56bd74740 1 -- 172.21.15.31:0/2568635009 >> 172.21.15.31:0/2568635009 conn(0x55c14c314560 msgr2=0x55c14c314940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.258+0000 7fe56bd74740 1 -- 172.21.15.31:0/2568635009 shutdown_connections 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.258+0000 7fe56bd74740 1 -- 172.21.15.31:0/2568635009 wait complete. 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.258+0000 7fe56bd74740 1 -- start start 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.258+0000 7fe56bd74740 1 Processor -- start 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318240 0x55c14c3201f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 0x55c14c320730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c320c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56bd74740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c14c325fc0 con 0x55c14c323230 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56bd74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c14c238b50 con 0x55c14c318b60 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56bd74740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c14c326140 con 0x55c14c318240 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56bd74740 10 client.0 _ll_register_callbacks cb 0x55c14c273ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c320c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c320c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51810/0 (socket says 172.21.15.31:51810) 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56359e640 1 -- 172.21.15.31:0/2144074662 learned_addr learned my addr 172.21.15.31:0/2144074662 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 0x55c14c320730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe562d9d640 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318240 0x55c14c3201f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56359e640 1 -- 172.21.15.31:0/2144074662 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318240 msgr2=0x55c14c3201f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56359e640 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318240 0x55c14c3201f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56359e640 1 -- 172.21.15.31:0/2144074662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 msgr2=0x55c14c320730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56359e640 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 0x55c14c320730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56359e640 1 -- 172.21.15.31:0/2144074662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c14c05d620 con 0x55c14c323230 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe56359e640 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c320c70 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fe55c008cc0 tx=0x7fe55c01a9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b31623521a55d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe546ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.259+0000 7fe546ff5640 1 -- 172.21.15.31:0/2144074662 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c020040 con 0x55c14c323230 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe546ff5640 1 -- 172.21.15.31:0/2144074662 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7fe55c01a3f0 con 0x55c14c323230 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe56bd74740 1 -- 172.21.15.31:0/2144074662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe55c019be0 con 0x55c14c323230 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe546ff5640 1 -- 172.21.15.31:0/2144074662 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe55c01a5f0 con 0x55c14c323230 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe56bd74740 10 client.17103 fetch_fsmap learned FSMap version 40 2024-04-25T02:06:35.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe56bd74740 1 -- 172.21.15.31:0/2144074662 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c14c332250 con 0x55c14c323230 2024-04-25T02:06:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe546ff5640 1 -- 172.21.15.31:0/2144074662 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe55c019be0 con 0x55c14c323230 2024-04-25T02:06:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe546ff5640 1 -- 172.21.15.31:0/2144074662 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fe55c024400 con 0x55c14c323230 2024-04-25T02:06:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe56bd74740 10 client.17103 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:06:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe56bd74740 10 client.17103 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe56bd74740 20 client.17103 populate_metadata read hostname 'smithi031' 2024-04-25T02:06:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe56bd74740 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c14c333660 0x55c14c335a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe56bd74740 4 client.17103 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:06:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.260+0000 7fe56bd74740 1 -- 172.21.15.31:0/2144074662 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c14c28a1f0 con 0x55c14c333660 2024-04-25T02:06:35.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.261+0000 7fe562d9d640 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c14c333660 0x55c14c335a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.261+0000 7fe546ff5640 10 client.17103 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:35.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.261+0000 7fe562d9d640 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c14c333660 0x55c14c335a50 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a68285acec245fc server_cookie=1337618e64ee740e in_seq=0 out_seq=0 2024-04-25T02:06:35.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe546ff5640 1 -- 172.21.15.31:0/2144074662 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c14c28a1f0 con 0x55c14c333660 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe546ff5640 10 client.17103 handle_command_reply: tid=0 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 1 client.17103 shutdown 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 1 -- 172.21.15.31:0/2144074662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 msgr2=0x55c14c320c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c320c70 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fe55c008cc0 tx=0x7fe55c01a9f0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 1 -- 172.21.15.31:0/2144074662 shutdown_connections 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c14c333660 0x55c14c335a50 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe546ff5640 0 client.17103 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c323230 0x55c14c320c70 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318b60 0x55c14c320730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 1 --2- 172.21.15.31:0/2144074662 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318240 0x55c14c3201f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 1 -- 172.21.15.31:0/2144074662 >> 172.21.15.31:0/2144074662 conn(0x55c14c314560 msgr2=0x55c14c31a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 1 -- 172.21.15.31:0/2144074662 shutdown_connections 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 1 -- 172.21.15.31:0/2144074662 wait complete. 2024-04-25T02:06:35.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.262+0000 7fe56bd74740 20 client.17103 trim_cache size 0 max 16384 2024-04-25T02:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe56bd74740 1 Processor -- start 2024-04-25T02:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe56bd74740 1 -- start start 2024-04-25T02:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318240 0x55c14c32eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318b60 0x55c14c320ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c323230 0x55c14c313440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe56bd74740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe5641af5b0 con 0x55c14c318b60 2024-04-25T02:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe56bd74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe5641b1270 con 0x55c14c318240 2024-04-25T02:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe56bd74740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe5641afc80 con 0x55c14c323230 2024-04-25T02:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe56259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318b60 0x55c14c320ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe562d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318240 0x55c14c32eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe56259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318b60 0x55c14c320ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51812/0 (socket says 172.21.15.31:51812) 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.279+0000 7fe56259c640 1 -- 172.21.15.31:0/1028770130 learned_addr learned my addr 172.21.15.31:0/1028770130 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.280+0000 7fe56359e640 1 --2- 172.21.15.31:0/1028770130 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c323230 0x55c14c313440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.280+0000 7fe562d9d640 1 -- 172.21.15.31:0/1028770130 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c323230 msgr2=0x55c14c313440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.280+0000 7fe562d9d640 1 --2- 172.21.15.31:0/1028770130 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c323230 0x55c14c313440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.280+0000 7fe562d9d640 1 -- 172.21.15.31:0/1028770130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318b60 msgr2=0x55c14c320ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.280+0000 7fe562d9d640 1 --2- 172.21.15.31:0/1028770130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318b60 0x55c14c320ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.280+0000 7fe562d9d640 1 -- 172.21.15.31:0/1028770130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c14c325620 con 0x55c14c318240 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.280+0000 7fe56259c640 1 --2- 172.21.15.31:0/1028770130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318b60 0x55c14c320ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.280+0000 7fe562d9d640 1 --2- 172.21.15.31:0/1028770130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318240 0x55c14c32eed0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe5540ccf20 tx=0x7fe554000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc265c9004784667 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.280+0000 7fe548ff9640 1 -- 172.21.15.31:0/1028770130 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5540a4960 con 0x55c14c318240 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.280+0000 7fe548ff9640 1 -- 172.21.15.31:0/1028770130 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe554099d90 con 0x55c14c318240 2024-04-25T02:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.280+0000 7fe548ff9640 1 -- 172.21.15.31:0/1028770130 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5540a44d0 con 0x55c14c318240 2024-04-25T02:06:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.281+0000 7fe56bd74740 1 -- 172.21.15.31:0/1028770130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318240 msgr2=0x55c14c32eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.281+0000 7fe56bd74740 1 --2- 172.21.15.31:0/1028770130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318240 0x55c14c32eed0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe5540ccf20 tx=0x7fe554000bc0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.281+0000 7fe56bd74740 1 -- 172.21.15.31:0/1028770130 shutdown_connections 2024-04-25T02:06:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.281+0000 7fe56bd74740 1 --2- 172.21.15.31:0/1028770130 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c323230 0x55c14c313440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.281+0000 7fe56bd74740 1 --2- 172.21.15.31:0/1028770130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318b60 0x55c14c320ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.281+0000 7fe56bd74740 1 --2- 172.21.15.31:0/1028770130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c318240 0x55c14c32eed0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.281+0000 7fe56bd74740 1 -- 172.21.15.31:0/1028770130 >> 172.21.15.31:0/1028770130 conn(0x55c14c314560 msgr2=0x55c14c312d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.281+0000 7fe56bd74740 1 -- 172.21.15.31:0/1028770130 shutdown_connections 2024-04-25T02:06:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.281+0000 7fe56bd74740 1 -- 172.21.15.31:0/1028770130 wait complete. 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.281+0000 7fe56bd74740 1 -- start start 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.281+0000 7fe56bd74740 1 Processor -- start 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318240 0x55c14c32b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318b60 0x55c14c31cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe56bd74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c323230 0x55c14c31d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe56bd74740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe5641b0ba0 con 0x55c14c318240 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe56bd74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe5641b0410 con 0x55c14c323230 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe56bd74740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe5641ad5b0 con 0x55c14c318b60 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe56bd74740 10 client.0 _ll_register_callbacks cb 0x7fe554005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe562d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318240 0x55c14c32b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe562d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318240 0x55c14c32b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51820/0 (socket says 172.21.15.31:51820) 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe562d9d640 1 -- 172.21.15.31:0/4070887618 learned_addr learned my addr 172.21.15.31:0/4070887618 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe56359e640 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c323230 0x55c14c31d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe56259c640 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318b60 0x55c14c31cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe562d9d640 1 -- 172.21.15.31:0/4070887618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318b60 msgr2=0x55c14c31cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe562d9d640 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318b60 0x55c14c31cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe562d9d640 1 -- 172.21.15.31:0/4070887618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c323230 msgr2=0x55c14c31d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe562d9d640 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c323230 0x55c14c31d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe562d9d640 1 -- 172.21.15.31:0/4070887618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c14c31d6a0 con 0x55c14c318240 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe56359e640 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c323230 0x55c14c31d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe562d9d640 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318240 0x55c14c32b640 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe5540a4430 tx=0x7fe554098000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58688cdb0e87179 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe546ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe546ff5640 1 -- 172.21.15.31:0/4070887618 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5540a4960 con 0x55c14c318240 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe546ff5640 1 -- 172.21.15.31:0/4070887618 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7fe5540a44d0 con 0x55c14c318240 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.282+0000 7fe56bd74740 1 -- 172.21.15.31:0/4070887618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe554099d90 con 0x55c14c318240 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe546ff5640 1 -- 172.21.15.31:0/4070887618 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe5540a46d0 con 0x55c14c318240 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe56bd74740 10 client.17109 fetch_fsmap learned FSMap version 40 2024-04-25T02:06:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe56bd74740 1 -- 172.21.15.31:0/4070887618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c14c3999c0 con 0x55c14c318240 2024-04-25T02:06:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe546ff5640 1 -- 172.21.15.31:0/4070887618 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe554099d90 con 0x55c14c318240 2024-04-25T02:06:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe546ff5640 1 -- 172.21.15.31:0/4070887618 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7fe554097b60 con 0x55c14c318240 2024-04-25T02:06:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe56bd74740 10 client.17109 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:06:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe56bd74740 10 client.17109 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe56bd74740 20 client.17109 populate_metadata read hostname 'smithi031' 2024-04-25T02:06:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe56bd74740 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c14c333660 0x55c14c39c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe56bd74740 4 client.17109 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:06:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe56bd74740 1 -- 172.21.15.31:0/4070887618 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c14c284540 con 0x55c14c333660 2024-04-25T02:06:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.283+0000 7fe56259c640 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c14c333660 0x55c14c39c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.284+0000 7fe546ff5640 10 client.17109 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.284+0000 7fe56259c640 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c14c333660 0x55c14c39c210 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52acf09a651df790 server_cookie=ab3be27da4386d9c in_seq=0 out_seq=0 2024-04-25T02:06:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe546ff5640 1 -- 172.21.15.31:0/4070887618 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c14c284540 con 0x55c14c333660 2024-04-25T02:06:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe546ff5640 10 client.17109 handle_command_reply: tid=0 2024-04-25T02:06:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 1 client.17109 shutdown 2024-04-25T02:06:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 1 -- 172.21.15.31:0/4070887618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318240 msgr2=0x55c14c32b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318240 0x55c14c32b640 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe5540a4430 tx=0x7fe554098000 comp rx=0 tx=0).stop 2024-04-25T02:06:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 1 -- 172.21.15.31:0/4070887618 shutdown_connections 2024-04-25T02:06:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c14c333660 0x55c14c39c210 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c14c323230 0x55c14c31d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c14c318b60 0x55c14c31cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 1 --2- 172.21.15.31:0/4070887618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c14c318240 0x55c14c32b640 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 1 -- 172.21.15.31:0/4070887618 >> 172.21.15.31:0/4070887618 conn(0x55c14c314560 msgr2=0x55c14c3135c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe546ff5640 0 client.17109 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 1 -- 172.21.15.31:0/4070887618 shutdown_connections 2024-04-25T02:06:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 1 -- 172.21.15.31:0/4070887618 wait complete. 2024-04-25T02:06:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.314+0000 7fe56bd74740 20 client.17109 trim_cache size 0 max 16384 2024-04-25T02:06:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:06:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:06:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:06:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:06:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.315+0000 7fe569064640 1 -- 172.21.15.31:0/3869176907 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe54c0a2dd0 msgr2=0x7fe54c0a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.315+0000 7fe569064640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe54c0a2dd0 0x7fe54c0a5290 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009fd0 tx=0x7fe55c009ec0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.315+0000 7fe569064640 1 -- 172.21.15.31:0/3869176907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5641029d0 msgr2=0x7fe564100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.315+0000 7fe569064640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5641029d0 0x7fe564100620 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe55400e9f0 tx=0x7fe55400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.315+0000 7fe569064640 1 -- 172.21.15.31:0/3869176907 shutdown_connections 2024-04-25T02:06:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.315+0000 7fe569064640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe54c0a2dd0 0x7fe54c0a5290 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.315+0000 7fe569064640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe56410bd10 0x7fe5641010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.315+0000 7fe569064640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5641032f0 0x7fe564100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.315+0000 7fe569064640 1 --2- 172.21.15.31:0/3869176907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5641029d0 0x7fe564100620 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.315+0000 7fe569064640 1 -- 172.21.15.31:0/3869176907 >> 172.21.15.31:0/3869176907 conn(0x7fe5640777c0 msgr2=0x7fe5640fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.316+0000 7fe569064640 1 -- 172.21.15.31:0/3869176907 shutdown_connections 2024-04-25T02:06:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:35.316+0000 7fe569064640 1 -- 172.21.15.31:0/3869176907 wait complete. 2024-04-25T02:06:35.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 248469 1.95 MB/sec execute 522 sec latency 2249.542 ms 2024-04-25T02:06:36.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:35 smithi040 ceph-mon[32418]: pgmap v377: 97 pgs: 97 active+clean; 139 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:36.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:35 smithi134 ceph-mon[33098]: pgmap v377: 97 pgs: 97 active+clean; 139 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:36.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:35 smithi031 ceph-mon[29204]: pgmap v377: 97 pgs: 97 active+clean; 139 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:36.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 248469 1.94 MB/sec execute 523 sec latency 1011.018 ms 2024-04-25T02:06:37.329 INFO:tasks.workunit.client.0.smithi031.stdout: 1 248469 1.94 MB/sec execute 524 sec latency 2011.097 ms 2024-04-25T02:06:38.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:37 smithi040 ceph-mon[32418]: pgmap v378: 97 pgs: 97 active+clean; 139 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1018 KiB/s wr, 92 op/s 2024-04-25T02:06:38.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:37 smithi134 ceph-mon[33098]: pgmap v378: 97 pgs: 97 active+clean; 139 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1018 KiB/s wr, 92 op/s 2024-04-25T02:06:38.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:37 smithi031 ceph-mon[29204]: pgmap v378: 97 pgs: 97 active+clean; 139 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1018 KiB/s wr, 92 op/s 2024-04-25T02:06:38.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 248469 1.94 MB/sec execute 525 sec latency 3011.217 ms 2024-04-25T02:06:39.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 248469 1.93 MB/sec execute 526 sec latency 4011.342 ms 2024-04-25T02:06:40.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:39 smithi040 ceph-mon[32418]: pgmap v379: 97 pgs: 97 active+clean; 139 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1018 KiB/s wr, 92 op/s 2024-04-25T02:06:40.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:39 smithi134 ceph-mon[33098]: pgmap v379: 97 pgs: 97 active+clean; 139 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1018 KiB/s wr, 92 op/s 2024-04-25T02:06:40.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:39 smithi031 ceph-mon[29204]: pgmap v379: 97 pgs: 97 active+clean; 139 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1018 KiB/s wr, 92 op/s 2024-04-25T02:06:40.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 248560 1.93 MB/sec execute 527 sec latency 4730.204 ms 2024-04-25T02:06:41.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 248791 1.93 MB/sec execute 528 sec latency 16.422 ms 2024-04-25T02:06:42.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:41 smithi040 ceph-mon[32418]: pgmap v380: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:42.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:41 smithi134 ceph-mon[33098]: pgmap v380: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:42.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:41 smithi031 ceph-mon[29204]: pgmap v380: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:06:42.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 249142 1.93 MB/sec execute 529 sec latency 18.815 ms 2024-04-25T02:06:43.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 249588 1.93 MB/sec execute 530 sec latency 20.265 ms 2024-04-25T02:06:44.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:43 smithi040 ceph-mon[32418]: pgmap v381: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 575 KiB/s wr, 52 op/s 2024-04-25T02:06:44.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:43 smithi134 ceph-mon[33098]: pgmap v381: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 575 KiB/s wr, 52 op/s 2024-04-25T02:06:44.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:43 smithi031 ceph-mon[29204]: pgmap v381: 97 pgs: 97 active+clean; 136 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 575 KiB/s wr, 52 op/s 2024-04-25T02:06:44.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 250042 1.93 MB/sec execute 531 sec latency 24.770 ms 2024-04-25T02:06:45.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 250651 1.93 MB/sec execute 532 sec latency 22.275 ms 2024-04-25T02:06:46.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:45 smithi134 ceph-mon[33098]: pgmap v382: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-25T02:06:46.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:45 smithi031 ceph-mon[29204]: pgmap v382: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-25T02:06:46.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 251198 1.93 MB/sec execute 533 sec latency 18.819 ms 2024-04-25T02:06:46.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:45 smithi040 ceph-mon[32418]: pgmap v382: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-25T02:06:47.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 251662 1.93 MB/sec execute 534 sec latency 19.135 ms 2024-04-25T02:06:48.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:47 smithi134 ceph-mon[33098]: pgmap v383: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 562 KiB/s wr, 70 op/s 2024-04-25T02:06:48.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:47 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:06:48.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:47 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:48.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:47 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:48.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:47 smithi031 ceph-mon[29204]: pgmap v383: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 562 KiB/s wr, 70 op/s 2024-04-25T02:06:48.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:47 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:06:48.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:47 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:48.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:47 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:48.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 251946 1.94 MB/sec execute 535 sec latency 477.640 ms 2024-04-25T02:06:48.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.406+0000 7f3b82c1c640 1 -- 172.21.15.31:0/27285260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b7c101000 msgr2=0x7f3b7c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.406+0000 7f3b82c1c640 1 --2- 172.21.15.31:0/27285260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b7c101000 0x7f3b7c10d510 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009cd0 tx=0x7f3b7402a8e0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.406+0000 7f3b82c1c640 1 -- 172.21.15.31:0/27285260 shutdown_connections 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.406+0000 7f3b82c1c640 1 --2- 172.21.15.31:0/27285260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b7c101000 0x7f3b7c10d510 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.406+0000 7f3b82c1c640 1 --2- 172.21.15.31:0/27285260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b7c100640 0x7f3b7c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.406+0000 7f3b82c1c640 1 --2- 172.21.15.31:0/27285260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c106650 0x7f3b7c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.406+0000 7f3b82c1c640 1 -- 172.21.15.31:0/27285260 >> 172.21.15.31:0/27285260 conn(0x7f3b7c0777c0 msgr2=0x7f3b7c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.406+0000 7f3b82c1c640 1 -- 172.21.15.31:0/27285260 shutdown_connections 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b82c1c640 1 -- 172.21.15.31:0/27285260 wait complete. 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b82c1c640 1 Processor -- start 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b82c1c640 1 -- start start 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b82c1c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c100640 0x7f3b7c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b82c1c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b7c101000 0x7f3b7c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b82c1c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b7c106650 0x7f3b7c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b82c1c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c110910 con 0x7f3b7c101000 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b82c1c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c110790 con 0x7f3b7c100640 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b82c1c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c110a90 con 0x7f3b7c106650 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b7bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b7c101000 0x7f3b7c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b7bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b7c101000 0x7f3b7c1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55338/0 (socket says 172.21.15.31:55338) 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2351496193 learned_addr learned my addr 172.21.15.31:0/2351496193 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.407+0000 7f3b809b8640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c100640 0x7f3b7c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b811b9640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b7c106650 0x7f3b7c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2351496193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b7c106650 msgr2=0x7f3b7c19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b7c106650 0x7f3b7c19c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2351496193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c100640 msgr2=0x7f3b7c197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c100640 0x7f3b7c197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2351496193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7c19ce30 con 0x7f3b7c101000 2024-04-25T02:06:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b7c101000 0x7f3b7c1983d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000e9f0 tx=0x7f3b7000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b79ffb640 1 -- 172.21.15.31:0/2351496193 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7000cd80 con 0x7f3b7c101000 2024-04-25T02:06:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b79ffb640 1 -- 172.21.15.31:0/2351496193 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b700054d0 con 0x7f3b7c101000 2024-04-25T02:06:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b82c1c640 1 -- 172.21.15.31:0/2351496193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b7c19d120 con 0x7f3b7c101000 2024-04-25T02:06:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b79ffb640 1 -- 172.21.15.31:0/2351496193 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70010490 con 0x7f3b7c101000 2024-04-25T02:06:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.408+0000 7f3b82c1c640 1 -- 172.21.15.31:0/2351496193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b7c1a49b0 con 0x7f3b7c101000 2024-04-25T02:06:48.419 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:47 smithi040 ceph-mon[32418]: pgmap v383: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 562 KiB/s wr, 70 op/s 2024-04-25T02:06:48.419 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:47 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:06:48.419 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:47 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:48.419 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:47 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.410+0000 7f3b79ffb640 1 -- 172.21.15.31:0/2351496193 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3b70005020 con 0x7f3b7c101000 2024-04-25T02:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b85a07740 1 Processor -- start 2024-04-25T02:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b85a07740 1 -- start start 2024-04-25T02:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8fdb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdb920 0x562ca8fea100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8feca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b85a07740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562ca8f3fcb0 con 0x562ca8fea640 2024-04-25T02:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b85a07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562ca8ee85b0 con 0x562ca8fdaf70 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b85a07740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562ca8f3bc70 con 0x562ca8fdb920 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b7bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8feca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b811b9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8fdb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b811b9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8fdb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51076/0 (socket says 172.21.15.31:51076) 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b811b9640 1 -- 172.21.15.31:0/948975055 learned_addr learned my addr 172.21.15.31:0/948975055 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b809b8640 1 --2- 172.21.15.31:0/948975055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdb920 0x562ca8fea100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.411+0000 7f3b79ffb640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3b640a2da0 0x7f3b640a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b7bfff640 1 -- 172.21.15.31:0/948975055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdb920 msgr2=0x562ca8fea100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/948975055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdb920 0x562ca8fea100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b7bfff640 1 -- 172.21.15.31:0/948975055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 msgr2=0x562ca8fdb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/948975055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8fdb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b7bfff640 1 -- 172.21.15.31:0/948975055 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ca8c54210 con 0x562ca8fea640 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b811b9640 1 --2- 172.21.15.31:0/948975055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8fdb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b79ffb640 1 -- 172.21.15.31:0/2351496193 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7f3b700cf840 con 0x7f3b7c101000 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b811b9640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3b640a2da0 0x7f3b640a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/948975055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8feca30 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3b700ccfb0 tx=0x7f3b7005be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa826f2d889b27be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b62ffd640 1 -- 172.21.15.31:0/948975055 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7009bb50 con 0x562ca8fea640 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b62ffd640 1 -- 172.21.15.31:0/948975055 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b70005310 con 0x562ca8fea640 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.412+0000 7f3b62ffd640 1 -- 172.21.15.31:0/948975055 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70007b00 con 0x562ca8fea640 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b85a07740 1 -- 172.21.15.31:0/948975055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 msgr2=0x562ca8feca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b811b9640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3b640a2da0 0x7f3b640a5260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3b74005e30 tx=0x7f3b7403f040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b85a07740 1 --2- 172.21.15.31:0/948975055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8feca30 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3b700ccfb0 tx=0x7f3b7005be60 comp rx=0 tx=0).stop 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b85a07740 1 -- 172.21.15.31:0/948975055 shutdown_connections 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b85a07740 1 --2- 172.21.15.31:0/948975055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8feca30 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b85a07740 1 --2- 172.21.15.31:0/948975055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdb920 0x562ca8fea100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b85a07740 1 --2- 172.21.15.31:0/948975055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8fdb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b85a07740 1 -- 172.21.15.31:0/948975055 >> 172.21.15.31:0/948975055 conn(0x562ca8fd7560 msgr2=0x562ca8fd7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b85a07740 1 -- 172.21.15.31:0/948975055 shutdown_connections 2024-04-25T02:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b85a07740 1 -- 172.21.15.31:0/948975055 wait complete. 2024-04-25T02:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b85a07740 1 -- start start 2024-04-25T02:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.413+0000 7f3b85a07740 1 Processor -- start 2024-04-25T02:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8fe75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdb920 0x562ca8fe7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8fe8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b85a07740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562ca8fed3a0 con 0x562ca8fea640 2024-04-25T02:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b85a07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562ca8efbb50 con 0x562ca8fdaf70 2024-04-25T02:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b85a07740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562ca8fed520 con 0x562ca8fdb920 2024-04-25T02:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b85a07740 10 client.0 _ll_register_callbacks cb 0x562ca8f36ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b811b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8fe8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b811b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8fe8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55360/0 (socket says 172.21.15.31:55360) 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b811b9640 1 -- 172.21.15.31:0/4126407906 learned_addr learned my addr 172.21.15.31:0/4126407906 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdb920 0x562ca8fe7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b809b8640 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8fe75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b811b9640 1 -- 172.21.15.31:0/4126407906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdb920 msgr2=0x562ca8fe7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b811b9640 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdb920 0x562ca8fe7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b811b9640 1 -- 172.21.15.31:0/4126407906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 msgr2=0x562ca8fe75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b811b9640 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8fe75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b811b9640 1 -- 172.21.15.31:0/4126407906 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ca8d20620 con 0x562ca8fea640 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b811b9640 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8fe8050 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009c50 tx=0x7f3b74005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64eb1e517c7e351 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b5bfff640 1 -- 172.21.15.31:0/4126407906 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74004750 con 0x562ca8fea640 2024-04-25T02:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.414+0000 7f3b5bfff640 1 -- 172.21.15.31:0/4126407906 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7f3b740305a0 con 0x562ca8fea640 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b85a07740 1 -- 172.21.15.31:0/4126407906 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562ca8f4d1f0 con 0x562ca8fea640 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b5bfff640 1 -- 172.21.15.31:0/4126407906 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3b740307a0 con 0x562ca8fea640 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b85a07740 10 client.17121 fetch_fsmap learned FSMap version 40 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b85a07740 1 -- 172.21.15.31:0/4126407906 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562ca8ff95d0 con 0x562ca8fea640 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b5bfff640 1 -- 172.21.15.31:0/4126407906 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562ca8f4d1f0 con 0x562ca8fea640 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b5bfff640 1 -- 172.21.15.31:0/4126407906 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f3b7404f020 con 0x562ca8fea640 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b85a07740 10 client.17121 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b85a07740 10 client.17121 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b85a07740 20 client.17121 populate_metadata read hostname 'smithi031' 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b85a07740 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562ca8ffab00 0x562ca8ffcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b85a07740 4 client.17121 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b85a07740 1 -- 172.21.15.31:0/4126407906 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ca8f47540 con 0x562ca8ffab00 2024-04-25T02:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.415+0000 7f3b809b8640 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562ca8ffab00 0x562ca8ffcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.416+0000 7f3b5bfff640 10 client.17121 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:48.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.416+0000 7f3b809b8640 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562ca8ffab00 0x562ca8ffcf90 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f257f7999124e997 server_cookie=1de7227f6b7797ae in_seq=0 out_seq=0 2024-04-25T02:06:48.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.416+0000 7f3b5bfff640 1 -- 172.21.15.31:0/4126407906 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562ca8f47540 con 0x562ca8ffab00 2024-04-25T02:06:48.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.416+0000 7f3b5bfff640 10 client.17121 handle_command_reply: tid=0 2024-04-25T02:06:48.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 1 client.17121 shutdown 2024-04-25T02:06:48.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 1 -- 172.21.15.31:0/4126407906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 msgr2=0x562ca8fe8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:48.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8fe8050 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009c50 tx=0x7f3b74005c30 comp rx=0 tx=0).stop 2024-04-25T02:06:48.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 1 -- 172.21.15.31:0/4126407906 shutdown_connections 2024-04-25T02:06:48.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562ca8ffab00 0x562ca8ffcf90 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8fe8050 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdb920 0x562ca8fe7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 1 --2- 172.21.15.31:0/4126407906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8fe75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 1 -- 172.21.15.31:0/4126407906 >> 172.21.15.31:0/4126407906 conn(0x562ca8fd7560 msgr2=0x562ca8fdd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b5bfff640 0 client.17121 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 1 -- 172.21.15.31:0/4126407906 shutdown_connections 2024-04-25T02:06:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 1 -- 172.21.15.31:0/4126407906 wait complete. 2024-04-25T02:06:48.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.417+0000 7f3b85a07740 20 client.17121 trim_cache size 0 max 16384 2024-04-25T02:06:48.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b85a07740 1 Processor -- start 2024-04-25T02:06:48.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b85a07740 1 -- start start 2024-04-25T02:06:48.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdaf70 0x562ca8ff62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fdb920 0x562ca8fe6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fea640 0x562ca9002860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b85a07740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c1ab030 con 0x562ca8fdb920 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b85a07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c1accf0 con 0x562ca8fea640 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b85a07740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c1ab700 con 0x562ca8fdaf70 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b7bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fdb920 0x562ca8fe6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b7bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fdb920 0x562ca8fe6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55368/0 (socket says 172.21.15.31:55368) 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2644791967 learned_addr learned my addr 172.21.15.31:0/2644791967 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b811b9640 1 --2- 172.21.15.31:0/2644791967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fea640 0x562ca9002860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2644791967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdaf70 msgr2=0x562ca8ff62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/2644791967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdaf70 0x562ca8ff62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2644791967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fea640 msgr2=0x562ca9002860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/2644791967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fea640 0x562ca9002860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.432+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2644791967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ca8fdc070 con 0x562ca8fdb920 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/2644791967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fdb920 0x562ca8fe6dc0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000c6a0 tx=0x7f3b700d3080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c345f93a40c6118e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b627fc640 1 -- 172.21.15.31:0/2644791967 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7001b990 con 0x562ca8fdb920 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b627fc640 1 -- 172.21.15.31:0/2644791967 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b7009c480 con 0x562ca8fdb920 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b627fc640 1 -- 172.21.15.31:0/2644791967 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7009b6e0 con 0x562ca8fdb920 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b85a07740 1 -- 172.21.15.31:0/2644791967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fdb920 msgr2=0x562ca8fe6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b85a07740 1 --2- 172.21.15.31:0/2644791967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fdb920 0x562ca8fe6dc0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000c6a0 tx=0x7f3b700d3080 comp rx=0 tx=0).stop 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b85a07740 1 -- 172.21.15.31:0/2644791967 shutdown_connections 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b85a07740 1 --2- 172.21.15.31:0/2644791967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fea640 0x562ca9002860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b85a07740 1 --2- 172.21.15.31:0/2644791967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fdb920 0x562ca8fe6dc0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b85a07740 1 --2- 172.21.15.31:0/2644791967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fdaf70 0x562ca8ff62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b85a07740 1 -- 172.21.15.31:0/2644791967 >> 172.21.15.31:0/2644791967 conn(0x562ca8fd7560 msgr2=0x562ca8fd65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b85a07740 1 -- 172.21.15.31:0/2644791967 shutdown_connections 2024-04-25T02:06:48.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.433+0000 7f3b85a07740 1 -- 172.21.15.31:0/2644791967 wait complete. 2024-04-25T02:06:48.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b85a07740 1 -- start start 2024-04-25T02:06:48.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b85a07740 1 Processor -- start 2024-04-25T02:06:48.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8ff27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8ff2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b85a07740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fe3880 0x562ca8fe3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b85a07740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c1ac620 con 0x562ca8fea640 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b85a07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c1abe90 con 0x562ca8fdaf70 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b85a07740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c1a9030 con 0x562ca8fe3880 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b85a07740 10 client.0 _ll_register_callbacks cb 0x7f3b70005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b7bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8ff2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b809b8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8ff27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b7bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8ff2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55370/0 (socket says 172.21.15.31:55370) 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.434+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2259608375 learned_addr learned my addr 172.21.15.31:0/2259608375 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b811b9640 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fe3880 0x562ca8fe3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2259608375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fe3880 msgr2=0x562ca8fe3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fe3880 0x562ca8fe3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2259608375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 msgr2=0x562ca8ff27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8ff27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b7bfff640 1 -- 172.21.15.31:0/2259608375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ca8fe4240 con 0x562ca8fea640 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b7bfff640 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8ff2cf0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f3b7009bd10 tx=0x7f3b700a0280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2622c9bcd377d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b5bfff640 1 -- 172.21.15.31:0/2259608375 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7001c020 con 0x562ca8fea640 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b5bfff640 1 -- 172.21.15.31:0/2259608375 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7f3b700cd230 con 0x562ca8fea640 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b85a07740 1 -- 172.21.15.31:0/2259608375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b7009c480 con 0x562ca8fea640 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b5bfff640 1 -- 172.21.15.31:0/2259608375 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3b7009b8c0 con 0x562ca8fea640 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b85a07740 10 client.17127 fetch_fsmap learned FSMap version 40 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b85a07740 1 -- 172.21.15.31:0/2259608375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562ca9060cc0 con 0x562ca8fea640 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b5bfff640 1 -- 172.21.15.31:0/2259608375 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b7009c480 con 0x562ca8fea640 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.435+0000 7f3b5bfff640 1 -- 172.21.15.31:0/2259608375 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f3b70097b90 con 0x562ca8fea640 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.436+0000 7f3b85a07740 10 client.17127 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:06:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.436+0000 7f3b85a07740 10 client.17127 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.436+0000 7f3b85a07740 20 client.17127 populate_metadata read hostname 'smithi031' 2024-04-25T02:06:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.436+0000 7f3b85a07740 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562ca9061080 0x562ca90634a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.436+0000 7f3b85a07740 4 client.17127 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:06:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.436+0000 7f3b85a07740 1 -- 172.21.15.31:0/2259608375 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562ca8ec58e0 con 0x562ca9061080 2024-04-25T02:06:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.436+0000 7f3b809b8640 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562ca9061080 0x562ca90634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.436+0000 7f3b809b8640 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562ca9061080 0x562ca90634a0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=646112b830a54065 server_cookie=a432812fceb75bd8 in_seq=0 out_seq=0 2024-04-25T02:06:48.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.436+0000 7f3b5bfff640 10 client.17127 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:48.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b5bfff640 1 -- 172.21.15.31:0/2259608375 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562ca8ec58e0 con 0x562ca9061080 2024-04-25T02:06:48.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b5bfff640 10 client.17127 handle_command_reply: tid=0 2024-04-25T02:06:48.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 1 client.17127 shutdown 2024-04-25T02:06:48.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 1 -- 172.21.15.31:0/2259608375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 msgr2=0x562ca8ff2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:48.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8ff2cf0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f3b7009bd10 tx=0x7f3b700a0280 comp rx=0 tx=0).stop 2024-04-25T02:06:48.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 1 -- 172.21.15.31:0/2259608375 shutdown_connections 2024-04-25T02:06:48.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562ca9061080 0x562ca90634a0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562ca8fe3880 0x562ca8fe3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ca8fea640 0x562ca8ff2cf0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b5bfff640 0 client.17127 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:06:48.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 1 --2- 172.21.15.31:0/2259608375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ca8fdaf70 0x562ca8ff27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 1 -- 172.21.15.31:0/2259608375 >> 172.21.15.31:0/2259608375 conn(0x562ca8fd7560 msgr2=0x562ca8fe8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:48.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 1 -- 172.21.15.31:0/2259608375 shutdown_connections 2024-04-25T02:06:48.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 1 -- 172.21.15.31:0/2259608375 wait complete. 2024-04-25T02:06:48.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.463+0000 7f3b85a07740 20 client.17127 trim_cache size 0 max 16384 2024-04-25T02:06:48.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:06:48.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:06:48.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:06:48.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:06:48.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 -- 172.21.15.31:0/2351496193 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3b640a2da0 msgr2=0x7f3b640a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:48.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3b640a2da0 0x7f3b640a5260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3b74005e30 tx=0x7f3b7403f040 comp rx=0 tx=0).stop 2024-04-25T02:06:48.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 -- 172.21.15.31:0/2351496193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b7c101000 msgr2=0x7f3b7c1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:48.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b7c101000 0x7f3b7c1983d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000e9f0 tx=0x7f3b7000edf0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 -- 172.21.15.31:0/2351496193 shutdown_connections 2024-04-25T02:06:48.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3b640a2da0 0x7f3b640a5260 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b7c106650 0x7f3b7c19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b7c101000 0x7f3b7c1983d0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 --2- 172.21.15.31:0/2351496193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b7c100640 0x7f3b7c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:48.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 -- 172.21.15.31:0/2351496193 >> 172.21.15.31:0/2351496193 conn(0x7f3b7c0777c0 msgr2=0x7f3b7c06a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:48.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 -- 172.21.15.31:0/2351496193 shutdown_connections 2024-04-25T02:06:48.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:06:48.464+0000 7f3b82c1c640 1 -- 172.21.15.31:0/2351496193 wait complete. 2024-04-25T02:06:49.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:48 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:49.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:48 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:49.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:48 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:06:49.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:48 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:06:49.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:48 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:49.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:48 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:49.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:48 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:49.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:48 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:06:49.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:48 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:06:49.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:48 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:49.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 251946 1.93 MB/sec execute 536 sec latency 1477.997 ms 2024-04-25T02:06:49.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:48 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:49.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:48 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:49.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:48 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:06:49.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:48 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:06:49.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:48 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:06:50.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:49 smithi134 ceph-mon[33098]: pgmap v384: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 562 KiB/s wr, 70 op/s 2024-04-25T02:06:50.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:49 smithi031 ceph-mon[29204]: pgmap v384: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 562 KiB/s wr, 70 op/s 2024-04-25T02:06:50.330 INFO:tasks.workunit.client.0.smithi031.stdout: 1 252013 1.93 MB/sec execute 537 sec latency 2196.442 ms 2024-04-25T02:06:50.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:49 smithi040 ceph-mon[32418]: pgmap v384: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 562 KiB/s wr, 70 op/s 2024-04-25T02:06:51.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 252013 1.93 MB/sec execute 538 sec latency 1030.035 ms 2024-04-25T02:06:52.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:51 smithi031 ceph-mon[29204]: pgmap v385: 97 pgs: 97 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-04-25T02:06:52.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 252013 1.92 MB/sec execute 539 sec latency 2030.151 ms 2024-04-25T02:06:52.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:51 smithi040 ceph-mon[32418]: pgmap v385: 97 pgs: 97 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-04-25T02:06:52.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:51 smithi134 ceph-mon[33098]: pgmap v385: 97 pgs: 97 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-04-25T02:06:53.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 252013 1.92 MB/sec execute 540 sec latency 3030.244 ms 2024-04-25T02:06:54.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:53 smithi031 ceph-mon[29204]: pgmap v386: 97 pgs: 97 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-25T02:06:54.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 252013 1.92 MB/sec execute 541 sec latency 4030.348 ms 2024-04-25T02:06:54.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:53 smithi040 ceph-mon[32418]: pgmap v386: 97 pgs: 97 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-25T02:06:54.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:53 smithi134 ceph-mon[33098]: pgmap v386: 97 pgs: 97 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.0 MiB/s wr, 98 op/s 2024-04-25T02:06:55.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 252110 1.91 MB/sec execute 542 sec latency 4745.987 ms 2024-04-25T02:06:56.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:55 smithi031 ceph-mon[29204]: pgmap v387: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-25T02:06:56.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 252350 1.91 MB/sec execute 543 sec latency 17.899 ms 2024-04-25T02:06:56.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:55 smithi040 ceph-mon[32418]: pgmap v387: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-25T02:06:56.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:55 smithi134 ceph-mon[33098]: pgmap v387: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.1 MiB/s wr, 109 op/s 2024-04-25T02:06:57.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 252752 1.91 MB/sec execute 544 sec latency 19.491 ms 2024-04-25T02:06:58.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:57 smithi031 ceph-mon[29204]: pgmap v388: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 580 KiB/s wr, 54 op/s 2024-04-25T02:06:58.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 253283 1.91 MB/sec execute 545 sec latency 18.662 ms 2024-04-25T02:06:58.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:57 smithi040 ceph-mon[32418]: pgmap v388: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 580 KiB/s wr, 54 op/s 2024-04-25T02:06:58.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:57 smithi134 ceph-mon[33098]: pgmap v388: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 580 KiB/s wr, 54 op/s 2024-04-25T02:06:59.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 253748 1.91 MB/sec execute 546 sec latency 24.672 ms 2024-04-25T02:07:00.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:06:59 smithi031 ceph-mon[29204]: pgmap v389: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 580 KiB/s wr, 54 op/s 2024-04-25T02:07:00.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 254309 1.91 MB/sec execute 547 sec latency 20.956 ms 2024-04-25T02:07:00.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:06:59 smithi040 ceph-mon[32418]: pgmap v389: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 580 KiB/s wr, 54 op/s 2024-04-25T02:07:00.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:06:59 smithi134 ceph-mon[33098]: pgmap v389: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 580 KiB/s wr, 54 op/s 2024-04-25T02:07:01.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 254891 1.92 MB/sec execute 548 sec latency 19.484 ms 2024-04-25T02:07:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.558+0000 7ffb715f9640 1 -- 172.21.15.31:0/1237244839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb6c103960 msgr2=0x7ffb6c103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.558+0000 7ffb715f9640 1 --2- 172.21.15.31:0/1237244839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb6c103960 0x7ffb6c103de0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009930 tx=0x7ffb6002a420 comp rx=0 tx=0).stop 2024-04-25T02:07:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.558+0000 7ffb715f9640 1 -- 172.21.15.31:0/1237244839 shutdown_connections 2024-04-25T02:07:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.558+0000 7ffb715f9640 1 --2- 172.21.15.31:0/1237244839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6c104320 0x7ffb6c110830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.558+0000 7ffb715f9640 1 --2- 172.21.15.31:0/1237244839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb6c103960 0x7ffb6c103de0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.558+0000 7ffb715f9640 1 --2- 172.21.15.31:0/1237244839 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb6c109970 0x7ffb6c109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.558+0000 7ffb715f9640 1 -- 172.21.15.31:0/1237244839 >> 172.21.15.31:0/1237244839 conn(0x7ffb6c0ff450 msgr2=0x7ffb6c101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.558+0000 7ffb715f9640 1 -- 172.21.15.31:0/1237244839 shutdown_connections 2024-04-25T02:07:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.558+0000 7ffb715f9640 1 -- 172.21.15.31:0/1237244839 wait complete. 2024-04-25T02:07:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.558+0000 7ffb715f9640 1 Processor -- start 2024-04-25T02:07:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb715f9640 1 -- start start 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb715f9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6c103960 0x7ffb6c19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb715f9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb6c104320 0x7ffb6c19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb715f9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb6c109970 0x7ffb6c19e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb715f9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c113c30 con 0x7ffb6c109970 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb715f9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c113ab0 con 0x7ffb6c103960 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb715f9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c113db0 con 0x7ffb6c104320 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6c103960 0x7ffb6c19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb6c109970 0x7ffb6c19e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6c103960 0x7ffb6c19a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52430/0 (socket says 172.21.15.31:52430) 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb6c109970 0x7ffb6c19e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42664/0 (socket says 172.21.15.31:42664) 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6affd640 1 -- 172.21.15.31:0/1874017024 learned_addr learned my addr 172.21.15.31:0/1874017024 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb6c104320 0x7ffb6c19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6b7fe640 1 -- 172.21.15.31:0/1874017024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb6c104320 msgr2=0x7ffb6c19a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6b7fe640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb6c104320 0x7ffb6c19a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6b7fe640 1 -- 172.21.15.31:0/1874017024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6c103960 msgr2=0x7ffb6c19a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6b7fe640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6c103960 0x7ffb6c19a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6b7fe640 1 -- 172.21.15.31:0/1874017024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6c19f0f0 con 0x7ffb6c109970 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.559+0000 7ffb6affd640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6c103960 0x7ffb6c19a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.560+0000 7ffb6b7fe640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb6c109970 0x7ffb6c19e9c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009fd0 tx=0x7ffb6400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.560+0000 7ffb53fff640 1 -- 172.21.15.31:0/1874017024 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6400ce20 con 0x7ffb6c109970 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.560+0000 7ffb53fff640 1 -- 172.21.15.31:0/1874017024 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb64005020 con 0x7ffb6c109970 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.560+0000 7ffb715f9640 1 -- 172.21.15.31:0/1874017024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6c19f3e0 con 0x7ffb6c109970 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.560+0000 7ffb53fff640 1 -- 172.21.15.31:0/1874017024 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb64010690 con 0x7ffb6c109970 2024-04-25T02:07:01.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.560+0000 7ffb715f9640 1 -- 172.21.15.31:0/1874017024 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb6c1a6c70 con 0x7ffb6c109970 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb53fff640 1 -- 172.21.15.31:0/1874017024 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ffb640051d0 con 0x7ffb6c109970 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb743e1740 1 Processor -- start 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb743e1740 1 -- start start 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x565138111620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x56513811bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56513811c230 0x56513811e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb743e1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x565138075cb0 con 0x565138111240 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb743e1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56513801e5b0 con 0x56513811c230 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb743e1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x565138071c70 con 0x565138111b60 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb6a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x565138111620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb6a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x565138111620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42680/0 (socket says 172.21.15.31:42680) 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/552587408 learned_addr learned my addr 172.21.15.31:0/552587408 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/552587408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56513811c230 0x56513811e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb6affd640 1 --2- 172.21.15.31:0/552587408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x56513811bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/552587408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 msgr2=0x56513811bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb53fff640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ffb540a2da0 0x7ffb540a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/552587408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x56513811bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/552587408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56513811c230 msgr2=0x56513811e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/552587408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56513811c230 0x56513811e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.563+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/552587408 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565137d8a210 con 0x565138111240 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.564+0000 7ffb6b7fe640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ffb540a2da0 0x7ffb540a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.564+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/552587408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x565138111620 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005ab0 tx=0x7ffb600058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92901eb79d63296c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.564+0000 7ffb53fff640 1 -- 172.21.15.31:0/1874017024 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7ffb640cfba0 con 0x7ffb6c109970 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.564+0000 7ffb517fa640 1 -- 172.21.15.31:0/552587408 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60035030 con 0x565138111240 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.564+0000 7ffb517fa640 1 -- 172.21.15.31:0/552587408 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb600035f0 con 0x565138111240 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.564+0000 7ffb517fa640 1 -- 172.21.15.31:0/552587408 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb600044e0 con 0x565138111240 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 -- 172.21.15.31:0/552587408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 msgr2=0x565138111620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb6b7fe640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ffb540a2da0 0x7ffb540a5260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffb640cd3c0 tx=0x7ffb6406f150 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 --2- 172.21.15.31:0/552587408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x565138111620 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005ab0 tx=0x7ffb600058b0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 -- 172.21.15.31:0/552587408 shutdown_connections 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 --2- 172.21.15.31:0/552587408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56513811c230 0x56513811e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 --2- 172.21.15.31:0/552587408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x56513811bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 --2- 172.21.15.31:0/552587408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x565138111620 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 -- 172.21.15.31:0/552587408 >> 172.21.15.31:0/552587408 conn(0x56513810d560 msgr2=0x56513810d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 -- 172.21.15.31:0/552587408 shutdown_connections 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 -- 172.21.15.31:0/552587408 wait complete. 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 -- start start 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 Processor -- start 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x5651381191c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x565138119700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56513811c230 0x565138119c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56513811efc0 con 0x565138111240 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565138031b50 con 0x56513811c230 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56513811f140 con 0x565138111b60 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.565+0000 7ffb743e1740 10 client.0 _ll_register_callbacks cb 0x56513806cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb6a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x5651381191c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb6a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x5651381191c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42690/0 (socket says 172.21.15.31:42690) 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/4142767290 learned_addr learned my addr 172.21.15.31:0/4142767290 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb6affd640 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x565138119700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56513811c230 0x565138119c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/4142767290 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 msgr2=0x565138119700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x565138119700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/4142767290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56513811c230 msgr2=0x565138119c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56513811c230 0x565138119c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/4142767290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565137e56620 con 0x565138111240 2024-04-25T02:07:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x5651381191c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ffb60031440 tx=0x7ffb600092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473ca0129d710c84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb4eff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb4eff5640 1 -- 172.21.15.31:0/4142767290 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60041020 con 0x565138111240 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb743e1740 1 -- 172.21.15.31:0/4142767290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb600035f0 con 0x565138111240 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.566+0000 7ffb4eff5640 1 -- 172.21.15.31:0/4142767290 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7ffb60005480 con 0x565138111240 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb4eff5640 1 -- 172.21.15.31:0/4142767290 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7ffb60035040 con 0x565138111240 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb743e1740 10 client.17139 fetch_fsmap learned FSMap version 40 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb743e1740 1 -- 172.21.15.31:0/4142767290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56513812b1c0 con 0x565138111240 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb4eff5640 1 -- 172.21.15.31:0/4142767290 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb600035f0 con 0x565138111240 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb4eff5640 1 -- 172.21.15.31:0/4142767290 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7ffb60053030 con 0x565138111240 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb743e1740 10 client.17139 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb743e1740 10 client.17139 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb743e1740 20 client.17139 populate_metadata read hostname 'smithi031' 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb743e1740 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56513812c510 0x56513812e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb743e1740 4 client.17139 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb743e1740 1 -- 172.21.15.31:0/4142767290 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651380831f0 con 0x56513812c510 2024-04-25T02:07:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.567+0000 7ffb6affd640 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56513812c510 0x56513812e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb4eff5640 10 client.17139 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb6affd640 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56513812c510 0x56513812e900 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b94603f6624aecc server_cookie=45801d7ba6200616 in_seq=0 out_seq=0 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb4eff5640 1 -- 172.21.15.31:0/4142767290 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651380831f0 con 0x56513812c510 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb4eff5640 10 client.17139 handle_command_reply: tid=0 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb743e1740 1 client.17139 shutdown 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb743e1740 1 -- 172.21.15.31:0/4142767290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 msgr2=0x5651381191c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb743e1740 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x5651381191c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ffb60031440 tx=0x7ffb600092d0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb743e1740 1 -- 172.21.15.31:0/4142767290 shutdown_connections 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb743e1740 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56513812c510 0x56513812e900 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb743e1740 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56513811c230 0x565138119c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb743e1740 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x565138119700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb743e1740 1 --2- 172.21.15.31:0/4142767290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565138111240 0x5651381191c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb743e1740 1 -- 172.21.15.31:0/4142767290 >> 172.21.15.31:0/4142767290 conn(0x56513810d560 msgr2=0x565138113360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.568+0000 7ffb4eff5640 0 client.17139 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.569+0000 7ffb743e1740 1 -- 172.21.15.31:0/4142767290 shutdown_connections 2024-04-25T02:07:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.569+0000 7ffb743e1740 1 -- 172.21.15.31:0/4142767290 wait complete. 2024-04-25T02:07:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.569+0000 7ffb743e1740 20 client.17139 trim_cache size 0 max 16384 2024-04-25T02:07:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb743e1740 1 Processor -- start 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb743e1740 1 -- start start 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111240 0x56513811b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565138111b60 0x56513810c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138127e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb743e1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1af410 con 0x56513811c230 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb743e1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1b10d0 con 0x565138111b60 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb743e1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1afae0 con 0x565138111240 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb6a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138127e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb6affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565138111b60 0x56513810c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb6a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138127e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42692/0 (socket says 172.21.15.31:42692) 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/1969400061 learned_addr learned my addr 172.21.15.31:0/1969400061 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:01.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/1969400061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111240 0x56513811b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/1969400061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111240 msgr2=0x56513811b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/1969400061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111240 0x56513811b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/1969400061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565138111b60 msgr2=0x56513810c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/1969400061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565138111b60 0x56513810c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/1969400061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565138111990 con 0x56513811c230 2024-04-25T02:07:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.588+0000 7ffb6affd640 1 --2- 172.21.15.31:0/1969400061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565138111b60 0x56513810c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.589+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/1969400061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138127e80 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005040 tx=0x7ffb600058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580d7d235af0b7ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.589+0000 7ffb50ff9640 1 -- 172.21.15.31:0/1969400061 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60041040 con 0x56513811c230 2024-04-25T02:07:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.589+0000 7ffb50ff9640 1 -- 172.21.15.31:0/1969400061 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb600044a0 con 0x56513811c230 2024-04-25T02:07:01.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.589+0000 7ffb50ff9640 1 -- 172.21.15.31:0/1969400061 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60003850 con 0x56513811c230 2024-04-25T02:07:01.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.589+0000 7ffb743e1740 1 -- 172.21.15.31:0/1969400061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 msgr2=0x565138127e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.589+0000 7ffb743e1740 1 --2- 172.21.15.31:0/1969400061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138127e80 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005040 tx=0x7ffb600058b0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.589+0000 7ffb743e1740 1 -- 172.21.15.31:0/1969400061 shutdown_connections 2024-04-25T02:07:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.589+0000 7ffb743e1740 1 --2- 172.21.15.31:0/1969400061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138127e80 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 --2- 172.21.15.31:0/1969400061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565138111b60 0x56513810c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 --2- 172.21.15.31:0/1969400061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111240 0x56513811b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 -- 172.21.15.31:0/1969400061 >> 172.21.15.31:0/1969400061 conn(0x56513810d560 msgr2=0x5651381181f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 -- 172.21.15.31:0/1969400061 shutdown_connections 2024-04-25T02:07:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 -- 172.21.15.31:0/1969400061 wait complete. 2024-04-25T02:07:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 -- start start 2024-04-25T02:07:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 Processor -- start 2024-04-25T02:07:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565138111240 0x56513811fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x5651381203d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138120910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1b0a00 con 0x56513811c230 2024-04-25T02:07:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1b0270 con 0x565138111240 2024-04-25T02:07:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1ad410 con 0x565138111b60 2024-04-25T02:07:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb743e1740 10 client.0 _ll_register_callbacks cb 0x7ffb60009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb6a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565138111240 0x56513811fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb6a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138120910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb6a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138120910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42704/0 (socket says 172.21.15.31:42704) 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/191536563 learned_addr learned my addr 172.21.15.31:0/191536563 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.590+0000 7ffb6affd640 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x5651381203d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/191536563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 msgr2=0x5651381203d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x5651381203d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/191536563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565138111240 msgr2=0x56513811fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565138111240 0x56513811fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb6a7fc640 1 -- 172.21.15.31:0/191536563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565138191960 con 0x56513811c230 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb6affd640 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x5651381203d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb6a7fc640 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138120910 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005ec0 tx=0x7ffb6002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f7c5afd1caa588 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb4eff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb4eff5640 1 -- 172.21.15.31:0/191536563 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60041040 con 0x56513811c230 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb4eff5640 1 -- 172.21.15.31:0/191536563 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8484+0+0 (secure 0 0 0) 0x7ffb600035f0 con 0x56513811c230 2024-04-25T02:07:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb743e1740 1 -- 172.21.15.31:0/191536563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb600044a0 con 0x56513811c230 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb4eff5640 1 -- 172.21.15.31:0/191536563 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7ffb600037f0 con 0x56513811c230 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb743e1740 10 client.17145 fetch_fsmap learned FSMap version 40 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb743e1740 1 -- 172.21.15.31:0/191536563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x565138192c70 con 0x56513811c230 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb4eff5640 1 -- 172.21.15.31:0/191536563 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb600044a0 con 0x56513811c230 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.591+0000 7ffb4eff5640 1 -- 172.21.15.31:0/191536563 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7ffb6005d040 con 0x56513811c230 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.592+0000 7ffb743e1740 10 client.17145 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.592+0000 7ffb743e1740 10 client.17145 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.592+0000 7ffb743e1740 20 client.17145 populate_metadata read hostname 'smithi031' 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.592+0000 7ffb743e1740 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565138193150 0x565138195570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.592+0000 7ffb743e1740 4 client.17145 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.592+0000 7ffb743e1740 1 -- 172.21.15.31:0/191536563 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56513807d540 con 0x565138193150 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.592+0000 7ffb6affd640 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565138193150 0x565138195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.592+0000 7ffb4eff5640 10 client.17145 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.592+0000 7ffb6affd640 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565138193150 0x565138195570 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d6ba12334933599 server_cookie=40303e92bdf85649 in_seq=0 out_seq=0 2024-04-25T02:07:01.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.621+0000 7ffb4eff5640 1 -- 172.21.15.31:0/191536563 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56513807d540 con 0x565138193150 2024-04-25T02:07:01.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.621+0000 7ffb4eff5640 10 client.17145 handle_command_reply: tid=0 2024-04-25T02:07:01.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.621+0000 7ffb743e1740 1 client.17145 shutdown 2024-04-25T02:07:01.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.621+0000 7ffb743e1740 1 -- 172.21.15.31:0/191536563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 msgr2=0x565138120910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:01.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.621+0000 7ffb743e1740 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138120910 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005ec0 tx=0x7ffb6002a420 comp rx=0 tx=0).stop 2024-04-25T02:07:01.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.621+0000 7ffb743e1740 1 -- 172.21.15.31:0/191536563 shutdown_connections 2024-04-25T02:07:01.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb743e1740 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x565138193150 0x565138195570 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb743e1740 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56513811c230 0x565138120910 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb4eff5640 0 client.17145 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:01.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb743e1740 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565138111b60 0x5651381203d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb743e1740 1 --2- 172.21.15.31:0/191536563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565138111240 0x56513811fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb743e1740 1 -- 172.21.15.31:0/191536563 >> 172.21.15.31:0/191536563 conn(0x56513810d560 msgr2=0x5651381128a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb743e1740 1 -- 172.21.15.31:0/191536563 shutdown_connections 2024-04-25T02:07:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb743e1740 1 -- 172.21.15.31:0/191536563 wait complete. 2024-04-25T02:07:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb743e1740 20 client.17145 trim_cache size 0 max 16384 2024-04-25T02:07:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:07:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:07:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:07:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:07:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb715f9640 1 -- 172.21.15.31:0/1874017024 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ffb540a2da0 msgr2=0x7ffb540a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb715f9640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ffb540a2da0 0x7ffb540a5260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffb640cd3c0 tx=0x7ffb6406f150 comp rx=0 tx=0).stop 2024-04-25T02:07:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb715f9640 1 -- 172.21.15.31:0/1874017024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb6c109970 msgr2=0x7ffb6c19e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb715f9640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb6c109970 0x7ffb6c19e9c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009fd0 tx=0x7ffb6400ef10 comp rx=0 tx=0).stop 2024-04-25T02:07:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb715f9640 1 -- 172.21.15.31:0/1874017024 shutdown_connections 2024-04-25T02:07:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb715f9640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ffb540a2da0 0x7ffb540a5260 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb715f9640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb6c109970 0x7ffb6c19e9c0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb715f9640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffb6c104320 0x7ffb6c19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb715f9640 1 --2- 172.21.15.31:0/1874017024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb6c103960 0x7ffb6c19a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.622+0000 7ffb715f9640 1 -- 172.21.15.31:0/1874017024 >> 172.21.15.31:0/1874017024 conn(0x7ffb6c0ff450 msgr2=0x7ffb6c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.623+0000 7ffb715f9640 1 -- 172.21.15.31:0/1874017024 shutdown_connections 2024-04-25T02:07:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:01.623+0000 7ffb715f9640 1 -- 172.21.15.31:0/1874017024 wait complete. 2024-04-25T02:07:02.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:01 smithi040 ceph-mon[32418]: pgmap v390: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:07:02.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:01 smithi031 ceph-mon[29204]: pgmap v390: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:07:02.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 255389 1.92 MB/sec execute 549 sec latency 28.014 ms 2024-04-25T02:07:02.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:01 smithi134 ceph-mon[33098]: pgmap v390: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:07:03.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:02 smithi031 ceph-mon[29204]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T02:07:03.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 255493 1.92 MB/sec execute 550 sec latency 805.636 ms 2024-04-25T02:07:03.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:02 smithi040 ceph-mon[32418]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T02:07:03.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:02 smithi134 ceph-mon[33098]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T02:07:04.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:03 smithi031 ceph-mon[29204]: pgmap v392: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 659 KiB/s wr, 82 op/s 2024-04-25T02:07:04.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 255493 1.91 MB/sec execute 551 sec latency 1805.754 ms 2024-04-25T02:07:04.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:03 smithi040 ceph-mon[32418]: pgmap v392: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 659 KiB/s wr, 82 op/s 2024-04-25T02:07:04.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:03 smithi134 ceph-mon[33098]: pgmap v392: 97 pgs: 97 active+clean; 137 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 659 KiB/s wr, 82 op/s 2024-04-25T02:07:05.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 255560 1.91 MB/sec execute 552 sec latency 2521.470 ms 2024-04-25T02:07:06.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:05 smithi031 ceph-mon[29204]: pgmap v393: 97 pgs: 97 active+clean; 141 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:07:06.331 INFO:tasks.workunit.client.0.smithi031.stdout: 1 255560 1.91 MB/sec execute 553 sec latency 1047.042 ms 2024-04-25T02:07:06.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:05 smithi040 ceph-mon[32418]: pgmap v393: 97 pgs: 97 active+clean; 141 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:07:06.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:05 smithi134 ceph-mon[33098]: pgmap v393: 97 pgs: 97 active+clean; 141 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:07:07.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 255560 1.90 MB/sec execute 554 sec latency 2047.129 ms 2024-04-25T02:07:08.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:07 smithi031 ceph-mon[29204]: pgmap v394: 97 pgs: 97 active+clean; 141 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:07:08.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 255560 1.90 MB/sec execute 555 sec latency 3047.208 ms 2024-04-25T02:07:08.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:07 smithi040 ceph-mon[32418]: pgmap v394: 97 pgs: 97 active+clean; 141 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:07:08.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:07 smithi134 ceph-mon[33098]: pgmap v394: 97 pgs: 97 active+clean; 141 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:07:09.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 255560 1.90 MB/sec execute 556 sec latency 4047.307 ms 2024-04-25T02:07:10.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:09 smithi040 ceph-mon[32418]: pgmap v395: 97 pgs: 97 active+clean; 141 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:07:10.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:09 smithi040 ceph-mon[32418]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T02:07:10.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:09 smithi134 ceph-mon[33098]: pgmap v395: 97 pgs: 97 active+clean; 141 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:07:10.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:09 smithi134 ceph-mon[33098]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T02:07:10.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:09 smithi031 ceph-mon[29204]: pgmap v395: 97 pgs: 97 active+clean; 141 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.3 MiB/s wr, 138 op/s 2024-04-25T02:07:10.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:09 smithi031 ceph-mon[29204]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T02:07:10.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 255654 1.90 MB/sec execute 557 sec latency 4764.112 ms 2024-04-25T02:07:11.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 255884 1.89 MB/sec execute 558 sec latency 16.723 ms 2024-04-25T02:07:12.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:11 smithi040 ceph-mon[32418]: pgmap v397: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 838 KiB/s wr, 93 op/s 2024-04-25T02:07:12.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:11 smithi134 ceph-mon[33098]: pgmap v397: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 838 KiB/s wr, 93 op/s 2024-04-25T02:07:12.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:11 smithi031 ceph-mon[29204]: pgmap v397: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 838 KiB/s wr, 93 op/s 2024-04-25T02:07:12.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 256230 1.89 MB/sec execute 559 sec latency 20.301 ms 2024-04-25T02:07:13.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 256686 1.89 MB/sec execute 560 sec latency 19.352 ms 2024-04-25T02:07:14.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:13 smithi040 ceph-mon[32418]: pgmap v398: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 738 KiB/s wr, 82 op/s 2024-04-25T02:07:14.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:13 smithi134 ceph-mon[33098]: pgmap v398: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 738 KiB/s wr, 82 op/s 2024-04-25T02:07:14.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:13 smithi031 ceph-mon[29204]: pgmap v398: 97 pgs: 97 active+clean; 142 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 738 KiB/s wr, 82 op/s 2024-04-25T02:07:14.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 257128 1.89 MB/sec execute 561 sec latency 26.195 ms 2024-04-25T02:07:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.719+0000 7f0944509640 1 -- 172.21.15.31:0/173788493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c107740 msgr2=0x7f093c107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.719+0000 7f0944509640 1 --2- 172.21.15.31:0/173788493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c107740 0x7f093c107b20 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0938009930 tx=0x7f093802a420 comp rx=0 tx=0).stop 2024-04-25T02:07:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.719+0000 7f0944509640 1 -- 172.21.15.31:0/173788493 shutdown_connections 2024-04-25T02:07:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.719+0000 7f0944509640 1 --2- 172.21.15.31:0/173788493 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f093c1020f0 0x7f093c10e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.719+0000 7f0944509640 1 --2- 172.21.15.31:0/173788493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f093c101730 0x7f093c101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.719+0000 7f0944509640 1 --2- 172.21.15.31:0/173788493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c107740 0x7f093c107b20 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.719+0000 7f0944509640 1 -- 172.21.15.31:0/173788493 >> 172.21.15.31:0/173788493 conn(0x7f093c0fd200 msgr2=0x7f093c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.719+0000 7f0944509640 1 -- 172.21.15.31:0/173788493 shutdown_connections 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.719+0000 7f0944509640 1 -- 172.21.15.31:0/173788493 wait complete. 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0944509640 1 Processor -- start 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0944509640 1 -- start start 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0944509640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f093c101730 0x7f093c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0944509640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f093c1020f0 0x7f093c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0944509640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c107740 0x7f093c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0944509640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1119c0 con 0x7f093c107740 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0944509640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f093c111840 con 0x7f093c1020f0 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0944509640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f093c111b40 con 0x7f093c101730 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0942aa6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c107740 0x7f093c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0941aa4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f093c1020f0 0x7f093c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0942aa6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c107740 0x7f093c19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43096/0 (socket says 172.21.15.31:43096) 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0942aa6640 1 -- 172.21.15.31:0/3815792553 learned_addr learned my addr 172.21.15.31:0/3815792553 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f09422a5640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f093c101730 0x7f093c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0942aa6640 1 -- 172.21.15.31:0/3815792553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f093c101730 msgr2=0x7f093c197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0942aa6640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f093c101730 0x7f093c197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0942aa6640 1 -- 172.21.15.31:0/3815792553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f093c1020f0 msgr2=0x7f093c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0942aa6640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f093c1020f0 0x7f093c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.720+0000 7f0942aa6640 1 -- 172.21.15.31:0/3815792553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093c19cf10 con 0x7f093c107740 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.721+0000 7f0941aa4640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f093c1020f0 0x7f093c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.721+0000 7f0942aa6640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c107740 0x7f093c19c7e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f093400cad0 tx=0x7f093400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.721+0000 7f09337fe640 1 -- 172.21.15.31:0/3815792553 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093400ae20 con 0x7f093c107740 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.721+0000 7f09337fe640 1 -- 172.21.15.31:0/3815792553 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0934005020 con 0x7f093c107740 2024-04-25T02:07:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.721+0000 7f0944509640 1 -- 172.21.15.31:0/3815792553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f093c19d200 con 0x7f093c107740 2024-04-25T02:07:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.721+0000 7f09337fe640 1 -- 172.21.15.31:0/3815792553 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093400f6b0 con 0x7f093c107740 2024-04-25T02:07:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.721+0000 7f0944509640 1 -- 172.21.15.31:0/3815792553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f093c1a4a90 con 0x7f093c107740 2024-04-25T02:07:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.724+0000 7f09337fe640 1 -- 172.21.15.31:0/3815792553 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f09340051d0 con 0x7f093c107740 2024-04-25T02:07:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.724+0000 7f0947210740 1 Processor -- start 2024-04-25T02:07:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f0947210740 1 -- start start 2024-04-25T02:07:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f0947210740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9f2240 0x55c08b9f2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f0947210740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9fccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f0947210740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9fd230 0x55c08b9ff620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f0947210740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c08b956cb0 con 0x55c08b9f2b60 2024-04-25T02:07:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f0947210740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c08b8ff5b0 con 0x55c08b9f2240 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f0947210740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c08b952c70 con 0x55c08b9fd230 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f09337fe640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09280a2da0 0x7f09280a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f09422a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9fccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f0941aa4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9f2240 0x55c08b9f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f09422a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9fccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43104/0 (socket says 172.21.15.31:43104) 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f09422a5640 1 -- 172.21.15.31:0/2859598927 learned_addr learned my addr 172.21.15.31:0/2859598927 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f0941aa4640 1 --2- 172.21.15.31:0/2859598927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9fd230 0x55c08b9ff620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f0942aa6640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09280a2da0 0x7f09280a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f09337fe640 1 -- 172.21.15.31:0/3815792553 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f09340cf370 con 0x7f093c107740 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f09422a5640 1 -- 172.21.15.31:0/2859598927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9fd230 msgr2=0x55c08b9ff620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f09422a5640 1 --2- 172.21.15.31:0/2859598927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9fd230 0x55c08b9ff620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f09422a5640 1 -- 172.21.15.31:0/2859598927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9f2240 msgr2=0x55c08b9f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f09422a5640 1 --2- 172.21.15.31:0/2859598927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9f2240 0x55c08b9f2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f09422a5640 1 -- 172.21.15.31:0/2859598927 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c08b66b210 con 0x55c08b9f2b60 2024-04-25T02:07:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.725+0000 7f0941aa4640 1 --2- 172.21.15.31:0/2859598927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9f2240 0x55c08b9f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.726+0000 7f09422a5640 1 --2- 172.21.15.31:0/2859598927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9fccf0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f0938009930 tx=0x7f0938005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9451626886b601 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.726+0000 7f0942aa6640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09280a2da0 0x7f09280a5260 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f09340cd0e0 tx=0x7f09340cf670 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.726+0000 7f0930bfe640 1 -- 172.21.15.31:0/2859598927 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938040070 con 0x55c08b9f2b60 2024-04-25T02:07:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.726+0000 7f0930bfe640 1 -- 172.21.15.31:0/2859598927 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09380316a0 con 0x55c08b9f2b60 2024-04-25T02:07:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.726+0000 7f0930bfe640 1 -- 172.21.15.31:0/2859598927 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938035040 con 0x55c08b9f2b60 2024-04-25T02:07:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.727+0000 7f0947210740 1 -- 172.21.15.31:0/2859598927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 msgr2=0x55c08b9fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.727+0000 7f0947210740 1 --2- 172.21.15.31:0/2859598927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9fccf0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f0938009930 tx=0x7f0938005fd0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.727+0000 7f0947210740 1 -- 172.21.15.31:0/2859598927 shutdown_connections 2024-04-25T02:07:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.727+0000 7f0947210740 1 --2- 172.21.15.31:0/2859598927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9fd230 0x55c08b9ff620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.727+0000 7f0947210740 1 --2- 172.21.15.31:0/2859598927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9fccf0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.727+0000 7f0947210740 1 --2- 172.21.15.31:0/2859598927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9f2240 0x55c08b9f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.727+0000 7f0947210740 1 -- 172.21.15.31:0/2859598927 >> 172.21.15.31:0/2859598927 conn(0x55c08b9ee560 msgr2=0x55c08b9ee940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.727+0000 7f0947210740 1 -- 172.21.15.31:0/2859598927 shutdown_connections 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.727+0000 7f0947210740 1 -- 172.21.15.31:0/2859598927 wait complete. 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.727+0000 7f0947210740 1 -- start start 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.727+0000 7f0947210740 1 Processor -- start 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0947210740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08b9fa100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0947210740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 0x55c08b9fa640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0947210740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9fab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0947210740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c08b9fffc0 con 0x55c08b9f2240 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0947210740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c08b912b50 con 0x55c08b9fd230 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0947210740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c08ba00140 con 0x55c08b9f2b60 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0947210740 10 client.0 _ll_register_callbacks cb 0x55c08b94dca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0941aa4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08b9fa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0941aa4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08b9fa100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43120/0 (socket says 172.21.15.31:43120) 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0941aa4640 1 -- 172.21.15.31:0/682020060 learned_addr learned my addr 172.21.15.31:0/682020060 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f09422a5640 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 0x55c08b9fa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0941aa4640 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9fab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f09422a5640 1 -- 172.21.15.31:0/682020060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 msgr2=0x55c08b9fab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f09422a5640 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9fab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f09422a5640 1 -- 172.21.15.31:0/682020060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 msgr2=0x55c08b9fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f09422a5640 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08b9fa100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f09422a5640 1 -- 172.21.15.31:0/682020060 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c08b737620 con 0x55c08b9f2b60 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.728+0000 7f0941aa4640 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08b9fa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.729+0000 7f09422a5640 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 0x55c08b9fa640 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0938009900 tx=0x7f0938004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3f0e0f4a6c0d987 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.729+0000 7f0925ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:07:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.729+0000 7f0925ffb640 1 -- 172.21.15.31:0/682020060 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093804c020 con 0x55c08b9f2b60 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.729+0000 7f0925ffb640 1 -- 172.21.15.31:0/682020060 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f0938005450 con 0x55c08b9f2b60 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.729+0000 7f0947210740 1 -- 172.21.15.31:0/682020060 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09380316a0 con 0x55c08b9f2b60 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.729+0000 7f0925ffb640 1 -- 172.21.15.31:0/682020060 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f0938057050 con 0x55c08b9f2b60 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.729+0000 7f0947210740 10 client.26515 fetch_fsmap learned FSMap version 40 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.729+0000 7f0947210740 1 -- 172.21.15.31:0/682020060 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c08b9fffc0 con 0x55c08b9f2b60 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.729+0000 7f0925ffb640 1 -- 172.21.15.31:0/682020060 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09380316a0 con 0x55c08b9f2b60 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.730+0000 7f0925ffb640 1 -- 172.21.15.31:0/682020060 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f093803c4d0 con 0x55c08b9f2b60 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.730+0000 7f0947210740 10 client.26515 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.730+0000 7f0947210740 10 client.26515 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.730+0000 7f0947210740 20 client.26515 populate_metadata read hostname 'smithi031' 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.730+0000 7f0947210740 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c08ba0d300 0x55c08ba0f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.730+0000 7f0947210740 4 client.26515 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.730+0000 7f0947210740 1 -- 172.21.15.31:0/682020060 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c08b9641f0 con 0x55c08ba0d300 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.730+0000 7f0941aa4640 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c08ba0d300 0x55c08ba0f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.730+0000 7f0925ffb640 10 client.26515 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.730+0000 7f0941aa4640 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c08ba0d300 0x55c08ba0f6f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89164a9f08a3471f server_cookie=23dafea56d9cb2c5 in_seq=0 out_seq=0 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0925ffb640 1 -- 172.21.15.31:0/682020060 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c08b9641f0 con 0x55c08ba0d300 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0925ffb640 10 client.26515 handle_command_reply: tid=0 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 1 client.26515 shutdown 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 1 -- 172.21.15.31:0/682020060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 msgr2=0x55c08b9fa640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 0x55c08b9fa640 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0938009900 tx=0x7f0938004580 comp rx=0 tx=0).stop 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 1 -- 172.21.15.31:0/682020060 shutdown_connections 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c08ba0d300 0x55c08ba0f6f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9fab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 0x55c08b9fa640 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0925ffb640 0 client.26515 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 1 --2- 172.21.15.31:0/682020060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08b9fa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 1 -- 172.21.15.31:0/682020060 >> 172.21.15.31:0/682020060 conn(0x55c08b9ee560 msgr2=0x55c08b9f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 1 -- 172.21.15.31:0/682020060 shutdown_connections 2024-04-25T02:07:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 1 -- 172.21.15.31:0/682020060 wait complete. 2024-04-25T02:07:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.731+0000 7f0947210740 20 client.26515 trim_cache size 0 max 16384 2024-04-25T02:07:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.747+0000 7f0947210740 1 Processor -- start 2024-04-25T02:07:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f0947210740 1 -- start start 2024-04-25T02:07:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f0947210740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2240 0x55c08ba08eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f0947210740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f0947210740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9ee940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f0947210740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1ad320 con 0x55c08b9f2b60 2024-04-25T02:07:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f0947210740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1aefe0 con 0x55c08b9fd230 2024-04-25T02:07:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f0947210740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1ad9f0 con 0x55c08b9f2240 2024-04-25T02:07:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f09422a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f09422a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9f44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43122/0 (socket says 172.21.15.31:43122) 2024-04-25T02:07:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f09422a5640 1 -- 172.21.15.31:0/1339434248 learned_addr learned my addr 172.21.15.31:0/1339434248 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f0941aa4640 1 --2- 172.21.15.31:0/1339434248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9ee940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f0941aa4640 1 --2- 172.21.15.31:0/1339434248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2240 0x55c08ba08eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f09422a5640 1 -- 172.21.15.31:0/1339434248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2240 msgr2=0x55c08ba08eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f09422a5640 1 --2- 172.21.15.31:0/1339434248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2240 0x55c08ba08eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f09422a5640 1 -- 172.21.15.31:0/1339434248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 msgr2=0x55c08b9ee940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f09422a5640 1 --2- 172.21.15.31:0/1339434248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9ee940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f09422a5640 1 -- 172.21.15.31:0/1339434248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c08b9f90b0 con 0x55c08b9f2b60 2024-04-25T02:07:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f09422a5640 1 --2- 172.21.15.31:0/1339434248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9f44e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f0938000c00 tx=0x7f093802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7bd7d73141c3c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.748+0000 7f0927fff640 1 -- 172.21.15.31:0/1339434248 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093804c040 con 0x55c08b9f2b60 2024-04-25T02:07:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.749+0000 7f0927fff640 1 -- 172.21.15.31:0/1339434248 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0938005850 con 0x55c08b9f2b60 2024-04-25T02:07:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.749+0000 7f0927fff640 1 -- 172.21.15.31:0/1339434248 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938040070 con 0x55c08b9f2b60 2024-04-25T02:07:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.749+0000 7f0947210740 1 -- 172.21.15.31:0/1339434248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 msgr2=0x55c08b9f44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.749+0000 7f0947210740 1 --2- 172.21.15.31:0/1339434248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9f44e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f0938000c00 tx=0x7f093802a420 comp rx=0 tx=0).stop 2024-04-25T02:07:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.749+0000 7f0947210740 1 -- 172.21.15.31:0/1339434248 shutdown_connections 2024-04-25T02:07:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.749+0000 7f0947210740 1 --2- 172.21.15.31:0/1339434248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9ee940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.749+0000 7f0947210740 1 --2- 172.21.15.31:0/1339434248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2b60 0x55c08b9f44e0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.749+0000 7f0947210740 1 --2- 172.21.15.31:0/1339434248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2240 0x55c08ba08eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.749+0000 7f0947210740 1 -- 172.21.15.31:0/1339434248 >> 172.21.15.31:0/1339434248 conn(0x55c08b9ee560 msgr2=0x55c08b9ed5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.749+0000 7f0947210740 1 -- 172.21.15.31:0/1339434248 shutdown_connections 2024-04-25T02:07:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.749+0000 7f0947210740 1 -- 172.21.15.31:0/1339434248 wait complete. 2024-04-25T02:07:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0947210740 1 -- start start 2024-04-25T02:07:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0947210740 1 Processor -- start 2024-04-25T02:07:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0947210740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08ba05890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0947210740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 0x55c08b9f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0947210740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0947210740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1ae910 con 0x55c08b9f2240 2024-04-25T02:07:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0947210740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1ae180 con 0x55c08b9fd230 2024-04-25T02:07:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0947210740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1ab320 con 0x55c08b9f2b60 2024-04-25T02:07:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0947210740 10 client.0 _ll_register_callbacks cb 0x7f092c009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0941aa4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08ba05890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0941aa4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08ba05890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43138/0 (socket says 172.21.15.31:43138) 2024-04-25T02:07:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0941aa4640 1 -- 172.21.15.31:0/2592965164 learned_addr learned my addr 172.21.15.31:0/2592965164 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f09422a5640 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 0x55c08b9f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0941aa4640 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0941aa4640 1 -- 172.21.15.31:0/2592965164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 msgr2=0x55c08b9f6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0941aa4640 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 0x55c08b9f6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0941aa4640 1 -- 172.21.15.31:0/2592965164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 msgr2=0x55c08b9f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0941aa4640 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9f70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f0941aa4640 1 -- 172.21.15.31:0/2592965164 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c08b9f7920 con 0x55c08b9f2240 2024-04-25T02:07:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.750+0000 7f09422a5640 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 0x55c08b9f6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0941aa4640 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08ba05890 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f092c000c00 tx=0x7f092c028040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d22f921f1c2a68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0925ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:07:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0925ffb640 1 -- 172.21.15.31:0/2592965164 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c005020 con 0x55c08b9f2240 2024-04-25T02:07:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0947210740 1 -- 172.21.15.31:0/2592965164 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0938005850 con 0x55c08b9f2240 2024-04-25T02:07:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0925ffb640 1 -- 172.21.15.31:0/2592965164 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f092c01a040 con 0x55c08b9f2240 2024-04-25T02:07:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0925ffb640 1 -- 172.21.15.31:0/2592965164 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f092c03a050 con 0x55c08b9f2240 2024-04-25T02:07:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0947210740 10 client.17163 fetch_fsmap learned FSMap version 40 2024-04-25T02:07:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0947210740 1 -- 172.21.15.31:0/2592965164 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c08ba73860 con 0x55c08b9f2240 2024-04-25T02:07:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0925ffb640 1 -- 172.21.15.31:0/2592965164 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0938005850 con 0x55c08b9f2240 2024-04-25T02:07:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0925ffb640 1 -- 172.21.15.31:0/2592965164 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f092c0250b0 con 0x55c08b9f2240 2024-04-25T02:07:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0947210740 10 client.17163 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:07:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0947210740 10 client.17163 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0947210740 20 client.17163 populate_metadata read hostname 'smithi031' 2024-04-25T02:07:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0947210740 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c08ba74080 0x55c08ba764a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0947210740 4 client.17163 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:07:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.751+0000 7f0947210740 1 -- 172.21.15.31:0/2592965164 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c08b9f4ee0 con 0x55c08ba74080 2024-04-25T02:07:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.752+0000 7f09422a5640 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c08ba74080 0x55c08ba764a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.752+0000 7f0925ffb640 10 client.17163 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.752+0000 7f09422a5640 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c08ba74080 0x55c08ba764a0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23a4322f5325368 server_cookie=ab7b15308d6eae9b in_seq=0 out_seq=0 2024-04-25T02:07:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0925ffb640 1 -- 172.21.15.31:0/2592965164 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c08b9f4ee0 con 0x55c08ba74080 2024-04-25T02:07:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0925ffb640 10 client.17163 handle_command_reply: tid=0 2024-04-25T02:07:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0947210740 1 client.17163 shutdown 2024-04-25T02:07:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0947210740 1 -- 172.21.15.31:0/2592965164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 msgr2=0x55c08ba05890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0947210740 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08ba05890 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f092c000c00 tx=0x7f092c028040 comp rx=0 tx=0).stop 2024-04-25T02:07:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0947210740 1 -- 172.21.15.31:0/2592965164 shutdown_connections 2024-04-25T02:07:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0947210740 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c08ba74080 0x55c08ba764a0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0947210740 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c08b9fd230 0x55c08b9f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0925ffb640 0 client.17163 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0947210740 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c08b9f2b60 0x55c08b9f6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0947210740 1 --2- 172.21.15.31:0/2592965164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c08b9f2240 0x55c08ba05890 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.782+0000 7f0947210740 1 -- 172.21.15.31:0/2592965164 >> 172.21.15.31:0/2592965164 conn(0x55c08b9ee560 msgr2=0x55c08b9ed5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0947210740 1 -- 172.21.15.31:0/2592965164 shutdown_connections 2024-04-25T02:07:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0947210740 1 -- 172.21.15.31:0/2592965164 wait complete. 2024-04-25T02:07:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0947210740 20 client.17163 trim_cache size 0 max 16384 2024-04-25T02:07:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:07:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:07:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:07:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:07:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0944509640 1 -- 172.21.15.31:0/3815792553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09280a2da0 msgr2=0x7f09280a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0944509640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09280a2da0 0x7f09280a5260 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f09340cd0e0 tx=0x7f09340cf670 comp rx=0 tx=0).stop 2024-04-25T02:07:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0944509640 1 -- 172.21.15.31:0/3815792553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c107740 msgr2=0x7f093c19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0944509640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c107740 0x7f093c19c7e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f093400cad0 tx=0x7f093400cf90 comp rx=0 tx=0).stop 2024-04-25T02:07:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0944509640 1 -- 172.21.15.31:0/3815792553 shutdown_connections 2024-04-25T02:07:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0944509640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09280a2da0 0x7f09280a5260 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0944509640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f093c107740 0x7f093c19c7e0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0944509640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f093c1020f0 0x7f093c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0944509640 1 --2- 172.21.15.31:0/3815792553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f093c101730 0x7f093c197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.783+0000 7f0944509640 1 -- 172.21.15.31:0/3815792553 >> 172.21.15.31:0/3815792553 conn(0x7f093c0fd200 msgr2=0x7f093c0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.784+0000 7f0944509640 1 -- 172.21.15.31:0/3815792553 shutdown_connections 2024-04-25T02:07:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:14.784+0000 7f0944509640 1 -- 172.21.15.31:0/3815792553 wait complete. 2024-04-25T02:07:15.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 257730 1.89 MB/sec execute 562 sec latency 23.509 ms 2024-04-25T02:07:16.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:15 smithi040 ceph-mon[32418]: pgmap v399: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 659 KiB/s wr, 80 op/s 2024-04-25T02:07:16.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:15 smithi134 ceph-mon[33098]: pgmap v399: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 659 KiB/s wr, 80 op/s 2024-04-25T02:07:16.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:15 smithi031 ceph-mon[29204]: pgmap v399: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 659 KiB/s wr, 80 op/s 2024-04-25T02:07:16.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 258216 1.90 MB/sec execute 563 sec latency 18.340 ms 2024-04-25T02:07:17.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 258751 1.90 MB/sec execute 564 sec latency 18.536 ms 2024-04-25T02:07:18.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:17 smithi040 ceph-mon[32418]: pgmap v400: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 659 KiB/s wr, 80 op/s 2024-04-25T02:07:18.198 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:17 smithi134 ceph-mon[33098]: pgmap v400: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 659 KiB/s wr, 80 op/s 2024-04-25T02:07:18.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:17 smithi031 ceph-mon[29204]: pgmap v400: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 659 KiB/s wr, 80 op/s 2024-04-25T02:07:18.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 259038 1.90 MB/sec execute 565 sec latency 490.827 ms 2024-04-25T02:07:19.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 259038 1.90 MB/sec execute 566 sec latency 1490.892 ms 2024-04-25T02:07:20.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:19 smithi040 ceph-mon[32418]: pgmap v401: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 659 KiB/s wr, 80 op/s 2024-04-25T02:07:20.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:19 smithi134 ceph-mon[33098]: pgmap v401: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 659 KiB/s wr, 80 op/s 2024-04-25T02:07:20.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:19 smithi031 ceph-mon[29204]: pgmap v401: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 659 KiB/s wr, 80 op/s 2024-04-25T02:07:20.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 259105 1.89 MB/sec execute 567 sec latency 2212.096 ms 2024-04-25T02:07:21.332 INFO:tasks.workunit.client.0.smithi031.stdout: 1 259105 1.89 MB/sec execute 568 sec latency 1022.731 ms 2024-04-25T02:07:22.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:21 smithi040 ceph-mon[32418]: pgmap v402: 97 pgs: 97 active+clean; 142 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:07:22.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:21 smithi134 ceph-mon[33098]: pgmap v402: 97 pgs: 97 active+clean; 142 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:07:22.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:21 smithi031 ceph-mon[29204]: pgmap v402: 97 pgs: 97 active+clean; 142 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:07:22.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 259105 1.89 MB/sec execute 569 sec latency 2022.767 ms 2024-04-25T02:07:23.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 259105 1.88 MB/sec execute 570 sec latency 3022.804 ms 2024-04-25T02:07:24.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:23 smithi040 ceph-mon[32418]: pgmap v403: 97 pgs: 97 active+clean; 142 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-04-25T02:07:24.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:23 smithi134 ceph-mon[33098]: pgmap v403: 97 pgs: 97 active+clean; 142 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-04-25T02:07:24.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:23 smithi031 ceph-mon[29204]: pgmap v403: 97 pgs: 97 active+clean; 142 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-04-25T02:07:24.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 259105 1.88 MB/sec execute 571 sec latency 4022.891 ms 2024-04-25T02:07:25.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 259195 1.88 MB/sec execute 572 sec latency 4738.696 ms 2024-04-25T02:07:26.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:25 smithi040 ceph-mon[32418]: pgmap v404: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:07:26.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:25 smithi134 ceph-mon[33098]: pgmap v404: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:07:26.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:25 smithi031 ceph-mon[29204]: pgmap v404: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-25T02:07:26.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 259416 1.88 MB/sec execute 573 sec latency 14.883 ms 2024-04-25T02:07:27.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 259793 1.87 MB/sec execute 574 sec latency 15.893 ms 2024-04-25T02:07:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.885+0000 7f1f5ae12640 1 -- 172.21.15.31:0/3774799131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f541032f0 msgr2=0x7f1f5410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.885+0000 7f1f5ae12640 1 --2- 172.21.15.31:0/3774799131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f541032f0 0x7f1f5410b7d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009900 tx=0x7f1f4802a510 comp rx=0 tx=0).stop 2024-04-25T02:07:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.885+0000 7f1f5ae12640 1 -- 172.21.15.31:0/3774799131 shutdown_connections 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.885+0000 7f1f5ae12640 1 --2- 172.21.15.31:0/3774799131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f5410bd10 0x7f1f5410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.885+0000 7f1f5ae12640 1 --2- 172.21.15.31:0/3774799131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f541032f0 0x7f1f5410b7d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.885+0000 7f1f5ae12640 1 --2- 172.21.15.31:0/3774799131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f541029d0 0x7f1f54102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.885+0000 7f1f5ae12640 1 -- 172.21.15.31:0/3774799131 >> 172.21.15.31:0/3774799131 conn(0x7f1f540777c0 msgr2=0x7f1f54077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.885+0000 7f1f5ae12640 1 -- 172.21.15.31:0/3774799131 shutdown_connections 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.885+0000 7f1f5ae12640 1 -- 172.21.15.31:0/3774799131 wait complete. 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f5ae12640 1 Processor -- start 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f5ae12640 1 -- start start 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f5ae12640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f541029d0 0x7f1f54100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f5ae12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f541032f0 0x7f1f54100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f5ae12640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f5410bd10 0x7f1f541010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f5ae12640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541119c0 con 0x7f1f541029d0 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f5ae12640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1f54111840 con 0x7f1f541032f0 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f5ae12640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1f54111b40 con 0x7f1f5410bd10 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f58bae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f541029d0 0x7f1f54100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f53fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f541032f0 0x7f1f54100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f58bae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f541029d0 0x7f1f54100620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36480/0 (socket says 172.21.15.31:36480) 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f58bae640 1 -- 172.21.15.31:0/717196140 learned_addr learned my addr 172.21.15.31:0/717196140 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f593af640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f5410bd10 0x7f1f541010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f58bae640 1 -- 172.21.15.31:0/717196140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f5410bd10 msgr2=0x7f1f541010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f58bae640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f5410bd10 0x7f1f541010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f58bae640 1 -- 172.21.15.31:0/717196140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f541032f0 msgr2=0x7f1f54100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f58bae640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f541032f0 0x7f1f54100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f58bae640 1 -- 172.21.15.31:0/717196140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f541a63e0 con 0x7f1f541029d0 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.886+0000 7f1f53fff640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f541032f0 0x7f1f54100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.887+0000 7f1f58bae640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f541029d0 0x7f1f54100620 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400e9f0 tx=0x7f1f4400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.887+0000 7f1f51ffb640 1 -- 172.21.15.31:0/717196140 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4400cd80 con 0x7f1f541029d0 2024-04-25T02:07:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.887+0000 7f1f5ae12640 1 -- 172.21.15.31:0/717196140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f541a66d0 con 0x7f1f541029d0 2024-04-25T02:07:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.887+0000 7f1f51ffb640 1 -- 172.21.15.31:0/717196140 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1f440054d0 con 0x7f1f541029d0 2024-04-25T02:07:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.887+0000 7f1f51ffb640 1 -- 172.21.15.31:0/717196140 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44010490 con 0x7f1f541029d0 2024-04-25T02:07:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.887+0000 7f1f5ae12640 1 -- 172.21.15.31:0/717196140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f541a6bf0 con 0x7f1f541029d0 2024-04-25T02:07:27.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.889+0000 7f1f51ffb640 1 -- 172.21.15.31:0/717196140 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1f44005020 con 0x7f1f541029d0 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f5dbf6740 1 Processor -- start 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f5dbf6740 1 -- start start 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033bf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f51ffb640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1f3c0a2da0 0x7f1f3c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556403323cb0 con 0x5564033bfb60 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5564032cc5b0 con 0x5564033bf240 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55640331fc70 con 0x5564033ca230 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f58bae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f593af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033bf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.890+0000 7f1f53fff640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1f3c0a2da0 0x7f1f3c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f593af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033bf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49110/0 (socket says 172.21.15.31:49110) 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f593af640 1 -- 172.21.15.31:0/875998008 learned_addr learned my addr 172.21.15.31:0/875998008 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f593af640 1 --2- 172.21.15.31:0/875998008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f51ffb640 1 -- 172.21.15.31:0/717196140 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f1f440cf790 con 0x7f1f541029d0 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f593af640 1 -- 172.21.15.31:0/875998008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 msgr2=0x5564033cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f593af640 1 --2- 172.21.15.31:0/875998008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f593af640 1 -- 172.21.15.31:0/875998008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 msgr2=0x5564033c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f593af640 1 --2- 172.21.15.31:0/875998008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f593af640 1 -- 172.21.15.31:0/875998008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556403038210 con 0x5564033bf240 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f593af640 1 --2- 172.21.15.31:0/875998008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033bf620 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c005f40 tx=0x7f1f4c005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b69af7d815c5dbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f53fff640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1f3c0a2da0 0x7f1f3c0a5260 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1f48031440 tx=0x7f1f48005b50 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f3affd640 1 -- 172.21.15.31:0/875998008 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c004b20 con 0x5564033bf240 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f3affd640 1 -- 172.21.15.31:0/875998008 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1f4c004cc0 con 0x5564033bf240 2024-04-25T02:07:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.891+0000 7f1f3affd640 1 -- 172.21.15.31:0/875998008 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c004050 con 0x5564033bf240 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.892+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/875998008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 msgr2=0x5564033bf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.892+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/875998008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033bf620 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c005f40 tx=0x7f1f4c005bd0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.892+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/875998008 shutdown_connections 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.892+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/875998008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.892+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/875998008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.892+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/875998008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033bf620 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.892+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/875998008 >> 172.21.15.31:0/875998008 conn(0x5564033bb560 msgr2=0x5564033bb940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.892+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/875998008 shutdown_connections 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.892+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/875998008 wait complete. 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f5dbf6740 1 -- start start 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f5dbf6740 1 Processor -- start 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5564033cd020 con 0x5564033bfb60 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5564032dfb50 con 0x5564033bf240 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5564033cd1a0 con 0x5564033ca230 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f5dbf6740 10 client.0 _ll_register_callbacks cb 0x55640331aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f58bae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f53fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f58bae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49112/0 (socket says 172.21.15.31:49112) 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f58bae640 1 -- 172.21.15.31:0/1215124959 learned_addr learned my addr 172.21.15.31:0/1215124959 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f593af640 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.893+0000 7f1f58bae640 1 -- 172.21.15.31:0/1215124959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 msgr2=0x5564033c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f58bae640 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f58bae640 1 -- 172.21.15.31:0/1215124959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 msgr2=0x5564033c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f58bae640 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f58bae640 1 -- 172.21.15.31:0/1215124959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556403104620 con 0x5564033bf240 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f58bae640 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033c71b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1f440986a0 tx=0x7f1f4409baa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ae726cc8fac39dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f33fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f33fff640 1 -- 172.21.15.31:0/1215124959 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44070060 con 0x5564033bf240 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f33fff640 1 -- 172.21.15.31:0/1215124959 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f1f44066a30 con 0x5564033bf240 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/1215124959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f4c004cc0 con 0x5564033bf240 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f33fff640 1 -- 172.21.15.31:0/1215124959 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1f4406a070 con 0x5564033bf240 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f5dbf6740 10 client.26750 fetch_fsmap learned FSMap version 40 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/1215124959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5564033d9260 con 0x5564033bf240 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.894+0000 7f1f33fff640 1 -- 172.21.15.31:0/1215124959 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1f4c004cc0 con 0x5564033bf240 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.895+0000 7f1f33fff640 1 -- 172.21.15.31:0/1215124959 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f1f440a44b0 con 0x5564033bf240 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.895+0000 7f1f5dbf6740 10 client.26750 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.895+0000 7f1f5dbf6740 10 client.26750 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.895+0000 7f1f5dbf6740 20 client.26750 populate_metadata read hostname 'smithi031' 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.895+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5564033da720 0x5564033dcb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.895+0000 7f1f5dbf6740 4 client.26750 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:07:27.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.895+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/1215124959 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564033311f0 con 0x5564033da720 2024-04-25T02:07:27.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.895+0000 7f1f53fff640 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5564033da720 0x5564033dcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.895+0000 7f1f33fff640 10 client.26750 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:27.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.895+0000 7f1f53fff640 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5564033da720 0x5564033dcb90 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddddc5a61e42c9b3 server_cookie=e7ae20ea1331016c in_seq=0 out_seq=0 2024-04-25T02:07:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f33fff640 1 -- 172.21.15.31:0/1215124959 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5564033311f0 con 0x5564033da720 2024-04-25T02:07:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f33fff640 10 client.26750 handle_command_reply: tid=0 2024-04-25T02:07:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 1 client.26750 shutdown 2024-04-25T02:07:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/1215124959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 msgr2=0x5564033c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033c71b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1f440986a0 tx=0x7f1f4409baa0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/1215124959 shutdown_connections 2024-04-25T02:07:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5564033da720 0x5564033dcb90 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f33fff640 0 client.26750 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/1215124959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033c71b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/1215124959 >> 172.21.15.31:0/1215124959 conn(0x5564033bb560 msgr2=0x5564033c1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/1215124959 shutdown_connections 2024-04-25T02:07:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/1215124959 wait complete. 2024-04-25T02:07:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.897+0000 7f1f5dbf6740 20 client.26750 trim_cache size 0 max 16384 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.913+0000 7f1f5dbf6740 1 Processor -- start 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f5dbf6740 1 -- start start 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bf240 0x5564033d5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bfb60 0x5564033c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033ba440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541af4a0 con 0x5564033bf240 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541b1160 con 0x5564033bfb60 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541afb70 con 0x5564033ca230 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f58bae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bf240 0x5564033d5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f58bae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bf240 0x5564033d5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36514/0 (socket says 172.21.15.31:36514) 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f53fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bfb60 0x5564033c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f58bae640 1 -- 172.21.15.31:0/2835541641 learned_addr learned my addr 172.21.15.31:0/2835541641 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f593af640 1 --2- 172.21.15.31:0/2835541641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033ba440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f58bae640 1 -- 172.21.15.31:0/2835541641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 msgr2=0x5564033ba440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f58bae640 1 --2- 172.21.15.31:0/2835541641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033ba440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f58bae640 1 -- 172.21.15.31:0/2835541641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bfb60 msgr2=0x5564033c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f58bae640 1 --2- 172.21.15.31:0/2835541641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bfb60 0x5564033c1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f58bae640 1 -- 172.21.15.31:0/2835541641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564033c1fa0 con 0x5564033bf240 2024-04-25T02:07:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f53fff640 1 --2- 172.21.15.31:0/2835541641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bfb60 0x5564033c1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.914+0000 7f1f58bae640 1 --2- 172.21.15.31:0/2835541641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bf240 0x5564033d5ee0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400c6a0 tx=0x7f1f4409a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1d3aabd37a5178 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.915+0000 7f1f3a7fc640 1 -- 172.21.15.31:0/2835541641 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f440b9070 con 0x5564033bf240 2024-04-25T02:07:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.915+0000 7f1f3a7fc640 1 -- 172.21.15.31:0/2835541641 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1f44005310 con 0x5564033bf240 2024-04-25T02:07:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.915+0000 7f1f3a7fc640 1 -- 172.21.15.31:0/2835541641 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f440a0080 con 0x5564033bf240 2024-04-25T02:07:27.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.915+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2835541641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bf240 msgr2=0x5564033d5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:27.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.915+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/2835541641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bf240 0x5564033d5ee0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400c6a0 tx=0x7f1f4409a1e0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.915+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2835541641 shutdown_connections 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.915+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/2835541641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033ba440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.915+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/2835541641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bfb60 0x5564033c1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.915+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/2835541641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bf240 0x5564033d5ee0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.915+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2835541641 >> 172.21.15.31:0/2835541641 conn(0x5564033bb560 msgr2=0x5564033b9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2835541641 shutdown_connections 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2835541641 wait complete. 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f5dbf6740 1 -- start start 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f5dbf6740 1 Processor -- start 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033d2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f5dbf6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541b0a90 con 0x5564033bfb60 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541b0300 con 0x5564033bf240 2024-04-25T02:07:27.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f5dbf6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541ad4a0 con 0x5564033ca230 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f5dbf6740 10 client.0 _ll_register_callbacks cb 0x7f1f44007f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f53fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.916+0000 7f1f58bae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033d2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f58bae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033d2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49132/0 (socket says 172.21.15.31:49132) 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f53fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36524/0 (socket says 172.21.15.31:36524) 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f58bae640 1 -- 172.21.15.31:0/2529680716 learned_addr learned my addr 172.21.15.31:0/2529680716 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f593af640 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f58bae640 1 -- 172.21.15.31:0/2529680716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 msgr2=0x5564033c3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f58bae640 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033c3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f58bae640 1 -- 172.21.15.31:0/2529680716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 msgr2=0x5564033c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f58bae640 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f58bae640 1 -- 172.21.15.31:0/2529680716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564033c45b0 con 0x5564033bf240 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f53fff640 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f58bae640 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033d2650 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1f44004a40 tx=0x7f1f44009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec34a55737d71492 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f33fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f33fff640 1 -- 172.21.15.31:0/2529680716 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f440b9070 con 0x5564033bf240 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2529680716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55640332b540 con 0x5564033bf240 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f33fff640 1 -- 172.21.15.31:0/2529680716 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f1f440a4b90 con 0x5564033bf240 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f33fff640 1 -- 172.21.15.31:0/2529680716 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1f440ac050 con 0x5564033bf240 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f5dbf6740 10 client.26756 fetch_fsmap learned FSMap version 40 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2529680716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x556403440bb0 con 0x5564033bf240 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.917+0000 7f1f33fff640 1 -- 172.21.15.31:0/2529680716 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55640332b540 con 0x5564033bf240 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.918+0000 7f1f33fff640 1 -- 172.21.15.31:0/2529680716 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f1f4409ab90 con 0x5564033bf240 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.918+0000 7f1f5dbf6740 10 client.26756 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.918+0000 7f1f5dbf6740 10 client.26756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.918+0000 7f1f5dbf6740 20 client.26756 populate_metadata read hostname 'smithi031' 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.918+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5564034410e0 0x556403443500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.918+0000 7f1f5dbf6740 4 client.26756 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.918+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2529680716 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5564032a98e0 con 0x5564034410e0 2024-04-25T02:07:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.918+0000 7f1f53fff640 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5564034410e0 0x556403443500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:27.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.918+0000 7f1f33fff640 10 client.26756 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:27.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.918+0000 7f1f53fff640 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5564034410e0 0x556403443500 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e5e421863140229 server_cookie=eafe617f196feee4 in_seq=0 out_seq=0 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.945+0000 7f1f33fff640 1 -- 172.21.15.31:0/2529680716 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5564032a98e0 con 0x5564034410e0 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.945+0000 7f1f33fff640 10 client.26756 handle_command_reply: tid=0 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.945+0000 7f1f5dbf6740 1 client.26756 shutdown 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.945+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2529680716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 msgr2=0x5564033d2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.945+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033d2650 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1f44004a40 tx=0x7f1f44009310 comp rx=0 tx=0).stop 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2529680716 shutdown_connections 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5564034410e0 0x556403443500 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5564033ca230 0x5564033c3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5564033bfb60 0x5564033c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5dbf6740 1 --2- 172.21.15.31:0/2529680716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564033bf240 0x5564033d2650 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f33fff640 0 client.26756 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2529680716 >> 172.21.15.31:0/2529680716 conn(0x5564033bb560 msgr2=0x5564033ba5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:27.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2529680716 shutdown_connections 2024-04-25T02:07:27.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5dbf6740 1 -- 172.21.15.31:0/2529680716 wait complete. 2024-04-25T02:07:27.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5dbf6740 20 client.26756 trim_cache size 0 max 16384 2024-04-25T02:07:27.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:07:27.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:07:27.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:07:27.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:07:27.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5ae12640 1 -- 172.21.15.31:0/717196140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1f3c0a2da0 msgr2=0x7f1f3c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:27.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5ae12640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1f3c0a2da0 0x7f1f3c0a5260 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1f48031440 tx=0x7f1f48005b50 comp rx=0 tx=0).stop 2024-04-25T02:07:27.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5ae12640 1 -- 172.21.15.31:0/717196140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f541029d0 msgr2=0x7f1f54100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:27.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5ae12640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f541029d0 0x7f1f54100620 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400e9f0 tx=0x7f1f4400edf0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5ae12640 1 -- 172.21.15.31:0/717196140 shutdown_connections 2024-04-25T02:07:27.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5ae12640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1f3c0a2da0 0x7f1f3c0a5260 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5ae12640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1f5410bd10 0x7f1f541010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.946+0000 7f1f5ae12640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f541032f0 0x7f1f54100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.947+0000 7f1f5ae12640 1 --2- 172.21.15.31:0/717196140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f541029d0 0x7f1f54100620 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:27.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.947+0000 7f1f5ae12640 1 -- 172.21.15.31:0/717196140 >> 172.21.15.31:0/717196140 conn(0x7f1f540777c0 msgr2=0x7f1f540fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:27.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.947+0000 7f1f5ae12640 1 -- 172.21.15.31:0/717196140 shutdown_connections 2024-04-25T02:07:27.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:27.947+0000 7f1f5ae12640 1 -- 172.21.15.31:0/717196140 wait complete. 2024-04-25T02:07:28.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:27 smithi040 ceph-mon[32418]: pgmap v405: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 575 KiB/s wr, 51 op/s 2024-04-25T02:07:28.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:27 smithi134 ceph-mon[33098]: pgmap v405: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 575 KiB/s wr, 51 op/s 2024-04-25T02:07:28.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:27 smithi031 ceph-mon[29204]: pgmap v405: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 575 KiB/s wr, 51 op/s 2024-04-25T02:07:28.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 260230 1.87 MB/sec execute 575 sec latency 19.565 ms 2024-04-25T02:07:29.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 260734 1.88 MB/sec execute 576 sec latency 25.593 ms 2024-04-25T02:07:30.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:29 smithi040 ceph-mon[32418]: pgmap v406: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 575 KiB/s wr, 51 op/s 2024-04-25T02:07:30.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:29 smithi134 ceph-mon[33098]: pgmap v406: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 575 KiB/s wr, 51 op/s 2024-04-25T02:07:30.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:29 smithi031 ceph-mon[29204]: pgmap v406: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 575 KiB/s wr, 51 op/s 2024-04-25T02:07:30.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 261305 1.88 MB/sec execute 577 sec latency 21.656 ms 2024-04-25T02:07:31.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 261847 1.88 MB/sec execute 578 sec latency 18.211 ms 2024-04-25T02:07:32.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:31 smithi040 ceph-mon[32418]: pgmap v407: 97 pgs: 97 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.1 MiB/s wr, 111 op/s 2024-04-25T02:07:32.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:31 smithi134 ceph-mon[33098]: pgmap v407: 97 pgs: 97 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.1 MiB/s wr, 111 op/s 2024-04-25T02:07:32.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:31 smithi031 ceph-mon[29204]: pgmap v407: 97 pgs: 97 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.1 MiB/s wr, 111 op/s 2024-04-25T02:07:32.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 262309 1.88 MB/sec execute 579 sec latency 11.675 ms 2024-04-25T02:07:33.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 262584 1.88 MB/sec execute 580 sec latency 543.275 ms 2024-04-25T02:07:34.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:33 smithi134 ceph-mon[33098]: pgmap v408: 97 pgs: 97 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 579 KiB/s wr, 71 op/s 2024-04-25T02:07:34.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:33 smithi031 ceph-mon[29204]: pgmap v408: 97 pgs: 97 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 579 KiB/s wr, 71 op/s 2024-04-25T02:07:34.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 262584 1.88 MB/sec execute 581 sec latency 1543.395 ms 2024-04-25T02:07:34.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:33 smithi040 ceph-mon[32418]: pgmap v408: 97 pgs: 97 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 579 KiB/s wr, 71 op/s 2024-04-25T02:07:35.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 262651 1.88 MB/sec execute 582 sec latency 2261.616 ms 2024-04-25T02:07:36.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:35 smithi134 ceph-mon[33098]: pgmap v409: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-25T02:07:36.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:35 smithi031 ceph-mon[29204]: pgmap v409: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-25T02:07:36.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 262651 1.88 MB/sec execute 583 sec latency 1030.792 ms 2024-04-25T02:07:36.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:35 smithi040 ceph-mon[32418]: pgmap v409: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-25T02:07:37.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 262651 1.87 MB/sec execute 584 sec latency 2030.935 ms 2024-04-25T02:07:38.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:37 smithi134 ceph-mon[33098]: pgmap v410: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-25T02:07:38.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:37 smithi031 ceph-mon[29204]: pgmap v410: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-25T02:07:38.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 262651 1.87 MB/sec execute 585 sec latency 3031.007 ms 2024-04-25T02:07:38.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:37 smithi040 ceph-mon[32418]: pgmap v410: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-25T02:07:39.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 262651 1.87 MB/sec execute 586 sec latency 4031.042 ms 2024-04-25T02:07:40.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:39 smithi134 ceph-mon[33098]: pgmap v411: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-25T02:07:40.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:39 smithi031 ceph-mon[29204]: pgmap v411: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-25T02:07:40.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 262744 1.86 MB/sec execute 587 sec latency 4749.013 ms 2024-04-25T02:07:40.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:39 smithi040 ceph-mon[32418]: pgmap v411: 97 pgs: 97 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 95 op/s 2024-04-25T02:07:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.040+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/3172473847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed0101030 msgr2=0x7f3ed010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.040+0000 7f3ed6bb7640 1 --2- 172.21.15.31:0/3172473847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed0101030 0x7f3ed010d4f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009cd0 tx=0x7f3ec802a8e0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.040+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/3172473847 shutdown_connections 2024-04-25T02:07:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.040+0000 7f3ed6bb7640 1 --2- 172.21.15.31:0/3172473847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed0101030 0x7f3ed010d4f0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.040+0000 7f3ed6bb7640 1 --2- 172.21.15.31:0/3172473847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ed0100670 0x7f3ed0100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.040+0000 7f3ed6bb7640 1 --2- 172.21.15.31:0/3172473847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed0106630 0x7f3ed0106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.040+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/3172473847 >> 172.21.15.31:0/3172473847 conn(0x7f3ed00777c0 msgr2=0x7f3ed0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.040+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/3172473847 shutdown_connections 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.040+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/3172473847 wait complete. 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed6bb7640 1 Processor -- start 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed6bb7640 1 -- start start 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed6bb7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed0100670 0x7f3ed00fd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed6bb7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed0101030 0x7f3ed00fdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed6bb7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ed0106630 0x7f3ed00fe080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed6bb7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01108f0 con 0x7f3ed0100670 2024-04-25T02:07:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed6bb7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed0110770 con 0x7f3ed0101030 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed6bb7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed0110a70 con 0x7f3ed0106630 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed4953640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed0100670 0x7f3ed00fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed4953640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed0100670 0x7f3ed00fd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48826/0 (socket says 172.21.15.31:48826) 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ecffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed0101030 0x7f3ed00fdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed4953640 1 -- 172.21.15.31:0/1732033411 learned_addr learned my addr 172.21.15.31:0/1732033411 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.041+0000 7f3ed5154640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ed0106630 0x7f3ed00fe080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.042+0000 7f3ecffff640 1 -- 172.21.15.31:0/1732033411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ed0106630 msgr2=0x7f3ed00fe080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.042+0000 7f3ecffff640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ed0106630 0x7f3ed00fe080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.042+0000 7f3ecffff640 1 -- 172.21.15.31:0/1732033411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed0100670 msgr2=0x7f3ed00fd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.042+0000 7f3ecffff640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed0100670 0x7f3ed00fd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.042+0000 7f3ecffff640 1 -- 172.21.15.31:0/1732033411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed01a63e0 con 0x7f3ed0101030 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.042+0000 7f3ecffff640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed0101030 0x7f3ed00fdb40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00e9f0 tx=0x7f3ebc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.042+0000 7f3ecdffb640 1 -- 172.21.15.31:0/1732033411 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc00ccc0 con 0x7f3ed0101030 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.042+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/1732033411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed01a66d0 con 0x7f3ed0101030 2024-04-25T02:07:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.042+0000 7f3ecdffb640 1 -- 172.21.15.31:0/1732033411 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ebc005020 con 0x7f3ed0101030 2024-04-25T02:07:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.042+0000 7f3ecdffb640 1 -- 172.21.15.31:0/1732033411 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc004a00 con 0x7f3ed0101030 2024-04-25T02:07:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.042+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/1732033411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed01a6bd0 con 0x7f3ed0101030 2024-04-25T02:07:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ecdffb640 1 -- 172.21.15.31:0/1732033411 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3ebc004ba0 con 0x7f3ed0101030 2024-04-25T02:07:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ed99a2740 1 Processor -- start 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ed99a2740 1 -- start start 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ecdffb640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ec00a2da0 0x7f3ec00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b51f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b529cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 0x55d53b52c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d53b483cb0 con 0x55d53b51fb60 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d53b42c5b0 con 0x55d53b52a230 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d53b47fc70 con 0x55d53b51f240 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ed5154640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ec00a2da0 0x7f3ec00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ecdffb640 1 -- 172.21.15.31:0/1732033411 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f3ebc0cf3e0 con 0x7f3ed0101030 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.045+0000 7f3ecffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b529cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ecffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b529cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48830/0 (socket says 172.21.15.31:48830) 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ecffff640 1 -- 172.21.15.31:0/3853659947 learned_addr learned my addr 172.21.15.31:0/3853659947 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ed4953640 1 --2- 172.21.15.31:0/3853659947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b51f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ed5154640 1 --2- 172.21.15.31:0/3853659947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 0x55d53b52c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ecffff640 1 -- 172.21.15.31:0/3853659947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 msgr2=0x55d53b51f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ecffff640 1 --2- 172.21.15.31:0/3853659947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b51f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ecffff640 1 -- 172.21.15.31:0/3853659947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 msgr2=0x55d53b52c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ecffff640 1 --2- 172.21.15.31:0/3853659947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 0x55d53b52c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ecffff640 1 -- 172.21.15.31:0/3853659947 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d53b198210 con 0x55d53b51fb60 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ed5154640 1 --2- 172.21.15.31:0/3853659947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 0x55d53b52c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ed4953640 1 --2- 172.21.15.31:0/3853659947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b51f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ecffff640 1 --2- 172.21.15.31:0/3853659947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b529cf0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc097d80 tx=0x7f3ebc09c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675376be13e53f27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ed5154640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ec00a2da0 0x7f3ec00a5260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8005e10 tx=0x7f3ec8005ef0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ebaffd640 1 -- 172.21.15.31:0/3853659947 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc01c030 con 0x55d53b51fb60 2024-04-25T02:07:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ebaffd640 1 -- 172.21.15.31:0/3853659947 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ebc09fcd0 con 0x55d53b51fb60 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.046+0000 7f3ebaffd640 1 -- 172.21.15.31:0/3853659947 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc0052f0 con 0x55d53b51fb60 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.047+0000 7f3ed99a2740 1 -- 172.21.15.31:0/3853659947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 msgr2=0x55d53b529cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.047+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/3853659947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b529cf0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc097d80 tx=0x7f3ebc09c0e0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.047+0000 7f3ed99a2740 1 -- 172.21.15.31:0/3853659947 shutdown_connections 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.047+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/3853659947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 0x55d53b52c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.047+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/3853659947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b529cf0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.047+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/3853659947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b51f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.047+0000 7f3ed99a2740 1 -- 172.21.15.31:0/3853659947 >> 172.21.15.31:0/3853659947 conn(0x55d53b51b560 msgr2=0x55d53b51b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.047+0000 7f3ed99a2740 1 -- 172.21.15.31:0/3853659947 shutdown_connections 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.047+0000 7f3ed99a2740 1 -- 172.21.15.31:0/3853659947 wait complete. 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.047+0000 7f3ed99a2740 1 -- start start 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed99a2740 1 Processor -- start 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b51f240 0x55d53b527210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b527750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b52a230 0x55d53b527c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d53b52d020 con 0x55d53b51fb60 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d53b43fb50 con 0x55d53b51f240 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d53b52d1a0 con 0x55d53b52a230 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed99a2740 10 client.0 _ll_register_callbacks cb 0x55d53b47aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ecffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b527750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed4953640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b51f240 0x55d53b527210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ecffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b527750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48844/0 (socket says 172.21.15.31:48844) 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ecffff640 1 -- 172.21.15.31:0/1703694383 learned_addr learned my addr 172.21.15.31:0/1703694383 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed5154640 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b52a230 0x55d53b527c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed4953640 1 -- 172.21.15.31:0/1703694383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b52a230 msgr2=0x55d53b527c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed4953640 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b52a230 0x55d53b527c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed4953640 1 -- 172.21.15.31:0/1703694383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 msgr2=0x55d53b527750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.048+0000 7f3ed4953640 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b527750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.049+0000 7f3ed4953640 1 -- 172.21.15.31:0/1703694383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d53b264620 con 0x55d53b51f240 2024-04-25T02:07:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.049+0000 7f3ed4953640 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b51f240 0x55d53b527210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4012740 tx=0x7f3ec4012c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0062972a900e80e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.049+0000 7f3eaffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.049+0000 7f3eaffff640 1 -- 172.21.15.31:0/1703694383 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec400f8c0 con 0x55d53b51f240 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.049+0000 7f3eaffff640 1 -- 172.21.15.31:0/1703694383 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f3ec400fa60 con 0x55d53b51f240 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.049+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1703694383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ebc09fcd0 con 0x55d53b51f240 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.049+0000 7f3eaffff640 1 -- 172.21.15.31:0/1703694383 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3ec400fc60 con 0x55d53b51f240 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.049+0000 7f3ed99a2740 10 client.26768 fetch_fsmap learned FSMap version 40 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.049+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1703694383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55d53b5392d0 con 0x55d53b51f240 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.049+0000 7f3eaffff640 1 -- 172.21.15.31:0/1703694383 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ebc09fcd0 con 0x55d53b51f240 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.050+0000 7f3eaffff640 1 -- 172.21.15.31:0/1703694383 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f3ec4010430 con 0x55d53b51f240 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.050+0000 7f3ed99a2740 10 client.26768 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.050+0000 7f3ed99a2740 10 client.26768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.050+0000 7f3ed99a2740 20 client.26768 populate_metadata read hostname 'smithi031' 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.050+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d53b53a790 0x55d53b53cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.050+0000 7f3ed99a2740 4 client.26768 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:07:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.050+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1703694383 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d53b4911f0 con 0x55d53b53a790 2024-04-25T02:07:41.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.050+0000 7f3ecffff640 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d53b53a790 0x55d53b53cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.050+0000 7f3ecffff640 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d53b53a790 0x55d53b53cc30 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f540ea911ce61e1 server_cookie=326d2d7771d40211 in_seq=0 out_seq=0 2024-04-25T02:07:41.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.050+0000 7f3eaffff640 10 client.26768 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.051+0000 7f3eaffff640 1 -- 172.21.15.31:0/1703694383 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d53b4911f0 con 0x55d53b53a790 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.051+0000 7f3eaffff640 10 client.26768 handle_command_reply: tid=0 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.051+0000 7f3ed99a2740 1 client.26768 shutdown 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.051+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1703694383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b51f240 msgr2=0x55d53b527210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.051+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b51f240 0x55d53b527210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4012740 tx=0x7f3ec4012c00 comp rx=0 tx=0).stop 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.052+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1703694383 shutdown_connections 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.052+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d53b53a790 0x55d53b53cc30 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.052+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b52a230 0x55d53b527c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.052+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b527750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.052+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1703694383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b51f240 0x55d53b527210 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.052+0000 7f3eaffff640 0 client.26768 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.052+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1703694383 >> 172.21.15.31:0/1703694383 conn(0x55d53b51b560 msgr2=0x55d53b521360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.052+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1703694383 shutdown_connections 2024-04-25T02:07:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.052+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1703694383 wait complete. 2024-04-25T02:07:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.052+0000 7f3ed99a2740 20 client.26768 trim_cache size 0 max 16384 2024-04-25T02:07:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.067+0000 7f3ed99a2740 1 Processor -- start 2024-04-25T02:07:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed99a2740 1 -- start start 2024-04-25T02:07:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b529940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b51fb60 0x55d53b535ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b52a230 0x55d53b526390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01af430 con 0x55d53b52a230 2024-04-25T02:07:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01b10f0 con 0x55d53b51fb60 2024-04-25T02:07:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01afb00 con 0x55d53b51f240 2024-04-25T02:07:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed4953640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b529940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed5154640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b52a230 0x55d53b526390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed4953640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b529940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:42920/0 (socket says 172.21.15.31:42920) 2024-04-25T02:07:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed4953640 1 -- 172.21.15.31:0/2072537650 learned_addr learned my addr 172.21.15.31:0/2072537650 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ecffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b51fb60 0x55d53b535ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed4953640 1 -- 172.21.15.31:0/2072537650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b51fb60 msgr2=0x55d53b535ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed4953640 1 --2- 172.21.15.31:0/2072537650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b51fb60 0x55d53b535ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed4953640 1 -- 172.21.15.31:0/2072537650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b52a230 msgr2=0x55d53b526390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed4953640 1 --2- 172.21.15.31:0/2072537650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b52a230 0x55d53b526390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed4953640 1 -- 172.21.15.31:0/2072537650 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d53b5158b0 con 0x55d53b51f240 2024-04-25T02:07:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed5154640 1 --2- 172.21.15.31:0/2072537650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b52a230 0x55d53b526390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.068+0000 7f3ed4953640 1 --2- 172.21.15.31:0/2072537650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b529940 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40108d0 tx=0x7f3ec400ff90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cd395ee5ae42aba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3eba7fc640 1 -- 172.21.15.31:0/2072537650 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4029030 con 0x55d53b51f240 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3eba7fc640 1 -- 172.21.15.31:0/2072537650 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ec4012e50 con 0x55d53b51f240 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3eba7fc640 1 -- 172.21.15.31:0/2072537650 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4005120 con 0x55d53b51f240 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3ed99a2740 1 -- 172.21.15.31:0/2072537650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 msgr2=0x55d53b529940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/2072537650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b529940 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40108d0 tx=0x7f3ec400ff90 comp rx=0 tx=0).stop 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3ed99a2740 1 -- 172.21.15.31:0/2072537650 shutdown_connections 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/2072537650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b52a230 0x55d53b526390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/2072537650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b51fb60 0x55d53b535ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/2072537650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b529940 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3ed99a2740 1 -- 172.21.15.31:0/2072537650 >> 172.21.15.31:0/2072537650 conn(0x55d53b51b560 msgr2=0x55d53b519d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3ed99a2740 1 -- 172.21.15.31:0/2072537650 shutdown_connections 2024-04-25T02:07:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.069+0000 7f3ed99a2740 1 -- 172.21.15.31:0/2072537650 wait complete. 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.070+0000 7f3ed99a2740 1 -- start start 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.070+0000 7f3ed99a2740 1 Processor -- start 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.070+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b5326c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.070+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b523ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.070+0000 7f3ed99a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 0x55d53b524020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.070+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01b0a20 con 0x55d53b51fb60 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.070+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01b0290 con 0x55d53b52a230 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.070+0000 7f3ed99a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01ad430 con 0x55d53b51f240 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.070+0000 7f3ed99a2740 10 client.0 _ll_register_callbacks cb 0x7f3ec4009940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.070+0000 7f3ecffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b523ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ed5154640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 0x55d53b524020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ed4953640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b5326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ecffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b523ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48854/0 (socket says 172.21.15.31:48854) 2024-04-25T02:07:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ecffff640 1 -- 172.21.15.31:0/1026343560 learned_addr learned my addr 172.21.15.31:0/1026343560 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ed5154640 1 -- 172.21.15.31:0/1026343560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 msgr2=0x55d53b5326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ed5154640 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b5326c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ed5154640 1 -- 172.21.15.31:0/1026343560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 msgr2=0x55d53b523ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ed5154640 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b523ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ed5154640 1 -- 172.21.15.31:0/1026343560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d53b524760 con 0x55d53b52a230 2024-04-25T02:07:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ecffff640 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b523ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ed4953640 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b5326c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ed5154640 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 0x55d53b524020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009ca0 tx=0x7f3ec8005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d67dde4c692c36c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3eaffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:07:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3eaffff640 1 -- 172.21.15.31:0/1026343560 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec805a060 con 0x55d53b52a230 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3eaffff640 1 -- 172.21.15.31:0/1026343560 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f3ec8004410 con 0x55d53b52a230 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.071+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1026343560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ec4012e50 con 0x55d53b52a230 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3eaffff640 1 -- 172.21.15.31:0/1026343560 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3ec8004610 con 0x55d53b52a230 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3ed99a2740 10 client.26774 fetch_fsmap learned FSMap version 40 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1026343560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55d53b5a0d10 con 0x55d53b52a230 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3eaffff640 1 -- 172.21.15.31:0/1026343560 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ec4012e50 con 0x55d53b52a230 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3eaffff640 1 -- 172.21.15.31:0/1026343560 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f3ec8030070 con 0x55d53b52a230 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3ed99a2740 10 client.26774 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3ed99a2740 10 client.26774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3ed99a2740 20 client.26774 populate_metadata read hostname 'smithi031' 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d53b5a1520 0x55d53b5a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3ed99a2740 4 client.26774 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1026343560 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d53b48b540 con 0x55d53b5a1520 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.072+0000 7f3ed4953640 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d53b5a1520 0x55d53b5a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.073+0000 7f3eaffff640 10 client.26774 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:41.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.073+0000 7f3ed4953640 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d53b5a1520 0x55d53b5a3940 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adf75655daafb67f server_cookie=30563f6d1b63d163 in_seq=0 out_seq=0 2024-04-25T02:07:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.101+0000 7f3eaffff640 1 -- 172.21.15.31:0/1026343560 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d53b48b540 con 0x55d53b5a1520 2024-04-25T02:07:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.101+0000 7f3eaffff640 10 client.26774 handle_command_reply: tid=0 2024-04-25T02:07:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.101+0000 7f3ed99a2740 1 client.26774 shutdown 2024-04-25T02:07:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1026343560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 msgr2=0x55d53b524020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 0x55d53b524020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009ca0 tx=0x7f3ec8005da0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1026343560 shutdown_connections 2024-04-25T02:07:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d53b5a1520 0x55d53b5a3940 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d53b52a230 0x55d53b524020 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d53b51fb60 0x55d53b523ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3eaffff640 0 client.26774 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3ed99a2740 1 --2- 172.21.15.31:0/1026343560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d53b51f240 0x55d53b5326c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1026343560 >> 172.21.15.31:0/1026343560 conn(0x55d53b51b560 msgr2=0x55d53b51a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1026343560 shutdown_connections 2024-04-25T02:07:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3ed99a2740 1 -- 172.21.15.31:0/1026343560 wait complete. 2024-04-25T02:07:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.102+0000 7f3ed99a2740 20 client.26774 trim_cache size 0 max 16384 2024-04-25T02:07:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:07:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:07:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:07:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/1732033411 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ec00a2da0 msgr2=0x7f3ec00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ec00a2da0 0x7f3ec00a5260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8005e10 tx=0x7f3ec8005ef0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/1732033411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed0101030 msgr2=0x7f3ed00fdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed0101030 0x7f3ed00fdb40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00e9f0 tx=0x7f3ebc00eeb0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/1732033411 shutdown_connections 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3ec00a2da0 0x7f3ec00a5260 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ed0106630 0x7f3ed00fe080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed0101030 0x7f3ed00fdb40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 --2- 172.21.15.31:0/1732033411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ed0100670 0x7f3ed00fd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/1732033411 >> 172.21.15.31:0/1732033411 conn(0x7f3ed00777c0 msgr2=0x7f3ed0069190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/1732033411 shutdown_connections 2024-04-25T02:07:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:41.103+0000 7f3ed6bb7640 1 -- 172.21.15.31:0/1732033411 wait complete. 2024-04-25T02:07:41.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 262963 1.86 MB/sec execute 588 sec latency 18.492 ms 2024-04-25T02:07:42.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:41 smithi134 ceph-mon[33098]: pgmap v412: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.1 MiB/s wr, 106 op/s 2024-04-25T02:07:42.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:41 smithi031 ceph-mon[29204]: pgmap v412: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.1 MiB/s wr, 106 op/s 2024-04-25T02:07:42.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 263325 1.86 MB/sec execute 589 sec latency 19.766 ms 2024-04-25T02:07:42.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:41 smithi040 ceph-mon[32418]: pgmap v412: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.1 MiB/s wr, 106 op/s 2024-04-25T02:07:43.333 INFO:tasks.workunit.client.0.smithi031.stdout: 1 263776 1.86 MB/sec execute 590 sec latency 18.493 ms 2024-04-25T02:07:44.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:43 smithi134 ceph-mon[33098]: pgmap v413: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 553 KiB/s wr, 46 op/s 2024-04-25T02:07:44.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:43 smithi031 ceph-mon[29204]: pgmap v413: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 553 KiB/s wr, 46 op/s 2024-04-25T02:07:44.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 264254 1.86 MB/sec execute 591 sec latency 22.257 ms 2024-04-25T02:07:44.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:43 smithi040 ceph-mon[32418]: pgmap v413: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 553 KiB/s wr, 46 op/s 2024-04-25T02:07:45.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 264849 1.86 MB/sec execute 592 sec latency 25.783 ms 2024-04-25T02:07:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:45 smithi134 ceph-mon[33098]: pgmap v414: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-04-25T02:07:46.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:45 smithi040 ceph-mon[32418]: pgmap v414: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-04-25T02:07:46.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:45 smithi031 ceph-mon[29204]: pgmap v414: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-04-25T02:07:46.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 265436 1.87 MB/sec execute 593 sec latency 17.993 ms 2024-04-25T02:07:47.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 265887 1.87 MB/sec execute 594 sec latency 19.093 ms 2024-04-25T02:07:48.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:47 smithi134 ceph-mon[33098]: pgmap v415: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 559 KiB/s wr, 71 op/s 2024-04-25T02:07:48.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:47 smithi031 ceph-mon[29204]: pgmap v415: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 559 KiB/s wr, 71 op/s 2024-04-25T02:07:48.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 266128 1.87 MB/sec execute 595 sec latency 663.597 ms 2024-04-25T02:07:48.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:47 smithi040 ceph-mon[32418]: pgmap v415: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 559 KiB/s wr, 71 op/s 2024-04-25T02:07:49.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:48 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:07:49.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:48 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:07:49.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 266128 1.87 MB/sec execute 596 sec latency 1663.642 ms 2024-04-25T02:07:49.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:48 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:07:50.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:49 smithi134 ceph-mon[33098]: pgmap v416: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 559 KiB/s wr, 71 op/s 2024-04-25T02:07:50.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:49 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:07:50.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:49 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:07:50.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:49 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:07:50.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:49 smithi031 ceph-mon[29204]: pgmap v416: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 559 KiB/s wr, 71 op/s 2024-04-25T02:07:50.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:49 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:07:50.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:49 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:07:50.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:49 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:07:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 266195 1.86 MB/sec execute 597 sec latency 2381.400 ms 2024-04-25T02:07:50.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:49 smithi040 ceph-mon[32418]: pgmap v416: 97 pgs: 97 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 559 KiB/s wr, 71 op/s 2024-04-25T02:07:50.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:49 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:07:50.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:49 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:07:50.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:49 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:07:51.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 266195 1.86 MB/sec execute 598 sec latency 1043.421 ms 2024-04-25T02:07:52.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:51 smithi134 ceph-mon[33098]: pgmap v417: 97 pgs: 97 active+clean; 148 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:07:52.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:51 smithi031 ceph-mon[29204]: pgmap v417: 97 pgs: 97 active+clean; 148 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:07:52.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 266195 1.86 MB/sec execute 599 sec latency 2043.516 ms 2024-04-25T02:07:52.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:51 smithi040 ceph-mon[32418]: pgmap v417: 97 pgs: 97 active+clean; 148 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-04-25T02:07:53.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 cleanup 600 sec 2024-04-25T02:07:54.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.201+0000 7f1eadde7640 1 -- 172.21.15.31:0/3848650188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8106670 msgr2=0x7f1ea8106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.201+0000 7f1eadde7640 1 --2- 172.21.15.31:0/3848650188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8106670 0x7f1ea8106a50 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f1e98009930 tx=0x7f1e9802a420 comp rx=0 tx=0).stop 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.201+0000 7f1eadde7640 1 -- 172.21.15.31:0/3848650188 shutdown_connections 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.201+0000 7f1eadde7640 1 --2- 172.21.15.31:0/3848650188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8101020 0x7f1ea810d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.201+0000 7f1eadde7640 1 --2- 172.21.15.31:0/3848650188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8100660 0x7f1ea8100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.201+0000 7f1eadde7640 1 --2- 172.21.15.31:0/3848650188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8106670 0x7f1ea8106a50 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.201+0000 7f1eadde7640 1 -- 172.21.15.31:0/3848650188 >> 172.21.15.31:0/3848650188 conn(0x7f1ea80769c0 msgr2=0x7f1ea8076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.202+0000 7f1eadde7640 1 -- 172.21.15.31:0/3848650188 shutdown_connections 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.202+0000 7f1eadde7640 1 -- 172.21.15.31:0/3848650188 wait complete. 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.202+0000 7f1eadde7640 1 Processor -- start 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.202+0000 7f1eadde7640 1 -- start start 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.202+0000 7f1eadde7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8100660 0x7f1ea8198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.202+0000 7f1eadde7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8101020 0x7f1ea8199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.202+0000 7f1eadde7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8106670 0x7f1ea81a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.202+0000 7f1eadde7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea8110980 con 0x7f1ea8101020 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.202+0000 7f1eadde7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea8110800 con 0x7f1ea8100660 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.202+0000 7f1eadde7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea8110b00 con 0x7f1ea8106670 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1e9ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8101020 0x7f1ea8199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1e9ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8101020 0x7f1ea8199200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35986/0 (socket says 172.21.15.31:35986) 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1e9ffff640 1 -- 172.21.15.31:0/2164785203 learned_addr learned my addr 172.21.15.31:0/2164785203 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1ea77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8100660 0x7f1ea8198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8106670 0x7f1ea81a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1e9ffff640 1 -- 172.21.15.31:0/2164785203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8106670 msgr2=0x7f1ea81a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1e9ffff640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8106670 0x7f1ea81a3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1e9ffff640 1 -- 172.21.15.31:0/2164785203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8100660 msgr2=0x7f1ea8198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1e9ffff640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8100660 0x7f1ea8198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1e9ffff640 1 -- 172.21.15.31:0/2164785203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea81a4280 con 0x7f1ea8101020 2024-04-25T02:07:54.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8106670 0x7f1ea81a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:07:54.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1e9ffff640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8101020 0x7f1ea8199200 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000d950 tx=0x7f1e9000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:54.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1ea57fa640 1 -- 172.21.15.31:0/2164785203 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90017070 con 0x7f1ea8101020 2024-04-25T02:07:54.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1eadde7640 1 -- 172.21.15.31:0/2164785203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea81a4450 con 0x7f1ea8101020 2024-04-25T02:07:54.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1ea57fa640 1 -- 172.21.15.31:0/2164785203 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e90009c30 con 0x7f1ea8101020 2024-04-25T02:07:54.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.203+0000 7f1ea57fa640 1 -- 172.21.15.31:0/2164785203 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9000f620 con 0x7f1ea8101020 2024-04-25T02:07:54.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.204+0000 7f1eadde7640 1 -- 172.21.15.31:0/2164785203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea81a49c0 con 0x7f1ea8101020 2024-04-25T02:07:54.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.206+0000 7f1ea57fa640 1 -- 172.21.15.31:0/2164785203 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1e900109a0 con 0x7f1ea8101020 2024-04-25T02:07:54.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.206+0000 7f1eb0bd1740 1 Processor -- start 2024-04-25T02:07:54.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.206+0000 7f1eb0bd1740 1 -- start start 2024-04-25T02:07:54.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.206+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52240 0x55ffc9b52620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.206+0000 7f1ea57fa640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e940a2da0 0x7f1e940a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.206+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b5ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ffc9ab6cb0 con 0x55ffc9b5d230 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ffc9a5f5b0 con 0x55ffc9b52b60 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ffc9ab2c70 con 0x55ffc9b52240 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea77fe640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e940a2da0 0x7f1e940a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35994/0 (socket says 172.21.15.31:35994) 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea7fff640 1 -- 172.21.15.31:0/701924881 learned_addr learned my addr 172.21.15.31:0/701924881 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/701924881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52240 0x55ffc9b52620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1e9ffff640 1 --2- 172.21.15.31:0/701924881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b5ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea57fa640 1 -- 172.21.15.31:0/2164785203 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f1e900cf060 con 0x7f1ea8101020 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea7fff640 1 -- 172.21.15.31:0/701924881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52240 msgr2=0x55ffc9b52620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/701924881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52240 0x55ffc9b52620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea7fff640 1 -- 172.21.15.31:0/701924881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 msgr2=0x55ffc9b5ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/701924881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b5ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea7fff640 1 -- 172.21.15.31:0/701924881 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffc97cb210 con 0x55ffc9b5d230 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea77fe640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e940a2da0 0x7f1e940a5260 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1e98031440 tx=0x7f1e98005a20 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/701924881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5f620 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0012a10 tx=0x7f1ea0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96000db163082f37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.207+0000 7f1e9dffb640 1 -- 172.21.15.31:0/701924881 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0004b20 con 0x55ffc9b5d230 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.208+0000 7f1e9dffb640 1 -- 172.21.15.31:0/701924881 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ea0004cc0 con 0x55ffc9b5d230 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.208+0000 7f1e9dffb640 1 -- 172.21.15.31:0/701924881 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea00054c0 con 0x55ffc9b5d230 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.208+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/701924881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 msgr2=0x55ffc9b5f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.208+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/701924881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5f620 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0012a10 tx=0x7f1ea0009750 comp rx=0 tx=0).stop 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.208+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/701924881 shutdown_connections 2024-04-25T02:07:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.208+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/701924881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5f620 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.208+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/701924881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b5ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.208+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/701924881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52240 0x55ffc9b52620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.208+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/701924881 >> 172.21.15.31:0/701924881 conn(0x55ffc9b4e560 msgr2=0x55ffc9b4e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.208+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/701924881 shutdown_connections 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.208+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/701924881 wait complete. 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1eb0bd1740 1 -- start start 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1eb0bd1740 1 Processor -- start 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52240 0x55ffc9b5a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b5a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ffc9b60020 con 0x55ffc9b5d230 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ffc9a72b50 con 0x55ffc9b52b60 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ffc9b601a0 con 0x55ffc9b52240 2024-04-25T02:07:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1eb0bd1740 10 client.0 _ll_register_callbacks cb 0x55ffc9aadca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1ea7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1ea7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35996/0 (socket says 172.21.15.31:35996) 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1ea7fff640 1 -- 172.21.15.31:0/2285496971 learned_addr learned my addr 172.21.15.31:0/2285496971 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.209+0000 7f1e9ffff640 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b5a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1ea77fe640 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52240 0x55ffc9b5a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1ea7fff640 1 -- 172.21.15.31:0/2285496971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52240 msgr2=0x55ffc9b5a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52240 0x55ffc9b5a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1ea7fff640 1 -- 172.21.15.31:0/2285496971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 msgr2=0x55ffc9b5a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b5a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1ea7fff640 1 -- 172.21.15.31:0/2285496971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffc9897620 con 0x55ffc9b5d230 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1e9ffff640 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b5a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5abd0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0005060 tx=0x7f1ea0004c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b466796495e0dc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1e86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1e86ffd640 1 -- 172.21.15.31:0/2285496971 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0004e70 con 0x55ffc9b5d230 2024-04-25T02:07:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/2285496971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ea0004cc0 con 0x55ffc9b5d230 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1e86ffd640 1 -- 172.21.15.31:0/2285496971 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f1ea0010a40 con 0x55ffc9b5d230 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1e86ffd640 1 -- 172.21.15.31:0/2285496971 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1ea0010c40 con 0x55ffc9b5d230 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1eb0bd1740 10 client.17211 fetch_fsmap learned FSMap version 40 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/2285496971 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ffc9b6c1b0 con 0x55ffc9b5d230 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.210+0000 7f1e86ffd640 1 -- 172.21.15.31:0/2285496971 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ea0004cc0 con 0x55ffc9b5d230 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.211+0000 7f1e86ffd640 1 -- 172.21.15.31:0/2285496971 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f1ea0030070 con 0x55ffc9b5d230 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.211+0000 7f1eb0bd1740 10 client.17211 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.211+0000 7f1eb0bd1740 10 client.17211 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.211+0000 7f1eb0bd1740 20 client.17211 populate_metadata read hostname 'smithi031' 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.211+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ffc9b6d620 0x55ffc9b6fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.211+0000 7f1eb0bd1740 4 client.17211 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.211+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/2285496971 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffc9ac41f0 con 0x55ffc9b6d620 2024-04-25T02:07:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.211+0000 7f1ea77fe640 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ffc9b6d620 0x55ffc9b6fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.211+0000 7f1e86ffd640 10 client.17211 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.211+0000 7f1ea77fe640 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ffc9b6d620 0x55ffc9b6fa10 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30f0edc2ef5038de server_cookie=68f26fdbe5539d7d in_seq=0 out_seq=0 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.212+0000 7f1e86ffd640 1 -- 172.21.15.31:0/2285496971 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ffc9ac41f0 con 0x55ffc9b6d620 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.212+0000 7f1e86ffd640 10 client.17211 handle_command_reply: tid=0 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.212+0000 7f1eb0bd1740 1 client.17211 shutdown 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.212+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/2285496971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 msgr2=0x55ffc9b5abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.212+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5abd0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0005060 tx=0x7f1ea0004c50 comp rx=0 tx=0).stop 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.212+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/2285496971 shutdown_connections 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.212+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ffc9b6d620 0x55ffc9b6fa10 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.212+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b5abd0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.212+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b5a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.212+0000 7f1e86ffd640 0 client.17211 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.212+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/2285496971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52240 0x55ffc9b5a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.213+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/2285496971 >> 172.21.15.31:0/2285496971 conn(0x55ffc9b4e560 msgr2=0x55ffc9b54360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.213+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/2285496971 shutdown_connections 2024-04-25T02:07:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.213+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/2285496971 wait complete. 2024-04-25T02:07:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.213+0000 7f1eb0bd1740 20 client.17211 trim_cache size 0 max 16384 2024-04-25T02:07:54.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1eb0bd1740 1 Processor -- start 2024-04-25T02:07:54.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1eb0bd1740 1 -- start start 2024-04-25T02:07:54.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52240 0x55ffc9b68ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52b60 0x55ffc9b75200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b75740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81ad2f0 con 0x55ffc9b5d230 2024-04-25T02:07:54.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81aefb0 con 0x55ffc9b52240 2024-04-25T02:07:54.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81ad9c0 con 0x55ffc9b52b60 2024-04-25T02:07:54.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1ea7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b75740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1ea7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b75740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36012/0 (socket says 172.21.15.31:36012) 2024-04-25T02:07:54.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1ea7fff640 1 -- 172.21.15.31:0/1796518179 learned_addr learned my addr 172.21.15.31:0/1796518179 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:54.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1e9ffff640 1 --2- 172.21.15.31:0/1796518179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52b60 0x55ffc9b75200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1ea77fe640 1 --2- 172.21.15.31:0/1796518179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52240 0x55ffc9b68ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1ea7fff640 1 -- 172.21.15.31:0/1796518179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52b60 msgr2=0x55ffc9b75200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:54.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/1796518179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52b60 0x55ffc9b75200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1ea7fff640 1 -- 172.21.15.31:0/1796518179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52240 msgr2=0x55ffc9b68ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:54.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/1796518179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52240 0x55ffc9b68ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1ea7fff640 1 -- 172.21.15.31:0/1796518179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffc9b6ffd0 con 0x55ffc9b5d230 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.228+0000 7f1ea77fe640 1 --2- 172.21.15.31:0/1796518179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52240 0x55ffc9b68ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/1796518179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b75740 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0005d70 tx=0x7f1ea0009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95182e98790c366 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1e9d7fa640 1 -- 172.21.15.31:0/1796518179 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0036020 con 0x55ffc9b5d230 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1e9d7fa640 1 -- 172.21.15.31:0/1796518179 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ea0016070 con 0x55ffc9b5d230 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1e9d7fa640 1 -- 172.21.15.31:0/1796518179 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea001f810 con 0x55ffc9b5d230 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1796518179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 msgr2=0x55ffc9b75740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/1796518179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b75740 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0005d70 tx=0x7f1ea0009e50 comp rx=0 tx=0).stop 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1796518179 shutdown_connections 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/1796518179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b5d230 0x55ffc9b75740 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/1796518179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b52b60 0x55ffc9b75200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/1796518179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52240 0x55ffc9b68ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1796518179 >> 172.21.15.31:0/1796518179 conn(0x55ffc9b4e560 msgr2=0x55ffc9b4d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1796518179 shutdown_connections 2024-04-25T02:07:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.229+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1796518179 wait complete. 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1eb0bd1740 1 -- start start 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1eb0bd1740 1 Processor -- start 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b52240 0x55ffc9b65790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b62b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1eb0bd1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b5d230 0x55ffc9b63060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81ae8e0 con 0x55ffc9b52240 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81ae150 con 0x55ffc9b52b60 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1eb0bd1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81ab2f0 con 0x55ffc9b5d230 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1eb0bd1740 10 client.0 _ll_register_callbacks cb 0x7f1e9000df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1ea77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b52240 0x55ffc9b65790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1ea77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b52240 0x55ffc9b65790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36026/0 (socket says 172.21.15.31:36026) 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1ea77fe640 1 -- 172.21.15.31:0/1539582685 learned_addr learned my addr 172.21.15.31:0/1539582685 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.230+0000 7f1e9ffff640 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b62b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b5d230 0x55ffc9b63060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1ea77fe640 1 -- 172.21.15.31:0/1539582685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b5d230 msgr2=0x55ffc9b63060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1ea77fe640 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b5d230 0x55ffc9b63060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1ea77fe640 1 -- 172.21.15.31:0/1539582685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 msgr2=0x55ffc9b62b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1ea77fe640 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b62b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1ea77fe640 1 -- 172.21.15.31:0/1539582685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffc9b63890 con 0x55ffc9b52240 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1ea7fff640 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b5d230 0x55ffc9b63060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1ea77fe640 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b52240 0x55ffc9b65790 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1e98006010 tx=0x7f1e9802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3401fc138c9190c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1e86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1e86ffd640 1 -- 172.21.15.31:0/1539582685 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98035040 con 0x55ffc9b52240 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1539582685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ea0016070 con 0x55ffc9b52240 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1e86ffd640 1 -- 172.21.15.31:0/1539582685 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8484+0+0 (secure 0 0 0) 0x7f1e98051d50 con 0x55ffc9b52240 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1e86ffd640 1 -- 172.21.15.31:0/1539582685 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1e98004750 con 0x55ffc9b52240 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1eb0bd1740 10 client.17217 fetch_fsmap learned FSMap version 40 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1539582685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ffc9bd3a50 con 0x55ffc9b52240 2024-04-25T02:07:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.231+0000 7f1e86ffd640 1 -- 172.21.15.31:0/1539582685 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ea0016070 con 0x55ffc9b52240 2024-04-25T02:07:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.232+0000 7f1e86ffd640 1 -- 172.21.15.31:0/1539582685 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 40) ==== 8386+0+0 (secure 0 0 0) 0x7f1e980516b0 con 0x55ffc9b52240 2024-04-25T02:07:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.232+0000 7f1eb0bd1740 10 client.17217 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:07:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.232+0000 7f1eb0bd1740 10 client.17217 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.232+0000 7f1eb0bd1740 20 client.17217 populate_metadata read hostname 'smithi031' 2024-04-25T02:07:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.232+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ffc9bd4240 0x55ffc9bd66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.232+0000 7f1eb0bd1740 4 client.17217 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:07:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.232+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1539582685 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ffc9abe540 con 0x55ffc9bd4240 2024-04-25T02:07:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.232+0000 7f1e9ffff640 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ffc9bd4240 0x55ffc9bd66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.233+0000 7f1e86ffd640 10 client.17217 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:54.233+0000 7f1e9ffff640 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ffc9bd4240 0x55ffc9bd66a0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70ac3a4a339e7335 server_cookie=1494eb2f27655f59 in_seq=0 out_seq=0 2024-04-25T02:07:54.247 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:53 smithi031 ceph-mon[29204]: pgmap v418: 97 pgs: 97 active+clean; 148 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-04-25T02:07:54.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 cleanup 601 sec 2024-04-25T02:07:54.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:53 smithi040 ceph-mon[32418]: pgmap v418: 97 pgs: 97 active+clean; 148 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-04-25T02:07:54.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:53 smithi134 ceph-mon[33098]: pgmap v418: 97 pgs: 97 active+clean; 148 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-04-25T02:07:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.055+0000 7f1e86ffd640 1 -- 172.21.15.31:0/1539582685 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ffc9abe540 con 0x55ffc9bd4240 2024-04-25T02:07:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.055+0000 7f1e86ffd640 10 client.17217 handle_command_reply: tid=0 2024-04-25T02:07:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 1 client.17217 shutdown 2024-04-25T02:07:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1539582685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b52240 msgr2=0x55ffc9b65790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b52240 0x55ffc9b65790 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1e98006010 tx=0x7f1e9802a420 comp rx=0 tx=0).stop 2024-04-25T02:07:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1539582685 shutdown_connections 2024-04-25T02:07:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ffc9bd4240 0x55ffc9bd66a0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ffc9b5d230 0x55ffc9b63060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ffc9b52b60 0x55ffc9b62b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 1 --2- 172.21.15.31:0/1539582685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ffc9b52240 0x55ffc9b65790 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1539582685 >> 172.21.15.31:0/1539582685 conn(0x55ffc9b4e560 msgr2=0x55ffc9b53a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1e86ffd640 0 client.17217 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:07:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1539582685 shutdown_connections 2024-04-25T02:07:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 1 -- 172.21.15.31:0/1539582685 wait complete. 2024-04-25T02:07:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.056+0000 7f1eb0bd1740 20 client.17217 trim_cache size 0 max 16384 2024-04-25T02:07:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:07:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:07:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:07:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:07:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 -- 172.21.15.31:0/2164785203 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e940a2da0 msgr2=0x7f1e940a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e940a2da0 0x7f1e940a5260 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1e98031440 tx=0x7f1e98005a20 comp rx=0 tx=0).stop 2024-04-25T02:07:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 -- 172.21.15.31:0/2164785203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8101020 msgr2=0x7f1ea8199200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8101020 0x7f1ea8199200 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000d950 tx=0x7f1e9000de10 comp rx=0 tx=0).stop 2024-04-25T02:07:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 -- 172.21.15.31:0/2164785203 shutdown_connections 2024-04-25T02:07:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f1e940a2da0 0x7f1e940a5260 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ea8106670 0x7f1ea81a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ea8101020 0x7f1ea8199200 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 --2- 172.21.15.31:0/2164785203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ea8100660 0x7f1ea8198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 -- 172.21.15.31:0/2164785203 >> 172.21.15.31:0/2164785203 conn(0x7f1ea80769c0 msgr2=0x7f1ea80fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 -- 172.21.15.31:0/2164785203 shutdown_connections 2024-04-25T02:07:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:07:55.057+0000 7f1eadde7640 1 -- 172.21.15.31:0/2164785203 wait complete. 2024-04-25T02:07:55.334 INFO:tasks.workunit.client.0.smithi031.stdout: 1 cleanup 602 sec 2024-04-25T02:07:55.987 INFO:tasks.workunit.client.0.smithi031.stdout: 0 cleanup 603 sec 2024-04-25T02:07:55.987 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T02:07:55.987 INFO:tasks.workunit.client.0.smithi031.stdout: Operation Count AvgLat MaxLat 2024-04-25T02:07:55.987 INFO:tasks.workunit.client.0.smithi031.stdout: ---------------------------------------- 2024-04-25T02:07:55.987 INFO:tasks.workunit.client.0.smithi031.stdout: NTCreateX 35639 3.894 60.421 2024-04-25T02:07:55.987 INFO:tasks.workunit.client.0.smithi031.stdout: Close 26181 0.047 0.614 2024-04-25T02:07:55.987 INFO:tasks.workunit.client.0.smithi031.stdout: Rename 1511 105.937 14955.472 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: Unlink 7202 4.315 133.160 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: Qpathinfo 32318 3.418 2684.916 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: Qfileinfo 5632 0.074 0.484 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: Qfsinfo 5935 0.314 5.245 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: Sfileinfo 2900 1.331 13.163 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: Find 12490 11.498 4842.333 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: WriteX 17597 0.173 4.477 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: ReadX 55960 0.004 0.139 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: LockX 116 0.740 5.922 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: UnlockX 116 0.579 1.778 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: Flush 2496 2.568 8.516 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T02:07:55.988 INFO:tasks.workunit.client.0.smithi031.stdout:Throughput 1.85726 MB/sec 1 clients 1 procs max_latency=14955.483 ms 2024-04-25T02:07:55.989 INFO:tasks.workunit.client.0.smithi031.stderr:+ dbench 10 2024-04-25T02:07:55.989 INFO:tasks.workunit.client.0.smithi031.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-25T02:07:55.989 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T02:07:55.989 INFO:tasks.workunit.client.0.smithi031.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-25T02:07:55.989 INFO:tasks.workunit.client.0.smithi031.stdout:8 of 10 processes prepared for launch 0 sec 2024-04-25T02:07:56.088 INFO:tasks.workunit.client.0.smithi031.stdout:10 of 10 processes prepared for launch 0 sec 2024-04-25T02:07:56.088 INFO:tasks.workunit.client.0.smithi031.stdout:releasing clients 2024-04-25T02:07:56.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:55 smithi031 ceph-mon[29204]: pgmap v419: 97 pgs: 97 active+clean; 149 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:07:56.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:55 smithi040 ceph-mon[32418]: pgmap v419: 97 pgs: 97 active+clean; 149 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:07:56.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:55 smithi134 ceph-mon[33098]: pgmap v419: 97 pgs: 97 active+clean; 149 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-04-25T02:07:57.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 223 86.50 MB/sec warmup 1 sec latency 119.057 ms 2024-04-25T02:07:58.088 INFO:tasks.workunit.client.0.smithi031.stdout: 10 381 67.99 MB/sec warmup 2 sec latency 122.990 ms 2024-04-25T02:07:58.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:57 smithi031 ceph-mon[29204]: pgmap v420: 97 pgs: 97 active+clean; 149 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 573 KiB/s wr, 52 op/s 2024-04-25T02:07:58.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:57 smithi040 ceph-mon[32418]: pgmap v420: 97 pgs: 97 active+clean; 149 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 573 KiB/s wr, 52 op/s 2024-04-25T02:07:58.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:57 smithi134 ceph-mon[33098]: pgmap v420: 97 pgs: 97 active+clean; 149 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 573 KiB/s wr, 52 op/s 2024-04-25T02:07:59.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 572 65.75 MB/sec warmup 3 sec latency 117.772 ms 2024-04-25T02:08:00.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 689 52.96 MB/sec warmup 4 sec latency 128.168 ms 2024-04-25T02:08:00.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:07:59 smithi031 ceph-mon[29204]: pgmap v421: 97 pgs: 97 active+clean; 149 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 573 KiB/s wr, 52 op/s 2024-04-25T02:08:00.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:07:59 smithi040 ceph-mon[32418]: pgmap v421: 97 pgs: 97 active+clean; 149 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 573 KiB/s wr, 52 op/s 2024-04-25T02:08:00.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:07:59 smithi134 ceph-mon[33098]: pgmap v421: 97 pgs: 97 active+clean; 149 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 573 KiB/s wr, 52 op/s 2024-04-25T02:08:01.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 782 42.84 MB/sec warmup 5 sec latency 418.077 ms 2024-04-25T02:08:02.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 877 36.07 MB/sec warmup 6 sec latency 785.400 ms 2024-04-25T02:08:02.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:01 smithi031 ceph-mon[29204]: pgmap v422: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 17 MiB/s wr, 143 op/s 2024-04-25T02:08:02.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:01 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:08:02.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:01 smithi040 ceph-mon[32418]: pgmap v422: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 17 MiB/s wr, 143 op/s 2024-04-25T02:08:02.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:01 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:08:02.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:01 smithi134 ceph-mon[33098]: pgmap v422: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 17 MiB/s wr, 143 op/s 2024-04-25T02:08:02.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:01 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:08:03.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 972 32.25 MB/sec warmup 7 sec latency 1785.428 ms 2024-04-25T02:08:03.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:02 smithi031 ceph-mon[29204]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T02:08:03.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:02 smithi040 ceph-mon[32418]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T02:08:03.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:02 smithi134 ceph-mon[33098]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T02:08:04.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 1065 28.33 MB/sec warmup 8 sec latency 2785.454 ms 2024-04-25T02:08:04.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:03 smithi031 ceph-mon[29204]: pgmap v424: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 20 MiB/s wr, 128 op/s 2024-04-25T02:08:04.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:03 smithi040 ceph-mon[32418]: pgmap v424: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 20 MiB/s wr, 128 op/s 2024-04-25T02:08:04.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:03 smithi134 ceph-mon[33098]: pgmap v424: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 20 MiB/s wr, 128 op/s 2024-04-25T02:08:05.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 1150 25.27 MB/sec warmup 9 sec latency 3785.481 ms 2024-04-25T02:08:06.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 1288 25.08 MB/sec warmup 10 sec latency 4785.510 ms 2024-04-25T02:08:06.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:05 smithi031 ceph-mon[29204]: pgmap v425: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 22 MiB/s wr, 214 op/s 2024-04-25T02:08:06.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:05 smithi040 ceph-mon[32418]: pgmap v425: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 22 MiB/s wr, 214 op/s 2024-04-25T02:08:06.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:05 smithi134 ceph-mon[33098]: pgmap v425: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 22 MiB/s wr, 214 op/s 2024-04-25T02:08:07.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 1397 22.87 MB/sec warmup 11 sec latency 5785.591 ms 2024-04-25T02:08:08.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 1509 21.04 MB/sec warmup 12 sec latency 6785.633 ms 2024-04-25T02:08:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.161+0000 7fdfda370640 1 -- 172.21.15.31:0/3893460517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfd4101730 msgr2=0x7fdfd4101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.161+0000 7fdfda370640 1 --2- 172.21.15.31:0/3893460517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfd4101730 0x7fdfd4101bb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009930 tx=0x7fdfc802a420 comp rx=0 tx=0).stop 2024-04-25T02:08:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.161+0000 7fdfda370640 1 -- 172.21.15.31:0/3893460517 shutdown_connections 2024-04-25T02:08:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.161+0000 7fdfda370640 1 --2- 172.21.15.31:0/3893460517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd41020f0 0x7fdfd410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.161+0000 7fdfda370640 1 --2- 172.21.15.31:0/3893460517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfd4101730 0x7fdfd4101bb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.161+0000 7fdfda370640 1 --2- 172.21.15.31:0/3893460517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd4107740 0x7fdfd4107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.161+0000 7fdfda370640 1 -- 172.21.15.31:0/3893460517 >> 172.21.15.31:0/3893460517 conn(0x7fdfd40fd200 msgr2=0x7fdfd40ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.161+0000 7fdfda370640 1 -- 172.21.15.31:0/3893460517 shutdown_connections 2024-04-25T02:08:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.162+0000 7fdfda370640 1 -- 172.21.15.31:0/3893460517 wait complete. 2024-04-25T02:08:08.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.162+0000 7fdfda370640 1 Processor -- start 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.162+0000 7fdfda370640 1 -- start start 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfda370640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd4101730 0x7fdfd4073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfda370640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd41020f0 0x7fdfd4073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfda370640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfd4107740 0x7fdfd40744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfda370640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd4111a00 con 0x7fdfd4101730 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfda370640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd4111880 con 0x7fdfd41020f0 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfda370640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd4111b80 con 0x7fdfd4107740 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfd3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd4101730 0x7fdfd4073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfd3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd4101730 0x7fdfd4073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53114/0 (socket says 172.21.15.31:53114) 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfd3fff640 1 -- 172.21.15.31:0/2184300975 learned_addr learned my addr 172.21.15.31:0/2184300975 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfd890d640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfd4107740 0x7fdfd40744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd41020f0 0x7fdfd4073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfd3fff640 1 -- 172.21.15.31:0/2184300975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfd4107740 msgr2=0x7fdfd40744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfd3fff640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfd4107740 0x7fdfd40744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfd3fff640 1 -- 172.21.15.31:0/2184300975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd41020f0 msgr2=0x7fdfd4073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfd3fff640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd41020f0 0x7fdfd4073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfd3fff640 1 -- 172.21.15.31:0/2184300975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd41aa880 con 0x7fdfd4101730 2024-04-25T02:08:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.163+0000 7fdfd890d640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfd4107740 0x7fdfd40744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.164+0000 7fdfd3fff640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd4101730 0x7fdfd4073a40 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400e9f0 tx=0x7fdfc400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.164+0000 7fdfd17fa640 1 -- 172.21.15.31:0/2184300975 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc400cd80 con 0x7fdfd4101730 2024-04-25T02:08:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.164+0000 7fdfd17fa640 1 -- 172.21.15.31:0/2184300975 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc40108e0 con 0x7fdfd4101730 2024-04-25T02:08:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.164+0000 7fdfda370640 1 -- 172.21.15.31:0/2184300975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd41aaab0 con 0x7fdfd4101730 2024-04-25T02:08:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.164+0000 7fdfd17fa640 1 -- 172.21.15.31:0/2184300975 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4018920 con 0x7fdfd4101730 2024-04-25T02:08:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.164+0000 7fdfda370640 1 -- 172.21.15.31:0/2184300975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd41ab020 con 0x7fdfd4101730 2024-04-25T02:08:08.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfd17fa640 1 -- 172.21.15.31:0/2184300975 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fdfc4010a90 con 0x7fdfd4101730 2024-04-25T02:08:08.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfdcd22740 1 Processor -- start 2024-04-25T02:08:08.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfdcd22740 1 -- start start 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a35709b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 0x55a3570a5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 0x55a3570a8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a356fffcb0 con 0x55a35709b240 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a356fa85b0 con 0x55a35709bb60 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a356ffbc70 con 0x55a3570a6230 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfd890d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a35709b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfd890d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a35709b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53124/0 (socket says 172.21.15.31:53124) 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfd890d640 1 -- 172.21.15.31:0/139977489 learned_addr learned my addr 172.21.15.31:0/139977489 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/139977489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 0x55a3570a8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.167+0000 7fdfd3fff640 1 --2- 172.21.15.31:0/139977489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 0x55a3570a5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfd17fa640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfbc0a2e20 0x7fdfbc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfd37fe640 1 -- 172.21.15.31:0/139977489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 msgr2=0x55a3570a5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/139977489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 0x55a3570a5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfd37fe640 1 -- 172.21.15.31:0/139977489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 msgr2=0x55a35709b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/139977489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a35709b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfd37fe640 1 -- 172.21.15.31:0/139977489 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a356d14210 con 0x55a3570a6230 2024-04-25T02:08:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfd890d640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfbc0a2e20 0x7fdfbc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfd17fa640 1 -- 172.21.15.31:0/2184300975 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8528+0+0 (secure 0 0 0) 0x7fdfc40cf630 con 0x7fdfd4101730 2024-04-25T02:08:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/139977489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 0x55a3570a8620 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8031440 tx=0x7fdfc8005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d64d2ba16fab937 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfba7fc640 1 -- 172.21.15.31:0/139977489 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8040070 con 0x55a3570a6230 2024-04-25T02:08:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfba7fc640 1 -- 172.21.15.31:0/139977489 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc8004030 con 0x55a3570a6230 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.168+0000 7fdfba7fc640 1 -- 172.21.15.31:0/139977489 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8035040 con 0x55a3570a6230 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.169+0000 7fdfdcd22740 1 -- 172.21.15.31:0/139977489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 msgr2=0x55a3570a8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.169+0000 7fdfd890d640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfbc0a2e20 0x7fdfbc0a52e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009fd0 tx=0x7fdfcc004a40 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.169+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/139977489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 0x55a3570a8620 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8031440 tx=0x7fdfc8005be0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.169+0000 7fdfdcd22740 1 -- 172.21.15.31:0/139977489 shutdown_connections 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.169+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/139977489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 0x55a3570a8620 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.169+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/139977489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 0x55a3570a5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.169+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/139977489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a35709b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.169+0000 7fdfdcd22740 1 -- 172.21.15.31:0/139977489 >> 172.21.15.31:0/139977489 conn(0x55a357097560 msgr2=0x55a357097940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.169+0000 7fdfdcd22740 1 -- 172.21.15.31:0/139977489 shutdown_connections 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.169+0000 7fdfdcd22740 1 -- 172.21.15.31:0/139977489 wait complete. 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.169+0000 7fdfdcd22740 1 -- start start 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfdcd22740 1 Processor -- start 2024-04-25T02:08:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a3570a3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 0x55a3570a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 0x55a3570a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a3570a8fc0 con 0x55a35709b240 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a356fbbb50 con 0x55a35709bb60 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a3570a9140 con 0x55a3570a6230 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfd37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 0x55a3570a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfd890d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 0x55a3570a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfd3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a3570a3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfd37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 0x55a3570a36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:50804/0 (socket says 172.21.15.31:50804) 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfd37fe640 1 -- 172.21.15.31:0/1976310733 learned_addr learned my addr 172.21.15.31:0/1976310733 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.170+0000 7fdfdcd22740 10 client.0 _ll_register_callbacks cb 0x55a356ff6ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfd37fe640 1 -- 172.21.15.31:0/1976310733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 msgr2=0x55a3570a3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 0x55a3570a3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfd37fe640 1 -- 172.21.15.31:0/1976310733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 msgr2=0x55a3570a3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a3570a3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfd37fe640 1 -- 172.21.15.31:0/1976310733 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a356de0620 con 0x55a35709bb60 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfd890d640 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 0x55a3570a3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 0x55a3570a36b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8005ec0 tx=0x7fdfc8004490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d09812501d70c32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1976310733 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8040030 con 0x55a35709bb60 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1976310733 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8528+0+0 (secure 0 0 0) 0x7fdfc80314a0 con 0x55a35709bb60 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1976310733 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc8004030 con 0x55a35709bb60 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1976310733 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fdfc8056050 con 0x55a35709bb60 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfdcd22740 10 client.26801 fetch_fsmap learned FSMap version 41 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.171+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1976310733 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55a3570b5160 con 0x55a35709bb60 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.172+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1976310733 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc8004030 con 0x55a35709bb60 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.172+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1976310733 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 41) ==== 8390+0+0 (secure 0 0 0) 0x7fdfc80035f0 con 0x55a35709bb60 2024-04-25T02:08:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.172+0000 7fdfdcd22740 10 client.26801 fetch_fsmap finished waiting for FSMap version 41 2024-04-25T02:08:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.172+0000 7fdfdcd22740 10 client.26801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.172+0000 7fdfdcd22740 20 client.26801 populate_metadata read hostname 'smithi031' 2024-04-25T02:08:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.172+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a3570b64b0 0x55a3570b88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.172+0000 7fdfdcd22740 4 client.26801 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:08:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.172+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1976310733 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a35700d1f0 con 0x55a3570b64b0 2024-04-25T02:08:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.172+0000 7fdfd3fff640 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a3570b64b0 0x55a3570b88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.172+0000 7fdfb37fe640 10 client.26801 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.173+0000 7fdfd3fff640 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a3570b64b0 0x55a3570b88a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bab08099c8d6c734 server_cookie=fec0fe52e5f1d471 in_seq=0 out_seq=0 2024-04-25T02:08:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1976310733 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a35700d1f0 con 0x55a3570b64b0 2024-04-25T02:08:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfb37fe640 10 client.26801 handle_command_reply: tid=0 2024-04-25T02:08:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 1 client.26801 shutdown 2024-04-25T02:08:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1976310733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 msgr2=0x55a3570a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 0x55a3570a36b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8005ec0 tx=0x7fdfc8004490 comp rx=0 tx=0).stop 2024-04-25T02:08:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1976310733 shutdown_connections 2024-04-25T02:08:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a3570b64b0 0x55a3570b88a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a3570a6230 0x55a3570a3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a35709bb60 0x55a3570a36b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfb37fe640 0 client.26801 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1976310733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a3570a3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1976310733 >> 172.21.15.31:0/1976310733 conn(0x55a357097560 msgr2=0x55a35709d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1976310733 shutdown_connections 2024-04-25T02:08:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1976310733 wait complete. 2024-04-25T02:08:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.174+0000 7fdfdcd22740 20 client.26801 trim_cache size 0 max 16384 2024-04-25T02:08:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.196+0000 7fdfdcd22740 1 Processor -- start 2024-04-25T02:08:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.196+0000 7fdfdcd22740 1 -- start start 2024-04-25T02:08:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.196+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a35709b240 0x55a3570a58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.196+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709bb60 0x55a3570b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3570a6230 0x55a3570a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41b3800 con 0x55a35709bb60 2024-04-25T02:08:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41b54c0 con 0x55a3570a6230 2024-04-25T02:08:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41b3ed0 con 0x55a35709b240 2024-04-25T02:08:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709bb60 0x55a3570b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709bb60 0x55a3570b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53150/0 (socket says 172.21.15.31:53150) 2024-04-25T02:08:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd37fe640 1 -- 172.21.15.31:0/1882763407 learned_addr learned my addr 172.21.15.31:0/1882763407 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd890d640 1 --2- 172.21.15.31:0/1882763407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3570a6230 0x55a3570a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd3fff640 1 --2- 172.21.15.31:0/1882763407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a35709b240 0x55a3570a58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd37fe640 1 -- 172.21.15.31:0/1882763407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a35709b240 msgr2=0x55a3570a58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/1882763407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a35709b240 0x55a3570a58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd37fe640 1 -- 172.21.15.31:0/1882763407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3570a6230 msgr2=0x55a3570a22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/1882763407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3570a6230 0x55a3570a22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd37fe640 1 -- 172.21.15.31:0/1882763407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a35709b990 con 0x55a35709bb60 2024-04-25T02:08:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd890d640 1 --2- 172.21.15.31:0/1882763407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3570a6230 0x55a3570a22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.197+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/1882763407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709bb60 0x55a3570b1e80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fdfc802a940 tx=0x7fdfc8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bb650450dc7437e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.198+0000 7fdfb9ffb640 1 -- 172.21.15.31:0/1882763407 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc805f070 con 0x55a35709bb60 2024-04-25T02:08:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.198+0000 7fdfb9ffb640 1 -- 172.21.15.31:0/1882763407 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc804d070 con 0x55a35709bb60 2024-04-25T02:08:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.198+0000 7fdfb9ffb640 1 -- 172.21.15.31:0/1882763407 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8040070 con 0x55a35709bb60 2024-04-25T02:08:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.199+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1882763407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709bb60 msgr2=0x55a3570b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.199+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1882763407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709bb60 0x55a3570b1e80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fdfc802a940 tx=0x7fdfc8005fd0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.199+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1882763407 shutdown_connections 2024-04-25T02:08:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.199+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1882763407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3570a6230 0x55a3570a22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.199+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1882763407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709bb60 0x55a3570b1e80 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.199+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1882763407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a35709b240 0x55a3570a58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.199+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1882763407 >> 172.21.15.31:0/1882763407 conn(0x55a357097560 msgr2=0x55a35709c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.199+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1882763407 shutdown_connections 2024-04-25T02:08:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.199+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1882763407 wait complete. 2024-04-25T02:08:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.199+0000 7fdfdcd22740 1 -- start start 2024-04-25T02:08:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.199+0000 7fdfdcd22740 1 Processor -- start 2024-04-25T02:08:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a357114c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a35709bb60 0x55a3571151c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfdcd22740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3570a6230 0x55a3570a9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41b4df0 con 0x55a35709b240 2024-04-25T02:08:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41b4660 con 0x55a3570a6230 2024-04-25T02:08:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfdcd22740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41b17e0 con 0x55a35709bb60 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfdcd22740 10 client.0 _ll_register_callbacks cb 0x7fdfcc009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfd3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a357114c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfd3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a357114c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53162/0 (socket says 172.21.15.31:53162) 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfd890d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3570a6230 0x55a3570a9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfd3fff640 1 -- 172.21.15.31:0/1066091449 learned_addr learned my addr 172.21.15.31:0/1066091449 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a35709bb60 0x55a3571151c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfd3fff640 1 -- 172.21.15.31:0/1066091449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a35709bb60 msgr2=0x55a3571151c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfd3fff640 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a35709bb60 0x55a3571151c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfd3fff640 1 -- 172.21.15.31:0/1066091449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3570a6230 msgr2=0x55a3570a9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfd3fff640 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3570a6230 0x55a3570a9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfd3fff640 1 -- 172.21.15.31:0/1066091449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3570a9ec0 con 0x55a35709b240 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfd3fff640 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a357114c80 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400c6a0 tx=0x7fdfc40d5530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2195c3615848d6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.200+0000 7fdfb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:08:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1066091449 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4010de0 con 0x55a35709b240 2024-04-25T02:08:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1066091449 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8528+0+0 (secure 0 0 0) 0x7fdfc40cd000 con 0x55a35709b240 2024-04-25T02:08:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1066091449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc804d070 con 0x55a35709b240 2024-04-25T02:08:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1066091449 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fdfc40b9050 con 0x55a35709b240 2024-04-25T02:08:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfdcd22740 10 client.17235 fetch_fsmap learned FSMap version 41 2024-04-25T02:08:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1066091449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55a35711c4f0 con 0x55a35709b240 2024-04-25T02:08:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1066091449 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc804d070 con 0x55a35709b240 2024-04-25T02:08:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1066091449 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 41) ==== 8390+0+0 (secure 0 0 0) 0x7fdfc40d1b30 con 0x55a35709b240 2024-04-25T02:08:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfdcd22740 10 client.17235 fetch_fsmap finished waiting for FSMap version 41 2024-04-25T02:08:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfdcd22740 10 client.17235 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfdcd22740 20 client.17235 populate_metadata read hostname 'smithi031' 2024-04-25T02:08:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a3570b64b0 0x55a35711ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfdcd22740 4 client.17235 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:08:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.201+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1066091449 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a357007540 con 0x55a3570b64b0 2024-04-25T02:08:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.202+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a3570b64b0 0x55a35711ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.202+0000 7fdfd37fe640 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a3570b64b0 0x55a35711ecf0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b45056233b65a31 server_cookie=5e3cddb4630cdaf0 in_seq=0 out_seq=0 2024-04-25T02:08:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:08.202+0000 7fdfb37fe640 10 client.17235 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:08.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:07 smithi031 ceph-mon[29204]: pgmap v426: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 22 MiB/s wr, 214 op/s 2024-04-25T02:08:08.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:07 smithi040 ceph-mon[32418]: pgmap v426: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 22 MiB/s wr, 214 op/s 2024-04-25T02:08:08.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:07 smithi134 ceph-mon[33098]: pgmap v426: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 22 MiB/s wr, 214 op/s 2024-04-25T02:08:09.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 1624 19.50 MB/sec warmup 13 sec latency 7785.661 ms 2024-04-25T02:08:10.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 1760 18.21 MB/sec warmup 14 sec latency 8785.692 ms 2024-04-25T02:08:10.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:09 smithi040 ceph-mon[32418]: pgmap v427: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 22 MiB/s wr, 214 op/s 2024-04-25T02:08:10.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:09 smithi040 ceph-mon[32418]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T02:08:10.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:09 smithi134 ceph-mon[33098]: pgmap v427: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 22 MiB/s wr, 214 op/s 2024-04-25T02:08:10.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:09 smithi134 ceph-mon[33098]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T02:08:10.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:09 smithi031 ceph-mon[29204]: pgmap v427: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 22 MiB/s wr, 214 op/s 2024-04-25T02:08:10.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:09 smithi031 ceph-mon[29204]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T02:08:11.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 1875 17.09 MB/sec warmup 15 sec latency 9785.724 ms 2024-04-25T02:08:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfb37fe640 1 -- 172.21.15.31:0/1066091449 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a357007540 con 0x55a3570b64b0 2024-04-25T02:08:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfb37fe640 10 client.17235 handle_command_reply: tid=0 2024-04-25T02:08:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 1 client.17235 shutdown 2024-04-25T02:08:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1066091449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 msgr2=0x55a357114c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a357114c80 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400c6a0 tx=0x7fdfc40d5530 comp rx=0 tx=0).stop 2024-04-25T02:08:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1066091449 shutdown_connections 2024-04-25T02:08:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a3570b64b0 0x55a35711ecf0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3570a6230 0x55a3570a9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a35709bb60 0x55a3571151c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 1 --2- 172.21.15.31:0/1066091449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35709b240 0x55a357114c80 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:11.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1066091449 >> 172.21.15.31:0/1066091449 conn(0x55a357097560 msgr2=0x55a3570a3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:11.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfb37fe640 0 client.17235 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:11.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1066091449 shutdown_connections 2024-04-25T02:08:11.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 1 -- 172.21.15.31:0/1066091449 wait complete. 2024-04-25T02:08:11.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.562+0000 7fdfdcd22740 20 client.17235 trim_cache size 0 max 16384 2024-04-25T02:08:11.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:08:11.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:08:11.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:08:11.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:08:11.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 -- 172.21.15.31:0/2184300975 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfbc0a2e20 msgr2=0x7fdfbc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:11.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfbc0a2e20 0x7fdfbc0a52e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009fd0 tx=0x7fdfcc004a40 comp rx=0 tx=0).stop 2024-04-25T02:08:11.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 -- 172.21.15.31:0/2184300975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd4101730 msgr2=0x7fdfd4073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:11.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd4101730 0x7fdfd4073a40 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400e9f0 tx=0x7fdfc400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:08:11.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 -- 172.21.15.31:0/2184300975 shutdown_connections 2024-04-25T02:08:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fdfbc0a2e20 0x7fdfbc0a52e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfd4107740 0x7fdfd40744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd41020f0 0x7fdfd4073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 --2- 172.21.15.31:0/2184300975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd4101730 0x7fdfd4073a40 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 -- 172.21.15.31:0/2184300975 >> 172.21.15.31:0/2184300975 conn(0x7fdfd40fd200 msgr2=0x7fdfd40fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 -- 172.21.15.31:0/2184300975 shutdown_connections 2024-04-25T02:08:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:11.563+0000 7fdfda370640 1 -- 172.21.15.31:0/2184300975 wait complete. 2024-04-25T02:08:12.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 2028 16.60 MB/sec warmup 16 sec latency 10331.543 ms 2024-04-25T02:08:12.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:11 smithi040 ceph-mon[32418]: pgmap v429: 97 pgs: 97 active+clean; 362 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 391 op/s 2024-04-25T02:08:12.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:11 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:08:12.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:11 smithi134 ceph-mon[33098]: pgmap v429: 97 pgs: 97 active+clean; 362 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 391 op/s 2024-04-25T02:08:12.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:11 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:08:12.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:11 smithi031 ceph-mon[29204]: pgmap v429: 97 pgs: 97 active+clean; 362 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 391 op/s 2024-04-25T02:08:12.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:11 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:08:13.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 2167 15.98 MB/sec warmup 17 sec latency 51.928 ms 2024-04-25T02:08:14.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 2363 15.95 MB/sec warmup 18 sec latency 52.571 ms 2024-04-25T02:08:14.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:13 smithi040 ceph-mon[32418]: pgmap v430: 97 pgs: 97 active+clean; 362 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 2.9 MiB/s wr, 345 op/s 2024-04-25T02:08:14.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:13 smithi134 ceph-mon[33098]: pgmap v430: 97 pgs: 97 active+clean; 362 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 2.9 MiB/s wr, 345 op/s 2024-04-25T02:08:14.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:13 smithi031 ceph-mon[29204]: pgmap v430: 97 pgs: 97 active+clean; 362 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 2.9 MiB/s wr, 345 op/s 2024-04-25T02:08:15.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 2646 16.56 MB/sec warmup 19 sec latency 116.131 ms 2024-04-25T02:08:16.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 2922 16.63 MB/sec warmup 20 sec latency 57.486 ms 2024-04-25T02:08:16.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:15 smithi040 ceph-mon[32418]: pgmap v431: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 3.5 MiB/s wr, 348 op/s 2024-04-25T02:08:16.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:15 smithi134 ceph-mon[33098]: pgmap v431: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 3.5 MiB/s wr, 348 op/s 2024-04-25T02:08:16.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:15 smithi031 ceph-mon[29204]: pgmap v431: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 3.5 MiB/s wr, 348 op/s 2024-04-25T02:08:17.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 3013 15.98 MB/sec warmup 21 sec latency 74.775 ms 2024-04-25T02:08:18.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 3160 15.96 MB/sec warmup 22 sec latency 151.210 ms 2024-04-25T02:08:18.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:17 smithi040 ceph-mon[32418]: pgmap v432: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 3.5 MiB/s wr, 348 op/s 2024-04-25T02:08:18.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:17 smithi134 ceph-mon[33098]: pgmap v432: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 3.5 MiB/s wr, 348 op/s 2024-04-25T02:08:18.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:17 smithi031 ceph-mon[29204]: pgmap v432: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 3.5 MiB/s wr, 348 op/s 2024-04-25T02:08:19.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 3331 15.60 MB/sec warmup 23 sec latency 56.435 ms 2024-04-25T02:08:20.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 3500 15.47 MB/sec warmup 24 sec latency 53.697 ms 2024-04-25T02:08:20.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:19 smithi040 ceph-mon[32418]: pgmap v433: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 3.5 MiB/s wr, 348 op/s 2024-04-25T02:08:20.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:19 smithi134 ceph-mon[33098]: pgmap v433: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 3.5 MiB/s wr, 348 op/s 2024-04-25T02:08:20.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:19 smithi031 ceph-mon[29204]: pgmap v433: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 3.5 MiB/s wr, 348 op/s 2024-04-25T02:08:21.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 3644 15.19 MB/sec warmup 25 sec latency 54.949 ms 2024-04-25T02:08:22.089 INFO:tasks.workunit.client.0.smithi031.stdout: 10 3766 14.70 MB/sec warmup 26 sec latency 60.476 ms 2024-04-25T02:08:22.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:21 smithi040 ceph-mon[32418]: pgmap v434: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.4 MiB/s wr, 393 op/s 2024-04-25T02:08:22.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:21 smithi134 ceph-mon[33098]: pgmap v434: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.4 MiB/s wr, 393 op/s 2024-04-25T02:08:22.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:21 smithi031 ceph-mon[29204]: pgmap v434: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.4 MiB/s wr, 393 op/s 2024-04-25T02:08:23.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 3923 14.28 MB/sec warmup 27 sec latency 113.941 ms 2024-04-25T02:08:24.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 4104 14.00 MB/sec warmup 28 sec latency 206.719 ms 2024-04-25T02:08:24.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:23 smithi040 ceph-mon[32418]: pgmap v435: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 4.8 MiB/s wr, 188 op/s 2024-04-25T02:08:24.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:23 smithi134 ceph-mon[33098]: pgmap v435: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 4.8 MiB/s wr, 188 op/s 2024-04-25T02:08:24.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:23 smithi031 ceph-mon[29204]: pgmap v435: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 4.8 MiB/s wr, 188 op/s 2024-04-25T02:08:24.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.697+0000 7f4cde714640 1 -- 172.21.15.31:0/2362476905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd8100670 msgr2=0x7f4cd8100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:24.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.697+0000 7f4cde714640 1 --2- 172.21.15.31:0/2362476905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd8100670 0x7f4cd8100af0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009900 tx=0x7f4ccc02a410 comp rx=0 tx=0).stop 2024-04-25T02:08:24.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.698+0000 7f4cde714640 1 -- 172.21.15.31:0/2362476905 shutdown_connections 2024-04-25T02:08:24.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.698+0000 7f4cde714640 1 --2- 172.21.15.31:0/2362476905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd8101030 0x7f4cd8068df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.698+0000 7f4cde714640 1 --2- 172.21.15.31:0/2362476905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd8100670 0x7f4cd8100af0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.698+0000 7f4cde714640 1 --2- 172.21.15.31:0/2362476905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd8106630 0x7f4cd8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.698+0000 7f4cde714640 1 -- 172.21.15.31:0/2362476905 >> 172.21.15.31:0/2362476905 conn(0x7f4cd80777c0 msgr2=0x7f4cd8077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:24.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.698+0000 7f4cde714640 1 -- 172.21.15.31:0/2362476905 shutdown_connections 2024-04-25T02:08:24.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.698+0000 7f4cde714640 1 -- 172.21.15.31:0/2362476905 wait complete. 2024-04-25T02:08:24.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.698+0000 7f4cde714640 1 Processor -- start 2024-04-25T02:08:24.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.698+0000 7f4cde714640 1 -- start start 2024-04-25T02:08:24.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cde714640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd8100670 0x7f4cd810b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cde714640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd8101030 0x7f4cd810baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cde714640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd8106630 0x7f4cd810c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cde714640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd8111a30 con 0x7f4cd8100670 2024-04-25T02:08:24.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cde714640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd81118b0 con 0x7f4cd8106630 2024-04-25T02:08:24.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cde714640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd8111bb0 con 0x7f4cd8101030 2024-04-25T02:08:24.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cd7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd8100670 0x7f4cd810b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cd77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd8101030 0x7f4cd810baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cd77fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd8101030 0x7f4cd810baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:47960/0 (socket says 172.21.15.31:47960) 2024-04-25T02:08:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cd7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd8100670 0x7f4cd810b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48512/0 (socket says 172.21.15.31:48512) 2024-04-25T02:08:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cd77fe640 1 -- 172.21.15.31:0/1485226858 learned_addr learned my addr 172.21.15.31:0/1485226858 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cdccb1640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd8106630 0x7f4cd810c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cd77fe640 1 -- 172.21.15.31:0/1485226858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd8106630 msgr2=0x7f4cd810c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd8106630 0x7f4cd810c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cd77fe640 1 -- 172.21.15.31:0/1485226858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd8100670 msgr2=0x7f4cd810b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd8100670 0x7f4cd810b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cd77fe640 1 -- 172.21.15.31:0/1485226858 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd81a6370 con 0x7f4cd8101030 2024-04-25T02:08:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.699+0000 7f4cd7fff640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd8100670 0x7f4cd810b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.700+0000 7f4cdccb1640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd8106630 0x7f4cd810c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.700+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd8101030 0x7f4cd810baf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc005ab0 tx=0x7f4ccc003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.700+0000 7f4cd57fa640 1 -- 172.21.15.31:0/1485226858 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc03f070 con 0x7f4cd8101030 2024-04-25T02:08:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.700+0000 7f4cd57fa640 1 -- 172.21.15.31:0/1485226858 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ccc031b60 con 0x7f4cd8101030 2024-04-25T02:08:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.700+0000 7f4cde714640 1 -- 172.21.15.31:0/1485226858 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd81a6600 con 0x7f4cd8101030 2024-04-25T02:08:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.700+0000 7f4cd57fa640 1 -- 172.21.15.31:0/1485226858 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc035030 con 0x7f4cd8101030 2024-04-25T02:08:24.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.700+0000 7f4cde714640 1 -- 172.21.15.31:0/1485226858 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd81a6af0 con 0x7f4cd8101030 2024-04-25T02:08:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.703+0000 7f4cd57fa640 1 -- 172.21.15.31:0/1485226858 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4ccc042070 con 0x7f4cd8101030 2024-04-25T02:08:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4ce1425740 1 Processor -- start 2024-04-25T02:08:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4ce1425740 1 -- start start 2024-04-25T02:08:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2d9de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9e370 0x5637f2da8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2da8a40 0x5637f2daae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4ce1425740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5637f2cf9cb0 con 0x5637f2d9e370 2024-04-25T02:08:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4ce1425740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5637f2ca25b0 con 0x5637f2d9da50 2024-04-25T02:08:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4ce1425740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5637f2cf5c70 con 0x5637f2da8a40 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4cd57fa640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cc00a2dd0 0x7f4cc00a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4cd7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9e370 0x5637f2da8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4cd7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9e370 0x5637f2da8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48520/0 (socket says 172.21.15.31:48520) 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4cd7fff640 1 -- 172.21.15.31:0/757018267 learned_addr learned my addr 172.21.15.31:0/757018267 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.704+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/757018267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2da8a40 0x5637f2daae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cdccb1640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cc00a2dd0 0x7f4cc00a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cd57fa640 1 -- 172.21.15.31:0/1485226858 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8484+0+0 (secure 0 0 0) 0x7f4ccc0efa60 con 0x7f4cd8101030 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cdccb1640 1 --2- 172.21.15.31:0/757018267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2d9de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cd7fff640 1 -- 172.21.15.31:0/757018267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2da8a40 msgr2=0x5637f2daae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cd7fff640 1 --2- 172.21.15.31:0/757018267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2da8a40 0x5637f2daae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cd7fff640 1 -- 172.21.15.31:0/757018267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 msgr2=0x5637f2d9de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cd7fff640 1 --2- 172.21.15.31:0/757018267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2d9de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cd7fff640 1 -- 172.21.15.31:0/757018267 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637f2a0e210 con 0x5637f2d9e370 2024-04-25T02:08:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cdccb1640 1 --2- 172.21.15.31:0/757018267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2d9de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:08:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cd7fff640 1 --2- 172.21.15.31:0/757018267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9e370 0x5637f2da8500 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80114f0 tx=0x7f4cc80119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7278ec8877fff69f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cbe7fc640 1 -- 172.21.15.31:0/757018267 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc801e070 con 0x5637f2d9e370 2024-04-25T02:08:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cdccb1640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cc00a2dd0 0x7f4cc00a5290 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0005fd0 tx=0x7f4cd0011440 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cbe7fc640 1 -- 172.21.15.31:0/757018267 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cc800fa70 con 0x5637f2d9e370 2024-04-25T02:08:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.705+0000 7f4cbe7fc640 1 -- 172.21.15.31:0/757018267 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8019aa0 con 0x5637f2d9e370 2024-04-25T02:08:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.706+0000 7f4ce1425740 1 -- 172.21.15.31:0/757018267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9e370 msgr2=0x5637f2da8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.706+0000 7f4ce1425740 1 --2- 172.21.15.31:0/757018267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9e370 0x5637f2da8500 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80114f0 tx=0x7f4cc80119b0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.706+0000 7f4ce1425740 1 -- 172.21.15.31:0/757018267 shutdown_connections 2024-04-25T02:08:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.706+0000 7f4ce1425740 1 --2- 172.21.15.31:0/757018267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2da8a40 0x5637f2daae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.706+0000 7f4ce1425740 1 --2- 172.21.15.31:0/757018267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9e370 0x5637f2da8500 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.706+0000 7f4ce1425740 1 --2- 172.21.15.31:0/757018267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2d9de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.706+0000 7f4ce1425740 1 -- 172.21.15.31:0/757018267 >> 172.21.15.31:0/757018267 conn(0x5637f2d99d70 msgr2=0x5637f2d9a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.706+0000 7f4ce1425740 1 -- 172.21.15.31:0/757018267 shutdown_connections 2024-04-25T02:08:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.706+0000 7f4ce1425740 1 -- 172.21.15.31:0/757018267 wait complete. 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.707+0000 7f4ce1425740 1 -- start start 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.707+0000 7f4ce1425740 1 Processor -- start 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.707+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9da50 0x5637f2da59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.707+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9e370 0x5637f2da5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.707+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2da8a40 0x5637f2da6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.707+0000 7f4ce1425740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5637f2dab830 con 0x5637f2d9da50 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.707+0000 7f4ce1425740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5637f2cb5b50 con 0x5637f2d9e370 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.707+0000 7f4ce1425740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5637f2dab9b0 con 0x5637f2da8a40 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.707+0000 7f4ce1425740 10 client.0 _ll_register_callbacks cb 0x5637f2cf0ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cd7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9da50 0x5637f2da59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cdccb1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2da8a40 0x5637f2da6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cd77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9e370 0x5637f2da5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cdccb1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2da8a40 0x5637f2da6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:47978/0 (socket says 172.21.15.31:47978) 2024-04-25T02:08:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cdccb1640 1 -- 172.21.15.31:0/3119145998 learned_addr learned my addr 172.21.15.31:0/3119145998 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cd7fff640 1 -- 172.21.15.31:0/3119145998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2da8a40 msgr2=0x5637f2da6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cd7fff640 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2da8a40 0x5637f2da6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cd7fff640 1 -- 172.21.15.31:0/3119145998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9e370 msgr2=0x5637f2da5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cd7fff640 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9e370 0x5637f2da5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cd7fff640 1 -- 172.21.15.31:0/3119145998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637f2ada620 con 0x5637f2d9da50 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9e370 0x5637f2da5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cd7fff640 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9da50 0x5637f2da59f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8011eb0 tx=0x7f4cc8019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39bfaa1dc5a706 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.708+0000 7f4cb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4cb77fe640 1 -- 172.21.15.31:0/3119145998 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc801e040 con 0x5637f2d9da50 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4cb77fe640 1 -- 172.21.15.31:0/3119145998 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8484+0+0 (secure 0 0 0) 0x7f4cc800f6c0 con 0x5637f2d9da50 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4ce1425740 1 -- 172.21.15.31:0/3119145998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cc800fa70 con 0x5637f2d9da50 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4cb77fe640 1 -- 172.21.15.31:0/3119145998 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4cc800f8c0 con 0x5637f2d9da50 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4ce1425740 10 client.17247 fetch_fsmap learned FSMap version 42 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4ce1425740 1 -- 172.21.15.31:0/3119145998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5637f2db7a50 con 0x5637f2d9da50 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4cb77fe640 1 -- 172.21.15.31:0/3119145998 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cc80197e0 con 0x5637f2d9da50 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4cb77fe640 1 -- 172.21.15.31:0/3119145998 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 42) ==== 8394+0+0 (secure 0 0 0) 0x7f4cc8021070 con 0x5637f2d9da50 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4ce1425740 10 client.17247 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T02:08:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4ce1425740 10 client.17247 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4ce1425740 20 client.17247 populate_metadata read hostname 'smithi031' 2024-04-25T02:08:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4ce1425740 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5637f2db8e60 0x5637f2dbb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4ce1425740 4 client.17247 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:08:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.709+0000 7f4ce1425740 1 -- 172.21.15.31:0/3119145998 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4cc800fa70 con 0x5637f2db8e60 2024-04-25T02:08:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.710+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5637f2db8e60 0x5637f2dbb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.710+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5637f2db8e60 0x5637f2dbb250 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c3d039de27e4af server_cookie=3b38ee40c060e1d6 in_seq=0 out_seq=0 2024-04-25T02:08:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.710+0000 7f4cb77fe640 10 client.17247 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4cb77fe640 1 -- 172.21.15.31:0/3119145998 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f4cc800fa70 con 0x5637f2db8e60 2024-04-25T02:08:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4cb77fe640 10 client.17247 handle_command_reply: tid=0 2024-04-25T02:08:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 1 client.17247 shutdown 2024-04-25T02:08:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 1 -- 172.21.15.31:0/3119145998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9da50 msgr2=0x5637f2da59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9da50 0x5637f2da59f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8011eb0 tx=0x7f4cc8019640 comp rx=0 tx=0).stop 2024-04-25T02:08:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 1 -- 172.21.15.31:0/3119145998 shutdown_connections 2024-04-25T02:08:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5637f2db8e60 0x5637f2dbb250 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2da8a40 0x5637f2da6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9e370 0x5637f2da5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4cb77fe640 0 client.17247 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 1 --2- 172.21.15.31:0/3119145998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2d9da50 0x5637f2da59f0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 1 -- 172.21.15.31:0/3119145998 >> 172.21.15.31:0/3119145998 conn(0x5637f2d99d70 msgr2=0x5637f2d9fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 1 -- 172.21.15.31:0/3119145998 shutdown_connections 2024-04-25T02:08:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 1 -- 172.21.15.31:0/3119145998 wait complete. 2024-04-25T02:08:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.712+0000 7f4ce1425740 20 client.17247 trim_cache size 0 max 16384 2024-04-25T02:08:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4ce1425740 1 Processor -- start 2024-04-25T02:08:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4ce1425740 1 -- start start 2024-04-25T02:08:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2db46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2d9e370 0x5637f2da01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 0x5637f2dc0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4ce1425740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd81ab0b0 con 0x5637f2da8a40 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4ce1425740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd81acd70 con 0x5637f2d9da50 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4ce1425740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd81ab780 con 0x5637f2d9e370 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4cd7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2db46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4cd7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2db46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44000/0 (socket says 172.21.15.31:44000) 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4cdccb1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 0x5637f2dc0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4cd7fff640 1 -- 172.21.15.31:0/1508507005 learned_addr learned my addr 172.21.15.31:0/1508507005 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.739+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/1508507005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2d9e370 0x5637f2da01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.740+0000 7f4cdccb1640 1 -- 172.21.15.31:0/1508507005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2d9e370 msgr2=0x5637f2da01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.740+0000 7f4cdccb1640 1 --2- 172.21.15.31:0/1508507005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2d9e370 0x5637f2da01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.740+0000 7f4cdccb1640 1 -- 172.21.15.31:0/1508507005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 msgr2=0x5637f2db46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.740+0000 7f4cdccb1640 1 --2- 172.21.15.31:0/1508507005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2db46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.740+0000 7f4cdccb1640 1 -- 172.21.15.31:0/1508507005 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637f2dbb5c0 con 0x5637f2da8a40 2024-04-25T02:08:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.740+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/1508507005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2d9e370 0x5637f2da01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.740+0000 7f4cdccb1640 1 --2- 172.21.15.31:0/1508507005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 0x5637f2dc0ab0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00099e0 tx=0x7f4cd0009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=395c74f364932706 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.740+0000 7f4cbdffb640 1 -- 172.21.15.31:0/1508507005 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd000fe30 con 0x5637f2da8a40 2024-04-25T02:08:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.740+0000 7f4cbdffb640 1 -- 172.21.15.31:0/1508507005 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cd0004040 con 0x5637f2da8a40 2024-04-25T02:08:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.740+0000 7f4cbdffb640 1 -- 172.21.15.31:0/1508507005 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0005020 con 0x5637f2da8a40 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.741+0000 7f4ce1425740 1 -- 172.21.15.31:0/1508507005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 msgr2=0x5637f2dc0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.741+0000 7f4ce1425740 1 --2- 172.21.15.31:0/1508507005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 0x5637f2dc0ab0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00099e0 tx=0x7f4cd0009500 comp rx=0 tx=0).stop 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.741+0000 7f4ce1425740 1 -- 172.21.15.31:0/1508507005 shutdown_connections 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.741+0000 7f4ce1425740 1 --2- 172.21.15.31:0/1508507005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 0x5637f2dc0ab0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.741+0000 7f4ce1425740 1 --2- 172.21.15.31:0/1508507005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2d9e370 0x5637f2da01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.741+0000 7f4ce1425740 1 --2- 172.21.15.31:0/1508507005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2db46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.741+0000 7f4ce1425740 1 -- 172.21.15.31:0/1508507005 >> 172.21.15.31:0/1508507005 conn(0x5637f2d99d70 msgr2=0x5637f2da0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.741+0000 7f4ce1425740 1 -- 172.21.15.31:0/1508507005 shutdown_connections 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.741+0000 7f4ce1425740 1 -- 172.21.15.31:0/1508507005 wait complete. 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.741+0000 7f4ce1425740 1 -- start start 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4ce1425740 1 Processor -- start 2024-04-25T02:08:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2db0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2d9e370 0x5637f2db1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4ce1425740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 0x5637f2da1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4ce1425740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd81ac6a0 con 0x5637f2da8a40 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4ce1425740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd81abf10 con 0x5637f2d9da50 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4ce1425740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd81a90d0 con 0x5637f2d9e370 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4ce1425740 10 client.0 _ll_register_callbacks cb 0x7f4cc801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4cdccb1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 0x5637f2da1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4cd7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2db0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4cdccb1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 0x5637f2da1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48546/0 (socket says 172.21.15.31:48546) 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4cdccb1640 1 -- 172.21.15.31:0/4169091864 learned_addr learned my addr 172.21.15.31:0/4169091864 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.742+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2d9e370 0x5637f2db1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4cd7fff640 1 -- 172.21.15.31:0/4169091864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2d9e370 msgr2=0x5637f2db1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4cd7fff640 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2d9e370 0x5637f2db1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4cd7fff640 1 -- 172.21.15.31:0/4169091864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 msgr2=0x5637f2da1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4cd7fff640 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 0x5637f2da1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4cd7fff640 1 -- 172.21.15.31:0/4169091864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637f2da21a0 con 0x5637f2d9da50 2024-04-25T02:08:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4cdccb1640 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 0x5637f2da1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4cd7fff640 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2db0c00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8003710 tx=0x7f4cc800ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7542eccb24875c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4cb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:08:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4cb77fe640 1 -- 172.21.15.31:0/4169091864 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc80052d0 con 0x5637f2d9da50 2024-04-25T02:08:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4ce1425740 1 -- 172.21.15.31:0/4169091864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cd0004040 con 0x5637f2d9da50 2024-04-25T02:08:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4cb77fe640 1 -- 172.21.15.31:0/4169091864 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8484+0+0 (secure 0 0 0) 0x7f4cc8005470 con 0x5637f2d9da50 2024-04-25T02:08:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4cb77fe640 1 -- 172.21.15.31:0/4169091864 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4cc803b050 con 0x5637f2d9da50 2024-04-25T02:08:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4ce1425740 10 client.26825 fetch_fsmap learned FSMap version 42 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.743+0000 7f4ce1425740 1 -- 172.21.15.31:0/4169091864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5637f2e1eb10 con 0x5637f2d9da50 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.744+0000 7f4cb77fe640 1 -- 172.21.15.31:0/4169091864 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cd0004040 con 0x5637f2d9da50 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.744+0000 7f4cb77fe640 1 -- 172.21.15.31:0/4169091864 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) ==== 8394+0+0 (secure 0 0 0) 0x7f4cc8032070 con 0x5637f2d9da50 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.744+0000 7f4ce1425740 10 client.26825 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.744+0000 7f4ce1425740 10 client.26825 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.744+0000 7f4ce1425740 20 client.26825 populate_metadata read hostname 'smithi031' 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.744+0000 7f4ce1425740 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5637f2e1f350 0x5637f2e21770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.744+0000 7f4ce1425740 4 client.26825 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.744+0000 7f4ce1425740 1 -- 172.21.15.31:0/4169091864 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5637f2d071f0 con 0x5637f2e1f350 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.744+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5637f2e1f350 0x5637f2e21770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.745+0000 7f4cb77fe640 10 client.26825 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.745+0000 7f4cd77fe640 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5637f2e1f350 0x5637f2e21770 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad5f86c77f97cb32 server_cookie=87696ac5961e4408 in_seq=0 out_seq=0 2024-04-25T02:08:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.829+0000 7f4cb77fe640 1 -- 172.21.15.31:0/4169091864 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5637f2d071f0 con 0x5637f2e1f350 2024-04-25T02:08:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.829+0000 7f4cb77fe640 10 client.26825 handle_command_reply: tid=0 2024-04-25T02:08:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.829+0000 7f4ce1425740 1 client.26825 shutdown 2024-04-25T02:08:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.829+0000 7f4ce1425740 1 -- 172.21.15.31:0/4169091864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 msgr2=0x5637f2db0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.829+0000 7f4ce1425740 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2db0c00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8003710 tx=0x7f4cc800ff70 comp rx=0 tx=0).stop 2024-04-25T02:08:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.829+0000 7f4ce1425740 1 -- 172.21.15.31:0/4169091864 shutdown_connections 2024-04-25T02:08:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.829+0000 7f4ce1425740 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5637f2e1f350 0x5637f2e21770 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4ce1425740 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637f2da8a40 0x5637f2da1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4ce1425740 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5637f2d9e370 0x5637f2db1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4ce1425740 1 --2- 172.21.15.31:0/4169091864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637f2d9da50 0x5637f2db0c00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4ce1425740 1 -- 172.21.15.31:0/4169091864 >> 172.21.15.31:0/4169091864 conn(0x5637f2d99d70 msgr2=0x5637f2d8fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4cb77fe640 0 client.26825 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4ce1425740 1 -- 172.21.15.31:0/4169091864 shutdown_connections 2024-04-25T02:08:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4ce1425740 1 -- 172.21.15.31:0/4169091864 wait complete. 2024-04-25T02:08:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4ce1425740 20 client.26825 trim_cache size 0 max 16384 2024-04-25T02:08:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:08:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:08:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:08:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:08:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4cde714640 1 -- 172.21.15.31:0/1485226858 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cc00a2dd0 msgr2=0x7f4cc00a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4cde714640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cc00a2dd0 0x7f4cc00a5290 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0005fd0 tx=0x7f4cd0011440 comp rx=0 tx=0).stop 2024-04-25T02:08:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4cde714640 1 -- 172.21.15.31:0/1485226858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd8101030 msgr2=0x7f4cd810baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.830+0000 7f4cde714640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd8101030 0x7f4cd810baf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc005ab0 tx=0x7f4ccc003c10 comp rx=0 tx=0).stop 2024-04-25T02:08:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.831+0000 7f4cde714640 1 -- 172.21.15.31:0/1485226858 shutdown_connections 2024-04-25T02:08:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.831+0000 7f4cde714640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f4cc00a2dd0 0x7f4cc00a5290 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.831+0000 7f4cde714640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd8106630 0x7f4cd810c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.831+0000 7f4cde714640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cd8101030 0x7f4cd810baf0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.831+0000 7f4cde714640 1 --2- 172.21.15.31:0/1485226858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cd8100670 0x7f4cd810b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.831+0000 7f4cde714640 1 -- 172.21.15.31:0/1485226858 >> 172.21.15.31:0/1485226858 conn(0x7f4cd80777c0 msgr2=0x7f4cd8104130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.831+0000 7f4cde714640 1 -- 172.21.15.31:0/1485226858 shutdown_connections 2024-04-25T02:08:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:24.831+0000 7f4cde714640 1 -- 172.21.15.31:0/1485226858 wait complete. 2024-04-25T02:08:25.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 4277 13.85 MB/sec warmup 29 sec latency 64.283 ms 2024-04-25T02:08:26.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 4444 13.62 MB/sec warmup 30 sec latency 78.374 ms 2024-04-25T02:08:26.165 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:25 smithi040 ceph-mon[32418]: pgmap v436: 97 pgs: 97 active+clean; 394 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.6 MiB/s wr, 299 op/s 2024-04-25T02:08:26.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:25 smithi134 ceph-mon[33098]: pgmap v436: 97 pgs: 97 active+clean; 394 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.6 MiB/s wr, 299 op/s 2024-04-25T02:08:26.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:25 smithi031 ceph-mon[29204]: pgmap v436: 97 pgs: 97 active+clean; 394 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.6 MiB/s wr, 299 op/s 2024-04-25T02:08:27.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 4634 13.87 MB/sec warmup 31 sec latency 72.427 ms 2024-04-25T02:08:28.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 4797 13.58 MB/sec warmup 32 sec latency 54.674 ms 2024-04-25T02:08:28.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:27 smithi134 ceph-mon[33098]: pgmap v437: 97 pgs: 97 active+clean; 394 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-04-25T02:08:28.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:27 smithi031 ceph-mon[29204]: pgmap v437: 97 pgs: 97 active+clean; 394 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-04-25T02:08:28.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:27 smithi040 ceph-mon[32418]: pgmap v437: 97 pgs: 97 active+clean; 394 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-04-25T02:08:29.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 4980 13.38 MB/sec warmup 33 sec latency 77.293 ms 2024-04-25T02:08:30.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 5138 13.06 MB/sec warmup 34 sec latency 52.458 ms 2024-04-25T02:08:30.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:29 smithi134 ceph-mon[33098]: pgmap v438: 97 pgs: 97 active+clean; 394 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-04-25T02:08:30.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:29 smithi031 ceph-mon[29204]: pgmap v438: 97 pgs: 97 active+clean; 394 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-04-25T02:08:30.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:29 smithi040 ceph-mon[32418]: pgmap v438: 97 pgs: 97 active+clean; 394 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-04-25T02:08:31.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 5311 12.75 MB/sec warmup 35 sec latency 55.461 ms 2024-04-25T02:08:32.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 5498 12.61 MB/sec warmup 36 sec latency 61.448 ms 2024-04-25T02:08:32.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:31 smithi134 ceph-mon[33098]: pgmap v439: 97 pgs: 97 active+clean; 379 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 359 op/s 2024-04-25T02:08:32.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:31 smithi031 ceph-mon[29204]: pgmap v439: 97 pgs: 97 active+clean; 379 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 359 op/s 2024-04-25T02:08:32.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:31 smithi040 ceph-mon[32418]: pgmap v439: 97 pgs: 97 active+clean; 379 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 359 op/s 2024-04-25T02:08:33.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 5645 12.40 MB/sec warmup 37 sec latency 66.074 ms 2024-04-25T02:08:34.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 5836 12.48 MB/sec warmup 38 sec latency 105.651 ms 2024-04-25T02:08:34.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:33 smithi134 ceph-mon[33098]: pgmap v440: 97 pgs: 97 active+clean; 379 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 260 op/s 2024-04-25T02:08:34.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:33 smithi031 ceph-mon[29204]: pgmap v440: 97 pgs: 97 active+clean; 379 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 260 op/s 2024-04-25T02:08:34.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:33 smithi040 ceph-mon[32418]: pgmap v440: 97 pgs: 97 active+clean; 379 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 260 op/s 2024-04-25T02:08:35.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 6099 12.75 MB/sec warmup 39 sec latency 52.490 ms 2024-04-25T02:08:36.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 6345 12.83 MB/sec warmup 40 sec latency 60.003 ms 2024-04-25T02:08:36.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:35 smithi134 ceph-mon[33098]: pgmap v441: 97 pgs: 97 active+clean; 391 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 369 op/s 2024-04-25T02:08:36.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:35 smithi031 ceph-mon[29204]: pgmap v441: 97 pgs: 97 active+clean; 391 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 369 op/s 2024-04-25T02:08:36.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:35 smithi040 ceph-mon[32418]: pgmap v441: 97 pgs: 97 active+clean; 391 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 369 op/s 2024-04-25T02:08:37.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 6455 12.65 MB/sec warmup 41 sec latency 54.622 ms 2024-04-25T02:08:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.964+0000 7f84c0142640 1 -- 172.21.15.31:0/3428732225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b810bd00 msgr2=0x7f84b810e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.964+0000 7f84c0142640 1 --2- 172.21.15.31:0/3428732225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b810bd00 0x7f84b810e1a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009ca0 tx=0x7f84b002a950 comp rx=0 tx=0).stop 2024-04-25T02:08:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.964+0000 7f84c0142640 1 -- 172.21.15.31:0/3428732225 shutdown_connections 2024-04-25T02:08:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.964+0000 7f84c0142640 1 --2- 172.21.15.31:0/3428732225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b810bd00 0x7f84b810e1a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.964+0000 7f84c0142640 1 --2- 172.21.15.31:0/3428732225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84b8103250 0x7f84b810b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.964+0000 7f84c0142640 1 --2- 172.21.15.31:0/3428732225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b8102930 0x7f84b8102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.965+0000 7f84c0142640 1 -- 172.21.15.31:0/3428732225 >> 172.21.15.31:0/3428732225 conn(0x7f84b80fc110 msgr2=0x7f84b80fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.965+0000 7f84c0142640 1 -- 172.21.15.31:0/3428732225 shutdown_connections 2024-04-25T02:08:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.965+0000 7f84c0142640 1 -- 172.21.15.31:0/3428732225 wait complete. 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.965+0000 7f84c0142640 1 Processor -- start 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.965+0000 7f84c0142640 1 -- start start 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84c0142640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b8102930 0x7f84b8197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84c0142640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b8103250 0x7f84b8198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84c0142640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84b810bd00 0x7f84b819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84c0142640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f84b8111a00 con 0x7f84b8103250 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84c0142640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f84b8111880 con 0x7f84b8102930 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84c0142640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f84b8111b80 con 0x7f84b810bd00 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84bdede640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b8102930 0x7f84b8197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84bd6dd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b8103250 0x7f84b8198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84bdede640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b8102930 0x7f84b8197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45418/0 (socket says 172.21.15.31:45418) 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84bdede640 1 -- 172.21.15.31:0/2223858783 learned_addr learned my addr 172.21.15.31:0/2223858783 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84be6df640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84b810bd00 0x7f84b819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84bdede640 1 -- 172.21.15.31:0/2223858783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84b810bd00 msgr2=0x7f84b819c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84bdede640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84b810bd00 0x7f84b819c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84bdede640 1 -- 172.21.15.31:0/2223858783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b8103250 msgr2=0x7f84b8198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84bdede640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b8103250 0x7f84b8198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84bdede640 1 -- 172.21.15.31:0/2223858783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b819ce30 con 0x7f84b8102930 2024-04-25T02:08:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.966+0000 7f84bdede640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b8102930 0x7f84b8197e40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f84b400ea60 tx=0x7f84b400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.967+0000 7f84aeffd640 1 -- 172.21.15.31:0/2223858783 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b400cde0 con 0x7f84b8102930 2024-04-25T02:08:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.967+0000 7f84aeffd640 1 -- 172.21.15.31:0/2223858783 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84b40035f0 con 0x7f84b8102930 2024-04-25T02:08:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.967+0000 7f84c0142640 1 -- 172.21.15.31:0/2223858783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84b819d120 con 0x7f84b8102930 2024-04-25T02:08:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.967+0000 7f84aeffd640 1 -- 172.21.15.31:0/2223858783 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b4010690 con 0x7f84b8102930 2024-04-25T02:08:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.967+0000 7f84c0142640 1 -- 172.21.15.31:0/2223858783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84b81004b0 con 0x7f84b8102930 2024-04-25T02:08:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.970+0000 7f84aeffd640 1 -- 172.21.15.31:0/2223858783 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f84b40037a0 con 0x7f84b8102930 2024-04-25T02:08:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.970+0000 7f84c2e50740 1 Processor -- start 2024-04-25T02:08:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84c2e50740 1 -- start start 2024-04-25T02:08:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84aeffd640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f84a40a2da0 0x7f84a40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a36620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a40cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a41230 0x55da66a43620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84c2e50740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55da6699acb0 con 0x55da66a36240 2024-04-25T02:08:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84c2e50740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55da669435b0 con 0x55da66a41230 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84c2e50740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55da66996c70 con 0x55da66a36b60 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84be6df640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f84a40a2da0 0x7f84a40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84bd6dd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a40cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84bdede640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a36620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84bd6dd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a40cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:57956/0 (socket says 172.21.15.31:57956) 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84bd6dd640 1 -- 172.21.15.31:0/3137927277 learned_addr learned my addr 172.21.15.31:0/3137927277 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84be6df640 1 --2- 172.21.15.31:0/3137927277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a41230 0x55da66a43620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84bd6dd640 1 -- 172.21.15.31:0/3137927277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a41230 msgr2=0x55da66a43620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84bd6dd640 1 --2- 172.21.15.31:0/3137927277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a41230 0x55da66a43620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84bd6dd640 1 -- 172.21.15.31:0/3137927277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 msgr2=0x55da66a36620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.971+0000 7f84bd6dd640 1 --2- 172.21.15.31:0/3137927277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a36620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.972+0000 7f84bd6dd640 1 -- 172.21.15.31:0/3137927277 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da666af210 con 0x55da66a36b60 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.972+0000 7f84bdede640 1 --2- 172.21.15.31:0/3137927277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a36620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.972+0000 7f84aeffd640 1 -- 172.21.15.31:0/2223858783 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8484+0+0 (secure 0 0 0) 0x7f84b40cf6c0 con 0x7f84b8102930 2024-04-25T02:08:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.972+0000 7f84be6df640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f84a40a2da0 0x7f84a40a5260 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f84b0006010 tx=0x7f84b0005f80 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.972+0000 7f84bd6dd640 1 --2- 172.21.15.31:0/3137927277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a40cf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f84a8009b90 tx=0x7f84a8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad7b219027c722c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.972+0000 7f84a3fff640 1 -- 172.21.15.31:0/3137927277 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a8004030 con 0x55da66a36b60 2024-04-25T02:08:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.972+0000 7f84a3fff640 1 -- 172.21.15.31:0/3137927277 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84a8005020 con 0x55da66a36b60 2024-04-25T02:08:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.972+0000 7f84a3fff640 1 -- 172.21.15.31:0/3137927277 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a8004cb0 con 0x55da66a36b60 2024-04-25T02:08:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.973+0000 7f84c2e50740 1 -- 172.21.15.31:0/3137927277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 msgr2=0x55da66a40cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.973+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3137927277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a40cf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f84a8009b90 tx=0x7f84a8011810 comp rx=0 tx=0).stop 2024-04-25T02:08:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.973+0000 7f84c2e50740 1 -- 172.21.15.31:0/3137927277 shutdown_connections 2024-04-25T02:08:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.973+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3137927277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a41230 0x55da66a43620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.973+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3137927277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a40cf0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.973+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3137927277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a36620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.973+0000 7f84c2e50740 1 -- 172.21.15.31:0/3137927277 >> 172.21.15.31:0/3137927277 conn(0x55da66a32560 msgr2=0x55da66a32940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.973+0000 7f84c2e50740 1 -- 172.21.15.31:0/3137927277 shutdown_connections 2024-04-25T02:08:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.973+0000 7f84c2e50740 1 -- 172.21.15.31:0/3137927277 wait complete. 2024-04-25T02:08:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.973+0000 7f84c2e50740 1 -- start start 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84c2e50740 1 Processor -- start 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a3e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a3e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a41230 0x55da66a3ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84c2e50740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55da66a44020 con 0x55da66a36240 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84c2e50740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55da66956b50 con 0x55da66a41230 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84c2e50740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55da66a441a0 con 0x55da66a36b60 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84c2e50740 10 client.0 _ll_register_callbacks cb 0x55da66991ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84bdede640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a3e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84bdede640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a3e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50540/0 (socket says 172.21.15.31:50540) 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84bdede640 1 -- 172.21.15.31:0/3947525984 learned_addr learned my addr 172.21.15.31:0/3947525984 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.974+0000 7f84bd6dd640 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a3e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84be6df640 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a41230 0x55da66a3ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84bdede640 1 -- 172.21.15.31:0/3947525984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 msgr2=0x55da66a3e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84bdede640 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a3e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84bdede640 1 -- 172.21.15.31:0/3947525984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a41230 msgr2=0x55da66a3ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84bdede640 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a41230 0x55da66a3ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84bdede640 1 -- 172.21.15.31:0/3947525984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da6677b620 con 0x55da66a36240 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84bdede640 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a3e1d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f84b400c760 tx=0x7f84b4005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2577c2761d2141d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84a17fa640 1 -- 172.21.15.31:0/3947525984 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b409c300 con 0x55da66a36240 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84a17fa640 1 -- 172.21.15.31:0/3947525984 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8484+0+0 (secure 0 0 0) 0x7f84b4098270 con 0x55da66a36240 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84c2e50740 1 -- 172.21.15.31:0/3947525984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84a8005020 con 0x55da66a36240 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.975+0000 7f84a17fa640 1 -- 172.21.15.31:0/3947525984 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f84b4003b30 con 0x55da66a36240 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.976+0000 7f84c2e50740 10 client.17265 fetch_fsmap learned FSMap version 42 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.976+0000 7f84c2e50740 1 -- 172.21.15.31:0/3947525984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55da66a50280 con 0x55da66a36240 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.976+0000 7f84a17fa640 1 -- 172.21.15.31:0/3947525984 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84b409bc00 con 0x55da66a36240 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.976+0000 7f84a17fa640 1 -- 172.21.15.31:0/3947525984 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 42) ==== 8394+0+0 (secure 0 0 0) 0x7f84b40a4e40 con 0x55da66a36240 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.976+0000 7f84c2e50740 10 client.17265 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T02:08:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.976+0000 7f84c2e50740 10 client.17265 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.976+0000 7f84c2e50740 20 client.17265 populate_metadata read hostname 'smithi031' 2024-04-25T02:08:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.976+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55da66a51740 0x55da66a53be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.977+0000 7f84c2e50740 4 client.17265 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:08:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.977+0000 7f84c2e50740 1 -- 172.21.15.31:0/3947525984 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f84a8005020 con 0x55da66a51740 2024-04-25T02:08:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.977+0000 7f84bd6dd640 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55da66a51740 0x55da66a53be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.977+0000 7f84a17fa640 10 client.17265 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.977+0000 7f84bd6dd640 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55da66a51740 0x55da66a53be0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b82ab39f165e34a server_cookie=381af009009eba73 in_seq=0 out_seq=0 2024-04-25T02:08:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.978+0000 7f84a17fa640 1 -- 172.21.15.31:0/3947525984 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f84a8005020 con 0x55da66a51740 2024-04-25T02:08:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.978+0000 7f84a17fa640 10 client.17265 handle_command_reply: tid=0 2024-04-25T02:08:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.978+0000 7f84c2e50740 1 client.17265 shutdown 2024-04-25T02:08:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.978+0000 7f84c2e50740 1 -- 172.21.15.31:0/3947525984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 msgr2=0x55da66a3e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.978+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a3e1d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f84b400c760 tx=0x7f84b4005790 comp rx=0 tx=0).stop 2024-04-25T02:08:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.978+0000 7f84c2e50740 1 -- 172.21.15.31:0/3947525984 shutdown_connections 2024-04-25T02:08:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.978+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55da66a51740 0x55da66a53be0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.978+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a41230 0x55da66a3ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.978+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a3e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.979+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3947525984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a3e1d0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.979+0000 7f84c2e50740 1 -- 172.21.15.31:0/3947525984 >> 172.21.15.31:0/3947525984 conn(0x55da66a32560 msgr2=0x55da66a38360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.979+0000 7f84a17fa640 0 client.17265 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.979+0000 7f84c2e50740 1 -- 172.21.15.31:0/3947525984 shutdown_connections 2024-04-25T02:08:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.979+0000 7f84c2e50740 1 -- 172.21.15.31:0/3947525984 wait complete. 2024-04-25T02:08:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:37.979+0000 7f84c2e50740 20 client.17265 trim_cache size 0 max 16384 2024-04-25T02:08:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.000+0000 7f84c2e50740 1 Processor -- start 2024-04-25T02:08:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84c2e50740 1 -- start start 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a4cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36b60 0x55da66a40890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a41230 0x55da66a3dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84c2e50740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f84b81af4b0 con 0x55da66a36240 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84c2e50740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f84b81b1170 con 0x55da66a36b60 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84c2e50740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f84b81afb80 con 0x55da66a41230 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84bdede640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a4cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84bdede640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a4cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50544/0 (socket says 172.21.15.31:50544) 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84bdede640 1 -- 172.21.15.31:0/3654860486 learned_addr learned my addr 172.21.15.31:0/3654860486 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84be6df640 1 --2- 172.21.15.31:0/3654860486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a41230 0x55da66a3dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84bd6dd640 1 --2- 172.21.15.31:0/3654860486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36b60 0x55da66a40890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84bdede640 1 -- 172.21.15.31:0/3654860486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a41230 msgr2=0x55da66a3dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84bdede640 1 --2- 172.21.15.31:0/3654860486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a41230 0x55da66a3dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84bdede640 1 -- 172.21.15.31:0/3654860486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36b60 msgr2=0x55da66a40890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84bdede640 1 --2- 172.21.15.31:0/3654860486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36b60 0x55da66a40890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.001+0000 7f84bdede640 1 -- 172.21.15.31:0/3654860486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da66a44140 con 0x55da66a36240 2024-04-25T02:08:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.002+0000 7f84bdede640 1 --2- 172.21.15.31:0/3654860486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a4cee0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009920 tx=0x7f84b409a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c277728d562cb62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.002+0000 7f84a37fe640 1 -- 172.21.15.31:0/3654860486 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b40a48c0 con 0x55da66a36240 2024-04-25T02:08:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.002+0000 7f84a37fe640 1 -- 172.21.15.31:0/3654860486 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84b40ca030 con 0x55da66a36240 2024-04-25T02:08:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.002+0000 7f84a37fe640 1 -- 172.21.15.31:0/3654860486 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b409ffb0 con 0x55da66a36240 2024-04-25T02:08:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.003+0000 7f84c2e50740 1 -- 172.21.15.31:0/3654860486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 msgr2=0x55da66a4cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.003+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3654860486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a4cee0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009920 tx=0x7f84b409a110 comp rx=0 tx=0).stop 2024-04-25T02:08:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.003+0000 7f84c2e50740 1 -- 172.21.15.31:0/3654860486 shutdown_connections 2024-04-25T02:08:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.003+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3654860486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a41230 0x55da66a3dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.003+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3654860486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36b60 0x55da66a40890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.003+0000 7f84c2e50740 1 --2- 172.21.15.31:0/3654860486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a36240 0x55da66a4cee0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.003+0000 7f84c2e50740 1 -- 172.21.15.31:0/3654860486 >> 172.21.15.31:0/3654860486 conn(0x55da66a32560 msgr2=0x55da66a315c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.003+0000 7f84c2e50740 1 -- 172.21.15.31:0/3654860486 shutdown_connections 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.003+0000 7f84c2e50740 1 -- 172.21.15.31:0/3654860486 wait complete. 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.003+0000 7f84c2e50740 1 -- start start 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.003+0000 7f84c2e50740 1 Processor -- start 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36240 0x55da66a44db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a452f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84c2e50740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a41230 0x55da66a45830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84c2e50740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f84b81b0aa0 con 0x55da66a41230 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84c2e50740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f84b81b0310 con 0x55da66a36240 2024-04-25T02:08:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84c2e50740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f84b81ad4b0 con 0x55da66a36b60 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84c2e50740 10 client.0 _ll_register_callbacks cb 0x7f84b4004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84bdede640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36240 0x55da66a44db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84bd6dd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84bdede640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36240 0x55da66a44db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45442/0 (socket says 172.21.15.31:45442) 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84bdede640 1 -- 172.21.15.31:0/122585132 learned_addr learned my addr 172.21.15.31:0/122585132 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84bd6dd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a452f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:57982/0 (socket says 172.21.15.31:57982) 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84be6df640 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a41230 0x55da66a45830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84bdede640 1 -- 172.21.15.31:0/122585132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 msgr2=0x55da66a452f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84bdede640 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a452f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84bdede640 1 -- 172.21.15.31:0/122585132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a41230 msgr2=0x55da66a45830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84bdede640 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a41230 0x55da66a45830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.004+0000 7f84bdede640 1 -- 172.21.15.31:0/122585132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da66ab6d60 con 0x55da66a36240 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.005+0000 7f84bdede640 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36240 0x55da66a44db0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f84b401b040 tx=0x7f84b409bc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b57193709149f753 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.005+0000 7f84a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:08:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.005+0000 7f84a17fa640 1 -- 172.21.15.31:0/122585132 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b40a48c0 con 0x55da66a36240 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.005+0000 7f84a17fa640 1 -- 172.21.15.31:0/122585132 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8484+0+0 (secure 0 0 0) 0x7f84b40a4480 con 0x55da66a36240 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.005+0000 7f84c2e50740 1 -- 172.21.15.31:0/122585132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84b40ca030 con 0x55da66a36240 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.005+0000 7f84a17fa640 1 -- 172.21.15.31:0/122585132 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f84b4097ea0 con 0x55da66a36240 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.005+0000 7f84c2e50740 10 client.26843 fetch_fsmap learned FSMap version 42 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.005+0000 7f84c2e50740 1 -- 172.21.15.31:0/122585132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55da66ab7fb0 con 0x55da66a36240 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.005+0000 7f84a17fa640 1 -- 172.21.15.31:0/122585132 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84b40ca030 con 0x55da66a36240 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.006+0000 7f84a17fa640 1 -- 172.21.15.31:0/122585132 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) ==== 8394+0+0 (secure 0 0 0) 0x7f84b40c2070 con 0x55da66a36240 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.006+0000 7f84c2e50740 10 client.26843 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.006+0000 7f84c2e50740 10 client.26843 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.006+0000 7f84c2e50740 20 client.26843 populate_metadata read hostname 'smithi031' 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.006+0000 7f84c2e50740 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55da66ab8530 0x55da66aba950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.006+0000 7f84c2e50740 4 client.26843 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.006+0000 7f84c2e50740 1 -- 172.21.15.31:0/122585132 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55da669a81f0 con 0x55da66ab8530 2024-04-25T02:08:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.006+0000 7f84bd6dd640 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55da66ab8530 0x55da66aba950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:38.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.006+0000 7f84bd6dd640 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55da66ab8530 0x55da66aba950 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=718792593117e45 server_cookie=3713255737a54d96 in_seq=0 out_seq=0 2024-04-25T02:08:38.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.006+0000 7f84a17fa640 10 client.26843 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84a17fa640 1 -- 172.21.15.31:0/122585132 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55da669a81f0 con 0x55da66ab8530 2024-04-25T02:08:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84a17fa640 10 client.26843 handle_command_reply: tid=0 2024-04-25T02:08:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84c2e50740 1 client.26843 shutdown 2024-04-25T02:08:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84c2e50740 1 -- 172.21.15.31:0/122585132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36240 msgr2=0x55da66a44db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84c2e50740 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36240 0x55da66a44db0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f84b401b040 tx=0x7f84b409bc00 comp rx=0 tx=0).stop 2024-04-25T02:08:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84c2e50740 1 -- 172.21.15.31:0/122585132 shutdown_connections 2024-04-25T02:08:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84c2e50740 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55da66ab8530 0x55da66aba950 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84c2e50740 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55da66a41230 0x55da66a45830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84c2e50740 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55da66a36b60 0x55da66a452f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84c2e50740 1 --2- 172.21.15.31:0/122585132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da66a36240 0x55da66a44db0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84a17fa640 0 client.26843 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84c2e50740 1 -- 172.21.15.31:0/122585132 >> 172.21.15.31:0/122585132 conn(0x55da66a32560 msgr2=0x55da66a37aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84c2e50740 1 -- 172.21.15.31:0/122585132 shutdown_connections 2024-04-25T02:08:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.045+0000 7f84c2e50740 1 -- 172.21.15.31:0/122585132 wait complete. 2024-04-25T02:08:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.046+0000 7f84c2e50740 20 client.26843 trim_cache size 0 max 16384 2024-04-25T02:08:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:08:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:08:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:08:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:08:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.046+0000 7f84c0142640 1 -- 172.21.15.31:0/2223858783 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f84a40a2da0 msgr2=0x7f84a40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.046+0000 7f84c0142640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f84a40a2da0 0x7f84a40a5260 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f84b0006010 tx=0x7f84b0005f80 comp rx=0 tx=0).stop 2024-04-25T02:08:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.046+0000 7f84c0142640 1 -- 172.21.15.31:0/2223858783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b8102930 msgr2=0x7f84b8197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.046+0000 7f84c0142640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b8102930 0x7f84b8197e40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f84b400ea60 tx=0x7f84b400ef20 comp rx=0 tx=0).stop 2024-04-25T02:08:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.046+0000 7f84c0142640 1 -- 172.21.15.31:0/2223858783 shutdown_connections 2024-04-25T02:08:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.046+0000 7f84c0142640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f84a40a2da0 0x7f84a40a5260 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.046+0000 7f84c0142640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84b810bd00 0x7f84b819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.046+0000 7f84c0142640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84b8103250 0x7f84b8198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.047+0000 7f84c0142640 1 --2- 172.21.15.31:0/2223858783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b8102930 0x7f84b8197e40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.047+0000 7f84c0142640 1 -- 172.21.15.31:0/2223858783 >> 172.21.15.31:0/2223858783 conn(0x7f84b80fc110 msgr2=0x7f84b80fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.047+0000 7f84c0142640 1 -- 172.21.15.31:0/2223858783 shutdown_connections 2024-04-25T02:08:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:38.047+0000 7f84c0142640 1 -- 172.21.15.31:0/2223858783 wait complete. 2024-04-25T02:08:38.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 6651 12.80 MB/sec warmup 42 sec latency 57.502 ms 2024-04-25T02:08:38.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:37 smithi134 ceph-mon[33098]: pgmap v442: 97 pgs: 97 active+clean; 391 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.7 MiB/s wr, 259 op/s 2024-04-25T02:08:38.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:37 smithi031 ceph-mon[29204]: pgmap v442: 97 pgs: 97 active+clean; 391 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.7 MiB/s wr, 259 op/s 2024-04-25T02:08:38.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:37 smithi040 ceph-mon[32418]: pgmap v442: 97 pgs: 97 active+clean; 391 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.7 MiB/s wr, 259 op/s 2024-04-25T02:08:39.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 6836 12.80 MB/sec warmup 43 sec latency 102.206 ms 2024-04-25T02:08:40.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 6966 12.64 MB/sec warmup 44 sec latency 54.719 ms 2024-04-25T02:08:40.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:39 smithi134 ceph-mon[33098]: pgmap v443: 97 pgs: 97 active+clean; 391 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.7 MiB/s wr, 259 op/s 2024-04-25T02:08:40.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:39 smithi031 ceph-mon[29204]: pgmap v443: 97 pgs: 97 active+clean; 391 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.7 MiB/s wr, 259 op/s 2024-04-25T02:08:40.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:39 smithi040 ceph-mon[32418]: pgmap v443: 97 pgs: 97 active+clean; 391 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.7 MiB/s wr, 259 op/s 2024-04-25T02:08:41.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 7121 12.60 MB/sec warmup 45 sec latency 53.381 ms 2024-04-25T02:08:42.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 7265 12.40 MB/sec warmup 46 sec latency 51.958 ms 2024-04-25T02:08:42.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:41 smithi134 ceph-mon[33098]: pgmap v444: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.3 MiB/s wr, 378 op/s 2024-04-25T02:08:42.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:41 smithi031 ceph-mon[29204]: pgmap v444: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.3 MiB/s wr, 378 op/s 2024-04-25T02:08:42.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:41 smithi040 ceph-mon[32418]: pgmap v444: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.3 MiB/s wr, 378 op/s 2024-04-25T02:08:43.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 7430 12.25 MB/sec warmup 47 sec latency 74.916 ms 2024-04-25T02:08:44.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 7579 12.08 MB/sec warmup 48 sec latency 80.894 ms 2024-04-25T02:08:44.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:43 smithi134 ceph-mon[33098]: pgmap v445: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 4.6 MiB/s wr, 228 op/s 2024-04-25T02:08:44.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:43 smithi031 ceph-mon[29204]: pgmap v445: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 4.6 MiB/s wr, 228 op/s 2024-04-25T02:08:44.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:43 smithi040 ceph-mon[32418]: pgmap v445: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 4.6 MiB/s wr, 228 op/s 2024-04-25T02:08:45.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 7741 12.02 MB/sec warmup 49 sec latency 56.582 ms 2024-04-25T02:08:46.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 7923 12.01 MB/sec warmup 50 sec latency 64.606 ms 2024-04-25T02:08:46.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:45 smithi134 ceph-mon[33098]: pgmap v446: 97 pgs: 97 active+clean; 412 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.6 MiB/s wr, 347 op/s 2024-04-25T02:08:46.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:45 smithi031 ceph-mon[29204]: pgmap v446: 97 pgs: 97 active+clean; 412 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.6 MiB/s wr, 347 op/s 2024-04-25T02:08:46.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:45 smithi040 ceph-mon[32418]: pgmap v446: 97 pgs: 97 active+clean; 412 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.6 MiB/s wr, 347 op/s 2024-04-25T02:08:47.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 8121 12.06 MB/sec warmup 51 sec latency 59.812 ms 2024-04-25T02:08:48.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 8269 11.92 MB/sec warmup 52 sec latency 90.537 ms 2024-04-25T02:08:48.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:47 smithi134 ceph-mon[33098]: pgmap v447: 97 pgs: 97 active+clean; 412 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 237 op/s 2024-04-25T02:08:48.217 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:47 smithi040 ceph-mon[32418]: pgmap v447: 97 pgs: 97 active+clean; 412 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 237 op/s 2024-04-25T02:08:48.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:47 smithi031 ceph-mon[29204]: pgmap v447: 97 pgs: 97 active+clean; 412 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 237 op/s 2024-04-25T02:08:49.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 8417 11.74 MB/sec warmup 53 sec latency 110.019 ms 2024-04-25T02:08:50.090 INFO:tasks.workunit.client.0.smithi031.stdout: 10 8570 11.62 MB/sec warmup 54 sec latency 145.656 ms 2024-04-25T02:08:50.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:49 smithi134 ceph-mon[33098]: pgmap v448: 97 pgs: 97 active+clean; 412 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 237 op/s 2024-04-25T02:08:50.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:49 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:08:50.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:49 smithi031 ceph-mon[29204]: pgmap v448: 97 pgs: 97 active+clean; 412 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 237 op/s 2024-04-25T02:08:50.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:49 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:08:50.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:49 smithi040 ceph-mon[32418]: pgmap v448: 97 pgs: 97 active+clean; 412 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 237 op/s 2024-04-25T02:08:50.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:49 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:08:51.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 8748 11.52 MB/sec warmup 55 sec latency 62.929 ms 2024-04-25T02:08:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.181+0000 7f017a84d640 1 -- 172.21.15.31:0/190207790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0174103960 msgr2=0x7f0174103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.181+0000 7f017a84d640 1 --2- 172.21.15.31:0/190207790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0174103960 0x7f0174103de0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0168009900 tx=0x7f016802a410 comp rx=0 tx=0).stop 2024-04-25T02:08:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.181+0000 7f017a84d640 1 -- 172.21.15.31:0/190207790 shutdown_connections 2024-04-25T02:08:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.181+0000 7f017a84d640 1 --2- 172.21.15.31:0/190207790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0174104320 0x7f0174110830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.181+0000 7f017a84d640 1 --2- 172.21.15.31:0/190207790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0174103960 0x7f0174103de0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.181+0000 7f017a84d640 1 --2- 172.21.15.31:0/190207790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0174109970 0x7f0174109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.181+0000 7f017a84d640 1 -- 172.21.15.31:0/190207790 >> 172.21.15.31:0/190207790 conn(0x7f01740ff450 msgr2=0x7f0174101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.182+0000 7f017a84d640 1 -- 172.21.15.31:0/190207790 shutdown_connections 2024-04-25T02:08:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.182+0000 7f017a84d640 1 -- 172.21.15.31:0/190207790 wait complete. 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.182+0000 7f017a84d640 1 Processor -- start 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f017a84d640 1 -- start start 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f017a84d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0174103960 0x7f017419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f017a84d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0174104320 0x7f017419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f017a84d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0174109970 0x7f017419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f017a84d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0174113c80 con 0x7f0174109970 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f017a84d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0174113b00 con 0x7f0174104320 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f017a84d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0174113e00 con 0x7f0174103960 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f0178dea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0174109970 0x7f017419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f0173fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0174103960 0x7f017419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f0173fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0174103960 0x7f017419a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:56526/0 (socket says 172.21.15.31:56526) 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f01737fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0174104320 0x7f017419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.183+0000 7f0173fff640 1 -- 172.21.15.31:0/4012531774 learned_addr learned my addr 172.21.15.31:0/4012531774 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f0178dea640 1 -- 172.21.15.31:0/4012531774 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0174103960 msgr2=0x7f017419a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f0178dea640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0174103960 0x7f017419a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f0178dea640 1 -- 172.21.15.31:0/4012531774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0174104320 msgr2=0x7f017419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f0178dea640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0174104320 0x7f017419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f0178dea640 1 -- 172.21.15.31:0/4012531774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017419f0e0 con 0x7f0174109970 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f01737fe640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0174104320 0x7f017419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f0178dea640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0174109970 0x7f017419e9b0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f016c009fd0 tx=0x7f016c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f01717fa640 1 -- 172.21.15.31:0/4012531774 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c00ce20 con 0x7f0174109970 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f01717fa640 1 -- 172.21.15.31:0/4012531774 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f016c005020 con 0x7f0174109970 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f01717fa640 1 -- 172.21.15.31:0/4012531774 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c010690 con 0x7f0174109970 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f017a84d640 1 -- 172.21.15.31:0/4012531774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f017419f3d0 con 0x7f0174109970 2024-04-25T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.184+0000 7f017a84d640 1 -- 172.21.15.31:0/4012531774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01741a6c60 con 0x7f0174109970 2024-04-25T02:08:51.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.187+0000 7f01717fa640 1 -- 172.21.15.31:0/4012531774 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f016c0051d0 con 0x7f0174109970 2024-04-25T02:08:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.188+0000 7f017d636740 1 Processor -- start 2024-04-25T02:08:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.188+0000 7f01717fa640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f015c0a2e20 0x7f015c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.188+0000 7f017d636740 1 -- start start 2024-04-25T02:08:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.188+0000 7f01737fe640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f015c0a2e20 0x7f015c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.188+0000 7f017d636740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 0x55c25e9a8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.188+0000 7f01717fa640 1 -- 172.21.15.31:0/4012531774 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8484+0+0 (secure 0 0 0) 0x7f016c0cfa30 con 0x7f0174109970 2024-04-25T02:08:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.188+0000 7f017d636740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 0x55c25e9bb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f017d636740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9bba40 0x55c25e9bde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f017d636740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c25e90ccb0 con 0x55c25e9bba40 2024-04-25T02:08:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f017d636740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c25e8b55b0 con 0x55c25e9a8940 2024-04-25T02:08:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f017d636740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c25e908c70 con 0x55c25e9a7f90 2024-04-25T02:08:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f0173fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 0x55c25e9a8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f01737fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 0x55c25e9bb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f0173fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 0x55c25e9a8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:56540/0 (socket says 172.21.15.31:56540) 2024-04-25T02:08:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f01737fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 0x55c25e9bb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58606/0 (socket says 172.21.15.31:58606) 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f0173fff640 1 -- 172.21.15.31:0/2782390908 learned_addr learned my addr 172.21.15.31:0/2782390908 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f0178dea640 1 --2- 172.21.15.31:0/2782390908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9bba40 0x55c25e9bde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f01737fe640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f015c0a2e20 0x7f015c0a52e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0168000c00 tx=0x7f0168004550 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f0173fff640 1 -- 172.21.15.31:0/2782390908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 msgr2=0x55c25e9bb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f0173fff640 1 --2- 172.21.15.31:0/2782390908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 0x55c25e9bb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f0173fff640 1 -- 172.21.15.31:0/2782390908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9bba40 msgr2=0x55c25e9bde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f0173fff640 1 --2- 172.21.15.31:0/2782390908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9bba40 0x55c25e9bde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f0173fff640 1 -- 172.21.15.31:0/2782390908 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c25e621210 con 0x55c25e9a7f90 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f0178dea640 1 --2- 172.21.15.31:0/2782390908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9bba40 0x55c25e9bde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.189+0000 7f01737fe640 1 --2- 172.21.15.31:0/2782390908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 0x55c25e9bb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.190+0000 7f0173fff640 1 --2- 172.21.15.31:0/2782390908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 0x55c25e9a8370 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0164009d90 tx=0x7f0164011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=397da006e734954b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.190+0000 7f015a7fc640 1 -- 172.21.15.31:0/2782390908 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016401e070 con 0x55c25e9a7f90 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.190+0000 7f015a7fc640 1 -- 172.21.15.31:0/2782390908 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0164019ae0 con 0x55c25e9a7f90 2024-04-25T02:08:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.190+0000 7f015a7fc640 1 -- 172.21.15.31:0/2782390908 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016400fb20 con 0x55c25e9a7f90 2024-04-25T02:08:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.191+0000 7f017d636740 1 -- 172.21.15.31:0/2782390908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 msgr2=0x55c25e9a8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.191+0000 7f017d636740 1 --2- 172.21.15.31:0/2782390908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 0x55c25e9a8370 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0164009d90 tx=0x7f0164011810 comp rx=0 tx=0).stop 2024-04-25T02:08:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.191+0000 7f017d636740 1 -- 172.21.15.31:0/2782390908 shutdown_connections 2024-04-25T02:08:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.191+0000 7f017d636740 1 --2- 172.21.15.31:0/2782390908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9bba40 0x55c25e9bde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.191+0000 7f017d636740 1 --2- 172.21.15.31:0/2782390908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 0x55c25e9bb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.191+0000 7f017d636740 1 --2- 172.21.15.31:0/2782390908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 0x55c25e9a8370 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.191+0000 7f017d636740 1 -- 172.21.15.31:0/2782390908 >> 172.21.15.31:0/2782390908 conn(0x55c25e9a4560 msgr2=0x55c25e9a4940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.191+0000 7f017d636740 1 -- 172.21.15.31:0/2782390908 shutdown_connections 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.191+0000 7f017d636740 1 -- 172.21.15.31:0/2782390908 wait complete. 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.191+0000 7f017d636740 1 -- start start 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.191+0000 7f017d636740 1 Processor -- start 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f017d636740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 0x55c25e9b89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f017d636740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 0x55c25e9b8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f017d636740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9bba40 0x55c25e9b9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f017d636740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c25e9be800 con 0x55c25e9a7f90 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f017d636740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c25e8c8b50 con 0x55c25e9a8940 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f017d636740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c25e9be980 con 0x55c25e9bba40 2024-04-25T02:08:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f017d636740 10 client.0 _ll_register_callbacks cb 0x55c25e903ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f0173fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 0x55c25e9b89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f0178dea640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9bba40 0x55c25e9b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f0173fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 0x55c25e9b89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46634/0 (socket says 172.21.15.31:46634) 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f0173fff640 1 -- 172.21.15.31:0/3144750429 learned_addr learned my addr 172.21.15.31:0/3144750429 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f0178dea640 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9bba40 0x55c25e9b9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:56542/0 (socket says 172.21.15.31:56542) 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f01737fe640 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 0x55c25e9b8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f0178dea640 1 -- 172.21.15.31:0/3144750429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 msgr2=0x55c25e9b8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f0178dea640 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 0x55c25e9b8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f0178dea640 1 -- 172.21.15.31:0/3144750429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 msgr2=0x55c25e9b89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f0178dea640 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 0x55c25e9b89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f0178dea640 1 -- 172.21.15.31:0/3144750429 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c25e6ed620 con 0x55c25e9bba40 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f0178dea640 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9bba40 0x55c25e9b9440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f016c0cd1a0 tx=0x7f016c09c030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa2d2de5ef8fc93d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.192+0000 7f01537fe640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:08:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f01537fe640 1 -- 172.21.15.31:0/3144750429 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c0b6020 con 0x55c25e9bba40 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f017d636740 1 -- 172.21.15.31:0/3144750429 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0164019ae0 con 0x55c25e9bba40 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f01537fe640 1 -- 172.21.15.31:0/3144750429 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8484+0+0 (secure 0 0 0) 0x7f016c0a4750 con 0x55c25e9bba40 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f01537fe640 1 -- 172.21.15.31:0/3144750429 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f016c081050 con 0x55c25e9bba40 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f017d636740 10 client.26638 fetch_fsmap learned FSMap version 42 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f017d636740 1 -- 172.21.15.31:0/3144750429 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55c25e9caa80 con 0x55c25e9bba40 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f01537fe640 1 -- 172.21.15.31:0/3144750429 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0164019ae0 con 0x55c25e9bba40 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f01537fe640 1 -- 172.21.15.31:0/3144750429 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 42) ==== 8394+0+0 (secure 0 0 0) 0x7f016c096ae0 con 0x55c25e9bba40 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f017d636740 10 client.26638 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f017d636740 10 client.26638 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f017d636740 20 client.26638 populate_metadata read hostname 'smithi031' 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f017d636740 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c25e9cc450 0x55c25e9ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f017d636740 4 client.26638 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.193+0000 7f017d636740 1 -- 172.21.15.31:0/3144750429 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c25e91a1f0 con 0x55c25e9cc450 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.194+0000 7f0173fff640 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c25e9cc450 0x55c25e9ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.194+0000 7f01537fe640 10 client.26638 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.194+0000 7f0173fff640 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c25e9cc450 0x55c25e9ce890 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9132656442761d08 server_cookie=e730639a96f89970 in_seq=0 out_seq=0 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f01537fe640 1 -- 172.21.15.31:0/3144750429 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c25e91a1f0 con 0x55c25e9cc450 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f01537fe640 10 client.26638 handle_command_reply: tid=0 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f017d636740 1 client.26638 shutdown 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f017d636740 1 -- 172.21.15.31:0/3144750429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9bba40 msgr2=0x55c25e9b9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f017d636740 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9bba40 0x55c25e9b9440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f016c0cd1a0 tx=0x7f016c09c030 comp rx=0 tx=0).stop 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f017d636740 1 -- 172.21.15.31:0/3144750429 shutdown_connections 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f017d636740 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c25e9cc450 0x55c25e9ce890 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f017d636740 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9bba40 0x55c25e9b9440 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f017d636740 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9a8940 0x55c25e9b8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f01537fe640 0 client.26638 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f017d636740 1 --2- 172.21.15.31:0/3144750429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 0x55c25e9b89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f017d636740 1 -- 172.21.15.31:0/3144750429 >> 172.21.15.31:0/3144750429 conn(0x55c25e9a4560 msgr2=0x55c25e9aa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f017d636740 1 -- 172.21.15.31:0/3144750429 shutdown_connections 2024-04-25T02:08:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.195+0000 7f017d636740 1 -- 172.21.15.31:0/3144750429 wait complete. 2024-04-25T02:08:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.196+0000 7f017d636740 20 client.26638 trim_cache size 0 max 16384 2024-04-25T02:08:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.224+0000 7f017d636740 1 Processor -- start 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.224+0000 7f017d636740 1 -- start start 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.224+0000 7f017d636740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 0x55c25e9bb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.224+0000 7f017d636740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a8940 0x55c25e9c8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.224+0000 7f017d636740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 0x55c25e9c7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.224+0000 7f017d636740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f01741ab100 con 0x55c25e9a8940 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.224+0000 7f017d636740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01741acdc0 con 0x55c25e9bba40 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.224+0000 7f017d636740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f01741ab7d0 con 0x55c25e9a7f90 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.224+0000 7f01737fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a8940 0x55c25e9c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f01737fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a8940 0x55c25e9c8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46648/0 (socket says 172.21.15.31:46648) 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f01737fe640 1 -- 172.21.15.31:0/2828185193 learned_addr learned my addr 172.21.15.31:0/2828185193 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f0173fff640 1 --2- 172.21.15.31:0/2828185193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 0x55c25e9bb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f0178dea640 1 --2- 172.21.15.31:0/2828185193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 0x55c25e9c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f0178dea640 1 -- 172.21.15.31:0/2828185193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 msgr2=0x55c25e9bb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f0178dea640 1 --2- 172.21.15.31:0/2828185193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 0x55c25e9bb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f0178dea640 1 -- 172.21.15.31:0/2828185193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a8940 msgr2=0x55c25e9c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f0178dea640 1 --2- 172.21.15.31:0/2828185193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a8940 0x55c25e9c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f0178dea640 1 -- 172.21.15.31:0/2828185193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c25e9b7970 con 0x55c25e9bba40 2024-04-25T02:08:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f01737fe640 1 --2- 172.21.15.31:0/2828185193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a8940 0x55c25e9c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f0178dea640 1 --2- 172.21.15.31:0/2828185193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 0x55c25e9c7700 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f016c09a220 tx=0x7f016c0a4d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9f05a5b1db96acf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.225+0000 7f0159ffb640 1 -- 172.21.15.31:0/2828185193 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c0b6040 con 0x55c25e9bba40 2024-04-25T02:08:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.226+0000 7f0159ffb640 1 -- 172.21.15.31:0/2828185193 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f016c096ae0 con 0x55c25e9bba40 2024-04-25T02:08:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.226+0000 7f0159ffb640 1 -- 172.21.15.31:0/2828185193 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c0b4070 con 0x55c25e9bba40 2024-04-25T02:08:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.226+0000 7f017d636740 1 -- 172.21.15.31:0/2828185193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 msgr2=0x55c25e9c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.226+0000 7f017d636740 1 --2- 172.21.15.31:0/2828185193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 0x55c25e9c7700 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f016c09a220 tx=0x7f016c0a4d60 comp rx=0 tx=0).stop 2024-04-25T02:08:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.227+0000 7f017d636740 1 -- 172.21.15.31:0/2828185193 shutdown_connections 2024-04-25T02:08:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.227+0000 7f017d636740 1 --2- 172.21.15.31:0/2828185193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 0x55c25e9c7700 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.227+0000 7f017d636740 1 --2- 172.21.15.31:0/2828185193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a8940 0x55c25e9c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.227+0000 7f017d636740 1 --2- 172.21.15.31:0/2828185193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a7f90 0x55c25e9bb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.227+0000 7f017d636740 1 -- 172.21.15.31:0/2828185193 >> 172.21.15.31:0/2828185193 conn(0x55c25e9a4560 msgr2=0x55c25e9a35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.227+0000 7f017d636740 1 -- 172.21.15.31:0/2828185193 shutdown_connections 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.227+0000 7f017d636740 1 -- 172.21.15.31:0/2828185193 wait complete. 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.227+0000 7f017d636740 1 -- start start 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f017d636740 1 Processor -- start 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f017d636740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 0x55c25e9c4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f017d636740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a8940 0x55c25e9b5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f017d636740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 0x55c25e9b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f017d636740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f01741ac6f0 con 0x55c25e9a7f90 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f0173fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 0x55c25e9c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f0173fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 0x55c25e9c4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46664/0 (socket says 172.21.15.31:46664) 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f0173fff640 1 -- 172.21.15.31:0/1015279999 learned_addr learned my addr 172.21.15.31:0/1015279999 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f01737fe640 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a8940 0x55c25e9b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f0178dea640 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 0x55c25e9b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f017d636740 1 -- 172.21.15.31:0/1015279999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01741abf60 con 0x55c25e9bba40 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f017d636740 1 -- 172.21.15.31:0/1015279999 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f01741a9100 con 0x55c25e9a8940 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.228+0000 7f017d636740 10 client.0 _ll_register_callbacks cb 0x7f016c005bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f0173fff640 1 -- 172.21.15.31:0/1015279999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a8940 msgr2=0x55c25e9b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f0173fff640 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a8940 0x55c25e9b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f0173fff640 1 -- 172.21.15.31:0/1015279999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 msgr2=0x55c25e9b5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f0173fff640 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 0x55c25e9b5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f0173fff640 1 -- 172.21.15.31:0/1015279999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c25ea31a10 con 0x55c25e9a7f90 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f01737fe640 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a8940 0x55c25e9b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f0178dea640 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 0x55c25e9b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f0173fff640 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 0x55c25e9c4130 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0164009b50 tx=0x7f016400f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ce27c737f3f2bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f01537fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:08:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f01537fe640 1 -- 172.21.15.31:0/1015279999 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016402f060 con 0x55c25e9a7f90 2024-04-25T02:08:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f01537fe640 1 -- 172.21.15.31:0/1015279999 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8484+0+0 (secure 0 0 0) 0x7f0164005310 con 0x55c25e9a7f90 2024-04-25T02:08:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.229+0000 7f017d636740 1 -- 172.21.15.31:0/1015279999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f016c096ae0 con 0x55c25e9a7f90 2024-04-25T02:08:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.230+0000 7f01537fe640 1 -- 172.21.15.31:0/1015279999 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f0164029e60 con 0x55c25e9a7f90 2024-04-25T02:08:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.230+0000 7f017d636740 10 client.17289 fetch_fsmap learned FSMap version 42 2024-04-25T02:08:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.230+0000 7f017d636740 1 -- 172.21.15.31:0/1015279999 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55c25ea32b10 con 0x55c25e9a7f90 2024-04-25T02:08:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.231+0000 7f01537fe640 1 -- 172.21.15.31:0/1015279999 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f016c096ae0 con 0x55c25e9a7f90 2024-04-25T02:08:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.231+0000 7f01537fe640 1 -- 172.21.15.31:0/1015279999 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 42) ==== 8394+0+0 (secure 0 0 0) 0x7f0164029170 con 0x55c25e9a7f90 2024-04-25T02:08:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.231+0000 7f017d636740 10 client.17289 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T02:08:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.231+0000 7f017d636740 10 client.17289 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.231+0000 7f017d636740 20 client.17289 populate_metadata read hostname 'smithi031' 2024-04-25T02:08:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.231+0000 7f017d636740 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c25ea33180 0x55c25ea355a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.231+0000 7f017d636740 4 client.17289 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:08:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.231+0000 7f017d636740 1 -- 172.21.15.31:0/1015279999 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c25e914540 con 0x55c25ea33180 2024-04-25T02:08:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.231+0000 7f01737fe640 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c25ea33180 0x55c25ea355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.232+0000 7f01537fe640 10 client.17289 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.232+0000 7f01737fe640 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c25ea33180 0x55c25ea355a0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81e2dc3ca248efc3 server_cookie=2103fcb7957998b5 in_seq=0 out_seq=0 2024-04-25T02:08:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.278+0000 7f01537fe640 1 -- 172.21.15.31:0/1015279999 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c25e914540 con 0x55c25ea33180 2024-04-25T02:08:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.278+0000 7f01537fe640 10 client.17289 handle_command_reply: tid=0 2024-04-25T02:08:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.278+0000 7f017d636740 1 client.17289 shutdown 2024-04-25T02:08:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f017d636740 1 -- 172.21.15.31:0/1015279999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 msgr2=0x55c25e9c4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f017d636740 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 0x55c25e9c4130 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0164009b50 tx=0x7f016400f490 comp rx=0 tx=0).stop 2024-04-25T02:08:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f017d636740 1 -- 172.21.15.31:0/1015279999 shutdown_connections 2024-04-25T02:08:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f017d636740 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c25ea33180 0x55c25ea355a0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f017d636740 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e9bba40 0x55c25e9b5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f017d636740 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c25e9a8940 0x55c25e9b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f017d636740 1 --2- 172.21.15.31:0/1015279999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c25e9a7f90 0x55c25e9c4130 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f01537fe640 0 client.17289 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:08:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f017d636740 1 -- 172.21.15.31:0/1015279999 >> 172.21.15.31:0/1015279999 conn(0x55c25e9a4560 msgr2=0x55c25e9a35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f017d636740 1 -- 172.21.15.31:0/1015279999 shutdown_connections 2024-04-25T02:08:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f017d636740 1 -- 172.21.15.31:0/1015279999 wait complete. 2024-04-25T02:08:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.279+0000 7f017d636740 20 client.17289 trim_cache size 0 max 16384 2024-04-25T02:08:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:08:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:08:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:08:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:08:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 -- 172.21.15.31:0/4012531774 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f015c0a2e20 msgr2=0x7f015c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f015c0a2e20 0x7f015c0a52e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0168000c00 tx=0x7f0168004550 comp rx=0 tx=0).stop 2024-04-25T02:08:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 -- 172.21.15.31:0/4012531774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0174109970 msgr2=0x7f017419e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0174109970 0x7f017419e9b0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f016c009fd0 tx=0x7f016c00ef10 comp rx=0 tx=0).stop 2024-04-25T02:08:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 -- 172.21.15.31:0/4012531774 shutdown_connections 2024-04-25T02:08:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f015c0a2e20 0x7f015c0a52e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0174109970 0x7f017419e9b0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0174104320 0x7f017419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 --2- 172.21.15.31:0/4012531774 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0174103960 0x7f017419a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 -- 172.21.15.31:0/4012531774 >> 172.21.15.31:0/4012531774 conn(0x7f01740ff450 msgr2=0x7f01740ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 -- 172.21.15.31:0/4012531774 shutdown_connections 2024-04-25T02:08:51.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:08:51.280+0000 7f017a84d640 1 -- 172.21.15.31:0/4012531774 wait complete. 2024-04-25T02:08:52.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 8920 11.48 MB/sec warmup 56 sec latency 69.370 ms 2024-04-25T02:08:52.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:51 smithi134 ceph-mon[33098]: pgmap v449: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.1 MiB/s wr, 365 op/s 2024-04-25T02:08:52.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:51 smithi031 ceph-mon[29204]: pgmap v449: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.1 MiB/s wr, 365 op/s 2024-04-25T02:08:52.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:51 smithi040 ceph-mon[32418]: pgmap v449: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.1 MiB/s wr, 365 op/s 2024-04-25T02:08:53.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 9076 11.38 MB/sec warmup 57 sec latency 57.462 ms 2024-04-25T02:08:54.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 9258 11.38 MB/sec warmup 58 sec latency 94.838 ms 2024-04-25T02:08:54.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:53 smithi031 ceph-mon[29204]: pgmap v450: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 247 op/s 2024-04-25T02:08:54.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:53 smithi040 ceph-mon[32418]: pgmap v450: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 247 op/s 2024-04-25T02:08:54.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:53 smithi134 ceph-mon[33098]: pgmap v450: 97 pgs: 97 active+clean; 405 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 247 op/s 2024-04-25T02:08:55.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 9503 11.54 MB/sec warmup 59 sec latency 143.463 ms 2024-04-25T02:08:56.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 9724 11.53 MB/sec warmup 60 sec latency 162.584 ms 2024-04-25T02:08:56.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:55 smithi031 ceph-mon[29204]: pgmap v451: 97 pgs: 97 active+clean; 411 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 361 op/s 2024-04-25T02:08:56.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:55 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:08:56.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:55 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:08:56.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:55 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:08:56.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:55 smithi040 ceph-mon[32418]: pgmap v451: 97 pgs: 97 active+clean; 411 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 361 op/s 2024-04-25T02:08:56.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:55 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:08:56.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:55 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:08:56.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:55 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:08:56.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:55 smithi134 ceph-mon[33098]: pgmap v451: 97 pgs: 97 active+clean; 411 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 361 op/s 2024-04-25T02:08:56.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:55 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:08:56.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:55 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:08:56.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:55 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:08:57.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 9851 11.45 MB/sec warmup 61 sec latency 67.178 ms 2024-04-25T02:08:58.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 9980 11.42 MB/sec warmup 62 sec latency 69.362 ms 2024-04-25T02:08:58.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:57 smithi031 ceph-mon[29204]: pgmap v452: 97 pgs: 97 active+clean; 411 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-04-25T02:08:58.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:57 smithi040 ceph-mon[32418]: pgmap v452: 97 pgs: 97 active+clean; 411 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-04-25T02:08:58.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:57 smithi134 ceph-mon[33098]: pgmap v452: 97 pgs: 97 active+clean; 411 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-04-25T02:08:59.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 10204 11.61 MB/sec warmup 63 sec latency 52.629 ms 2024-04-25T02:09:00.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 10392 11.61 MB/sec warmup 64 sec latency 50.897 ms 2024-04-25T02:09:00.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:08:59 smithi031 ceph-mon[29204]: pgmap v453: 97 pgs: 97 active+clean; 411 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-04-25T02:09:00.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:08:59 smithi040 ceph-mon[32418]: pgmap v453: 97 pgs: 97 active+clean; 411 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-04-25T02:09:00.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:08:59 smithi134 ceph-mon[33098]: pgmap v453: 97 pgs: 97 active+clean; 411 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.5 MiB/s wr, 243 op/s 2024-04-25T02:09:01.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 10527 11.49 MB/sec warmup 65 sec latency 85.343 ms 2024-04-25T02:09:02.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 10663 11.42 MB/sec warmup 66 sec latency 70.662 ms 2024-04-25T02:09:02.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:01 smithi031 ceph-mon[29204]: pgmap v454: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 350 op/s 2024-04-25T02:09:02.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:01 smithi040 ceph-mon[32418]: pgmap v454: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 350 op/s 2024-04-25T02:09:02.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:01 smithi134 ceph-mon[33098]: pgmap v454: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 350 op/s 2024-04-25T02:09:03.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 10825 11.33 MB/sec warmup 67 sec latency 78.631 ms 2024-04-25T02:09:03.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:02 smithi031 ceph-mon[29204]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T02:09:03.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:02 smithi040 ceph-mon[32418]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T02:09:03.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:02 smithi134 ceph-mon[33098]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T02:09:04.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 10990 11.27 MB/sec warmup 68 sec latency 81.269 ms 2024-04-25T02:09:04.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:03 smithi031 ceph-mon[29204]: pgmap v456: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 266 op/s 2024-04-25T02:09:04.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:03 smithi040 ceph-mon[32418]: pgmap v456: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 266 op/s 2024-04-25T02:09:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.413+0000 7fe916f1b640 1 -- 172.21.15.31:0/1419358162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910101760 msgr2=0x7fe910101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.413+0000 7fe916f1b640 1 --2- 172.21.15.31:0/1419358162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910101760 0x7fe910101be0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009930 tx=0x7fe8fc02a420 comp rx=0 tx=0).stop 2024-04-25T02:09:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.413+0000 7fe916f1b640 1 -- 172.21.15.31:0/1419358162 shutdown_connections 2024-04-25T02:09:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.413+0000 7fe916f1b640 1 --2- 172.21.15.31:0/1419358162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe910102120 0x7fe91010e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.413+0000 7fe916f1b640 1 --2- 172.21.15.31:0/1419358162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910101760 0x7fe910101be0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.413+0000 7fe916f1b640 1 --2- 172.21.15.31:0/1419358162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe910107770 0x7fe910107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.413+0000 7fe916f1b640 1 -- 172.21.15.31:0/1419358162 >> 172.21.15.31:0/1419358162 conn(0x7fe9100fd1f0 msgr2=0x7fe9100ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.413+0000 7fe916f1b640 1 -- 172.21.15.31:0/1419358162 shutdown_connections 2024-04-25T02:09:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.413+0000 7fe916f1b640 1 -- 172.21.15.31:0/1419358162 wait complete. 2024-04-25T02:09:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe916f1b640 1 Processor -- start 2024-04-25T02:09:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe916f1b640 1 -- start start 2024-04-25T02:09:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe916f1b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe910101760 0x7fe910197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe916f1b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910102120 0x7fe910198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe916f1b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe910107770 0x7fe91019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe916f1b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe910111a30 con 0x7fe910102120 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe916f1b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe9101118b0 con 0x7fe910107770 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe916f1b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe910111bb0 con 0x7fe910101760 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe90ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910102120 0x7fe910198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe90ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910102120 0x7fe910198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50620/0 (socket says 172.21.15.31:50620) 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe9154b8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe910107770 0x7fe91019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe90ffff640 1 -- 172.21.15.31:0/4239305243 learned_addr learned my addr 172.21.15.31:0/4239305243 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.414+0000 7fe914cb7640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe910101760 0x7fe910197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.415+0000 7fe9154b8640 1 -- 172.21.15.31:0/4239305243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe910101760 msgr2=0x7fe910197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.415+0000 7fe9154b8640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe910101760 0x7fe910197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.415+0000 7fe9154b8640 1 -- 172.21.15.31:0/4239305243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910102120 msgr2=0x7fe910198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.415+0000 7fe9154b8640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910102120 0x7fe910198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.415+0000 7fe9154b8640 1 -- 172.21.15.31:0/4239305243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91019cee0 con 0x7fe910107770 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.415+0000 7fe90ffff640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910102120 0x7fe910198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.415+0000 7fe9154b8640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe910107770 0x7fe91019c7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe908009fd0 tx=0x7fe90800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.415+0000 7fe90dffb640 1 -- 172.21.15.31:0/4239305243 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90800ce20 con 0x7fe910107770 2024-04-25T02:09:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.415+0000 7fe916f1b640 1 -- 172.21.15.31:0/4239305243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe91019d1d0 con 0x7fe910107770 2024-04-25T02:09:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.415+0000 7fe90dffb640 1 -- 172.21.15.31:0/4239305243 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe908005020 con 0x7fe910107770 2024-04-25T02:09:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.415+0000 7fe90dffb640 1 -- 172.21.15.31:0/4239305243 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908010690 con 0x7fe910107770 2024-04-25T02:09:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.416+0000 7fe916f1b640 1 -- 172.21.15.31:0/4239305243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9101a4a60 con 0x7fe910107770 2024-04-25T02:09:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.418+0000 7fe90dffb640 1 -- 172.21.15.31:0/4239305243 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe9080051d0 con 0x7fe910107770 2024-04-25T02:09:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe919c2c740 1 Processor -- start 2024-04-25T02:09:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe919c2c740 1 -- start start 2024-04-25T02:09:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe90dffb640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe9000a2e20 0x7fe9000a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd30596620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd305a3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe919c2c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dd304facb0 con 0x55dd30596b60 2024-04-25T02:09:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe919c2c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dd304a35b0 con 0x55dd305a1230 2024-04-25T02:09:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe919c2c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dd304f6c70 con 0x55dd30596240 2024-04-25T02:09:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe90ffff640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe9000a2e20 0x7fe9000a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe90dffb640 1 -- 172.21.15.31:0/4239305243 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8528+0+0 (secure 0 0 0) 0x7fe9080cfa40 con 0x7fe910107770 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe914cb7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd30596620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe90ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe914cb7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd30596620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:41248/0 (socket says 172.21.15.31:41248) 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe914cb7640 1 -- 172.21.15.31:0/1702829512 learned_addr learned my addr 172.21.15.31:0/1702829512 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.419+0000 7fe9154b8640 1 --2- 172.21.15.31:0/1702829512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd305a3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.420+0000 7fe914cb7640 1 -- 172.21.15.31:0/1702829512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 msgr2=0x55dd305a3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.420+0000 7fe914cb7640 1 --2- 172.21.15.31:0/1702829512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd305a3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.420+0000 7fe914cb7640 1 -- 172.21.15.31:0/1702829512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 msgr2=0x55dd305a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.420+0000 7fe914cb7640 1 --2- 172.21.15.31:0/1702829512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.420+0000 7fe914cb7640 1 -- 172.21.15.31:0/1702829512 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd3020f210 con 0x55dd30596240 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.420+0000 7fe9154b8640 1 --2- 172.21.15.31:0/1702829512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd305a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.420+0000 7fe90ffff640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe9000a2e20 0x7fe9000a52e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc000c00 tx=0x7fe8fc031400 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.420+0000 7fe914cb7640 1 --2- 172.21.15.31:0/1702829512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd30596620 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe904009c70 tx=0x7fe904011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbac8c854bcc60e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.420+0000 7fe8faffd640 1 -- 172.21.15.31:0/1702829512 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904004230 con 0x55dd30596240 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.420+0000 7fe8faffd640 1 -- 172.21.15.31:0/1702829512 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9040043d0 con 0x55dd30596240 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.420+0000 7fe8faffd640 1 -- 172.21.15.31:0/1702829512 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9040196b0 con 0x55dd30596240 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.421+0000 7fe919c2c740 1 -- 172.21.15.31:0/1702829512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 msgr2=0x55dd30596620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.421+0000 7fe919c2c740 1 --2- 172.21.15.31:0/1702829512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd30596620 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe904009c70 tx=0x7fe904011810 comp rx=0 tx=0).stop 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.421+0000 7fe919c2c740 1 -- 172.21.15.31:0/1702829512 shutdown_connections 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.421+0000 7fe919c2c740 1 --2- 172.21.15.31:0/1702829512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd305a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.421+0000 7fe919c2c740 1 --2- 172.21.15.31:0/1702829512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.421+0000 7fe919c2c740 1 --2- 172.21.15.31:0/1702829512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd30596620 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.421+0000 7fe919c2c740 1 -- 172.21.15.31:0/1702829512 >> 172.21.15.31:0/1702829512 conn(0x55dd30592560 msgr2=0x55dd30592940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.421+0000 7fe919c2c740 1 -- 172.21.15.31:0/1702829512 shutdown_connections 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.421+0000 7fe919c2c740 1 -- 172.21.15.31:0/1702829512 wait complete. 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.421+0000 7fe919c2c740 1 -- start start 2024-04-25T02:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.422+0000 7fe919c2c740 1 Processor -- start 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.422+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd3059e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.422+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd3059e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.422+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd3059ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.422+0000 7fe919c2c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dd305a8430 con 0x55dd30596b60 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.422+0000 7fe919c2c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dd304b6b50 con 0x55dd305a1230 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.422+0000 7fe919c2c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dd305a85b0 con 0x55dd30596240 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.422+0000 7fe9154b8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd3059ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.422+0000 7fe90ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd3059e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.422+0000 7fe914cb7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd3059e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.422+0000 7fe9154b8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd3059ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41558/0 (socket says 172.21.15.31:41558) 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe9154b8640 1 -- 172.21.15.31:0/504877737 learned_addr learned my addr 172.21.15.31:0/504877737 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe919c2c740 10 client.0 _ll_register_callbacks cb 0x55dd304f1ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe9154b8640 1 -- 172.21.15.31:0/504877737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 msgr2=0x55dd3059e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe9154b8640 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd3059e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe9154b8640 1 -- 172.21.15.31:0/504877737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 msgr2=0x55dd3059e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe9154b8640 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd3059e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe9154b8640 1 -- 172.21.15.31:0/504877737 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd302db620 con 0x55dd305a1230 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe90ffff640 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd3059e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe914cb7640 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd3059e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe9154b8640 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd3059ec40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c760 tx=0x7fe9080bc190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e844355a0b083a66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe8effff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe8effff640 1 -- 172.21.15.31:0/504877737 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9080b4030 con 0x55dd305a1230 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe8effff640 1 -- 172.21.15.31:0/504877737 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8528+0+0 (secure 0 0 0) 0x7fe9080a4740 con 0x55dd305a1230 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.423+0000 7fe919c2c740 1 -- 172.21.15.31:0/504877737 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9040043d0 con 0x55dd305a1230 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.424+0000 7fe8effff640 1 -- 172.21.15.31:0/504877737 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fe9080a4bd0 con 0x55dd305a1230 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.424+0000 7fe919c2c740 10 client.26873 fetch_fsmap learned FSMap version 42 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.424+0000 7fe919c2c740 1 -- 172.21.15.31:0/504877737 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55dd305b4670 con 0x55dd305a1230 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.424+0000 7fe8effff640 1 -- 172.21.15.31:0/504877737 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9080a4940 con 0x55dd305a1230 2024-04-25T02:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.424+0000 7fe8effff640 1 -- 172.21.15.31:0/504877737 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) ==== 8394+0+0 (secure 0 0 0) 0x7fe908097cb0 con 0x55dd305a1230 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.424+0000 7fe919c2c740 10 client.26873 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.424+0000 7fe919c2c740 10 client.26873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.424+0000 7fe919c2c740 20 client.26873 populate_metadata read hostname 'smithi031' 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.424+0000 7fe919c2c740 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55dd305b5ba0 0x55dd305b7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.424+0000 7fe919c2c740 4 client.26873 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.424+0000 7fe919c2c740 1 -- 172.21.15.31:0/504877737 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe9040043d0 con 0x55dd305b5ba0 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.425+0000 7fe914cb7640 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55dd305b5ba0 0x55dd305b7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.425+0000 7fe914cb7640 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55dd305b5ba0 0x55dd305b7fd0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fd43f80efda12a6 server_cookie=1b030c36de220a42 in_seq=0 out_seq=0 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.425+0000 7fe8effff640 10 client.26873 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe8effff640 1 -- 172.21.15.31:0/504877737 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fe9040043d0 con 0x55dd305b5ba0 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe8effff640 10 client.26873 handle_command_reply: tid=0 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe919c2c740 1 client.26873 shutdown 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe919c2c740 1 -- 172.21.15.31:0/504877737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 msgr2=0x55dd3059ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe919c2c740 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd3059ec40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe90800c760 tx=0x7fe9080bc190 comp rx=0 tx=0).stop 2024-04-25T02:09:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe919c2c740 1 -- 172.21.15.31:0/504877737 shutdown_connections 2024-04-25T02:09:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe919c2c740 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55dd305b5ba0 0x55dd305b7fd0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe919c2c740 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd3059ec40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe8effff640 0 client.26873 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe919c2c740 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd3059e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe919c2c740 1 --2- 172.21.15.31:0/504877737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd3059e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe919c2c740 1 -- 172.21.15.31:0/504877737 >> 172.21.15.31:0/504877737 conn(0x55dd30592560 msgr2=0x55dd30598360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe919c2c740 1 -- 172.21.15.31:0/504877737 shutdown_connections 2024-04-25T02:09:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.428+0000 7fe919c2c740 1 -- 172.21.15.31:0/504877737 wait complete. 2024-04-25T02:09:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.429+0000 7fe919c2c740 20 client.26873 trim_cache size 0 max 16384 2024-04-25T02:09:04.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:03 smithi134 ceph-mon[33098]: pgmap v456: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 266 op/s 2024-04-25T02:09:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.451+0000 7fe919c2c740 1 Processor -- start 2024-04-25T02:09:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.451+0000 7fe919c2c740 1 -- start start 2024-04-25T02:09:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.451+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd305a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.451+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.451+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd3059d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.451+0000 7fe919c2c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe9101ab200 con 0x55dd30596b60 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.451+0000 7fe919c2c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe9101acec0 con 0x55dd305a1230 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.451+0000 7fe919c2c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe9101ab8d0 con 0x55dd30596240 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe90ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe90ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50630/0 (socket says 172.21.15.31:50630) 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe90ffff640 1 -- 172.21.15.31:0/3342579220 learned_addr learned my addr 172.21.15.31:0/3342579220 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe9154b8640 1 --2- 172.21.15.31:0/3342579220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd3059d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe914cb7640 1 --2- 172.21.15.31:0/3342579220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd305a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe90ffff640 1 -- 172.21.15.31:0/3342579220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 msgr2=0x55dd305a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe90ffff640 1 --2- 172.21.15.31:0/3342579220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd305a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe90ffff640 1 -- 172.21.15.31:0/3342579220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 msgr2=0x55dd3059d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe90ffff640 1 --2- 172.21.15.31:0/3342579220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd3059d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe90ffff640 1 -- 172.21.15.31:0/3342579220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd30596990 con 0x55dd30596b60 2024-04-25T02:09:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe90ffff640 1 --2- 172.21.15.31:0/3342579220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305b12f0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009a50 tx=0x7fe8fc02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e7a2ac7f34a700 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.452+0000 7fe8fa7fc640 1 -- 172.21.15.31:0/3342579220 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc003970 con 0x55dd30596b60 2024-04-25T02:09:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.453+0000 7fe8fa7fc640 1 -- 172.21.15.31:0/3342579220 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8fc057480 con 0x55dd30596b60 2024-04-25T02:09:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.453+0000 7fe8fa7fc640 1 -- 172.21.15.31:0/3342579220 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc03d6b0 con 0x55dd30596b60 2024-04-25T02:09:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.453+0000 7fe919c2c740 1 -- 172.21.15.31:0/3342579220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 msgr2=0x55dd305b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.453+0000 7fe919c2c740 1 --2- 172.21.15.31:0/3342579220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305b12f0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009a50 tx=0x7fe8fc02a420 comp rx=0 tx=0).stop 2024-04-25T02:09:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.454+0000 7fe919c2c740 1 -- 172.21.15.31:0/3342579220 shutdown_connections 2024-04-25T02:09:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.454+0000 7fe919c2c740 1 --2- 172.21.15.31:0/3342579220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd305a1230 0x55dd3059d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.454+0000 7fe919c2c740 1 --2- 172.21.15.31:0/3342579220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305b12f0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.454+0000 7fe919c2c740 1 --2- 172.21.15.31:0/3342579220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd30596240 0x55dd305a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.454+0000 7fe919c2c740 1 -- 172.21.15.31:0/3342579220 >> 172.21.15.31:0/3342579220 conn(0x55dd30592560 msgr2=0x55dd30590d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.454+0000 7fe919c2c740 1 -- 172.21.15.31:0/3342579220 shutdown_connections 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.454+0000 7fe919c2c740 1 -- 172.21.15.31:0/3342579220 wait complete. 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.454+0000 7fe919c2c740 1 -- start start 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.454+0000 7fe919c2c740 1 Processor -- start 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd30596240 0x55dd305ad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305a8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe919c2c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd305a1230 0x55dd305a9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe919c2c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe9101ac7f0 con 0x55dd30596b60 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe919c2c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe9101ac060 con 0x55dd30596240 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe919c2c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe9101a91e0 con 0x55dd305a1230 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe919c2c740 10 client.0 _ll_register_callbacks cb 0x7fe90809bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe914cb7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd30596240 0x55dd305ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe9154b8640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd305a1230 0x55dd305a9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe914cb7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd30596240 0x55dd305ad870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41572/0 (socket says 172.21.15.31:41572) 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe914cb7640 1 -- 172.21.15.31:0/3172942661 learned_addr learned my addr 172.21.15.31:0/3172942661 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe90ffff640 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305a8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe914cb7640 1 -- 172.21.15.31:0/3172942661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd305a1230 msgr2=0x55dd305a9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe914cb7640 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd305a1230 0x55dd305a9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe914cb7640 1 -- 172.21.15.31:0/3172942661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 msgr2=0x55dd305a8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe914cb7640 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305a8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe914cb7640 1 -- 172.21.15.31:0/3172942661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd305a9650 con 0x55dd30596240 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.455+0000 7fe90ffff640 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305a8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.456+0000 7fe914cb7640 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd30596240 0x55dd305ad870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe904005fd0 tx=0x7fe9040099d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1118f64ab73fa439 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.456+0000 7fe8effff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.456+0000 7fe8effff640 1 -- 172.21.15.31:0/3172942661 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904003d00 con 0x55dd30596240 2024-04-25T02:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.456+0000 7fe8effff640 1 -- 172.21.15.31:0/3172942661 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8528+0+0 (secure 0 0 0) 0x7fe90401f040 con 0x55dd30596240 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.456+0000 7fe919c2c740 1 -- 172.21.15.31:0/3172942661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8fc057480 con 0x55dd30596240 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.456+0000 7fe8effff640 1 -- 172.21.15.31:0/3172942661 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fe9040052d0 con 0x55dd30596240 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.456+0000 7fe919c2c740 10 client.26879 fetch_fsmap learned FSMap version 42 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.456+0000 7fe919c2c740 1 -- 172.21.15.31:0/3172942661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55dd3061bd70 con 0x55dd30596240 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.456+0000 7fe8effff640 1 -- 172.21.15.31:0/3172942661 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8fc057480 con 0x55dd30596240 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.457+0000 7fe8effff640 1 -- 172.21.15.31:0/3172942661 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) ==== 8394+0+0 (secure 0 0 0) 0x7fe904018070 con 0x55dd30596240 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.457+0000 7fe919c2c740 10 client.26879 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.457+0000 7fe919c2c740 10 client.26879 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.457+0000 7fe919c2c740 20 client.26879 populate_metadata read hostname 'smithi031' 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.457+0000 7fe919c2c740 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55dd305b5ba0 0x55dd3061e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.457+0000 7fe919c2c740 4 client.26879 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.457+0000 7fe919c2c740 1 -- 172.21.15.31:0/3172942661 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd305081f0 con 0x55dd305b5ba0 2024-04-25T02:09:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.457+0000 7fe90ffff640 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55dd305b5ba0 0x55dd3061e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.457+0000 7fe90ffff640 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55dd305b5ba0 0x55dd3061e390 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d87d1c0b19872abb server_cookie=51e03107e3e69ee5 in_seq=0 out_seq=0 2024-04-25T02:09:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.457+0000 7fe8effff640 10 client.26879 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe8effff640 1 -- 172.21.15.31:0/3172942661 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd305081f0 con 0x55dd305b5ba0 2024-04-25T02:09:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe8effff640 10 client.26879 handle_command_reply: tid=0 2024-04-25T02:09:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe919c2c740 1 client.26879 shutdown 2024-04-25T02:09:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe919c2c740 1 -- 172.21.15.31:0/3172942661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd30596240 msgr2=0x55dd305ad870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe919c2c740 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd30596240 0x55dd305ad870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe904005fd0 tx=0x7fe9040099d0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe919c2c740 1 -- 172.21.15.31:0/3172942661 shutdown_connections 2024-04-25T02:09:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe919c2c740 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55dd305b5ba0 0x55dd3061e390 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe919c2c740 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dd305a1230 0x55dd305a9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe919c2c740 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd30596b60 0x55dd305a8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe8effff640 0 client.26879 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe919c2c740 1 --2- 172.21.15.31:0/3172942661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd30596240 0x55dd305ad870 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.518+0000 7fe919c2c740 1 -- 172.21.15.31:0/3172942661 >> 172.21.15.31:0/3172942661 conn(0x55dd30592560 msgr2=0x55dd30590d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.519+0000 7fe919c2c740 1 -- 172.21.15.31:0/3172942661 shutdown_connections 2024-04-25T02:09:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.519+0000 7fe919c2c740 1 -- 172.21.15.31:0/3172942661 wait complete. 2024-04-25T02:09:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.519+0000 7fe919c2c740 20 client.26879 trim_cache size 0 max 16384 2024-04-25T02:09:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:09:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:09:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:09:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:09:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 -- 172.21.15.31:0/4239305243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe9000a2e20 msgr2=0x7fe9000a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe9000a2e20 0x7fe9000a52e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc000c00 tx=0x7fe8fc031400 comp rx=0 tx=0).stop 2024-04-25T02:09:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 -- 172.21.15.31:0/4239305243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe910107770 msgr2=0x7fe91019c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe910107770 0x7fe91019c7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe908009fd0 tx=0x7fe90800ef10 comp rx=0 tx=0).stop 2024-04-25T02:09:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 -- 172.21.15.31:0/4239305243 shutdown_connections 2024-04-25T02:09:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe9000a2e20 0x7fe9000a52e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe910107770 0x7fe91019c7b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910102120 0x7fe910198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 --2- 172.21.15.31:0/4239305243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe910101760 0x7fe910197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 -- 172.21.15.31:0/4239305243 >> 172.21.15.31:0/4239305243 conn(0x7fe9100fd1f0 msgr2=0x7fe9100fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 -- 172.21.15.31:0/4239305243 shutdown_connections 2024-04-25T02:09:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:04.520+0000 7fe916f1b640 1 -- 172.21.15.31:0/4239305243 wait complete. 2024-04-25T02:09:05.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 11148 11.18 MB/sec warmup 69 sec latency 74.520 ms 2024-04-25T02:09:06.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 11317 11.18 MB/sec warmup 70 sec latency 78.417 ms 2024-04-25T02:09:06.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:05 smithi031 ceph-mon[29204]: pgmap v457: 97 pgs: 97 active+clean; 441 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 294 op/s 2024-04-25T02:09:06.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:05 smithi040 ceph-mon[32418]: pgmap v457: 97 pgs: 97 active+clean; 441 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 294 op/s 2024-04-25T02:09:06.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:05 smithi134 ceph-mon[33098]: pgmap v457: 97 pgs: 97 active+clean; 441 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 294 op/s 2024-04-25T02:09:07.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 11486 11.17 MB/sec warmup 71 sec latency 62.666 ms 2024-04-25T02:09:08.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 11668 11.22 MB/sec warmup 72 sec latency 53.100 ms 2024-04-25T02:09:08.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:07 smithi031 ceph-mon[29204]: pgmap v458: 97 pgs: 97 active+clean; 441 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 294 op/s 2024-04-25T02:09:08.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:07 smithi040 ceph-mon[32418]: pgmap v458: 97 pgs: 97 active+clean; 441 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 294 op/s 2024-04-25T02:09:08.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:07 smithi134 ceph-mon[33098]: pgmap v458: 97 pgs: 97 active+clean; 441 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 294 op/s 2024-04-25T02:09:09.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 11834 11.11 MB/sec warmup 73 sec latency 54.449 ms 2024-04-25T02:09:10.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 11999 10.99 MB/sec warmup 74 sec latency 57.823 ms 2024-04-25T02:09:10.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:09 smithi040 ceph-mon[32418]: pgmap v459: 97 pgs: 97 active+clean; 441 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 294 op/s 2024-04-25T02:09:10.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:09 smithi040 ceph-mon[32418]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T02:09:10.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:09 smithi134 ceph-mon[33098]: pgmap v459: 97 pgs: 97 active+clean; 441 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 294 op/s 2024-04-25T02:09:10.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:09 smithi134 ceph-mon[33098]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T02:09:10.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:09 smithi031 ceph-mon[29204]: pgmap v459: 97 pgs: 97 active+clean; 441 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 294 op/s 2024-04-25T02:09:10.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:09 smithi031 ceph-mon[29204]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T02:09:11.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 12170 10.90 MB/sec warmup 75 sec latency 60.022 ms 2024-04-25T02:09:12.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 12349 10.86 MB/sec warmup 76 sec latency 59.931 ms 2024-04-25T02:09:12.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:11 smithi040 ceph-mon[32418]: pgmap v461: 97 pgs: 97 active+clean; 457 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 366 op/s 2024-04-25T02:09:12.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:11 smithi134 ceph-mon[33098]: pgmap v461: 97 pgs: 97 active+clean; 457 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 366 op/s 2024-04-25T02:09:12.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:11 smithi031 ceph-mon[29204]: pgmap v461: 97 pgs: 97 active+clean; 457 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 366 op/s 2024-04-25T02:09:13.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 12516 10.84 MB/sec warmup 77 sec latency 63.673 ms 2024-04-25T02:09:14.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 12719 10.89 MB/sec warmup 78 sec latency 58.938 ms 2024-04-25T02:09:14.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:13 smithi134 ceph-mon[33098]: pgmap v462: 97 pgs: 97 active+clean; 457 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 324 op/s 2024-04-25T02:09:14.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:13 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:09:14.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:13 smithi031 ceph-mon[29204]: pgmap v462: 97 pgs: 97 active+clean; 457 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 324 op/s 2024-04-25T02:09:14.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:13 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:09:14.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:13 smithi040 ceph-mon[32418]: pgmap v462: 97 pgs: 97 active+clean; 457 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 324 op/s 2024-04-25T02:09:14.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:13 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:09:15.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 12956 11.01 MB/sec warmup 79 sec latency 52.134 ms 2024-04-25T02:09:16.091 INFO:tasks.workunit.client.0.smithi031.stdout: 10 13173 11.01 MB/sec warmup 80 sec latency 72.221 ms 2024-04-25T02:09:16.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:15 smithi134 ceph-mon[33098]: pgmap v463: 97 pgs: 97 active+clean; 470 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-04-25T02:09:16.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:15 smithi031 ceph-mon[29204]: pgmap v463: 97 pgs: 97 active+clean; 470 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-04-25T02:09:16.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:15 smithi040 ceph-mon[32418]: pgmap v463: 97 pgs: 97 active+clean; 470 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-04-25T02:09:17.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 13328 11.00 MB/sec warmup 81 sec latency 54.297 ms 2024-04-25T02:09:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.645+0000 7fc51d5b7640 1 -- 172.21.15.31:0/651057773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5181029d0 msgr2=0x7fc518102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.645+0000 7fc51d5b7640 1 --2- 172.21.15.31:0/651057773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5181029d0 0x7fc518102db0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc508009900 tx=0x7fc50802a480 comp rx=0 tx=0).stop 2024-04-25T02:09:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.645+0000 7fc51d5b7640 1 -- 172.21.15.31:0/651057773 shutdown_connections 2024-04-25T02:09:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.645+0000 7fc51d5b7640 1 --2- 172.21.15.31:0/651057773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc51810bda0 0x7fc51810e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.645+0000 7fc51d5b7640 1 --2- 172.21.15.31:0/651057773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5181032f0 0x7fc51810b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.645+0000 7fc51d5b7640 1 --2- 172.21.15.31:0/651057773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5181029d0 0x7fc518102db0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.645+0000 7fc51d5b7640 1 -- 172.21.15.31:0/651057773 >> 172.21.15.31:0/651057773 conn(0x7fc518077760 msgr2=0x7fc518077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.645+0000 7fc51d5b7640 1 -- 172.21.15.31:0/651057773 shutdown_connections 2024-04-25T02:09:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.645+0000 7fc51d5b7640 1 -- 172.21.15.31:0/651057773 wait complete. 2024-04-25T02:09:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.646+0000 7fc51d5b7640 1 Processor -- start 2024-04-25T02:09:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.646+0000 7fc51d5b7640 1 -- start start 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.646+0000 7fc51d5b7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5181029d0 0x7fc518100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.646+0000 7fc51d5b7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5181032f0 0x7fc518100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.646+0000 7fc51d5b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc51810bda0 0x7fc5181010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.646+0000 7fc51d5b7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181119e0 con 0x7fc51810bda0 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.646+0000 7fc51d5b7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc518111860 con 0x7fc5181032f0 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.646+0000 7fc51d5b7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc518111b60 con 0x7fc5181029d0 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc516ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5181029d0 0x7fc518100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc5167fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5181032f0 0x7fc518100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc516ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5181029d0 0x7fc518100620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:37842/0 (socket says 172.21.15.31:37842) 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc5167fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5181032f0 0x7fc518100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:32884/0 (socket says 172.21.15.31:32884) 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc516ffd640 1 -- 172.21.15.31:0/2246286265 learned_addr learned my addr 172.21.15.31:0/2246286265 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc5177fe640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc51810bda0 0x7fc5181010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc5167fc640 1 -- 172.21.15.31:0/2246286265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5181029d0 msgr2=0x7fc518100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc5167fc640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5181029d0 0x7fc518100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc5167fc640 1 -- 172.21.15.31:0/2246286265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc51810bda0 msgr2=0x7fc5181010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc5167fc640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc51810bda0 0x7fc5181010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc5167fc640 1 -- 172.21.15.31:0/2246286265 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5181a63a0 con 0x7fc5181032f0 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc5177fe640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc51810bda0 0x7fc5181010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.647+0000 7fc5167fc640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5181032f0 0x7fc518100b60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00e9f0 tx=0x7fc50c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.648+0000 7fc4fffff640 1 -- 172.21.15.31:0/2246286265 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c00ccc0 con 0x7fc5181032f0 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.648+0000 7fc4fffff640 1 -- 172.21.15.31:0/2246286265 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc50c0108e0 con 0x7fc5181032f0 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.648+0000 7fc51d5b7640 1 -- 172.21.15.31:0/2246286265 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5181a6690 con 0x7fc5181032f0 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.648+0000 7fc4fffff640 1 -- 172.21.15.31:0/2246286265 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c018880 con 0x7fc5181032f0 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.648+0000 7fc51d5b7640 1 -- 172.21.15.31:0/2246286265 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5181a6be0 con 0x7fc5181032f0 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc4fffff640 1 -- 172.21.15.31:0/2246286265 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc50c010a90 con 0x7fc5181032f0 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc5203a3740 1 Processor -- start 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc5203a3740 1 -- start start 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c57620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57b60 0x55b3c2c61cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 0x55b3c2c64620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc5203a3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b3c2bbbcb0 con 0x55b3c2c62230 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc5203a3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b3c2b645b0 con 0x55b3c2c57240 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc5203a3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b3c2bb7c70 con 0x55b3c2c57b60 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc5177fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c57620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc516ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57b60 0x55b3c2c61cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc5177fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c57620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:32890/0 (socket says 172.21.15.31:32890) 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.651+0000 7fc4fffff640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc5000a2da0 0x7fc5000a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc5177fe640 1 -- 172.21.15.31:0/3550665251 learned_addr learned my addr 172.21.15.31:0/3550665251 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc5167fc640 1 --2- 172.21.15.31:0/3550665251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 0x55b3c2c64620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc5177fe640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc5000a2da0 0x7fc5000a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc4fffff640 1 -- 172.21.15.31:0/2246286265 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7fc50c0cf780 con 0x7fc5181032f0 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc5177fe640 1 -- 172.21.15.31:0/3550665251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57b60 msgr2=0x55b3c2c61cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc5177fe640 1 --2- 172.21.15.31:0/3550665251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57b60 0x55b3c2c61cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc5177fe640 1 -- 172.21.15.31:0/3550665251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 msgr2=0x55b3c2c64620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc5177fe640 1 --2- 172.21.15.31:0/3550665251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 0x55b3c2c64620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc5177fe640 1 -- 172.21.15.31:0/3550665251 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3c28d0210 con 0x55b3c2c57240 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc516ffd640 1 --2- 172.21.15.31:0/3550665251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57b60 0x55b3c2c61cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc5167fc640 1 --2- 172.21.15.31:0/3550665251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 0x55b3c2c64620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.652+0000 7fc5177fe640 1 --2- 172.21.15.31:0/3550665251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c57620 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc510013e40 tx=0x7fc51000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8d0950ea89594b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.653+0000 7fc4fd7fa640 1 -- 172.21.15.31:0/3550665251 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc510023020 con 0x55b3c2c57240 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.653+0000 7fc4fd7fa640 1 -- 172.21.15.31:0/3550665251 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc51001f930 con 0x55b3c2c57240 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.653+0000 7fc4fd7fa640 1 -- 172.21.15.31:0/3550665251 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51001e640 con 0x55b3c2c57240 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.653+0000 7fc5203a3740 1 -- 172.21.15.31:0/3550665251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 msgr2=0x55b3c2c57620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.653+0000 7fc5177fe640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc5000a2da0 0x7fc5000a5260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc510024000 tx=0x7fc510022000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.653+0000 7fc5203a3740 1 --2- 172.21.15.31:0/3550665251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c57620 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc510013e40 tx=0x7fc51000f790 comp rx=0 tx=0).stop 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.654+0000 7fc5203a3740 1 -- 172.21.15.31:0/3550665251 shutdown_connections 2024-04-25T02:09:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.654+0000 7fc5203a3740 1 --2- 172.21.15.31:0/3550665251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 0x55b3c2c64620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.654+0000 7fc5203a3740 1 --2- 172.21.15.31:0/3550665251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57b60 0x55b3c2c61cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.654+0000 7fc5203a3740 1 --2- 172.21.15.31:0/3550665251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c57620 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.654+0000 7fc5203a3740 1 -- 172.21.15.31:0/3550665251 >> 172.21.15.31:0/3550665251 conn(0x55b3c2c53560 msgr2=0x55b3c2c53940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.654+0000 7fc5203a3740 1 -- 172.21.15.31:0/3550665251 shutdown_connections 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.654+0000 7fc5203a3740 1 -- 172.21.15.31:0/3550665251 wait complete. 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.654+0000 7fc5203a3740 1 -- start start 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.654+0000 7fc5203a3740 1 Processor -- start 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c5f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 0x55b3c2c5f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c62230 0x55b3c2c5fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc5203a3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b3c2c65020 con 0x55b3c2c57b60 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc5203a3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b3c2b77b50 con 0x55b3c2c57240 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc5203a3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b3c2c651a0 con 0x55b3c2c62230 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc5203a3740 10 client.0 _ll_register_callbacks cb 0x55b3c2bb2ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc516ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c5f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc5167fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 0x55b3c2c5f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc516ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c5f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:32906/0 (socket says 172.21.15.31:32906) 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc516ffd640 1 -- 172.21.15.31:0/3268290321 learned_addr learned my addr 172.21.15.31:0/3268290321 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc5177fe640 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c62230 0x55b3c2c5fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc516ffd640 1 -- 172.21.15.31:0/3268290321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c62230 msgr2=0x55b3c2c5fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc516ffd640 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c62230 0x55b3c2c5fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc516ffd640 1 -- 172.21.15.31:0/3268290321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 msgr2=0x55b3c2c5f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc516ffd640 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 0x55b3c2c5f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc516ffd640 1 -- 172.21.15.31:0/3268290321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3c299c620 con 0x55b3c2c57240 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.655+0000 7fc5167fc640 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 0x55b3c2c5f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc516ffd640 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c5f1b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc508000c00 tx=0x7fc508031ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbdc5cfc114beaf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc4faff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc4faff5640 1 -- 172.21.15.31:0/3268290321 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508035040 con 0x55b3c2c57240 2024-04-25T02:09:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc5203a3740 1 -- 172.21.15.31:0/3268290321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc51001f930 con 0x55b3c2c57240 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc4faff5640 1 -- 172.21.15.31:0/3268290321 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7fc5080317a0 con 0x55b3c2c57240 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc4faff5640 1 -- 172.21.15.31:0/3268290321 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fc508004040 con 0x55b3c2c57240 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc5203a3740 10 client.26891 fetch_fsmap learned FSMap version 43 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc5203a3740 1 -- 172.21.15.31:0/3268290321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55b3c2c71200 con 0x55b3c2c57240 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc4faff5640 1 -- 172.21.15.31:0/3268290321 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc51001f930 con 0x55b3c2c57240 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc4faff5640 1 -- 172.21.15.31:0/3268290321 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fc50804d020 con 0x55b3c2c57240 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc5203a3740 10 client.26891 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.656+0000 7fc5203a3740 10 client.26891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.657+0000 7fc5203a3740 20 client.26891 populate_metadata read hostname 'smithi031' 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.657+0000 7fc5203a3740 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3c2c72b40 0x55b3c2c74fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.657+0000 7fc5203a3740 4 client.26891 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.657+0000 7fc5203a3740 1 -- 172.21.15.31:0/3268290321 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3c2bc91f0 con 0x55b3c2c72b40 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.657+0000 7fc5167fc640 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3c2c72b40 0x55b3c2c74fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.657+0000 7fc4faff5640 10 client.26891 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.657+0000 7fc5167fc640 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3c2c72b40 0x55b3c2c74fc0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daa02847a6ebed8f server_cookie=13ffeb573d6823f6 in_seq=0 out_seq=0 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc4faff5640 1 -- 172.21.15.31:0/3268290321 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3c2bc91f0 con 0x55b3c2c72b40 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc4faff5640 10 client.26891 handle_command_reply: tid=0 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 1 client.26891 shutdown 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 1 -- 172.21.15.31:0/3268290321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 msgr2=0x55b3c2c5f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c5f1b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc508000c00 tx=0x7fc508031ef0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 1 -- 172.21.15.31:0/3268290321 shutdown_connections 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3c2c72b40 0x55b3c2c74fc0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c62230 0x55b3c2c5fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc4faff5640 0 client.26891 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 0x55b3c2c5f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 1 --2- 172.21.15.31:0/3268290321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57240 0x55b3c2c5f1b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 1 -- 172.21.15.31:0/3268290321 >> 172.21.15.31:0/3268290321 conn(0x55b3c2c53560 msgr2=0x55b3c2c59360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 1 -- 172.21.15.31:0/3268290321 shutdown_connections 2024-04-25T02:09:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 1 -- 172.21.15.31:0/3268290321 wait complete. 2024-04-25T02:09:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.658+0000 7fc5203a3740 20 client.26891 trim_cache size 0 max 16384 2024-04-25T02:09:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.680+0000 7fc5203a3740 1 Processor -- start 2024-04-25T02:09:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.680+0000 7fc5203a3740 1 -- start start 2024-04-25T02:09:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.680+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57240 0x55b3c2c6dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.680+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57b60 0x55b3c2c59ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.680+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 0x55b3c2c53940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.680+0000 7fc5203a3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181af400 con 0x55b3c2c62230 2024-04-25T02:09:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.680+0000 7fc5203a3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181b1060 con 0x55b3c2c57b60 2024-04-25T02:09:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.680+0000 7fc5203a3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181afad0 con 0x55b3c2c57240 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.680+0000 7fc5177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 0x55b3c2c53940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc5177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 0x55b3c2c53940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34182/0 (socket says 172.21.15.31:34182) 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc5167fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57b60 0x55b3c2c59ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc5177fe640 1 -- 172.21.15.31:0/1486131137 learned_addr learned my addr 172.21.15.31:0/1486131137 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc516ffd640 1 --2- 172.21.15.31:0/1486131137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57240 0x55b3c2c6dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc5177fe640 1 -- 172.21.15.31:0/1486131137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57240 msgr2=0x55b3c2c6dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc5177fe640 1 --2- 172.21.15.31:0/1486131137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57240 0x55b3c2c6dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc5177fe640 1 -- 172.21.15.31:0/1486131137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57b60 msgr2=0x55b3c2c59ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc5177fe640 1 --2- 172.21.15.31:0/1486131137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57b60 0x55b3c2c59ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc5177fe640 1 -- 172.21.15.31:0/1486131137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3c2c5e180 con 0x55b3c2c62230 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc5167fc640 1 --2- 172.21.15.31:0/1486131137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57b60 0x55b3c2c59ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc5177fe640 1 --2- 172.21.15.31:0/1486131137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 0x55b3c2c53940 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fc5100138b0 tx=0x7fc510012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=300b75b5c08cdd31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.681+0000 7fc4fcff9640 1 -- 172.21.15.31:0/1486131137 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc510023050 con 0x55b3c2c62230 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.682+0000 7fc4fcff9640 1 -- 172.21.15.31:0/1486131137 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5100139e0 con 0x55b3c2c62230 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.682+0000 7fc4fcff9640 1 -- 172.21.15.31:0/1486131137 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51001f6c0 con 0x55b3c2c62230 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.682+0000 7fc5203a3740 1 -- 172.21.15.31:0/1486131137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 msgr2=0x55b3c2c53940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.682+0000 7fc5203a3740 1 --2- 172.21.15.31:0/1486131137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 0x55b3c2c53940 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fc5100138b0 tx=0x7fc510012000 comp rx=0 tx=0).stop 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.682+0000 7fc5203a3740 1 -- 172.21.15.31:0/1486131137 shutdown_connections 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.682+0000 7fc5203a3740 1 --2- 172.21.15.31:0/1486131137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c62230 0x55b3c2c53940 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.682+0000 7fc5203a3740 1 --2- 172.21.15.31:0/1486131137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c57b60 0x55b3c2c59ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.682+0000 7fc5203a3740 1 --2- 172.21.15.31:0/1486131137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57240 0x55b3c2c6dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.682+0000 7fc5203a3740 1 -- 172.21.15.31:0/1486131137 >> 172.21.15.31:0/1486131137 conn(0x55b3c2c53560 msgr2=0x55b3c2c525c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5203a3740 1 -- 172.21.15.31:0/1486131137 shutdown_connections 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5203a3740 1 -- 172.21.15.31:0/1486131137 wait complete. 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5203a3740 1 -- start start 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5203a3740 1 Processor -- start 2024-04-25T02:09:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57240 0x55b3c2c6a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 0x55b3c2c5bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5203a3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c62230 0x55b3c2c5c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5203a3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181b0990 con 0x55b3c2c57b60 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5203a3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181b0200 con 0x55b3c2c62230 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5203a3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181ad400 con 0x55b3c2c57240 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5203a3740 10 client.0 _ll_register_callbacks cb 0x7fc508009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.683+0000 7fc5167fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 0x55b3c2c5bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc5167fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 0x55b3c2c5bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34198/0 (socket says 172.21.15.31:34198) 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc5167fc640 1 -- 172.21.15.31:0/2331620543 learned_addr learned my addr 172.21.15.31:0/2331620543 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc516ffd640 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57240 0x55b3c2c6a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc5177fe640 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c62230 0x55b3c2c5c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc5167fc640 1 -- 172.21.15.31:0/2331620543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57240 msgr2=0x55b3c2c6a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc5167fc640 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57240 0x55b3c2c6a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc5167fc640 1 -- 172.21.15.31:0/2331620543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c62230 msgr2=0x55b3c2c5c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc5167fc640 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c62230 0x55b3c2c5c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc5167fc640 1 -- 172.21.15.31:0/2331620543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3c2c64990 con 0x55b3c2c57b60 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc5167fc640 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 0x55b3c2c5bd90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc50c09ba70 tx=0x7fc50c009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6dfc2735d331f41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc4faff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc4faff5640 1 -- 172.21.15.31:0/2331620543 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c0a4930 con 0x55b3c2c57b60 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc4faff5640 1 -- 172.21.15.31:0/2331620543 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7fc50c0983e0 con 0x55b3c2c57b60 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc5203a3740 1 -- 172.21.15.31:0/2331620543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5100139e0 con 0x55b3c2c57b60 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.684+0000 7fc4faff5640 1 -- 172.21.15.31:0/2331620543 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fc50c018a20 con 0x55b3c2c57b60 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.685+0000 7fc5203a3740 10 client.17325 fetch_fsmap learned FSMap version 43 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.685+0000 7fc5203a3740 1 -- 172.21.15.31:0/2331620543 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55b3c2cd91a0 con 0x55b3c2c57b60 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.685+0000 7fc4faff5640 1 -- 172.21.15.31:0/2331620543 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5100139e0 con 0x55b3c2c57b60 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.685+0000 7fc4faff5640 1 -- 172.21.15.31:0/2331620543 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fc50c09b2e0 con 0x55b3c2c57b60 2024-04-25T02:09:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.685+0000 7fc5203a3740 10 client.17325 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:09:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.685+0000 7fc5203a3740 10 client.17325 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.685+0000 7fc5203a3740 20 client.17325 populate_metadata read hostname 'smithi031' 2024-04-25T02:09:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.685+0000 7fc5203a3740 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3c2cd9990 0x55b3c2cdbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.685+0000 7fc5203a3740 4 client.17325 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:09:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.685+0000 7fc5203a3740 1 -- 172.21.15.31:0/2331620543 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b3c2bc3540 con 0x55b3c2cd9990 2024-04-25T02:09:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.685+0000 7fc516ffd640 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3c2cd9990 0x55b3c2cdbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.686+0000 7fc4faff5640 10 client.17325 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.686+0000 7fc516ffd640 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3c2cd9990 0x55b3c2cdbdb0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffbc455443ea2062 server_cookie=711240fb0688f5c8 in_seq=0 out_seq=0 2024-04-25T02:09:17.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.747+0000 7fc4faff5640 1 -- 172.21.15.31:0/2331620543 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b3c2bc3540 con 0x55b3c2cd9990 2024-04-25T02:09:17.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.747+0000 7fc4faff5640 10 client.17325 handle_command_reply: tid=0 2024-04-25T02:09:17.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.747+0000 7fc5203a3740 1 client.17325 shutdown 2024-04-25T02:09:17.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.747+0000 7fc5203a3740 1 -- 172.21.15.31:0/2331620543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 msgr2=0x55b3c2c5bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:17.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.747+0000 7fc5203a3740 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 0x55b3c2c5bd90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc50c09ba70 tx=0x7fc50c009700 comp rx=0 tx=0).stop 2024-04-25T02:09:17.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.747+0000 7fc5203a3740 1 -- 172.21.15.31:0/2331620543 shutdown_connections 2024-04-25T02:09:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.747+0000 7fc5203a3740 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3c2cd9990 0x55b3c2cdbdb0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.747+0000 7fc5203a3740 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3c2c62230 0x55b3c2c5c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.747+0000 7fc5203a3740 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3c2c57b60 0x55b3c2c5bd90 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.747+0000 7fc5203a3740 1 --2- 172.21.15.31:0/2331620543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3c2c57240 0x55b3c2c6a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.748+0000 7fc4faff5640 0 client.17325 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.748+0000 7fc5203a3740 1 -- 172.21.15.31:0/2331620543 >> 172.21.15.31:0/2331620543 conn(0x55b3c2c53560 msgr2=0x55b3c2c525c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.748+0000 7fc5203a3740 1 -- 172.21.15.31:0/2331620543 shutdown_connections 2024-04-25T02:09:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.748+0000 7fc5203a3740 1 -- 172.21.15.31:0/2331620543 wait complete. 2024-04-25T02:09:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.748+0000 7fc5203a3740 20 client.17325 trim_cache size 0 max 16384 2024-04-25T02:09:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:09:17.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:09:17.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:09:17.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:09:17.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.748+0000 7fc51d5b7640 1 -- 172.21.15.31:0/2246286265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc5000a2da0 msgr2=0x7fc5000a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.748+0000 7fc51d5b7640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc5000a2da0 0x7fc5000a5260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc510024000 tx=0x7fc510022000 comp rx=0 tx=0).stop 2024-04-25T02:09:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.749+0000 7fc51d5b7640 1 -- 172.21.15.31:0/2246286265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5181032f0 msgr2=0x7fc518100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.749+0000 7fc51d5b7640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5181032f0 0x7fc518100b60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00e9f0 tx=0x7fc50c00eeb0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.749+0000 7fc51d5b7640 1 -- 172.21.15.31:0/2246286265 shutdown_connections 2024-04-25T02:09:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.749+0000 7fc51d5b7640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc5000a2da0 0x7fc5000a5260 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.749+0000 7fc51d5b7640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc51810bda0 0x7fc5181010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.749+0000 7fc51d5b7640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5181032f0 0x7fc518100b60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.749+0000 7fc51d5b7640 1 --2- 172.21.15.31:0/2246286265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5181029d0 0x7fc518100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.749+0000 7fc51d5b7640 1 -- 172.21.15.31:0/2246286265 >> 172.21.15.31:0/2246286265 conn(0x7fc518077760 msgr2=0x7fc5180fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.749+0000 7fc51d5b7640 1 -- 172.21.15.31:0/2246286265 shutdown_connections 2024-04-25T02:09:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:17.749+0000 7fc51d5b7640 1 -- 172.21.15.31:0/2246286265 wait complete. 2024-04-25T02:09:18.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 13462 10.93 MB/sec warmup 82 sec latency 73.487 ms 2024-04-25T02:09:18.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:17 smithi134 ceph-mon[33098]: pgmap v464: 97 pgs: 97 active+clean; 470 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-04-25T02:09:18.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:17 smithi040 ceph-mon[32418]: pgmap v464: 97 pgs: 97 active+clean; 470 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-04-25T02:09:18.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:17 smithi031 ceph-mon[29204]: pgmap v464: 97 pgs: 97 active+clean; 470 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-04-25T02:09:19.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 13653 11.00 MB/sec warmup 83 sec latency 57.304 ms 2024-04-25T02:09:20.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 13820 11.00 MB/sec warmup 84 sec latency 70.809 ms 2024-04-25T02:09:20.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:19 smithi134 ceph-mon[33098]: pgmap v465: 97 pgs: 97 active+clean; 470 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-04-25T02:09:20.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:19 smithi031 ceph-mon[29204]: pgmap v465: 97 pgs: 97 active+clean; 470 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-04-25T02:09:20.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:19 smithi040 ceph-mon[32418]: pgmap v465: 97 pgs: 97 active+clean; 470 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-04-25T02:09:21.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 13955 10.94 MB/sec warmup 85 sec latency 76.379 ms 2024-04-25T02:09:22.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 14117 10.90 MB/sec warmup 86 sec latency 74.891 ms 2024-04-25T02:09:22.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:21 smithi134 ceph-mon[33098]: pgmap v466: 97 pgs: 97 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.5 MiB/s wr, 371 op/s 2024-04-25T02:09:22.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:21 smithi031 ceph-mon[29204]: pgmap v466: 97 pgs: 97 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.5 MiB/s wr, 371 op/s 2024-04-25T02:09:22.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:21 smithi040 ceph-mon[32418]: pgmap v466: 97 pgs: 97 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.5 MiB/s wr, 371 op/s 2024-04-25T02:09:23.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 14273 10.84 MB/sec warmup 87 sec latency 58.093 ms 2024-04-25T02:09:24.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 14435 10.81 MB/sec warmup 88 sec latency 77.443 ms 2024-04-25T02:09:24.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:23 smithi134 ceph-mon[33098]: pgmap v467: 97 pgs: 97 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 235 op/s 2024-04-25T02:09:24.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:23 smithi031 ceph-mon[29204]: pgmap v467: 97 pgs: 97 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 235 op/s 2024-04-25T02:09:24.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:23 smithi040 ceph-mon[32418]: pgmap v467: 97 pgs: 97 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 235 op/s 2024-04-25T02:09:25.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 14609 10.75 MB/sec warmup 89 sec latency 74.412 ms 2024-04-25T02:09:26.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 14779 10.80 MB/sec warmup 90 sec latency 122.695 ms 2024-04-25T02:09:26.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:25 smithi134 ceph-mon[33098]: pgmap v468: 97 pgs: 97 active+clean; 489 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.7 MiB/s wr, 344 op/s 2024-04-25T02:09:26.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:25 smithi031 ceph-mon[29204]: pgmap v468: 97 pgs: 97 active+clean; 489 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.7 MiB/s wr, 344 op/s 2024-04-25T02:09:26.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:25 smithi040 ceph-mon[32418]: pgmap v468: 97 pgs: 97 active+clean; 489 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.7 MiB/s wr, 344 op/s 2024-04-25T02:09:27.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 14922 10.73 MB/sec warmup 91 sec latency 57.776 ms 2024-04-25T02:09:28.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 15102 10.78 MB/sec warmup 92 sec latency 89.982 ms 2024-04-25T02:09:28.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:27 smithi134 ceph-mon[33098]: pgmap v469: 97 pgs: 97 active+clean; 489 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 224 op/s 2024-04-25T02:09:28.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:27 smithi031 ceph-mon[29204]: pgmap v469: 97 pgs: 97 active+clean; 489 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 224 op/s 2024-04-25T02:09:28.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:27 smithi040 ceph-mon[32418]: pgmap v469: 97 pgs: 97 active+clean; 489 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 224 op/s 2024-04-25T02:09:29.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 15252 10.72 MB/sec warmup 93 sec latency 171.274 ms 2024-04-25T02:09:30.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 15425 10.65 MB/sec warmup 94 sec latency 94.854 ms 2024-04-25T02:09:30.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:29 smithi134 ceph-mon[33098]: pgmap v470: 97 pgs: 97 active+clean; 489 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 224 op/s 2024-04-25T02:09:30.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:29 smithi031 ceph-mon[29204]: pgmap v470: 97 pgs: 97 active+clean; 489 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 224 op/s 2024-04-25T02:09:30.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:29 smithi040 ceph-mon[32418]: pgmap v470: 97 pgs: 97 active+clean; 489 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 224 op/s 2024-04-25T02:09:30.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.882+0000 7fe6a1037640 1 -- 172.21.15.31:0/3893142925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c101760 msgr2=0x7fe69c101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.882+0000 7fe6a1037640 1 --2- 172.21.15.31:0/3893142925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c101760 0x7fe69c101be0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe690009930 tx=0x7fe69002a420 comp rx=0 tx=0).stop 2024-04-25T02:09:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.882+0000 7fe6a1037640 1 -- 172.21.15.31:0/3893142925 shutdown_connections 2024-04-25T02:09:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.882+0000 7fe6a1037640 1 --2- 172.21.15.31:0/3893142925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe69c102120 0x7fe69c10e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.882+0000 7fe6a1037640 1 --2- 172.21.15.31:0/3893142925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c101760 0x7fe69c101be0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.882+0000 7fe6a1037640 1 --2- 172.21.15.31:0/3893142925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe69c107770 0x7fe69c107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.882+0000 7fe6a1037640 1 -- 172.21.15.31:0/3893142925 >> 172.21.15.31:0/3893142925 conn(0x7fe69c0fd1f0 msgr2=0x7fe69c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.882+0000 7fe6a1037640 1 -- 172.21.15.31:0/3893142925 shutdown_connections 2024-04-25T02:09:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.882+0000 7fe6a1037640 1 -- 172.21.15.31:0/3893142925 wait complete. 2024-04-25T02:09:30.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe6a1037640 1 Processor -- start 2024-04-25T02:09:30.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe6a1037640 1 -- start start 2024-04-25T02:09:30.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe6a1037640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe69c101760 0x7fe69c19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe6a1037640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe69c102120 0x7fe69c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe6a1037640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c107770 0x7fe69c19e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe6a1037640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c111a30 con 0x7fe69c107770 2024-04-25T02:09:30.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe6a1037640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c1118b0 con 0x7fe69c101760 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe6a1037640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c111bb0 con 0x7fe69c102120 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe69a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe69c102120 0x7fe69c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe69ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe69c101760 0x7fe69c19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe69a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe69c102120 0x7fe69c19a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:57190/0 (socket says 172.21.15.31:57190) 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe69a59c640 1 -- 172.21.15.31:0/2013153044 learned_addr learned my addr 172.21.15.31:0/2013153044 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.883+0000 7fe69b59e640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c107770 0x7fe69c19e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.884+0000 7fe69ad9d640 1 -- 172.21.15.31:0/2013153044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe69c102120 msgr2=0x7fe69c19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.884+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe69c102120 0x7fe69c19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.884+0000 7fe69ad9d640 1 -- 172.21.15.31:0/2013153044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c107770 msgr2=0x7fe69c19e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.884+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c107770 0x7fe69c19e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.884+0000 7fe69ad9d640 1 -- 172.21.15.31:0/2013153044 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69c19f0c0 con 0x7fe69c101760 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.884+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe69c101760 0x7fe69c19a0d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00e9f0 tx=0x7fe68c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.884+0000 7fe683fff640 1 -- 172.21.15.31:0/2013153044 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c00ccc0 con 0x7fe69c101760 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.884+0000 7fe6a1037640 1 -- 172.21.15.31:0/2013153044 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe69c19f3b0 con 0x7fe69c101760 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.884+0000 7fe683fff640 1 -- 172.21.15.31:0/2013153044 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe68c0108e0 con 0x7fe69c101760 2024-04-25T02:09:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.884+0000 7fe683fff640 1 -- 172.21.15.31:0/2013153044 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c018880 con 0x7fe69c101760 2024-04-25T02:09:30.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.884+0000 7fe6a1037640 1 -- 172.21.15.31:0/2013153044 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe69c1a6c40 con 0x7fe69c101760 2024-04-25T02:09:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.887+0000 7fe683fff640 1 -- 172.21.15.31:0/2013153044 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe68c010a90 con 0x7fe69c101760 2024-04-25T02:09:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.887+0000 7fe6a3e16740 1 Processor -- start 2024-04-25T02:09:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.887+0000 7fe6a3e16740 1 -- start start 2024-04-25T02:09:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75c2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75c2b60 0x55baf75cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75cd230 0x55baf75cf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55baf7526cb0 con 0x55baf75c2240 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55baf74cf5b0 con 0x55baf75c2b60 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55baf7522c70 con 0x55baf75cd230 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe69ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75c2b60 0x55baf75cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe69b59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe69b59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75c2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53512/0 (socket says 172.21.15.31:53512) 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe69b59e640 1 -- 172.21.15.31:0/2883060212 learned_addr learned my addr 172.21.15.31:0/2883060212 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe69a59c640 1 --2- 172.21.15.31:0/2883060212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75cd230 0x55baf75cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe683fff640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe6840a2da0 0x7fe6840a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe69ad9d640 1 -- 172.21.15.31:0/2883060212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75cd230 msgr2=0x55baf75cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/2883060212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75cd230 0x55baf75cf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe69ad9d640 1 -- 172.21.15.31:0/2883060212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 msgr2=0x55baf75c2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:30.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/2883060212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75c2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe69ad9d640 1 -- 172.21.15.31:0/2883060212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baf723b210 con 0x55baf75c2b60 2024-04-25T02:09:30.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.888+0000 7fe69b59e640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe6840a2da0 0x7fe6840a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.889+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/2883060212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75c2b60 0x55baf75cccf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0cce60 tx=0x7fe68c066910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48848de8caf321e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:30.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.889+0000 7fe683fff640 1 -- 172.21.15.31:0/2013153044 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7fe68c0cf6f0 con 0x7fe69c101760 2024-04-25T02:09:30.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.889+0000 7fe6817fa640 1 -- 172.21.15.31:0/2883060212 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c026030 con 0x55baf75c2b60 2024-04-25T02:09:30.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.889+0000 7fe6817fa640 1 -- 172.21.15.31:0/2883060212 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe68c010e00 con 0x55baf75c2b60 2024-04-25T02:09:30.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.889+0000 7fe6817fa640 1 -- 172.21.15.31:0/2883060212 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c0a48a0 con 0x55baf75c2b60 2024-04-25T02:09:30.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.889+0000 7fe69b59e640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe6840a2da0 0x7fe6840a5260 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe694003b10 tx=0x7fe694004ab0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:30.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.890+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2883060212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75c2b60 msgr2=0x55baf75cccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:30.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.890+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/2883060212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75c2b60 0x55baf75cccf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0cce60 tx=0x7fe68c066910 comp rx=0 tx=0).stop 2024-04-25T02:09:30.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.890+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2883060212 shutdown_connections 2024-04-25T02:09:30.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.890+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/2883060212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75cd230 0x55baf75cf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.890+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/2883060212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75c2b60 0x55baf75cccf0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.890+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/2883060212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75c2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.890+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2883060212 >> 172.21.15.31:0/2883060212 conn(0x55baf75be560 msgr2=0x55baf75be940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.890+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2883060212 shutdown_connections 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.890+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2883060212 wait complete. 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.890+0000 7fe6a3e16740 1 -- start start 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.890+0000 7fe6a3e16740 1 Processor -- start 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 0x55baf75ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75cd230 0x55baf75cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55baf75cffc0 con 0x55baf75c2240 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55baf74e2b50 con 0x55baf75cd230 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55baf75d0140 con 0x55baf75c2b60 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe69a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 0x55baf75ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe69a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 0x55baf75ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:57210/0 (socket says 172.21.15.31:57210) 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe69ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe69a59c640 1 -- 172.21.15.31:0/1761877361 learned_addr learned my addr 172.21.15.31:0/1761877361 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe69b59e640 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75cd230 0x55baf75cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe6a3e16740 10 client.0 _ll_register_callbacks cb 0x55baf751dca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe69ad9d640 1 -- 172.21.15.31:0/1761877361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 msgr2=0x55baf75ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 0x55baf75ca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe69ad9d640 1 -- 172.21.15.31:0/1761877361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75cd230 msgr2=0x55baf75cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75cd230 0x55baf75cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe69ad9d640 1 -- 172.21.15.31:0/1761877361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baf7307620 con 0x55baf75c2240 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.891+0000 7fe69a59c640 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 0x55baf75ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.892+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75ca1d0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x55baf75c3800 tx=0x7fe68c066620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6656d161823f1be7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.892+0000 7fe67eff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.892+0000 7fe67eff5640 1 -- 172.21.15.31:0/1761877361 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c06f250 con 0x55baf75c2240 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.892+0000 7fe67eff5640 1 -- 172.21.15.31:0/1761877361 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7fe68c09bf40 con 0x55baf75c2240 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.892+0000 7fe6a3e16740 1 -- 172.21.15.31:0/1761877361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55baf75341f0 con 0x55baf75c2240 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.892+0000 7fe67eff5640 1 -- 172.21.15.31:0/1761877361 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fe68c09bce0 con 0x55baf75c2240 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.892+0000 7fe6a3e16740 10 client.17337 fetch_fsmap learned FSMap version 43 2024-04-25T02:09:30.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.892+0000 7fe6a3e16740 1 -- 172.21.15.31:0/1761877361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55baf75dc1a0 con 0x55baf75c2240 2024-04-25T02:09:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.893+0000 7fe67eff5640 1 -- 172.21.15.31:0/1761877361 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55baf75341f0 con 0x55baf75c2240 2024-04-25T02:09:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.893+0000 7fe67eff5640 1 -- 172.21.15.31:0/1761877361 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fe68c09b130 con 0x55baf75c2240 2024-04-25T02:09:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.893+0000 7fe6a3e16740 10 client.17337 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:09:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.893+0000 7fe6a3e16740 10 client.17337 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.893+0000 7fe6a3e16740 20 client.17337 populate_metadata read hostname 'smithi031' 2024-04-25T02:09:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.893+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55baf75dd4f0 0x55baf75df8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.893+0000 7fe6a3e16740 4 client.17337 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:09:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.893+0000 7fe6a3e16740 1 -- 172.21.15.31:0/1761877361 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55baf752e540 con 0x55baf75dd4f0 2024-04-25T02:09:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.893+0000 7fe69a59c640 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55baf75dd4f0 0x55baf75df8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.894+0000 7fe69a59c640 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55baf75dd4f0 0x55baf75df8e0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7901197a8a95bf4 server_cookie=76c9da66211d1593 in_seq=0 out_seq=0 2024-04-25T02:09:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.894+0000 7fe67eff5640 10 client.17337 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:30.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe67eff5640 1 -- 172.21.15.31:0/1761877361 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55baf752e540 con 0x55baf75dd4f0 2024-04-25T02:09:30.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe67eff5640 10 client.17337 handle_command_reply: tid=0 2024-04-25T02:09:30.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 1 client.17337 shutdown 2024-04-25T02:09:30.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 1 -- 172.21.15.31:0/1761877361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 msgr2=0x55baf75ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75ca1d0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x55baf75c3800 tx=0x7fe68c066620 comp rx=0 tx=0).stop 2024-04-25T02:09:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 1 -- 172.21.15.31:0/1761877361 shutdown_connections 2024-04-25T02:09:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55baf75dd4f0 0x55baf75df8e0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75cd230 0x55baf75cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 0x55baf75ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/1761877361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75ca1d0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe67eff5640 0 client.17337 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 1 -- 172.21.15.31:0/1761877361 >> 172.21.15.31:0/1761877361 conn(0x55baf75be560 msgr2=0x55baf75c4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 1 -- 172.21.15.31:0/1761877361 shutdown_connections 2024-04-25T02:09:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 1 -- 172.21.15.31:0/1761877361 wait complete. 2024-04-25T02:09:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.895+0000 7fe6a3e16740 20 client.17337 trim_cache size 0 max 16384 2024-04-25T02:09:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.918+0000 7fe6a3e16740 1 Processor -- start 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.918+0000 7fe6a3e16740 1 -- start start 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75c2240 0x55baf75cc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2b60 0x55baf75d8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75cd230 0x55baf75e4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c1af440 con 0x55baf75c2b60 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c1b10a0 con 0x55baf75c2240 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c1afb10 con 0x55baf75cd230 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe69a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2b60 0x55baf75d8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe69a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2b60 0x55baf75d8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53524/0 (socket says 172.21.15.31:53524) 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe69a59c640 1 -- 172.21.15.31:0/3468566182 learned_addr learned my addr 172.21.15.31:0/3468566182 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/3468566182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75c2240 0x55baf75cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe69b59e640 1 --2- 172.21.15.31:0/3468566182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75cd230 0x55baf75e4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe69a59c640 1 -- 172.21.15.31:0/3468566182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75cd230 msgr2=0x55baf75e4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe69a59c640 1 --2- 172.21.15.31:0/3468566182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75cd230 0x55baf75e4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe69a59c640 1 -- 172.21.15.31:0/3468566182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75c2240 msgr2=0x55baf75cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:30.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe69a59c640 1 --2- 172.21.15.31:0/3468566182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75c2240 0x55baf75cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe69a59c640 1 -- 172.21.15.31:0/3468566182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baf75da2b0 con 0x55baf75c2b60 2024-04-25T02:09:30.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.919+0000 7fe69a59c640 1 --2- 172.21.15.31:0/3468566182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2b60 0x55baf75d8e80 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe690009230 tx=0x7fe6900058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fef5f09b8d6067d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.920+0000 7fe680ff9640 1 -- 172.21.15.31:0/3468566182 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69004a020 con 0x55baf75c2b60 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.920+0000 7fe680ff9640 1 -- 172.21.15.31:0/3468566182 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe690046340 con 0x55baf75c2b60 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.920+0000 7fe680ff9640 1 -- 172.21.15.31:0/3468566182 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe690035040 con 0x55baf75c2b60 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 -- 172.21.15.31:0/3468566182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2b60 msgr2=0x55baf75d8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/3468566182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2b60 0x55baf75d8e80 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe690009230 tx=0x7fe6900058d0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 -- 172.21.15.31:0/3468566182 shutdown_connections 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/3468566182 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75cd230 0x55baf75e4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/3468566182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2b60 0x55baf75d8e80 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/3468566182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75c2240 0x55baf75cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 -- 172.21.15.31:0/3468566182 >> 172.21.15.31:0/3468566182 conn(0x55baf75be560 msgr2=0x55baf75be940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 -- 172.21.15.31:0/3468566182 shutdown_connections 2024-04-25T02:09:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 -- 172.21.15.31:0/3468566182 wait complete. 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 -- start start 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 Processor -- start 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75d55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 0x55baf75c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75cd230 0x55baf75c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c1b09d0 con 0x55baf75c2240 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c1b0240 con 0x55baf75cd230 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c1ad440 con 0x55baf75c2b60 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.921+0000 7fe6a3e16740 10 client.0 _ll_register_callbacks cb 0x7fe694009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75d55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75d55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53528/0 (socket says 172.21.15.31:53528) 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69ad9d640 1 -- 172.21.15.31:0/2517298601 learned_addr learned my addr 172.21.15.31:0/2517298601 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69a59c640 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 0x55baf75c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69b59e640 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75cd230 0x55baf75c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69ad9d640 1 -- 172.21.15.31:0/2517298601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 msgr2=0x55baf75c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 0x55baf75c6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69ad9d640 1 -- 172.21.15.31:0/2517298601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75cd230 msgr2=0x55baf75c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75cd230 0x55baf75c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69ad9d640 1 -- 172.21.15.31:0/2517298601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baf75c75b0 con 0x55baf75c2240 2024-04-25T02:09:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69b59e640 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75cd230 0x55baf75c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:09:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69a59c640 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 0x55baf75c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe69ad9d640 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75d55f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0cf040 tx=0x7fe68c0cf9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189aec75e3c91700 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe67eff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:09:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe67eff5640 1 -- 172.21.15.31:0/2517298601 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c097f80 con 0x55baf75c2240 2024-04-25T02:09:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe67eff5640 1 -- 172.21.15.31:0/2517298601 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7fe68c010430 con 0x55baf75c2240 2024-04-25T02:09:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.922+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2517298601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe690046340 con 0x55baf75c2240 2024-04-25T02:09:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.923+0000 7fe67eff5640 1 -- 172.21.15.31:0/2517298601 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fe68c09bab0 con 0x55baf75c2240 2024-04-25T02:09:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.923+0000 7fe6a3e16740 10 client.17343 fetch_fsmap learned FSMap version 43 2024-04-25T02:09:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.923+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2517298601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55baf76436c0 con 0x55baf75c2240 2024-04-25T02:09:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.923+0000 7fe67eff5640 1 -- 172.21.15.31:0/2517298601 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe68c06e070 con 0x55baf75c2240 2024-04-25T02:09:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.923+0000 7fe67eff5640 1 -- 172.21.15.31:0/2517298601 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fe68c09ad30 con 0x55baf75c2240 2024-04-25T02:09:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.923+0000 7fe6a3e16740 10 client.17343 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:09:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.923+0000 7fe6a3e16740 10 client.17343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.923+0000 7fe6a3e16740 20 client.17343 populate_metadata read hostname 'smithi031' 2024-04-25T02:09:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.923+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55baf7643c20 0x55baf7646040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.923+0000 7fe6a3e16740 4 client.17343 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:09:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.923+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2517298601 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fe690046340 con 0x55baf7643c20 2024-04-25T02:09:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.924+0000 7fe69a59c640 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55baf7643c20 0x55baf7646040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.924+0000 7fe69a59c640 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55baf7643c20 0x55baf7646040 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e149aee653cff9b5 server_cookie=2ceb7c2a7c9c5529 in_seq=0 out_seq=0 2024-04-25T02:09:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.924+0000 7fe67eff5640 10 client.17343 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe67eff5640 1 -- 172.21.15.31:0/2517298601 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fe690046340 con 0x55baf7643c20 2024-04-25T02:09:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe67eff5640 10 client.17343 handle_command_reply: tid=0 2024-04-25T02:09:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 1 client.17343 shutdown 2024-04-25T02:09:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2517298601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 msgr2=0x55baf75d55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75d55f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0cf040 tx=0x7fe68c0cf9f0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2517298601 shutdown_connections 2024-04-25T02:09:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55baf7643c20 0x55baf7646040 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55baf75cd230 0x55baf75c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe67eff5640 0 client.17343 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55baf75c2b60 0x55baf75c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 1 --2- 172.21.15.31:0/2517298601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baf75c2240 0x55baf75d55f0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2517298601 >> 172.21.15.31:0/2517298601 conn(0x55baf75be560 msgr2=0x55baf75c3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2517298601 shutdown_connections 2024-04-25T02:09:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 1 -- 172.21.15.31:0/2517298601 wait complete. 2024-04-25T02:09:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.967+0000 7fe6a3e16740 20 client.17343 trim_cache size 0 max 16384 2024-04-25T02:09:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:09:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:09:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:09:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:09:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.968+0000 7fe6a1037640 1 -- 172.21.15.31:0/2013153044 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe6840a2da0 msgr2=0x7fe6840a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.968+0000 7fe6a1037640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe6840a2da0 0x7fe6840a5260 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe694003b10 tx=0x7fe694004ab0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.968+0000 7fe6a1037640 1 -- 172.21.15.31:0/2013153044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe69c101760 msgr2=0x7fe69c19a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.968+0000 7fe6a1037640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe69c101760 0x7fe69c19a0d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00e9f0 tx=0x7fe68c00eeb0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.968+0000 7fe6a1037640 1 -- 172.21.15.31:0/2013153044 shutdown_connections 2024-04-25T02:09:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.968+0000 7fe6a1037640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe6840a2da0 0x7fe6840a5260 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.968+0000 7fe6a1037640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe69c107770 0x7fe69c19e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.968+0000 7fe6a1037640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe69c102120 0x7fe69c19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.968+0000 7fe6a1037640 1 --2- 172.21.15.31:0/2013153044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe69c101760 0x7fe69c19a0d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.968+0000 7fe6a1037640 1 -- 172.21.15.31:0/2013153044 >> 172.21.15.31:0/2013153044 conn(0x7fe69c0fd1f0 msgr2=0x7fe69c10ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.968+0000 7fe6a1037640 1 -- 172.21.15.31:0/2013153044 shutdown_connections 2024-04-25T02:09:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:30.969+0000 7fe6a1037640 1 -- 172.21.15.31:0/2013153044 wait complete. 2024-04-25T02:09:31.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 15583 10.59 MB/sec warmup 95 sec latency 130.790 ms 2024-04-25T02:09:32.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 15731 10.52 MB/sec warmup 96 sec latency 59.577 ms 2024-04-25T02:09:32.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:31 smithi134 ceph-mon[33098]: pgmap v471: 97 pgs: 97 active+clean; 487 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 345 op/s 2024-04-25T02:09:32.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:31 smithi031 ceph-mon[29204]: pgmap v471: 97 pgs: 97 active+clean; 487 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 345 op/s 2024-04-25T02:09:32.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:31 smithi040 ceph-mon[32418]: pgmap v471: 97 pgs: 97 active+clean; 487 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 345 op/s 2024-04-25T02:09:33.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 15915 10.54 MB/sec warmup 97 sec latency 57.879 ms 2024-04-25T02:09:34.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16088 10.55 MB/sec warmup 98 sec latency 61.172 ms 2024-04-25T02:09:34.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:33 smithi134 ceph-mon[33098]: pgmap v472: 97 pgs: 97 active+clean; 487 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 229 op/s 2024-04-25T02:09:34.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:33 smithi031 ceph-mon[29204]: pgmap v472: 97 pgs: 97 active+clean; 487 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 229 op/s 2024-04-25T02:09:34.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:33 smithi040 ceph-mon[32418]: pgmap v472: 97 pgs: 97 active+clean; 487 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 229 op/s 2024-04-25T02:09:35.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16282 10.55 MB/sec warmup 99 sec latency 51.822 ms 2024-04-25T02:09:36.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16446 10.60 MB/sec warmup 100 sec latency 319.956 ms 2024-04-25T02:09:36.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:35 smithi134 ceph-mon[33098]: pgmap v473: 97 pgs: 97 active+clean; 492 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.0 MiB/s wr, 352 op/s 2024-04-25T02:09:36.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:35 smithi031 ceph-mon[29204]: pgmap v473: 97 pgs: 97 active+clean; 492 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.0 MiB/s wr, 352 op/s 2024-04-25T02:09:36.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:35 smithi040 ceph-mon[32418]: pgmap v473: 97 pgs: 97 active+clean; 492 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.0 MiB/s wr, 352 op/s 2024-04-25T02:09:37.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16648 10.60 MB/sec warmup 101 sec latency 158.631 ms 2024-04-25T02:09:38.092 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16822 10.60 MB/sec warmup 102 sec latency 169.817 ms 2024-04-25T02:09:38.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:37 smithi134 ceph-mon[33098]: pgmap v474: 97 pgs: 97 active+clean; 492 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 243 op/s 2024-04-25T02:09:38.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:37 smithi031 ceph-mon[29204]: pgmap v474: 97 pgs: 97 active+clean; 492 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 243 op/s 2024-04-25T02:09:38.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:37 smithi040 ceph-mon[32418]: pgmap v474: 97 pgs: 97 active+clean; 492 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 243 op/s 2024-04-25T02:09:39.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16935 10.53 MB/sec warmup 103 sec latency 58.769 ms 2024-04-25T02:09:40.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 17122 10.60 MB/sec warmup 104 sec latency 56.355 ms 2024-04-25T02:09:40.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:39 smithi134 ceph-mon[33098]: pgmap v475: 97 pgs: 97 active+clean; 492 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 243 op/s 2024-04-25T02:09:40.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:39 smithi031 ceph-mon[29204]: pgmap v475: 97 pgs: 97 active+clean; 492 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 243 op/s 2024-04-25T02:09:40.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:39 smithi040 ceph-mon[32418]: pgmap v475: 97 pgs: 97 active+clean; 492 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 243 op/s 2024-04-25T02:09:41.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 17277 10.55 MB/sec warmup 105 sec latency 79.684 ms 2024-04-25T02:09:42.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 17435 10.52 MB/sec warmup 106 sec latency 56.989 ms 2024-04-25T02:09:42.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:41 smithi031 ceph-mon[29204]: pgmap v476: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 345 op/s 2024-04-25T02:09:42.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:41 smithi040 ceph-mon[32418]: pgmap v476: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 345 op/s 2024-04-25T02:09:42.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:41 smithi134 ceph-mon[33098]: pgmap v476: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 345 op/s 2024-04-25T02:09:43.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 17612 10.53 MB/sec warmup 107 sec latency 57.086 ms 2024-04-25T02:09:44.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 17789 10.48 MB/sec warmup 108 sec latency 58.165 ms 2024-04-25T02:09:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.095+0000 7f41c719e640 1 -- 172.21.15.31:0/2569255996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c0106660 msgr2=0x7f41c0106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.095+0000 7f41c719e640 1 --2- 172.21.15.31:0/2569255996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c0106660 0x7f41c0106a40 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f41b4009900 tx=0x7f41b402a480 comp rx=0 tx=0).stop 2024-04-25T02:09:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.095+0000 7f41c719e640 1 -- 172.21.15.31:0/2569255996 shutdown_connections 2024-04-25T02:09:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.095+0000 7f41c719e640 1 --2- 172.21.15.31:0/2569255996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41c0101730 0x7f41c010d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.095+0000 7f41c719e640 1 --2- 172.21.15.31:0/2569255996 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41c0100d70 0x7f41c01011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.095+0000 7f41c719e640 1 --2- 172.21.15.31:0/2569255996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c0106660 0x7f41c0106a40 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.095+0000 7f41c719e640 1 -- 172.21.15.31:0/2569255996 >> 172.21.15.31:0/2569255996 conn(0x7f41c0076bd0 msgr2=0x7f41c00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:44.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.096+0000 7f41c719e640 1 -- 172.21.15.31:0/2569255996 shutdown_connections 2024-04-25T02:09:44.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.096+0000 7f41c719e640 1 -- 172.21.15.31:0/2569255996 wait complete. 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.096+0000 7f41c719e640 1 Processor -- start 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.096+0000 7f41c719e640 1 -- start start 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c719e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c0100d70 0x7f41c019e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c719e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41c0101730 0x7f41c019e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c719e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41c0106660 0x7f41c01a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c719e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41c010f7e0 con 0x7f41c0100d70 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c719e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f41c010f660 con 0x7f41c0106660 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c719e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f41c010f960 con 0x7f41c0101730 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c4f3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c0100d70 0x7f41c019e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41bffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41c0101730 0x7f41c019e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c4f3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c0100d70 0x7f41c019e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49386/0 (socket says 172.21.15.31:49386) 2024-04-25T02:09:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2465075782 learned_addr learned my addr 172.21.15.31:0/2465075782 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c573b640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41c0106660 0x7f41c01a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2465075782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41c0101730 msgr2=0x7f41c019e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41c0101730 0x7f41c019e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2465075782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41c0106660 msgr2=0x7f41c01a2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41c0106660 0x7f41c01a2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2465075782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c01a3480 con 0x7f41c0100d70 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c573b640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41c0106660 0x7f41c01a2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.097+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c0100d70 0x7f41c019e490 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f41b4006010 tx=0x7f41b4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.098+0000 7f41bdffb640 1 -- 172.21.15.31:0/2465075782 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b403f030 con 0x7f41c0100d70 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.098+0000 7f41bdffb640 1 -- 172.21.15.31:0/2465075782 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41b4003aa0 con 0x7f41c0100d70 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.098+0000 7f41bdffb640 1 -- 172.21.15.31:0/2465075782 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b4035070 con 0x7f41c0100d70 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.098+0000 7f41c719e640 1 -- 172.21.15.31:0/2465075782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c01a3710 con 0x7f41c0100d70 2024-04-25T02:09:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.098+0000 7f41c719e640 1 -- 172.21.15.31:0/2465075782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c01ab000 con 0x7f41c0100d70 2024-04-25T02:09:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.101+0000 7f41bdffb640 1 -- 172.21.15.31:0/2465075782 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f41b4042030 con 0x7f41c0100d70 2024-04-25T02:09:44.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.101+0000 7f41c9f87740 1 Processor -- start 2024-04-25T02:09:44.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.101+0000 7f41c9f87740 1 -- start start 2024-04-25T02:09:44.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.101+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0240 0x5615a6ec0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.101+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 0x5615a6ecacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ecb230 0x5615a6ecd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c9f87740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5615a6e24cb0 con 0x5615a6ecb230 2024-04-25T02:09:44.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c9f87740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5615a6dcd5b0 con 0x5615a6ec0b60 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c9f87740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5615a6e20c70 con 0x5615a6ec0240 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c573b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0240 0x5615a6ec0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41bdffb640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b00a2e20 0x7f41b00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c4f3a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 0x5615a6ecacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41bffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ecb230 0x5615a6ecd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c4f3a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 0x5615a6ecacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44346/0 (socket says 172.21.15.31:44346) 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2680422153 learned_addr learned my addr 172.21.15.31:0/2680422153 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c573b640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b00a2e20 0x7f41b00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2680422153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0240 msgr2=0x5615a6ec0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/2680422153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0240 0x5615a6ec0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2680422153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ecb230 msgr2=0x5615a6ecd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/2680422153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ecb230 0x5615a6ecd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2680422153 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615a6b39210 con 0x5615a6ec0b60 2024-04-25T02:09:44.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.102+0000 7f41bdffb640 1 -- 172.21.15.31:0/2465075782 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7f41b40ef8c0 con 0x7f41c0100d70 2024-04-25T02:09:44.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.103+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/2680422153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 0x5615a6ecacf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f41b40c1880 tx=0x7f41b4090b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94b30df51be8cbe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:44.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.103+0000 7f41aaffd640 1 -- 172.21.15.31:0/2680422153 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b4049070 con 0x5615a6ec0b60 2024-04-25T02:09:44.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.103+0000 7f41aaffd640 1 -- 172.21.15.31:0/2680422153 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41b40bbeb0 con 0x5615a6ec0b60 2024-04-25T02:09:44.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.103+0000 7f41aaffd640 1 -- 172.21.15.31:0/2680422153 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b40c4a70 con 0x5615a6ec0b60 2024-04-25T02:09:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c9f87740 1 -- 172.21.15.31:0/2680422153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 msgr2=0x5615a6ecacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c9f87740 1 --2- 172.21.15.31:0/2680422153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 0x5615a6ecacf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f41b40c1880 tx=0x7f41b4090b00 comp rx=0 tx=0).stop 2024-04-25T02:09:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c573b640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b00a2e20 0x7f41b00a52e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009fd0 tx=0x7f41b8004ab0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c9f87740 1 -- 172.21.15.31:0/2680422153 shutdown_connections 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c9f87740 1 --2- 172.21.15.31:0/2680422153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ecb230 0x5615a6ecd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c9f87740 1 --2- 172.21.15.31:0/2680422153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 0x5615a6ecacf0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c9f87740 1 --2- 172.21.15.31:0/2680422153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0240 0x5615a6ec0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c9f87740 1 -- 172.21.15.31:0/2680422153 >> 172.21.15.31:0/2680422153 conn(0x5615a6ebc560 msgr2=0x5615a6ebc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c9f87740 1 -- 172.21.15.31:0/2680422153 shutdown_connections 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c9f87740 1 -- 172.21.15.31:0/2680422153 wait complete. 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c9f87740 1 -- start start 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.104+0000 7f41c9f87740 1 Processor -- start 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ec80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 0x5615a6ec8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 0x5615a6ec8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c9f87740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5615a6ecdde0 con 0x5615a6ec0240 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c9f87740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5615a6de0b50 con 0x5615a6ecb230 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c9f87740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5615a6ecdf60 con 0x5615a6ec0b60 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c9f87740 10 client.0 _ll_register_callbacks cb 0x5615a6e1bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c4f3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ec80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c4f3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ec80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49396/0 (socket says 172.21.15.31:49396) 2024-04-25T02:09:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c4f3a640 1 -- 172.21.15.31:0/578647387 learned_addr learned my addr 172.21.15.31:0/578647387 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41bffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 0x5615a6ec8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c573b640 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 0x5615a6ec8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c4f3a640 1 -- 172.21.15.31:0/578647387 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 msgr2=0x5615a6ec8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 0x5615a6ec8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c4f3a640 1 -- 172.21.15.31:0/578647387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 msgr2=0x5615a6ec8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 0x5615a6ec8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c4f3a640 1 -- 172.21.15.31:0/578647387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615a6c05620 con 0x5615a6ec0240 2024-04-25T02:09:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.105+0000 7f41c573b640 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 0x5615a6ec8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.106+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ec80c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f41b40c2910 tx=0x7f41b40c01b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4627a5e091945fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.106+0000 7f419ffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.106+0000 7f419ffff640 1 -- 172.21.15.31:0/578647387 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b409f060 con 0x5615a6ec0240 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.106+0000 7f419ffff640 1 -- 172.21.15.31:0/578647387 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7f41b40b8520 con 0x5615a6ec0240 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.106+0000 7f41c9f87740 1 -- 172.21.15.31:0/578647387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41b40bbeb0 con 0x5615a6ec0240 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.106+0000 7f419ffff640 1 -- 172.21.15.31:0/578647387 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f41b4049070 con 0x5615a6ec0240 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.106+0000 7f41c9f87740 10 client.17355 fetch_fsmap learned FSMap version 43 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.106+0000 7f41c9f87740 1 -- 172.21.15.31:0/578647387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5615a6ecdde0 con 0x5615a6ec0240 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.106+0000 7f419ffff640 1 -- 172.21.15.31:0/578647387 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41b40a51a0 con 0x5615a6ec0240 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.106+0000 7f419ffff640 1 -- 172.21.15.31:0/578647387 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f41b403bc90 con 0x5615a6ec0240 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.107+0000 7f41c9f87740 10 client.17355 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.107+0000 7f41c9f87740 10 client.17355 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.107+0000 7f41c9f87740 20 client.17355 populate_metadata read hostname 'smithi031' 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.107+0000 7f41c9f87740 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615a6edb200 0x5615a6edd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.107+0000 7f41c9f87740 4 client.17355 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.107+0000 7f41c9f87740 1 -- 172.21.15.31:0/578647387 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f41b40bbeb0 con 0x5615a6edb200 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.107+0000 7f41bffff640 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615a6edb200 0x5615a6edd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.107+0000 7f419ffff640 10 client.17355 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.107+0000 7f41bffff640 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615a6edb200 0x5615a6edd640 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab80dd85049dd48e server_cookie=c0130d4e4554180e in_seq=0 out_seq=0 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f419ffff640 1 -- 172.21.15.31:0/578647387 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f41b40bbeb0 con 0x5615a6edb200 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f419ffff640 10 client.17355 handle_command_reply: tid=0 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f41c9f87740 1 client.17355 shutdown 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f41c9f87740 1 -- 172.21.15.31:0/578647387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 msgr2=0x5615a6ec80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f41c9f87740 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ec80c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f41b40c2910 tx=0x7f41b40c01b0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f41c9f87740 1 -- 172.21.15.31:0/578647387 shutdown_connections 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f41c9f87740 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615a6edb200 0x5615a6edd640 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f41c9f87740 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 0x5615a6ec8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f41c9f87740 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 0x5615a6ec8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f419ffff640 0 client.17355 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f41c9f87740 1 --2- 172.21.15.31:0/578647387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ec80c0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.108+0000 7f41c9f87740 1 -- 172.21.15.31:0/578647387 >> 172.21.15.31:0/578647387 conn(0x5615a6ebc560 msgr2=0x5615a6ec2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.109+0000 7f41c9f87740 1 -- 172.21.15.31:0/578647387 shutdown_connections 2024-04-25T02:09:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.109+0000 7f41c9f87740 1 -- 172.21.15.31:0/578647387 wait complete. 2024-04-25T02:09:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.109+0000 7f41c9f87740 20 client.17355 trim_cache size 0 max 16384 2024-04-25T02:09:44.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c9f87740 1 Processor -- start 2024-04-25T02:09:44.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c9f87740 1 -- start start 2024-04-25T02:09:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6eca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 0x5615a6ed6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ecb230 0x5615a6ec7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c9f87740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01b3850 con 0x5615a6ec0240 2024-04-25T02:09:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c9f87740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01b54b0 con 0x5615a6ec0b60 2024-04-25T02:09:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c9f87740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01b3f20 con 0x5615a6ecb230 2024-04-25T02:09:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c4f3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6eca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c4f3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6eca7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49410/0 (socket says 172.21.15.31:49410) 2024-04-25T02:09:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2104084159 learned_addr learned my addr 172.21.15.31:0/2104084159 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41c573b640 1 --2- 172.21.15.31:0/2104084159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ecb230 0x5615a6ec7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.134+0000 7f41bffff640 1 --2- 172.21.15.31:0/2104084159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 0x5615a6ed6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.135+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2104084159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ecb230 msgr2=0x5615a6ec7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.135+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/2104084159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ecb230 0x5615a6ec7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.135+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2104084159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 msgr2=0x5615a6ed6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.135+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/2104084159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 0x5615a6ed6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.135+0000 7f41c4f3a640 1 -- 172.21.15.31:0/2104084159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615a6ec1700 con 0x5615a6ec0240 2024-04-25T02:09:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.135+0000 7f41bffff640 1 --2- 172.21.15.31:0/2104084159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 0x5615a6ed6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.135+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/2104084159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6eca7f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f41b403d040 tx=0x7f41b40ba3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befff9cfb2d6773c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.135+0000 7f41aa7fc640 1 -- 172.21.15.31:0/2104084159 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b409f040 con 0x5615a6ec0240 2024-04-25T02:09:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.135+0000 7f41aa7fc640 1 -- 172.21.15.31:0/2104084159 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41b40a51a0 con 0x5615a6ec0240 2024-04-25T02:09:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.135+0000 7f41aa7fc640 1 -- 172.21.15.31:0/2104084159 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b40bbc80 con 0x5615a6ec0240 2024-04-25T02:09:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.136+0000 7f41c9f87740 1 -- 172.21.15.31:0/2104084159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 msgr2=0x5615a6eca7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.136+0000 7f41c9f87740 1 --2- 172.21.15.31:0/2104084159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6eca7f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f41b403d040 tx=0x7f41b40ba3c0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.136+0000 7f41c9f87740 1 -- 172.21.15.31:0/2104084159 shutdown_connections 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.136+0000 7f41c9f87740 1 --2- 172.21.15.31:0/2104084159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ecb230 0x5615a6ec7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.136+0000 7f41c9f87740 1 --2- 172.21.15.31:0/2104084159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ec0b60 0x5615a6ed6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.136+0000 7f41c9f87740 1 --2- 172.21.15.31:0/2104084159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6eca7f0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.136+0000 7f41c9f87740 1 -- 172.21.15.31:0/2104084159 >> 172.21.15.31:0/2104084159 conn(0x5615a6ebc560 msgr2=0x5615a6ec1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.136+0000 7f41c9f87740 1 -- 172.21.15.31:0/2104084159 shutdown_connections 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.136+0000 7f41c9f87740 1 -- 172.21.15.31:0/2104084159 wait complete. 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.137+0000 7f41c9f87740 1 -- start start 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.137+0000 7f41c9f87740 1 Processor -- start 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.137+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ed2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.137+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 0x5615a6ed3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41c9f87740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 0x5615a6ece470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41c4f3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ed2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41c4f3a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ed2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49416/0 (socket says 172.21.15.31:49416) 2024-04-25T02:09:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41c573b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 0x5615a6ece470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41bffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 0x5615a6ed3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41bffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 0x5615a6ed3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:44502/0 (socket says 172.21.15.31:44502) 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41c573b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 0x5615a6ece470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44366/0 (socket says 172.21.15.31:44366) 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41c4f3a640 1 -- 172.21.15.31:0/1331106052 learned_addr learned my addr 172.21.15.31:0/1331106052 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41c9f87740 1 -- 172.21.15.31:0/1331106052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01b4de0 con 0x5615a6ec0240 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41c9f87740 1 -- 172.21.15.31:0/1331106052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01b4650 con 0x5615a6ecb230 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41c9f87740 1 -- 172.21.15.31:0/1331106052 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01b1850 con 0x5615a6ec0b60 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41bffff640 1 -- 172.21.15.31:0/1331106052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 msgr2=0x5615a6ece470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41bffff640 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 0x5615a6ece470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41bffff640 1 -- 172.21.15.31:0/1331106052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 msgr2=0x5615a6ed2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41bffff640 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ed2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.138+0000 7f41bffff640 1 -- 172.21.15.31:0/1331106052 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615a6ecec70 con 0x5615a6ec0b60 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.139+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ed2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.139+0000 7f41c9f87740 10 client.0 _ll_register_callbacks cb 0x7f41ac00b7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:44.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.139+0000 7f41c9f87740 1 -- 172.21.15.31:0/1331106052 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41b40a51a0 con 0x5615a6ec0b60 2024-04-25T02:09:44.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.139+0000 7f41bffff640 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 0x5615a6ed3400 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x5615a6eddeb0 tx=0x7f41ac00b3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c987cbecbeb31ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:44.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.139+0000 7f419ffff640 10 client.26716 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:09:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.139+0000 7f419ffff640 1 -- 172.21.15.31:0/1331106052 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac005220 con 0x5615a6ec0b60 2024-04-25T02:09:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.139+0000 7f419ffff640 1 -- 172.21.15.31:0/1331106052 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7f41ac0053c0 con 0x5615a6ec0b60 2024-04-25T02:09:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.140+0000 7f419ffff640 1 -- 172.21.15.31:0/1331106052 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f41ac0055c0 con 0x5615a6ec0b60 2024-04-25T02:09:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.140+0000 7f41c9f87740 10 client.26716 fetch_fsmap learned FSMap version 43 2024-04-25T02:09:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.140+0000 7f41c9f87740 1 -- 172.21.15.31:0/1331106052 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({fsmap=43}) -- 0x5615a6f410c0 con 0x5615a6ec0b60 2024-04-25T02:09:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.140+0000 7f419ffff640 1 -- 172.21.15.31:0/1331106052 <== mon.1 v2:172.21.15.134:3300/0 4 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f41ac0047a0 con 0x5615a6ec0b60 2024-04-25T02:09:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.140+0000 7f41c9f87740 10 client.26716 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:09:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.140+0000 7f41c9f87740 10 client.26716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.140+0000 7f41c9f87740 20 client.26716 populate_metadata read hostname 'smithi031' 2024-04-25T02:09:44.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.140+0000 7f41c9f87740 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615a6f417c0 0x5615a6f43be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:44.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.141+0000 7f41c9f87740 4 client.26716 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:09:44.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.141+0000 7f41c9f87740 1 -- 172.21.15.31:0/1331106052 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f41b40a51a0 con 0x5615a6f417c0 2024-04-25T02:09:44.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.141+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615a6f417c0 0x5615a6f43be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:44.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.141+0000 7f419ffff640 10 client.26716 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:44.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.141+0000 7f41c4f3a640 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615a6f417c0 0x5615a6f43be0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3610150fdb6e0161 server_cookie=5490f7f420cae5df in_seq=0 out_seq=0 2024-04-25T02:09:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.189+0000 7f419ffff640 1 -- 172.21.15.31:0/1331106052 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f41b40a51a0 con 0x5615a6f417c0 2024-04-25T02:09:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.189+0000 7f419ffff640 10 client.26716 handle_command_reply: tid=0 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.189+0000 7f41c9f87740 1 client.26716 shutdown 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.189+0000 7f41c9f87740 1 -- 172.21.15.31:0/1331106052 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 msgr2=0x5615a6ed3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.189+0000 7f41c9f87740 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 0x5615a6ed3400 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x5615a6eddeb0 tx=0x7f41ac00b3f0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.190+0000 7f41c9f87740 1 -- 172.21.15.31:0/1331106052 shutdown_connections 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.190+0000 7f41c9f87740 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5615a6f417c0 0x5615a6f43be0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.190+0000 7f41c9f87740 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615a6ecb230 0x5615a6ece470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.190+0000 7f41c9f87740 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a6ec0b60 0x5615a6ed3400 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.190+0000 7f419ffff640 0 client.26716 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.190+0000 7f41c9f87740 1 --2- 172.21.15.31:0/1331106052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5615a6ec0240 0x5615a6ed2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.190+0000 7f41c9f87740 1 -- 172.21.15.31:0/1331106052 >> 172.21.15.31:0/1331106052 conn(0x5615a6ebc560 msgr2=0x5615a6ec2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.190+0000 7f41c9f87740 1 -- 172.21.15.31:0/1331106052 shutdown_connections 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.190+0000 7f41c9f87740 1 -- 172.21.15.31:0/1331106052 wait complete. 2024-04-25T02:09:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.190+0000 7f41c9f87740 20 client.26716 trim_cache size 0 max 16384 2024-04-25T02:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 -- 172.21.15.31:0/2465075782 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b00a2e20 msgr2=0x7f41b00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b00a2e20 0x7f41b00a52e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009fd0 tx=0x7f41b8004ab0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 -- 172.21.15.31:0/2465075782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c0100d70 msgr2=0x7f41c019e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c0100d70 0x7f41c019e490 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f41b4006010 tx=0x7f41b4004270 comp rx=0 tx=0).stop 2024-04-25T02:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 -- 172.21.15.31:0/2465075782 shutdown_connections 2024-04-25T02:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f41b00a2e20 0x7f41b00a52e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41c0106660 0x7f41c01a2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41c0101730 0x7f41c019e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 --2- 172.21.15.31:0/2465075782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41c0100d70 0x7f41c019e490 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 -- 172.21.15.31:0/2465075782 >> 172.21.15.31:0/2465075782 conn(0x7f41c0076bd0 msgr2=0x7f41c00fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 -- 172.21.15.31:0/2465075782 shutdown_connections 2024-04-25T02:09:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:44.191+0000 7f41c719e640 1 -- 172.21.15.31:0/2465075782 wait complete. 2024-04-25T02:09:44.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:43 smithi031 ceph-mon[29204]: pgmap v477: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.3 MiB/s wr, 224 op/s 2024-04-25T02:09:44.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:43 smithi040 ceph-mon[32418]: pgmap v477: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.3 MiB/s wr, 224 op/s 2024-04-25T02:09:44.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:43 smithi134 ceph-mon[33098]: pgmap v477: 97 pgs: 97 active+clean; 503 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.3 MiB/s wr, 224 op/s 2024-04-25T02:09:45.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 17929 10.44 MB/sec warmup 109 sec latency 90.082 ms 2024-04-25T02:09:46.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 18086 10.40 MB/sec warmup 110 sec latency 59.768 ms 2024-04-25T02:09:46.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:45 smithi031 ceph-mon[29204]: pgmap v478: 97 pgs: 97 active+clean; 510 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 342 op/s 2024-04-25T02:09:46.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:45 smithi040 ceph-mon[32418]: pgmap v478: 97 pgs: 97 active+clean; 510 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 342 op/s 2024-04-25T02:09:46.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:45 smithi134 ceph-mon[33098]: pgmap v478: 97 pgs: 97 active+clean; 510 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 342 op/s 2024-04-25T02:09:47.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 18271 10.43 MB/sec warmup 111 sec latency 66.323 ms 2024-04-25T02:09:48.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 18449 10.40 MB/sec warmup 112 sec latency 60.334 ms 2024-04-25T02:09:48.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:47 smithi040 ceph-mon[32418]: pgmap v479: 97 pgs: 97 active+clean; 510 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 219 op/s 2024-04-25T02:09:48.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:47 smithi031 ceph-mon[29204]: pgmap v479: 97 pgs: 97 active+clean; 510 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 219 op/s 2024-04-25T02:09:48.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:47 smithi134 ceph-mon[33098]: pgmap v479: 97 pgs: 97 active+clean; 510 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 219 op/s 2024-04-25T02:09:49.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 18645 10.44 MB/sec warmup 113 sec latency 59.850 ms 2024-04-25T02:09:50.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 18792 10.38 MB/sec warmup 114 sec latency 56.503 ms 2024-04-25T02:09:50.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:49 smithi031 ceph-mon[29204]: pgmap v480: 97 pgs: 97 active+clean; 510 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 219 op/s 2024-04-25T02:09:50.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:49 smithi040 ceph-mon[32418]: pgmap v480: 97 pgs: 97 active+clean; 510 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 219 op/s 2024-04-25T02:09:50.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:49 smithi134 ceph-mon[33098]: pgmap v480: 97 pgs: 97 active+clean; 510 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 219 op/s 2024-04-25T02:09:51.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 18955 10.33 MB/sec warmup 115 sec latency 62.135 ms 2024-04-25T02:09:52.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 19136 10.31 MB/sec warmup 116 sec latency 56.968 ms 2024-04-25T02:09:52.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:51 smithi031 ceph-mon[29204]: pgmap v481: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 346 op/s 2024-04-25T02:09:52.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:51 smithi040 ceph-mon[32418]: pgmap v481: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 346 op/s 2024-04-25T02:09:52.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:51 smithi134 ceph-mon[33098]: pgmap v481: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 346 op/s 2024-04-25T02:09:53.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 19347 10.32 MB/sec warmup 117 sec latency 99.789 ms 2024-04-25T02:09:54.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 19547 10.33 MB/sec warmup 118 sec latency 57.048 ms 2024-04-25T02:09:54.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:53 smithi031 ceph-mon[29204]: pgmap v482: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 244 op/s 2024-04-25T02:09:54.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:53 smithi040 ceph-mon[32418]: pgmap v482: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 244 op/s 2024-04-25T02:09:54.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:53 smithi134 ceph-mon[33098]: pgmap v482: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 244 op/s 2024-04-25T02:09:55.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 19686 10.29 MB/sec warmup 119 sec latency 49.783 ms 2024-04-25T02:09:56.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:55 smithi031 ceph-mon[29204]: pgmap v483: 97 pgs: 97 active+clean; 517 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.0 MiB/s wr, 363 op/s 2024-04-25T02:09:56.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:55 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:09:56.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:55 smithi040 ceph-mon[32418]: pgmap v483: 97 pgs: 97 active+clean; 517 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.0 MiB/s wr, 363 op/s 2024-04-25T02:09:56.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:55 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:09:56.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:55 smithi134 ceph-mon[33098]: pgmap v483: 97 pgs: 97 active+clean; 517 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.0 MiB/s wr, 363 op/s 2024-04-25T02:09:56.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:55 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:09:57.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 20063 14.86 MB/sec execute 1 sec latency 66.554 ms 2024-04-25T02:09:57.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:56 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:09:57.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:56 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:09:57.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:56 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:09:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.325+0000 7f3fcf590640 1 -- 172.21.15.31:0/1807869094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fc8101030 msgr2=0x7f3fc810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.325+0000 7f3fcf590640 1 --2- 172.21.15.31:0/1807869094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fc8101030 0x7f3fc810d4f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009cd0 tx=0x7f3fc002a8e0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.325+0000 7f3fcf590640 1 -- 172.21.15.31:0/1807869094 shutdown_connections 2024-04-25T02:09:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.325+0000 7f3fcf590640 1 --2- 172.21.15.31:0/1807869094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fc8101030 0x7f3fc810d4f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.325+0000 7f3fcf590640 1 --2- 172.21.15.31:0/1807869094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fc8100670 0x7f3fc8100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.325+0000 7f3fcf590640 1 --2- 172.21.15.31:0/1807869094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc8106630 0x7f3fc8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.325+0000 7f3fcf590640 1 -- 172.21.15.31:0/1807869094 >> 172.21.15.31:0/1807869094 conn(0x7f3fc80777c0 msgr2=0x7f3fc8077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.325+0000 7f3fcf590640 1 -- 172.21.15.31:0/1807869094 shutdown_connections 2024-04-25T02:09:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.326+0000 7f3fcf590640 1 -- 172.21.15.31:0/1807869094 wait complete. 2024-04-25T02:09:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.326+0000 7f3fcf590640 1 Processor -- start 2024-04-25T02:09:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.326+0000 7f3fcf590640 1 -- start start 2024-04-25T02:09:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcf590640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fc8100670 0x7f3fc819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcf590640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fc8101030 0x7f3fc819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcf590640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc8106630 0x7f3fc819e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcf590640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc8113c40 con 0x7f3fc8101030 2024-04-25T02:09:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcf590640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc8113ac0 con 0x7f3fc8106630 2024-04-25T02:09:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcf590640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc8113dc0 con 0x7f3fc8100670 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fccb2b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fc8101030 0x7f3fc819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcd32c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fc8100670 0x7f3fc819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fccb2b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fc8101030 0x7f3fc819a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45902/0 (socket says 172.21.15.31:45902) 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcd32c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fc8100670 0x7f3fc819a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59204/0 (socket says 172.21.15.31:59204) 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcdb2d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc8106630 0x7f3fc819e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fccb2b640 1 -- 172.21.15.31:0/935108074 learned_addr learned my addr 172.21.15.31:0/935108074 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcd32c640 1 -- 172.21.15.31:0/935108074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc8106630 msgr2=0x7f3fc819e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcd32c640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc8106630 0x7f3fc819e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcd32c640 1 -- 172.21.15.31:0/935108074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fc8101030 msgr2=0x7f3fc819a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcd32c640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fc8101030 0x7f3fc819a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.327+0000 7f3fcd32c640 1 -- 172.21.15.31:0/935108074 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc819f030 con 0x7f3fc8100670 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.328+0000 7f3fcd32c640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fc8100670 0x7f3fc819a040 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400e9f0 tx=0x7f3fc400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.328+0000 7f3fbe7fc640 1 -- 172.21.15.31:0/935108074 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc400ccc0 con 0x7f3fc8100670 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.328+0000 7f3fcf590640 1 -- 172.21.15.31:0/935108074 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fc819f320 con 0x7f3fc8100670 2024-04-25T02:09:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.328+0000 7f3fbe7fc640 1 -- 172.21.15.31:0/935108074 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fc40108e0 con 0x7f3fc8100670 2024-04-25T02:09:57.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.328+0000 7f3fbe7fc640 1 -- 172.21.15.31:0/935108074 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4018880 con 0x7f3fc8100670 2024-04-25T02:09:57.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.328+0000 7f3fcf590640 1 -- 172.21.15.31:0/935108074 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fc81a6bb0 con 0x7f3fc8100670 2024-04-25T02:09:57.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.332+0000 7f3fd237b740 1 Processor -- start 2024-04-25T02:09:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.332+0000 7f3fd237b740 1 -- start start 2024-04-25T02:09:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.332+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd4578620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.332+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4578b60 0x562fd4582cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.332+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4583230 0x562fd4585620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.332+0000 7f3fd237b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562fd44dccb0 con 0x562fd4578b60 2024-04-25T02:09:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.332+0000 7f3fd237b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562fd44855b0 con 0x562fd4583230 2024-04-25T02:09:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.332+0000 7f3fd237b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562fd44d8c70 con 0x562fd4578240 2024-04-25T02:09:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.333+0000 7f3fcdb2d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd4578620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.333+0000 7f3fcdb2d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd4578620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59212/0 (socket says 172.21.15.31:59212) 2024-04-25T02:09:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.333+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/320029329 learned_addr learned my addr 172.21.15.31:0/320029329 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.333+0000 7f3fccb2b640 1 --2- 172.21.15.31:0/320029329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4583230 0x562fd4585620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.333+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/320029329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4583230 msgr2=0x562fd4585620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.333+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/320029329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4583230 0x562fd4585620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.333+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/320029329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4578b60 msgr2=0x562fd4582cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:09:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.333+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/320029329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4578b60 0x562fd4582cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.333+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/320029329 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fd41f1210 con 0x562fd4578240 2024-04-25T02:09:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.333+0000 7f3fccb2b640 1 --2- 172.21.15.31:0/320029329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4583230 0x562fd4585620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:09:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.333+0000 7f3fbe7fc640 1 -- 172.21.15.31:0/935108074 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3fc4010a90 con 0x7f3fc8100670 2024-04-25T02:09:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.335+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/320029329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd4578620 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009cd0 tx=0x7f3fc0005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5667200fbf3be7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.335+0000 7f3fbe7fc640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3fb40a2e20 0x7f3fb40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.335+0000 7f3fcd32c640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3fb40a2e20 0x7f3fb40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.335+0000 7f3fbe7fc640 1 -- 172.21.15.31:0/935108074 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7f3fc40cf750 con 0x7f3fc8100670 2024-04-25T02:09:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.335+0000 7f3fb37fe640 1 -- 172.21.15.31:0/320029329 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0048070 con 0x562fd4578240 2024-04-25T02:09:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.335+0000 7f3fb37fe640 1 -- 172.21.15.31:0/320029329 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fc0036070 con 0x562fd4578240 2024-04-25T02:09:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.335+0000 7f3fb37fe640 1 -- 172.21.15.31:0/320029329 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0004460 con 0x562fd4578240 2024-04-25T02:09:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.336+0000 7f3fd237b740 1 -- 172.21.15.31:0/320029329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 msgr2=0x562fd4578620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.336+0000 7f3fcd32c640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3fb40a2e20 0x7f3fb40a52e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x562fd4579800 tx=0x7f3fc4009700 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.336+0000 7f3fd237b740 1 --2- 172.21.15.31:0/320029329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd4578620 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009cd0 tx=0x7f3fc0005f80 comp rx=0 tx=0).stop 2024-04-25T02:09:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.336+0000 7f3fd237b740 1 -- 172.21.15.31:0/320029329 shutdown_connections 2024-04-25T02:09:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.336+0000 7f3fd237b740 1 --2- 172.21.15.31:0/320029329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4583230 0x562fd4585620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.336+0000 7f3fd237b740 1 --2- 172.21.15.31:0/320029329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4578b60 0x562fd4582cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.337+0000 7f3fd237b740 1 --2- 172.21.15.31:0/320029329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd4578620 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.337+0000 7f3fd237b740 1 -- 172.21.15.31:0/320029329 >> 172.21.15.31:0/320029329 conn(0x562fd4574560 msgr2=0x562fd4574940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.337+0000 7f3fd237b740 1 -- 172.21.15.31:0/320029329 shutdown_connections 2024-04-25T02:09:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.337+0000 7f3fd237b740 1 -- 172.21.15.31:0/320029329 wait complete. 2024-04-25T02:09:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.337+0000 7f3fd237b740 1 -- start start 2024-04-25T02:09:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.337+0000 7f3fd237b740 1 Processor -- start 2024-04-25T02:09:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4578240 0x562fd4580260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 0x562fd45807a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4583230 0x562fd4580ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fd237b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562fd4596fd0 con 0x562fd4578240 2024-04-25T02:09:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fd237b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562fd4498b50 con 0x562fd4583230 2024-04-25T02:09:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fd237b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562fd4597150 con 0x562fd4578b60 2024-04-25T02:09:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fcdb2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4578240 0x562fd4580260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fcdb2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4578240 0x562fd4580260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45910/0 (socket says 172.21.15.31:45910) 2024-04-25T02:09:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/1492183458 learned_addr learned my addr 172.21.15.31:0/1492183458 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fd237b740 10 client.0 _ll_register_callbacks cb 0x562fd44d3ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fccb2b640 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 0x562fd45807a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4583230 0x562fd4580ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.338+0000 7f3fccb2b640 1 -- 172.21.15.31:0/1492183458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4583230 msgr2=0x562fd4580ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fccb2b640 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4583230 0x562fd4580ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fccb2b640 1 -- 172.21.15.31:0/1492183458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4578240 msgr2=0x562fd4580260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fccb2b640 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4578240 0x562fd4580260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fccb2b640 1 -- 172.21.15.31:0/1492183458 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fd42bd620 con 0x562fd4578b60 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4578240 0x562fd4580260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fccb2b640 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 0x562fd45807a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8077d50 tx=0x7f3fb8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a278248c718672d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/1492183458 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8004020 con 0x562fd4578b60 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fd237b740 1 -- 172.21.15.31:0/1492183458 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fc0036070 con 0x562fd4578b60 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/1492183458 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7f3fb80041c0 con 0x562fd4578b60 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/1492183458 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3fb80043c0 con 0x562fd4578b60 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fd237b740 10 client.26728 fetch_fsmap learned FSMap version 43 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.339+0000 7f3fd237b740 1 -- 172.21.15.31:0/1492183458 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562fd45a3220 con 0x562fd4578b60 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.340+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/1492183458 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fc0036070 con 0x562fd4578b60 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.340+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/1492183458 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f3fb8022030 con 0x562fd4578b60 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.340+0000 7f3fd237b740 10 client.26728 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.340+0000 7f3fd237b740 10 client.26728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.340+0000 7f3fd237b740 20 client.26728 populate_metadata read hostname 'smithi031' 2024-04-25T02:09:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.340+0000 7f3fd237b740 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562fd45a47a0 0x562fd45a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.340+0000 7f3fd237b740 4 client.26728 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:09:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.340+0000 7f3fd237b740 1 -- 172.21.15.31:0/1492183458 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fd44ea1f0 con 0x562fd45a47a0 2024-04-25T02:09:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.340+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562fd45a47a0 0x562fd45a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.341+0000 7f3fb0ff9640 10 client.26728 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.341+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562fd45a47a0 0x562fd45a6c20 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb96ba80db05c07e server_cookie=d63b8aece550dbbf in_seq=0 out_seq=0 2024-04-25T02:09:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.341+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/1492183458 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562fd44ea1f0 con 0x562fd45a47a0 2024-04-25T02:09:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.341+0000 7f3fb0ff9640 10 client.26728 handle_command_reply: tid=0 2024-04-25T02:09:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.341+0000 7f3fd237b740 1 client.26728 shutdown 2024-04-25T02:09:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fd237b740 1 -- 172.21.15.31:0/1492183458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 msgr2=0x562fd45807a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fd237b740 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 0x562fd45807a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8077d50 tx=0x7f3fb8009f90 comp rx=0 tx=0).stop 2024-04-25T02:09:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fd237b740 1 -- 172.21.15.31:0/1492183458 shutdown_connections 2024-04-25T02:09:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fd237b740 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562fd45a47a0 0x562fd45a6c20 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fd237b740 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4583230 0x562fd4580ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fd237b740 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 0x562fd45807a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fd237b740 1 --2- 172.21.15.31:0/1492183458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4578240 0x562fd4580260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fb0ff9640 0 client.26728 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fd237b740 1 -- 172.21.15.31:0/1492183458 >> 172.21.15.31:0/1492183458 conn(0x562fd4574560 msgr2=0x562fd457a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fd237b740 1 -- 172.21.15.31:0/1492183458 shutdown_connections 2024-04-25T02:09:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fd237b740 1 -- 172.21.15.31:0/1492183458 wait complete. 2024-04-25T02:09:57.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.342+0000 7f3fd237b740 20 client.26728 trim_cache size 0 max 16384 2024-04-25T02:09:57.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.368+0000 7f3fd237b740 1 Processor -- start 2024-04-25T02:09:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.368+0000 7f3fd237b740 1 -- start start 2024-04-25T02:09:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.368+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd459fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.368+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4578b60 0x562fd457f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.368+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4583230 0x562fd457aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.368+0000 7f3fd237b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc8073710 con 0x562fd4583230 2024-04-25T02:09:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.368+0000 7f3fd237b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc81a87f0 con 0x562fd4578b60 2024-04-25T02:09:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.368+0000 7f3fd237b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc81a7260 con 0x562fd4578240 2024-04-25T02:09:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.369+0000 7f3fcdb2d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd459fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.369+0000 7f3fcdb2d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd459fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59230/0 (socket says 172.21.15.31:59230) 2024-04-25T02:09:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.369+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/621423501 learned_addr learned my addr 172.21.15.31:0/621423501 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.369+0000 7f3fccb2b640 1 --2- 172.21.15.31:0/621423501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4578b60 0x562fd457f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.369+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/621423501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4583230 0x562fd457aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.369+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/621423501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4578b60 msgr2=0x562fd457f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.369+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/621423501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4578b60 0x562fd457f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.369+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/621423501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4583230 msgr2=0x562fd457aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.369+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/621423501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4583230 0x562fd457aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.369+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/621423501 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fd4578fa0 con 0x562fd4578240 2024-04-25T02:09:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.369+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/621423501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd459fe90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0003dd0 tx=0x7f3fc002a8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b477b8f96e45eaed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.370+0000 7f3fb2ffd640 1 -- 172.21.15.31:0/621423501 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0048040 con 0x562fd4578240 2024-04-25T02:09:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.370+0000 7f3fb2ffd640 1 -- 172.21.15.31:0/621423501 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fc0036070 con 0x562fd4578240 2024-04-25T02:09:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.370+0000 7f3fb2ffd640 1 -- 172.21.15.31:0/621423501 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0004800 con 0x562fd4578240 2024-04-25T02:09:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.371+0000 7f3fd237b740 1 -- 172.21.15.31:0/621423501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 msgr2=0x562fd459fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.371+0000 7f3fd237b740 1 --2- 172.21.15.31:0/621423501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd459fe90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0003dd0 tx=0x7f3fc002a8e0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.371+0000 7f3fd237b740 1 -- 172.21.15.31:0/621423501 shutdown_connections 2024-04-25T02:09:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.371+0000 7f3fd237b740 1 --2- 172.21.15.31:0/621423501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4583230 0x562fd457aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.371+0000 7f3fd237b740 1 --2- 172.21.15.31:0/621423501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4578b60 0x562fd457f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.371+0000 7f3fd237b740 1 --2- 172.21.15.31:0/621423501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578240 0x562fd459fe90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.371+0000 7f3fd237b740 1 -- 172.21.15.31:0/621423501 >> 172.21.15.31:0/621423501 conn(0x562fd4574560 msgr2=0x562fd4572d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.371+0000 7f3fd237b740 1 -- 172.21.15.31:0/621423501 shutdown_connections 2024-04-25T02:09:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.371+0000 7f3fd237b740 1 -- 172.21.15.31:0/621423501 wait complete. 2024-04-25T02:09:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.371+0000 7f3fd237b740 1 -- start start 2024-04-25T02:09:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.372+0000 7f3fd237b740 1 Processor -- start 2024-04-25T02:09:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.372+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4578240 0x562fd459c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.372+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 0x562fd457c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.372+0000 7f3fd237b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4583230 0x562fd457ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fccb2b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 0x562fd457c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fcdb2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4583230 0x562fd457ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fccb2b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 0x562fd457c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59240/0 (socket says 172.21.15.31:59240) 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fccb2b640 1 -- 172.21.15.31:0/3927617968 learned_addr learned my addr 172.21.15.31:0/3927617968 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4578240 0x562fd459c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fd237b740 1 -- 172.21.15.31:0/3927617968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc81a8120 con 0x562fd4583230 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fd237b740 1 -- 172.21.15.31:0/3927617968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc81a7990 con 0x562fd4578240 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fd237b740 1 -- 172.21.15.31:0/3927617968 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc8071710 con 0x562fd4578b60 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fd237b740 10 client.0 _ll_register_callbacks cb 0x7f3fb8009470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/3927617968 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 msgr2=0x562fd457c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 0x562fd457c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/3927617968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4578240 msgr2=0x562fd459c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4578240 0x562fd459c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fcdb2d640 1 -- 172.21.15.31:0/3927617968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fd457d380 con 0x562fd4583230 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fcdb2d640 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4583230 0x562fd457ce40 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0031fd0 tx=0x7f3fc0003c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b283c30a06e74331 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.373+0000 7f3fb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:09:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.374+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/3927617968 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0048070 con 0x562fd4583230 2024-04-25T02:09:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.374+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/3927617968 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8484+0+0 (secure 0 0 0) 0x7f3fc0054310 con 0x562fd4583230 2024-04-25T02:09:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.374+0000 7f3fd237b740 1 -- 172.21.15.31:0/3927617968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fc0036070 con 0x562fd4583230 2024-04-25T02:09:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.375+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/3927617968 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3fc0054510 con 0x562fd4583230 2024-04-25T02:09:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.376+0000 7f3fd237b740 10 client.17376 fetch_fsmap learned FSMap version 43 2024-04-25T02:09:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.376+0000 7f3fd237b740 1 -- 172.21.15.31:0/3927617968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562fd460aa10 con 0x562fd4583230 2024-04-25T02:09:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.376+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/3927617968 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fc0036070 con 0x562fd4583230 2024-04-25T02:09:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.376+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/3927617968 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7f3fc0069070 con 0x562fd4583230 2024-04-25T02:09:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.376+0000 7f3fd237b740 10 client.17376 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:09:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.377+0000 7f3fd237b740 10 client.17376 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.377+0000 7f3fd237b740 20 client.17376 populate_metadata read hostname 'smithi031' 2024-04-25T02:09:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.377+0000 7f3fd237b740 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562fd460ac40 0x562fd460d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.377+0000 7f3fd237b740 4 client.17376 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:09:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.377+0000 7f3fd237b740 1 -- 172.21.15.31:0/3927617968 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562fd44e4540 con 0x562fd460ac40 2024-04-25T02:09:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.377+0000 7f3fccb2b640 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562fd460ac40 0x562fd460d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.377+0000 7f3fccb2b640 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562fd460ac40 0x562fd460d030 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acd6eea7f243b30b server_cookie=4eb171c20c827f65 in_seq=0 out_seq=0 2024-04-25T02:09:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.378+0000 7f3fb0ff9640 10 client.17376 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:57.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:56 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:09:57.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:56 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:09:57.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:56 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:09:57.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.430+0000 7f3fb0ff9640 1 -- 172.21.15.31:0/3927617968 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562fd44e4540 con 0x562fd460ac40 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.430+0000 7f3fb0ff9640 10 client.17376 handle_command_reply: tid=0 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.430+0000 7f3fd237b740 1 client.17376 shutdown 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.430+0000 7f3fd237b740 1 -- 172.21.15.31:0/3927617968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4583230 msgr2=0x562fd457ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.430+0000 7f3fd237b740 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4583230 0x562fd457ce40 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0031fd0 tx=0x7f3fc0003c50 comp rx=0 tx=0).stop 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.430+0000 7f3fd237b740 1 -- 172.21.15.31:0/3927617968 shutdown_connections 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.430+0000 7f3fd237b740 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x562fd460ac40 0x562fd460d030 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.430+0000 7f3fd237b740 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fd4583230 0x562fd457ce40 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.430+0000 7f3fb0ff9640 0 client.17376 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.430+0000 7f3fd237b740 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562fd4578b60 0x562fd457c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.431+0000 7f3fd237b740 1 --2- 172.21.15.31:0/3927617968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562fd4578240 0x562fd459c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.431+0000 7f3fd237b740 1 -- 172.21.15.31:0/3927617968 >> 172.21.15.31:0/3927617968 conn(0x562fd4574560 msgr2=0x562fd45735c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.431+0000 7f3fd237b740 1 -- 172.21.15.31:0/3927617968 shutdown_connections 2024-04-25T02:09:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.431+0000 7f3fd237b740 1 -- 172.21.15.31:0/3927617968 wait complete. 2024-04-25T02:09:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.431+0000 7f3fd237b740 20 client.17376 trim_cache size 0 max 16384 2024-04-25T02:09:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:09:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:09:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:09:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 -- 172.21.15.31:0/935108074 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3fb40a2e20 msgr2=0x7f3fb40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3fb40a2e20 0x7f3fb40a52e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x562fd4579800 tx=0x7f3fc4009700 comp rx=0 tx=0).stop 2024-04-25T02:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 -- 172.21.15.31:0/935108074 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fc8100670 msgr2=0x7f3fc819a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fc8100670 0x7f3fc819a040 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400e9f0 tx=0x7f3fc400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 -- 172.21.15.31:0/935108074 shutdown_connections 2024-04-25T02:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3fb40a2e20 0x7f3fb40a52e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc8106630 0x7f3fc819e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fc8101030 0x7f3fc819a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 --2- 172.21.15.31:0/935108074 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fc8100670 0x7f3fc819a040 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 -- 172.21.15.31:0/935108074 >> 172.21.15.31:0/935108074 conn(0x7f3fc80777c0 msgr2=0x7f3fc8103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 -- 172.21.15.31:0/935108074 shutdown_connections 2024-04-25T02:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:09:57.432+0000 7f3fcf590640 1 -- 172.21.15.31:0/935108074 wait complete. 2024-04-25T02:09:57.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:56 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:09:57.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:56 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:09:57.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:56 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:09:58.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 20283 13.10 MB/sec execute 2 sec latency 83.294 ms 2024-04-25T02:09:58.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:57 smithi031 ceph-mon[29204]: pgmap v484: 97 pgs: 97 active+clean; 517 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 245 op/s 2024-04-25T02:09:58.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:57 smithi040 ceph-mon[32418]: pgmap v484: 97 pgs: 97 active+clean; 517 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 245 op/s 2024-04-25T02:09:58.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:57 smithi134 ceph-mon[33098]: pgmap v484: 97 pgs: 97 active+clean; 517 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 245 op/s 2024-04-25T02:09:59.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 20392 9.45 MB/sec execute 3 sec latency 75.130 ms 2024-04-25T02:10:00.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 20497 7.46 MB/sec execute 4 sec latency 129.685 ms 2024-04-25T02:10:00.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:09:59 smithi031 ceph-mon[29204]: pgmap v485: 97 pgs: 97 active+clean; 517 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 245 op/s 2024-04-25T02:10:00.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a[29200]: 2024-04-25T02:09:59.999+0000 7fc418dd9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:10:00.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:09:59 smithi040 ceph-mon[32418]: pgmap v485: 97 pgs: 97 active+clean; 517 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 245 op/s 2024-04-25T02:10:00.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:09:59 smithi134 ceph-mon[33098]: pgmap v485: 97 pgs: 97 active+clean; 517 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 245 op/s 2024-04-25T02:10:01.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 20660 9.10 MB/sec execute 5 sec latency 200.880 ms 2024-04-25T02:10:01.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:00 smithi031 ceph-mon[29204]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:10:01.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:00 smithi040 ceph-mon[32418]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:10:01.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:00 smithi134 ceph-mon[33098]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:10:02.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 20816 8.76 MB/sec execute 6 sec latency 60.787 ms 2024-04-25T02:10:02.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:01 smithi031 ceph-mon[29204]: pgmap v486: 97 pgs: 97 active+clean; 529 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 357 op/s 2024-04-25T02:10:02.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:01 smithi040 ceph-mon[32418]: pgmap v486: 97 pgs: 97 active+clean; 529 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 357 op/s 2024-04-25T02:10:02.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:01 smithi134 ceph-mon[33098]: pgmap v486: 97 pgs: 97 active+clean; 529 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 357 op/s 2024-04-25T02:10:03.093 INFO:tasks.workunit.client.0.smithi031.stdout: 10 20968 8.40 MB/sec execute 7 sec latency 60.190 ms 2024-04-25T02:10:03.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:02 smithi031 ceph-mon[29204]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T02:10:03.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:02 smithi040 ceph-mon[32418]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T02:10:03.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:02 smithi134 ceph-mon[33098]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T02:10:04.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 21131 8.71 MB/sec execute 8 sec latency 61.904 ms 2024-04-25T02:10:04.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:03 smithi031 ceph-mon[29204]: pgmap v488: 97 pgs: 97 active+clean; 529 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.0 MiB/s wr, 276 op/s 2024-04-25T02:10:04.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:03 smithi040 ceph-mon[32418]: pgmap v488: 97 pgs: 97 active+clean; 529 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.0 MiB/s wr, 276 op/s 2024-04-25T02:10:04.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:03 smithi134 ceph-mon[33098]: pgmap v488: 97 pgs: 97 active+clean; 529 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.0 MiB/s wr, 276 op/s 2024-04-25T02:10:05.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 21308 8.63 MB/sec execute 9 sec latency 49.675 ms 2024-04-25T02:10:06.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 21485 8.69 MB/sec execute 10 sec latency 107.697 ms 2024-04-25T02:10:06.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:05 smithi031 ceph-mon[29204]: pgmap v489: 97 pgs: 97 active+clean; 546 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 287 op/s 2024-04-25T02:10:06.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:05 smithi040 ceph-mon[32418]: pgmap v489: 97 pgs: 97 active+clean; 546 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 287 op/s 2024-04-25T02:10:06.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:05 smithi134 ceph-mon[33098]: pgmap v489: 97 pgs: 97 active+clean; 546 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 287 op/s 2024-04-25T02:10:07.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 21639 8.14 MB/sec execute 11 sec latency 59.278 ms 2024-04-25T02:10:08.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 21811 8.17 MB/sec execute 12 sec latency 85.139 ms 2024-04-25T02:10:08.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:07 smithi031 ceph-mon[29204]: pgmap v490: 97 pgs: 97 active+clean; 546 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 287 op/s 2024-04-25T02:10:08.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:07 smithi040 ceph-mon[32418]: pgmap v490: 97 pgs: 97 active+clean; 546 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 287 op/s 2024-04-25T02:10:08.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:07 smithi134 ceph-mon[33098]: pgmap v490: 97 pgs: 97 active+clean; 546 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 287 op/s 2024-04-25T02:10:09.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 21985 8.06 MB/sec execute 13 sec latency 59.614 ms 2024-04-25T02:10:10.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 22173 8.61 MB/sec execute 14 sec latency 66.398 ms 2024-04-25T02:10:10.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:09 smithi134 ceph-mon[33098]: pgmap v491: 97 pgs: 97 active+clean; 546 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 287 op/s 2024-04-25T02:10:10.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:09 smithi134 ceph-mon[33098]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T02:10:10.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:09 smithi031 ceph-mon[29204]: pgmap v491: 97 pgs: 97 active+clean; 546 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 287 op/s 2024-04-25T02:10:10.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:09 smithi031 ceph-mon[29204]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T02:10:10.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:09 smithi040 ceph-mon[32418]: pgmap v491: 97 pgs: 97 active+clean; 546 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 287 op/s 2024-04-25T02:10:10.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:09 smithi040 ceph-mon[32418]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T02:10:10.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.546+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/764487559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdc4109940 msgr2=0x7fbdc4109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.546+0000 7fbdcb9c1640 1 --2- 172.21.15.31:0/764487559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdc4109940 0x7fbdc4109d20 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0009930 tx=0x7fbdc002a420 comp rx=0 tx=0).stop 2024-04-25T02:10:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.546+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/764487559 shutdown_connections 2024-04-25T02:10:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.546+0000 7fbdcb9c1640 1 --2- 172.21.15.31:0/764487559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdc41042f0 0x7fbdc4110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.546+0000 7fbdcb9c1640 1 --2- 172.21.15.31:0/764487559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdc4103930 0x7fbdc4103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.546+0000 7fbdcb9c1640 1 --2- 172.21.15.31:0/764487559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdc4109940 0x7fbdc4109d20 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.546+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/764487559 >> 172.21.15.31:0/764487559 conn(0x7fbdc40ff400 msgr2=0x7fbdc4101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.546+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/764487559 shutdown_connections 2024-04-25T02:10:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.546+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/764487559 wait complete. 2024-04-25T02:10:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.546+0000 7fbdcb9c1640 1 Processor -- start 2024-04-25T02:10:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdcb9c1640 1 -- start start 2024-04-25T02:10:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdcb9c1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdc4103930 0x7fbdc419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdcb9c1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdc41042f0 0x7fbdc419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdcb9c1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdc419eba0 0x7fbdc419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdcb9c1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc4113c00 con 0x7fbdc4103930 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdcb9c1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc4113a80 con 0x7fbdc419eba0 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdcb9c1640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc4113d80 con 0x7fbdc41042f0 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdc975d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdc4103930 0x7fbdc419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdc975d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdc4103930 0x7fbdc419a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36556/0 (socket says 172.21.15.31:36556) 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdc975d640 1 -- 172.21.15.31:0/44304598 learned_addr learned my addr 172.21.15.31:0/44304598 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdc41042f0 0x7fbdc419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdc9f5e640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdc419eba0 0x7fbdc419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdc975d640 1 -- 172.21.15.31:0/44304598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdc41042f0 msgr2=0x7fbdc419a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdc975d640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdc41042f0 0x7fbdc419a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.547+0000 7fbdc975d640 1 -- 172.21.15.31:0/44304598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdc419eba0 msgr2=0x7fbdc419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.548+0000 7fbdc975d640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdc419eba0 0x7fbdc419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.548+0000 7fbdc975d640 1 -- 172.21.15.31:0/44304598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc419f690 con 0x7fbdc4103930 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.548+0000 7fbdc9f5e640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdc419eba0 0x7fbdc419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.548+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdc41042f0 0x7fbdc419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.548+0000 7fbdc975d640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdc4103930 0x7fbdc419a2e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0005940 tx=0x7fbdc0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.548+0000 7fbdba7fc640 1 -- 172.21.15.31:0/44304598 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc003f070 con 0x7fbdc4103930 2024-04-25T02:10:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.548+0000 7fbdba7fc640 1 -- 172.21.15.31:0/44304598 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbdc0003aa0 con 0x7fbdc4103930 2024-04-25T02:10:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.548+0000 7fbdba7fc640 1 -- 172.21.15.31:0/44304598 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0035070 con 0x7fbdc4103930 2024-04-25T02:10:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.548+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/44304598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc419f920 con 0x7fbdc4103930 2024-04-25T02:10:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.548+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/44304598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc41a7090 con 0x7fbdc4103930 2024-04-25T02:10:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.551+0000 7fbdba7fc640 1 -- 172.21.15.31:0/44304598 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fbdc003b460 con 0x7fbdc4103930 2024-04-25T02:10:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.552+0000 7fbdce7ae740 1 Processor -- start 2024-04-25T02:10:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdce7ae740 1 -- start start 2024-04-25T02:10:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdba7fc640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbdb00a2e20 0x7fbdb00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 0x557adf030620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 0x557adf03acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf03d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557adef94cb0 con 0x557adf030240 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557adef3d5b0 con 0x557adf030b60 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557adef90c70 con 0x557adf03b230 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdc8f5c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 0x557adf03acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdc975d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 0x557adf030620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdc8f5c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 0x557adf03acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33728/0 (socket says 172.21.15.31:33728) 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdc8f5c640 1 -- 172.21.15.31:0/2802342380 learned_addr learned my addr 172.21.15.31:0/2802342380 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdc9f5e640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbdb00a2e20 0x7fbdb00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdc9f5e640 1 --2- 172.21.15.31:0/2802342380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf03d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.553+0000 7fbdba7fc640 1 -- 172.21.15.31:0/44304598 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fbdc00ef760 con 0x7fbdc4103930 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.554+0000 7fbdc8f5c640 1 -- 172.21.15.31:0/2802342380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 msgr2=0x557adf03d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.554+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/2802342380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf03d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.554+0000 7fbdc8f5c640 1 -- 172.21.15.31:0/2802342380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 msgr2=0x557adf030620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:10.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.554+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/2802342380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 0x557adf030620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.554+0000 7fbdc8f5c640 1 -- 172.21.15.31:0/2802342380 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557adeca9210 con 0x557adf030b60 2024-04-25T02:10:10.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.554+0000 7fbdc9f5e640 1 --2- 172.21.15.31:0/2802342380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf03d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:10:10.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.554+0000 7fbdc9f5e640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbdb00a2e20 0x7fbdb00a52e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x557adf02d6f0 tx=0x7fbdbc009f60 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:10.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.554+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/2802342380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 0x557adf03acf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40114f0 tx=0x7fbdb40119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4a19f5eab48d769 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:10.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.554+0000 7fbdaf7fe640 1 -- 172.21.15.31:0/2802342380 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4004020 con 0x557adf030b60 2024-04-25T02:10:10.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.554+0000 7fbdaf7fe640 1 -- 172.21.15.31:0/2802342380 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbdb40041c0 con 0x557adf030b60 2024-04-25T02:10:10.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.554+0000 7fbdaf7fe640 1 -- 172.21.15.31:0/2802342380 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb400f8a0 con 0x557adf030b60 2024-04-25T02:10:10.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.555+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2802342380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 msgr2=0x557adf03acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:10.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.555+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/2802342380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 0x557adf03acf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40114f0 tx=0x7fbdb40119b0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.555+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2802342380 shutdown_connections 2024-04-25T02:10:10.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.555+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/2802342380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf03d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.555+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/2802342380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 0x557adf03acf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.555+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/2802342380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 0x557adf030620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.555+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2802342380 >> 172.21.15.31:0/2802342380 conn(0x557adf02c560 msgr2=0x557adf02c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:10.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.555+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2802342380 shutdown_connections 2024-04-25T02:10:10.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.555+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2802342380 wait complete. 2024-04-25T02:10:10.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.556+0000 7fbdce7ae740 1 -- start start 2024-04-25T02:10:10.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.556+0000 7fbdce7ae740 1 Processor -- start 2024-04-25T02:10:10.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 0x557adf038150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 0x557adf038690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557adf03e020 con 0x557adf030240 2024-04-25T02:10:10.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557adef50b50 con 0x557adf030b60 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557adf03e1a0 con 0x557adf03b230 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdc975d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 0x557adf038150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdc9f5e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdc975d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 0x557adf038150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36576/0 (socket says 172.21.15.31:36576) 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdce7ae740 10 client.0 _ll_register_callbacks cb 0x557adef8bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdc975d640 1 -- 172.21.15.31:0/3498331810 learned_addr learned my addr 172.21.15.31:0/3498331810 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.557+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 0x557adf038690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdc975d640 1 -- 172.21.15.31:0/3498331810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 msgr2=0x557adf038bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdc975d640 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf038bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdc975d640 1 -- 172.21.15.31:0/3498331810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 msgr2=0x557adf038690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdc975d640 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 0x557adf038690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdc975d640 1 -- 172.21.15.31:0/3498331810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557aded75620 con 0x557adf030240 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdc9f5e640 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:10.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdc975d640 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 0x557adf038150 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fbdc00ecf30 tx=0x7fbdc002ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8789a9549f73867f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdacff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdacff9640 1 -- 172.21.15.31:0/3498331810 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0048040 con 0x557adf030240 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdacff9640 1 -- 172.21.15.31:0/3498331810 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fbdc00ed150 con 0x557adf030240 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdce7ae740 1 -- 172.21.15.31:0/3498331810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdb40041c0 con 0x557adf030240 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdacff9640 1 -- 172.21.15.31:0/3498331810 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fbdc00bb6b0 con 0x557adf030240 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdce7ae740 10 client.17388 fetch_fsmap learned FSMap version 43 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.558+0000 7fbdce7ae740 1 -- 172.21.15.31:0/3498331810 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x557adf04a1e0 con 0x557adf030240 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.559+0000 7fbdacff9640 1 -- 172.21.15.31:0/3498331810 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbdb40041c0 con 0x557adf030240 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.559+0000 7fbdacff9640 1 -- 172.21.15.31:0/3498331810 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fbdc004b060 con 0x557adf030240 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.559+0000 7fbdce7ae740 10 client.17388 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.559+0000 7fbdce7ae740 10 client.17388 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.559+0000 7fbdce7ae740 20 client.17388 populate_metadata read hostname 'smithi031' 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.559+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557adf04b690 0x557adf04db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.559+0000 7fbdce7ae740 4 client.17388 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.559+0000 7fbdce7ae740 1 -- 172.21.15.31:0/3498331810 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557adefa21f0 con 0x557adf04b690 2024-04-25T02:10:10.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.559+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557adf04b690 0x557adf04db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.560+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557adf04b690 0x557adf04db30 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ce2fe9861e7fda4 server_cookie=fb6b20647e8e9e94 in_seq=0 out_seq=0 2024-04-25T02:10:10.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.560+0000 7fbdacff9640 10 client.17388 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.560+0000 7fbdacff9640 1 -- 172.21.15.31:0/3498331810 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557adefa21f0 con 0x557adf04b690 2024-04-25T02:10:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.560+0000 7fbdacff9640 10 client.17388 handle_command_reply: tid=0 2024-04-25T02:10:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.560+0000 7fbdce7ae740 1 client.17388 shutdown 2024-04-25T02:10:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdce7ae740 1 -- 172.21.15.31:0/3498331810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 msgr2=0x557adf038150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 0x557adf038150 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fbdc00ecf30 tx=0x7fbdc002ff50 comp rx=0 tx=0).stop 2024-04-25T02:10:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdce7ae740 1 -- 172.21.15.31:0/3498331810 shutdown_connections 2024-04-25T02:10:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557adf04b690 0x557adf04db30 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030b60 0x557adf038690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/3498331810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030240 0x557adf038150 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdce7ae740 1 -- 172.21.15.31:0/3498331810 >> 172.21.15.31:0/3498331810 conn(0x557adf02c560 msgr2=0x557adf032260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdacff9640 0 client.17388 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdce7ae740 1 -- 172.21.15.31:0/3498331810 shutdown_connections 2024-04-25T02:10:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdce7ae740 1 -- 172.21.15.31:0/3498331810 wait complete. 2024-04-25T02:10:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.561+0000 7fbdce7ae740 20 client.17388 trim_cache size 0 max 16384 2024-04-25T02:10:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.590+0000 7fbdce7ae740 1 Processor -- start 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.590+0000 7fbdce7ae740 1 -- start start 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.590+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 0x557adf03a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.590+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf030b60 0x557adf046f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.590+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf03b230 0x557adf053440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.590+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc41ab610 con 0x557adf03b230 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.590+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc41ad270 con 0x557adf030240 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.590+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc41abce0 con 0x557adf030b60 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc975d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 0x557adf03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc8f5c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf030b60 0x557adf046f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc975d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 0x557adf03a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33748/0 (socket says 172.21.15.31:33748) 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc8f5c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf030b60 0x557adf046f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:37644/0 (socket says 172.21.15.31:37644) 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc975d640 1 -- 172.21.15.31:0/4142438235 learned_addr learned my addr 172.21.15.31:0/4142438235 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc9f5e640 1 --2- 172.21.15.31:0/4142438235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf03b230 0x557adf053440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc8f5c640 1 -- 172.21.15.31:0/4142438235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 msgr2=0x557adf03a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/4142438235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 0x557adf03a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc8f5c640 1 -- 172.21.15.31:0/4142438235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf03b230 msgr2=0x557adf053440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/4142438235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf03b230 0x557adf053440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc8f5c640 1 -- 172.21.15.31:0/4142438235 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557adf03d990 con 0x557adf030b60 2024-04-25T02:10:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/4142438235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf030b60 0x557adf046f10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40114c0 tx=0x7fbdb4004500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd5cb6edce6c0df0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:10.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdaeffd640 1 -- 172.21.15.31:0/4142438235 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4004210 con 0x557adf030b60 2024-04-25T02:10:10.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdaeffd640 1 -- 172.21.15.31:0/4142438235 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbdb40054f0 con 0x557adf030b60 2024-04-25T02:10:10.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.591+0000 7fbdaeffd640 1 -- 172.21.15.31:0/4142438235 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb40048c0 con 0x557adf030b60 2024-04-25T02:10:10.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.592+0000 7fbdce7ae740 1 -- 172.21.15.31:0/4142438235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf030b60 msgr2=0x557adf046f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:10.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.592+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/4142438235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf030b60 0x557adf046f10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40114c0 tx=0x7fbdb4004500 comp rx=0 tx=0).stop 2024-04-25T02:10:10.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.592+0000 7fbdce7ae740 1 -- 172.21.15.31:0/4142438235 shutdown_connections 2024-04-25T02:10:10.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.592+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/4142438235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf03b230 0x557adf053440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.592+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/4142438235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf030b60 0x557adf046f10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.592+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/4142438235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 0x557adf03a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.592+0000 7fbdce7ae740 1 -- 172.21.15.31:0/4142438235 >> 172.21.15.31:0/4142438235 conn(0x557adf02c560 msgr2=0x557adf02ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.592+0000 7fbdce7ae740 1 -- 172.21.15.31:0/4142438235 shutdown_connections 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.592+0000 7fbdce7ae740 1 -- 172.21.15.31:0/4142438235 wait complete. 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.593+0000 7fbdce7ae740 1 -- start start 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.593+0000 7fbdce7ae740 1 Processor -- start 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.593+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 0x557adf043420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.593+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030b60 0x557adf043960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.593+0000 7fbdce7ae740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf0343e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.593+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc41acba0 con 0x557adf030b60 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.593+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc41ac410 con 0x557adf030240 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.593+0000 7fbdce7ae740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc41a9610 con 0x557adf03b230 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.593+0000 7fbdce7ae740 10 client.0 _ll_register_callbacks cb 0x7fbdc002eee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc8f5c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030b60 0x557adf043960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc975d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 0x557adf043420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc8f5c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030b60 0x557adf043960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36590/0 (socket says 172.21.15.31:36590) 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc8f5c640 1 -- 172.21.15.31:0/2545535195 learned_addr learned my addr 172.21.15.31:0/2545535195 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc9f5e640 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf0343e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc8f5c640 1 -- 172.21.15.31:0/2545535195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 msgr2=0x557adf0343e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf0343e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc8f5c640 1 -- 172.21.15.31:0/2545535195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 msgr2=0x557adf043420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 0x557adf043420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc8f5c640 1 -- 172.21.15.31:0/2545535195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557adf034920 con 0x557adf030b60 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc975d640 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 0x557adf043420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdc8f5c640 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030b60 0x557adf043960 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4004590 tx=0x7fbdb4005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee3233548ae47f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdacff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdacff9640 1 -- 172.21.15.31:0/2545535195 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4009be0 con 0x557adf030b60 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdacff9640 1 -- 172.21.15.31:0/2545535195 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fbdb4005040 con 0x557adf030b60 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.594+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2545535195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdb40054f0 con 0x557adf030b60 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.595+0000 7fbdacff9640 1 -- 172.21.15.31:0/2545535195 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fbdb4005240 con 0x557adf030b60 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.595+0000 7fbdce7ae740 10 client.17394 fetch_fsmap learned FSMap version 43 2024-04-25T02:10:10.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.595+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2545535195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x557adf0b1480 con 0x557adf030b60 2024-04-25T02:10:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.595+0000 7fbdacff9640 1 -- 172.21.15.31:0/2545535195 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbdb40054f0 con 0x557adf030b60 2024-04-25T02:10:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.595+0000 7fbdacff9640 1 -- 172.21.15.31:0/2545535195 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) ==== 8386+0+0 (secure 0 0 0) 0x7fbdb403f020 con 0x557adf030b60 2024-04-25T02:10:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.595+0000 7fbdce7ae740 10 client.17394 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:10:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.595+0000 7fbdce7ae740 10 client.17394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.595+0000 7fbdce7ae740 20 client.17394 populate_metadata read hostname 'smithi031' 2024-04-25T02:10:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.595+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557adf0b1a80 0x557adf0b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.595+0000 7fbdce7ae740 4 client.17394 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:10:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.595+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2545535195 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557adef9c540 con 0x557adf0b1a80 2024-04-25T02:10:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.596+0000 7fbdc975d640 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557adf0b1a80 0x557adf0b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.596+0000 7fbdc975d640 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557adf0b1a80 0x557adf0b3ea0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fd36c4ec38b1451 server_cookie=4154d0d349619f9a in_seq=0 out_seq=0 2024-04-25T02:10:10.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.596+0000 7fbdacff9640 10 client.17394 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdacff9640 1 -- 172.21.15.31:0/2545535195 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557adef9c540 con 0x557adf0b1a80 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdacff9640 10 client.17394 handle_command_reply: tid=0 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 1 client.17394 shutdown 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2545535195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030b60 msgr2=0x557adf043960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030b60 0x557adf043960 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4004590 tx=0x7fbdb4005730 comp rx=0 tx=0).stop 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2545535195 shutdown_connections 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557adf0b1a80 0x557adf0b3ea0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557adf03b230 0x557adf0343e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdacff9640 0 client.17394 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557adf030b60 0x557adf043960 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 1 --2- 172.21.15.31:0/2545535195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557adf030240 0x557adf043420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2545535195 >> 172.21.15.31:0/2545535195 conn(0x557adf02c560 msgr2=0x557adf036910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2545535195 shutdown_connections 2024-04-25T02:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 1 -- 172.21.15.31:0/2545535195 wait complete. 2024-04-25T02:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.655+0000 7fbdce7ae740 20 client.17394 trim_cache size 0 max 16384 2024-04-25T02:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/44304598 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbdb00a2e20 msgr2=0x7fbdb00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbdb00a2e20 0x7fbdb00a52e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x557adf02d6f0 tx=0x7fbdbc009f60 comp rx=0 tx=0).stop 2024-04-25T02:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/44304598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdc4103930 msgr2=0x7fbdc419a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdc4103930 0x7fbdc419a2e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0005940 tx=0x7fbdc0004040 comp rx=0 tx=0).stop 2024-04-25T02:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/44304598 shutdown_connections 2024-04-25T02:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fbdb00a2e20 0x7fbdb00a52e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbdc419eba0 0x7fbdc419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdc41042f0 0x7fbdc419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 --2- 172.21.15.31:0/44304598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbdc4103930 0x7fbdc419a2e0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/44304598 >> 172.21.15.31:0/44304598 conn(0x7fbdc40ff400 msgr2=0x7fbdc40ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/44304598 shutdown_connections 2024-04-25T02:10:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:10.656+0000 7fbdcb9c1640 1 -- 172.21.15.31:0/44304598 wait complete. 2024-04-25T02:10:11.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 22311 8.26 MB/sec execute 15 sec latency 81.619 ms 2024-04-25T02:10:12.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 22489 8.17 MB/sec execute 16 sec latency 63.711 ms 2024-04-25T02:10:12.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:11 smithi134 ceph-mon[33098]: pgmap v493: 97 pgs: 97 active+clean; 555 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.9 MiB/s wr, 338 op/s 2024-04-25T02:10:12.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:11 smithi031 ceph-mon[29204]: pgmap v493: 97 pgs: 97 active+clean; 555 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.9 MiB/s wr, 338 op/s 2024-04-25T02:10:12.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:11 smithi040 ceph-mon[32418]: pgmap v493: 97 pgs: 97 active+clean; 555 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.9 MiB/s wr, 338 op/s 2024-04-25T02:10:13.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 22681 8.12 MB/sec execute 17 sec latency 53.451 ms 2024-04-25T02:10:14.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 22897 8.41 MB/sec execute 18 sec latency 57.562 ms 2024-04-25T02:10:14.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:13 smithi134 ceph-mon[33098]: pgmap v494: 97 pgs: 97 active+clean; 555 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.2 MiB/s wr, 298 op/s 2024-04-25T02:10:14.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:13 smithi031 ceph-mon[29204]: pgmap v494: 97 pgs: 97 active+clean; 555 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.2 MiB/s wr, 298 op/s 2024-04-25T02:10:14.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:13 smithi040 ceph-mon[32418]: pgmap v494: 97 pgs: 97 active+clean; 555 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.2 MiB/s wr, 298 op/s 2024-04-25T02:10:15.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 23089 8.56 MB/sec execute 19 sec latency 50.380 ms 2024-04-25T02:10:16.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 23223 8.56 MB/sec execute 20 sec latency 60.566 ms 2024-04-25T02:10:16.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:15 smithi134 ceph-mon[33098]: pgmap v495: 97 pgs: 97 active+clean; 569 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 294 op/s 2024-04-25T02:10:16.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:15 smithi031 ceph-mon[29204]: pgmap v495: 97 pgs: 97 active+clean; 569 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 294 op/s 2024-04-25T02:10:16.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:15 smithi040 ceph-mon[32418]: pgmap v495: 97 pgs: 97 active+clean; 569 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 294 op/s 2024-04-25T02:10:17.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 23456 9.43 MB/sec execute 21 sec latency 54.638 ms 2024-04-25T02:10:18.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 23683 9.58 MB/sec execute 22 sec latency 51.441 ms 2024-04-25T02:10:18.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:17 smithi134 ceph-mon[33098]: pgmap v496: 97 pgs: 97 active+clean; 569 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 294 op/s 2024-04-25T02:10:18.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:17 smithi040 ceph-mon[32418]: pgmap v496: 97 pgs: 97 active+clean; 569 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 294 op/s 2024-04-25T02:10:18.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:17 smithi031 ceph-mon[29204]: pgmap v496: 97 pgs: 97 active+clean; 569 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 294 op/s 2024-04-25T02:10:19.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 23830 9.49 MB/sec execute 23 sec latency 52.934 ms 2024-04-25T02:10:20.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 23929 9.16 MB/sec execute 24 sec latency 115.740 ms 2024-04-25T02:10:20.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:19 smithi134 ceph-mon[33098]: pgmap v497: 97 pgs: 97 active+clean; 569 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 294 op/s 2024-04-25T02:10:20.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:19 smithi031 ceph-mon[29204]: pgmap v497: 97 pgs: 97 active+clean; 569 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 294 op/s 2024-04-25T02:10:20.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:19 smithi040 ceph-mon[32418]: pgmap v497: 97 pgs: 97 active+clean; 569 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 294 op/s 2024-04-25T02:10:21.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 24098 9.33 MB/sec execute 25 sec latency 110.184 ms 2024-04-25T02:10:22.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 24262 9.18 MB/sec execute 26 sec latency 100.620 ms 2024-04-25T02:10:22.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:21 smithi134 ceph-mon[33098]: pgmap v498: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 362 op/s 2024-04-25T02:10:22.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:21 smithi031 ceph-mon[29204]: pgmap v498: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 362 op/s 2024-04-25T02:10:22.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:21 smithi040 ceph-mon[32418]: pgmap v498: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 362 op/s 2024-04-25T02:10:23.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 24412 9.04 MB/sec execute 27 sec latency 53.538 ms 2024-04-25T02:10:23.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.787+0000 7fc4f32da640 1 -- 172.21.15.31:0/2520387560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ec102110 msgr2=0x7fc4ec10e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:23.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.787+0000 7fc4f32da640 1 --2- 172.21.15.31:0/2520387560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ec102110 0x7fc4ec10e620 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009c10 tx=0x7fc4e402a6e0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.787+0000 7fc4f32da640 1 -- 172.21.15.31:0/2520387560 shutdown_connections 2024-04-25T02:10:23.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.787+0000 7fc4f32da640 1 --2- 172.21.15.31:0/2520387560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ec102110 0x7fc4ec10e620 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.787+0000 7fc4f32da640 1 --2- 172.21.15.31:0/2520387560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec101750 0x7fc4ec101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.787+0000 7fc4f32da640 1 --2- 172.21.15.31:0/2520387560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec107760 0x7fc4ec107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.787+0000 7fc4f32da640 1 -- 172.21.15.31:0/2520387560 >> 172.21.15.31:0/2520387560 conn(0x7fc4ec0fd240 msgr2=0x7fc4ec0ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:23.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.787+0000 7fc4f32da640 1 -- 172.21.15.31:0/2520387560 shutdown_connections 2024-04-25T02:10:23.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.787+0000 7fc4f32da640 1 -- 172.21.15.31:0/2520387560 wait complete. 2024-04-25T02:10:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f32da640 1 Processor -- start 2024-04-25T02:10:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f32da640 1 -- start start 2024-04-25T02:10:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f32da640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec101750 0x7fc4ec114c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f32da640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec102110 0x7fc4ec111120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f32da640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ec107760 0x7fc4ec111660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f32da640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec074230 con 0x7fc4ec102110 2024-04-25T02:10:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f32da640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec0740b0 con 0x7fc4ec101750 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f32da640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec0743b0 con 0x7fc4ec107760 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f0875640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec102110 0x7fc4ec111120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f0875640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec102110 0x7fc4ec111120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50758/0 (socket says 172.21.15.31:50758) 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f0875640 1 -- 172.21.15.31:0/2139052214 learned_addr learned my addr 172.21.15.31:0/2139052214 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f1076640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec101750 0x7fc4ec114c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f1877640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ec107760 0x7fc4ec111660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f1877640 1 -- 172.21.15.31:0/2139052214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec101750 msgr2=0x7fc4ec114c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.788+0000 7fc4f1877640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec101750 0x7fc4ec114c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.789+0000 7fc4f1877640 1 -- 172.21.15.31:0/2139052214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec102110 msgr2=0x7fc4ec111120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.789+0000 7fc4f1877640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec102110 0x7fc4ec111120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.789+0000 7fc4f1877640 1 -- 172.21.15.31:0/2139052214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ec111d90 con 0x7fc4ec107760 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.789+0000 7fc4f0875640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec102110 0x7fc4ec111120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.789+0000 7fc4f1877640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ec107760 0x7fc4ec111660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4034040 tx=0x7fc4e4003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.789+0000 7fc4e27fc640 1 -- 172.21.15.31:0/2139052214 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4041020 con 0x7fc4ec107760 2024-04-25T02:10:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.789+0000 7fc4e27fc640 1 -- 172.21.15.31:0/2139052214 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4e4037040 con 0x7fc4ec107760 2024-04-25T02:10:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.789+0000 7fc4f32da640 1 -- 172.21.15.31:0/2139052214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ec1aaaa0 con 0x7fc4ec107760 2024-04-25T02:10:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.789+0000 7fc4e27fc640 1 -- 172.21.15.31:0/2139052214 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4036070 con 0x7fc4ec107760 2024-04-25T02:10:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.789+0000 7fc4f32da640 1 -- 172.21.15.31:0/2139052214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ec1aaea0 con 0x7fc4ec107760 2024-04-25T02:10:23.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.792+0000 7fc4e27fc640 1 -- 172.21.15.31:0/2139052214 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc4e4044020 con 0x7fc4ec107760 2024-04-25T02:10:23.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.792+0000 7fc4f5fef740 1 Processor -- start 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f5fef740 1 -- start start 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc7660620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 0x559bc766acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc766d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559bc75c4cb0 con 0x559bc7660240 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559bc756d5b0 con 0x559bc766b230 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559bc75c0c70 con 0x559bc7660b60 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f0875640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc7660620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f0875640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc7660620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50762/0 (socket says 172.21.15.31:50762) 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f0875640 1 -- 172.21.15.31:0/3741654032 learned_addr learned my addr 172.21.15.31:0/3741654032 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f1076640 1 --2- 172.21.15.31:0/3741654032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 0x559bc766acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4e27fc640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc4d80a2e20 0x7fc4d80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f0875640 1 --2- 172.21.15.31:0/3741654032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc766d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f0875640 1 -- 172.21.15.31:0/3741654032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 msgr2=0x559bc766acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f0875640 1 --2- 172.21.15.31:0/3741654032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 0x559bc766acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f1877640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc4d80a2e20 0x7fc4d80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f0875640 1 -- 172.21.15.31:0/3741654032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 msgr2=0x559bc766d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f0875640 1 --2- 172.21.15.31:0/3741654032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc766d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f1076640 1 --2- 172.21.15.31:0/3741654032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 0x559bc766acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:10:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.793+0000 7fc4f0875640 1 -- 172.21.15.31:0/3741654032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bc72d9210 con 0x559bc7660240 2024-04-25T02:10:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.794+0000 7fc4e27fc640 1 -- 172.21.15.31:0/2139052214 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fc4e40efb20 con 0x7fc4ec107760 2024-04-25T02:10:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.794+0000 7fc4f0875640 1 --2- 172.21.15.31:0/3741654032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc7660620 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc014aa0 tx=0x7fc4dc00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1561c03937eb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.794+0000 7fc4d77fe640 1 -- 172.21.15.31:0/3741654032 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc01a030 con 0x559bc7660240 2024-04-25T02:10:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.794+0000 7fc4d77fe640 1 -- 172.21.15.31:0/3741654032 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4dc014170 con 0x559bc7660240 2024-04-25T02:10:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.794+0000 7fc4d77fe640 1 -- 172.21.15.31:0/3741654032 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc005280 con 0x559bc7660240 2024-04-25T02:10:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.794+0000 7fc4f1877640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc4d80a2e20 0x7fc4d80a52e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc4e40ed890 tx=0x7fc4e408e0a0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.795+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3741654032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 msgr2=0x559bc7660620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.795+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/3741654032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc7660620 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc014aa0 tx=0x7fc4dc00bda0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.795+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3741654032 shutdown_connections 2024-04-25T02:10:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.795+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/3741654032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc766d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.795+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/3741654032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 0x559bc766acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.795+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/3741654032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc7660620 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.795+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3741654032 >> 172.21.15.31:0/3741654032 conn(0x559bc765c560 msgr2=0x559bc765c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.795+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3741654032 shutdown_connections 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.795+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3741654032 wait complete. 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.795+0000 7fc4f5fef740 1 -- start start 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f5fef740 1 Processor -- start 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc7668200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 0x559bc7668740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc7668c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559bc766dfc0 con 0x559bc7660240 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559bc7580b50 con 0x559bc766b230 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559bc766e140 con 0x559bc7660b60 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f0875640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc7668200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f1076640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 0x559bc7668740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f5fef740 10 client.0 _ll_register_callbacks cb 0x559bc75bbca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f0875640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc7668200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50766/0 (socket says 172.21.15.31:50766) 2024-04-25T02:10:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f0875640 1 -- 172.21.15.31:0/123749464 learned_addr learned my addr 172.21.15.31:0/123749464 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.796+0000 7fc4f0875640 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc7668c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4f1076640 1 -- 172.21.15.31:0/123749464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 msgr2=0x559bc7668c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4f1076640 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc7668c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4f1076640 1 -- 172.21.15.31:0/123749464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 msgr2=0x559bc7668200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4f1076640 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc7668200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4f1076640 1 -- 172.21.15.31:0/123749464 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bc73a5620 con 0x559bc7660b60 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4f1076640 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 0x559bc7668740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8005fd0 tx=0x7fc4e8011b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cea8cb229cc0c3f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/123749464 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e800fd30 con 0x559bc7660b60 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4f5fef740 1 -- 172.21.15.31:0/123749464 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4dc014170 con 0x559bc7660b60 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/123749464 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fc4e8004030 con 0x559bc7660b60 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/123749464 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fc4e8004230 con 0x559bc7660b60 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4f5fef740 10 client.26764 fetch_fsmap learned FSMap version 44 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.797+0000 7fc4f5fef740 1 -- 172.21.15.31:0/123749464 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x559bc767a200 con 0x559bc7660b60 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.798+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/123749464 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4dc014170 con 0x559bc7660b60 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.798+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/123749464 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 44) ==== 8382+0+0 (secure 0 0 0) 0x7fc4e8021020 con 0x559bc7660b60 2024-04-25T02:10:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.798+0000 7fc4f5fef740 10 client.26764 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T02:10:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.798+0000 7fc4f5fef740 10 client.26764 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.798+0000 7fc4f5fef740 20 client.26764 populate_metadata read hostname 'smithi031' 2024-04-25T02:10:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.798+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559bc767b780 0x559bc767dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.798+0000 7fc4f5fef740 4 client.26764 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:10:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.798+0000 7fc4f5fef740 1 -- 172.21.15.31:0/123749464 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bc75d21f0 con 0x559bc767b780 2024-04-25T02:10:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.798+0000 7fc4f0875640 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559bc767b780 0x559bc767dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.798+0000 7fc4d4ff9640 10 client.26764 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.798+0000 7fc4f0875640 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559bc767b780 0x559bc767dc00 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6e03f919d5c9d55 server_cookie=b6d47ca3153638d8 in_seq=0 out_seq=0 2024-04-25T02:10:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.799+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/123749464 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559bc75d21f0 con 0x559bc767b780 2024-04-25T02:10:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.799+0000 7fc4d4ff9640 10 client.26764 handle_command_reply: tid=0 2024-04-25T02:10:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.799+0000 7fc4f5fef740 1 client.26764 shutdown 2024-04-25T02:10:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.799+0000 7fc4f5fef740 1 -- 172.21.15.31:0/123749464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 msgr2=0x559bc7668740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.799+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 0x559bc7668740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8005fd0 tx=0x7fc4e8011b50 comp rx=0 tx=0).stop 2024-04-25T02:10:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.800+0000 7fc4f5fef740 1 -- 172.21.15.31:0/123749464 shutdown_connections 2024-04-25T02:10:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.800+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559bc767b780 0x559bc767dc00 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.800+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc7668c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.800+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660b60 0x559bc7668740 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.800+0000 7fc4d4ff9640 0 client.26764 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.800+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/123749464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc7668200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.800+0000 7fc4f5fef740 1 -- 172.21.15.31:0/123749464 >> 172.21.15.31:0/123749464 conn(0x559bc765c560 msgr2=0x559bc7662360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.800+0000 7fc4f5fef740 1 -- 172.21.15.31:0/123749464 shutdown_connections 2024-04-25T02:10:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.800+0000 7fc4f5fef740 1 -- 172.21.15.31:0/123749464 wait complete. 2024-04-25T02:10:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.800+0000 7fc4f5fef740 20 client.26764 trim_cache size 0 max 16384 2024-04-25T02:10:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.823+0000 7fc4f5fef740 1 Processor -- start 2024-04-25T02:10:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.823+0000 7fc4f5fef740 1 -- start start 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660240 0x559bc766a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660b60 0x559bc7676e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc7667360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b37a0 con 0x559bc7660b60 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b53d0 con 0x559bc766b230 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b3e70 con 0x559bc7660240 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f0875640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660240 0x559bc766a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f0875640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660240 0x559bc766a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:52272/0 (socket says 172.21.15.31:52272) 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f0875640 1 -- 172.21.15.31:0/1868124518 learned_addr learned my addr 172.21.15.31:0/1868124518 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f1076640 1 --2- 172.21.15.31:0/1868124518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660b60 0x559bc7676e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f0875640 1 --2- 172.21.15.31:0/1868124518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc7667360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f1076640 1 -- 172.21.15.31:0/1868124518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660240 msgr2=0x559bc766a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f1076640 1 --2- 172.21.15.31:0/1868124518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660240 0x559bc766a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f1076640 1 -- 172.21.15.31:0/1868124518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 msgr2=0x559bc7667360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f1076640 1 --2- 172.21.15.31:0/1868124518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc7667360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f1076640 1 -- 172.21.15.31:0/1868124518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bc7663dc0 con 0x559bc7660b60 2024-04-25T02:10:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.824+0000 7fc4f0875640 1 --2- 172.21.15.31:0/1868124518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660240 0x559bc766a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.825+0000 7fc4f1076640 1 --2- 172.21.15.31:0/1868124518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660b60 0x559bc7676e80 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fc4e801a450 tx=0x7fc4e80093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30b57698f060c6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.825+0000 7fc4d6ffd640 1 -- 172.21.15.31:0/1868124518 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8005450 con 0x559bc7660b60 2024-04-25T02:10:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.825+0000 7fc4d6ffd640 1 -- 172.21.15.31:0/1868124518 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4e80055f0 con 0x559bc7660b60 2024-04-25T02:10:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.825+0000 7fc4d6ffd640 1 -- 172.21.15.31:0/1868124518 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8023c40 con 0x559bc7660b60 2024-04-25T02:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.826+0000 7fc4f5fef740 1 -- 172.21.15.31:0/1868124518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660b60 msgr2=0x559bc7676e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.826+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/1868124518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660b60 0x559bc7676e80 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fc4e801a450 tx=0x7fc4e80093f0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.826+0000 7fc4f5fef740 1 -- 172.21.15.31:0/1868124518 shutdown_connections 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.826+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/1868124518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc766b230 0x559bc7667360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.826+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/1868124518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660b60 0x559bc7676e80 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.826+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/1868124518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc7660240 0x559bc766a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.826+0000 7fc4f5fef740 1 -- 172.21.15.31:0/1868124518 >> 172.21.15.31:0/1868124518 conn(0x559bc765c560 msgr2=0x559bc765ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.826+0000 7fc4f5fef740 1 -- 172.21.15.31:0/1868124518 shutdown_connections 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.826+0000 7fc4f5fef740 1 -- 172.21.15.31:0/1868124518 wait complete. 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.826+0000 7fc4f5fef740 1 -- start start 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.826+0000 7fc4f5fef740 1 Processor -- start 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc766e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc7660b60 0x559bc766eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f5fef740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc766b230 0x559bc76dfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b4d00 con 0x559bc7660240 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b4570 con 0x559bc7660b60 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f5fef740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b17a0 con 0x559bc766b230 2024-04-25T02:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f5fef740 10 client.0 _ll_register_callbacks cb 0x7fc4e8009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f0875640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc766e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f1076640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc7660b60 0x559bc766eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f1076640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc7660b60 0x559bc766eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:50552/0 (socket says 172.21.15.31:50552) 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f1076640 1 -- 172.21.15.31:0/3796274203 learned_addr learned my addr 172.21.15.31:0/3796274203 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.827+0000 7fc4f0875640 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc766b230 0x559bc76dfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.828+0000 7fc4f1076640 1 -- 172.21.15.31:0/3796274203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc766b230 msgr2=0x559bc76dfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.828+0000 7fc4f1076640 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc766b230 0x559bc76dfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.828+0000 7fc4f1076640 1 -- 172.21.15.31:0/3796274203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 msgr2=0x559bc766e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.828+0000 7fc4f1076640 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc766e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.828+0000 7fc4f1076640 1 -- 172.21.15.31:0/3796274203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bc76e0340 con 0x559bc7660b60 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.828+0000 7fc4f0875640 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc766e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.828+0000 7fc4f1076640 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc7660b60 0x559bc766eae0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8011b90 tx=0x7fc4e8005570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f2253cab0cd8bae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.828+0000 7fc4d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.828+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/3796274203 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e803a060 con 0x559bc7660b60 2024-04-25T02:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.828+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/3796274203 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fc4e8031040 con 0x559bc7660b60 2024-04-25T02:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.828+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3796274203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4e80055f0 con 0x559bc7660b60 2024-04-25T02:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.829+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/3796274203 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fc4e801a6c0 con 0x559bc7660b60 2024-04-25T02:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.829+0000 7fc4f5fef740 10 client.26984 fetch_fsmap learned FSMap version 44 2024-04-25T02:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.829+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3796274203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x559bc76e1650 con 0x559bc7660b60 2024-04-25T02:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.829+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/3796274203 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4e80055f0 con 0x559bc7660b60 2024-04-25T02:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.829+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/3796274203 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 44) ==== 8382+0+0 (secure 0 0 0) 0x7fc4e8019070 con 0x559bc7660b60 2024-04-25T02:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.829+0000 7fc4f5fef740 10 client.26984 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T02:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.829+0000 7fc4f5fef740 10 client.26984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.829+0000 7fc4f5fef740 20 client.26984 populate_metadata read hostname 'smithi031' 2024-04-25T02:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.829+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559bc76e1d20 0x559bc76e4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.829+0000 7fc4f5fef740 4 client.26984 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.829+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3796274203 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559bc75cc540 con 0x559bc76e1d20 2024-04-25T02:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.830+0000 7fc4f0875640 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559bc76e1d20 0x559bc76e4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.830+0000 7fc4f0875640 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559bc76e1d20 0x559bc76e4140 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dc346d9c7b5d243 server_cookie=1661937be6ad6e74 in_seq=0 out_seq=0 2024-04-25T02:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.830+0000 7fc4d4ff9640 10 client.26984 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.886+0000 7fc4d4ff9640 1 -- 172.21.15.31:0/3796274203 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559bc75cc540 con 0x559bc76e1d20 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.886+0000 7fc4d4ff9640 10 client.26984 handle_command_reply: tid=0 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.886+0000 7fc4f5fef740 1 client.26984 shutdown 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.886+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3796274203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc7660b60 msgr2=0x559bc766eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.887+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc7660b60 0x559bc766eae0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8011b90 tx=0x7fc4e8005570 comp rx=0 tx=0).stop 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.887+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3796274203 shutdown_connections 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.887+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559bc76e1d20 0x559bc76e4140 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.887+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559bc766b230 0x559bc76dfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.887+0000 7fc4d4ff9640 0 client.26984 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.887+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559bc7660b60 0x559bc766eae0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.887+0000 7fc4f5fef740 1 --2- 172.21.15.31:0/3796274203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559bc7660240 0x559bc766e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.887+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3796274203 >> 172.21.15.31:0/3796274203 conn(0x559bc765c560 msgr2=0x559bc765b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.887+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3796274203 shutdown_connections 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.887+0000 7fc4f5fef740 1 -- 172.21.15.31:0/3796274203 wait complete. 2024-04-25T02:10:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.887+0000 7fc4f5fef740 20 client.26984 trim_cache size 0 max 16384 2024-04-25T02:10:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:10:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:10:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:10:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:10:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 -- 172.21.15.31:0/2139052214 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc4d80a2e20 msgr2=0x7fc4d80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc4d80a2e20 0x7fc4d80a52e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc4e40ed890 tx=0x7fc4e408e0a0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 -- 172.21.15.31:0/2139052214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ec107760 msgr2=0x7fc4ec111660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ec107760 0x7fc4ec111660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4034040 tx=0x7fc4e4003f10 comp rx=0 tx=0).stop 2024-04-25T02:10:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 -- 172.21.15.31:0/2139052214 shutdown_connections 2024-04-25T02:10:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc4d80a2e20 0x7fc4d80a52e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4ec107760 0x7fc4ec111660 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec102110 0x7fc4ec111120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 --2- 172.21.15.31:0/2139052214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4ec101750 0x7fc4ec114c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 -- 172.21.15.31:0/2139052214 >> 172.21.15.31:0/2139052214 conn(0x7fc4ec0fd240 msgr2=0x7fc4ec105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 -- 172.21.15.31:0/2139052214 shutdown_connections 2024-04-25T02:10:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:23.888+0000 7fc4f32da640 1 -- 172.21.15.31:0/2139052214 wait complete. 2024-04-25T02:10:24.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 24563 9.02 MB/sec execute 28 sec latency 64.657 ms 2024-04-25T02:10:24.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:23 smithi134 ceph-mon[33098]: pgmap v499: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 237 op/s 2024-04-25T02:10:24.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:23 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:24.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:23 smithi031 ceph-mon[29204]: pgmap v499: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 237 op/s 2024-04-25T02:10:24.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:23 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:24.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:23 smithi040 ceph-mon[32418]: pgmap v499: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 237 op/s 2024-04-25T02:10:24.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:23 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:25.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 24722 8.90 MB/sec execute 29 sec latency 63.393 ms 2024-04-25T02:10:26.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 24895 8.96 MB/sec execute 30 sec latency 82.475 ms 2024-04-25T02:10:26.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:25 smithi031 ceph-mon[29204]: pgmap v500: 97 pgs: 97 active+clean; 587 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 7.1 MiB/s wr, 366 op/s 2024-04-25T02:10:26.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:25 smithi040 ceph-mon[32418]: pgmap v500: 97 pgs: 97 active+clean; 587 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 7.1 MiB/s wr, 366 op/s 2024-04-25T02:10:26.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:25 smithi134 ceph-mon[33098]: pgmap v500: 97 pgs: 97 active+clean; 587 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 7.1 MiB/s wr, 366 op/s 2024-04-25T02:10:27.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 25085 8.99 MB/sec execute 31 sec latency 53.662 ms 2024-04-25T02:10:27.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:26 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:27.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:26 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:27.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:26 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:28.094 INFO:tasks.workunit.client.0.smithi031.stdout: 10 25253 9.01 MB/sec execute 32 sec latency 81.821 ms 2024-04-25T02:10:28.166 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:27 smithi040 ceph-mon[32418]: pgmap v501: 97 pgs: 97 active+clean; 587 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 241 op/s 2024-04-25T02:10:28.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:27 smithi031 ceph-mon[29204]: pgmap v501: 97 pgs: 97 active+clean; 587 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 241 op/s 2024-04-25T02:10:28.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:27 smithi134 ceph-mon[33098]: pgmap v501: 97 pgs: 97 active+clean; 587 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 241 op/s 2024-04-25T02:10:29.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 25414 8.82 MB/sec execute 33 sec latency 120.435 ms 2024-04-25T02:10:30.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 25596 8.99 MB/sec execute 34 sec latency 107.482 ms 2024-04-25T02:10:30.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:29 smithi031 ceph-mon[29204]: pgmap v502: 97 pgs: 97 active+clean; 587 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 241 op/s 2024-04-25T02:10:30.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:29 smithi040 ceph-mon[32418]: pgmap v502: 97 pgs: 97 active+clean; 587 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 241 op/s 2024-04-25T02:10:30.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:29 smithi134 ceph-mon[33098]: pgmap v502: 97 pgs: 97 active+clean; 587 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 241 op/s 2024-04-25T02:10:31.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 25752 8.87 MB/sec execute 35 sec latency 107.703 ms 2024-04-25T02:10:32.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 25927 8.74 MB/sec execute 36 sec latency 124.169 ms 2024-04-25T02:10:32.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:31 smithi031 ceph-mon[29204]: pgmap v503: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.8 MiB/s wr, 365 op/s 2024-04-25T02:10:32.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:31 smithi040 ceph-mon[32418]: pgmap v503: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.8 MiB/s wr, 365 op/s 2024-04-25T02:10:32.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:31 smithi134 ceph-mon[33098]: pgmap v503: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.8 MiB/s wr, 365 op/s 2024-04-25T02:10:33.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 26106 8.77 MB/sec execute 37 sec latency 173.357 ms 2024-04-25T02:10:34.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 26294 8.71 MB/sec execute 38 sec latency 51.747 ms 2024-04-25T02:10:34.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:33 smithi031 ceph-mon[29204]: pgmap v504: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-04-25T02:10:34.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:33 smithi040 ceph-mon[32418]: pgmap v504: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-04-25T02:10:34.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:33 smithi134 ceph-mon[33098]: pgmap v504: 97 pgs: 97 active+clean; 585 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-04-25T02:10:35.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 26477 8.81 MB/sec execute 39 sec latency 58.851 ms 2024-04-25T02:10:36.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 26660 8.86 MB/sec execute 40 sec latency 52.103 ms 2024-04-25T02:10:36.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:35 smithi031 ceph-mon[29204]: pgmap v505: 97 pgs: 97 active+clean; 596 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 361 op/s 2024-04-25T02:10:36.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:35 smithi040 ceph-mon[32418]: pgmap v505: 97 pgs: 97 active+clean; 596 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 361 op/s 2024-04-25T02:10:36.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:35 smithi134 ceph-mon[33098]: pgmap v505: 97 pgs: 97 active+clean; 596 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 361 op/s 2024-04-25T02:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.022+0000 7fe451f14640 1 -- 172.21.15.31:0/100599971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44c100670 msgr2=0x7fe44c100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.022+0000 7fe451f14640 1 --2- 172.21.15.31:0/100599971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44c100670 0x7fe44c100af0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe440009860 tx=0x7fe44002a340 comp rx=0 tx=0).stop 2024-04-25T02:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.022+0000 7fe451f14640 1 -- 172.21.15.31:0/100599971 shutdown_connections 2024-04-25T02:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.022+0000 7fe451f14640 1 --2- 172.21.15.31:0/100599971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c101030 0x7fe44c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.022+0000 7fe451f14640 1 --2- 172.21.15.31:0/100599971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44c100670 0x7fe44c100af0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.022+0000 7fe451f14640 1 --2- 172.21.15.31:0/100599971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe44c106630 0x7fe44c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.022+0000 7fe451f14640 1 -- 172.21.15.31:0/100599971 >> 172.21.15.31:0/100599971 conn(0x7fe44c0777c0 msgr2=0x7fe44c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.022+0000 7fe451f14640 1 -- 172.21.15.31:0/100599971 shutdown_connections 2024-04-25T02:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.022+0000 7fe451f14640 1 -- 172.21.15.31:0/100599971 wait complete. 2024-04-25T02:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.023+0000 7fe451f14640 1 Processor -- start 2024-04-25T02:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.023+0000 7fe451f14640 1 -- start start 2024-04-25T02:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe451f14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44c100670 0x7fe44c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe451f14640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe44c101030 0x7fe44c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe451f14640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c106630 0x7fe44c19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe451f14640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c1108f0 con 0x7fe44c106630 2024-04-25T02:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe451f14640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c110770 con 0x7fe44c100670 2024-04-25T02:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe451f14640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c110a70 con 0x7fe44c101030 2024-04-25T02:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44c100670 0x7fe44c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe44bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c106630 0x7fe44c19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44c100670 0x7fe44c197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51256/0 (socket says 172.21.15.31:51256) 2024-04-25T02:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe44b7fe640 1 -- 172.21.15.31:0/3746630279 learned_addr learned my addr 172.21.15.31:0/3746630279 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe44bfff640 1 -- 172.21.15.31:0/3746630279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe44c101030 msgr2=0x7fe44c198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe44affd640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe44c101030 0x7fe44c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe44bfff640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe44c101030 0x7fe44c198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe44bfff640 1 -- 172.21.15.31:0/3746630279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44c100670 msgr2=0x7fe44c197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe44bfff640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44c100670 0x7fe44c197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.024+0000 7fe44bfff640 1 -- 172.21.15.31:0/3746630279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe44c19cec0 con 0x7fe44c106630 2024-04-25T02:10:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.025+0000 7fe44b7fe640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44c100670 0x7fe44c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.025+0000 7fe44bfff640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c106630 0x7fe44c19c790 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe44400efc0 tx=0x7fe44400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.025+0000 7fe448ff9640 1 -- 172.21.15.31:0/3746630279 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe444003970 con 0x7fe44c106630 2024-04-25T02:10:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.025+0000 7fe448ff9640 1 -- 172.21.15.31:0/3746630279 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe444004020 con 0x7fe44c106630 2024-04-25T02:10:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.025+0000 7fe448ff9640 1 -- 172.21.15.31:0/3746630279 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44400b940 con 0x7fe44c106630 2024-04-25T02:10:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.025+0000 7fe451f14640 1 -- 172.21.15.31:0/3746630279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe44c19d1b0 con 0x7fe44c106630 2024-04-25T02:10:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.025+0000 7fe451f14640 1 -- 172.21.15.31:0/3746630279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe44c1a4a40 con 0x7fe44c106630 2024-04-25T02:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.028+0000 7fe448ff9640 1 -- 172.21.15.31:0/3746630279 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe44401d020 con 0x7fe44c106630 2024-04-25T02:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe454c24740 1 Processor -- start 2024-04-25T02:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe454c24740 1 -- start start 2024-04-25T02:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe454c24740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e00e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe454c24740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00eb60 0x55cd8e018cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe454c24740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 0x55cd8e01b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe454c24740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cd8df72cb0 con 0x55cd8e019230 2024-04-25T02:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe454c24740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cd8df1b5b0 con 0x55cd8e00e240 2024-04-25T02:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe454c24740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55cd8df6ec70 con 0x55cd8e00eb60 2024-04-25T02:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe44affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e00e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00eb60 0x55cd8e018cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe44affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 0x55cd8e01b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00eb60 0x55cd8e018cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:42874/0 (socket says 172.21.15.31:42874) 2024-04-25T02:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe44b7fe640 1 -- 172.21.15.31:0/2339686216 learned_addr learned my addr 172.21.15.31:0/2339686216 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.029+0000 7fe448ff9640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe4340a2f00 0x7fe4340a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe44bfff640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe4340a2f00 0x7fe4340a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe44b7fe640 1 -- 172.21.15.31:0/2339686216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 msgr2=0x55cd8e00e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe44b7fe640 1 --2- 172.21.15.31:0/2339686216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e00e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe44b7fe640 1 -- 172.21.15.31:0/2339686216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 msgr2=0x55cd8e01b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe44b7fe640 1 --2- 172.21.15.31:0/2339686216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 0x55cd8e01b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe44b7fe640 1 -- 172.21.15.31:0/2339686216 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd8dc87210 con 0x55cd8e00eb60 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe44affd640 1 --2- 172.21.15.31:0/2339686216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 0x55cd8e01b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe448ff9640 1 -- 172.21.15.31:0/3746630279 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fe4440cf280 con 0x7fe44c106630 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe44b7fe640 1 --2- 172.21.15.31:0/2339686216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00eb60 0x55cd8e018cf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe43c0115f0 tx=0x7fe43c011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=944554e2567cd49b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe44bfff640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe4340a2f00 0x7fe4340a53c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe4440a3f00 tx=0x7fe444021f20 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe431ffb640 1 -- 172.21.15.31:0/2339686216 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00f8f0 con 0x55cd8e00eb60 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe431ffb640 1 -- 172.21.15.31:0/2339686216 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe43c01a070 con 0x55cd8e00eb60 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.030+0000 7fe431ffb640 1 -- 172.21.15.31:0/2339686216 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c016d00 con 0x55cd8e00eb60 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.031+0000 7fe454c24740 1 -- 172.21.15.31:0/2339686216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00eb60 msgr2=0x55cd8e018cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.031+0000 7fe454c24740 1 --2- 172.21.15.31:0/2339686216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00eb60 0x55cd8e018cf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe43c0115f0 tx=0x7fe43c011ab0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.031+0000 7fe454c24740 1 -- 172.21.15.31:0/2339686216 shutdown_connections 2024-04-25T02:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.031+0000 7fe454c24740 1 --2- 172.21.15.31:0/2339686216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 0x55cd8e01b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.031+0000 7fe454c24740 1 --2- 172.21.15.31:0/2339686216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00eb60 0x55cd8e018cf0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.031+0000 7fe454c24740 1 --2- 172.21.15.31:0/2339686216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e00e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.031+0000 7fe454c24740 1 -- 172.21.15.31:0/2339686216 >> 172.21.15.31:0/2339686216 conn(0x55cd8e00a560 msgr2=0x55cd8e00a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.032+0000 7fe454c24740 1 -- 172.21.15.31:0/2339686216 shutdown_connections 2024-04-25T02:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.032+0000 7fe454c24740 1 -- 172.21.15.31:0/2339686216 wait complete. 2024-04-25T02:10:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.032+0000 7fe454c24740 1 -- start start 2024-04-25T02:10:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.032+0000 7fe454c24740 1 Processor -- start 2024-04-25T02:10:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe454c24740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e016190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe454c24740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e0166d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe454c24740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e019230 0x55cd8e016c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe454c24740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cd8e01c020 con 0x55cd8e00eb60 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe454c24740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cd8df2eb50 con 0x55cd8e00e240 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe454c24740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55cd8e01c1a0 con 0x55cd8e019230 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe454c24740 10 client.0 _ll_register_callbacks cb 0x55cd8df69ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e0166d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e0166d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59546/0 (socket says 172.21.15.31:59546) 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe44b7fe640 1 -- 172.21.15.31:0/1116487250 learned_addr learned my addr 172.21.15.31:0/1116487250 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe44affd640 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e016190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe44affd640 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e019230 0x55cd8e016c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe44b7fe640 1 -- 172.21.15.31:0/1116487250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e019230 msgr2=0x55cd8e016c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe44b7fe640 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e019230 0x55cd8e016c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe44b7fe640 1 -- 172.21.15.31:0/1116487250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 msgr2=0x55cd8e016190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe44b7fe640 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e016190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.033+0000 7fe44b7fe640 1 -- 172.21.15.31:0/1116487250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd8dd53620 con 0x55cd8e00eb60 2024-04-25T02:10:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.034+0000 7fe44b7fe640 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e0166d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe43c016680 tx=0x7fe43c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c93ece3f50fc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.034+0000 7fe42affd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:10:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.034+0000 7fe42affd640 1 -- 172.21.15.31:0/1116487250 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c030070 con 0x55cd8e00eb60 2024-04-25T02:10:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.034+0000 7fe42affd640 1 -- 172.21.15.31:0/1116487250 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fe43c02ccc0 con 0x55cd8e00eb60 2024-04-25T02:10:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.034+0000 7fe454c24740 1 -- 172.21.15.31:0/1116487250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe43c01a070 con 0x55cd8e00eb60 2024-04-25T02:10:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.035+0000 7fe42affd640 1 -- 172.21.15.31:0/1116487250 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe43c03c050 con 0x55cd8e00eb60 2024-04-25T02:10:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.035+0000 7fe454c24740 10 client.17424 fetch_fsmap learned FSMap version 45 2024-04-25T02:10:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.035+0000 7fe454c24740 1 -- 172.21.15.31:0/1116487250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55cd8e0281f0 con 0x55cd8e00eb60 2024-04-25T02:10:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.035+0000 7fe42affd640 1 -- 172.21.15.31:0/1116487250 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe43c01a070 con 0x55cd8e00eb60 2024-04-25T02:10:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.035+0000 7fe42affd640 1 -- 172.21.15.31:0/1116487250 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7fe43c0047e0 con 0x55cd8e00eb60 2024-04-25T02:10:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.035+0000 7fe454c24740 10 client.17424 fetch_fsmap finished waiting for FSMap version 45 2024-04-25T02:10:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.035+0000 7fe454c24740 10 client.17424 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.035+0000 7fe454c24740 20 client.17424 populate_metadata read hostname 'smithi031' 2024-04-25T02:10:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.035+0000 7fe454c24740 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cd8e0296b0 0x55cd8e02bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.035+0000 7fe454c24740 4 client.17424 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:10:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.035+0000 7fe454c24740 1 -- 172.21.15.31:0/1116487250 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd8df801f0 con 0x55cd8e0296b0 2024-04-25T02:10:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.036+0000 7fe44affd640 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cd8e0296b0 0x55cd8e02bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.036+0000 7fe44affd640 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cd8e0296b0 0x55cd8e02bac0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10f59c7a96123df5 server_cookie=36b676de2f92f33d in_seq=0 out_seq=0 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.036+0000 7fe42affd640 10 client.17424 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.037+0000 7fe42affd640 1 -- 172.21.15.31:0/1116487250 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd8df801f0 con 0x55cd8e0296b0 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.037+0000 7fe42affd640 10 client.17424 handle_command_reply: tid=0 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.037+0000 7fe454c24740 1 client.17424 shutdown 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe454c24740 1 -- 172.21.15.31:0/1116487250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 msgr2=0x55cd8e0166d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe454c24740 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e0166d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe43c016680 tx=0x7fe43c005d50 comp rx=0 tx=0).stop 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe454c24740 1 -- 172.21.15.31:0/1116487250 shutdown_connections 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe454c24740 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cd8e0296b0 0x55cd8e02bac0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe454c24740 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e019230 0x55cd8e016c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe454c24740 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e0166d0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe42affd640 0 client.17424 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe454c24740 1 --2- 172.21.15.31:0/1116487250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e016190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe454c24740 1 -- 172.21.15.31:0/1116487250 >> 172.21.15.31:0/1116487250 conn(0x55cd8e00a560 msgr2=0x55cd8e010360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe454c24740 1 -- 172.21.15.31:0/1116487250 shutdown_connections 2024-04-25T02:10:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe454c24740 1 -- 172.21.15.31:0/1116487250 wait complete. 2024-04-25T02:10:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.038+0000 7fe454c24740 20 client.17424 trim_cache size 0 max 16384 2024-04-25T02:10:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe454c24740 1 Processor -- start 2024-04-25T02:10:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe454c24740 1 -- start start 2024-04-25T02:10:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe454c24740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00e240 0x55cd8e024ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe454c24740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00eb60 0x55cd8e010a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe454c24740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 0x55cd8e014a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe454c24740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c1ab180 con 0x55cd8e019230 2024-04-25T02:10:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe454c24740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c1acdb0 con 0x55cd8e00eb60 2024-04-25T02:10:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe454c24740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c1ab850 con 0x55cd8e00e240 2024-04-25T02:10:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe44affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 0x55cd8e014a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00eb60 0x55cd8e010a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe44affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 0x55cd8e014a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59562/0 (socket says 172.21.15.31:59562) 2024-04-25T02:10:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00eb60 0x55cd8e010a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51274/0 (socket says 172.21.15.31:51274) 2024-04-25T02:10:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe44affd640 1 -- 172.21.15.31:0/1711332667 learned_addr learned my addr 172.21.15.31:0/1711332667 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.060+0000 7fe44affd640 1 --2- 172.21.15.31:0/1711332667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00e240 0x55cd8e024ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.061+0000 7fe44affd640 1 -- 172.21.15.31:0/1711332667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00e240 msgr2=0x55cd8e024ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.061+0000 7fe44affd640 1 --2- 172.21.15.31:0/1711332667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00e240 0x55cd8e024ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.061+0000 7fe44affd640 1 -- 172.21.15.31:0/1711332667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00eb60 msgr2=0x55cd8e010a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.061+0000 7fe44affd640 1 --2- 172.21.15.31:0/1711332667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00eb60 0x55cd8e010a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.061+0000 7fe44affd640 1 -- 172.21.15.31:0/1711332667 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd8e015140 con 0x55cd8e019230 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.061+0000 7fe44b7fe640 1 --2- 172.21.15.31:0/1711332667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00eb60 0x55cd8e010a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.061+0000 7fe44affd640 1 --2- 172.21.15.31:0/1711332667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 0x55cd8e014a60 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe440005e00 tx=0x7fe44002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e48c6da9ecd0407 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.061+0000 7fe4317fa640 1 -- 172.21.15.31:0/1711332667 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440037040 con 0x55cd8e019230 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.061+0000 7fe4317fa640 1 -- 172.21.15.31:0/1711332667 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe440030700 con 0x55cd8e019230 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.061+0000 7fe4317fa640 1 -- 172.21.15.31:0/1711332667 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440048740 con 0x55cd8e019230 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.062+0000 7fe454c24740 1 -- 172.21.15.31:0/1711332667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 msgr2=0x55cd8e014a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.062+0000 7fe454c24740 1 --2- 172.21.15.31:0/1711332667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 0x55cd8e014a60 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe440005e00 tx=0x7fe44002c040 comp rx=0 tx=0).stop 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.062+0000 7fe454c24740 1 -- 172.21.15.31:0/1711332667 shutdown_connections 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.062+0000 7fe454c24740 1 --2- 172.21.15.31:0/1711332667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e019230 0x55cd8e014a60 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.062+0000 7fe454c24740 1 --2- 172.21.15.31:0/1711332667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00eb60 0x55cd8e010a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.062+0000 7fe454c24740 1 --2- 172.21.15.31:0/1711332667 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e00e240 0x55cd8e024ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.062+0000 7fe454c24740 1 -- 172.21.15.31:0/1711332667 >> 172.21.15.31:0/1711332667 conn(0x55cd8e00a560 msgr2=0x55cd8e0095c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.062+0000 7fe454c24740 1 -- 172.21.15.31:0/1711332667 shutdown_connections 2024-04-25T02:10:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.062+0000 7fe454c24740 1 -- 172.21.15.31:0/1711332667 wait complete. 2024-04-25T02:10:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.063+0000 7fe454c24740 1 -- start start 2024-04-25T02:10:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.063+0000 7fe454c24740 1 Processor -- start 2024-04-25T02:10:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.063+0000 7fe454c24740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e021900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.063+0000 7fe454c24740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e01cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe454c24740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e019230 0x55cd8e01d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe454c24740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c1ac6e0 con 0x55cd8e00eb60 2024-04-25T02:10:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe454c24740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c1abf50 con 0x55cd8e00e240 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e021900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e01cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe454c24740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c1a9180 con 0x55cd8e019230 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e01cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59570/0 (socket says 172.21.15.31:59570) 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e021900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51282/0 (socket says 172.21.15.31:51282) 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44b7fe640 1 -- 172.21.15.31:0/1450340206 learned_addr learned my addr 172.21.15.31:0/1450340206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44affd640 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e019230 0x55cd8e01d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe454c24740 10 client.0 _ll_register_callbacks cb 0x7fe440035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44b7fe640 1 -- 172.21.15.31:0/1450340206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e019230 msgr2=0x55cd8e01d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44b7fe640 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e019230 0x55cd8e01d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44b7fe640 1 -- 172.21.15.31:0/1450340206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 msgr2=0x55cd8e021900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44b7fe640 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e021900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44b7fe640 1 -- 172.21.15.31:0/1450340206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd8e08ec00 con 0x55cd8e00eb60 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe44b7fe640 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e01cc70 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x55cd8e017760 tx=0x7fe43c00f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d632008d80d960d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe42affd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.064+0000 7fe42affd640 1 -- 172.21.15.31:0/1450340206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c030040 con 0x55cd8e00eb60 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe42affd640 1 -- 172.21.15.31:0/1450340206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fe43c004880 con 0x55cd8e00eb60 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe454c24740 1 -- 172.21.15.31:0/1450340206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe440030700 con 0x55cd8e00eb60 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe42affd640 1 -- 172.21.15.31:0/1450340206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe43c016260 con 0x55cd8e00eb60 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe454c24740 10 client.17430 fetch_fsmap learned FSMap version 45 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe454c24740 1 -- 172.21.15.31:0/1450340206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55cd8e08fd10 con 0x55cd8e00eb60 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe42affd640 1 -- 172.21.15.31:0/1450340206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe440030700 con 0x55cd8e00eb60 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe42affd640 1 -- 172.21.15.31:0/1450340206 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7fe43c00f760 con 0x55cd8e00eb60 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe454c24740 10 client.17430 fetch_fsmap finished waiting for FSMap version 45 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe454c24740 10 client.17430 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe454c24740 20 client.17430 populate_metadata read hostname 'smithi031' 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe454c24740 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cd8e090520 0x55cd8e092940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe454c24740 4 client.17430 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:10:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.065+0000 7fe454c24740 1 -- 172.21.15.31:0/1450340206 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cd8df7a540 con 0x55cd8e090520 2024-04-25T02:10:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.066+0000 7fe44affd640 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cd8e090520 0x55cd8e092940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.066+0000 7fe44affd640 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cd8e090520 0x55cd8e092940 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6400b4f94504f53 server_cookie=738fd16c7fbecda5 in_seq=0 out_seq=0 2024-04-25T02:10:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.066+0000 7fe42affd640 10 client.17430 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:37.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 26881 9.17 MB/sec execute 41 sec latency 57.752 ms 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.128+0000 7fe42affd640 1 -- 172.21.15.31:0/1450340206 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cd8df7a540 con 0x55cd8e090520 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.128+0000 7fe42affd640 10 client.17430 handle_command_reply: tid=0 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.128+0000 7fe454c24740 1 client.17430 shutdown 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.128+0000 7fe454c24740 1 -- 172.21.15.31:0/1450340206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 msgr2=0x55cd8e01cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.128+0000 7fe454c24740 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e01cc70 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x55cd8e017760 tx=0x7fe43c00f490 comp rx=0 tx=0).stop 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe454c24740 1 -- 172.21.15.31:0/1450340206 shutdown_connections 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe454c24740 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cd8e090520 0x55cd8e092940 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe42affd640 0 client.17430 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe454c24740 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cd8e019230 0x55cd8e01d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe454c24740 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd8e00eb60 0x55cd8e01cc70 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe454c24740 1 --2- 172.21.15.31:0/1450340206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cd8e00e240 0x55cd8e021900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe454c24740 1 -- 172.21.15.31:0/1450340206 >> 172.21.15.31:0/1450340206 conn(0x55cd8e00a560 msgr2=0x55cd8e0095c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe454c24740 1 -- 172.21.15.31:0/1450340206 shutdown_connections 2024-04-25T02:10:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe454c24740 1 -- 172.21.15.31:0/1450340206 wait complete. 2024-04-25T02:10:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe454c24740 20 client.17430 trim_cache size 0 max 16384 2024-04-25T02:10:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:10:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:10:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:10:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:10:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe451f14640 1 -- 172.21.15.31:0/3746630279 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe4340a2f00 msgr2=0x7fe4340a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe451f14640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe4340a2f00 0x7fe4340a53c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe4440a3f00 tx=0x7fe444021f20 comp rx=0 tx=0).stop 2024-04-25T02:10:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe451f14640 1 -- 172.21.15.31:0/3746630279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c106630 msgr2=0x7fe44c19c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.129+0000 7fe451f14640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c106630 0x7fe44c19c790 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe44400efc0 tx=0x7fe44400c800 comp rx=0 tx=0).stop 2024-04-25T02:10:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.130+0000 7fe451f14640 1 -- 172.21.15.31:0/3746630279 shutdown_connections 2024-04-25T02:10:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.130+0000 7fe451f14640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe4340a2f00 0x7fe4340a53c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.130+0000 7fe451f14640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe44c106630 0x7fe44c19c790 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.130+0000 7fe451f14640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe44c101030 0x7fe44c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.130+0000 7fe451f14640 1 --2- 172.21.15.31:0/3746630279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44c100670 0x7fe44c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.130+0000 7fe451f14640 1 -- 172.21.15.31:0/3746630279 >> 172.21.15.31:0/3746630279 conn(0x7fe44c0777c0 msgr2=0x7fe44c069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.130+0000 7fe451f14640 1 -- 172.21.15.31:0/3746630279 shutdown_connections 2024-04-25T02:10:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:37.130+0000 7fe451f14640 1 -- 172.21.15.31:0/3746630279 wait complete. 2024-04-25T02:10:38.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 27087 9.31 MB/sec execute 42 sec latency 61.885 ms 2024-04-25T02:10:38.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:37 smithi031 ceph-mon[29204]: pgmap v506: 97 pgs: 97 active+clean; 596 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-04-25T02:10:38.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:37 smithi040 ceph-mon[32418]: pgmap v506: 97 pgs: 97 active+clean; 596 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-04-25T02:10:38.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:37 smithi134 ceph-mon[33098]: pgmap v506: 97 pgs: 97 active+clean; 596 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-04-25T02:10:39.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 27253 9.31 MB/sec execute 43 sec latency 55.785 ms 2024-04-25T02:10:40.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 27364 9.26 MB/sec execute 44 sec latency 79.485 ms 2024-04-25T02:10:40.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:39 smithi031 ceph-mon[29204]: pgmap v507: 97 pgs: 97 active+clean; 596 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-04-25T02:10:40.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:39 smithi040 ceph-mon[32418]: pgmap v507: 97 pgs: 97 active+clean; 596 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-04-25T02:10:40.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:39 smithi134 ceph-mon[33098]: pgmap v507: 97 pgs: 97 active+clean; 596 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-04-25T02:10:41.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 27516 9.30 MB/sec execute 45 sec latency 92.326 ms 2024-04-25T02:10:42.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 27679 9.30 MB/sec execute 46 sec latency 185.959 ms 2024-04-25T02:10:42.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:41 smithi031 ceph-mon[29204]: pgmap v508: 97 pgs: 97 active+clean; 610 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 351 op/s 2024-04-25T02:10:42.247 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:41 smithi134 ceph-mon[33098]: pgmap v508: 97 pgs: 97 active+clean; 610 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 351 op/s 2024-04-25T02:10:42.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:41 smithi040 ceph-mon[32418]: pgmap v508: 97 pgs: 97 active+clean; 610 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 351 op/s 2024-04-25T02:10:43.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 27828 9.19 MB/sec execute 47 sec latency 47.297 ms 2024-04-25T02:10:44.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 27970 9.15 MB/sec execute 48 sec latency 52.970 ms 2024-04-25T02:10:44.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:43 smithi031 ceph-mon[29204]: pgmap v509: 97 pgs: 97 active+clean; 610 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 227 op/s 2024-04-25T02:10:44.245 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:43 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:44.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:43 smithi040 ceph-mon[32418]: pgmap v509: 97 pgs: 97 active+clean; 610 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 227 op/s 2024-04-25T02:10:44.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:43 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:44.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:43 smithi134 ceph-mon[33098]: pgmap v509: 97 pgs: 97 active+clean; 610 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 227 op/s 2024-04-25T02:10:44.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:43 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:45.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 28131 9.05 MB/sec execute 49 sec latency 63.993 ms 2024-04-25T02:10:46.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 28291 9.04 MB/sec execute 50 sec latency 62.822 ms 2024-04-25T02:10:46.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:45 smithi031 ceph-mon[29204]: pgmap v510: 97 pgs: 97 active+clean; 619 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.9 MiB/s wr, 338 op/s 2024-04-25T02:10:46.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:45 smithi040 ceph-mon[32418]: pgmap v510: 97 pgs: 97 active+clean; 619 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.9 MiB/s wr, 338 op/s 2024-04-25T02:10:46.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:45 smithi134 ceph-mon[33098]: pgmap v510: 97 pgs: 97 active+clean; 619 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.9 MiB/s wr, 338 op/s 2024-04-25T02:10:47.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 28452 8.95 MB/sec execute 51 sec latency 63.974 ms 2024-04-25T02:10:48.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 28633 8.98 MB/sec execute 52 sec latency 66.891 ms 2024-04-25T02:10:48.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:47 smithi031 ceph-mon[29204]: pgmap v511: 97 pgs: 97 active+clean; 619 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 229 op/s 2024-04-25T02:10:48.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:47 smithi040 ceph-mon[32418]: pgmap v511: 97 pgs: 97 active+clean; 619 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 229 op/s 2024-04-25T02:10:48.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:47 smithi134 ceph-mon[33098]: pgmap v511: 97 pgs: 97 active+clean; 619 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 229 op/s 2024-04-25T02:10:49.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 28810 8.98 MB/sec execute 53 sec latency 66.139 ms 2024-04-25T02:10:50.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 29007 9.08 MB/sec execute 54 sec latency 59.300 ms 2024-04-25T02:10:50.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:49 smithi031 ceph-mon[29204]: pgmap v512: 97 pgs: 97 active+clean; 619 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 229 op/s 2024-04-25T02:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.263+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1793799406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7d4101030 msgr2=0x7fe7d410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.263+0000 7fe7dbf22640 1 --2- 172.21.15.31:0/1793799406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7d4101030 0x7fe7d410d4f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009cd0 tx=0x7fe7cc02a8e0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.263+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1793799406 shutdown_connections 2024-04-25T02:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.263+0000 7fe7dbf22640 1 --2- 172.21.15.31:0/1793799406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7d4101030 0x7fe7d410d4f0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.263+0000 7fe7dbf22640 1 --2- 172.21.15.31:0/1793799406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7d4100670 0x7fe7d4100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.263+0000 7fe7dbf22640 1 --2- 172.21.15.31:0/1793799406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d4106630 0x7fe7d4106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.263+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1793799406 >> 172.21.15.31:0/1793799406 conn(0x7fe7d40777c0 msgr2=0x7fe7d4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.263+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1793799406 shutdown_connections 2024-04-25T02:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.263+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1793799406 wait complete. 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7dbf22640 1 Processor -- start 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7dbf22640 1 -- start start 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7dbf22640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7d4100670 0x7fe7d4197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7dbf22640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d4101030 0x7fe7d4198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7dbf22640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7d4106630 0x7fe7d419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7dbf22640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4110940 con 0x7fe7d4106630 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7dbf22640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41107c0 con 0x7fe7d4101030 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7dbf22640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4110ac0 con 0x7fe7d4100670 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7da4bf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7d4106630 0x7fe7d419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7d9cbe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7d4100670 0x7fe7d4197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7d94bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d4101030 0x7fe7d4198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7da4bf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7d4106630 0x7fe7d419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40920/0 (socket says 172.21.15.31:40920) 2024-04-25T02:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7d9cbe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7d4100670 0x7fe7d4197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:55396/0 (socket says 172.21.15.31:55396) 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.264+0000 7fe7da4bf640 1 -- 172.21.15.31:0/1744252098 learned_addr learned my addr 172.21.15.31:0/1744252098 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7d94bd640 1 -- 172.21.15.31:0/1744252098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7d4100670 msgr2=0x7fe7d4197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7d94bd640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7d4100670 0x7fe7d4197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7d94bd640 1 -- 172.21.15.31:0/1744252098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7d4106630 msgr2=0x7fe7d419c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7d94bd640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7d4106630 0x7fe7d419c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7d94bd640 1 -- 172.21.15.31:0/1744252098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d419cdc0 con 0x7fe7d4101030 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7d4106630 0x7fe7d419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7d9cbe640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7d4100670 0x7fe7d4197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7d94bd640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d4101030 0x7fe7d4198360 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400e9f0 tx=0x7fe7c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7caffd640 1 -- 172.21.15.31:0/1744252098 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c400ccc0 con 0x7fe7d4101030 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7caffd640 1 -- 172.21.15.31:0/1744252098 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7c40108e0 con 0x7fe7d4101030 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1744252098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d419d0b0 con 0x7fe7d4101030 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7caffd640 1 -- 172.21.15.31:0/1744252098 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4018880 con 0x7fe7d4101030 2024-04-25T02:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.265+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1744252098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d41a4940 con 0x7fe7d4101030 2024-04-25T02:10:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.268+0000 7fe7caffd640 1 -- 172.21.15.31:0/1744252098 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe7c4010a90 con 0x7fe7d4101030 2024-04-25T02:10:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.269+0000 7fe7dec33740 1 Processor -- start 2024-04-25T02:10:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.269+0000 7fe7dec33740 1 -- start start 2024-04-25T02:10:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.269+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cbe500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.269+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05ccd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.269+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05ccfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.269+0000 7fe7dec33740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ee05c22cb0 con 0x55ee05cbea40 2024-04-25T02:10:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.269+0000 7fe7dec33740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ee05bcb5b0 con 0x55ee05cbe120 2024-04-25T02:10:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.269+0000 7fe7dec33740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ee05c1ec70 con 0x55ee05ccd640 2024-04-25T02:10:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7da4bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cbe500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7d9cbe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05ccd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7da4bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cbe500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51342/0 (socket says 172.21.15.31:51342) 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7d94bd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05ccfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7da4bf640 1 -- 172.21.15.31:0/823398383 learned_addr learned my addr 172.21.15.31:0/823398383 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7caffd640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe7c00a2e20 0x7fe7c00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe7c00a2e20 0x7fe7c00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7da4bf640 1 -- 172.21.15.31:0/823398383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 msgr2=0x55ee05ccfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/823398383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05ccfa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7da4bf640 1 -- 172.21.15.31:0/823398383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 msgr2=0x55ee05ccd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/823398383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05ccd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7da4bf640 1 -- 172.21.15.31:0/823398383 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee05937210 con 0x55ee05cbe120 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7d94bd640 1 --2- 172.21.15.31:0/823398383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05ccfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7d9cbe640 1 --2- 172.21.15.31:0/823398383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05ccd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/823398383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cbe500 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc005fe0 tx=0x7fe7cc031cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb3412db62b726e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.270+0000 7fe7caffd640 1 -- 172.21.15.31:0/1744252098 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fe7c40cf6c0 con 0x7fe7d4101030 2024-04-25T02:10:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.271+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe7c00a2e20 0x7fe7c00a52e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc047000 tx=0x7fe7cc04a000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.271+0000 7fe7bffff640 1 -- 172.21.15.31:0/823398383 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0046d0 con 0x55ee05cbe120 2024-04-25T02:10:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.271+0000 7fe7bffff640 1 -- 172.21.15.31:0/823398383 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7cc036070 con 0x55ee05cbe120 2024-04-25T02:10:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.271+0000 7fe7bffff640 1 -- 172.21.15.31:0/823398383 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc04a670 con 0x55ee05cbe120 2024-04-25T02:10:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.272+0000 7fe7dec33740 1 -- 172.21.15.31:0/823398383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 msgr2=0x55ee05cbe500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.272+0000 7fe7dec33740 1 --2- 172.21.15.31:0/823398383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cbe500 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc005fe0 tx=0x7fe7cc031cf0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.272+0000 7fe7dec33740 1 -- 172.21.15.31:0/823398383 shutdown_connections 2024-04-25T02:10:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.272+0000 7fe7dec33740 1 --2- 172.21.15.31:0/823398383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05ccfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.272+0000 7fe7dec33740 1 --2- 172.21.15.31:0/823398383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05ccd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.272+0000 7fe7dec33740 1 --2- 172.21.15.31:0/823398383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cbe500 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.272+0000 7fe7dec33740 1 -- 172.21.15.31:0/823398383 >> 172.21.15.31:0/823398383 conn(0x55ee05cba560 msgr2=0x55ee05cba940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.272+0000 7fe7dec33740 1 -- 172.21.15.31:0/823398383 shutdown_connections 2024-04-25T02:10:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.272+0000 7fe7dec33740 1 -- 172.21.15.31:0/823398383 wait complete. 2024-04-25T02:10:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.273+0000 7fe7dec33740 1 -- start start 2024-04-25T02:10:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.273+0000 7fe7dec33740 1 Processor -- start 2024-04-25T02:10:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.274+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05cbe120 0x55ee05cca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.274+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05ccaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.274+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05ccd640 0x55ee05ccb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.274+0000 7fe7d9cbe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05cbe120 0x55ee05cca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.274+0000 7fe7d9cbe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05cbe120 0x55ee05cca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:55414/0 (socket says 172.21.15.31:55414) 2024-04-25T02:10:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.274+0000 7fe7d94bd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05ccaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.274+0000 7fe7d94bd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05ccaae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40932/0 (socket says 172.21.15.31:40932) 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.274+0000 7fe7da4bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05ccd640 0x55ee05ccb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.275+0000 7fe7da4bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05ccd640 0x55ee05ccb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51358/0 (socket says 172.21.15.31:51358) 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.275+0000 7fe7d9cbe640 1 -- 172.21.15.31:0/461211644 learned_addr learned my addr 172.21.15.31:0/461211644 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.275+0000 7fe7dec33740 1 -- 172.21.15.31:0/461211644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ee05cd03c0 con 0x55ee05cbea40 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.275+0000 7fe7dec33740 1 -- 172.21.15.31:0/461211644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ee05bdeb50 con 0x55ee05ccd640 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.275+0000 7fe7dec33740 1 -- 172.21.15.31:0/461211644 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ee05cd0540 con 0x55ee05cbe120 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.275+0000 7fe7dec33740 10 client.0 _ll_register_callbacks cb 0x55ee05c19ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.275+0000 7fe7da4bf640 1 -- 172.21.15.31:0/461211644 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05cbe120 msgr2=0x55ee05cca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.275+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/461211644 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05cbe120 0x55ee05cca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.275+0000 7fe7da4bf640 1 -- 172.21.15.31:0/461211644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 msgr2=0x55ee05ccaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.275+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/461211644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05ccaae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.275+0000 7fe7da4bf640 1 -- 172.21.15.31:0/461211644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee05a03620 con 0x55ee05ccd640 2024-04-25T02:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.276+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/461211644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05ccd640 0x55ee05ccb020 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009a60 tx=0x7fe7cc02a8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44833245f0a7a71c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.276+0000 7fe7bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.276+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/461211644 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc058070 con 0x55ee05ccd640 2024-04-25T02:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.276+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/461211644 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fe7cc04a650 con 0x55ee05ccd640 2024-04-25T02:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.276+0000 7fe7dec33740 1 -- 172.21.15.31:0/461211644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7cc036070 con 0x55ee05ccd640 2024-04-25T02:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.276+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/461211644 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fe7cc004860 con 0x55ee05ccd640 2024-04-25T02:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.276+0000 7fe7dec33740 10 client.27014 fetch_fsmap learned FSMap version 46 2024-04-25T02:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.276+0000 7fe7dec33740 1 -- 172.21.15.31:0/461211644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55ee05cd03c0 con 0x55ee05ccd640 2024-04-25T02:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.277+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/461211644 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7cc036070 con 0x55ee05ccd640 2024-04-25T02:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.277+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/461211644 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 8374+0+0 (secure 0 0 0) 0x7fe7cc030070 con 0x55ee05ccd640 2024-04-25T02:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.277+0000 7fe7dec33740 10 client.27014 fetch_fsmap finished waiting for FSMap version 46 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.277+0000 7fe7dec33740 10 client.27014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.277+0000 7fe7dec33740 20 client.27014 populate_metadata read hostname 'smithi031' 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.277+0000 7fe7dec33740 1 --2- 172.21.15.31:0/461211644 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ee05cdd780 0x55ee05cdfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.277+0000 7fe7dec33740 4 client.27014 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.277+0000 7fe7dec33740 1 -- 172.21.15.31:0/461211644 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee05c301f0 con 0x55ee05cdd780 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.277+0000 7fe7d9cbe640 1 --2- 172.21.15.31:0/461211644 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ee05cdd780 0x55ee05cdfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.277+0000 7fe7d9cbe640 1 --2- 172.21.15.31:0/461211644 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ee05cdd780 0x55ee05cdfb70 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d4d5e032f7a6e3a server_cookie=c349cbcf36a32fa4 in_seq=0 out_seq=0 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.278+0000 7fe7bd7fa640 10 client.27014 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.278+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/461211644 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee05c301f0 con 0x55ee05cdd780 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.278+0000 7fe7bd7fa640 10 client.27014 handle_command_reply: tid=0 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.278+0000 7fe7dec33740 1 client.27014 shutdown 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.278+0000 7fe7dec33740 1 -- 172.21.15.31:0/461211644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05ccd640 msgr2=0x55ee05ccb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.278+0000 7fe7dec33740 1 --2- 172.21.15.31:0/461211644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05ccd640 0x55ee05ccb020 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009a60 tx=0x7fe7cc02a8e0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.279+0000 7fe7dec33740 1 -- 172.21.15.31:0/461211644 shutdown_connections 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.279+0000 7fe7dec33740 1 --2- 172.21.15.31:0/461211644 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ee05cdd780 0x55ee05cdfb70 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.279+0000 7fe7dec33740 1 --2- 172.21.15.31:0/461211644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05ccd640 0x55ee05ccb020 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.279+0000 7fe7dec33740 1 --2- 172.21.15.31:0/461211644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05ccaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.279+0000 7fe7dec33740 1 --2- 172.21.15.31:0/461211644 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05cbe120 0x55ee05cca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.279+0000 7fe7dec33740 1 -- 172.21.15.31:0/461211644 >> 172.21.15.31:0/461211644 conn(0x55ee05cba560 msgr2=0x55ee05cc0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.279+0000 7fe7bd7fa640 0 client.27014 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.279+0000 7fe7dec33740 1 -- 172.21.15.31:0/461211644 shutdown_connections 2024-04-25T02:10:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.279+0000 7fe7dec33740 1 -- 172.21.15.31:0/461211644 wait complete. 2024-04-25T02:10:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.279+0000 7fe7dec33740 20 client.27014 trim_cache size 0 max 16384 2024-04-25T02:10:50.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.307+0000 7fe7dec33740 1 Processor -- start 2024-04-25T02:10:50.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.307+0000 7fe7dec33740 1 -- start start 2024-04-25T02:10:50.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbe120 0x55ee05cd92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbea40 0x55ee05cb9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05ce5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7dec33740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41aaff0 con 0x55ee05cbe120 2024-04-25T02:10:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7dec33740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41acbf0 con 0x55ee05cbea40 2024-04-25T02:10:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7dec33740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41ab6c0 con 0x55ee05ccd640 2024-04-25T02:10:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7d9cbe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbe120 0x55ee05cd92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7d94bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbea40 0x55ee05cb9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7d94bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbea40 0x55ee05cb9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51372/0 (socket says 172.21.15.31:51372) 2024-04-25T02:10:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7d94bd640 1 -- 172.21.15.31:0/2810342576 learned_addr learned my addr 172.21.15.31:0/2810342576 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/2810342576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05ce5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7d94bd640 1 -- 172.21.15.31:0/2810342576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 msgr2=0x55ee05ce5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7d94bd640 1 --2- 172.21.15.31:0/2810342576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05ce5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7d94bd640 1 -- 172.21.15.31:0/2810342576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbe120 msgr2=0x55ee05cd92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7d94bd640 1 --2- 172.21.15.31:0/2810342576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbe120 0x55ee05cd92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7d94bd640 1 -- 172.21.15.31:0/2810342576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee05cbf6d0 con 0x55ee05cbea40 2024-04-25T02:10:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/2810342576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05ce5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.308+0000 7fe7d9cbe640 1 --2- 172.21.15.31:0/2810342576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbe120 0x55ee05cd92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.309+0000 7fe7d94bd640 1 --2- 172.21.15.31:0/2810342576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbea40 0x55ee05cb9440 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe7c409bc40 tx=0x7fe7c4098080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd18886a36e9c409 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.309+0000 7fe7bf7fe640 1 -- 172.21.15.31:0/2810342576 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c40b2070 con 0x55ee05cbea40 2024-04-25T02:10:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.309+0000 7fe7bf7fe640 1 -- 172.21.15.31:0/2810342576 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7c409bcb0 con 0x55ee05cbea40 2024-04-25T02:10:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.309+0000 7fe7bf7fe640 1 -- 172.21.15.31:0/2810342576 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c40d5040 con 0x55ee05cbea40 2024-04-25T02:10:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.310+0000 7fe7dec33740 1 -- 172.21.15.31:0/2810342576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbea40 msgr2=0x55ee05cb9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.310+0000 7fe7dec33740 1 --2- 172.21.15.31:0/2810342576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbea40 0x55ee05cb9440 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe7c409bc40 tx=0x7fe7c4098080 comp rx=0 tx=0).stop 2024-04-25T02:10:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.310+0000 7fe7dec33740 1 -- 172.21.15.31:0/2810342576 shutdown_connections 2024-04-25T02:10:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.310+0000 7fe7dec33740 1 --2- 172.21.15.31:0/2810342576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05ce5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.310+0000 7fe7dec33740 1 --2- 172.21.15.31:0/2810342576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbea40 0x55ee05cb9440 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.310+0000 7fe7dec33740 1 --2- 172.21.15.31:0/2810342576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbe120 0x55ee05cd92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.310+0000 7fe7dec33740 1 -- 172.21.15.31:0/2810342576 >> 172.21.15.31:0/2810342576 conn(0x55ee05cba560 msgr2=0x55ee05cb8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.310+0000 7fe7dec33740 1 -- 172.21.15.31:0/2810342576 shutdown_connections 2024-04-25T02:10:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.310+0000 7fe7dec33740 1 -- 172.21.15.31:0/2810342576 wait complete. 2024-04-25T02:10:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.310+0000 7fe7dec33740 1 -- start start 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.310+0000 7fe7dec33740 1 Processor -- start 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cd5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05cd0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7dec33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05cd1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7dec33740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41ac520 con 0x55ee05cbea40 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7dec33740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41abd90 con 0x55ee05cbe120 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7dec33740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41a8ff0 con 0x55ee05ccd640 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7dec33740 10 client.0 _ll_register_callbacks cb 0x7fe7d0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7d9cbe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cd5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7d9cbe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cd5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51376/0 (socket says 172.21.15.31:51376) 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7d94bd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05cd0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7da4bf640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05cd1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7d9cbe640 1 -- 172.21.15.31:0/2374449082 learned_addr learned my addr 172.21.15.31:0/2374449082 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:10:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7da4bf640 1 -- 172.21.15.31:0/2374449082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 msgr2=0x55ee05cd5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cd5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7da4bf640 1 -- 172.21.15.31:0/2374449082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 msgr2=0x55ee05cd0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05cd0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7da4bf640 1 -- 172.21.15.31:0/2374449082 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee05cd18f0 con 0x55ee05ccd640 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7d9cbe640 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cd5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.311+0000 7fe7d94bd640 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05cd0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.312+0000 7fe7da4bf640 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05cd1300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc005e30 tx=0x7fe7cc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4836ec36fac1bd92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.312+0000 7fe7bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.312+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/2374449082 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc058040 con 0x55ee05ccd640 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.312+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/2374449082 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8484+0+0 (secure 0 0 0) 0x7fe7cc052cb0 con 0x55ee05ccd640 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.312+0000 7fe7dec33740 1 -- 172.21.15.31:0/2374449082 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7c409bcb0 con 0x55ee05ccd640 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.312+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/2374449082 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fe7cc031c70 con 0x55ee05ccd640 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.312+0000 7fe7dec33740 10 client.26803 fetch_fsmap learned FSMap version 46 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.312+0000 7fe7dec33740 1 -- 172.21.15.31:0/2374449082 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55ee05d439f0 con 0x55ee05ccd640 2024-04-25T02:10:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.312+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/2374449082 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7c409bcb0 con 0x55ee05ccd640 2024-04-25T02:10:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.313+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/2374449082 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 46) ==== 8374+0+0 (secure 0 0 0) 0x7fe7cc063020 con 0x55ee05ccd640 2024-04-25T02:10:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.313+0000 7fe7dec33740 10 client.26803 fetch_fsmap finished waiting for FSMap version 46 2024-04-25T02:10:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.313+0000 7fe7dec33740 10 client.26803 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.313+0000 7fe7dec33740 20 client.26803 populate_metadata read hostname 'smithi031' 2024-04-25T02:10:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.313+0000 7fe7dec33740 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ee05d441c0 0x55ee05d465e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.313+0000 7fe7dec33740 4 client.26803 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:10:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.313+0000 7fe7dec33740 1 -- 172.21.15.31:0/2374449082 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ee05c2a540 con 0x55ee05d441c0 2024-04-25T02:10:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.313+0000 7fe7d9cbe640 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ee05d441c0 0x55ee05d465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.313+0000 7fe7bd7fa640 10 client.26803 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.313+0000 7fe7d9cbe640 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ee05d441c0 0x55ee05d465e0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21a700db2fb6f61f server_cookie=669bcac8e1cee8f3 in_seq=0 out_seq=0 2024-04-25T02:10:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7bd7fa640 1 -- 172.21.15.31:0/2374449082 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ee05c2a540 con 0x55ee05d441c0 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7bd7fa640 10 client.26803 handle_command_reply: tid=0 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7dec33740 1 client.26803 shutdown 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7dec33740 1 -- 172.21.15.31:0/2374449082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 msgr2=0x55ee05cd1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7dec33740 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05cd1300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc005e30 tx=0x7fe7cc004a40 comp rx=0 tx=0).stop 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7dec33740 1 -- 172.21.15.31:0/2374449082 shutdown_connections 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7dec33740 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ee05d441c0 0x55ee05d465e0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7dec33740 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ee05ccd640 0x55ee05cd1300 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7dec33740 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee05cbea40 0x55ee05cd0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7bd7fa640 0 client.26803 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7dec33740 1 --2- 172.21.15.31:0/2374449082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee05cbe120 0x55ee05cd5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.372+0000 7fe7dec33740 1 -- 172.21.15.31:0/2374449082 >> 172.21.15.31:0/2374449082 conn(0x55ee05cba560 msgr2=0x55ee05ccaea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.373+0000 7fe7dec33740 1 -- 172.21.15.31:0/2374449082 shutdown_connections 2024-04-25T02:10:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.373+0000 7fe7dec33740 1 -- 172.21.15.31:0/2374449082 wait complete. 2024-04-25T02:10:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.373+0000 7fe7dec33740 20 client.26803 trim_cache size 0 max 16384 2024-04-25T02:10:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:10:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:10:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:10:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:10:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.373+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1744252098 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe7c00a2e20 msgr2=0x7fe7c00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.373+0000 7fe7dbf22640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe7c00a2e20 0x7fe7c00a52e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc047000 tx=0x7fe7cc04a000 comp rx=0 tx=0).stop 2024-04-25T02:10:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.373+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1744252098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d4101030 msgr2=0x7fe7d4198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.373+0000 7fe7dbf22640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d4101030 0x7fe7d4198360 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400e9f0 tx=0x7fe7c400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.374+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1744252098 shutdown_connections 2024-04-25T02:10:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.374+0000 7fe7dbf22640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe7c00a2e20 0x7fe7c00a52e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.374+0000 7fe7dbf22640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7d4106630 0x7fe7d419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.374+0000 7fe7dbf22640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7d4101030 0x7fe7d4198360 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.374+0000 7fe7dbf22640 1 --2- 172.21.15.31:0/1744252098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7d4100670 0x7fe7d4197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.374+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1744252098 >> 172.21.15.31:0/1744252098 conn(0x7fe7d40777c0 msgr2=0x7fe7d40ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.374+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1744252098 shutdown_connections 2024-04-25T02:10:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:10:50.374+0000 7fe7dbf22640 1 -- 172.21.15.31:0/1744252098 wait complete. 2024-04-25T02:10:50.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:49 smithi040 ceph-mon[32418]: pgmap v512: 97 pgs: 97 active+clean; 619 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 229 op/s 2024-04-25T02:10:50.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:49 smithi134 ceph-mon[33098]: pgmap v512: 97 pgs: 97 active+clean; 619 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.0 MiB/s wr, 229 op/s 2024-04-25T02:10:51.095 INFO:tasks.workunit.client.0.smithi031.stdout: 10 29144 8.99 MB/sec execute 55 sec latency 65.156 ms 2024-04-25T02:10:52.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 29305 8.98 MB/sec execute 56 sec latency 197.474 ms 2024-04-25T02:10:52.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:51 smithi040 ceph-mon[32418]: pgmap v513: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.1 MiB/s wr, 369 op/s 2024-04-25T02:10:52.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:51 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:52.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:51 smithi134 ceph-mon[33098]: pgmap v513: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.1 MiB/s wr, 369 op/s 2024-04-25T02:10:52.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:51 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:52.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:51 smithi031 ceph-mon[29204]: pgmap v513: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.1 MiB/s wr, 369 op/s 2024-04-25T02:10:52.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:51 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:10:53.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 29501 8.96 MB/sec execute 57 sec latency 64.945 ms 2024-04-25T02:10:54.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 29701 8.92 MB/sec execute 58 sec latency 55.701 ms 2024-04-25T02:10:54.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:53 smithi040 ceph-mon[32418]: pgmap v514: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 250 op/s 2024-04-25T02:10:54.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:53 smithi134 ceph-mon[33098]: pgmap v514: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 250 op/s 2024-04-25T02:10:54.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:53 smithi031 ceph-mon[29204]: pgmap v514: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 250 op/s 2024-04-25T02:10:55.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 29874 8.93 MB/sec execute 59 sec latency 69.628 ms 2024-04-25T02:10:56.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 29988 8.85 MB/sec execute 60 sec latency 116.944 ms 2024-04-25T02:10:56.417 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:55 smithi040 ceph-mon[32418]: pgmap v515: 97 pgs: 97 active+clean; 625 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 358 op/s 2024-04-25T02:10:56.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:55 smithi134 ceph-mon[33098]: pgmap v515: 97 pgs: 97 active+clean; 625 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 358 op/s 2024-04-25T02:10:56.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:55 smithi031 ceph-mon[29204]: pgmap v515: 97 pgs: 97 active+clean; 625 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 358 op/s 2024-04-25T02:10:57.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 30147 8.89 MB/sec execute 61 sec latency 102.896 ms 2024-04-25T02:10:57.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:56 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:10:57.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:56 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:10:57.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:56 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:10:58.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 30377 9.13 MB/sec execute 62 sec latency 117.702 ms 2024-04-25T02:10:58.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:57 smithi040 ceph-mon[32418]: pgmap v516: 97 pgs: 97 active+clean; 625 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 247 op/s 2024-04-25T02:10:58.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:57 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:10:58.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:57 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:10:58.417 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:10:57 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:10:58.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:57 smithi134 ceph-mon[33098]: pgmap v516: 97 pgs: 97 active+clean; 625 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 247 op/s 2024-04-25T02:10:58.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:57 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:10:58.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:57 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:10:58.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:10:57 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:10:58.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:57 smithi031 ceph-mon[29204]: pgmap v516: 97 pgs: 97 active+clean; 625 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 247 op/s 2024-04-25T02:10:58.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:57 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:10:58.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:57 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:10:58.495 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:10:57 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:10:59.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 30602 9.17 MB/sec execute 63 sec latency 57.851 ms 2024-04-25T02:11:00.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 30747 9.15 MB/sec execute 64 sec latency 69.645 ms 2024-04-25T02:11:00.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:00 smithi040 ceph-mon[32418]: pgmap v517: 97 pgs: 97 active+clean; 625 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 247 op/s 2024-04-25T02:11:00.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:00 smithi134 ceph-mon[33098]: pgmap v517: 97 pgs: 97 active+clean; 625 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 247 op/s 2024-04-25T02:11:00.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:00 smithi031 ceph-mon[29204]: pgmap v517: 97 pgs: 97 active+clean; 625 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 247 op/s 2024-04-25T02:11:01.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 30856 9.09 MB/sec execute 65 sec latency 100.229 ms 2024-04-25T02:11:02.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 31062 9.24 MB/sec execute 66 sec latency 76.123 ms 2024-04-25T02:11:02.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:02 smithi040 ceph-mon[32418]: pgmap v518: 97 pgs: 97 active+clean; 641 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.3 MiB/s wr, 355 op/s 2024-04-25T02:11:02.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:02 smithi134 ceph-mon[33098]: pgmap v518: 97 pgs: 97 active+clean; 641 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.3 MiB/s wr, 355 op/s 2024-04-25T02:11:02.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:02 smithi031 ceph-mon[29204]: pgmap v518: 97 pgs: 97 active+clean; 641 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.3 MiB/s wr, 355 op/s 2024-04-25T02:11:03.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 31215 9.20 MB/sec execute 67 sec latency 58.936 ms 2024-04-25T02:11:03.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:03 smithi040 ceph-mon[32418]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T02:11:03.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:03 smithi134 ceph-mon[33098]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T02:11:03.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:03 smithi031 ceph-mon[29204]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T02:11:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.506+0000 7f54ed64d640 1 -- 172.21.15.31:0/3517167739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54e8106630 msgr2=0x7f54e8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.506+0000 7f54ed64d640 1 --2- 172.21.15.31:0/3517167739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54e8106630 0x7f54e8106a10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f54d800b880 tx=0x7f54d8005f90 comp rx=0 tx=0).stop 2024-04-25T02:11:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.506+0000 7f54ed64d640 1 -- 172.21.15.31:0/3517167739 shutdown_connections 2024-04-25T02:11:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.506+0000 7f54ed64d640 1 --2- 172.21.15.31:0/3517167739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54e8101030 0x7f54e810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.506+0000 7f54ed64d640 1 --2- 172.21.15.31:0/3517167739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e8100670 0x7f54e8100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.506+0000 7f54ed64d640 1 --2- 172.21.15.31:0/3517167739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54e8106630 0x7f54e8106a10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.506+0000 7f54ed64d640 1 -- 172.21.15.31:0/3517167739 >> 172.21.15.31:0/3517167739 conn(0x7f54e80777c0 msgr2=0x7f54e8077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.506+0000 7f54ed64d640 1 -- 172.21.15.31:0/3517167739 shutdown_connections 2024-04-25T02:11:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.506+0000 7f54ed64d640 1 -- 172.21.15.31:0/3517167739 wait complete. 2024-04-25T02:11:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.507+0000 7f54ed64d640 1 Processor -- start 2024-04-25T02:11:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.507+0000 7f54ed64d640 1 -- start start 2024-04-25T02:11:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.507+0000 7f54ed64d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e8100670 0x7f54e819b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.507+0000 7f54ed64d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54e8101030 0x7f54e819be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.507+0000 7f54ed64d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54e8106630 0x7f54e8195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.507+0000 7f54ed64d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54e810f830 con 0x7f54e8100670 2024-04-25T02:11:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.507+0000 7f54ed64d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f54e810f6b0 con 0x7f54e8101030 2024-04-25T02:11:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54ed64d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f54e810f9b0 con 0x7f54e8106630 2024-04-25T02:11:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e8100670 0x7f54e819b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54dffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54e8101030 0x7f54e819be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54dffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54e8101030 0x7f54e819be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60264/0 (socket says 172.21.15.31:60264) 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54dffff640 1 -- 172.21.15.31:0/2951191370 learned_addr learned my addr 172.21.15.31:0/2951191370 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e8100670 0x7f54e819b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43384/0 (socket says 172.21.15.31:43384) 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54e77fe640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54e8106630 0x7f54e8195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54dffff640 1 -- 172.21.15.31:0/2951191370 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54e8106630 msgr2=0x7f54e8195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54dffff640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54e8106630 0x7f54e8195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54dffff640 1 -- 172.21.15.31:0/2951191370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e8100670 msgr2=0x7f54e819b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54dffff640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e8100670 0x7f54e819b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54dffff640 1 -- 172.21.15.31:0/2951191370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e8196210 con 0x7f54e8101030 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54dffff640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54e8101030 0x7f54e819be20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009ad0 tx=0x7f54d0009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54e4ff9640 1 -- 172.21.15.31:0/2951191370 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d000cde0 con 0x7f54e8101030 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.508+0000 7f54e4ff9640 1 -- 172.21.15.31:0/2951191370 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54d00035f0 con 0x7f54e8101030 2024-04-25T02:11:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.509+0000 7f54e4ff9640 1 -- 172.21.15.31:0/2951191370 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0017640 con 0x7f54e8101030 2024-04-25T02:11:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.509+0000 7f54ed64d640 1 -- 172.21.15.31:0/2951191370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e8196500 con 0x7f54e8101030 2024-04-25T02:11:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.509+0000 7f54ed64d640 1 -- 172.21.15.31:0/2951191370 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e81a27a0 con 0x7f54e8101030 2024-04-25T02:11:03.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.511+0000 7f54e4ff9640 1 -- 172.21.15.31:0/2951191370 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f54d0004000 con 0x7f54e8101030 2024-04-25T02:11:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.512+0000 7f54efffd740 1 Processor -- start 2024-04-25T02:11:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.512+0000 7f54efffd740 1 -- start start 2024-04-25T02:11:03.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.512+0000 7f54e4ff9640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54d40a2e20 0x7f54d40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.512+0000 7f54e4ff9640 1 -- 172.21.15.31:0/2951191370 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8528+0+0 (secure 0 0 0) 0x7f54d00cf0d0 con 0x7f54e8101030 2024-04-25T02:11:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.512+0000 7f54e77fe640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54d40a2e20 0x7f54d40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.512+0000 7f54efffd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4240 0x55cafa9a4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.512+0000 7f54efffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54efffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9af230 0x55cafa9b1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54efffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cafa908cb0 con 0x55cafa9a4b60 2024-04-25T02:11:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54efffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cafa8b15b0 con 0x55cafa9af230 2024-04-25T02:11:03.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54efffd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55cafa904c70 con 0x55cafa9a4240 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4240 0x55cafa9a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9aecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43392/0 (socket says 172.21.15.31:43392) 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54dffff640 1 -- 172.21.15.31:0/3972099489 learned_addr learned my addr 172.21.15.31:0/3972099489 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54dffff640 1 -- 172.21.15.31:0/3972099489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4240 msgr2=0x55cafa9a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54dffff640 1 --2- 172.21.15.31:0/3972099489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4240 0x55cafa9a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54dffff640 1 -- 172.21.15.31:0/3972099489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9af230 msgr2=0x55cafa9b1620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54dffff640 1 --2- 172.21.15.31:0/3972099489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9af230 0x55cafa9b1620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54dffff640 1 -- 172.21.15.31:0/3972099489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cafa61d210 con 0x55cafa9a4b60 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54e6ffd640 1 --2- 172.21.15.31:0/3972099489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4240 0x55cafa9a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54e77fe640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54d40a2e20 0x7f54d40a52e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f54e0005fd0 tx=0x7f54e0011040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.513+0000 7f54dffff640 1 --2- 172.21.15.31:0/3972099489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9aecf0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f54d00a3dd0 tx=0x7f54d009b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d419bbe5fa9a6c89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.514+0000 7f54dd7fa640 1 -- 172.21.15.31:0/3972099489 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0086070 con 0x55cafa9a4b60 2024-04-25T02:11:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.514+0000 7f54dd7fa640 1 -- 172.21.15.31:0/3972099489 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54d0099a40 con 0x55cafa9a4b60 2024-04-25T02:11:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.514+0000 7f54dd7fa640 1 -- 172.21.15.31:0/3972099489 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0097db0 con 0x55cafa9a4b60 2024-04-25T02:11:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.515+0000 7f54efffd740 1 -- 172.21.15.31:0/3972099489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 msgr2=0x55cafa9aecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:03.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.515+0000 7f54efffd740 1 --2- 172.21.15.31:0/3972099489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9aecf0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f54d00a3dd0 tx=0x7f54d009b800 comp rx=0 tx=0).stop 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.515+0000 7f54efffd740 1 -- 172.21.15.31:0/3972099489 shutdown_connections 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.515+0000 7f54efffd740 1 --2- 172.21.15.31:0/3972099489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9af230 0x55cafa9b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.515+0000 7f54efffd740 1 --2- 172.21.15.31:0/3972099489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9aecf0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.515+0000 7f54efffd740 1 --2- 172.21.15.31:0/3972099489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4240 0x55cafa9a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.515+0000 7f54efffd740 1 -- 172.21.15.31:0/3972099489 >> 172.21.15.31:0/3972099489 conn(0x55cafa9a0560 msgr2=0x55cafa9a0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.515+0000 7f54efffd740 1 -- 172.21.15.31:0/3972099489 shutdown_connections 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.515+0000 7f54efffd740 1 -- 172.21.15.31:0/3972099489 wait complete. 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.515+0000 7f54efffd740 1 -- start start 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54efffd740 1 Processor -- start 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54efffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9ac2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54efffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54efffd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9af230 0x55cafa9acd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54efffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cafa9b1fc0 con 0x55cafa9a4b60 2024-04-25T02:11:03.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54efffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cafa8c4b50 con 0x55cafa9a4240 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54efffd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55cafa9b2140 con 0x55cafa9af230 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54efffd740 10 client.0 _ll_register_callbacks cb 0x55cafa8ffca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43402/0 (socket says 172.21.15.31:43402) 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54dffff640 1 -- 172.21.15.31:0/1905536512 learned_addr learned my addr 172.21.15.31:0/1905536512 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54e6ffd640 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9ac2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54e77fe640 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9af230 0x55cafa9acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54dffff640 1 -- 172.21.15.31:0/1905536512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9af230 msgr2=0x55cafa9acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54dffff640 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9af230 0x55cafa9acd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54dffff640 1 -- 172.21.15.31:0/1905536512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 msgr2=0x55cafa9ac2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54dffff640 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9ac2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.516+0000 7f54dffff640 1 -- 172.21.15.31:0/1905536512 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cafa6e9620 con 0x55cafa9a4b60 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.517+0000 7f54dffff640 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9ac7e0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f54d000c640 tx=0x7f54d00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e813ea68c6b8fbe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.517+0000 7f54ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:11:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.517+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1905536512 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d00b9060 con 0x55cafa9a4b60 2024-04-25T02:11:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.517+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1905536512 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8528+0+0 (secure 0 0 0) 0x7f54d0097db0 con 0x55cafa9a4b60 2024-04-25T02:11:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.517+0000 7f54efffd740 1 -- 172.21.15.31:0/1905536512 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54d0099a40 con 0x55cafa9a4b60 2024-04-25T02:11:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.517+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1905536512 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f54d00037a0 con 0x55cafa9a4b60 2024-04-25T02:11:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.517+0000 7f54efffd740 10 client.17460 fetch_fsmap learned FSMap version 47 2024-04-25T02:11:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.517+0000 7f54efffd740 1 -- 172.21.15.31:0/1905536512 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55cafa9be2b0 con 0x55cafa9a4b60 2024-04-25T02:11:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.517+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1905536512 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54d0099a40 con 0x55cafa9a4b60 2024-04-25T02:11:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.518+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1905536512 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f54d0097490 con 0x55cafa9a4b60 2024-04-25T02:11:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.518+0000 7f54efffd740 10 client.17460 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T02:11:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.518+0000 7f54efffd740 10 client.17460 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:11:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.518+0000 7f54efffd740 20 client.17460 populate_metadata read hostname 'smithi031' 2024-04-25T02:11:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.518+0000 7f54efffd740 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cafa9bf9b0 0x55cafa9c1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.518+0000 7f54efffd740 4 client.17460 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:11:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.518+0000 7f54efffd740 1 -- 172.21.15.31:0/1905536512 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cafa9161f0 con 0x55cafa9bf9b0 2024-04-25T02:11:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.518+0000 7f54e6ffd640 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cafa9bf9b0 0x55cafa9c1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.519+0000 7f54ce7fc640 10 client.17460 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.519+0000 7f54e6ffd640 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cafa9bf9b0 0x55cafa9c1dd0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b1cff0e8ce87fee server_cookie=caf45f70b77d88b0 in_seq=0 out_seq=0 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.519+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1905536512 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cafa9161f0 con 0x55cafa9bf9b0 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.519+0000 7f54ce7fc640 10 client.17460 handle_command_reply: tid=0 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.519+0000 7f54efffd740 1 client.17460 shutdown 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54efffd740 1 -- 172.21.15.31:0/1905536512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 msgr2=0x55cafa9ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54efffd740 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9ac7e0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f54d000c640 tx=0x7f54d00039e0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54efffd740 1 -- 172.21.15.31:0/1905536512 shutdown_connections 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54efffd740 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cafa9bf9b0 0x55cafa9c1dd0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54efffd740 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9af230 0x55cafa9acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54efffd740 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9a4b60 0x55cafa9ac7e0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54efffd740 1 --2- 172.21.15.31:0/1905536512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9ac2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54efffd740 1 -- 172.21.15.31:0/1905536512 >> 172.21.15.31:0/1905536512 conn(0x55cafa9a0560 msgr2=0x55cafa9a6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54ce7fc640 0 client.17460 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54efffd740 1 -- 172.21.15.31:0/1905536512 shutdown_connections 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54efffd740 1 -- 172.21.15.31:0/1905536512 wait complete. 2024-04-25T02:11:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.520+0000 7f54efffd740 20 client.17460 trim_cache size 0 max 16384 2024-04-25T02:11:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.543+0000 7f54efffd740 1 Processor -- start 2024-04-25T02:11:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.543+0000 7f54efffd740 1 -- start start 2024-04-25T02:11:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.543+0000 7f54efffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9bae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.543+0000 7f54efffd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4b60 0x55cafa9acb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.543+0000 7f54efffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9af230 0x55cafa99f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.543+0000 7f54efffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81ab250 con 0x55cafa9af230 2024-04-25T02:11:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.543+0000 7f54efffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81ace50 con 0x55cafa9a4240 2024-04-25T02:11:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.543+0000 7f54efffd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81ab920 con 0x55cafa9a4b60 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9bae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54dffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4b60 0x55cafa9acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9bae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60280/0 (socket says 172.21.15.31:60280) 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54e6ffd640 1 -- 172.21.15.31:0/1273348620 learned_addr learned my addr 172.21.15.31:0/1273348620 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54e77fe640 1 --2- 172.21.15.31:0/1273348620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9af230 0x55cafa99f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54e6ffd640 1 -- 172.21.15.31:0/1273348620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4b60 msgr2=0x55cafa9acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54e6ffd640 1 --2- 172.21.15.31:0/1273348620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4b60 0x55cafa9acb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54e6ffd640 1 -- 172.21.15.31:0/1273348620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9af230 msgr2=0x55cafa99f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54e6ffd640 1 --2- 172.21.15.31:0/1273348620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9af230 0x55cafa99f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54e6ffd640 1 -- 172.21.15.31:0/1273348620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cafa9a4960 con 0x55cafa9a4240 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54e6ffd640 1 --2- 172.21.15.31:0/1273348620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9bae80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f54d802a3b0 tx=0x7f54d80056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1471df402e53727 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54dcff9640 1 -- 172.21.15.31:0/1273348620 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8032e40 con 0x55cafa9a4240 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54dcff9640 1 -- 172.21.15.31:0/1273348620 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54d804a070 con 0x55cafa9a4240 2024-04-25T02:11:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.544+0000 7f54dcff9640 1 -- 172.21.15.31:0/1273348620 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8045490 con 0x55cafa9a4240 2024-04-25T02:11:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.545+0000 7f54efffd740 1 -- 172.21.15.31:0/1273348620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 msgr2=0x55cafa9bae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.545+0000 7f54efffd740 1 --2- 172.21.15.31:0/1273348620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9bae80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f54d802a3b0 tx=0x7f54d80056d0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.545+0000 7f54efffd740 1 -- 172.21.15.31:0/1273348620 shutdown_connections 2024-04-25T02:11:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.545+0000 7f54efffd740 1 --2- 172.21.15.31:0/1273348620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9af230 0x55cafa99f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.545+0000 7f54efffd740 1 --2- 172.21.15.31:0/1273348620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4b60 0x55cafa9acb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.545+0000 7f54efffd740 1 --2- 172.21.15.31:0/1273348620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9bae80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.545+0000 7f54efffd740 1 -- 172.21.15.31:0/1273348620 >> 172.21.15.31:0/1273348620 conn(0x55cafa9a0560 msgr2=0x55cafa99ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.545+0000 7f54efffd740 1 -- 172.21.15.31:0/1273348620 shutdown_connections 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54efffd740 1 -- 172.21.15.31:0/1273348620 wait complete. 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54efffd740 1 -- start start 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54efffd740 1 Processor -- start 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54efffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9b7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54efffd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4b60 0x55cafa9a8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54efffd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9af230 0x55cafa9a9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54efffd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81ac780 con 0x55cafa9af230 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54efffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81abff0 con 0x55cafa9a4240 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54efffd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81a9230 con 0x55cafa9a4b60 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54efffd740 10 client.0 _ll_register_callbacks cb 0x7f54d800b6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54dffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4b60 0x55cafa9a8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9b7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54e77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9af230 0x55cafa9a9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54dffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4b60 0x55cafa9a8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:45744/0 (socket says 172.21.15.31:45744) 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9b7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60296/0 (socket says 172.21.15.31:60296) 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.546+0000 7f54dffff640 1 -- 172.21.15.31:0/1606465948 learned_addr learned my addr 172.21.15.31:0/1606465948 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54e77fe640 1 -- 172.21.15.31:0/1606465948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4b60 msgr2=0x55cafa9a8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54e77fe640 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4b60 0x55cafa9a8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54e77fe640 1 -- 172.21.15.31:0/1606465948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 msgr2=0x55cafa9b7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54e77fe640 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9b7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54e77fe640 1 -- 172.21.15.31:0/1606465948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cafa9a9590 con 0x55cafa9af230 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54e6ffd640 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9b7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54e77fe640 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9af230 0x55cafa9a9050 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f54e000ff10 tx=0x7f54e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8d09d14ace80db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1606465948 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e0030020 con 0x55cafa9af230 2024-04-25T02:11:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1606465948 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8528+0+0 (secure 0 0 0) 0x7f54e0004800 con 0x55cafa9af230 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54efffd740 1 -- 172.21.15.31:0/1606465948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54d804a070 con 0x55cafa9af230 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.547+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1606465948 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f54e003b050 con 0x55cafa9af230 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.548+0000 7f54efffd740 10 client.17466 fetch_fsmap learned FSMap version 47 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.548+0000 7f54efffd740 1 -- 172.21.15.31:0/1606465948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55cafaa25c70 con 0x55cafa9af230 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.548+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1606465948 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54d804a070 con 0x55cafa9af230 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.548+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1606465948 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f54e0018440 con 0x55cafa9af230 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.548+0000 7f54efffd740 10 client.17466 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.548+0000 7f54efffd740 10 client.17466 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.548+0000 7f54efffd740 20 client.17466 populate_metadata read hostname 'smithi031' 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.548+0000 7f54efffd740 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cafaa26500 0x55cafaa28920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.548+0000 7f54efffd740 4 client.17466 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.548+0000 7f54efffd740 1 -- 172.21.15.31:0/1606465948 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cafa910540 con 0x55cafaa26500 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.548+0000 7f54e6ffd640 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cafaa26500 0x55cafaa28920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.549+0000 7f54e6ffd640 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cafaa26500 0x55cafaa28920 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c634cb46cf4b31ad server_cookie=dfa0df9cf6f17042 in_seq=0 out_seq=0 2024-04-25T02:11:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.549+0000 7f54ce7fc640 10 client.17466 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54ce7fc640 1 -- 172.21.15.31:0/1606465948 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cafa910540 con 0x55cafaa26500 2024-04-25T02:11:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54ce7fc640 10 client.17466 handle_command_reply: tid=0 2024-04-25T02:11:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54efffd740 1 client.17466 shutdown 2024-04-25T02:11:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54efffd740 1 -- 172.21.15.31:0/1606465948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9af230 msgr2=0x55cafa9a9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54efffd740 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9af230 0x55cafa9a9050 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f54e000ff10 tx=0x7f54e0004a40 comp rx=0 tx=0).stop 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54efffd740 1 -- 172.21.15.31:0/1606465948 shutdown_connections 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54efffd740 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55cafaa26500 0x55cafaa28920 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54efffd740 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cafa9af230 0x55cafa9a9050 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54efffd740 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55cafa9a4b60 0x55cafa9a8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54efffd740 1 --2- 172.21.15.31:0/1606465948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cafa9a4240 0x55cafa9b7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54efffd740 1 -- 172.21.15.31:0/1606465948 >> 172.21.15.31:0/1606465948 conn(0x55cafa9a0560 msgr2=0x55cafa99f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54ce7fc640 0 client.17466 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54efffd740 1 -- 172.21.15.31:0/1606465948 shutdown_connections 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.591+0000 7f54efffd740 1 -- 172.21.15.31:0/1606465948 wait complete. 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54efffd740 20 client.17466 trim_cache size 0 max 16384 2024-04-25T02:11:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:11:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:11:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:11:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:11:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 -- 172.21.15.31:0/2951191370 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54d40a2e20 msgr2=0x7f54d40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54d40a2e20 0x7f54d40a52e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f54e0005fd0 tx=0x7f54e0011040 comp rx=0 tx=0).stop 2024-04-25T02:11:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 -- 172.21.15.31:0/2951191370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54e8101030 msgr2=0x7f54e819be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54e8101030 0x7f54e819be20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009ad0 tx=0x7f54d0009f90 comp rx=0 tx=0).stop 2024-04-25T02:11:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 -- 172.21.15.31:0/2951191370 shutdown_connections 2024-04-25T02:11:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54d40a2e20 0x7f54d40a52e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54e8106630 0x7f54e8195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54e8101030 0x7f54e819be20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 --2- 172.21.15.31:0/2951191370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e8100670 0x7f54e819b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 -- 172.21.15.31:0/2951191370 >> 172.21.15.31:0/2951191370 conn(0x7f54e80777c0 msgr2=0x7f54e8103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 -- 172.21.15.31:0/2951191370 shutdown_connections 2024-04-25T02:11:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:03.592+0000 7f54ed64d640 1 -- 172.21.15.31:0/2951191370 wait complete. 2024-04-25T02:11:04.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 31349 9.14 MB/sec execute 68 sec latency 70.404 ms 2024-04-25T02:11:04.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:04 smithi040 ceph-mon[32418]: pgmap v520: 97 pgs: 97 active+clean; 641 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 259 op/s 2024-04-25T02:11:04.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:04 smithi134 ceph-mon[33098]: pgmap v520: 97 pgs: 97 active+clean; 641 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 259 op/s 2024-04-25T02:11:04.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:04 smithi031 ceph-mon[29204]: pgmap v520: 97 pgs: 97 active+clean; 641 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 259 op/s 2024-04-25T02:11:05.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 31510 9.13 MB/sec execute 69 sec latency 52.162 ms 2024-04-25T02:11:06.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 31681 9.13 MB/sec execute 70 sec latency 65.183 ms 2024-04-25T02:11:06.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:06 smithi040 ceph-mon[32418]: pgmap v521: 97 pgs: 97 active+clean; 660 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 291 op/s 2024-04-25T02:11:06.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:06 smithi134 ceph-mon[33098]: pgmap v521: 97 pgs: 97 active+clean; 660 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 291 op/s 2024-04-25T02:11:06.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:06 smithi031 ceph-mon[29204]: pgmap v521: 97 pgs: 97 active+clean; 660 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 291 op/s 2024-04-25T02:11:07.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 31813 9.05 MB/sec execute 71 sec latency 114.956 ms 2024-04-25T02:11:08.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 31971 8.97 MB/sec execute 72 sec latency 53.359 ms 2024-04-25T02:11:08.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:08 smithi031 ceph-mon[29204]: pgmap v522: 97 pgs: 97 active+clean; 660 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 291 op/s 2024-04-25T02:11:08.296 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:08 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:11:08.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:08 smithi040 ceph-mon[32418]: pgmap v522: 97 pgs: 97 active+clean; 660 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 291 op/s 2024-04-25T02:11:08.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:08 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:11:08.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:08 smithi134 ceph-mon[33098]: pgmap v522: 97 pgs: 97 active+clean; 660 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 291 op/s 2024-04-25T02:11:08.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:08 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:11:09.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 32133 8.97 MB/sec execute 73 sec latency 64.745 ms 2024-04-25T02:11:10.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 32311 8.96 MB/sec execute 74 sec latency 55.082 ms 2024-04-25T02:11:10.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:09 smithi134 ceph-mon[33098]: pgmap v523: 97 pgs: 97 active+clean; 660 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 291 op/s 2024-04-25T02:11:10.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:09 smithi134 ceph-mon[33098]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T02:11:10.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:09 smithi031 ceph-mon[29204]: pgmap v523: 97 pgs: 97 active+clean; 660 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 291 op/s 2024-04-25T02:11:10.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:09 smithi031 ceph-mon[29204]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T02:11:10.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:09 smithi040 ceph-mon[32418]: pgmap v523: 97 pgs: 97 active+clean; 660 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.5 MiB/s wr, 291 op/s 2024-04-25T02:11:10.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:09 smithi040 ceph-mon[32418]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T02:11:11.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 32516 9.09 MB/sec execute 75 sec latency 61.758 ms 2024-04-25T02:11:12.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 32658 9.02 MB/sec execute 76 sec latency 93.448 ms 2024-04-25T02:11:12.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:11 smithi134 ceph-mon[33098]: pgmap v525: 97 pgs: 97 active+clean; 673 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 362 op/s 2024-04-25T02:11:12.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:11 smithi031 ceph-mon[29204]: pgmap v525: 97 pgs: 97 active+clean; 673 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 362 op/s 2024-04-25T02:11:12.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:11 smithi040 ceph-mon[32418]: pgmap v525: 97 pgs: 97 active+clean; 673 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 362 op/s 2024-04-25T02:11:13.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 32825 9.00 MB/sec execute 77 sec latency 63.029 ms 2024-04-25T02:11:14.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 33036 9.00 MB/sec execute 78 sec latency 62.019 ms 2024-04-25T02:11:14.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:13 smithi134 ceph-mon[33098]: pgmap v526: 97 pgs: 97 active+clean; 673 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 319 op/s 2024-04-25T02:11:14.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:13 smithi031 ceph-mon[29204]: pgmap v526: 97 pgs: 97 active+clean; 673 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 319 op/s 2024-04-25T02:11:14.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:13 smithi040 ceph-mon[32418]: pgmap v526: 97 pgs: 97 active+clean; 673 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 319 op/s 2024-04-25T02:11:15.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 33232 8.99 MB/sec execute 79 sec latency 69.570 ms 2024-04-25T02:11:16.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 33376 8.97 MB/sec execute 80 sec latency 62.948 ms 2024-04-25T02:11:16.196 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:15 smithi134 ceph-mon[33098]: pgmap v527: 97 pgs: 97 active+clean; 687 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-04-25T02:11:16.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:15 smithi031 ceph-mon[29204]: pgmap v527: 97 pgs: 97 active+clean; 687 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-04-25T02:11:16.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:15 smithi040 ceph-mon[32418]: pgmap v527: 97 pgs: 97 active+clean; 687 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-04-25T02:11:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.718+0000 7f7084cd4640 1 -- 172.21.15.31:0/1391047439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080101760 msgr2=0x7f7080101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.718+0000 7f7084cd4640 1 --2- 172.21.15.31:0/1391047439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080101760 0x7f7080101be0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f7074009930 tx=0x7f707402a420 comp rx=0 tx=0).stop 2024-04-25T02:11:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.719+0000 7f7084cd4640 1 -- 172.21.15.31:0/1391047439 shutdown_connections 2024-04-25T02:11:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.719+0000 7f7084cd4640 1 --2- 172.21.15.31:0/1391047439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7080102120 0x7f708010e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.719+0000 7f7084cd4640 1 --2- 172.21.15.31:0/1391047439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080101760 0x7f7080101be0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.719+0000 7f7084cd4640 1 --2- 172.21.15.31:0/1391047439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7080107770 0x7f7080107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.719+0000 7f7084cd4640 1 -- 172.21.15.31:0/1391047439 >> 172.21.15.31:0/1391047439 conn(0x7f70800fd1f0 msgr2=0x7f70800ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:16.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.719+0000 7f7084cd4640 1 -- 172.21.15.31:0/1391047439 shutdown_connections 2024-04-25T02:11:16.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.719+0000 7f7084cd4640 1 -- 172.21.15.31:0/1391047439 wait complete. 2024-04-25T02:11:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f7084cd4640 1 Processor -- start 2024-04-25T02:11:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f7084cd4640 1 -- start start 2024-04-25T02:11:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f7084cd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7080101760 0x7f7080197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f7084cd4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7080102120 0x7f7080198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f7084cd4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080107770 0x7f708019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f7084cd4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7080111a40 con 0x7f7080107770 2024-04-25T02:11:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f7084cd4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f70801118c0 con 0x7f7080101760 2024-04-25T02:11:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f7084cd4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7080111bc0 con 0x7f7080102120 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f707e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7080101760 0x7f7080197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f707dd9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7080102120 0x7f7080198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f707e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7080101760 0x7f7080197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:33724/0 (socket says 172.21.15.31:33724) 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f707e59c640 1 -- 172.21.15.31:0/96494117 learned_addr learned my addr 172.21.15.31:0/96494117 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f707ed9d640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080107770 0x7f708019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f707e59c640 1 -- 172.21.15.31:0/96494117 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7080102120 msgr2=0x7f7080198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f707e59c640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7080102120 0x7f7080198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f707e59c640 1 -- 172.21.15.31:0/96494117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080107770 msgr2=0x7f708019c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f707e59c640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080107770 0x7f708019c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f707e59c640 1 -- 172.21.15.31:0/96494117 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708019cee0 con 0x7f7080101760 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.720+0000 7f707ed9d640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080107770 0x7f708019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.721+0000 7f707e59c640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7080101760 0x7f7080197f40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f707000e9f0 tx=0x7f707000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.721+0000 7f706f7fe640 1 -- 172.21.15.31:0/96494117 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707000cd80 con 0x7f7080101760 2024-04-25T02:11:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.721+0000 7f7084cd4640 1 -- 172.21.15.31:0/96494117 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f708019d1d0 con 0x7f7080101760 2024-04-25T02:11:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.721+0000 7f706f7fe640 1 -- 172.21.15.31:0/96494117 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70700108e0 con 0x7f7080101760 2024-04-25T02:11:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.721+0000 7f706f7fe640 1 -- 172.21.15.31:0/96494117 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070018920 con 0x7f7080101760 2024-04-25T02:11:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.721+0000 7f7084cd4640 1 -- 172.21.15.31:0/96494117 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70801a4a60 con 0x7f7080101760 2024-04-25T02:11:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.723+0000 7f706f7fe640 1 -- 172.21.15.31:0/96494117 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7070010a90 con 0x7f7080101760 2024-04-25T02:11:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.723+0000 7f70879e3740 1 Processor -- start 2024-04-25T02:11:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f70879e3740 1 -- start start 2024-04-25T02:11:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f70879e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 0x5620846b6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f70879e3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6a40 0x5620846c5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f70879e3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846c5640 0x5620846c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f70879e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56208461acb0 con 0x5620846b6120 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f70879e3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5620845c35b0 con 0x5620846c5640 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f70879e3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562084616c70 con 0x5620846b6a40 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f707ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 0x5620846b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f706f7fe640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70680a2da0 0x7f70680a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f707e59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6a40 0x5620846c5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f707dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846c5640 0x5620846c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f707e59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6a40 0x5620846c5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36982/0 (socket says 172.21.15.31:36982) 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f707e59c640 1 -- 172.21.15.31:0/1537763832 learned_addr learned my addr 172.21.15.31:0/1537763832 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f707ed9d640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70680a2da0 0x7f70680a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f706f7fe640 1 -- 172.21.15.31:0/96494117 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7f70700cf720 con 0x7f7080101760 2024-04-25T02:11:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f707e59c640 1 -- 172.21.15.31:0/1537763832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846c5640 msgr2=0x5620846c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f707e59c640 1 --2- 172.21.15.31:0/1537763832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846c5640 0x5620846c7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f707e59c640 1 -- 172.21.15.31:0/1537763832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 msgr2=0x5620846b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f707e59c640 1 --2- 172.21.15.31:0/1537763832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 0x5620846b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.724+0000 7f707e59c640 1 -- 172.21.15.31:0/1537763832 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56208432f210 con 0x5620846b6a40 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.725+0000 7f707e59c640 1 --2- 172.21.15.31:0/1537763832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6a40 0x5620846c5100 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f70700ccdd0 tx=0x7f707009bae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1199c8c6c83273f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.725+0000 7f707ed9d640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70680a2da0 0x7f70680a5260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f70780099e0 tx=0x7f7078004ab0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.725+0000 7f706cff9640 1 -- 172.21.15.31:0/1537763832 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707009bcb0 con 0x5620846b6a40 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.725+0000 7f706cff9640 1 -- 172.21.15.31:0/1537763832 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70700a4870 con 0x5620846b6a40 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.725+0000 7f706cff9640 1 -- 172.21.15.31:0/1537763832 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70700cfb10 con 0x5620846b6a40 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.725+0000 7f70879e3740 1 -- 172.21.15.31:0/1537763832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6a40 msgr2=0x5620846c5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.725+0000 7f70879e3740 1 --2- 172.21.15.31:0/1537763832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6a40 0x5620846c5100 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f70700ccdd0 tx=0x7f707009bae0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.725+0000 7f70879e3740 1 -- 172.21.15.31:0/1537763832 shutdown_connections 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.725+0000 7f70879e3740 1 --2- 172.21.15.31:0/1537763832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846c5640 0x5620846c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.725+0000 7f70879e3740 1 --2- 172.21.15.31:0/1537763832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6a40 0x5620846c5100 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.725+0000 7f70879e3740 1 --2- 172.21.15.31:0/1537763832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 0x5620846b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 1 -- 172.21.15.31:0/1537763832 >> 172.21.15.31:0/1537763832 conn(0x5620846b2560 msgr2=0x5620846b2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 1 -- 172.21.15.31:0/1537763832 shutdown_connections 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 1 -- 172.21.15.31:0/1537763832 wait complete. 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 1 -- start start 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 1 Processor -- start 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 0x5620846c2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846b6a40 0x5620846c2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846c5640 0x5620846c3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5620846c8420 con 0x5620846c5640 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5620845d6b50 con 0x5620846b6a40 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5620846c85a0 con 0x5620846b6120 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f70879e3740 10 client.0 _ll_register_callbacks cb 0x562084611ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f707ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846c5640 0x5620846c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f707ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846c5640 0x5620846c3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34924/0 (socket says 172.21.15.31:34924) 2024-04-25T02:11:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f707ed9d640 1 -- 172.21.15.31:0/4113203763 learned_addr learned my addr 172.21.15.31:0/4113203763 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f707e59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 0x5620846c2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.726+0000 7f707dd9b640 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846b6a40 0x5620846c2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f707ed9d640 1 -- 172.21.15.31:0/4113203763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 msgr2=0x5620846c2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f707ed9d640 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 0x5620846c2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f707ed9d640 1 -- 172.21.15.31:0/4113203763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846b6a40 msgr2=0x5620846c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f707ed9d640 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846b6a40 0x5620846c2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f707ed9d640 1 -- 172.21.15.31:0/4113203763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620843fb620 con 0x5620846c5640 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f707e59c640 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 0x5620846c2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f707ed9d640 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846c5640 0x5620846c3000 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f7078003bf0 tx=0x7f7078004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2aa2bf8d9b4377f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f7065ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f7065ffb640 1 -- 172.21.15.31:0/4113203763 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7078027070 con 0x5620846c5640 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f7065ffb640 1 -- 172.21.15.31:0/4113203763 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7f7078004b20 con 0x5620846c5640 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f70879e3740 1 -- 172.21.15.31:0/4113203763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70700a4870 con 0x5620846c5640 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f7065ffb640 1 -- 172.21.15.31:0/4113203763 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f7078032050 con 0x5620846c5640 2024-04-25T02:11:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f70879e3740 10 client.17478 fetch_fsmap learned FSMap version 48 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f70879e3740 1 -- 172.21.15.31:0/4113203763 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5620846d4650 con 0x5620846c5640 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f7065ffb640 1 -- 172.21.15.31:0/4113203763 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70700a4870 con 0x5620846c5640 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.727+0000 7f7065ffb640 1 -- 172.21.15.31:0/4113203763 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 48) ==== 8358+0+0 (secure 0 0 0) 0x7f7078005800 con 0x5620846c5640 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.728+0000 7f70879e3740 10 client.17478 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.728+0000 7f70879e3740 10 client.17478 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.728+0000 7f70879e3740 20 client.17478 populate_metadata read hostname 'smithi031' 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.728+0000 7f70879e3740 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5620846d5b30 0x5620846d7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.728+0000 7f70879e3740 4 client.17478 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.728+0000 7f70879e3740 1 -- 172.21.15.31:0/4113203763 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620846281f0 con 0x5620846d5b30 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.728+0000 7f707e59c640 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5620846d5b30 0x5620846d7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.728+0000 7f707e59c640 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5620846d5b30 0x5620846d7f70 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15af80a69390bb86 server_cookie=eb65f600b85f213d in_seq=0 out_seq=0 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.728+0000 7f7065ffb640 10 client.17478 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f7065ffb640 1 -- 172.21.15.31:0/4113203763 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5620846281f0 con 0x5620846d5b30 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f7065ffb640 10 client.17478 handle_command_reply: tid=0 2024-04-25T02:11:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 1 client.17478 shutdown 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 1 -- 172.21.15.31:0/4113203763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846c5640 msgr2=0x5620846c3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846c5640 0x5620846c3000 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f7078003bf0 tx=0x7f7078004a40 comp rx=0 tx=0).stop 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 1 -- 172.21.15.31:0/4113203763 shutdown_connections 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5620846d5b30 0x5620846d7f70 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846c5640 0x5620846c3000 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846b6a40 0x5620846c2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 1 --2- 172.21.15.31:0/4113203763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 0x5620846c2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f7065ffb640 0 client.17478 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 1 -- 172.21.15.31:0/4113203763 >> 172.21.15.31:0/4113203763 conn(0x5620846b2560 msgr2=0x5620846b82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 1 -- 172.21.15.31:0/4113203763 shutdown_connections 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 1 -- 172.21.15.31:0/4113203763 wait complete. 2024-04-25T02:11:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.729+0000 7f70879e3740 20 client.17478 trim_cache size 0 max 16384 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f70879e3740 1 Processor -- start 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f70879e3740 1 -- start start 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f70879e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 0x5620846c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f70879e3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846b6a40 0x5620846d1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f70879e3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846c5640 0x5620846dd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f70879e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f70801ab0d0 con 0x5620846b6120 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f70879e3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f70801acc40 con 0x5620846b6a40 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f70879e3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f70801ab7a0 con 0x5620846c5640 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f707e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 0x5620846c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f707e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 0x5620846c4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34936/0 (socket says 172.21.15.31:34936) 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f707e59c640 1 -- 172.21.15.31:0/3378027869 learned_addr learned my addr 172.21.15.31:0/3378027869 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f707e59c640 1 -- 172.21.15.31:0/3378027869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846c5640 msgr2=0x5620846dd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f707ed9d640 1 --2- 172.21.15.31:0/3378027869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846c5640 0x5620846dd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f707e59c640 1 --2- 172.21.15.31:0/3378027869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846c5640 0x5620846dd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f707e59c640 1 -- 172.21.15.31:0/3378027869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846b6a40 msgr2=0x5620846d1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f707e59c640 1 --2- 172.21.15.31:0/3378027869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846b6a40 0x5620846d1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f707e59c640 1 -- 172.21.15.31:0/3378027869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620846b8a20 con 0x5620846b6120 2024-04-25T02:11:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f707ed9d640 1 --2- 172.21.15.31:0/3378027869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846c5640 0x5620846dd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:11:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.744+0000 7f707e59c640 1 --2- 172.21.15.31:0/3378027869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 0x5620846c4cb0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f7070098110 tx=0x7f70700cce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc33ccca28df172 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.745+0000 7f7067fff640 1 -- 172.21.15.31:0/3378027869 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70700b9070 con 0x5620846b6120 2024-04-25T02:11:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.745+0000 7f7067fff640 1 -- 172.21.15.31:0/3378027869 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f707009bcb0 con 0x5620846b6120 2024-04-25T02:11:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.745+0000 7f7067fff640 1 -- 172.21.15.31:0/3378027869 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070097a00 con 0x5620846b6120 2024-04-25T02:11:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.745+0000 7f70879e3740 1 -- 172.21.15.31:0/3378027869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 msgr2=0x5620846c4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.745+0000 7f70879e3740 1 --2- 172.21.15.31:0/3378027869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 0x5620846c4cb0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f7070098110 tx=0x7f70700cce20 comp rx=0 tx=0).stop 2024-04-25T02:11:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.745+0000 7f70879e3740 1 -- 172.21.15.31:0/3378027869 shutdown_connections 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.745+0000 7f70879e3740 1 --2- 172.21.15.31:0/3378027869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846c5640 0x5620846dd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.745+0000 7f70879e3740 1 --2- 172.21.15.31:0/3378027869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846b6a40 0x5620846d1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.745+0000 7f70879e3740 1 --2- 172.21.15.31:0/3378027869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6120 0x5620846c4cb0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.746+0000 7f70879e3740 1 -- 172.21.15.31:0/3378027869 >> 172.21.15.31:0/3378027869 conn(0x5620846b2560 msgr2=0x5620846c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.746+0000 7f70879e3740 1 -- 172.21.15.31:0/3378027869 shutdown_connections 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.746+0000 7f70879e3740 1 -- 172.21.15.31:0/3378027869 wait complete. 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.746+0000 7f70879e3740 1 -- start start 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.746+0000 7f70879e3740 1 Processor -- start 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f70879e3740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 0x5620846cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f70879e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6a40 0x5620846cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f70879e3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846c5640 0x5620846bec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f70879e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f70801ac570 con 0x5620846b6a40 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f70879e3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f70801abe70 con 0x5620846c5640 2024-04-25T02:11:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f70879e3740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f70801a90f0 con 0x5620846b6120 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f707e59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 0x5620846cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f707e59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 0x5620846cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36992/0 (socket says 172.21.15.31:36992) 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f707e59c640 1 -- 172.21.15.31:0/2350223334 learned_addr learned my addr 172.21.15.31:0/2350223334 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f707ed9d640 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846c5640 0x5620846bec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f707dd9b640 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6a40 0x5620846cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f70879e3740 10 client.0 _ll_register_callbacks cb 0x7f7078013d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f707e59c640 1 -- 172.21.15.31:0/2350223334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846c5640 msgr2=0x5620846bec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f707e59c640 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846c5640 0x5620846bec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f707e59c640 1 -- 172.21.15.31:0/2350223334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6a40 msgr2=0x5620846cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f707e59c640 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6a40 0x5620846cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f707e59c640 1 -- 172.21.15.31:0/2350223334 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620846bf460 con 0x5620846b6120 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.747+0000 7f707dd9b640 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6a40 0x5620846cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.748+0000 7f707e59c640 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 0x5620846cd860 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7070099f60 tx=0x7f70700a48f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb325dc1428ca02d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.748+0000 7f7065ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.748+0000 7f7065ffb640 1 -- 172.21.15.31:0/2350223334 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70700aa060 con 0x5620846b6120 2024-04-25T02:11:16.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.748+0000 7f70879e3740 1 -- 172.21.15.31:0/2350223334 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f707009bcb0 con 0x5620846b6120 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.748+0000 7f7065ffb640 1 -- 172.21.15.31:0/2350223334 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7f707009b100 con 0x5620846b6120 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.748+0000 7f7065ffb640 1 -- 172.21.15.31:0/2350223334 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f70700b9070 con 0x5620846b6120 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.748+0000 7f70879e3740 10 client.26836 fetch_fsmap learned FSMap version 48 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.748+0000 7f70879e3740 1 -- 172.21.15.31:0/2350223334 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56208473bc30 con 0x5620846b6120 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.748+0000 7f7065ffb640 1 -- 172.21.15.31:0/2350223334 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f707009bcb0 con 0x5620846b6120 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.749+0000 7f7065ffb640 1 -- 172.21.15.31:0/2350223334 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 48) ==== 8358+0+0 (secure 0 0 0) 0x7f7070018ac0 con 0x5620846b6120 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.749+0000 7f70879e3740 10 client.26836 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.749+0000 7f70879e3740 10 client.26836 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.749+0000 7f70879e3740 20 client.26836 populate_metadata read hostname 'smithi031' 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.749+0000 7f70879e3740 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56208473bf70 0x56208473e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.749+0000 7f70879e3740 4 client.26836 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.749+0000 7f70879e3740 1 -- 172.21.15.31:0/2350223334 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562084622540 con 0x56208473bf70 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.749+0000 7f707dd9b640 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56208473bf70 0x56208473e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.749+0000 7f7065ffb640 10 client.26836 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:16.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.749+0000 7f707dd9b640 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56208473bf70 0x56208473e360 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d4555f15106c22b server_cookie=9ae1d53cd568450c in_seq=0 out_seq=0 2024-04-25T02:11:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f7065ffb640 1 -- 172.21.15.31:0/2350223334 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562084622540 con 0x56208473bf70 2024-04-25T02:11:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f7065ffb640 10 client.26836 handle_command_reply: tid=0 2024-04-25T02:11:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f70879e3740 1 client.26836 shutdown 2024-04-25T02:11:16.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f70879e3740 1 -- 172.21.15.31:0/2350223334 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 msgr2=0x5620846cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f70879e3740 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 0x5620846cd860 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7070099f60 tx=0x7f70700a48f0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f70879e3740 1 -- 172.21.15.31:0/2350223334 shutdown_connections 2024-04-25T02:11:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f70879e3740 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56208473bf70 0x56208473e360 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f70879e3740 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620846c5640 0x5620846bec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f70879e3740 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620846b6a40 0x5620846cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f70879e3740 1 --2- 172.21.15.31:0/2350223334 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5620846b6120 0x5620846cd860 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f70879e3740 1 -- 172.21.15.31:0/2350223334 >> 172.21.15.31:0/2350223334 conn(0x5620846b2560 msgr2=0x5620846c2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f7065ffb640 0 client.26836 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f70879e3740 1 -- 172.21.15.31:0/2350223334 shutdown_connections 2024-04-25T02:11:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.825+0000 7f70879e3740 1 -- 172.21.15.31:0/2350223334 wait complete. 2024-04-25T02:11:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.826+0000 7f70879e3740 20 client.26836 trim_cache size 0 max 16384 2024-04-25T02:11:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:11:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:11:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:11:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:11:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 -- 172.21.15.31:0/96494117 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70680a2da0 msgr2=0x7f70680a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70680a2da0 0x7f70680a5260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f70780099e0 tx=0x7f7078004ab0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 -- 172.21.15.31:0/96494117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7080101760 msgr2=0x7f7080197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7080101760 0x7f7080197f40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f707000e9f0 tx=0x7f707000eeb0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 -- 172.21.15.31:0/96494117 shutdown_connections 2024-04-25T02:11:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f70680a2da0 0x7f70680a5260 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7080107770 0x7f708019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7080102120 0x7f7080198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 --2- 172.21.15.31:0/96494117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7080101760 0x7f7080197f40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 -- 172.21.15.31:0/96494117 >> 172.21.15.31:0/96494117 conn(0x7f70800fd1f0 msgr2=0x7f70800fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 -- 172.21.15.31:0/96494117 shutdown_connections 2024-04-25T02:11:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:16.827+0000 7f7084cd4640 1 -- 172.21.15.31:0/96494117 wait complete. 2024-04-25T02:11:17.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 33533 8.99 MB/sec execute 81 sec latency 74.326 ms 2024-04-25T02:11:18.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 33741 9.08 MB/sec execute 82 sec latency 59.444 ms 2024-04-25T02:11:18.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:17 smithi040 ceph-mon[32418]: pgmap v528: 97 pgs: 97 active+clean; 687 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-04-25T02:11:18.218 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:17 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:11:18.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:17 smithi031 ceph-mon[29204]: pgmap v528: 97 pgs: 97 active+clean; 687 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-04-25T02:11:18.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:17 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:11:18.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:17 smithi134 ceph-mon[33098]: pgmap v528: 97 pgs: 97 active+clean; 687 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-04-25T02:11:18.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:17 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:11:19.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 33965 9.15 MB/sec execute 83 sec latency 50.070 ms 2024-04-25T02:11:20.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 34150 9.15 MB/sec execute 84 sec latency 59.309 ms 2024-04-25T02:11:20.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:19 smithi031 ceph-mon[29204]: pgmap v529: 97 pgs: 97 active+clean; 687 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-04-25T02:11:20.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:19 smithi040 ceph-mon[32418]: pgmap v529: 97 pgs: 97 active+clean; 687 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-04-25T02:11:20.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:19 smithi134 ceph-mon[33098]: pgmap v529: 97 pgs: 97 active+clean; 687 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-04-25T02:11:21.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 34278 9.09 MB/sec execute 85 sec latency 64.848 ms 2024-04-25T02:11:22.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 34454 9.15 MB/sec execute 86 sec latency 74.912 ms 2024-04-25T02:11:22.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:21 smithi031 ceph-mon[29204]: pgmap v530: 97 pgs: 97 active+clean; 702 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.8 MiB/s wr, 384 op/s 2024-04-25T02:11:22.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:21 smithi040 ceph-mon[32418]: pgmap v530: 97 pgs: 97 active+clean; 702 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.8 MiB/s wr, 384 op/s 2024-04-25T02:11:22.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:21 smithi134 ceph-mon[33098]: pgmap v530: 97 pgs: 97 active+clean; 702 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.8 MiB/s wr, 384 op/s 2024-04-25T02:11:23.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 34642 9.19 MB/sec execute 87 sec latency 56.127 ms 2024-04-25T02:11:24.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 34804 9.20 MB/sec execute 88 sec latency 51.984 ms 2024-04-25T02:11:24.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:23 smithi031 ceph-mon[29204]: pgmap v531: 97 pgs: 97 active+clean; 702 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 250 op/s 2024-04-25T02:11:24.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:23 smithi040 ceph-mon[32418]: pgmap v531: 97 pgs: 97 active+clean; 702 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 250 op/s 2024-04-25T02:11:24.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:23 smithi134 ceph-mon[33098]: pgmap v531: 97 pgs: 97 active+clean; 702 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 250 op/s 2024-04-25T02:11:25.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 34966 9.16 MB/sec execute 89 sec latency 52.222 ms 2024-04-25T02:11:26.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 35123 9.16 MB/sec execute 90 sec latency 53.605 ms 2024-04-25T02:11:26.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:25 smithi031 ceph-mon[29204]: pgmap v532: 97 pgs: 97 active+clean; 709 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.3 MiB/s wr, 365 op/s 2024-04-25T02:11:26.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:25 smithi040 ceph-mon[32418]: pgmap v532: 97 pgs: 97 active+clean; 709 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.3 MiB/s wr, 365 op/s 2024-04-25T02:11:26.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:25 smithi134 ceph-mon[33098]: pgmap v532: 97 pgs: 97 active+clean; 709 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.3 MiB/s wr, 365 op/s 2024-04-25T02:11:27.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 35285 9.10 MB/sec execute 91 sec latency 83.604 ms 2024-04-25T02:11:28.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 35456 9.06 MB/sec execute 92 sec latency 56.170 ms 2024-04-25T02:11:28.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:27 smithi031 ceph-mon[29204]: pgmap v533: 97 pgs: 97 active+clean; 709 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 246 op/s 2024-04-25T02:11:28.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:27 smithi040 ceph-mon[32418]: pgmap v533: 97 pgs: 97 active+clean; 709 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 246 op/s 2024-04-25T02:11:28.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:27 smithi134 ceph-mon[33098]: pgmap v533: 97 pgs: 97 active+clean; 709 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 246 op/s 2024-04-25T02:11:29.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 35632 9.08 MB/sec execute 93 sec latency 74.586 ms 2024-04-25T02:11:29.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.951+0000 7f1a15763640 1 -- 172.21.15.31:0/1360555061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a10100e50 msgr2=0x7f1a1010d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:29.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.951+0000 7f1a15763640 1 --2- 172.21.15.31:0/1360555061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a10100e50 0x7f1a1010d310 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009ca0 tx=0x7f1a0802a8c0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.952+0000 7f1a15763640 1 -- 172.21.15.31:0/1360555061 shutdown_connections 2024-04-25T02:11:29.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.952+0000 7f1a15763640 1 --2- 172.21.15.31:0/1360555061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a10100e50 0x7f1a1010d310 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.952+0000 7f1a15763640 1 --2- 172.21.15.31:0/1360555061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a10100490 0x7f1a10100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.952+0000 7f1a15763640 1 --2- 172.21.15.31:0/1360555061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10106550 0x7f1a10106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.952+0000 7f1a15763640 1 -- 172.21.15.31:0/1360555061 >> 172.21.15.31:0/1360555061 conn(0x7f1a100fbf80 msgr2=0x7f1a100fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:29.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.952+0000 7f1a15763640 1 -- 172.21.15.31:0/1360555061 shutdown_connections 2024-04-25T02:11:29.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.952+0000 7f1a15763640 1 -- 172.21.15.31:0/1360555061 wait complete. 2024-04-25T02:11:29.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.952+0000 7f1a15763640 1 Processor -- start 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a15763640 1 -- start start 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a15763640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10100490 0x7f1a10199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a15763640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a10100e50 0x7f1a1019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a15763640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a10106550 0x7f1a1019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a15763640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1a10110730 con 0x7f1a10106550 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a15763640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101105b0 con 0x7f1a10100e50 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a15763640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101108b0 con 0x7f1a10100490 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10100490 0x7f1a10199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a10100e50 0x7f1a1019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10100490 0x7f1a10199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:40688/0 (socket says 172.21.15.31:40688) 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a0effd640 1 -- 172.21.15.31:0/1545710295 learned_addr learned my addr 172.21.15.31:0/1545710295 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.953+0000 7f1a0f7fe640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a10106550 0x7f1a1019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.954+0000 7f1a07fff640 1 -- 172.21.15.31:0/1545710295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10100490 msgr2=0x7f1a10199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.954+0000 7f1a07fff640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10100490 0x7f1a10199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.954+0000 7f1a07fff640 1 -- 172.21.15.31:0/1545710295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a10106550 msgr2=0x7f1a1019e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.954+0000 7f1a07fff640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a10106550 0x7f1a1019e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.954+0000 7f1a07fff640 1 -- 172.21.15.31:0/1545710295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1019ee60 con 0x7f1a10100e50 2024-04-25T02:11:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.954+0000 7f1a07fff640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a10100e50 0x7f1a1019a3b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f19f800e9f0 tx=0x7f19f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.957+0000 7f1a0cff9640 1 -- 172.21.15.31:0/1545710295 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f800cd80 con 0x7f1a10100e50 2024-04-25T02:11:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.957+0000 7f1a0cff9640 1 -- 172.21.15.31:0/1545710295 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19f80035f0 con 0x7f1a10100e50 2024-04-25T02:11:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.957+0000 7f1a0cff9640 1 -- 172.21.15.31:0/1545710295 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f8010670 con 0x7f1a10100e50 2024-04-25T02:11:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.957+0000 7f1a15763640 1 -- 172.21.15.31:0/1545710295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a1019f150 con 0x7f1a10100e50 2024-04-25T02:11:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.957+0000 7f1a15763640 1 -- 172.21.15.31:0/1545710295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a101a69e0 con 0x7f1a10100e50 2024-04-25T02:11:29.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.960+0000 7f1a0cff9640 1 -- 172.21.15.31:0/1545710295 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f19f80037a0 con 0x7f1a10100e50 2024-04-25T02:11:29.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a1846a740 1 Processor -- start 2024-04-25T02:11:29.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a1846a740 1 -- start start 2024-04-25T02:11:29.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a0cff9640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f19fc0a2d70 0x7f19fc0a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a0f7fe640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f19fc0a2d70 0x7f19fc0a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a0cff9640 1 -- 172.21.15.31:0/1545710295 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7f19f80cf600 con 0x7f1a10100e50 2024-04-25T02:11:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09420 0x563533b09800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09d40 0x563533b13ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b16800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a1846a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563533a6dcb0 con 0x563533b14410 2024-04-25T02:11:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a1846a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563533a165b0 con 0x563533b09420 2024-04-25T02:11:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a1846a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563533a69c70 con 0x563533b09d40 2024-04-25T02:11:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09420 0x563533b09800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09420 0x563533b09800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47878/0 (socket says 172.21.15.31:47878) 2024-04-25T02:11:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a0effd640 1 -- 172.21.15.31:0/2038614442 learned_addr learned my addr 172.21.15.31:0/2038614442 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a0f7fe640 1 --2- 172.21.15.31:0/2038614442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b16800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.961+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09d40 0x563533b13ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.962+0000 7f1a0effd640 1 -- 172.21.15.31:0/2038614442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09d40 msgr2=0x563533b13ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.962+0000 7f1a0effd640 1 --2- 172.21.15.31:0/2038614442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09d40 0x563533b13ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.962+0000 7f1a0effd640 1 -- 172.21.15.31:0/2038614442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 msgr2=0x563533b16800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.962+0000 7f1a0effd640 1 --2- 172.21.15.31:0/2038614442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b16800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.962+0000 7f1a0effd640 1 -- 172.21.15.31:0/2038614442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563533782210 con 0x563533b09420 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.962+0000 7f1a0f7fe640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f19fc0a2d70 0x7f19fc0a5230 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1a08006010 tx=0x7f1a08031690 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.962+0000 7f1a0effd640 1 --2- 172.21.15.31:0/2038614442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09420 0x563533b09800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009e10 tx=0x7f1a00011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad7b2a93ac668846 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.962+0000 7f1a057fa640 1 -- 172.21.15.31:0/2038614442 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0001e070 con 0x563533b09420 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.962+0000 7f1a057fa640 1 -- 172.21.15.31:0/2038614442 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a00019ae0 con 0x563533b09420 2024-04-25T02:11:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.962+0000 7f1a057fa640 1 -- 172.21.15.31:0/2038614442 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0000f9c0 con 0x563533b09420 2024-04-25T02:11:29.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.963+0000 7f1a1846a740 1 -- 172.21.15.31:0/2038614442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09420 msgr2=0x563533b09800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:29.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.963+0000 7f1a1846a740 1 --2- 172.21.15.31:0/2038614442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09420 0x563533b09800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009e10 tx=0x7f1a00011810 comp rx=0 tx=0).stop 2024-04-25T02:11:29.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.963+0000 7f1a1846a740 1 -- 172.21.15.31:0/2038614442 shutdown_connections 2024-04-25T02:11:29.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.963+0000 7f1a1846a740 1 --2- 172.21.15.31:0/2038614442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b16800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.963+0000 7f1a1846a740 1 --2- 172.21.15.31:0/2038614442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09d40 0x563533b13ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.963+0000 7f1a1846a740 1 --2- 172.21.15.31:0/2038614442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09420 0x563533b09800 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.963+0000 7f1a1846a740 1 -- 172.21.15.31:0/2038614442 >> 172.21.15.31:0/2038614442 conn(0x563533b05740 msgr2=0x563533b05b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.963+0000 7f1a1846a740 1 -- 172.21.15.31:0/2038614442 shutdown_connections 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.964+0000 7f1a1846a740 1 -- 172.21.15.31:0/2038614442 wait complete. 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.964+0000 7f1a1846a740 1 -- start start 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.964+0000 7f1a1846a740 1 Processor -- start 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.964+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b11300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.964+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b09d40 0x563533b11840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.964+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b14410 0x563533b11d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a1846a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563533b17200 con 0x563533b09d40 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a1846a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563533a29b50 con 0x563533b14410 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a1846a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563533b17380 con 0x563533b09420 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b11300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b11300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:40700/0 (socket says 172.21.15.31:40700) 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a0effd640 1 -- 172.21.15.31:0/25859821 learned_addr learned my addr 172.21.15.31:0/25859821 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:29.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a07fff640 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b09d40 0x563533b11840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a1846a740 10 client.0 _ll_register_callbacks cb 0x563533a64ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a0f7fe640 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b14410 0x563533b11d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a0effd640 1 -- 172.21.15.31:0/25859821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b14410 msgr2=0x563533b11d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a0effd640 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b14410 0x563533b11d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a0effd640 1 -- 172.21.15.31:0/25859821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b09d40 msgr2=0x563533b11840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a0effd640 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b09d40 0x563533b11840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a0effd640 1 -- 172.21.15.31:0/25859821 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56353384e620 con 0x563533b09420 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a07fff640 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b09d40 0x563533b11840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f1a0effd640 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b11300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000f620 tx=0x7f1a00019660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a20784f32f377aa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.965+0000 7f19f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.966+0000 7f19f67fc640 1 -- 172.21.15.31:0/25859821 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0001e050 con 0x563533b09420 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.966+0000 7f19f67fc640 1 -- 172.21.15.31:0/25859821 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7f1a00005240 con 0x563533b09420 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.966+0000 7f1a1846a740 1 -- 172.21.15.31:0/25859821 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a00019ae0 con 0x563533b09420 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.966+0000 7f19f67fc640 1 -- 172.21.15.31:0/25859821 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1a00033050 con 0x563533b09420 2024-04-25T02:11:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.966+0000 7f1a1846a740 10 client.26848 fetch_fsmap learned FSMap version 49 2024-04-25T02:11:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.966+0000 7f1a1846a740 1 -- 172.21.15.31:0/25859821 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563533b2bb40 con 0x563533b09420 2024-04-25T02:11:29.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.967+0000 7f19f67fc640 1 -- 172.21.15.31:0/25859821 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a00019ae0 con 0x563533b09420 2024-04-25T02:11:29.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.967+0000 7f19f67fc640 1 -- 172.21.15.31:0/25859821 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f1a0000f6c0 con 0x563533b09420 2024-04-25T02:11:29.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.967+0000 7f1a1846a740 10 client.26848 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:11:29.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.967+0000 7f1a1846a740 10 client.26848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.967+0000 7f1a1846a740 20 client.26848 populate_metadata read hostname 'smithi031' 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.967+0000 7f1a1846a740 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563533b2d3d0 0x563533b2f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.967+0000 7f1a1846a740 4 client.26848 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.967+0000 7f1a1846a740 1 -- 172.21.15.31:0/25859821 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563533a7b1f0 con 0x563533b2d3d0 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.968+0000 7f1a07fff640 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563533b2d3d0 0x563533b2f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.968+0000 7f1a07fff640 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563533b2d3d0 0x563533b2f860 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0a826defc82d064 server_cookie=5ad8399fe7feec08 in_seq=0 out_seq=0 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.968+0000 7f19f67fc640 10 client.26848 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f19f67fc640 1 -- 172.21.15.31:0/25859821 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563533a7b1f0 con 0x563533b2d3d0 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f19f67fc640 10 client.26848 handle_command_reply: tid=0 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 1 client.26848 shutdown 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 1 -- 172.21.15.31:0/25859821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 msgr2=0x563533b11300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b11300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000f620 tx=0x7f1a00019660 comp rx=0 tx=0).stop 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 1 -- 172.21.15.31:0/25859821 shutdown_connections 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563533b2d3d0 0x563533b2f860 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b14410 0x563533b11d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f19f67fc640 0 client.26848 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b09d40 0x563533b11840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 1 --2- 172.21.15.31:0/25859821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b11300 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 1 -- 172.21.15.31:0/25859821 >> 172.21.15.31:0/25859821 conn(0x563533b05740 msgr2=0x563533b0b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 1 -- 172.21.15.31:0/25859821 shutdown_connections 2024-04-25T02:11:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 1 -- 172.21.15.31:0/25859821 wait complete. 2024-04-25T02:11:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.969+0000 7f1a1846a740 20 client.26848 trim_cache size 0 max 16384 2024-04-25T02:11:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a1846a740 1 Processor -- start 2024-04-25T02:11:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a1846a740 1 -- start start 2024-04-25T02:11:30.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b13a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b05020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b0b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a1846a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101aaf90 con 0x563533b14410 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a1846a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101acad0 con 0x563533b09d40 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a1846a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101ab660 con 0x563533b09420 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b05020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a0f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b0b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b13a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b05020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47892/0 (socket says 172.21.15.31:47892) 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.995+0000 7f1a07fff640 1 -- 172.21.15.31:0/3865763436 learned_addr learned my addr 172.21.15.31:0/3865763436 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.996+0000 7f1a07fff640 1 -- 172.21.15.31:0/3865763436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 msgr2=0x563533b13a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.996+0000 7f1a07fff640 1 --2- 172.21.15.31:0/3865763436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b13a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.996+0000 7f1a07fff640 1 -- 172.21.15.31:0/3865763436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 msgr2=0x563533b0b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.996+0000 7f1a07fff640 1 --2- 172.21.15.31:0/3865763436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b0b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.996+0000 7f1a07fff640 1 -- 172.21.15.31:0/3865763436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563533b102d0 con 0x563533b09d40 2024-04-25T02:11:30.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.996+0000 7f1a07fff640 1 --2- 172.21.15.31:0/3865763436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b05020 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f19f8098510 tx=0x7f19f8005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b3b75ba6c6f270e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.996+0000 7f1a04ff9640 1 -- 172.21.15.31:0/3865763436 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f80a47b0 con 0x563533b09d40 2024-04-25T02:11:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.996+0000 7f1a04ff9640 1 -- 172.21.15.31:0/3865763436 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19f8099e40 con 0x563533b09d40 2024-04-25T02:11:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.996+0000 7f1a04ff9640 1 -- 172.21.15.31:0/3865763436 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f80ba330 con 0x563533b09d40 2024-04-25T02:11:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.997+0000 7f1a1846a740 1 -- 172.21.15.31:0/3865763436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 msgr2=0x563533b05020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.997+0000 7f1a1846a740 1 --2- 172.21.15.31:0/3865763436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b05020 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f19f8098510 tx=0x7f19f8005bd0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.997+0000 7f1a1846a740 1 -- 172.21.15.31:0/3865763436 shutdown_connections 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.997+0000 7f1a1846a740 1 --2- 172.21.15.31:0/3865763436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b0b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.997+0000 7f1a1846a740 1 --2- 172.21.15.31:0/3865763436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b05020 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.997+0000 7f1a1846a740 1 --2- 172.21.15.31:0/3865763436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b13a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.997+0000 7f1a1846a740 1 -- 172.21.15.31:0/3865763436 >> 172.21.15.31:0/3865763436 conn(0x563533b05740 msgr2=0x563533b05b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.997+0000 7f1a1846a740 1 -- 172.21.15.31:0/3865763436 shutdown_connections 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.997+0000 7f1a1846a740 1 -- 172.21.15.31:0/3865763436 wait complete. 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a1846a740 1 -- start start 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a1846a740 1 Processor -- start 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b0df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b0e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a1846a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b0e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a1846a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101ac400 con 0x563533b14410 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a1846a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101abd30 con 0x563533b09d40 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a1846a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101a8fb0 con 0x563533b09420 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a1846a740 10 client.0 _ll_register_callbacks cb 0x7f1a0001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:11:30.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b0df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b0e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a0f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b0e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b0e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47908/0 (socket says 172.21.15.31:47908) 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b0df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:40716/0 (socket says 172.21.15.31:40716) 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a07fff640 1 -- 172.21.15.31:0/3344328634 learned_addr learned my addr 172.21.15.31:0/3344328634 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a07fff640 1 -- 172.21.15.31:0/3344328634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 msgr2=0x563533b0df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a07fff640 1 --2- 172.21.15.31:0/3344328634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b0df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a07fff640 1 -- 172.21.15.31:0/3344328634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 msgr2=0x563533b0e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a07fff640 1 --2- 172.21.15.31:0/3344328634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b0e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.998+0000 7f1a07fff640 1 -- 172.21.15.31:0/3344328634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563533b929e0 con 0x563533b09d40 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.999+0000 7f1a07fff640 1 --2- 172.21.15.31:0/3344328634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b0e450 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f19f80cd370 tx=0x7f19f80a47b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b691f7e5f80f22f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.999+0000 7f19f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.999+0000 7f19f67fc640 1 -- 172.21.15.31:0/3344328634 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f80981b0 con 0x563533b09d40 2024-04-25T02:11:30.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.999+0000 7f1a1846a740 1 -- 172.21.15.31:0/3344328634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19f8099e40 con 0x563533b09d40 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.999+0000 7f19f67fc640 1 -- 172.21.15.31:0/3344328634 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7f19f80ba330 con 0x563533b09d40 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.999+0000 7f19f67fc640 1 -- 172.21.15.31:0/3344328634 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f19f8098350 con 0x563533b09d40 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.999+0000 7f1a1846a740 10 client.27068 fetch_fsmap learned FSMap version 49 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.999+0000 7f1a1846a740 1 -- 172.21.15.31:0/3344328634 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563533b93c70 con 0x563533b09d40 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:29.999+0000 7f19f67fc640 1 -- 172.21.15.31:0/3344328634 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19f8099e40 con 0x563533b09d40 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.000+0000 7f19f67fc640 1 -- 172.21.15.31:0/3344328634 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f19f8097a20 con 0x563533b09d40 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.000+0000 7f1a1846a740 10 client.27068 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.000+0000 7f1a1846a740 10 client.27068 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.000+0000 7f1a1846a740 20 client.27068 populate_metadata read hostname 'smithi031' 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.000+0000 7f1a1846a740 1 --2- 172.21.15.31:0/3344328634 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563533b2d3d0 0x563533b96380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.000+0000 7f1a1846a740 4 client.27068 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.000+0000 7f1a1846a740 1 -- 172.21.15.31:0/3344328634 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563533a75540 con 0x563533b2d3d0 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.000+0000 7f1a0effd640 1 --2- 172.21.15.31:0/3344328634 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563533b2d3d0 0x563533b96380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:30.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.001+0000 7f19f67fc640 10 client.27068 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:30.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.001+0000 7f1a0effd640 1 --2- 172.21.15.31:0/3344328634 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563533b2d3d0 0x563533b96380 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7e70910c26ec9af server_cookie=bf74395550f224ca in_seq=0 out_seq=0 2024-04-25T02:11:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f19f67fc640 1 -- 172.21.15.31:0/3344328634 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563533a75540 con 0x563533b2d3d0 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f19f67fc640 10 client.27068 handle_command_reply: tid=0 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 1 client.27068 shutdown 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 1 -- 172.21.15.31:0/3344328634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 msgr2=0x563533b0e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 1 --2- 172.21.15.31:0/3344328634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b0e450 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f19f80cd370 tx=0x7f19f80a47b0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 1 -- 172.21.15.31:0/3344328634 shutdown_connections 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 1 --2- 172.21.15.31:0/3344328634 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563533b2d3d0 0x563533b96380 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 1 --2- 172.21.15.31:0/3344328634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563533b14410 0x563533b0e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 1 --2- 172.21.15.31:0/3344328634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563533b09d40 0x563533b0e450 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 1 --2- 172.21.15.31:0/3344328634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563533b09420 0x563533b0df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 1 -- 172.21.15.31:0/3344328634 >> 172.21.15.31:0/3344328634 conn(0x563533b05740 msgr2=0x563533b11b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f19f67fc640 0 client.27068 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 1 -- 172.21.15.31:0/3344328634 shutdown_connections 2024-04-25T02:11:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 1 -- 172.21.15.31:0/3344328634 wait complete. 2024-04-25T02:11:30.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.053+0000 7f1a1846a740 20 client.27068 trim_cache size 0 max 16384 2024-04-25T02:11:30.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:11:30.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:11:30.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:11:30.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:11:30.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 -- 172.21.15.31:0/1545710295 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f19fc0a2d70 msgr2=0x7f19fc0a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:30.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f19fc0a2d70 0x7f19fc0a5230 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1a08006010 tx=0x7f1a08031690 comp rx=0 tx=0).stop 2024-04-25T02:11:30.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 -- 172.21.15.31:0/1545710295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a10100e50 msgr2=0x7f1a1019a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:30.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a10100e50 0x7f1a1019a3b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f19f800e9f0 tx=0x7f19f800eeb0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 -- 172.21.15.31:0/1545710295 shutdown_connections 2024-04-25T02:11:30.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f19fc0a2d70 0x7f19fc0a5230 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a10106550 0x7f1a1019e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a10100e50 0x7f1a1019a3b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 --2- 172.21.15.31:0/1545710295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10100490 0x7f1a10199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 -- 172.21.15.31:0/1545710295 >> 172.21.15.31:0/1545710295 conn(0x7f1a100fbf80 msgr2=0x7f1a100fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 -- 172.21.15.31:0/1545710295 shutdown_connections 2024-04-25T02:11:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:30.054+0000 7f1a15763640 1 -- 172.21.15.31:0/1545710295 wait complete. 2024-04-25T02:11:30.096 INFO:tasks.workunit.client.0.smithi031.stdout: 10 35797 9.08 MB/sec execute 94 sec latency 83.624 ms 2024-04-25T02:11:30.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:29 smithi031 ceph-mon[29204]: pgmap v534: 97 pgs: 97 active+clean; 709 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 246 op/s 2024-04-25T02:11:30.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:29 smithi040 ceph-mon[32418]: pgmap v534: 97 pgs: 97 active+clean; 709 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 246 op/s 2024-04-25T02:11:30.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:29 smithi134 ceph-mon[33098]: pgmap v534: 97 pgs: 97 active+clean; 709 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 246 op/s 2024-04-25T02:11:31.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 35974 9.13 MB/sec execute 95 sec latency 85.693 ms 2024-04-25T02:11:32.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 36125 9.08 MB/sec execute 96 sec latency 57.478 ms 2024-04-25T02:11:32.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:31 smithi031 ceph-mon[29204]: pgmap v535: 97 pgs: 97 active+clean; 708 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 373 op/s 2024-04-25T02:11:32.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:31 smithi040 ceph-mon[32418]: pgmap v535: 97 pgs: 97 active+clean; 708 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 373 op/s 2024-04-25T02:11:32.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:31 smithi134 ceph-mon[33098]: pgmap v535: 97 pgs: 97 active+clean; 708 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 373 op/s 2024-04-25T02:11:33.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 36325 9.11 MB/sec execute 97 sec latency 70.964 ms 2024-04-25T02:11:34.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 36523 9.07 MB/sec execute 98 sec latency 54.942 ms 2024-04-25T02:11:34.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:33 smithi031 ceph-mon[29204]: pgmap v536: 97 pgs: 97 active+clean; 708 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 242 op/s 2024-04-25T02:11:34.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:33 smithi040 ceph-mon[32418]: pgmap v536: 97 pgs: 97 active+clean; 708 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 242 op/s 2024-04-25T02:11:34.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:33 smithi134 ceph-mon[33098]: pgmap v536: 97 pgs: 97 active+clean; 708 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 242 op/s 2024-04-25T02:11:35.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 36691 9.04 MB/sec execute 99 sec latency 52.631 ms 2024-04-25T02:11:36.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 36840 9.03 MB/sec execute 100 sec latency 57.800 ms 2024-04-25T02:11:36.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:35 smithi031 ceph-mon[29204]: pgmap v537: 97 pgs: 97 active+clean; 719 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.4 MiB/s wr, 363 op/s 2024-04-25T02:11:36.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:35 smithi040 ceph-mon[32418]: pgmap v537: 97 pgs: 97 active+clean; 719 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.4 MiB/s wr, 363 op/s 2024-04-25T02:11:36.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:35 smithi134 ceph-mon[33098]: pgmap v537: 97 pgs: 97 active+clean; 719 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.4 MiB/s wr, 363 op/s 2024-04-25T02:11:37.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 37023 9.07 MB/sec execute 101 sec latency 63.347 ms 2024-04-25T02:11:38.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 37243 9.20 MB/sec execute 102 sec latency 108.874 ms 2024-04-25T02:11:38.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:37 smithi031 ceph-mon[29204]: pgmap v538: 97 pgs: 97 active+clean; 719 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-04-25T02:11:38.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:37 smithi040 ceph-mon[32418]: pgmap v538: 97 pgs: 97 active+clean; 719 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-04-25T02:11:38.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:37 smithi134 ceph-mon[33098]: pgmap v538: 97 pgs: 97 active+clean; 719 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-04-25T02:11:39.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 37444 9.22 MB/sec execute 103 sec latency 115.340 ms 2024-04-25T02:11:40.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 37587 9.20 MB/sec execute 104 sec latency 65.265 ms 2024-04-25T02:11:40.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:39 smithi031 ceph-mon[29204]: pgmap v539: 97 pgs: 97 active+clean; 719 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-04-25T02:11:40.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:39 smithi040 ceph-mon[32418]: pgmap v539: 97 pgs: 97 active+clean; 719 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-04-25T02:11:40.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:39 smithi134 ceph-mon[33098]: pgmap v539: 97 pgs: 97 active+clean; 719 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-04-25T02:11:41.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 37715 9.14 MB/sec execute 105 sec latency 70.246 ms 2024-04-25T02:11:42.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 37928 9.20 MB/sec execute 106 sec latency 88.095 ms 2024-04-25T02:11:42.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:41 smithi031 ceph-mon[29204]: pgmap v540: 97 pgs: 97 active+clean; 734 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.6 MiB/s wr, 355 op/s 2024-04-25T02:11:42.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:41 smithi040 ceph-mon[32418]: pgmap v540: 97 pgs: 97 active+clean; 734 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.6 MiB/s wr, 355 op/s 2024-04-25T02:11:42.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:41 smithi134 ceph-mon[33098]: pgmap v540: 97 pgs: 97 active+clean; 734 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.6 MiB/s wr, 355 op/s 2024-04-25T02:11:43.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 38093 9.19 MB/sec execute 107 sec latency 54.844 ms 2024-04-25T02:11:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.188+0000 7fe729a2c640 1 -- 172.21.15.31:0/1400470278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7241029d0 msgr2=0x7fe724102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.188+0000 7fe729a2c640 1 --2- 172.21.15.31:0/1400470278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7241029d0 0x7fe724102db0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fe714009900 tx=0x7fe71402a510 comp rx=0 tx=0).stop 2024-04-25T02:11:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.188+0000 7fe729a2c640 1 -- 172.21.15.31:0/1400470278 shutdown_connections 2024-04-25T02:11:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.188+0000 7fe729a2c640 1 --2- 172.21.15.31:0/1400470278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe72410bd10 0x7fe72410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.188+0000 7fe729a2c640 1 --2- 172.21.15.31:0/1400470278 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7241032f0 0x7fe72410b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.188+0000 7fe729a2c640 1 --2- 172.21.15.31:0/1400470278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7241029d0 0x7fe724102db0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.188+0000 7fe729a2c640 1 -- 172.21.15.31:0/1400470278 >> 172.21.15.31:0/1400470278 conn(0x7fe7240777c0 msgr2=0x7fe724077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.188+0000 7fe729a2c640 1 -- 172.21.15.31:0/1400470278 shutdown_connections 2024-04-25T02:11:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.189+0000 7fe729a2c640 1 -- 172.21.15.31:0/1400470278 wait complete. 2024-04-25T02:11:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.189+0000 7fe729a2c640 1 Processor -- start 2024-04-25T02:11:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.189+0000 7fe729a2c640 1 -- start start 2024-04-25T02:11:43.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe729a2c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7241029d0 0x7fe724100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe729a2c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7241032f0 0x7fe724100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe729a2c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe72410bd10 0x7fe7241010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe729a2c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe7241119c0 con 0x7fe7241029d0 2024-04-25T02:11:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe729a2c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe724111840 con 0x7fe72410bd10 2024-04-25T02:11:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe729a2c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe724111b40 con 0x7fe7241032f0 2024-04-25T02:11:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe722ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7241029d0 0x7fe724100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe7227fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7241032f0 0x7fe724100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe722ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7241029d0 0x7fe724100620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46458/0 (socket says 172.21.15.31:46458) 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe7237fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe72410bd10 0x7fe7241010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe722ffd640 1 -- 172.21.15.31:0/2195540378 learned_addr learned my addr 172.21.15.31:0/2195540378 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe7227fc640 1 -- 172.21.15.31:0/2195540378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe72410bd10 msgr2=0x7fe7241010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe7227fc640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe72410bd10 0x7fe7241010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe7227fc640 1 -- 172.21.15.31:0/2195540378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7241029d0 msgr2=0x7fe724100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe7227fc640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7241029d0 0x7fe724100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe7227fc640 1 -- 172.21.15.31:0/2195540378 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7241a63a0 con 0x7fe7241032f0 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.190+0000 7fe722ffd640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7241029d0 0x7fe724100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.191+0000 7fe7227fc640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7241032f0 0x7fe724100b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe71800e9f0 tx=0x7fe71800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.191+0000 7fe728a2a640 1 -- 172.21.15.31:0/2195540378 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71800ccc0 con 0x7fe7241032f0 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.191+0000 7fe729a2c640 1 -- 172.21.15.31:0/2195540378 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7241a6690 con 0x7fe7241032f0 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.191+0000 7fe728a2a640 1 -- 172.21.15.31:0/2195540378 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7180108e0 con 0x7fe7241032f0 2024-04-25T02:11:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.191+0000 7fe729a2c640 1 -- 172.21.15.31:0/2195540378 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7241a6c30 con 0x7fe7241032f0 2024-04-25T02:11:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.191+0000 7fe728a2a640 1 -- 172.21.15.31:0/2195540378 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe718018660 con 0x7fe7241032f0 2024-04-25T02:11:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.194+0000 7fe728a2a640 1 -- 172.21.15.31:0/2195540378 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe718010430 con 0x7fe7241032f0 2024-04-25T02:11:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe72c810740 1 Processor -- start 2024-04-25T02:11:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe72c810740 1 -- start start 2024-04-25T02:11:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe728a2a640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe70c0a2dd0 0x7fe70c0a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe72c810740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96240 0x55bc36f96620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe72c810740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96b60 0x55bc36fa0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe728a2a640 1 -- 172.21.15.31:0/2195540378 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7fe7180cf630 con 0x7fe7241032f0 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe72c810740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36fa1230 0x55bc36fa3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe72c810740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bc36efacb0 con 0x55bc36fa1230 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe72c810740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bc36ea35b0 con 0x55bc36f96b60 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe72c810740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bc36ef6c70 con 0x55bc36f96240 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe7237fe640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe70c0a2dd0 0x7fe70c0a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe722ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96240 0x55bc36f96620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe7227fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96b60 0x55bc36fa0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe722ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96240 0x55bc36f96620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59032/0 (socket says 172.21.15.31:59032) 2024-04-25T02:11:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe722ffd640 1 -- 172.21.15.31:0/1183492391 learned_addr learned my addr 172.21.15.31:0/1183492391 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.195+0000 7fe7237fe640 1 --2- 172.21.15.31:0/1183492391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36fa1230 0x55bc36fa3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.196+0000 7fe7227fc640 1 -- 172.21.15.31:0/1183492391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96240 msgr2=0x55bc36f96620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.196+0000 7fe7227fc640 1 --2- 172.21.15.31:0/1183492391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96240 0x55bc36f96620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.196+0000 7fe7227fc640 1 -- 172.21.15.31:0/1183492391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36fa1230 msgr2=0x55bc36fa3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.196+0000 7fe7227fc640 1 --2- 172.21.15.31:0/1183492391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36fa1230 0x55bc36fa3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.196+0000 7fe7227fc640 1 -- 172.21.15.31:0/1183492391 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc36c0f210 con 0x55bc36f96b60 2024-04-25T02:11:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.196+0000 7fe7237fe640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe70c0a2dd0 0x7fe70c0a5290 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe71c005fd0 tx=0x7fe71c013b00 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.196+0000 7fe722ffd640 1 --2- 172.21.15.31:0/1183492391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96240 0x55bc36f96620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.196+0000 7fe7227fc640 1 --2- 172.21.15.31:0/1183492391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96b60 0x55bc36fa0cf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe7180ccd40 tx=0x7fe7180107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc590b88c2a7e31d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:43.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.196+0000 7fe709ffb640 1 -- 172.21.15.31:0/1183492391 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71809fec0 con 0x55bc36f96b60 2024-04-25T02:11:43.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.196+0000 7fe709ffb640 1 -- 172.21.15.31:0/1183492391 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7180b4070 con 0x55bc36f96b60 2024-04-25T02:11:43.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.196+0000 7fe709ffb640 1 -- 172.21.15.31:0/1183492391 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7180cf350 con 0x55bc36f96b60 2024-04-25T02:11:43.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.197+0000 7fe72c810740 1 -- 172.21.15.31:0/1183492391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96b60 msgr2=0x55bc36fa0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:43.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.197+0000 7fe72c810740 1 --2- 172.21.15.31:0/1183492391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96b60 0x55bc36fa0cf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe7180ccd40 tx=0x7fe7180107a0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.197+0000 7fe72c810740 1 -- 172.21.15.31:0/1183492391 shutdown_connections 2024-04-25T02:11:43.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.197+0000 7fe72c810740 1 --2- 172.21.15.31:0/1183492391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36fa1230 0x55bc36fa3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.197+0000 7fe72c810740 1 --2- 172.21.15.31:0/1183492391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96b60 0x55bc36fa0cf0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.197+0000 7fe72c810740 1 --2- 172.21.15.31:0/1183492391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96240 0x55bc36f96620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.197+0000 7fe72c810740 1 -- 172.21.15.31:0/1183492391 >> 172.21.15.31:0/1183492391 conn(0x55bc36f92560 msgr2=0x55bc36f92940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.197+0000 7fe72c810740 1 -- 172.21.15.31:0/1183492391 shutdown_connections 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.197+0000 7fe72c810740 1 -- 172.21.15.31:0/1183492391 wait complete. 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.198+0000 7fe72c810740 1 -- start start 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.198+0000 7fe72c810740 1 Processor -- start 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.198+0000 7fe72c810740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36f9e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.198+0000 7fe72c810740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f9e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe722ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36f9e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe722ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36f9e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36158/0 (socket says 172.21.15.31:36158) 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe7227fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f9e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe7227fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f9e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46472/0 (socket says 172.21.15.31:46472) 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe722ffd640 1 -- 172.21.15.31:0/1323875018 learned_addr learned my addr 172.21.15.31:0/1323875018 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe72c810740 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 0x55bc36f9ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe72c810740 1 -- 172.21.15.31:0/1323875018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bc36fa3fc0 con 0x55bc36f96b60 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe72c810740 1 -- 172.21.15.31:0/1323875018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bc36eb6b50 con 0x55bc36f96240 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe72c810740 1 -- 172.21.15.31:0/1323875018 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bc36fa4140 con 0x55bc36fa1230 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe7237fe640 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 0x55bc36f9ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe7227fc640 1 -- 172.21.15.31:0/1323875018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 msgr2=0x55bc36f9ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe7227fc640 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 0x55bc36f9ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe7227fc640 1 -- 172.21.15.31:0/1323875018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 msgr2=0x55bc36f9e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe7227fc640 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36f9e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe7227fc640 1 -- 172.21.15.31:0/1323875018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc36cdb620 con 0x55bc36f96b60 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.199+0000 7fe7237fe640 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 0x55bc36f9ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.200+0000 7fe72c810740 10 client.0 _ll_register_callbacks cb 0x55bc36ef1ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.200+0000 7fe72c810740 1 -- 172.21.15.31:0/1323875018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7180b4070 con 0x55bc36f96b60 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.200+0000 7fe7227fc640 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f9e6c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe71809b950 tx=0x7fe7180cf370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6fb3a3c736f9a77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.200+0000 7fe702ffd640 10 client.17514 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.200+0000 7fe702ffd640 1 -- 172.21.15.31:0/1323875018 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7180a4780 con 0x55bc36f96b60 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.200+0000 7fe702ffd640 1 -- 172.21.15.31:0/1323875018 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7fe7180ba300 con 0x55bc36f96b60 2024-04-25T02:11:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.200+0000 7fe702ffd640 1 -- 172.21.15.31:0/1323875018 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fe71809b260 con 0x55bc36f96b60 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.200+0000 7fe72c810740 10 client.17514 fetch_fsmap learned FSMap version 49 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.200+0000 7fe72c810740 1 -- 172.21.15.31:0/1323875018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=49}) -- 0x55bc36fafe20 con 0x55bc36f96b60 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.201+0000 7fe702ffd640 1 -- 172.21.15.31:0/1323875018 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fe718097860 con 0x55bc36f96b60 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.201+0000 7fe72c810740 10 client.17514 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.201+0000 7fe72c810740 10 client.17514 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.201+0000 7fe72c810740 20 client.17514 populate_metadata read hostname 'smithi031' 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.201+0000 7fe72c810740 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bc36fb1170 0x55bc36fb3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.201+0000 7fe72c810740 4 client.17514 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.201+0000 7fe72c810740 1 -- 172.21.15.31:0/1323875018 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc36f081f0 con 0x55bc36fb1170 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.201+0000 7fe722ffd640 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bc36fb1170 0x55bc36fb3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.201+0000 7fe702ffd640 10 client.17514 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.201+0000 7fe722ffd640 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bc36fb1170 0x55bc36fb3560 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6855534bd24b6996 server_cookie=3425b5625eb88ec3 in_seq=0 out_seq=0 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.202+0000 7fe702ffd640 1 -- 172.21.15.31:0/1323875018 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc36f081f0 con 0x55bc36fb1170 2024-04-25T02:11:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.202+0000 7fe702ffd640 10 client.17514 handle_command_reply: tid=0 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.202+0000 7fe72c810740 1 client.17514 shutdown 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe72c810740 1 -- 172.21.15.31:0/1323875018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 msgr2=0x55bc36f9e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe72c810740 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f9e6c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe71809b950 tx=0x7fe7180cf370 comp rx=0 tx=0).stop 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe72c810740 1 -- 172.21.15.31:0/1323875018 shutdown_connections 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe72c810740 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bc36fb1170 0x55bc36fb3560 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe702ffd640 0 client.17514 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe72c810740 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 0x55bc36f9ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe72c810740 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f9e6c0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe72c810740 1 --2- 172.21.15.31:0/1323875018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36f9e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe72c810740 1 -- 172.21.15.31:0/1323875018 >> 172.21.15.31:0/1323875018 conn(0x55bc36f92560 msgr2=0x55bc36f98260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe72c810740 1 -- 172.21.15.31:0/1323875018 shutdown_connections 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe72c810740 1 -- 172.21.15.31:0/1323875018 wait complete. 2024-04-25T02:11:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.203+0000 7fe72c810740 20 client.17514 trim_cache size 0 max 16384 2024-04-25T02:11:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.229+0000 7fe72c810740 1 Processor -- start 2024-04-25T02:11:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.229+0000 7fe72c810740 1 -- start start 2024-04-25T02:11:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe72c810740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36f9da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe72c810740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f989a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe72c810740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 0x55bc36fb8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe72c810740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe71000e650 con 0x55bc36f96b60 2024-04-25T02:11:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe72c810740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe710010190 con 0x55bc36f96240 2024-04-25T02:11:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe72c810740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe71000ed20 con 0x55bc36fa1230 2024-04-25T02:11:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe7227fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe7237fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 0x55bc36fb8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe7227fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46474/0 (socket says 172.21.15.31:46474) 2024-04-25T02:11:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe7227fc640 1 -- 172.21.15.31:0/1765752070 learned_addr learned my addr 172.21.15.31:0/1765752070 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe7237fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 0x55bc36fb8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59060/0 (socket says 172.21.15.31:59060) 2024-04-25T02:11:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe722ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36f9da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe7227fc640 1 -- 172.21.15.31:0/1765752070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 msgr2=0x55bc36fb8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe7227fc640 1 --2- 172.21.15.31:0/1765752070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 0x55bc36fb8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe7227fc640 1 -- 172.21.15.31:0/1765752070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 msgr2=0x55bc36f9da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe7227fc640 1 --2- 172.21.15.31:0/1765752070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36f9da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe7227fc640 1 -- 172.21.15.31:0/1765752070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc36f9f0e0 con 0x55bc36f96b60 2024-04-25T02:11:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.230+0000 7fe7227fc640 1 --2- 172.21.15.31:0/1765752070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f989a0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe7180984e0 tx=0x7fe7180ce000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9b982f2ddcb971 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.231+0000 7fe7097fa640 1 -- 172.21.15.31:0/1765752070 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71809fec0 con 0x55bc36f96b60 2024-04-25T02:11:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.231+0000 7fe7097fa640 1 -- 172.21.15.31:0/1765752070 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7180b4070 con 0x55bc36f96b60 2024-04-25T02:11:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.231+0000 7fe7097fa640 1 -- 172.21.15.31:0/1765752070 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe718010a90 con 0x55bc36f96b60 2024-04-25T02:11:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.232+0000 7fe72c810740 1 -- 172.21.15.31:0/1765752070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 msgr2=0x55bc36f989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.232+0000 7fe72c810740 1 --2- 172.21.15.31:0/1765752070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f989a0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe7180984e0 tx=0x7fe7180ce000 comp rx=0 tx=0).stop 2024-04-25T02:11:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.232+0000 7fe72c810740 1 -- 172.21.15.31:0/1765752070 shutdown_connections 2024-04-25T02:11:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.232+0000 7fe72c810740 1 --2- 172.21.15.31:0/1765752070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36fa1230 0x55bc36fb8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.232+0000 7fe72c810740 1 --2- 172.21.15.31:0/1765752070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36f96b60 0x55bc36f989a0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.232+0000 7fe72c810740 1 --2- 172.21.15.31:0/1765752070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36f9da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.232+0000 7fe72c810740 1 -- 172.21.15.31:0/1765752070 >> 172.21.15.31:0/1765752070 conn(0x55bc36f92560 msgr2=0x55bc36f9e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.232+0000 7fe72c810740 1 -- 172.21.15.31:0/1765752070 shutdown_connections 2024-04-25T02:11:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.232+0000 7fe72c810740 1 -- 172.21.15.31:0/1765752070 wait complete. 2024-04-25T02:11:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.232+0000 7fe72c810740 1 -- start start 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe72c810740 1 Processor -- start 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe72c810740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36fa95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe72c810740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96b60 0x55bc36f9aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe72c810740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36fa1230 0x55bc36f9af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe72c810740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe71000fac0 con 0x55bc36fa1230 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe72c810740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe71000f3f0 con 0x55bc36f96240 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe72c810740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe71000c650 con 0x55bc36f96b60 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe72c810740 10 client.0 _ll_register_callbacks cb 0x7fe718005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe722ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36fa95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe7237fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36fa1230 0x55bc36f9af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe7227fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96b60 0x55bc36f9aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe722ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36fa95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36170/0 (socket says 172.21.15.31:36170) 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe722ffd640 1 -- 172.21.15.31:0/408928894 learned_addr learned my addr 172.21.15.31:0/408928894 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe7227fc640 1 -- 172.21.15.31:0/408928894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 msgr2=0x55bc36fa95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe7227fc640 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36fa95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe7227fc640 1 -- 172.21.15.31:0/408928894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36fa1230 msgr2=0x55bc36f9af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe7227fc640 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36fa1230 0x55bc36f9af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe7227fc640 1 -- 172.21.15.31:0/408928894 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc36f9b560 con 0x55bc36f96b60 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.233+0000 7fe722ffd640 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36fa95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.234+0000 7fe7227fc640 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96b60 0x55bc36f9aa30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55bc36f92bd0 tx=0x7fe7180cf530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28decfd0d4c070fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.234+0000 7fe702ffd640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.234+0000 7fe702ffd640 1 -- 172.21.15.31:0/408928894 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71809fec0 con 0x55bc36f96b60 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.234+0000 7fe702ffd640 1 -- 172.21.15.31:0/408928894 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7fe718097860 con 0x55bc36f96b60 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.234+0000 7fe72c810740 1 -- 172.21.15.31:0/408928894 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7180b4070 con 0x55bc36f96b60 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.234+0000 7fe702ffd640 1 -- 172.21.15.31:0/408928894 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fe71809abe0 con 0x55bc36f96b60 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.234+0000 7fe72c810740 10 client.26872 fetch_fsmap learned FSMap version 49 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.234+0000 7fe72c810740 1 -- 172.21.15.31:0/408928894 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55bc36f9b560 con 0x55bc36f96b60 2024-04-25T02:11:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.234+0000 7fe702ffd640 1 -- 172.21.15.31:0/408928894 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7180b4070 con 0x55bc36f96b60 2024-04-25T02:11:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.234+0000 7fe702ffd640 1 -- 172.21.15.31:0/408928894 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fe7180ab030 con 0x55bc36f96b60 2024-04-25T02:11:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.234+0000 7fe72c810740 10 client.26872 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:11:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.235+0000 7fe72c810740 10 client.26872 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:11:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.235+0000 7fe72c810740 20 client.26872 populate_metadata read hostname 'smithi031' 2024-04-25T02:11:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.235+0000 7fe72c810740 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bc36fb1170 0x55bc370196a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.235+0000 7fe72c810740 4 client.26872 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:11:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.235+0000 7fe72c810740 1 -- 172.21.15.31:0/408928894 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bc36f02540 con 0x55bc36fb1170 2024-04-25T02:11:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.235+0000 7fe722ffd640 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bc36fb1170 0x55bc370196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.235+0000 7fe702ffd640 10 client.26872 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.235+0000 7fe722ffd640 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bc36fb1170 0x55bc370196a0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=689abe4d6c1e5e8 server_cookie=2b5d37b5bc74aabe in_seq=0 out_seq=0 2024-04-25T02:11:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.294+0000 7fe702ffd640 1 -- 172.21.15.31:0/408928894 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bc36f02540 con 0x55bc36fb1170 2024-04-25T02:11:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.294+0000 7fe702ffd640 10 client.26872 handle_command_reply: tid=0 2024-04-25T02:11:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.294+0000 7fe72c810740 1 client.26872 shutdown 2024-04-25T02:11:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe72c810740 1 -- 172.21.15.31:0/408928894 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96b60 msgr2=0x55bc36f9aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe72c810740 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96b60 0x55bc36f9aa30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55bc36f92bd0 tx=0x7fe7180cf530 comp rx=0 tx=0).stop 2024-04-25T02:11:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe72c810740 1 -- 172.21.15.31:0/408928894 shutdown_connections 2024-04-25T02:11:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe72c810740 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bc36fb1170 0x55bc370196a0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe72c810740 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc36fa1230 0x55bc36f9af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe72c810740 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bc36f96b60 0x55bc36f9aa30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe702ffd640 0 client.26872 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe72c810740 1 --2- 172.21.15.31:0/408928894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc36f96240 0x55bc36fa95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe72c810740 1 -- 172.21.15.31:0/408928894 >> 172.21.15.31:0/408928894 conn(0x55bc36f92560 msgr2=0x55bc36f9ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe72c810740 1 -- 172.21.15.31:0/408928894 shutdown_connections 2024-04-25T02:11:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe72c810740 1 -- 172.21.15.31:0/408928894 wait complete. 2024-04-25T02:11:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.295+0000 7fe72c810740 20 client.26872 trim_cache size 0 max 16384 2024-04-25T02:11:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:11:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:11:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:11:43.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:11:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 -- 172.21.15.31:0/2195540378 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe70c0a2dd0 msgr2=0x7fe70c0a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe70c0a2dd0 0x7fe70c0a5290 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe71c005fd0 tx=0x7fe71c013b00 comp rx=0 tx=0).stop 2024-04-25T02:11:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 -- 172.21.15.31:0/2195540378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7241032f0 msgr2=0x7fe724100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7241032f0 0x7fe724100b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe71800e9f0 tx=0x7fe71800edf0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 -- 172.21.15.31:0/2195540378 shutdown_connections 2024-04-25T02:11:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe70c0a2dd0 0x7fe70c0a5290 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe72410bd10 0x7fe7241010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7241032f0 0x7fe724100b60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 --2- 172.21.15.31:0/2195540378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7241029d0 0x7fe724100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:43.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 -- 172.21.15.31:0/2195540378 >> 172.21.15.31:0/2195540378 conn(0x7fe7240777c0 msgr2=0x7fe7240fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:43.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 -- 172.21.15.31:0/2195540378 shutdown_connections 2024-04-25T02:11:43.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:43.296+0000 7fe729a2c640 1 -- 172.21.15.31:0/2195540378 wait complete. 2024-04-25T02:11:44.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 38225 9.16 MB/sec execute 108 sec latency 71.872 ms 2024-04-25T02:11:44.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:43 smithi031 ceph-mon[29204]: pgmap v541: 97 pgs: 97 active+clean; 734 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 229 op/s 2024-04-25T02:11:44.246 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:43 smithi134 ceph-mon[33098]: pgmap v541: 97 pgs: 97 active+clean; 734 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 229 op/s 2024-04-25T02:11:44.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:43 smithi040 ceph-mon[32418]: pgmap v541: 97 pgs: 97 active+clean; 734 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 229 op/s 2024-04-25T02:11:45.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 38368 9.14 MB/sec execute 109 sec latency 59.316 ms 2024-04-25T02:11:46.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 38533 9.17 MB/sec execute 110 sec latency 71.648 ms 2024-04-25T02:11:46.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:45 smithi031 ceph-mon[29204]: pgmap v542: 97 pgs: 97 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.2 MiB/s wr, 352 op/s 2024-04-25T02:11:46.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:45 smithi040 ceph-mon[32418]: pgmap v542: 97 pgs: 97 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.2 MiB/s wr, 352 op/s 2024-04-25T02:11:46.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:45 smithi134 ceph-mon[33098]: pgmap v542: 97 pgs: 97 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.2 MiB/s wr, 352 op/s 2024-04-25T02:11:47.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 38693 9.14 MB/sec execute 111 sec latency 104.660 ms 2024-04-25T02:11:48.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 38852 9.10 MB/sec execute 112 sec latency 89.951 ms 2024-04-25T02:11:48.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:47 smithi031 ceph-mon[29204]: pgmap v543: 97 pgs: 97 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-04-25T02:11:48.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:47 smithi040 ceph-mon[32418]: pgmap v543: 97 pgs: 97 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-04-25T02:11:48.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:47 smithi134 ceph-mon[33098]: pgmap v543: 97 pgs: 97 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-04-25T02:11:49.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 39010 9.07 MB/sec execute 113 sec latency 88.624 ms 2024-04-25T02:11:50.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 39173 9.04 MB/sec execute 114 sec latency 73.830 ms 2024-04-25T02:11:50.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:49 smithi031 ceph-mon[29204]: pgmap v544: 97 pgs: 97 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-04-25T02:11:50.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:49 smithi040 ceph-mon[32418]: pgmap v544: 97 pgs: 97 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-04-25T02:11:50.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:49 smithi134 ceph-mon[33098]: pgmap v544: 97 pgs: 97 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-04-25T02:11:51.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 39358 9.11 MB/sec execute 115 sec latency 56.806 ms 2024-04-25T02:11:52.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 39491 9.09 MB/sec execute 116 sec latency 57.844 ms 2024-04-25T02:11:52.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:51 smithi031 ceph-mon[29204]: pgmap v545: 97 pgs: 97 active+clean; 740 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.2 MiB/s wr, 348 op/s 2024-04-25T02:11:52.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:51 smithi040 ceph-mon[32418]: pgmap v545: 97 pgs: 97 active+clean; 740 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.2 MiB/s wr, 348 op/s 2024-04-25T02:11:52.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:51 smithi134 ceph-mon[33098]: pgmap v545: 97 pgs: 97 active+clean; 740 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.2 MiB/s wr, 348 op/s 2024-04-25T02:11:53.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 39657 9.06 MB/sec execute 117 sec latency 57.371 ms 2024-04-25T02:11:54.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 39856 9.09 MB/sec execute 118 sec latency 56.613 ms 2024-04-25T02:11:54.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:53 smithi031 ceph-mon[29204]: pgmap v546: 97 pgs: 97 active+clean; 740 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 240 op/s 2024-04-25T02:11:54.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:53 smithi040 ceph-mon[32418]: pgmap v546: 97 pgs: 97 active+clean; 740 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 240 op/s 2024-04-25T02:11:54.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:53 smithi134 ceph-mon[33098]: pgmap v546: 97 pgs: 97 active+clean; 740 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 240 op/s 2024-04-25T02:11:55.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 40055 9.09 MB/sec execute 119 sec latency 56.461 ms 2024-04-25T02:11:56.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 40190 9.05 MB/sec execute 120 sec latency 64.930 ms 2024-04-25T02:11:56.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:55 smithi031 ceph-mon[29204]: pgmap v547: 97 pgs: 97 active+clean; 752 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 353 op/s 2024-04-25T02:11:56.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:55 smithi040 ceph-mon[32418]: pgmap v547: 97 pgs: 97 active+clean; 752 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 353 op/s 2024-04-25T02:11:56.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:55 smithi134 ceph-mon[33098]: pgmap v547: 97 pgs: 97 active+clean; 752 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 353 op/s 2024-04-25T02:11:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.577+0000 7f3896fce640 1 -- 172.21.15.31:0/432002134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3890101720 msgr2=0x7f3890101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.577+0000 7f3896fce640 1 --2- 172.21.15.31:0/432002134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3890101720 0x7f3890101ba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f387c009900 tx=0x7f387c02a410 comp rx=0 tx=0).stop 2024-04-25T02:11:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.577+0000 7f3896fce640 1 -- 172.21.15.31:0/432002134 shutdown_connections 2024-04-25T02:11:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.577+0000 7f3896fce640 1 --2- 172.21.15.31:0/432002134 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38901020e0 0x7f389010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.577+0000 7f3896fce640 1 --2- 172.21.15.31:0/432002134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3890101720 0x7f3890101ba0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.577+0000 7f3896fce640 1 --2- 172.21.15.31:0/432002134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3890107730 0x7f3890107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.577+0000 7f3896fce640 1 -- 172.21.15.31:0/432002134 >> 172.21.15.31:0/432002134 conn(0x7f38900fd1f0 msgr2=0x7f38900ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.577+0000 7f3896fce640 1 -- 172.21.15.31:0/432002134 shutdown_connections 2024-04-25T02:11:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.577+0000 7f3896fce640 1 -- 172.21.15.31:0/432002134 wait complete. 2024-04-25T02:11:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.578+0000 7f3896fce640 1 Processor -- start 2024-04-25T02:11:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.578+0000 7f3896fce640 1 -- start start 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f3896fce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3890101720 0x7f3890073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f3896fce640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38901020e0 0x7f3890073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f3896fce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3890107730 0x7f38900744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f3896fce640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38901119f0 con 0x7f3890101720 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f3896fce640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3890111870 con 0x7f3890107730 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f3896fce640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3890111b70 con 0x7f38901020e0 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f3894d6a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3890101720 0x7f3890073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f3894d6a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3890101720 0x7f3890073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53398/0 (socket says 172.21.15.31:53398) 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f3894d6a640 1 -- 172.21.15.31:0/1776275497 learned_addr learned my addr 172.21.15.31:0/1776275497 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f388ffff640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38901020e0 0x7f3890073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f389556b640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3890107730 0x7f38900744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f388ffff640 1 -- 172.21.15.31:0/1776275497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3890107730 msgr2=0x7f38900744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f388ffff640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3890107730 0x7f38900744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f388ffff640 1 -- 172.21.15.31:0/1776275497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3890101720 msgr2=0x7f3890073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:56.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f388ffff640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3890101720 0x7f3890073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f388ffff640 1 -- 172.21.15.31:0/1776275497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38901a6150 con 0x7f38901020e0 2024-04-25T02:11:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f389556b640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3890107730 0x7f38900744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.579+0000 7f3894d6a640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3890101720 0x7f3890073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.580+0000 7f388ffff640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38901020e0 0x7f3890073f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f387c009850 tx=0x7f387c030650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.580+0000 7f388dffb640 1 -- 172.21.15.31:0/1776275497 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c03f070 con 0x7f38901020e0 2024-04-25T02:11:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.580+0000 7f388dffb640 1 -- 172.21.15.31:0/1776275497 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f387c004460 con 0x7f38901020e0 2024-04-25T02:11:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.580+0000 7f3896fce640 1 -- 172.21.15.31:0/1776275497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38901a63e0 con 0x7f38901020e0 2024-04-25T02:11:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.580+0000 7f388dffb640 1 -- 172.21.15.31:0/1776275497 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c035030 con 0x7f38901020e0 2024-04-25T02:11:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.580+0000 7f3896fce640 1 -- 172.21.15.31:0/1776275497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38901a6920 con 0x7f38901020e0 2024-04-25T02:11:56.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.583+0000 7f388dffb640 1 -- 172.21.15.31:0/1776275497 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f387c042070 con 0x7f38901020e0 2024-04-25T02:11:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.584+0000 7f388dffb640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38800a2e20 0x7f38800a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.584+0000 7f388dffb640 1 -- 172.21.15.31:0/1776275497 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7f387c0f0cb0 con 0x7f38901020e0 2024-04-25T02:11:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.584+0000 7f3894d6a640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38800a2e20 0x7f38800a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.599+0000 7f3894d6a640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38800a2e20 0x7f38800a52e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f38840095f0 tx=0x7f3884005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:56.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f3899dbc740 1 Processor -- start 2024-04-25T02:11:56.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f3899dbc740 1 -- start start 2024-04-25T02:11:56.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4240 0x56016e3f4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e3fecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e401620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f3899dbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56016e358cb0 con 0x56016e3ff230 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f3899dbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56016e3015b0 con 0x56016e3f4b60 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f3899dbc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56016e354c70 con 0x56016e3f4240 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f389556b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e401620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f389556b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e401620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53410/0 (socket says 172.21.15.31:53410) 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f389556b640 1 -- 172.21.15.31:0/505576195 learned_addr learned my addr 172.21.15.31:0/505576195 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f3894d6a640 1 --2- 172.21.15.31:0/505576195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4240 0x56016e3f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f388ffff640 1 --2- 172.21.15.31:0/505576195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e3fecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f389556b640 1 -- 172.21.15.31:0/505576195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4240 msgr2=0x56016e3f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f389556b640 1 --2- 172.21.15.31:0/505576195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4240 0x56016e3f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.600+0000 7f389556b640 1 -- 172.21.15.31:0/505576195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 msgr2=0x56016e3fecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.601+0000 7f389556b640 1 --2- 172.21.15.31:0/505576195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e3fecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.601+0000 7f389556b640 1 -- 172.21.15.31:0/505576195 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56016e06d210 con 0x56016e3ff230 2024-04-25T02:11:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.601+0000 7f388ffff640 1 --2- 172.21.15.31:0/505576195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e3fecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:11:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.601+0000 7f389556b640 1 --2- 172.21.15.31:0/505576195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e401620 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f38880117e0 tx=0x7f3888011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1b2c5ddd90a348 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.601+0000 7f387affd640 1 -- 172.21.15.31:0/505576195 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888011310 con 0x56016e3ff230 2024-04-25T02:11:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.601+0000 7f387affd640 1 -- 172.21.15.31:0/505576195 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3888003e20 con 0x56016e3ff230 2024-04-25T02:11:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.601+0000 7f387affd640 1 -- 172.21.15.31:0/505576195 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38880044f0 con 0x56016e3ff230 2024-04-25T02:11:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.602+0000 7f3899dbc740 1 -- 172.21.15.31:0/505576195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 msgr2=0x56016e401620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.602+0000 7f3899dbc740 1 --2- 172.21.15.31:0/505576195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e401620 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f38880117e0 tx=0x7f3888011ca0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.602+0000 7f3899dbc740 1 -- 172.21.15.31:0/505576195 shutdown_connections 2024-04-25T02:11:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.602+0000 7f3899dbc740 1 --2- 172.21.15.31:0/505576195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e401620 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.602+0000 7f3899dbc740 1 --2- 172.21.15.31:0/505576195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e3fecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.602+0000 7f3899dbc740 1 --2- 172.21.15.31:0/505576195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4240 0x56016e3f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.602+0000 7f3899dbc740 1 -- 172.21.15.31:0/505576195 >> 172.21.15.31:0/505576195 conn(0x56016e3f0560 msgr2=0x56016e3f0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.602+0000 7f3899dbc740 1 -- 172.21.15.31:0/505576195 shutdown_connections 2024-04-25T02:11:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.602+0000 7f3899dbc740 1 -- 172.21.15.31:0/505576195 wait complete. 2024-04-25T02:11:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.602+0000 7f3899dbc740 1 -- start start 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.602+0000 7f3899dbc740 1 Processor -- start 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4240 0x56016e3fc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4b60 0x56016e3fc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3fcca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f3899dbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56016e402020 con 0x56016e3ff230 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f3899dbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56016e314b50 con 0x56016e3f4240 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f3899dbc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56016e4021a0 con 0x56016e3f4b60 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f3899dbc740 10 client.0 _ll_register_callbacks cb 0x56016e34fca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f389556b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3fcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f389556b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3fcca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53424/0 (socket says 172.21.15.31:53424) 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f389556b640 1 -- 172.21.15.31:0/1968525353 learned_addr learned my addr 172.21.15.31:0/1968525353 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f3894d6a640 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4240 0x56016e3fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f388ffff640 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4b60 0x56016e3fc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f389556b640 1 -- 172.21.15.31:0/1968525353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4b60 msgr2=0x56016e3fc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f389556b640 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4b60 0x56016e3fc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f389556b640 1 -- 172.21.15.31:0/1968525353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4240 msgr2=0x56016e3fc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f389556b640 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4240 0x56016e3fc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f389556b640 1 -- 172.21.15.31:0/1968525353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56016e139620 con 0x56016e3ff230 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f389556b640 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3fcca0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f388800a010 tx=0x7f3888011340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0bb866aac00b078 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f386ffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f386ffff640 1 -- 172.21.15.31:0/1968525353 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388802a060 con 0x56016e3ff230 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f386ffff640 1 -- 172.21.15.31:0/1968525353 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7f388800fdb0 con 0x56016e3ff230 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.603+0000 7f3899dbc740 1 -- 172.21.15.31:0/1968525353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3888003e20 con 0x56016e3ff230 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.604+0000 7f386ffff640 1 -- 172.21.15.31:0/1968525353 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f38880114b0 con 0x56016e3ff230 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.604+0000 7f3899dbc740 10 client.17532 fetch_fsmap learned FSMap version 49 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.604+0000 7f3899dbc740 1 -- 172.21.15.31:0/1968525353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56016e40e1e0 con 0x56016e3ff230 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.604+0000 7f386ffff640 1 -- 172.21.15.31:0/1968525353 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3888003e20 con 0x56016e3ff230 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.604+0000 7f386ffff640 1 -- 172.21.15.31:0/1968525353 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f3888018020 con 0x56016e3ff230 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.604+0000 7f3899dbc740 10 client.17532 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:11:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.604+0000 7f3899dbc740 10 client.17532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:11:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.604+0000 7f3899dbc740 20 client.17532 populate_metadata read hostname 'smithi031' 2024-04-25T02:11:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.604+0000 7f3899dbc740 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56016e40f530 0x56016e411920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.604+0000 7f3899dbc740 4 client.17532 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:11:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.604+0000 7f3899dbc740 1 -- 172.21.15.31:0/1968525353 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56016e3661f0 con 0x56016e40f530 2024-04-25T02:11:56.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.664+0000 7f3894d6a640 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56016e40f530 0x56016e411920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.665+0000 7f3894d6a640 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56016e40f530 0x56016e411920 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3df85da381b67ea0 server_cookie=c7ac7b497af59d0d in_seq=0 out_seq=0 2024-04-25T02:11:56.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.665+0000 7f386ffff640 10 client.17532 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:56.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.666+0000 7f386ffff640 1 -- 172.21.15.31:0/1968525353 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56016e3661f0 con 0x56016e40f530 2024-04-25T02:11:56.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.666+0000 7f386ffff640 10 client.17532 handle_command_reply: tid=0 2024-04-25T02:11:56.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.666+0000 7f3899dbc740 1 client.17532 shutdown 2024-04-25T02:11:56.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.666+0000 7f3899dbc740 1 -- 172.21.15.31:0/1968525353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 msgr2=0x56016e3fcca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:56.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.666+0000 7f3899dbc740 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3fcca0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f388800a010 tx=0x7f3888011340 comp rx=0 tx=0).stop 2024-04-25T02:11:56.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.666+0000 7f3899dbc740 1 -- 172.21.15.31:0/1968525353 shutdown_connections 2024-04-25T02:11:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.666+0000 7f3899dbc740 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56016e40f530 0x56016e411920 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.666+0000 7f3899dbc740 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3fcca0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.667+0000 7f3899dbc740 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4b60 0x56016e3fc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.667+0000 7f3899dbc740 1 --2- 172.21.15.31:0/1968525353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4240 0x56016e3fc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.667+0000 7f3899dbc740 1 -- 172.21.15.31:0/1968525353 >> 172.21.15.31:0/1968525353 conn(0x56016e3f0560 msgr2=0x56016e3f6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.667+0000 7f386ffff640 0 client.17532 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.667+0000 7f3899dbc740 1 -- 172.21.15.31:0/1968525353 shutdown_connections 2024-04-25T02:11:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.667+0000 7f3899dbc740 1 -- 172.21.15.31:0/1968525353 wait complete. 2024-04-25T02:11:56.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.667+0000 7f3899dbc740 20 client.17532 trim_cache size 0 max 16384 2024-04-25T02:11:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f3899dbc740 1 Processor -- start 2024-04-25T02:11:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f3899dbc740 1 -- start start 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4240 0x56016e40aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e3f6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3ef440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f3899dbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38901af280 con 0x56016e3ff230 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f3899dbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f38901b0dc0 con 0x56016e3f4b60 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f3899dbc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f38901af950 con 0x56016e3f4240 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f389556b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3ef440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f389556b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3ef440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53436/0 (socket says 172.21.15.31:53436) 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f389556b640 1 -- 172.21.15.31:0/673062755 learned_addr learned my addr 172.21.15.31:0/673062755 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f3894d6a640 1 --2- 172.21.15.31:0/673062755 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4240 0x56016e40aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f388ffff640 1 --2- 172.21.15.31:0/673062755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e3f6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f389556b640 1 -- 172.21.15.31:0/673062755 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4240 msgr2=0x56016e40aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f389556b640 1 --2- 172.21.15.31:0/673062755 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4240 0x56016e40aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f389556b640 1 -- 172.21.15.31:0/673062755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 msgr2=0x56016e3f6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f389556b640 1 --2- 172.21.15.31:0/673062755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e3f6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.695+0000 7f389556b640 1 -- 172.21.15.31:0/673062755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56016e3f4040 con 0x56016e3ff230 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f389556b640 1 --2- 172.21.15.31:0/673062755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3ef440 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f3888006010 tx=0x7f3888009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5cc51b53ae298b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f387a7fc640 1 -- 172.21.15.31:0/673062755 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388802a070 con 0x56016e3ff230 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f387a7fc640 1 -- 172.21.15.31:0/673062755 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f388800fbb0 con 0x56016e3ff230 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f387a7fc640 1 -- 172.21.15.31:0/673062755 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888005490 con 0x56016e3ff230 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f3899dbc740 1 -- 172.21.15.31:0/673062755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 msgr2=0x56016e3ef440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f3899dbc740 1 --2- 172.21.15.31:0/673062755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3ef440 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f3888006010 tx=0x7f3888009500 comp rx=0 tx=0).stop 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f3899dbc740 1 -- 172.21.15.31:0/673062755 shutdown_connections 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f3899dbc740 1 --2- 172.21.15.31:0/673062755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3ff230 0x56016e3ef440 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f3899dbc740 1 --2- 172.21.15.31:0/673062755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e3f6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f3899dbc740 1 --2- 172.21.15.31:0/673062755 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3f4240 0x56016e40aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f3899dbc740 1 -- 172.21.15.31:0/673062755 >> 172.21.15.31:0/673062755 conn(0x56016e3f0560 msgr2=0x56016e3eed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f3899dbc740 1 -- 172.21.15.31:0/673062755 shutdown_connections 2024-04-25T02:11:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.696+0000 7f3899dbc740 1 -- 172.21.15.31:0/673062755 wait complete. 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3899dbc740 1 -- start start 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3899dbc740 1 Processor -- start 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3f4240 0x56016e4073f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e407930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3899dbc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3ff230 0x56016e3f8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3899dbc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38901b06f0 con 0x56016e3f4240 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3899dbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f38901b0020 con 0x56016e3f4b60 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3899dbc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f38901ad280 con 0x56016e3ff230 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3899dbc740 10 client.0 _ll_register_callbacks cb 0x7f388400f480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3894d6a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3f4240 0x56016e4073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3894d6a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3f4240 0x56016e4073f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53450/0 (socket says 172.21.15.31:53450) 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f3894d6a640 1 -- 172.21.15.31:0/56656038 learned_addr learned my addr 172.21.15.31:0/56656038 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f388ffff640 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e407930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.697+0000 7f389556b640 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3ff230 0x56016e3f8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f3894d6a640 1 -- 172.21.15.31:0/56656038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3ff230 msgr2=0x56016e3f8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f3894d6a640 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3ff230 0x56016e3f8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f3894d6a640 1 -- 172.21.15.31:0/56656038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 msgr2=0x56016e407930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f3894d6a640 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e407930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f3894d6a640 1 -- 172.21.15.31:0/56656038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56016e3f8bc0 con 0x56016e3f4240 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f3894d6a640 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3f4240 0x56016e4073f0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f388400f880 tx=0x7f3884013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efaa03e9c729a79d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f386ffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f386ffff640 1 -- 172.21.15.31:0/56656038 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388402b330 con 0x56016e3f4240 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f386ffff640 1 -- 172.21.15.31:0/56656038 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8484+0+0 (secure 0 0 0) 0x7f388402b4d0 con 0x56016e3f4240 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f3899dbc740 1 -- 172.21.15.31:0/56656038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f388800fbb0 con 0x56016e3f4240 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f386ffff640 1 -- 172.21.15.31:0/56656038 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f388403b050 con 0x56016e3f4240 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f3899dbc740 10 client.17538 fetch_fsmap learned FSMap version 49 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f3899dbc740 1 -- 172.21.15.31:0/56656038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56016e475360 con 0x56016e3f4240 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f386ffff640 1 -- 172.21.15.31:0/56656038 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f388800fbb0 con 0x56016e3f4240 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.698+0000 7f386ffff640 1 -- 172.21.15.31:0/56656038 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f388401cc90 con 0x56016e3f4240 2024-04-25T02:11:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.699+0000 7f3899dbc740 10 client.17538 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:11:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.699+0000 7f3899dbc740 10 client.17538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:11:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.699+0000 7f3899dbc740 20 client.17538 populate_metadata read hostname 'smithi031' 2024-04-25T02:11:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.699+0000 7f3899dbc740 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56016e475ac0 0x56016e477ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.699+0000 7f3899dbc740 4 client.17538 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:11:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.699+0000 7f3899dbc740 1 -- 172.21.15.31:0/56656038 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56016e360540 con 0x56016e475ac0 2024-04-25T02:11:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.699+0000 7f388ffff640 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56016e475ac0 0x56016e477ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.699+0000 7f386ffff640 10 client.17538 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.699+0000 7f388ffff640 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56016e475ac0 0x56016e477ee0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98210ee86431b04c server_cookie=900aa63c96276248 in_seq=0 out_seq=0 2024-04-25T02:11:56.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f386ffff640 1 -- 172.21.15.31:0/56656038 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56016e360540 con 0x56016e475ac0 2024-04-25T02:11:56.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f386ffff640 10 client.17538 handle_command_reply: tid=0 2024-04-25T02:11:56.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 1 client.17538 shutdown 2024-04-25T02:11:56.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 1 -- 172.21.15.31:0/56656038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3f4240 msgr2=0x56016e4073f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:56.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3f4240 0x56016e4073f0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f388400f880 tx=0x7f3884013040 comp rx=0 tx=0).stop 2024-04-25T02:11:56.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 1 -- 172.21.15.31:0/56656038 shutdown_connections 2024-04-25T02:11:56.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56016e475ac0 0x56016e477ee0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56016e3ff230 0x56016e3f8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56016e3f4b60 0x56016e407930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 1 --2- 172.21.15.31:0/56656038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56016e3f4240 0x56016e4073f0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 1 -- 172.21.15.31:0/56656038 >> 172.21.15.31:0/56656038 conn(0x56016e3f0560 msgr2=0x56016e3ef5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f386ffff640 0 client.17538 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:11:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 1 -- 172.21.15.31:0/56656038 shutdown_connections 2024-04-25T02:11:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 1 -- 172.21.15.31:0/56656038 wait complete. 2024-04-25T02:11:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.754+0000 7f3899dbc740 20 client.17538 trim_cache size 0 max 16384 2024-04-25T02:11:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:11:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:11:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:11:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:11:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 -- 172.21.15.31:0/1776275497 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38800a2e20 msgr2=0x7f38800a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38800a2e20 0x7f38800a52e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f38840095f0 tx=0x7f3884005f90 comp rx=0 tx=0).stop 2024-04-25T02:11:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 -- 172.21.15.31:0/1776275497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38901020e0 msgr2=0x7f3890073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38901020e0 0x7f3890073f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f387c009850 tx=0x7f387c030650 comp rx=0 tx=0).stop 2024-04-25T02:11:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 -- 172.21.15.31:0/1776275497 shutdown_connections 2024-04-25T02:11:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f38800a2e20 0x7f38800a52e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3890107730 0x7f38900744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38901020e0 0x7f3890073f80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 --2- 172.21.15.31:0/1776275497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3890101720 0x7f3890073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 -- 172.21.15.31:0/1776275497 >> 172.21.15.31:0/1776275497 conn(0x7f38900fd1f0 msgr2=0x7f38900fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 -- 172.21.15.31:0/1776275497 shutdown_connections 2024-04-25T02:11:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:11:56.755+0000 7f3896fce640 1 -- 172.21.15.31:0/1776275497 wait complete. 2024-04-25T02:11:57.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 40315 9.04 MB/sec execute 121 sec latency 129.673 ms 2024-04-25T02:11:58.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 40479 9.05 MB/sec execute 122 sec latency 58.301 ms 2024-04-25T02:11:58.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:57 smithi031 ceph-mon[29204]: pgmap v548: 97 pgs: 97 active+clean; 752 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 230 op/s 2024-04-25T02:11:58.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:57 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:11:58.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:57 smithi040 ceph-mon[32418]: pgmap v548: 97 pgs: 97 active+clean; 752 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 230 op/s 2024-04-25T02:11:58.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:57 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:11:58.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:57 smithi134 ceph-mon[33098]: pgmap v548: 97 pgs: 97 active+clean; 752 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 230 op/s 2024-04-25T02:11:58.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:57 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:11:59.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 40667 9.12 MB/sec execute 123 sec latency 97.026 ms 2024-04-25T02:12:00.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 40872 9.14 MB/sec execute 124 sec latency 61.126 ms 2024-04-25T02:12:00.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:59 smithi031 ceph-mon[29204]: pgmap v549: 97 pgs: 97 active+clean; 752 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 230 op/s 2024-04-25T02:12:00.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:59 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:12:00.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:59 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:12:00.244 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:11:59 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:12:00.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:59 smithi040 ceph-mon[32418]: pgmap v549: 97 pgs: 97 active+clean; 752 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 230 op/s 2024-04-25T02:12:00.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:59 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:12:00.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:59 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:12:00.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:11:59 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:12:00.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:59 smithi134 ceph-mon[33098]: pgmap v549: 97 pgs: 97 active+clean; 752 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 230 op/s 2024-04-25T02:12:00.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:59 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:12:00.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:59 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:12:00.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:11:59 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:12:01.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 40986 9.11 MB/sec execute 125 sec latency 268.885 ms 2024-04-25T02:12:02.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 40986 9.03 MB/sec execute 126 sec latency 1268.993 ms 2024-04-25T02:12:02.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:01 smithi040 ceph-mon[32418]: pgmap v550: 97 pgs: 97 active+clean; 763 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.3 MiB/s wr, 349 op/s 2024-04-25T02:12:02.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:01 smithi134 ceph-mon[33098]: pgmap v550: 97 pgs: 97 active+clean; 763 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.3 MiB/s wr, 349 op/s 2024-04-25T02:12:02.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:01 smithi031 ceph-mon[29204]: pgmap v550: 97 pgs: 97 active+clean; 763 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.3 MiB/s wr, 349 op/s 2024-04-25T02:12:03.097 INFO:tasks.workunit.client.0.smithi031.stdout: 10 41026 8.98 MB/sec execute 127 sec latency 1937.990 ms 2024-04-25T02:12:04.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 41134 8.92 MB/sec execute 128 sec latency 62.553 ms 2024-04-25T02:12:04.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:03 smithi040 ceph-mon[32418]: pgmap v551: 97 pgs: 97 active+clean; 763 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 231 op/s 2024-04-25T02:12:04.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:03 smithi134 ceph-mon[33098]: pgmap v551: 97 pgs: 97 active+clean; 763 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 231 op/s 2024-04-25T02:12:04.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:03 smithi031 ceph-mon[29204]: pgmap v551: 97 pgs: 97 active+clean; 763 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 231 op/s 2024-04-25T02:12:05.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 41343 9.00 MB/sec execute 129 sec latency 55.822 ms 2024-04-25T02:12:06.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 41532 9.01 MB/sec execute 130 sec latency 57.028 ms 2024-04-25T02:12:06.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:05 smithi040 ceph-mon[32418]: pgmap v552: 97 pgs: 97 active+clean; 776 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 315 op/s 2024-04-25T02:12:06.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:05 smithi134 ceph-mon[33098]: pgmap v552: 97 pgs: 97 active+clean; 776 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 315 op/s 2024-04-25T02:12:06.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:05 smithi031 ceph-mon[29204]: pgmap v552: 97 pgs: 97 active+clean; 776 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.0 MiB/s wr, 315 op/s 2024-04-25T02:12:07.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 41666 8.98 MB/sec execute 131 sec latency 83.882 ms 2024-04-25T02:12:08.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 41816 8.98 MB/sec execute 132 sec latency 55.480 ms 2024-04-25T02:12:08.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:08 smithi031 ceph-mon[29204]: pgmap v553: 97 pgs: 97 active+clean; 776 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 202 op/s 2024-04-25T02:12:08.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:08 smithi031 ceph-mon[29204]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T02:12:08.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:08 smithi040 ceph-mon[32418]: pgmap v553: 97 pgs: 97 active+clean; 776 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 202 op/s 2024-04-25T02:12:08.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:08 smithi040 ceph-mon[32418]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T02:12:08.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:08 smithi134 ceph-mon[33098]: pgmap v553: 97 pgs: 97 active+clean; 776 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 202 op/s 2024-04-25T02:12:08.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:08 smithi134 ceph-mon[33098]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T02:12:09.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 41979 8.97 MB/sec execute 133 sec latency 70.671 ms 2024-04-25T02:12:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.934+0000 7febabcb2640 1 -- 172.21.15.31:0/3310027031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba41006a0 msgr2=0x7feba4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.934+0000 7febabcb2640 1 --2- 172.21.15.31:0/3310027031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba41006a0 0x7feba4100b20 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7feb94009930 tx=0x7feb9402a420 comp rx=0 tx=0).stop 2024-04-25T02:12:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.934+0000 7febabcb2640 1 -- 172.21.15.31:0/3310027031 shutdown_connections 2024-04-25T02:12:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.934+0000 7febabcb2640 1 --2- 172.21.15.31:0/3310027031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba4101060 0x7feba410d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.934+0000 7febabcb2640 1 --2- 172.21.15.31:0/3310027031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba41006a0 0x7feba4100b20 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.934+0000 7febabcb2640 1 --2- 172.21.15.31:0/3310027031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feba4106660 0x7feba4106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.934+0000 7febabcb2640 1 -- 172.21.15.31:0/3310027031 >> 172.21.15.31:0/3310027031 conn(0x7feba4077790 msgr2=0x7feba4077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.934+0000 7febabcb2640 1 -- 172.21.15.31:0/3310027031 shutdown_connections 2024-04-25T02:12:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.934+0000 7febabcb2640 1 -- 172.21.15.31:0/3310027031 wait complete. 2024-04-25T02:12:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.934+0000 7febabcb2640 1 Processor -- start 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7febabcb2640 1 -- start start 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7febabcb2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba41006a0 0x7feba4197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7febabcb2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba4101060 0x7feba41983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7febabcb2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feba4106660 0x7feba419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7febabcb2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feba4110920 con 0x7feba41006a0 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7febabcb2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feba41107a0 con 0x7feba4101060 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7febabcb2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7feba4110aa0 con 0x7feba4106660 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7feba9a4e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba41006a0 0x7feba4197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7feba9a4e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba41006a0 0x7feba4197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48064/0 (socket says 172.21.15.31:48064) 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7feba9a4e640 1 -- 172.21.15.31:0/4174250770 learned_addr learned my addr 172.21.15.31:0/4174250770 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7febaa24f640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feba4106660 0x7feba419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7feba924d640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba4101060 0x7feba41983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7feba9a4e640 1 -- 172.21.15.31:0/4174250770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feba4106660 msgr2=0x7feba419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7feba9a4e640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feba4106660 0x7feba419c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7feba9a4e640 1 -- 172.21.15.31:0/4174250770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba4101060 msgr2=0x7feba41983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7feba9a4e640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba4101060 0x7feba41983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7feba9a4e640 1 -- 172.21.15.31:0/4174250770 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba419cea0 con 0x7feba41006a0 2024-04-25T02:12:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.935+0000 7febaa24f640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feba4106660 0x7feba419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:12:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.936+0000 7feba9a4e640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba41006a0 0x7feba4197eb0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7feba000e9f0 tx=0x7feba000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.936+0000 7feb9affd640 1 -- 172.21.15.31:0/4174250770 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba000cd80 con 0x7feba41006a0 2024-04-25T02:12:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.936+0000 7febabcb2640 1 -- 172.21.15.31:0/4174250770 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba419d190 con 0x7feba41006a0 2024-04-25T02:12:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.936+0000 7febabcb2640 1 -- 172.21.15.31:0/4174250770 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba41a4a20 con 0x7feba41006a0 2024-04-25T02:12:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.936+0000 7feb9affd640 1 -- 172.21.15.31:0/4174250770 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feba00054d0 con 0x7feba41006a0 2024-04-25T02:12:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.936+0000 7feb9affd640 1 -- 172.21.15.31:0/4174250770 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba0010490 con 0x7feba41006a0 2024-04-25T02:12:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.939+0000 7feb9affd640 1 -- 172.21.15.31:0/4174250770 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7feba0005020 con 0x7feba41006a0 2024-04-25T02:12:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.939+0000 7febae9bd740 1 Processor -- start 2024-04-25T02:12:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7febae9bd740 1 -- start start 2024-04-25T02:12:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7febae9bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 0x564b73c072f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7febae9bd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c1a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7febae9bd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c1d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7febae9bd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564b73b6bcb0 con 0x564b73c1ae10 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7febae9bd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564b73b145b0 con 0x564b73c07830 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7febae9bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564b73b67c70 con 0x564b73c06f10 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7feb9affd640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7feb900a2e20 0x7feb900a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7feba9a4e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c1a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7febaa24f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 0x564b73c072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7feba9a4e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c1a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55684/0 (socket says 172.21.15.31:55684) 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7febaa24f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 0x564b73c072f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:41880/0 (socket says 172.21.15.31:41880) 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7feba9a4e640 1 -- 172.21.15.31:0/3032486475 learned_addr learned my addr 172.21.15.31:0/3032486475 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7febaa24f640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7feb900a2e20 0x7feb900a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.940+0000 7feba924d640 1 --2- 172.21.15.31:0/3032486475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c1d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7febaa24f640 1 -- 172.21.15.31:0/3032486475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 msgr2=0x564b73c1a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7febaa24f640 1 --2- 172.21.15.31:0/3032486475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c1a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7febaa24f640 1 -- 172.21.15.31:0/3032486475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 msgr2=0x564b73c1d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7febaa24f640 1 --2- 172.21.15.31:0/3032486475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c1d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7febaa24f640 1 -- 172.21.15.31:0/3032486475 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b73880210 con 0x564b73c06f10 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7feba9a4e640 1 --2- 172.21.15.31:0/3032486475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c1a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7febaa24f640 1 --2- 172.21.15.31:0/3032486475 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 0x564b73c072f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feb9c005fd0 tx=0x7feb9c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd5fd8ece9b95ea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7feb9affd640 1 -- 172.21.15.31:0/4174250770 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8528+0+0 (secure 0 0 0) 0x7feba00cf6d0 con 0x7feba41006a0 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7febaa24f640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7feb900a2e20 0x7feb900a52e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb9c010ca0 tx=0x7feb9c008000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7feb8ffff640 1 -- 172.21.15.31:0/3032486475 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c024070 con 0x564b73c06f10 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7feb8ffff640 1 -- 172.21.15.31:0/3032486475 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb9c009d20 con 0x564b73c06f10 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.941+0000 7feb8ffff640 1 -- 172.21.15.31:0/3032486475 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c01fb90 con 0x564b73c06f10 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.942+0000 7febae9bd740 1 -- 172.21.15.31:0/3032486475 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 msgr2=0x564b73c072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.942+0000 7febae9bd740 1 --2- 172.21.15.31:0/3032486475 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 0x564b73c072f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feb9c005fd0 tx=0x7feb9c0107a0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.943+0000 7febae9bd740 1 -- 172.21.15.31:0/3032486475 shutdown_connections 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.943+0000 7febae9bd740 1 --2- 172.21.15.31:0/3032486475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c1d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.943+0000 7febae9bd740 1 --2- 172.21.15.31:0/3032486475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c1a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.943+0000 7febae9bd740 1 --2- 172.21.15.31:0/3032486475 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 0x564b73c072f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.943+0000 7febae9bd740 1 -- 172.21.15.31:0/3032486475 >> 172.21.15.31:0/3032486475 conn(0x564b73c03560 msgr2=0x564b73c03940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.943+0000 7febae9bd740 1 -- 172.21.15.31:0/3032486475 shutdown_connections 2024-04-25T02:12:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.943+0000 7febae9bd740 1 -- 172.21.15.31:0/3032486475 wait complete. 2024-04-25T02:12:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.943+0000 7febae9bd740 1 -- start start 2024-04-25T02:12:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.943+0000 7febae9bd740 1 Processor -- start 2024-04-25T02:12:09.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7febae9bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 0x564b73c17d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7febae9bd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c182b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7febae9bd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c187f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7febae9bd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564b73c1d7d0 con 0x564b73c1ae10 2024-04-25T02:12:09.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7febae9bd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564b73b27b50 con 0x564b73c07830 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7febae9bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564b73c1d950 con 0x564b73c06f10 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7febae9bd740 10 client.0 _ll_register_callbacks cb 0x564b73b62ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7feba924d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c182b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7feba924d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c182b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:55694/0 (socket says 172.21.15.31:55694) 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7feba9a4e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 0x564b73c17d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7feba924d640 1 -- 172.21.15.31:0/380567285 learned_addr learned my addr 172.21.15.31:0/380567285 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7febaa24f640 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c187f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7feba924d640 1 -- 172.21.15.31:0/380567285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 msgr2=0x564b73c17d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.944+0000 7feba924d640 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 0x564b73c17d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.945+0000 7feba924d640 1 -- 172.21.15.31:0/380567285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 msgr2=0x564b73c187f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.945+0000 7feba924d640 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c187f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.945+0000 7feba924d640 1 -- 172.21.15.31:0/380567285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b7394c620 con 0x564b73c07830 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.945+0000 7feba924d640 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c182b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feb94009930 tx=0x7feb94031ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8409b33aa1871164 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.945+0000 7feb8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.945+0000 7feb8d7fa640 1 -- 172.21.15.31:0/380567285 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb94035040 con 0x564b73c07830 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.945+0000 7feb8d7fa640 1 -- 172.21.15.31:0/380567285 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8528+0+0 (secure 0 0 0) 0x7feb94004050 con 0x564b73c07830 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.945+0000 7febae9bd740 1 -- 172.21.15.31:0/380567285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb9c009d20 con 0x564b73c07830 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.945+0000 7feb8d7fa640 1 -- 172.21.15.31:0/380567285 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7feb94004250 con 0x564b73c07830 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.945+0000 7febae9bd740 10 client.27113 fetch_fsmap learned FSMap version 49 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.945+0000 7febae9bd740 1 -- 172.21.15.31:0/380567285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x564b73c29a40 con 0x564b73c07830 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.946+0000 7feb8d7fa640 1 -- 172.21.15.31:0/380567285 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb9c009d20 con 0x564b73c07830 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.946+0000 7feb8d7fa640 1 -- 172.21.15.31:0/380567285 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7feb94004960 con 0x564b73c07830 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.946+0000 7febae9bd740 10 client.27113 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.946+0000 7febae9bd740 10 client.27113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.946+0000 7febae9bd740 20 client.27113 populate_metadata read hostname 'smithi031' 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.946+0000 7febae9bd740 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564b73c2af50 0x564b73c2d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.946+0000 7febae9bd740 4 client.27113 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:12:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.946+0000 7febae9bd740 1 -- 172.21.15.31:0/380567285 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b73b791f0 con 0x564b73c2af50 2024-04-25T02:12:09.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.947+0000 7feba9a4e640 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564b73c2af50 0x564b73c2d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.947+0000 7feba9a4e640 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564b73c2af50 0x564b73c2d360 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc71ac9f24345318 server_cookie=45509aaac873427b in_seq=0 out_seq=0 2024-04-25T02:12:09.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.947+0000 7feb8d7fa640 10 client.27113 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7feb8d7fa640 1 -- 172.21.15.31:0/380567285 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564b73b791f0 con 0x564b73c2af50 2024-04-25T02:12:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7feb8d7fa640 10 client.27113 handle_command_reply: tid=0 2024-04-25T02:12:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 1 client.27113 shutdown 2024-04-25T02:12:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 1 -- 172.21.15.31:0/380567285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 msgr2=0x564b73c182b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c182b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feb94009930 tx=0x7feb94031ef0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 1 -- 172.21.15.31:0/380567285 shutdown_connections 2024-04-25T02:12:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564b73c2af50 0x564b73c2d360 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c187f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c182b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 1 --2- 172.21.15.31:0/380567285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c06f10 0x564b73c17d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 1 -- 172.21.15.31:0/380567285 >> 172.21.15.31:0/380567285 conn(0x564b73c03560 msgr2=0x564b73c093d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:09.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7feb8d7fa640 0 client.27113 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:09.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 1 -- 172.21.15.31:0/380567285 shutdown_connections 2024-04-25T02:12:09.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 1 -- 172.21.15.31:0/380567285 wait complete. 2024-04-25T02:12:09.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.948+0000 7febae9bd740 20 client.27113 trim_cache size 0 max 16384 2024-04-25T02:12:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.971+0000 7febae9bd740 1 Processor -- start 2024-04-25T02:12:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.971+0000 7febae9bd740 1 -- start start 2024-04-25T02:12:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.971+0000 7febae9bd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c06f10 0x564b73c1a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.971+0000 7febae9bd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c26700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.971+0000 7febae9bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c1ae10 0x564b73c32c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.971+0000 7febae9bd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feb8800e790 con 0x564b73c06f10 2024-04-25T02:12:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7febae9bd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb880102d0 con 0x564b73c07830 2024-04-25T02:12:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7febae9bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7feb8800ee60 con 0x564b73c1ae10 2024-04-25T02:12:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feba9a4e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c06f10 0x564b73c1a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feba9a4e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c06f10 0x564b73c1a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48098/0 (socket says 172.21.15.31:48098) 2024-04-25T02:12:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feba9a4e640 1 -- 172.21.15.31:0/2622694788 learned_addr learned my addr 172.21.15.31:0/2622694788 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feba924d640 1 --2- 172.21.15.31:0/2622694788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c26700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7febaa24f640 1 --2- 172.21.15.31:0/2622694788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c1ae10 0x564b73c32c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feba9a4e640 1 -- 172.21.15.31:0/2622694788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c1ae10 msgr2=0x564b73c32c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feba9a4e640 1 --2- 172.21.15.31:0/2622694788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c1ae10 0x564b73c32c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feba9a4e640 1 -- 172.21.15.31:0/2622694788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 msgr2=0x564b73c26700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feba9a4e640 1 --2- 172.21.15.31:0/2622694788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c26700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feba9a4e640 1 -- 172.21.15.31:0/2622694788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b73c1d570 con 0x564b73c06f10 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feba924d640 1 --2- 172.21.15.31:0/2622694788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c26700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feba9a4e640 1 --2- 172.21.15.31:0/2622694788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c06f10 0x564b73c1a4a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7feba0098580 tx=0x7feba00cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cea0fe921ce7be7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feb8f7fe640 1 -- 172.21.15.31:0/2622694788 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba009c2b0 con 0x564b73c06f10 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feb8f7fe640 1 -- 172.21.15.31:0/2622694788 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feba009bbb0 con 0x564b73c06f10 2024-04-25T02:12:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.972+0000 7feb8f7fe640 1 -- 172.21.15.31:0/2622694788 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba00a4390 con 0x564b73c06f10 2024-04-25T02:12:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.973+0000 7febae9bd740 1 -- 172.21.15.31:0/2622694788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c06f10 msgr2=0x564b73c1a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.973+0000 7febae9bd740 1 --2- 172.21.15.31:0/2622694788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c06f10 0x564b73c1a4a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7feba0098580 tx=0x7feba00cf040 comp rx=0 tx=0).stop 2024-04-25T02:12:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.973+0000 7febae9bd740 1 -- 172.21.15.31:0/2622694788 shutdown_connections 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.973+0000 7febae9bd740 1 --2- 172.21.15.31:0/2622694788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c1ae10 0x564b73c32c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.973+0000 7febae9bd740 1 --2- 172.21.15.31:0/2622694788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c07830 0x564b73c26700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.973+0000 7febae9bd740 1 --2- 172.21.15.31:0/2622694788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c06f10 0x564b73c1a4a0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.973+0000 7febae9bd740 1 -- 172.21.15.31:0/2622694788 >> 172.21.15.31:0/2622694788 conn(0x564b73c03560 msgr2=0x564b73c0a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.973+0000 7febae9bd740 1 -- 172.21.15.31:0/2622694788 shutdown_connections 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.973+0000 7febae9bd740 1 -- 172.21.15.31:0/2622694788 wait complete. 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.974+0000 7febae9bd740 1 -- start start 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.974+0000 7febae9bd740 1 Processor -- start 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7febae9bd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c06f10 0x564b73c22d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7febae9bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c07830 0x564b73c1e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7febae9bd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c1e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7febae9bd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feb8800fc00 con 0x564b73c1ae10 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7febae9bd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb8800f530 con 0x564b73c06f10 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7febae9bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7feb8800c770 con 0x564b73c07830 2024-04-25T02:12:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7febae9bd740 10 client.0 _ll_register_callbacks cb 0x7feb94009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7feba9a4e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c06f10 0x564b73c22d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7feba924d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c07830 0x564b73c1e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7feba924d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c07830 0x564b73c1e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:41916/0 (socket says 172.21.15.31:41916) 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7feba924d640 1 -- 172.21.15.31:0/2087127628 learned_addr learned my addr 172.21.15.31:0/2087127628 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7febaa24f640 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c1e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7feba924d640 1 -- 172.21.15.31:0/2087127628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c06f10 msgr2=0x564b73c22d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7feba924d640 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c06f10 0x564b73c22d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7feba924d640 1 -- 172.21.15.31:0/2087127628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 msgr2=0x564b73c1e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7feba924d640 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c1e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7feba924d640 1 -- 172.21.15.31:0/2087127628 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b73c1ecf0 con 0x564b73c07830 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7feba924d640 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c07830 0x564b73c1e1f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feb94000c00 tx=0x7feb94032f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29405c87dbf89c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.975+0000 7feb8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.976+0000 7feb8d7fa640 1 -- 172.21.15.31:0/2087127628 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb94035050 con 0x564b73c07830 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.976+0000 7feb8d7fa640 1 -- 172.21.15.31:0/2087127628 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8528+0+0 (secure 0 0 0) 0x7feb94004770 con 0x564b73c07830 2024-04-25T02:12:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.976+0000 7febae9bd740 1 -- 172.21.15.31:0/2087127628 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feba009bbb0 con 0x564b73c07830 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.976+0000 7feb8d7fa640 1 -- 172.21.15.31:0/2087127628 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7feb94052040 con 0x564b73c07830 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.976+0000 7febae9bd740 10 client.26908 fetch_fsmap learned FSMap version 49 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.976+0000 7febae9bd740 1 -- 172.21.15.31:0/2087127628 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x564b73c912a0 con 0x564b73c07830 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.977+0000 7feb8d7fa640 1 -- 172.21.15.31:0/2087127628 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feba009bbb0 con 0x564b73c07830 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.977+0000 7feb8d7fa640 1 -- 172.21.15.31:0/2087127628 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7feb94055070 con 0x564b73c07830 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.977+0000 7febae9bd740 10 client.26908 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.977+0000 7febae9bd740 10 client.26908 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.977+0000 7febae9bd740 20 client.26908 populate_metadata read hostname 'smithi031' 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.977+0000 7febae9bd740 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564b73c91a10 0x564b73c93e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.977+0000 7febae9bd740 4 client.26908 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.977+0000 7febae9bd740 1 -- 172.21.15.31:0/2087127628 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564b73b73540 con 0x564b73c91a10 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.977+0000 7feba9a4e640 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564b73c91a10 0x564b73c93e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.977+0000 7feb8d7fa640 10 client.26908 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:09.977+0000 7feba9a4e640 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564b73c91a10 0x564b73c93e30 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e48283f2d748192 server_cookie=9eee6d0c72bb59ae in_seq=0 out_seq=0 2024-04-25T02:12:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.053+0000 7feb8d7fa640 1 -- 172.21.15.31:0/2087127628 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564b73b73540 con 0x564b73c91a10 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.053+0000 7feb8d7fa640 10 client.26908 handle_command_reply: tid=0 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.053+0000 7febae9bd740 1 client.26908 shutdown 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7febae9bd740 1 -- 172.21.15.31:0/2087127628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c07830 msgr2=0x564b73c1e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7febae9bd740 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c07830 0x564b73c1e1f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feb94000c00 tx=0x7feb94032f70 comp rx=0 tx=0).stop 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7febae9bd740 1 -- 172.21.15.31:0/2087127628 shutdown_connections 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7febae9bd740 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x564b73c91a10 0x564b73c93e30 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7febae9bd740 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564b73c1ae10 0x564b73c1e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7feb8d7fa640 0 client.26908 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7febae9bd740 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564b73c07830 0x564b73c1e1f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7febae9bd740 1 --2- 172.21.15.31:0/2087127628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b73c06f10 0x564b73c22d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7febae9bd740 1 -- 172.21.15.31:0/2087127628 >> 172.21.15.31:0/2087127628 conn(0x564b73c03560 msgr2=0x564b73c08ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:10.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7febae9bd740 1 -- 172.21.15.31:0/2087127628 shutdown_connections 2024-04-25T02:12:10.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7febae9bd740 1 -- 172.21.15.31:0/2087127628 wait complete. 2024-04-25T02:12:10.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.054+0000 7febae9bd740 20 client.26908 trim_cache size 0 max 16384 2024-04-25T02:12:10.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:12:10.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:12:10.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:12:10.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:12:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.055+0000 7febabcb2640 1 -- 172.21.15.31:0/4174250770 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7feb900a2e20 msgr2=0x7feb900a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.055+0000 7febabcb2640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7feb900a2e20 0x7feb900a52e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb9c010ca0 tx=0x7feb9c008000 comp rx=0 tx=0).stop 2024-04-25T02:12:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.055+0000 7febabcb2640 1 -- 172.21.15.31:0/4174250770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba41006a0 msgr2=0x7feba4197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.055+0000 7febabcb2640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba41006a0 0x7feba4197eb0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7feba000e9f0 tx=0x7feba000edf0 comp rx=0 tx=0).stop 2024-04-25T02:12:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.056+0000 7febabcb2640 1 -- 172.21.15.31:0/4174250770 shutdown_connections 2024-04-25T02:12:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.056+0000 7febabcb2640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7feb900a2e20 0x7feb900a52e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.056+0000 7febabcb2640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feba4106660 0x7feba419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.056+0000 7febabcb2640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba4101060 0x7feba41983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.056+0000 7febabcb2640 1 --2- 172.21.15.31:0/4174250770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feba41006a0 0x7feba4197eb0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.056+0000 7febabcb2640 1 -- 172.21.15.31:0/4174250770 >> 172.21.15.31:0/4174250770 conn(0x7feba4077790 msgr2=0x7feba4069ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:10.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.059+0000 7febabcb2640 1 -- 172.21.15.31:0/4174250770 shutdown_connections 2024-04-25T02:12:10.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:10.060+0000 7febabcb2640 1 -- 172.21.15.31:0/4174250770 wait complete. 2024-04-25T02:12:10.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 42154 8.96 MB/sec execute 134 sec latency 100.099 ms 2024-04-25T02:12:10.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:10 smithi040 ceph-mon[32418]: pgmap v555: 97 pgs: 97 active+clean; 776 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 242 op/s 2024-04-25T02:12:10.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:10 smithi134 ceph-mon[33098]: pgmap v555: 97 pgs: 97 active+clean; 776 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 242 op/s 2024-04-25T02:12:10.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:10 smithi031 ceph-mon[29204]: pgmap v555: 97 pgs: 97 active+clean; 776 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 242 op/s 2024-04-25T02:12:11.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 42318 8.92 MB/sec execute 135 sec latency 66.830 ms 2024-04-25T02:12:12.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 42465 8.90 MB/sec execute 136 sec latency 68.161 ms 2024-04-25T02:12:12.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:12 smithi040 ceph-mon[32418]: pgmap v556: 97 pgs: 97 active+clean; 799 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 235 op/s 2024-04-25T02:12:12.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:12 smithi134 ceph-mon[33098]: pgmap v556: 97 pgs: 97 active+clean; 799 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 235 op/s 2024-04-25T02:12:12.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:12 smithi031 ceph-mon[29204]: pgmap v556: 97 pgs: 97 active+clean; 799 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 235 op/s 2024-04-25T02:12:13.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 42638 8.89 MB/sec execute 137 sec latency 97.710 ms 2024-04-25T02:12:14.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 42836 8.96 MB/sec execute 138 sec latency 50.357 ms 2024-04-25T02:12:14.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:14 smithi040 ceph-mon[32418]: pgmap v557: 97 pgs: 97 active+clean; 799 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 235 op/s 2024-04-25T02:12:14.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:14 smithi040 ceph-mon[32418]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T02:12:14.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:14 smithi134 ceph-mon[33098]: pgmap v557: 97 pgs: 97 active+clean; 799 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 235 op/s 2024-04-25T02:12:14.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:14 smithi134 ceph-mon[33098]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T02:12:14.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:14 smithi031 ceph-mon[29204]: pgmap v557: 97 pgs: 97 active+clean; 799 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 235 op/s 2024-04-25T02:12:14.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:14 smithi031 ceph-mon[29204]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T02:12:15.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 43003 8.93 MB/sec execute 139 sec latency 65.838 ms 2024-04-25T02:12:16.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 43182 8.94 MB/sec execute 140 sec latency 96.879 ms 2024-04-25T02:12:16.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:16 smithi040 ceph-mon[32418]: pgmap v559: 97 pgs: 97 active+clean; 809 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.9 MiB/s wr, 383 op/s 2024-04-25T02:12:16.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:16 smithi134 ceph-mon[33098]: pgmap v559: 97 pgs: 97 active+clean; 809 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.9 MiB/s wr, 383 op/s 2024-04-25T02:12:16.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:16 smithi031 ceph-mon[29204]: pgmap v559: 97 pgs: 97 active+clean; 809 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.9 MiB/s wr, 383 op/s 2024-04-25T02:12:17.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 43369 8.93 MB/sec execute 141 sec latency 59.863 ms 2024-04-25T02:12:18.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 43523 8.92 MB/sec execute 142 sec latency 61.787 ms 2024-04-25T02:12:18.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:18 smithi040 ceph-mon[32418]: pgmap v560: 97 pgs: 97 active+clean; 809 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 311 op/s 2024-04-25T02:12:18.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:18 smithi134 ceph-mon[33098]: pgmap v560: 97 pgs: 97 active+clean; 809 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 311 op/s 2024-04-25T02:12:18.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:18 smithi031 ceph-mon[29204]: pgmap v560: 97 pgs: 97 active+clean; 809 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 311 op/s 2024-04-25T02:12:19.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 43705 8.94 MB/sec execute 143 sec latency 119.411 ms 2024-04-25T02:12:20.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 43890 8.95 MB/sec execute 144 sec latency 56.489 ms 2024-04-25T02:12:20.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:20 smithi040 ceph-mon[32418]: pgmap v561: 97 pgs: 97 active+clean; 809 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 306 op/s 2024-04-25T02:12:20.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:20 smithi134 ceph-mon[33098]: pgmap v561: 97 pgs: 97 active+clean; 809 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 306 op/s 2024-04-25T02:12:20.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:20 smithi031 ceph-mon[29204]: pgmap v561: 97 pgs: 97 active+clean; 809 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 306 op/s 2024-04-25T02:12:21.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 44053 8.97 MB/sec execute 145 sec latency 57.575 ms 2024-04-25T02:12:22.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 44256 9.01 MB/sec execute 146 sec latency 115.226 ms 2024-04-25T02:12:22.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:22 smithi040 ceph-mon[32418]: pgmap v562: 97 pgs: 97 active+clean; 821 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 311 op/s 2024-04-25T02:12:22.455 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:22 smithi134 ceph-mon[33098]: pgmap v562: 97 pgs: 97 active+clean; 821 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 311 op/s 2024-04-25T02:12:22.498 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:22 smithi031 ceph-mon[29204]: pgmap v562: 97 pgs: 97 active+clean; 821 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 311 op/s 2024-04-25T02:12:23.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 44444 9.02 MB/sec execute 147 sec latency 54.169 ms 2024-04-25T02:12:23.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.188+0000 7fabc95dc640 1 -- 172.21.15.31:0/1535023313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabc4107830 msgr2=0x7fabc4107c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:23.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.188+0000 7fabc95dc640 1 --2- 172.21.15.31:0/1535023313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabc4107830 0x7fabc4107c10 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009900 tx=0x7fabb402a480 comp rx=0 tx=0).stop 2024-04-25T02:12:23.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.188+0000 7fabc95dc640 1 -- 172.21.15.31:0/1535023313 shutdown_connections 2024-04-25T02:12:23.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.188+0000 7fabc95dc640 1 --2- 172.21.15.31:0/1535023313 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabc406a2e0 0x7fabc410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.188+0000 7fabc95dc640 1 --2- 172.21.15.31:0/1535023313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc4069940 0x7fabc4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.188+0000 7fabc95dc640 1 --2- 172.21.15.31:0/1535023313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabc4107830 0x7fabc4107c10 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.188+0000 7fabc95dc640 1 -- 172.21.15.31:0/1535023313 >> 172.21.15.31:0/1535023313 conn(0x7fabc4077520 msgr2=0x7fabc4077930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:23.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.188+0000 7fabc95dc640 1 -- 172.21.15.31:0/1535023313 shutdown_connections 2024-04-25T02:12:23.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.188+0000 7fabc95dc640 1 -- 172.21.15.31:0/1535023313 wait complete. 2024-04-25T02:12:23.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.189+0000 7fabc95dc640 1 Processor -- start 2024-04-25T02:12:23.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc95dc640 1 -- start start 2024-04-25T02:12:23.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc95dc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabc4069940 0x7fabc4108600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc95dc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc406a2e0 0x7fabc4108b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc95dc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabc4107830 0x7fabc4109080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc95dc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fabc4111a30 con 0x7fabc4107830 2024-04-25T02:12:23.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc95dc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fabc41118b0 con 0x7fabc406a2e0 2024-04-25T02:12:23.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc95dc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fabc4111bb0 con 0x7fabc4069940 2024-04-25T02:12:23.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc2ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabc4069940 0x7fabc4108600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc2ffd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabc4069940 0x7fabc4108600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:50582/0 (socket says 172.21.15.31:50582) 2024-04-25T02:12:23.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc2ffd640 1 -- 172.21.15.31:0/3140953404 learned_addr learned my addr 172.21.15.31:0/3140953404 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:23.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc27fc640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc406a2e0 0x7fabc4108b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.190+0000 7fabc37fe640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabc4107830 0x7fabc4109080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.191+0000 7fabc2ffd640 1 -- 172.21.15.31:0/3140953404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc406a2e0 msgr2=0x7fabc4108b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.191+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc406a2e0 0x7fabc4108b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.191+0000 7fabc2ffd640 1 -- 172.21.15.31:0/3140953404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabc4107830 msgr2=0x7fabc4109080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.191+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabc4107830 0x7fabc4109080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.191+0000 7fabc2ffd640 1 -- 172.21.15.31:0/3140953404 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc41aa730 con 0x7fabc4069940 2024-04-25T02:12:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.191+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabc4069940 0x7fabc4108600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fabb4006010 tx=0x7fabb4003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.191+0000 7fababfff640 1 -- 172.21.15.31:0/3140953404 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb403f070 con 0x7fabc4069940 2024-04-25T02:12:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.191+0000 7fabc95dc640 1 -- 172.21.15.31:0/3140953404 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc41aa9c0 con 0x7fabc4069940 2024-04-25T02:12:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.191+0000 7fababfff640 1 -- 172.21.15.31:0/3140953404 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabb403bb10 con 0x7fabc4069940 2024-04-25T02:12:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.191+0000 7fabc95dc640 1 -- 172.21.15.31:0/3140953404 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc41aaf00 con 0x7fabc4069940 2024-04-25T02:12:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.191+0000 7fababfff640 1 -- 172.21.15.31:0/3140953404 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb4035030 con 0x7fabc4069940 2024-04-25T02:12:23.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.194+0000 7fababfff640 1 -- 172.21.15.31:0/3140953404 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fabb4042070 con 0x7fabc4069940 2024-04-25T02:12:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.195+0000 7fabcc3c6740 1 Processor -- start 2024-04-25T02:12:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.195+0000 7fabcc3c6740 1 -- start start 2024-04-25T02:12:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.195+0000 7fababfff640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabac0a2da0 0x7fabac0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.195+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 0x555f4269b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.195+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269bb60 0x555f426a5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.195+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f426a6230 0x555f426a8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.195+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555f425ffcb0 con 0x555f4269bb60 2024-04-25T02:12:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.195+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555f425a85b0 con 0x555f4269b240 2024-04-25T02:12:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.195+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555f425fbc70 con 0x555f426a6230 2024-04-25T02:12:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 0x555f4269b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc37fe640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabac0a2da0 0x7fabac0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc27fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269bb60 0x555f426a5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 0x555f4269b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:59990/0 (socket says 172.21.15.31:59990) 2024-04-25T02:12:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc2ffd640 1 -- 172.21.15.31:0/1777100891 learned_addr learned my addr 172.21.15.31:0/1777100891 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fababfff640 1 -- 172.21.15.31:0/3140953404 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7fabb40ef9d0 con 0x7fabc4069940 2024-04-25T02:12:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc37fe640 1 --2- 172.21.15.31:0/1777100891 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f426a6230 0x555f426a8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc2ffd640 1 -- 172.21.15.31:0/1777100891 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f426a6230 msgr2=0x555f426a8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/1777100891 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f426a6230 0x555f426a8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc2ffd640 1 -- 172.21.15.31:0/1777100891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269bb60 msgr2=0x555f426a5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/1777100891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269bb60 0x555f426a5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc2ffd640 1 -- 172.21.15.31:0/1777100891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f42314210 con 0x555f4269b240 2024-04-25T02:12:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc37fe640 1 --2- 172.21.15.31:0/1777100891 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f426a6230 0x555f426a8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:12:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc27fc640 1 --2- 172.21.15.31:0/1777100891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269bb60 0x555f426a5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:12:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/1777100891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 0x555f4269b620 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fabb4004650 tx=0x7fabb402a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebfacd318d836900 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7fabc37fe640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabac0a2da0 0x7fabac0a5260 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x555f426986f0 tx=0x7fabbc009a90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.196+0000 7faba97fa640 1 -- 172.21.15.31:0/1777100891 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb4049030 con 0x555f4269b240 2024-04-25T02:12:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.197+0000 7faba97fa640 1 -- 172.21.15.31:0/1777100891 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabb40c5a40 con 0x555f4269b240 2024-04-25T02:12:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.197+0000 7faba97fa640 1 -- 172.21.15.31:0/1777100891 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb40ba350 con 0x555f4269b240 2024-04-25T02:12:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.197+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1777100891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 msgr2=0x555f4269b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.197+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1777100891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 0x555f4269b620 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fabb4004650 tx=0x7fabb402a480 comp rx=0 tx=0).stop 2024-04-25T02:12:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.198+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1777100891 shutdown_connections 2024-04-25T02:12:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.198+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1777100891 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f426a6230 0x555f426a8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.198+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1777100891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269bb60 0x555f426a5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.198+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1777100891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 0x555f4269b620 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.198+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1777100891 >> 172.21.15.31:0/1777100891 conn(0x555f42697560 msgr2=0x555f42697940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.198+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1777100891 shutdown_connections 2024-04-25T02:12:23.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.198+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1777100891 wait complete. 2024-04-25T02:12:23.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.198+0000 7fabcc3c6740 1 -- start start 2024-04-25T02:12:23.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.198+0000 7fabcc3c6740 1 Processor -- start 2024-04-25T02:12:23.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269b240 0x555f426a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269bb60 0x555f426a3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 0x555f426a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555f426a8de0 con 0x555f426a6230 2024-04-25T02:12:23.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555f425bbb50 con 0x555f4269bb60 2024-04-25T02:12:23.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555f426a8f60 con 0x555f4269b240 2024-04-25T02:12:23.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabcc3c6740 10 client.0 _ll_register_callbacks cb 0x555f425f6ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:12:23.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabc37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 0x555f426a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabc37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 0x555f426a3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57584/0 (socket says 172.21.15.31:57584) 2024-04-25T02:12:23.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabc27fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269bb60 0x555f426a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabc37fe640 1 -- 172.21.15.31:0/1808252395 learned_addr learned my addr 172.21.15.31:0/1808252395 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269b240 0x555f426a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabc37fe640 1 -- 172.21.15.31:0/1808252395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269b240 msgr2=0x555f426a31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabc37fe640 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269b240 0x555f426a31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabc37fe640 1 -- 172.21.15.31:0/1808252395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269bb60 msgr2=0x555f426a3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabc37fe640 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269bb60 0x555f426a3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabc37fe640 1 -- 172.21.15.31:0/1808252395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f423e0620 con 0x555f426a6230 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.199+0000 7fabc37fe640 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 0x555f426a3c50 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fabbc0109a0 tx=0x7fabbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ebaa12ba29771 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.200+0000 7faba6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.200+0000 7faba6ff5640 1 -- 172.21.15.31:0/1808252395 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabbc01a070 con 0x555f426a6230 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.200+0000 7faba6ff5640 1 -- 172.21.15.31:0/1808252395 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7fabbc00eb40 con 0x555f426a6230 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.200+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1808252395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabb40c5a40 con 0x555f426a6230 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.200+0000 7faba6ff5640 1 -- 172.21.15.31:0/1808252395 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fabbc02e050 con 0x555f426a6230 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.200+0000 7fabcc3c6740 10 client.17568 fetch_fsmap learned FSMap version 49 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.200+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1808252395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x555f426b4f20 con 0x555f426a6230 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.201+0000 7faba6ff5640 1 -- 172.21.15.31:0/1808252395 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabb40c5a40 con 0x555f426a6230 2024-04-25T02:12:23.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.201+0000 7faba6ff5640 1 -- 172.21.15.31:0/1808252395 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fabbc0044e0 con 0x555f426a6230 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.201+0000 7fabcc3c6740 10 client.17568 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.201+0000 7fabcc3c6740 10 client.17568 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.201+0000 7fabcc3c6740 20 client.17568 populate_metadata read hostname 'smithi031' 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.201+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555f426b68a0 0x555f426b8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.201+0000 7fabcc3c6740 4 client.17568 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.201+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1808252395 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f4260d1f0 con 0x555f426b68a0 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.201+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555f426b68a0 0x555f426b8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.202+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555f426b68a0 0x555f426b8d00 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e18a0c1ad62d323 server_cookie=97e813d5ab3ea539 in_seq=0 out_seq=0 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.202+0000 7faba6ff5640 10 client.17568 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.202+0000 7faba6ff5640 1 -- 172.21.15.31:0/1808252395 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f4260d1f0 con 0x555f426b68a0 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7faba6ff5640 10 client.17568 handle_command_reply: tid=0 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 1 client.17568 shutdown 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1808252395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 msgr2=0x555f426a3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 0x555f426a3c50 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fabbc0109a0 tx=0x7fabbc004a40 comp rx=0 tx=0).stop 2024-04-25T02:12:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1808252395 shutdown_connections 2024-04-25T02:12:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555f426b68a0 0x555f426b8d00 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 0x555f426a3c50 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269bb60 0x555f426a3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1808252395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269b240 0x555f426a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1808252395 >> 172.21.15.31:0/1808252395 conn(0x555f42697560 msgr2=0x555f4269d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7faba6ff5640 0 client.17568 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1808252395 shutdown_connections 2024-04-25T02:12:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1808252395 wait complete. 2024-04-25T02:12:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.203+0000 7fabcc3c6740 20 client.17568 trim_cache size 0 max 16384 2024-04-25T02:12:23.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.225+0000 7fabcc3c6740 1 Processor -- start 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.225+0000 7fabcc3c6740 1 -- start start 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.225+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269b240 0x555f426a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.225+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269bb60 0x555f4269d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.225+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f426a6230 0x555f426b1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fabc41b38b0 con 0x555f4269b240 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fabc41b53f0 con 0x555f4269bb60 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fabc41b3f80 con 0x555f426a6230 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269b240 0x555f426a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269b240 0x555f426a1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57586/0 (socket says 172.21.15.31:57586) 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc2ffd640 1 -- 172.21.15.31:0/2366772888 learned_addr learned my addr 172.21.15.31:0/2366772888 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc27fc640 1 --2- 172.21.15.31:0/2366772888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269bb60 0x555f4269d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc37fe640 1 --2- 172.21.15.31:0/2366772888 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f426a6230 0x555f426b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc2ffd640 1 -- 172.21.15.31:0/2366772888 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f426a6230 msgr2=0x555f426b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/2366772888 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f426a6230 0x555f426b1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc2ffd640 1 -- 172.21.15.31:0/2366772888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269bb60 msgr2=0x555f4269d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/2366772888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269bb60 0x555f4269d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc2ffd640 1 -- 172.21.15.31:0/2366772888 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f426a2180 con 0x555f4269b240 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc27fc640 1 --2- 172.21.15.31:0/2366772888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269bb60 0x555f4269d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.226+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/2366772888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269b240 0x555f426a1aa0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fabb40b83d0 tx=0x7fabb40045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc2ba4b73a17f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.227+0000 7faba8ff9640 1 -- 172.21.15.31:0/2366772888 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb40c9070 con 0x555f4269b240 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.227+0000 7faba8ff9640 1 -- 172.21.15.31:0/2366772888 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabb40c02c0 con 0x555f4269b240 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.227+0000 7faba8ff9640 1 -- 172.21.15.31:0/2366772888 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb4049040 con 0x555f4269b240 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.227+0000 7fabcc3c6740 1 -- 172.21.15.31:0/2366772888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269b240 msgr2=0x555f426a1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.227+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/2366772888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269b240 0x555f426a1aa0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fabb40b83d0 tx=0x7fabb40045a0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.227+0000 7fabcc3c6740 1 -- 172.21.15.31:0/2366772888 shutdown_connections 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.227+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/2366772888 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f426a6230 0x555f426b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.227+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/2366772888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269bb60 0x555f4269d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.227+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/2366772888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f4269b240 0x555f426a1aa0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.228+0000 7fabcc3c6740 1 -- 172.21.15.31:0/2366772888 >> 172.21.15.31:0/2366772888 conn(0x555f42697560 msgr2=0x555f426965c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:23.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.228+0000 7fabcc3c6740 1 -- 172.21.15.31:0/2366772888 shutdown_connections 2024-04-25T02:12:23.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.228+0000 7fabcc3c6740 1 -- 172.21.15.31:0/2366772888 wait complete. 2024-04-25T02:12:23.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.228+0000 7fabcc3c6740 1 -- start start 2024-04-25T02:12:23.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.228+0000 7fabcc3c6740 1 Processor -- start 2024-04-25T02:12:23.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.228+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 0x555f426ae010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269bb60 0x555f426a9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabcc3c6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 0x555f426a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fabc41b4d20 con 0x555f426a6230 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fabc41b4650 con 0x555f4269b240 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabcc3c6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fabc41b18b0 con 0x555f4269bb60 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabcc3c6740 10 client.0 _ll_register_callbacks cb 0x7fabbc010650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc27fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269bb60 0x555f426a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 0x555f426ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc27fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269bb60 0x555f426a9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:50608/0 (socket says 172.21.15.31:50608) 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc27fc640 1 -- 172.21.15.31:0/1124164059 learned_addr learned my addr 172.21.15.31:0/1124164059 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 0x555f426ae010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:60006/0 (socket says 172.21.15.31:60006) 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc37fe640 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 0x555f426a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc27fc640 1 -- 172.21.15.31:0/1124164059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 msgr2=0x555f426ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc27fc640 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 0x555f426ae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc27fc640 1 -- 172.21.15.31:0/1124164059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 msgr2=0x555f426a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc27fc640 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 0x555f426a98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc27fc640 1 -- 172.21.15.31:0/1124164059 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f426a9f00 con 0x555f4269bb60 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.229+0000 7fabc37fe640 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 0x555f426a98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.230+0000 7fabc27fc640 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269bb60 0x555f426a9380 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fabc4109d70 tx=0x7fabb8012d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a342ac4e4f8e13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.230+0000 7faba6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.230+0000 7faba6ff5640 1 -- 172.21.15.31:0/1124164059 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb8023060 con 0x555f4269bb60 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.230+0000 7faba6ff5640 1 -- 172.21.15.31:0/1124164059 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7fabb8004230 con 0x555f4269bb60 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.230+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1124164059 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabb40c02c0 con 0x555f4269bb60 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.230+0000 7faba6ff5640 1 -- 172.21.15.31:0/1124164059 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fabb8004430 con 0x555f4269bb60 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.230+0000 7fabcc3c6740 10 client.26923 fetch_fsmap learned FSMap version 49 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.230+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1124164059 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x555f4271cd50 con 0x555f4269bb60 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.230+0000 7faba6ff5640 1 -- 172.21.15.31:0/1124164059 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabb40c02c0 con 0x555f4269bb60 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.230+0000 7faba6ff5640 1 -- 172.21.15.31:0/1124164059 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fabb8003bc0 con 0x555f4269bb60 2024-04-25T02:12:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.231+0000 7fabcc3c6740 10 client.26923 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:12:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.231+0000 7fabcc3c6740 10 client.26923 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:12:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.231+0000 7fabcc3c6740 20 client.26923 populate_metadata read hostname 'smithi031' 2024-04-25T02:12:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.231+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555f4271d530 0x555f4271f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.231+0000 7fabcc3c6740 4 client.26923 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:12:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.231+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1124164059 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555f42607540 con 0x555f4271d530 2024-04-25T02:12:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.231+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555f4271d530 0x555f4271f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.231+0000 7faba6ff5640 10 client.26923 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.231+0000 7fabc2ffd640 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555f4271d530 0x555f4271f950 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26725b0ba1a5de8f server_cookie=a4517f2a33a3f5e1 in_seq=0 out_seq=0 2024-04-25T02:12:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7faba6ff5640 1 -- 172.21.15.31:0/1124164059 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555f42607540 con 0x555f4271d530 2024-04-25T02:12:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7faba6ff5640 10 client.26923 handle_command_reply: tid=0 2024-04-25T02:12:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 1 client.26923 shutdown 2024-04-25T02:12:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1124164059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269bb60 msgr2=0x555f426a9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269bb60 0x555f426a9380 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fabc4109d70 tx=0x7fabb8012d70 comp rx=0 tx=0).stop 2024-04-25T02:12:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1124164059 shutdown_connections 2024-04-25T02:12:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x555f4271d530 0x555f4271f950 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555f426a6230 0x555f426a98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7faba6ff5640 0 client.26923 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f4269bb60 0x555f426a9380 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 1 --2- 172.21.15.31:0/1124164059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555f4269b240 0x555f426ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1124164059 >> 172.21.15.31:0/1124164059 conn(0x555f42697560 msgr2=0x555f426965c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1124164059 shutdown_connections 2024-04-25T02:12:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 1 -- 172.21.15.31:0/1124164059 wait complete. 2024-04-25T02:12:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.283+0000 7fabcc3c6740 20 client.26923 trim_cache size 0 max 16384 2024-04-25T02:12:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:12:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:12:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:12:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:12:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 -- 172.21.15.31:0/3140953404 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabac0a2da0 msgr2=0x7fabac0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabac0a2da0 0x7fabac0a5260 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x555f426986f0 tx=0x7fabbc009a90 comp rx=0 tx=0).stop 2024-04-25T02:12:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 -- 172.21.15.31:0/3140953404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabc4069940 msgr2=0x7fabc4108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabc4069940 0x7fabc4108600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fabb4006010 tx=0x7fabb4003c10 comp rx=0 tx=0).stop 2024-04-25T02:12:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 -- 172.21.15.31:0/3140953404 shutdown_connections 2024-04-25T02:12:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fabac0a2da0 0x7fabac0a5260 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabc4107830 0x7fabc4109080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc406a2e0 0x7fabc4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 --2- 172.21.15.31:0/3140953404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabc4069940 0x7fabc4108600 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 -- 172.21.15.31:0/3140953404 >> 172.21.15.31:0/3140953404 conn(0x7fabc4077520 msgr2=0x7fabc410cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 -- 172.21.15.31:0/3140953404 shutdown_connections 2024-04-25T02:12:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:23.284+0000 7fabc95dc640 1 -- 172.21.15.31:0/3140953404 wait complete. 2024-04-25T02:12:24.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 44567 8.99 MB/sec execute 148 sec latency 82.173 ms 2024-04-25T02:12:24.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:24 smithi040 ceph-mon[32418]: pgmap v563: 97 pgs: 97 active+clean; 821 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 311 op/s 2024-04-25T02:12:24.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:24 smithi134 ceph-mon[33098]: pgmap v563: 97 pgs: 97 active+clean; 821 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 311 op/s 2024-04-25T02:12:24.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:24 smithi031 ceph-mon[29204]: pgmap v563: 97 pgs: 97 active+clean; 821 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 311 op/s 2024-04-25T02:12:25.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 44720 8.99 MB/sec execute 149 sec latency 78.698 ms 2024-04-25T02:12:26.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 44897 9.02 MB/sec execute 150 sec latency 63.948 ms 2024-04-25T02:12:26.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:26 smithi040 ceph-mon[32418]: pgmap v564: 97 pgs: 97 active+clean; 838 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 8.0 MiB/s wr, 412 op/s 2024-04-25T02:12:26.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:26 smithi134 ceph-mon[33098]: pgmap v564: 97 pgs: 97 active+clean; 838 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 8.0 MiB/s wr, 412 op/s 2024-04-25T02:12:26.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:26 smithi031 ceph-mon[29204]: pgmap v564: 97 pgs: 97 active+clean; 838 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 8.0 MiB/s wr, 412 op/s 2024-04-25T02:12:27.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 45055 9.01 MB/sec execute 151 sec latency 96.482 ms 2024-04-25T02:12:28.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 45215 9.01 MB/sec execute 152 sec latency 61.834 ms 2024-04-25T02:12:28.416 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:28 smithi040 ceph-mon[32418]: pgmap v565: 97 pgs: 97 active+clean; 838 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:12:28.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:28 smithi134 ceph-mon[33098]: pgmap v565: 97 pgs: 97 active+clean; 838 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:12:28.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:28 smithi031 ceph-mon[29204]: pgmap v565: 97 pgs: 97 active+clean; 838 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:12:29.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 45343 8.99 MB/sec execute 153 sec latency 122.540 ms 2024-04-25T02:12:30.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 45493 8.97 MB/sec execute 154 sec latency 87.318 ms 2024-04-25T02:12:30.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:30 smithi040 ceph-mon[32418]: pgmap v566: 97 pgs: 97 active+clean; 838 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:12:30.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:30 smithi134 ceph-mon[33098]: pgmap v566: 97 pgs: 97 active+clean; 838 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:12:30.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:30 smithi031 ceph-mon[29204]: pgmap v566: 97 pgs: 97 active+clean; 838 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:12:31.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 45676 8.96 MB/sec execute 155 sec latency 122.652 ms 2024-04-25T02:12:32.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 45844 8.94 MB/sec execute 156 sec latency 61.417 ms 2024-04-25T02:12:32.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:32 smithi040 ceph-mon[32418]: pgmap v567: 97 pgs: 97 active+clean; 842 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.1 MiB/s wr, 345 op/s 2024-04-25T02:12:32.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:32 smithi134 ceph-mon[33098]: pgmap v567: 97 pgs: 97 active+clean; 842 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.1 MiB/s wr, 345 op/s 2024-04-25T02:12:32.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:32 smithi031 ceph-mon[29204]: pgmap v567: 97 pgs: 97 active+clean; 842 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.1 MiB/s wr, 345 op/s 2024-04-25T02:12:33.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 46000 8.93 MB/sec execute 157 sec latency 74.041 ms 2024-04-25T02:12:34.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 46170 8.94 MB/sec execute 158 sec latency 56.330 ms 2024-04-25T02:12:34.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:34 smithi040 ceph-mon[32418]: pgmap v568: 97 pgs: 97 active+clean; 842 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:12:34.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:34 smithi134 ceph-mon[33098]: pgmap v568: 97 pgs: 97 active+clean; 842 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:12:34.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:34 smithi031 ceph-mon[29204]: pgmap v568: 97 pgs: 97 active+clean; 842 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:12:35.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 46332 8.95 MB/sec execute 159 sec latency 112.941 ms 2024-04-25T02:12:36.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 46513 8.96 MB/sec execute 160 sec latency 58.905 ms 2024-04-25T02:12:36.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:36 smithi040 ceph-mon[32418]: pgmap v569: 97 pgs: 97 active+clean; 845 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 346 op/s 2024-04-25T02:12:36.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.437+0000 7f2b10838640 1 -- 172.21.15.31:0/712990326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c101760 msgr2=0x7f2b0c101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:36.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.437+0000 7f2b10838640 1 --2- 172.21.15.31:0/712990326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c101760 0x7f2b0c101be0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009930 tx=0x7f2b0002a420 comp rx=0 tx=0).stop 2024-04-25T02:12:36.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.437+0000 7f2b10838640 1 -- 172.21.15.31:0/712990326 shutdown_connections 2024-04-25T02:12:36.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.437+0000 7f2b10838640 1 --2- 172.21.15.31:0/712990326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b0c102120 0x7f2b0c10e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.437+0000 7f2b10838640 1 --2- 172.21.15.31:0/712990326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c101760 0x7f2b0c101be0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.437+0000 7f2b10838640 1 --2- 172.21.15.31:0/712990326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0c107770 0x7f2b0c107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.437+0000 7f2b10838640 1 -- 172.21.15.31:0/712990326 >> 172.21.15.31:0/712990326 conn(0x7f2b0c0fd1f0 msgr2=0x7f2b0c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.437+0000 7f2b10838640 1 -- 172.21.15.31:0/712990326 shutdown_connections 2024-04-25T02:12:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.437+0000 7f2b10838640 1 -- 172.21.15.31:0/712990326 wait complete. 2024-04-25T02:12:36.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:36 smithi134 ceph-mon[33098]: pgmap v569: 97 pgs: 97 active+clean; 845 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 346 op/s 2024-04-25T02:12:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.438+0000 7f2b10838640 1 Processor -- start 2024-04-25T02:12:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.438+0000 7f2b10838640 1 -- start start 2024-04-25T02:12:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b10838640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b0c101760 0x7f2b0c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b10838640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0c102120 0x7f2b0c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b10838640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c107770 0x7f2b0c19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b10838640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c111a40 con 0x7f2b0c107770 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b10838640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1118c0 con 0x7f2b0c102120 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b10838640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c111bc0 con 0x7f2b0c101760 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b0ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c107770 0x7f2b0c19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b0a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b0c101760 0x7f2b0c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b0a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b0c101760 0x7f2b0c197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59644/0 (socket says 172.21.15.31:59644) 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b0ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c107770 0x7f2b0c19c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51852/0 (socket says 172.21.15.31:51852) 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b0a59c640 1 -- 172.21.15.31:0/3575757735 learned_addr learned my addr 172.21.15.31:0/3575757735 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b09d9b640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0c102120 0x7f2b0c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b0ad9d640 1 -- 172.21.15.31:0/3575757735 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b0c101760 msgr2=0x7f2b0c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b0ad9d640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b0c101760 0x7f2b0c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b0ad9d640 1 -- 172.21.15.31:0/3575757735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0c102120 msgr2=0x7f2b0c198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b0ad9d640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0c102120 0x7f2b0c198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b0ad9d640 1 -- 172.21.15.31:0/3575757735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0c19cee0 con 0x7f2b0c107770 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.439+0000 7f2b0ad9d640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c107770 0x7f2b0c19c7b0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009fd0 tx=0x7f2b0400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.440+0000 7f2afb7fe640 1 -- 172.21.15.31:0/3575757735 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0400ce20 con 0x7f2b0c107770 2024-04-25T02:12:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.440+0000 7f2afb7fe640 1 -- 172.21.15.31:0/3575757735 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b04005020 con 0x7f2b0c107770 2024-04-25T02:12:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.440+0000 7f2afb7fe640 1 -- 172.21.15.31:0/3575757735 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04010690 con 0x7f2b0c107770 2024-04-25T02:12:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.440+0000 7f2b10838640 1 -- 172.21.15.31:0/3575757735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0c19d1d0 con 0x7f2b0c107770 2024-04-25T02:12:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.440+0000 7f2b10838640 1 -- 172.21.15.31:0/3575757735 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b0c1a4a60 con 0x7f2b0c107770 2024-04-25T02:12:36.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.444+0000 7f2b13627740 1 Processor -- start 2024-04-25T02:12:36.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.444+0000 7f2afb7fe640 1 -- 172.21.15.31:0/3575757735 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2b04010830 con 0x7f2b0c107770 2024-04-25T02:12:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.445+0000 7f2b13627740 1 -- start start 2024-04-25T02:12:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.445+0000 7f2b13627740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0240 0x55bedeec0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.445+0000 7f2b13627740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeecacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.445+0000 7f2b13627740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 0x55bedeecd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.445+0000 7f2b13627740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bedee24cb0 con 0x55bedeecb230 2024-04-25T02:12:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.445+0000 7f2b13627740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bededcd5b0 con 0x55bedeec0b60 2024-04-25T02:12:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.445+0000 7f2b13627740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bedee20c70 con 0x55bedeec0240 2024-04-25T02:12:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.445+0000 7f2b0a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeecacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.445+0000 7f2b09d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 0x55bedeecd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.445+0000 7f2b0a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeecacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38088/0 (socket says 172.21.15.31:38088) 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.445+0000 7f2b0a59c640 1 -- 172.21.15.31:0/2730769149 learned_addr learned my addr 172.21.15.31:0/2730769149 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2b0a59c640 1 -- 172.21.15.31:0/2730769149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0240 msgr2=0x55bedeec0620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2b0a59c640 1 --2- 172.21.15.31:0/2730769149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0240 0x55bedeec0620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2b0a59c640 1 -- 172.21.15.31:0/2730769149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 msgr2=0x55bedeecd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2b0a59c640 1 --2- 172.21.15.31:0/2730769149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 0x55bedeecd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2b0a59c640 1 -- 172.21.15.31:0/2730769149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bedeb39210 con 0x55bedeec0b60 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2b09d9b640 1 --2- 172.21.15.31:0/2730769149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 0x55bedeecd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2b0a59c640 1 --2- 172.21.15.31:0/2730769149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeecacf0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2afc009d40 tx=0x7f2afc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31697b9fc2c1523c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2afb7fe640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2af40a2ea0 0x7f2af40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2af8ff9640 1 -- 172.21.15.31:0/2730769149 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc004220 con 0x55bedeec0b60 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2af8ff9640 1 -- 172.21.15.31:0/2730769149 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2afc0043c0 con 0x55bedeec0b60 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2af8ff9640 1 -- 172.21.15.31:0/2730769149 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc004e50 con 0x55bedeec0b60 2024-04-25T02:12:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2afb7fe640 1 -- 172.21.15.31:0/3575757735 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f2b040cfae0 con 0x7f2b0c107770 2024-04-25T02:12:36.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.446+0000 7f2b09d9b640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2af40a2ea0 0x7f2af40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.447+0000 7f2b13627740 1 -- 172.21.15.31:0/2730769149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 msgr2=0x55bedeecacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:36.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.447+0000 7f2b13627740 1 --2- 172.21.15.31:0/2730769149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeecacf0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2afc009d40 tx=0x7f2afc011810 comp rx=0 tx=0).stop 2024-04-25T02:12:36.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.448+0000 7f2b09d9b640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2af40a2ea0 0x7f2af40a5360 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2b00031440 tx=0x7f2b00005a20 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:36.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.448+0000 7f2b13627740 1 -- 172.21.15.31:0/2730769149 shutdown_connections 2024-04-25T02:12:36.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.448+0000 7f2b13627740 1 --2- 172.21.15.31:0/2730769149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 0x55bedeecd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.448+0000 7f2b13627740 1 --2- 172.21.15.31:0/2730769149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeecacf0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.448+0000 7f2b13627740 1 --2- 172.21.15.31:0/2730769149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0240 0x55bedeec0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.448+0000 7f2b13627740 1 -- 172.21.15.31:0/2730769149 >> 172.21.15.31:0/2730769149 conn(0x55bedeebc560 msgr2=0x55bedeebc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.448+0000 7f2b13627740 1 -- 172.21.15.31:0/2730769149 shutdown_connections 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.448+0000 7f2b13627740 1 -- 172.21.15.31:0/2730769149 wait complete. 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b13627740 1 -- start start 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b13627740 1 Processor -- start 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b13627740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeec0240 0x55bedeec8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b13627740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0b60 0x55bedeec8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b13627740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeecb230 0x55bedeec8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b13627740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bedeedf030 con 0x55bedeec0240 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b13627740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bedede0b50 con 0x55bedeecb230 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b13627740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bedeedf1b0 con 0x55bedeec0b60 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b13627740 10 client.0 _ll_register_callbacks cb 0x55bedee1bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:12:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b09d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0b60 0x55bedeec8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b0ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeecb230 0x55bedeec8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b0a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeec0240 0x55bedeec8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b09d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0b60 0x55bedeec8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59660/0 (socket says 172.21.15.31:59660) 2024-04-25T02:12:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.449+0000 7f2b09d9b640 1 -- 172.21.15.31:0/245561496 learned_addr learned my addr 172.21.15.31:0/245561496 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.450+0000 7f2b09d9b640 1 -- 172.21.15.31:0/245561496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeecb230 msgr2=0x55bedeec8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.450+0000 7f2b09d9b640 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeecb230 0x55bedeec8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.450+0000 7f2b09d9b640 1 -- 172.21.15.31:0/245561496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeec0240 msgr2=0x55bedeec8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.450+0000 7f2b09d9b640 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeec0240 0x55bedeec8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.450+0000 7f2b09d9b640 1 -- 172.21.15.31:0/245561496 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bedec05620 con 0x55bedeec0b60 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.450+0000 7f2b0ad9d640 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeecb230 0x55bedeec8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.450+0000 7f2b09d9b640 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0b60 0x55bedeec8750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009930 tx=0x7f2b00009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e71353a5af80bf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.450+0000 7f2af1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.450+0000 7f2af1ffb640 1 -- 172.21.15.31:0/245561496 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b00035040 con 0x55bedeec0b60 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2b13627740 1 -- 172.21.15.31:0/245561496 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2afc0043c0 con 0x55bedeec0b60 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2af1ffb640 1 -- 172.21.15.31:0/245561496 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f2b00004b60 con 0x55bedeec0b60 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2af1ffb640 1 -- 172.21.15.31:0/245561496 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f2b00004d60 con 0x55bedeec0b60 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2b13627740 10 client.26932 fetch_fsmap learned FSMap version 49 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2b13627740 1 -- 172.21.15.31:0/245561496 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55bedeeeb070 con 0x55bedeec0b60 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2af1ffb640 1 -- 172.21.15.31:0/245561496 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2afc0043c0 con 0x55bedeec0b60 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2af1ffb640 1 -- 172.21.15.31:0/245561496 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f2b00044030 con 0x55bedeec0b60 2024-04-25T02:12:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2b13627740 10 client.26932 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:12:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2b13627740 10 client.26932 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:12:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2b13627740 20 client.26932 populate_metadata read hostname 'smithi031' 2024-04-25T02:12:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2b13627740 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bedeeec900 0x55bedeeeeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2b13627740 4 client.26932 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:12:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.451+0000 7f2b13627740 1 -- 172.21.15.31:0/245561496 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bedee321f0 con 0x55bedeeec900 2024-04-25T02:12:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.452+0000 7f2b0a59c640 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bedeeec900 0x55bedeeeeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.452+0000 7f2af1ffb640 10 client.26932 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:36.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.452+0000 7f2b0a59c640 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bedeeec900 0x55bedeeeeda0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a47683862f9c8e9 server_cookie=f0929fc212263cfb in_seq=0 out_seq=0 2024-04-25T02:12:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.453+0000 7f2af1ffb640 1 -- 172.21.15.31:0/245561496 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bedee321f0 con 0x55bedeeec900 2024-04-25T02:12:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.453+0000 7f2af1ffb640 10 client.26932 handle_command_reply: tid=0 2024-04-25T02:12:36.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.453+0000 7f2b13627740 1 client.26932 shutdown 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.453+0000 7f2b13627740 1 -- 172.21.15.31:0/245561496 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0b60 msgr2=0x55bedeec8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.453+0000 7f2b13627740 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0b60 0x55bedeec8750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009930 tx=0x7f2b00009260 comp rx=0 tx=0).stop 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.453+0000 7f2b13627740 1 -- 172.21.15.31:0/245561496 shutdown_connections 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.453+0000 7f2b13627740 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bedeeec900 0x55bedeeeeda0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.453+0000 7f2b13627740 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeecb230 0x55bedeec8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.453+0000 7f2af1ffb640 0 client.26932 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.453+0000 7f2b13627740 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0b60 0x55bedeec8750 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.454+0000 7f2b13627740 1 --2- 172.21.15.31:0/245561496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeec0240 0x55bedeec8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.454+0000 7f2b13627740 1 -- 172.21.15.31:0/245561496 >> 172.21.15.31:0/245561496 conn(0x55bedeebc560 msgr2=0x55bedeec2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.454+0000 7f2b13627740 1 -- 172.21.15.31:0/245561496 shutdown_connections 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.454+0000 7f2b13627740 1 -- 172.21.15.31:0/245561496 wait complete. 2024-04-25T02:12:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.454+0000 7f2b13627740 20 client.26932 trim_cache size 0 max 16384 2024-04-25T02:12:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.475+0000 7f2b13627740 1 Processor -- start 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.475+0000 7f2b13627740 1 -- start start 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b13627740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeec0240 0x55bedeeca940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b13627740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeee7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b13627740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeecb230 0x55bedeee47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b13627740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c0737c0 con 0x55bedeec0240 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b13627740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1a65c0 con 0x55bedeec0b60 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b13627740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1a5150 con 0x55bedeecb230 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b09d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeee7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b0ad9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeecb230 0x55bedeee47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b09d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeee7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38114/0 (socket says 172.21.15.31:38114) 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b0a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeec0240 0x55bedeeca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b0ad9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeecb230 0x55bedeee47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59668/0 (socket says 172.21.15.31:59668) 2024-04-25T02:12:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b09d9b640 1 -- 172.21.15.31:0/3705053234 learned_addr learned my addr 172.21.15.31:0/3705053234 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b0a59c640 1 -- 172.21.15.31:0/3705053234 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeecb230 msgr2=0x55bedeee47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b0a59c640 1 --2- 172.21.15.31:0/3705053234 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeecb230 0x55bedeee47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b0a59c640 1 -- 172.21.15.31:0/3705053234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 msgr2=0x55bedeee7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b0a59c640 1 --2- 172.21.15.31:0/3705053234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeee7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.476+0000 7f2b0a59c640 1 -- 172.21.15.31:0/3705053234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bedeec0040 con 0x55bedeec0240 2024-04-25T02:12:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.477+0000 7f2b0a59c640 1 --2- 172.21.15.31:0/3705053234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeec0240 0x55bedeeca940 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f2afc009620 tx=0x7f2afc0049b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98330da61ef4dda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.477+0000 7f2af3fff640 1 -- 172.21.15.31:0/3705053234 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc033070 con 0x55bedeec0240 2024-04-25T02:12:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.477+0000 7f2af3fff640 1 -- 172.21.15.31:0/3705053234 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2afc01e040 con 0x55bedeec0240 2024-04-25T02:12:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.477+0000 7f2af3fff640 1 -- 172.21.15.31:0/3705053234 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc01d040 con 0x55bedeec0240 2024-04-25T02:12:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.478+0000 7f2b13627740 1 -- 172.21.15.31:0/3705053234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeec0240 msgr2=0x55bedeeca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.478+0000 7f2b13627740 1 --2- 172.21.15.31:0/3705053234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeec0240 0x55bedeeca940 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f2afc009620 tx=0x7f2afc0049b0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.478+0000 7f2b13627740 1 -- 172.21.15.31:0/3705053234 shutdown_connections 2024-04-25T02:12:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.478+0000 7f2b13627740 1 --2- 172.21.15.31:0/3705053234 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeecb230 0x55bedeee47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.478+0000 7f2b13627740 1 --2- 172.21.15.31:0/3705053234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeee7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.478+0000 7f2b13627740 1 --2- 172.21.15.31:0/3705053234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeec0240 0x55bedeeca940 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.478+0000 7f2b13627740 1 -- 172.21.15.31:0/3705053234 >> 172.21.15.31:0/3705053234 conn(0x55bedeebc560 msgr2=0x55bedeebc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.478+0000 7f2b13627740 1 -- 172.21.15.31:0/3705053234 shutdown_connections 2024-04-25T02:12:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.478+0000 7f2b13627740 1 -- 172.21.15.31:0/3705053234 wait complete. 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.478+0000 7f2b13627740 1 -- start start 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b13627740 1 Processor -- start 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b13627740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0240 0x55bedeec4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b13627740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeec49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b13627740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 0x55bedeec4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b13627740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1a5ef0 con 0x55bedeecb230 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b13627740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1a5820 con 0x55bedeec0b60 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b13627740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c0717c0 con 0x55bedeec0240 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b13627740 10 client.0 _ll_register_callbacks cb 0x7f2b00009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b0a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0240 0x55bedeec4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b0a59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0240 0x55bedeec4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59680/0 (socket says 172.21.15.31:59680) 2024-04-25T02:12:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b0a59c640 1 -- 172.21.15.31:0/2775064292 learned_addr learned my addr 172.21.15.31:0/2775064292 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b09d9b640 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeec49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b0ad9d640 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 0x55bedeec4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b0a59c640 1 -- 172.21.15.31:0/2775064292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 msgr2=0x55bedeec49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b0a59c640 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeec49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b0a59c640 1 -- 172.21.15.31:0/2775064292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 msgr2=0x55bedeec4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b0a59c640 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 0x55bedeec4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b0a59c640 1 -- 172.21.15.31:0/2775064292 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bedeec5540 con 0x55bedeec0240 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.479+0000 7f2b0ad9d640 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 0x55bedeec4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.480+0000 7f2b0a59c640 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0240 0x55bedeec4480 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2afc009d40 tx=0x7f2afc026680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2978fee067673c3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.480+0000 7f2af1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.480+0000 7f2af1ffb640 1 -- 172.21.15.31:0/2775064292 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc033030 con 0x55bedeec0240 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.480+0000 7f2af1ffb640 1 -- 172.21.15.31:0/2775064292 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f2afc026ab0 con 0x55bedeec0240 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.480+0000 7f2b13627740 1 -- 172.21.15.31:0/2775064292 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2afc01e040 con 0x55bedeec0240 2024-04-25T02:12:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.480+0000 7f2af1ffb640 1 -- 172.21.15.31:0/2775064292 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f2afc01d040 con 0x55bedeec0240 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.480+0000 7f2b13627740 10 client.26938 fetch_fsmap learned FSMap version 49 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.480+0000 7f2b13627740 1 -- 172.21.15.31:0/2775064292 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55bedef57330 con 0x55bedeec0240 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.481+0000 7f2af1ffb640 1 -- 172.21.15.31:0/2775064292 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2afc01e040 con 0x55bedeec0240 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.481+0000 7f2af1ffb640 1 -- 172.21.15.31:0/2775064292 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f2afc040020 con 0x55bedeec0240 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.481+0000 7f2b13627740 10 client.26938 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.481+0000 7f2b13627740 10 client.26938 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.481+0000 7f2b13627740 20 client.26938 populate_metadata read hostname 'smithi031' 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.481+0000 7f2b13627740 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bedeeec900 0x55bedef59900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.481+0000 7f2b13627740 4 client.26938 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.481+0000 7f2b13627740 1 -- 172.21.15.31:0/2775064292 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bedee2c540 con 0x55bedeeec900 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.481+0000 7f2b09d9b640 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bedeeec900 0x55bedef59900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.482+0000 7f2b09d9b640 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bedeeec900 0x55bedef59900 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d09675adc9b974bf server_cookie=ad07286aadc2ecf5 in_seq=0 out_seq=0 2024-04-25T02:12:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.482+0000 7f2af1ffb640 10 client.26938 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:36.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:36 smithi031 ceph-mon[29204]: pgmap v569: 97 pgs: 97 active+clean; 845 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 346 op/s 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.537+0000 7f2af1ffb640 1 -- 172.21.15.31:0/2775064292 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bedee2c540 con 0x55bedeeec900 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.537+0000 7f2af1ffb640 10 client.26938 handle_command_reply: tid=0 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.537+0000 7f2b13627740 1 client.26938 shutdown 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2b13627740 1 -- 172.21.15.31:0/2775064292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0240 msgr2=0x55bedeec4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2b13627740 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0240 0x55bedeec4480 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2afc009d40 tx=0x7f2afc026680 comp rx=0 tx=0).stop 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2b13627740 1 -- 172.21.15.31:0/2775064292 shutdown_connections 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2b13627740 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55bedeeec900 0x55bedef59900 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2b13627740 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bedeecb230 0x55bedeec4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2b13627740 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bedeec0b60 0x55bedeec49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2b13627740 1 --2- 172.21.15.31:0/2775064292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bedeec0240 0x55bedeec4480 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2b13627740 1 -- 172.21.15.31:0/2775064292 >> 172.21.15.31:0/2775064292 conn(0x55bedeebc560 msgr2=0x55bedeec81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2af1ffb640 0 client.26938 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2b13627740 1 -- 172.21.15.31:0/2775064292 shutdown_connections 2024-04-25T02:12:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2b13627740 1 -- 172.21.15.31:0/2775064292 wait complete. 2024-04-25T02:12:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.538+0000 7f2b13627740 20 client.26938 trim_cache size 0 max 16384 2024-04-25T02:12:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:12:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:12:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:12:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:12:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 -- 172.21.15.31:0/3575757735 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2af40a2ea0 msgr2=0x7f2af40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2af40a2ea0 0x7f2af40a5360 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2b00031440 tx=0x7f2b00005a20 comp rx=0 tx=0).stop 2024-04-25T02:12:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 -- 172.21.15.31:0/3575757735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c107770 msgr2=0x7f2b0c19c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c107770 0x7f2b0c19c7b0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009fd0 tx=0x7f2b0400ef10 comp rx=0 tx=0).stop 2024-04-25T02:12:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 -- 172.21.15.31:0/3575757735 shutdown_connections 2024-04-25T02:12:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2af40a2ea0 0x7f2af40a5360 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b0c107770 0x7f2b0c19c7b0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0c102120 0x7f2b0c198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 --2- 172.21.15.31:0/3575757735 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b0c101760 0x7f2b0c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 -- 172.21.15.31:0/3575757735 >> 172.21.15.31:0/3575757735 conn(0x7f2b0c0fd1f0 msgr2=0x7f2b0c0fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 -- 172.21.15.31:0/3575757735 shutdown_connections 2024-04-25T02:12:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:36.539+0000 7f2b10838640 1 -- 172.21.15.31:0/3575757735 wait complete. 2024-04-25T02:12:37.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 46694 8.93 MB/sec execute 161 sec latency 100.683 ms 2024-04-25T02:12:38.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 46851 8.92 MB/sec execute 162 sec latency 54.455 ms 2024-04-25T02:12:38.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:38 smithi031 ceph-mon[29204]: pgmap v570: 97 pgs: 97 active+clean; 845 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 231 op/s 2024-04-25T02:12:38.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:38 smithi040 ceph-mon[32418]: pgmap v570: 97 pgs: 97 active+clean; 845 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 231 op/s 2024-04-25T02:12:38.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:38 smithi134 ceph-mon[33098]: pgmap v570: 97 pgs: 97 active+clean; 845 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 231 op/s 2024-04-25T02:12:39.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 47008 8.89 MB/sec execute 163 sec latency 59.227 ms 2024-04-25T02:12:40.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 47208 8.94 MB/sec execute 164 sec latency 61.003 ms 2024-04-25T02:12:40.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:40 smithi040 ceph-mon[32418]: pgmap v571: 97 pgs: 97 active+clean; 845 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 231 op/s 2024-04-25T02:12:40.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:40 smithi134 ceph-mon[33098]: pgmap v571: 97 pgs: 97 active+clean; 845 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 231 op/s 2024-04-25T02:12:40.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:40 smithi031 ceph-mon[29204]: pgmap v571: 97 pgs: 97 active+clean; 845 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 231 op/s 2024-04-25T02:12:41.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 47372 8.95 MB/sec execute 165 sec latency 51.176 ms 2024-04-25T02:12:42.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 47551 8.97 MB/sec execute 166 sec latency 54.010 ms 2024-04-25T02:12:42.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:42 smithi040 ceph-mon[32418]: pgmap v572: 97 pgs: 97 active+clean; 853 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 354 op/s 2024-04-25T02:12:42.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:42 smithi134 ceph-mon[33098]: pgmap v572: 97 pgs: 97 active+clean; 853 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 354 op/s 2024-04-25T02:12:42.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:42 smithi031 ceph-mon[29204]: pgmap v572: 97 pgs: 97 active+clean; 853 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 354 op/s 2024-04-25T02:12:43.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 47755 9.02 MB/sec execute 167 sec latency 54.819 ms 2024-04-25T02:12:44.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 47953 9.02 MB/sec execute 168 sec latency 74.110 ms 2024-04-25T02:12:44.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:44 smithi040 ceph-mon[32418]: pgmap v573: 97 pgs: 97 active+clean; 853 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.2 MiB/s wr, 240 op/s 2024-04-25T02:12:44.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:44 smithi134 ceph-mon[33098]: pgmap v573: 97 pgs: 97 active+clean; 853 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.2 MiB/s wr, 240 op/s 2024-04-25T02:12:44.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:44 smithi031 ceph-mon[29204]: pgmap v573: 97 pgs: 97 active+clean; 853 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.2 MiB/s wr, 240 op/s 2024-04-25T02:12:45.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 48087 8.99 MB/sec execute 169 sec latency 70.698 ms 2024-04-25T02:12:46.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 48234 9.00 MB/sec execute 170 sec latency 62.173 ms 2024-04-25T02:12:46.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:46 smithi040 ceph-mon[32418]: pgmap v574: 97 pgs: 97 active+clean; 873 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.0 MiB/s wr, 346 op/s 2024-04-25T02:12:46.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:46 smithi134 ceph-mon[33098]: pgmap v574: 97 pgs: 97 active+clean; 873 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.0 MiB/s wr, 346 op/s 2024-04-25T02:12:46.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:46 smithi031 ceph-mon[29204]: pgmap v574: 97 pgs: 97 active+clean; 873 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.0 MiB/s wr, 346 op/s 2024-04-25T02:12:47.098 INFO:tasks.workunit.client.0.smithi031.stdout: 10 48408 9.01 MB/sec execute 171 sec latency 57.166 ms 2024-04-25T02:12:48.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 48583 9.04 MB/sec execute 172 sec latency 57.531 ms 2024-04-25T02:12:48.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:48 smithi040 ceph-mon[32418]: pgmap v575: 97 pgs: 97 active+clean; 873 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:12:48.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:48 smithi134 ceph-mon[33098]: pgmap v575: 97 pgs: 97 active+clean; 873 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:12:48.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:48 smithi031 ceph-mon[29204]: pgmap v575: 97 pgs: 97 active+clean; 873 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:12:49.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 48726 9.01 MB/sec execute 173 sec latency 58.116 ms 2024-04-25T02:12:49.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.671+0000 7f58e1058640 1 -- 172.21.15.31:0/709676077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58dc1029d0 msgr2=0x7f58dc102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:49.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.671+0000 7f58e1058640 1 --2- 172.21.15.31:0/709676077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58dc1029d0 0x7f58dc102db0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009900 tx=0x7f58cc02a480 comp rx=0 tx=0).stop 2024-04-25T02:12:49.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.671+0000 7f58e1058640 1 -- 172.21.15.31:0/709676077 shutdown_connections 2024-04-25T02:12:49.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.671+0000 7f58e1058640 1 --2- 172.21.15.31:0/709676077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58dc10bda0 0x7f58dc10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.671+0000 7f58e1058640 1 --2- 172.21.15.31:0/709676077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58dc1032f0 0x7f58dc10b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.671+0000 7f58e1058640 1 --2- 172.21.15.31:0/709676077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58dc1029d0 0x7f58dc102db0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.671+0000 7f58e1058640 1 -- 172.21.15.31:0/709676077 >> 172.21.15.31:0/709676077 conn(0x7f58dc077760 msgr2=0x7f58dc077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:49.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.671+0000 7f58e1058640 1 -- 172.21.15.31:0/709676077 shutdown_connections 2024-04-25T02:12:49.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.671+0000 7f58e1058640 1 -- 172.21.15.31:0/709676077 wait complete. 2024-04-25T02:12:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58e1058640 1 Processor -- start 2024-04-25T02:12:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58e1058640 1 -- start start 2024-04-25T02:12:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58e1058640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58dc1029d0 0x7f58dc19b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58e1058640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58dc1032f0 0x7f58dc19be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58e1058640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58dc10bda0 0x7f58dc195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58e1058640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc101e80 con 0x7f58dc1029d0 2024-04-25T02:12:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58e1058640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc101d00 con 0x7f58dc10bda0 2024-04-25T02:12:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58e1058640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc102000 con 0x7f58dc1032f0 2024-04-25T02:12:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58dad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58dc1029d0 0x7f58dc19b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58da59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58dc1032f0 0x7f58dc19be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58db59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58dc10bda0 0x7f58dc195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58dad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58dc1029d0 0x7f58dc19b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48908/0 (socket says 172.21.15.31:48908) 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.672+0000 7f58dad9d640 1 -- 172.21.15.31:0/2705054137 learned_addr learned my addr 172.21.15.31:0/2705054137 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58dad9d640 1 -- 172.21.15.31:0/2705054137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58dc1032f0 msgr2=0x7f58dc19be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58dad9d640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58dc1032f0 0x7f58dc19be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58dad9d640 1 -- 172.21.15.31:0/2705054137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58dc10bda0 msgr2=0x7f58dc195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58dad9d640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58dc10bda0 0x7f58dc195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58dad9d640 1 -- 172.21.15.31:0/2705054137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58dc1962c0 con 0x7f58dc1029d0 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58db59e640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58dc10bda0 0x7f58dc195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58dad9d640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58dc1029d0 0x7f58dc19b940 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f58cc005940 tx=0x7f58cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58c3fff640 1 -- 172.21.15.31:0/2705054137 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc03f030 con 0x7f58dc1029d0 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58e1058640 1 -- 172.21.15.31:0/2705054137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58dc196550 con 0x7f58dc1029d0 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58c3fff640 1 -- 172.21.15.31:0/2705054137 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58cc003cb0 con 0x7f58dc1029d0 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58c3fff640 1 -- 172.21.15.31:0/2705054137 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc035070 con 0x7f58dc1029d0 2024-04-25T02:12:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.673+0000 7f58e1058640 1 -- 172.21.15.31:0/2705054137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58dc1a2800 con 0x7f58dc1029d0 2024-04-25T02:12:49.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.677+0000 7f58c3fff640 1 -- 172.21.15.31:0/2705054137 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f58cc042030 con 0x7f58dc1029d0 2024-04-25T02:12:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.677+0000 7f58e3d66740 1 Processor -- start 2024-04-25T02:12:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.677+0000 7f58e3d66740 1 -- start start 2024-04-25T02:12:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.677+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4581620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.677+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a458bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.677+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a458c230 0x5595a458e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.677+0000 7f58e3d66740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5595a44e5cb0 con 0x5595a4581b60 2024-04-25T02:12:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.677+0000 7f58e3d66740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5595a448e5b0 con 0x5595a4581240 2024-04-25T02:12:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.677+0000 7f58e3d66740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5595a44e1c70 con 0x5595a458c230 2024-04-25T02:12:49.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.677+0000 7f58da59c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a458c230 0x5595a458e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.677+0000 7f58db59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4581620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.678+0000 7f58db59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4581620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39296/0 (socket says 172.21.15.31:39296) 2024-04-25T02:12:49.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.678+0000 7f58db59e640 1 -- 172.21.15.31:0/637491106 learned_addr learned my addr 172.21.15.31:0/637491106 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.678+0000 7f58dad9d640 1 --2- 172.21.15.31:0/637491106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a458bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.678+0000 7f58da59c640 1 -- 172.21.15.31:0/637491106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 msgr2=0x5595a4581620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.678+0000 7f58da59c640 1 --2- 172.21.15.31:0/637491106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4581620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.678+0000 7f58da59c640 1 -- 172.21.15.31:0/637491106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 msgr2=0x5595a458bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.678+0000 7f58da59c640 1 --2- 172.21.15.31:0/637491106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a458bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.678+0000 7f58da59c640 1 -- 172.21.15.31:0/637491106 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595a41fa210 con 0x5595a458c230 2024-04-25T02:12:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.678+0000 7f58db59e640 1 --2- 172.21.15.31:0/637491106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4581620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:12:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.678+0000 7f58c3fff640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f58c40a7230 0x7f58c40a96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.678+0000 7f58da59c640 1 --2- 172.21.15.31:0/637491106 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a458c230 0x5595a458e620 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f58dc1000e0 tx=0x7f58d00107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c85733df08c37973 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.679+0000 7f58db59e640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f58c40a7230 0x7f58c40a96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.679+0000 7f58c3fff640 1 -- 172.21.15.31:0/2705054137 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f58cc0efad0 con 0x7f58dc1029d0 2024-04-25T02:12:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.679+0000 7f58c17fa640 1 -- 172.21.15.31:0/637491106 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d001c480 con 0x5595a458c230 2024-04-25T02:12:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.679+0000 7f58c17fa640 1 -- 172.21.15.31:0/637491106 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58d001cad0 con 0x5595a458c230 2024-04-25T02:12:49.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.679+0000 7f58c17fa640 1 -- 172.21.15.31:0/637491106 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0011b50 con 0x5595a458c230 2024-04-25T02:12:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.680+0000 7f58e3d66740 1 -- 172.21.15.31:0/637491106 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a458c230 msgr2=0x5595a458e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.680+0000 7f58db59e640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f58c40a7230 0x7f58c40a96f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009fd0 tx=0x7f58d4009500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.680+0000 7f58e3d66740 1 --2- 172.21.15.31:0/637491106 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a458c230 0x5595a458e620 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f58dc1000e0 tx=0x7f58d00107a0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.680+0000 7f58e3d66740 1 -- 172.21.15.31:0/637491106 shutdown_connections 2024-04-25T02:12:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.680+0000 7f58e3d66740 1 --2- 172.21.15.31:0/637491106 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a458c230 0x5595a458e620 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.680+0000 7f58e3d66740 1 --2- 172.21.15.31:0/637491106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a458bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.680+0000 7f58e3d66740 1 --2- 172.21.15.31:0/637491106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4581620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.680+0000 7f58e3d66740 1 -- 172.21.15.31:0/637491106 >> 172.21.15.31:0/637491106 conn(0x5595a457d560 msgr2=0x5595a457d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.680+0000 7f58e3d66740 1 -- 172.21.15.31:0/637491106 shutdown_connections 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.680+0000 7f58e3d66740 1 -- 172.21.15.31:0/637491106 wait complete. 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.681+0000 7f58e3d66740 1 -- start start 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.681+0000 7f58e3d66740 1 Processor -- start 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.681+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4589170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.681+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a45896b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.681+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a458c230 0x5595a4589bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.681+0000 7f58e3d66740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5595a458f020 con 0x5595a4581b60 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.681+0000 7f58e3d66740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5595a44a1b50 con 0x5595a4581240 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.681+0000 7f58e3d66740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5595a458f1a0 con 0x5595a458c230 2024-04-25T02:12:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58e3d66740 10 client.0 _ll_register_callbacks cb 0x5595a44dcca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58dad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4589170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58da59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a45896b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58dad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4589170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39302/0 (socket says 172.21.15.31:39302) 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58dad9d640 1 -- 172.21.15.31:0/2173180761 learned_addr learned my addr 172.21.15.31:0/2173180761 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58db59e640 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a458c230 0x5595a4589bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58dad9d640 1 -- 172.21.15.31:0/2173180761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a458c230 msgr2=0x5595a4589bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58dad9d640 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a458c230 0x5595a4589bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58dad9d640 1 -- 172.21.15.31:0/2173180761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 msgr2=0x5595a45896b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58dad9d640 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a45896b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58dad9d640 1 -- 172.21.15.31:0/2173180761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595a42c6620 con 0x5595a4581240 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58dad9d640 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4589170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f58cc006010 tx=0x7f58cc009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=261571237b188920 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.682+0000 7f58beff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58beff5640 1 -- 172.21.15.31:0/2173180761 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc049070 con 0x5595a4581240 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58beff5640 1 -- 172.21.15.31:0/2173180761 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f58cc0b8730 con 0x5595a4581240 2024-04-25T02:12:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58e3d66740 1 -- 172.21.15.31:0/2173180761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58d001cad0 con 0x5595a4581240 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58beff5640 1 -- 172.21.15.31:0/2173180761 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f58cc0bbe90 con 0x5595a4581240 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58e3d66740 10 client.27167 fetch_fsmap learned FSMap version 49 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58e3d66740 1 -- 172.21.15.31:0/2173180761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5595a459b1e0 con 0x5595a4581240 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58beff5640 1 -- 172.21.15.31:0/2173180761 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58d001cad0 con 0x5595a4581240 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58beff5640 1 -- 172.21.15.31:0/2173180761 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f58cc003800 con 0x5595a4581240 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58e3d66740 10 client.27167 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58e3d66740 10 client.27167 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58e3d66740 20 client.27167 populate_metadata read hostname 'smithi031' 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58e3d66740 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5595a459c6e0 0x5595a459ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58e3d66740 4 client.27167 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58e3d66740 1 -- 172.21.15.31:0/2173180761 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595a44f31f0 con 0x5595a459c6e0 2024-04-25T02:12:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.683+0000 7f58da59c640 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5595a459c6e0 0x5595a459ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.684+0000 7f58beff5640 10 client.27167 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.684+0000 7f58da59c640 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5595a459c6e0 0x5595a459ead0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=624d92b99c830333 server_cookie=53f6abde7f41e68a in_seq=0 out_seq=0 2024-04-25T02:12:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58beff5640 1 -- 172.21.15.31:0/2173180761 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595a44f31f0 con 0x5595a459c6e0 2024-04-25T02:12:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58beff5640 10 client.27167 handle_command_reply: tid=0 2024-04-25T02:12:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 1 client.27167 shutdown 2024-04-25T02:12:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 1 -- 172.21.15.31:0/2173180761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 msgr2=0x5595a4589170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4589170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f58cc006010 tx=0x7f58cc009620 comp rx=0 tx=0).stop 2024-04-25T02:12:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 1 -- 172.21.15.31:0/2173180761 shutdown_connections 2024-04-25T02:12:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5595a459c6e0 0x5595a459ead0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a458c230 0x5595a4589bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a45896b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 1 --2- 172.21.15.31:0/2173180761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4589170 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 1 -- 172.21.15.31:0/2173180761 >> 172.21.15.31:0/2173180761 conn(0x5595a457d560 msgr2=0x5595a4583260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58beff5640 0 client.27167 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 1 -- 172.21.15.31:0/2173180761 shutdown_connections 2024-04-25T02:12:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 1 -- 172.21.15.31:0/2173180761 wait complete. 2024-04-25T02:12:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.685+0000 7f58e3d66740 20 client.27167 trim_cache size 0 max 16384 2024-04-25T02:12:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.711+0000 7f58e3d66740 1 Processor -- start 2024-04-25T02:12:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.711+0000 7f58e3d66740 1 -- start start 2024-04-25T02:12:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a4581240 0x5595a4597ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a4588a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a458c230 0x5595a45a43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58e3d66740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1ab000 con 0x5595a4581b60 2024-04-25T02:12:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58e3d66740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1acb40 con 0x5595a458c230 2024-04-25T02:12:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58e3d66740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1ab6d0 con 0x5595a4581240 2024-04-25T02:12:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58da59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a4588a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58da59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a4588a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48932/0 (socket says 172.21.15.31:48932) 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58da59c640 1 -- 172.21.15.31:0/1175530998 learned_addr learned my addr 172.21.15.31:0/1175530998 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58dad9d640 1 --2- 172.21.15.31:0/1175530998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a4581240 0x5595a4597ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58db59e640 1 --2- 172.21.15.31:0/1175530998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a458c230 0x5595a45a43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58da59c640 1 -- 172.21.15.31:0/1175530998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a4581240 msgr2=0x5595a4597ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58da59c640 1 --2- 172.21.15.31:0/1175530998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a4581240 0x5595a4597ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58da59c640 1 -- 172.21.15.31:0/1175530998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a458c230 msgr2=0x5595a45a43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58da59c640 1 --2- 172.21.15.31:0/1175530998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a458c230 0x5595a45a43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58da59c640 1 -- 172.21.15.31:0/1175530998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595a458a0d0 con 0x5595a4581b60 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.712+0000 7f58dad9d640 1 --2- 172.21.15.31:0/1175530998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a4581240 0x5595a4597ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.713+0000 7f58da59c640 1 --2- 172.21.15.31:0/1175530998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a4588a50 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x5595a457e6f0 tx=0x7f58d00115d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17cf69c405421fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.713+0000 7f58c0ff9640 1 -- 172.21.15.31:0/1175530998 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0004020 con 0x5595a4581b60 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.713+0000 7f58c0ff9640 1 -- 172.21.15.31:0/1175530998 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58d00054f0 con 0x5595a4581b60 2024-04-25T02:12:49.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.713+0000 7f58c0ff9640 1 -- 172.21.15.31:0/1175530998 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0011a80 con 0x5595a4581b60 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.714+0000 7f58e3d66740 1 -- 172.21.15.31:0/1175530998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 msgr2=0x5595a4588a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.714+0000 7f58e3d66740 1 --2- 172.21.15.31:0/1175530998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a4588a50 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x5595a457e6f0 tx=0x7f58d00115d0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.714+0000 7f58e3d66740 1 -- 172.21.15.31:0/1175530998 shutdown_connections 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.714+0000 7f58e3d66740 1 --2- 172.21.15.31:0/1175530998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a458c230 0x5595a45a43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.714+0000 7f58e3d66740 1 --2- 172.21.15.31:0/1175530998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a4581b60 0x5595a4588a50 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.714+0000 7f58e3d66740 1 --2- 172.21.15.31:0/1175530998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a4581240 0x5595a4597ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.714+0000 7f58e3d66740 1 -- 172.21.15.31:0/1175530998 >> 172.21.15.31:0/1175530998 conn(0x5595a457d560 msgr2=0x5595a45899c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.714+0000 7f58e3d66740 1 -- 172.21.15.31:0/1175530998 shutdown_connections 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.714+0000 7f58e3d66740 1 -- 172.21.15.31:0/1175530998 wait complete. 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.714+0000 7f58e3d66740 1 -- start start 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.714+0000 7f58e3d66740 1 Processor -- start 2024-04-25T02:12:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4594960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a4581b60 0x5595a4585c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58e3d66740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a458c230 0x5595a4586350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58dad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4594960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58dad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4594960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39328/0 (socket says 172.21.15.31:39328) 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58dad9d640 1 -- 172.21.15.31:0/4258091386 learned_addr learned my addr 172.21.15.31:0/4258091386 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58db59e640 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a458c230 0x5595a4586350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58e3d66740 1 -- 172.21.15.31:0/4258091386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1ac470 con 0x5595a458c230 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58e3d66740 1 -- 172.21.15.31:0/4258091386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1abda0 con 0x5595a4581240 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58e3d66740 1 -- 172.21.15.31:0/4258091386 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc1a9000 con 0x5595a4581b60 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58da59c640 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a4581b60 0x5595a4585c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58e3d66740 10 client.0 _ll_register_callbacks cb 0x7f58cc0f3310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58dad9d640 1 -- 172.21.15.31:0/4258091386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a4581b60 msgr2=0x5595a4585c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58dad9d640 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a4581b60 0x5595a4585c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58dad9d640 1 -- 172.21.15.31:0/4258091386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a458c230 msgr2=0x5595a4586350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:12:49.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58dad9d640 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a458c230 0x5595a4586350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58dad9d640 1 -- 172.21.15.31:0/4258091386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595a4601c30 con 0x5595a4581240 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58dad9d640 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4594960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0bc050 tx=0x7f58cc005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9e692d199ca6086 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.715+0000 7f58beff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.716+0000 7f58beff5640 1 -- 172.21.15.31:0/4258091386 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc049070 con 0x5595a4581240 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.716+0000 7f58beff5640 1 -- 172.21.15.31:0/4258091386 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8484+0+0 (secure 0 0 0) 0x7f58cc0c47f0 con 0x5595a4581240 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.716+0000 7f58e3d66740 1 -- 172.21.15.31:0/4258091386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58d00054f0 con 0x5595a4581240 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.716+0000 7f58beff5640 1 -- 172.21.15.31:0/4258091386 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f58cc09f050 con 0x5595a4581240 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.716+0000 7f58e3d66740 10 client.27173 fetch_fsmap learned FSMap version 49 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.716+0000 7f58e3d66740 1 -- 172.21.15.31:0/4258091386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5595a4601c30 con 0x5595a4581240 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.716+0000 7f58beff5640 1 -- 172.21.15.31:0/4258091386 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58d00054f0 con 0x5595a4581240 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.717+0000 7f58beff5640 1 -- 172.21.15.31:0/4258091386 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f58cc0bb510 con 0x5595a4581240 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.717+0000 7f58e3d66740 10 client.27173 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.717+0000 7f58e3d66740 10 client.27173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.717+0000 7f58e3d66740 20 client.27173 populate_metadata read hostname 'smithi031' 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.717+0000 7f58e3d66740 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5595a4603160 0x5595a4605570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:12:49.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.717+0000 7f58e3d66740 4 client.27173 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:12:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.717+0000 7f58e3d66740 1 -- 172.21.15.31:0/4258091386 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5595a44ed540 con 0x5595a4603160 2024-04-25T02:12:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.717+0000 7f58da59c640 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5595a4603160 0x5595a4605570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:12:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.717+0000 7f58beff5640 10 client.27173 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:49.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.717+0000 7f58da59c640 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5595a4603160 0x5595a4605570 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d045318a9c5d97a8 server_cookie=f47f84396d89720d in_seq=0 out_seq=0 2024-04-25T02:12:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.787+0000 7f58beff5640 1 -- 172.21.15.31:0/4258091386 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5595a44ed540 con 0x5595a4603160 2024-04-25T02:12:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.787+0000 7f58beff5640 10 client.27173 handle_command_reply: tid=0 2024-04-25T02:12:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.787+0000 7f58e3d66740 1 client.27173 shutdown 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.787+0000 7f58e3d66740 1 -- 172.21.15.31:0/4258091386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 msgr2=0x5595a4594960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.787+0000 7f58e3d66740 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4594960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0bc050 tx=0x7f58cc005c90 comp rx=0 tx=0).stop 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.788+0000 7f58e3d66740 1 -- 172.21.15.31:0/4258091386 shutdown_connections 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.788+0000 7f58e3d66740 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5595a4603160 0x5595a4605570 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.788+0000 7f58e3d66740 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5595a458c230 0x5595a4586350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.788+0000 7f58e3d66740 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5595a4581b60 0x5595a4585c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.788+0000 7f58beff5640 0 client.27173 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.788+0000 7f58e3d66740 1 --2- 172.21.15.31:0/4258091386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5595a4581240 0x5595a4594960 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.788+0000 7f58e3d66740 1 -- 172.21.15.31:0/4258091386 >> 172.21.15.31:0/4258091386 conn(0x5595a457d560 msgr2=0x5595a457bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.788+0000 7f58e3d66740 1 -- 172.21.15.31:0/4258091386 shutdown_connections 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.788+0000 7f58e3d66740 1 -- 172.21.15.31:0/4258091386 wait complete. 2024-04-25T02:12:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.788+0000 7f58e3d66740 20 client.27173 trim_cache size 0 max 16384 2024-04-25T02:12:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:12:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:12:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:12:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:12:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.793+0000 7f58e1058640 1 -- 172.21.15.31:0/2705054137 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f58c40a7230 msgr2=0x7f58c40a96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.793+0000 7f58e1058640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f58c40a7230 0x7f58c40a96f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009fd0 tx=0x7f58d4009500 comp rx=0 tx=0).stop 2024-04-25T02:12:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.793+0000 7f58e1058640 1 -- 172.21.15.31:0/2705054137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58dc1029d0 msgr2=0x7f58dc19b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:12:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.793+0000 7f58e1058640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58dc1029d0 0x7f58dc19b940 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f58cc005940 tx=0x7f58cc0042f0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.794+0000 7f58e1058640 1 -- 172.21.15.31:0/2705054137 shutdown_connections 2024-04-25T02:12:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.794+0000 7f58e1058640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f58c40a7230 0x7f58c40a96f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.794+0000 7f58e1058640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58dc10bda0 0x7f58dc195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.794+0000 7f58e1058640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58dc1032f0 0x7f58dc19be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.794+0000 7f58e1058640 1 --2- 172.21.15.31:0/2705054137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58dc1029d0 0x7f58dc19b940 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:12:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.794+0000 7f58e1058640 1 -- 172.21.15.31:0/2705054137 >> 172.21.15.31:0/2705054137 conn(0x7f58dc077760 msgr2=0x7f58dc0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:12:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.794+0000 7f58e1058640 1 -- 172.21.15.31:0/2705054137 shutdown_connections 2024-04-25T02:12:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:12:49.794+0000 7f58e1058640 1 -- 172.21.15.31:0/2705054137 wait complete. 2024-04-25T02:12:50.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 48854 8.97 MB/sec execute 174 sec latency 74.917 ms 2024-04-25T02:12:50.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:50 smithi040 ceph-mon[32418]: pgmap v576: 97 pgs: 97 active+clean; 873 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:12:50.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:50 smithi134 ceph-mon[33098]: pgmap v576: 97 pgs: 97 active+clean; 873 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:12:50.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:50 smithi031 ceph-mon[29204]: pgmap v576: 97 pgs: 97 active+clean; 873 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:12:51.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 49017 8.96 MB/sec execute 175 sec latency 62.136 ms 2024-04-25T02:12:52.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 49209 8.96 MB/sec execute 176 sec latency 56.609 ms 2024-04-25T02:12:52.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:52 smithi040 ceph-mon[32418]: pgmap v577: 97 pgs: 97 active+clean; 882 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 345 op/s 2024-04-25T02:12:52.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:52 smithi134 ceph-mon[33098]: pgmap v577: 97 pgs: 97 active+clean; 882 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 345 op/s 2024-04-25T02:12:52.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:52 smithi031 ceph-mon[29204]: pgmap v577: 97 pgs: 97 active+clean; 882 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.4 MiB/s wr, 345 op/s 2024-04-25T02:12:53.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 49363 8.93 MB/sec execute 177 sec latency 61.382 ms 2024-04-25T02:12:54.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 49507 8.95 MB/sec execute 178 sec latency 199.591 ms 2024-04-25T02:12:54.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:54 smithi040 ceph-mon[32418]: pgmap v578: 97 pgs: 97 active+clean; 882 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 222 op/s 2024-04-25T02:12:54.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:54 smithi134 ceph-mon[33098]: pgmap v578: 97 pgs: 97 active+clean; 882 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 222 op/s 2024-04-25T02:12:54.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:54 smithi031 ceph-mon[29204]: pgmap v578: 97 pgs: 97 active+clean; 882 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 222 op/s 2024-04-25T02:12:55.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 49694 8.98 MB/sec execute 179 sec latency 53.791 ms 2024-04-25T02:12:56.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 49879 9.00 MB/sec execute 180 sec latency 58.094 ms 2024-04-25T02:12:56.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:56 smithi040 ceph-mon[32418]: pgmap v579: 97 pgs: 97 active+clean; 877 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.7 MiB/s wr, 358 op/s 2024-04-25T02:12:56.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:56 smithi134 ceph-mon[33098]: pgmap v579: 97 pgs: 97 active+clean; 877 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.7 MiB/s wr, 358 op/s 2024-04-25T02:12:56.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:56 smithi031 ceph-mon[29204]: pgmap v579: 97 pgs: 97 active+clean; 877 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.7 MiB/s wr, 358 op/s 2024-04-25T02:12:57.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 50055 8.98 MB/sec execute 181 sec latency 64.981 ms 2024-04-25T02:12:58.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 50207 8.94 MB/sec execute 182 sec latency 63.693 ms 2024-04-25T02:12:58.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:12:58 smithi040 ceph-mon[32418]: pgmap v580: 97 pgs: 97 active+clean; 877 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-04-25T02:12:58.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:12:58 smithi134 ceph-mon[33098]: pgmap v580: 97 pgs: 97 active+clean; 877 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-04-25T02:12:58.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:12:58 smithi031 ceph-mon[29204]: pgmap v580: 97 pgs: 97 active+clean; 877 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-04-25T02:12:59.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 50348 8.92 MB/sec execute 183 sec latency 229.998 ms 2024-04-25T02:13:00.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 50499 8.91 MB/sec execute 184 sec latency 155.630 ms 2024-04-25T02:13:00.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:00 smithi040 ceph-mon[32418]: pgmap v581: 97 pgs: 97 active+clean; 877 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-04-25T02:13:00.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:00 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:13:00.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:00 smithi134 ceph-mon[33098]: pgmap v581: 97 pgs: 97 active+clean; 877 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-04-25T02:13:00.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:00 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:13:00.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:00 smithi031 ceph-mon[29204]: pgmap v581: 97 pgs: 97 active+clean; 877 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 253 op/s 2024-04-25T02:13:00.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:00 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:13:01.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 50662 8.91 MB/sec execute 185 sec latency 100.423 ms 2024-04-25T02:13:02.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 50812 8.92 MB/sec execute 186 sec latency 57.571 ms 2024-04-25T02:13:02.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:02 smithi040 ceph-mon[32418]: pgmap v582: 97 pgs: 97 active+clean; 888 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 349 op/s 2024-04-25T02:13:02.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:02 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:13:02.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:02 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:13:02.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:02 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:13:02.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:02 smithi134 ceph-mon[33098]: pgmap v582: 97 pgs: 97 active+clean; 888 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 349 op/s 2024-04-25T02:13:02.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:02 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:13:02.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:02 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:13:02.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:02 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:13:02.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:02 smithi031 ceph-mon[29204]: pgmap v582: 97 pgs: 97 active+clean; 888 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 349 op/s 2024-04-25T02:13:02.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:02 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:13:02.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:02 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:13:02.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:02 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:13:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.914+0000 7f02a63fb640 1 -- 172.21.15.31:0/761096051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02a010bd10 msgr2=0x7f02a010e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.914+0000 7f02a63fb640 1 --2- 172.21.15.31:0/761096051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02a010bd10 0x7f02a010e1b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0298009ca0 tx=0x7f029802a9e0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.914+0000 7f02a63fb640 1 -- 172.21.15.31:0/761096051 shutdown_connections 2024-04-25T02:13:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.914+0000 7f02a63fb640 1 --2- 172.21.15.31:0/761096051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02a010bd10 0x7f02a010e1b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.914+0000 7f02a63fb640 1 --2- 172.21.15.31:0/761096051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02a01032f0 0x7f02a010b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.914+0000 7f02a63fb640 1 --2- 172.21.15.31:0/761096051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a01029d0 0x7f02a0102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.914+0000 7f02a63fb640 1 -- 172.21.15.31:0/761096051 >> 172.21.15.31:0/761096051 conn(0x7f02a00777c0 msgr2=0x7f02a0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.914+0000 7f02a63fb640 1 -- 172.21.15.31:0/761096051 shutdown_connections 2024-04-25T02:13:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.914+0000 7f02a63fb640 1 -- 172.21.15.31:0/761096051 wait complete. 2024-04-25T02:13:02.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.914+0000 7f02a63fb640 1 Processor -- start 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.914+0000 7f02a63fb640 1 -- start start 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f02a63fb640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02a01029d0 0x7f02a019a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f02a63fb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a01032f0 0x7f02a019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f02a63fb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02a010bd10 0x7f02a019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f02a63fb640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f02a0111a10 con 0x7f02a01032f0 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f02a63fb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f02a0111890 con 0x7f02a010bd10 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f02a63fb640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f02a0111b90 con 0x7f02a01029d0 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a01032f0 0x7f02a019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f029ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02a01029d0 0x7f02a019a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a01032f0 0x7f02a019a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33194/0 (socket says 172.21.15.31:33194) 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f029f7fe640 1 -- 172.21.15.31:0/2510213138 learned_addr learned my addr 172.21.15.31:0/2510213138 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f029ffff640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02a01029d0 0x7f02a019a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:51734/0 (socket says 172.21.15.31:51734) 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.915+0000 7f02a4998640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02a010bd10 0x7f02a019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f029f7fe640 1 -- 172.21.15.31:0/2510213138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02a01029d0 msgr2=0x7f02a019a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f029f7fe640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02a01029d0 0x7f02a019a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f029f7fe640 1 -- 172.21.15.31:0/2510213138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02a010bd10 msgr2=0x7f02a019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f029f7fe640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02a010bd10 0x7f02a019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f029f7fe640 1 -- 172.21.15.31:0/2510213138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a019f090 con 0x7f02a01032f0 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f029ffff640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02a01029d0 0x7f02a019a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f02a4998640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02a010bd10 0x7f02a019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f029f7fe640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a01032f0 0x7f02a019a5e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f029400e9f0 tx=0x7f029400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f029d7fa640 1 -- 172.21.15.31:0/2510213138 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029400cd80 con 0x7f02a01032f0 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f02a63fb640 1 -- 172.21.15.31:0/2510213138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a019f380 con 0x7f02a01032f0 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f029d7fa640 1 -- 172.21.15.31:0/2510213138 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02940054d0 con 0x7f02a01032f0 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f029d7fa640 1 -- 172.21.15.31:0/2510213138 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0294010490 con 0x7f02a01032f0 2024-04-25T02:13:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.916+0000 7f02a63fb640 1 -- 172.21.15.31:0/2510213138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a01a6c10 con 0x7f02a01032f0 2024-04-25T02:13:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.919+0000 7f029d7fa640 1 -- 172.21.15.31:0/2510213138 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0294005020 con 0x7f02a01032f0 2024-04-25T02:13:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f02a91e0740 1 Processor -- start 2024-04-25T02:13:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f02a91e0740 1 -- start start 2024-04-25T02:13:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86146240 0x563c86146620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 0x563c86150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86151230 0x563c86153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f02a91e0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563c860aacb0 con 0x563c86146240 2024-04-25T02:13:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f02a91e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563c860535b0 con 0x563c86151230 2024-04-25T02:13:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f02a91e0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c860a6c70 con 0x563c86146b60 2024-04-25T02:13:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f029d7fa640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02880a2da0 0x7f02880a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f029ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 0x563c86150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f029ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 0x563c86150cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:51742/0 (socket says 172.21.15.31:51742) 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.920+0000 7f029ffff640 1 -- 172.21.15.31:0/2850996972 learned_addr learned my addr 172.21.15.31:0/2850996972 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f02a4998640 1 --2- 172.21.15.31:0/2850996972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86146240 0x563c86146620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f029f7fe640 1 --2- 172.21.15.31:0/2850996972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86151230 0x563c86153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f029ffff640 1 -- 172.21.15.31:0/2850996972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86151230 msgr2=0x563c86153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f029ffff640 1 --2- 172.21.15.31:0/2850996972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86151230 0x563c86153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f029ffff640 1 -- 172.21.15.31:0/2850996972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86146240 msgr2=0x563c86146620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f029ffff640 1 --2- 172.21.15.31:0/2850996972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86146240 0x563c86146620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f029ffff640 1 -- 172.21.15.31:0/2850996972 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c85dbf210 con 0x563c86146b60 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f02a4998640 1 --2- 172.21.15.31:0/2850996972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86146240 0x563c86146620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f02a4998640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02880a2da0 0x7f02880a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f029f7fe640 1 --2- 172.21.15.31:0/2850996972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86151230 0x563c86153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:13:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f029d7fa640 1 -- 172.21.15.31:0/2510213138 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8528+0+0 (secure 0 0 0) 0x7f02940cf760 con 0x7f02a01032f0 2024-04-25T02:13:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f029ffff640 1 --2- 172.21.15.31:0/2850996972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 0x563c86150cf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f02900115f0 tx=0x7f0290011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3342f047124e8167 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.921+0000 7f02867fc640 1 -- 172.21.15.31:0/2850996972 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029001e070 con 0x563c86146b60 2024-04-25T02:13:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.922+0000 7f02867fc640 1 -- 172.21.15.31:0/2850996972 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f029000faf0 con 0x563c86146b60 2024-04-25T02:13:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.922+0000 7f02a4998640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02880a2da0 0x7f02880a5260 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0298006010 tx=0x7f029803b040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.922+0000 7f02867fc640 1 -- 172.21.15.31:0/2850996972 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0290019bc0 con 0x563c86146b60 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.922+0000 7f02a91e0740 1 -- 172.21.15.31:0/2850996972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 msgr2=0x563c86150cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.922+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2850996972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 0x563c86150cf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f02900115f0 tx=0x7f0290011ab0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.923+0000 7f02a91e0740 1 -- 172.21.15.31:0/2850996972 shutdown_connections 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.923+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2850996972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86151230 0x563c86153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.923+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2850996972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 0x563c86150cf0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.923+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2850996972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86146240 0x563c86146620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.923+0000 7f02a91e0740 1 -- 172.21.15.31:0/2850996972 >> 172.21.15.31:0/2850996972 conn(0x563c86142560 msgr2=0x563c86142940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.923+0000 7f02a91e0740 1 -- 172.21.15.31:0/2850996972 shutdown_connections 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.923+0000 7f02a91e0740 1 -- 172.21.15.31:0/2850996972 wait complete. 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.923+0000 7f02a91e0740 1 -- start start 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.923+0000 7f02a91e0740 1 Processor -- start 2024-04-25T02:13:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146240 0x563c8614e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c8614ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f02a91e0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563c86154020 con 0x563c86151230 2024-04-25T02:13:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f02a91e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563c86066b50 con 0x563c86146b60 2024-04-25T02:13:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f02a91e0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c861541a0 con 0x563c86146240 2024-04-25T02:13:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f02a91e0740 10 client.0 _ll_register_callbacks cb 0x563c860a1ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:13:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f02a4998640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c8614ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f029ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146240 0x563c8614e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41548/0 (socket says 172.21.15.31:41548) 2024-04-25T02:13:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f029f7fe640 1 -- 172.21.15.31:0/2204713262 learned_addr learned my addr 172.21.15.31:0/2204713262 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f029f7fe640 1 -- 172.21.15.31:0/2204713262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146240 msgr2=0x563c8614e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f029f7fe640 1 --2- 172.21.15.31:0/2204713262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146240 0x563c8614e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f029f7fe640 1 -- 172.21.15.31:0/2204713262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 msgr2=0x563c8614ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f029f7fe640 1 --2- 172.21.15.31:0/2204713262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c8614ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.924+0000 7f029f7fe640 1 -- 172.21.15.31:0/2204713262 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c85e8b620 con 0x563c86146b60 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f029f7fe640 1 --2- 172.21.15.31:0/2204713262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614e6e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f02940cced0 tx=0x7f029406bb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=784af4b1210a4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f027f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f027f7fe640 1 -- 172.21.15.31:0/2204713262 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029409c340 con 0x563c86146b60 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f02a91e0740 1 -- 172.21.15.31:0/2204713262 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f029000faf0 con 0x563c86146b60 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f027f7fe640 1 -- 172.21.15.31:0/2204713262 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8528+0+0 (secure 0 0 0) 0x7f029407b170 con 0x563c86146b60 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f027f7fe640 1 -- 172.21.15.31:0/2204713262 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f029409bc40 con 0x563c86146b60 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f02a91e0740 10 client.27185 fetch_fsmap learned FSMap version 49 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f02a91e0740 1 -- 172.21.15.31:0/2204713262 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563c86160240 con 0x563c86146b60 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f027f7fe640 1 -- 172.21.15.31:0/2204713262 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f029000faf0 con 0x563c86146b60 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f027f7fe640 1 -- 172.21.15.31:0/2204713262 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f029406bbd0 con 0x563c86146b60 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f02a91e0740 10 client.27185 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f02a91e0740 10 client.27185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f02a91e0740 20 client.27185 populate_metadata read hostname 'smithi031' 2024-04-25T02:13:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2204713262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c86161bb0 0x563c86163fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f02a91e0740 4 client.27185 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:13:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.925+0000 7f02a91e0740 1 -- 172.21.15.31:0/2204713262 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c860b81f0 con 0x563c86161bb0 2024-04-25T02:13:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.926+0000 7f029ffff640 1 --2- 172.21.15.31:0/2204713262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c86161bb0 0x563c86163fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.926+0000 7f027f7fe640 10 client.27185 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.926+0000 7f029ffff640 1 --2- 172.21.15.31:0/2204713262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c86161bb0 0x563c86163fd0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99d26b8c23de6571 server_cookie=ceb8e909e0b6df6d in_seq=0 out_seq=0 2024-04-25T02:13:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.927+0000 7f027f7fe640 1 -- 172.21.15.31:0/2204713262 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c860b81f0 con 0x563c86161bb0 2024-04-25T02:13:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.927+0000 7f027f7fe640 10 client.27185 handle_command_reply: tid=0 2024-04-25T02:13:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.927+0000 7f02a91e0740 1 client.27185 shutdown 2024-04-25T02:13:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.927+0000 7f02a91e0740 1 -- 172.21.15.31:0/2204713262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 msgr2=0x563c8614e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.927+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2204713262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614e6e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f02940cced0 tx=0x7f029406bb60 comp rx=0 tx=0).stop 2024-04-25T02:13:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.927+0000 7f02a91e0740 1 -- 172.21.15.31:0/2204713262 shutdown_connections 2024-04-25T02:13:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.927+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2204713262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c86161bb0 0x563c86163fd0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.928+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2204713262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c8614ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.928+0000 7f027f7fe640 0 client.27185 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.928+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2204713262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614e6e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.928+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2204713262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146240 0x563c8614e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.928+0000 7f02a91e0740 1 -- 172.21.15.31:0/2204713262 >> 172.21.15.31:0/2204713262 conn(0x563c86142560 msgr2=0x563c86148360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.928+0000 7f02a91e0740 1 -- 172.21.15.31:0/2204713262 shutdown_connections 2024-04-25T02:13:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.928+0000 7f02a91e0740 1 -- 172.21.15.31:0/2204713262 wait complete. 2024-04-25T02:13:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.928+0000 7f02a91e0740 20 client.27185 trim_cache size 0 max 16384 2024-04-25T02:13:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.951+0000 7f02a91e0740 1 Processor -- start 2024-04-25T02:13:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.951+0000 7f02a91e0740 1 -- start start 2024-04-25T02:13:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.951+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146240 0x563c861508d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.951+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 0x563c8615cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.951+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c861698e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.951+0000 7f02a91e0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f02a01af550 con 0x563c86151230 2024-04-25T02:13:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.951+0000 7f02a91e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f02a01b1090 con 0x563c86146240 2024-04-25T02:13:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.951+0000 7f02a91e0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f02a01afc20 con 0x563c86146b60 2024-04-25T02:13:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f029ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146240 0x563c861508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f02a4998640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c861698e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f029ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146240 0x563c861508d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41556/0 (socket says 172.21.15.31:41556) 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f029ffff640 1 -- 172.21.15.31:0/2464325052 learned_addr learned my addr 172.21.15.31:0/2464325052 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f029f7fe640 1 --2- 172.21.15.31:0/2464325052 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 0x563c8615cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f029ffff640 1 -- 172.21.15.31:0/2464325052 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 msgr2=0x563c8615cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f029ffff640 1 --2- 172.21.15.31:0/2464325052 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 0x563c8615cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f029ffff640 1 -- 172.21.15.31:0/2464325052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 msgr2=0x563c861698e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f029ffff640 1 --2- 172.21.15.31:0/2464325052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c861698e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f029ffff640 1 -- 172.21.15.31:0/2464325052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c86146040 con 0x563c86146240 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f02a4998640 1 --2- 172.21.15.31:0/2464325052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c861698e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.952+0000 7f029ffff640 1 --2- 172.21.15.31:0/2464325052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146240 0x563c861508d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f029001c040 tx=0x7f029000f440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cb679b78d0cc05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.953+0000 7f0285ffb640 1 -- 172.21.15.31:0/2464325052 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029002f070 con 0x563c86146240 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.953+0000 7f0285ffb640 1 -- 172.21.15.31:0/2464325052 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02900054f0 con 0x563c86146240 2024-04-25T02:13:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.953+0000 7f0285ffb640 1 -- 172.21.15.31:0/2464325052 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029001e040 con 0x563c86146240 2024-04-25T02:13:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.953+0000 7f02a91e0740 1 -- 172.21.15.31:0/2464325052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146240 msgr2=0x563c861508d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.953+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2464325052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146240 0x563c861508d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f029001c040 tx=0x7f029000f440 comp rx=0 tx=0).stop 2024-04-25T02:13:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.954+0000 7f02a91e0740 1 -- 172.21.15.31:0/2464325052 shutdown_connections 2024-04-25T02:13:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.954+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2464325052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c861698e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.954+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2464325052 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146b60 0x563c8615cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.954+0000 7f02a91e0740 1 --2- 172.21.15.31:0/2464325052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146240 0x563c861508d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.954+0000 7f02a91e0740 1 -- 172.21.15.31:0/2464325052 >> 172.21.15.31:0/2464325052 conn(0x563c86142560 msgr2=0x563c86142940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.954+0000 7f02a91e0740 1 -- 172.21.15.31:0/2464325052 shutdown_connections 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.954+0000 7f02a91e0740 1 -- 172.21.15.31:0/2464325052 wait complete. 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.954+0000 7f02a91e0740 1 -- start start 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.954+0000 7f02a91e0740 1 Processor -- start 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146240 0x563c861595e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f02a91e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c8614ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f02a91e0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f02a01b09c0 con 0x563c86151230 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f029ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146240 0x563c861595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:41572/0 (socket says 172.21.15.31:41572) 2024-04-25T02:13:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f029ffff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146240 0x563c861595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:51778/0 (socket says 172.21.15.31:51778) 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f029f7fe640 1 -- 172.21.15.31:0/3480028950 learned_addr learned my addr 172.21.15.31:0/3480028950 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f02a91e0740 1 -- 172.21.15.31:0/3480028950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f02a01b02f0 con 0x563c86146b60 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f02a91e0740 1 -- 172.21.15.31:0/3480028950 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f02a01ad530 con 0x563c86146240 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f02a4998640 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c8614ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f02a91e0740 10 client.0 _ll_register_callbacks cb 0x7f029400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f029f7fe640 1 -- 172.21.15.31:0/3480028950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146240 msgr2=0x563c861595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f029f7fe640 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146240 0x563c861595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f029f7fe640 1 -- 172.21.15.31:0/3480028950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 msgr2=0x563c8614ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f029f7fe640 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c8614ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f029f7fe640 1 -- 172.21.15.31:0/3480028950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c8614b260 con 0x563c86146b60 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.955+0000 7f02a4998640 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c8614ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.956+0000 7f029f7fe640 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614a7e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f029400e9c0 tx=0x7f02940053a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a1f500333005ef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.956+0000 7f027f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.956+0000 7f027f7fe640 1 -- 172.21.15.31:0/3480028950 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029407c060 con 0x563c86146b60 2024-04-25T02:13:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.956+0000 7f02a91e0740 1 -- 172.21.15.31:0/3480028950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02900054f0 con 0x563c86146b60 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.956+0000 7f027f7fe640 1 -- 172.21.15.31:0/3480028950 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8528+0+0 (secure 0 0 0) 0x7f029406b8d0 con 0x563c86146b60 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.956+0000 7f027f7fe640 1 -- 172.21.15.31:0/3480028950 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f0294097ee0 con 0x563c86146b60 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.956+0000 7f02a91e0740 10 client.27191 fetch_fsmap learned FSMap version 49 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.956+0000 7f02a91e0740 1 -- 172.21.15.31:0/3480028950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563c861c7db0 con 0x563c86146b60 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.957+0000 7f027f7fe640 1 -- 172.21.15.31:0/3480028950 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02900054f0 con 0x563c86146b60 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.957+0000 7f027f7fe640 1 -- 172.21.15.31:0/3480028950 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f02940cd0f0 con 0x563c86146b60 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.957+0000 7f02a91e0740 10 client.27191 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.957+0000 7f02a91e0740 10 client.27191 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.957+0000 7f02a91e0740 20 client.27191 populate_metadata read hostname 'smithi031' 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.957+0000 7f02a91e0740 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c86161bb0 0x563c861ca530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.957+0000 7f02a91e0740 4 client.27191 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.957+0000 7f02a91e0740 1 -- 172.21.15.31:0/3480028950 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563c860b2540 con 0x563c86161bb0 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.957+0000 7f029ffff640 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c86161bb0 0x563c861ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.957+0000 7f027f7fe640 10 client.27191 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:02.957+0000 7f029ffff640 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c86161bb0 0x563c861ca530 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=424286a9b224f40f server_cookie=8d3ff4030e5c6e4c in_seq=0 out_seq=0 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.025+0000 7f027f7fe640 1 -- 172.21.15.31:0/3480028950 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563c860b2540 con 0x563c86161bb0 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.025+0000 7f027f7fe640 10 client.27191 handle_command_reply: tid=0 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.025+0000 7f02a91e0740 1 client.27191 shutdown 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.025+0000 7f02a91e0740 1 -- 172.21.15.31:0/3480028950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 msgr2=0x563c8614a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.025+0000 7f02a91e0740 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614a7e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f029400e9c0 tx=0x7f02940053a0 comp rx=0 tx=0).stop 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.025+0000 7f02a91e0740 1 -- 172.21.15.31:0/3480028950 shutdown_connections 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.025+0000 7f02a91e0740 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563c86161bb0 0x563c861ca530 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.026+0000 7f02a91e0740 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c86151230 0x563c8614ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.026+0000 7f027f7fe640 0 client.27191 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.026+0000 7f02a91e0740 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563c86146b60 0x563c8614a7e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.026+0000 7f02a91e0740 1 --2- 172.21.15.31:0/3480028950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c86146240 0x563c861595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.026+0000 7f02a91e0740 1 -- 172.21.15.31:0/3480028950 >> 172.21.15.31:0/3480028950 conn(0x563c86142560 msgr2=0x563c86140d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.026+0000 7f02a91e0740 1 -- 172.21.15.31:0/3480028950 shutdown_connections 2024-04-25T02:13:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.026+0000 7f02a91e0740 1 -- 172.21.15.31:0/3480028950 wait complete. 2024-04-25T02:13:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.026+0000 7f02a91e0740 20 client.27191 trim_cache size 0 max 16384 2024-04-25T02:13:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:13:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:13:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:13:03.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:13:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 -- 172.21.15.31:0/2510213138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02880a2da0 msgr2=0x7f02880a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02880a2da0 0x7f02880a5260 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0298006010 tx=0x7f029803b040 comp rx=0 tx=0).stop 2024-04-25T02:13:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 -- 172.21.15.31:0/2510213138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a01032f0 msgr2=0x7f02a019a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a01032f0 0x7f02a019a5e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f029400e9f0 tx=0x7f029400edf0 comp rx=0 tx=0).stop 2024-04-25T02:13:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 -- 172.21.15.31:0/2510213138 shutdown_connections 2024-04-25T02:13:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02880a2da0 0x7f02880a5260 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02a010bd10 0x7f02a019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a01032f0 0x7f02a019a5e0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 --2- 172.21.15.31:0/2510213138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02a01029d0 0x7f02a019a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 -- 172.21.15.31:0/2510213138 >> 172.21.15.31:0/2510213138 conn(0x7f02a00777c0 msgr2=0x7f02a0107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 -- 172.21.15.31:0/2510213138 shutdown_connections 2024-04-25T02:13:03.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:03.027+0000 7f02a63fb640 1 -- 172.21.15.31:0/2510213138 wait complete. 2024-04-25T02:13:03.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 50963 8.92 MB/sec execute 187 sec latency 79.482 ms 2024-04-25T02:13:03.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:03 smithi040 ceph-mon[32418]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T02:13:03.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:03 smithi134 ceph-mon[33098]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T02:13:03.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:03 smithi031 ceph-mon[29204]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T02:13:04.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 51199 8.99 MB/sec execute 188 sec latency 217.568 ms 2024-04-25T02:13:04.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:04 smithi040 ceph-mon[32418]: pgmap v584: 97 pgs: 97 active+clean; 888 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 279 op/s 2024-04-25T02:13:04.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:04 smithi134 ceph-mon[33098]: pgmap v584: 97 pgs: 97 active+clean; 888 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 279 op/s 2024-04-25T02:13:04.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:04 smithi031 ceph-mon[29204]: pgmap v584: 97 pgs: 97 active+clean; 888 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 279 op/s 2024-04-25T02:13:05.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 51422 9.00 MB/sec execute 189 sec latency 123.724 ms 2024-04-25T02:13:06.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 51529 8.96 MB/sec execute 190 sec latency 91.030 ms 2024-04-25T02:13:06.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:06 smithi040 ceph-mon[32418]: pgmap v585: 97 pgs: 97 active+clean; 910 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.6 MiB/s wr, 279 op/s 2024-04-25T02:13:06.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:06 smithi134 ceph-mon[33098]: pgmap v585: 97 pgs: 97 active+clean; 910 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.6 MiB/s wr, 279 op/s 2024-04-25T02:13:06.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:06 smithi031 ceph-mon[29204]: pgmap v585: 97 pgs: 97 active+clean; 910 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.6 MiB/s wr, 279 op/s 2024-04-25T02:13:07.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 51656 8.95 MB/sec execute 191 sec latency 55.474 ms 2024-04-25T02:13:08.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 51860 9.02 MB/sec execute 192 sec latency 173.256 ms 2024-04-25T02:13:08.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:08 smithi040 ceph-mon[32418]: pgmap v586: 97 pgs: 97 active+clean; 910 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.6 MiB/s wr, 279 op/s 2024-04-25T02:13:08.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:08 smithi134 ceph-mon[33098]: pgmap v586: 97 pgs: 97 active+clean; 910 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.6 MiB/s wr, 279 op/s 2024-04-25T02:13:08.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:08 smithi031 ceph-mon[29204]: pgmap v586: 97 pgs: 97 active+clean; 910 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.6 MiB/s wr, 279 op/s 2024-04-25T02:13:09.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 52024 9.02 MB/sec execute 193 sec latency 81.368 ms 2024-04-25T02:13:10.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 52150 8.99 MB/sec execute 194 sec latency 54.673 ms 2024-04-25T02:13:10.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:09 smithi031 ceph-mon[29204]: pgmap v587: 97 pgs: 97 active+clean; 910 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.6 MiB/s wr, 279 op/s 2024-04-25T02:13:10.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:09 smithi031 ceph-mon[29204]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T02:13:10.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:09 smithi040 ceph-mon[32418]: pgmap v587: 97 pgs: 97 active+clean; 910 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.6 MiB/s wr, 279 op/s 2024-04-25T02:13:10.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:09 smithi040 ceph-mon[32418]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T02:13:10.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:09 smithi134 ceph-mon[33098]: pgmap v587: 97 pgs: 97 active+clean; 910 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.6 MiB/s wr, 279 op/s 2024-04-25T02:13:10.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:09 smithi134 ceph-mon[33098]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T02:13:11.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 52305 8.98 MB/sec execute 195 sec latency 54.175 ms 2024-04-25T02:13:12.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 52476 8.96 MB/sec execute 196 sec latency 60.250 ms 2024-04-25T02:13:12.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:11 smithi031 ceph-mon[29204]: pgmap v589: 97 pgs: 97 active+clean; 931 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.6 MiB/s wr, 332 op/s 2024-04-25T02:13:12.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:11 smithi040 ceph-mon[32418]: pgmap v589: 97 pgs: 97 active+clean; 931 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.6 MiB/s wr, 332 op/s 2024-04-25T02:13:12.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:11 smithi134 ceph-mon[33098]: pgmap v589: 97 pgs: 97 active+clean; 931 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.6 MiB/s wr, 332 op/s 2024-04-25T02:13:13.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 52642 8.94 MB/sec execute 197 sec latency 81.040 ms 2024-04-25T02:13:14.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 52793 8.92 MB/sec execute 198 sec latency 71.380 ms 2024-04-25T02:13:14.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:13 smithi031 ceph-mon[29204]: pgmap v590: 97 pgs: 97 active+clean; 931 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.7 MiB/s wr, 292 op/s 2024-04-25T02:13:14.246 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:13 smithi134 ceph-mon[33098]: pgmap v590: 97 pgs: 97 active+clean; 931 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.7 MiB/s wr, 292 op/s 2024-04-25T02:13:14.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:13 smithi040 ceph-mon[32418]: pgmap v590: 97 pgs: 97 active+clean; 931 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.7 MiB/s wr, 292 op/s 2024-04-25T02:13:15.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 52961 8.93 MB/sec execute 199 sec latency 93.707 ms 2024-04-25T02:13:16.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 53159 8.95 MB/sec execute 200 sec latency 55.057 ms 2024-04-25T02:13:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.156+0000 7fcb3be12640 1 -- 172.21.15.31:0/4196965657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb34107740 msgr2=0x7fcb34107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.156+0000 7fcb3be12640 1 --2- 172.21.15.31:0/4196965657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb34107740 0x7fcb34107b20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009900 tx=0x7fcb3002a410 comp rx=0 tx=0).stop 2024-04-25T02:13:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.156+0000 7fcb3be12640 1 -- 172.21.15.31:0/4196965657 shutdown_connections 2024-04-25T02:13:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.156+0000 7fcb3be12640 1 --2- 172.21.15.31:0/4196965657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34100600 0x7fcb3410e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.156+0000 7fcb3be12640 1 --2- 172.21.15.31:0/4196965657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb340ffc40 0x7fcb341000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.156+0000 7fcb3be12640 1 --2- 172.21.15.31:0/4196965657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb34107740 0x7fcb34107b20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.156+0000 7fcb3be12640 1 -- 172.21.15.31:0/4196965657 >> 172.21.15.31:0/4196965657 conn(0x7fcb34076b80 msgr2=0x7fcb340fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.156+0000 7fcb3be12640 1 -- 172.21.15.31:0/4196965657 shutdown_connections 2024-04-25T02:13:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.156+0000 7fcb3be12640 1 -- 172.21.15.31:0/4196965657 wait complete. 2024-04-25T02:13:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.157+0000 7fcb3be12640 1 Processor -- start 2024-04-25T02:13:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.157+0000 7fcb3be12640 1 -- start start 2024-04-25T02:13:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.157+0000 7fcb3be12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb340ffc40 0x7fcb34197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.157+0000 7fcb3be12640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34100600 0x7fcb34198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.157+0000 7fcb3be12640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb34107740 0x7fcb3419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.157+0000 7fcb39bae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb340ffc40 0x7fcb34197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.157+0000 7fcb3be12640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb341119e0 con 0x7fcb34100600 2024-04-25T02:13:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.157+0000 7fcb3be12640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcb34111860 con 0x7fcb340ffc40 2024-04-25T02:13:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb39bae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb340ffc40 0x7fcb34197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:53018/0 (socket says 172.21.15.31:53018) 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb393ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34100600 0x7fcb34198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb3a3af640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb34107740 0x7fcb3419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb39bae640 1 -- 172.21.15.31:0/354108204 learned_addr learned my addr 172.21.15.31:0/354108204 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb3be12640 1 -- 172.21.15.31:0/354108204 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcb34111b60 con 0x7fcb34107740 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb3a3af640 1 -- 172.21.15.31:0/354108204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb340ffc40 msgr2=0x7fcb34197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb3a3af640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb340ffc40 0x7fcb34197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb3a3af640 1 -- 172.21.15.31:0/354108204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34100600 msgr2=0x7fcb34198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb3a3af640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34100600 0x7fcb34198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb3a3af640 1 -- 172.21.15.31:0/354108204 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb3419cea0 con 0x7fcb34107740 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb3a3af640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb34107740 0x7fcb3419c770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c00ec50 tx=0x7fcb2c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb2affd640 1 -- 172.21.15.31:0/354108204 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c003b70 con 0x7fcb34107740 2024-04-25T02:13:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb2affd640 1 -- 172.21.15.31:0/354108204 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb2c003d10 con 0x7fcb34107740 2024-04-25T02:13:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb3be12640 1 -- 172.21.15.31:0/354108204 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb3419d190 con 0x7fcb34107740 2024-04-25T02:13:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb2affd640 1 -- 172.21.15.31:0/354108204 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c010760 con 0x7fcb34107740 2024-04-25T02:13:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.158+0000 7fcb3be12640 1 -- 172.21.15.31:0/354108204 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb341a4a20 con 0x7fcb34107740 2024-04-25T02:13:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.161+0000 7fcb2affd640 1 -- 172.21.15.31:0/354108204 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcb2c004020 con 0x7fcb34107740 2024-04-25T02:13:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.161+0000 7fcb3ebfb740 1 Processor -- start 2024-04-25T02:13:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.161+0000 7fcb3ebfb740 1 -- start start 2024-04-25T02:13:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed34f70 0x55599ed35350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 0x55599ed44100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed44640 0x55599ed46a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55599ec99cb0 con 0x55599ed44640 2024-04-25T02:13:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55599ec425b0 con 0x55599ed34f70 2024-04-25T02:13:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55599ec95c70 con 0x55599ed35920 2024-04-25T02:13:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb393ad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed34f70 0x55599ed35350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb39bae640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 0x55599ed44100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb393ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed44640 0x55599ed46a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb2affd640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcb200a2e20 0x7fcb200a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb39bae640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 0x55599ed44100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59456/0 (socket says 172.21.15.31:59456) 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb393ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed44640 0x55599ed46a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48184/0 (socket says 172.21.15.31:48184) 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb39bae640 1 -- 172.21.15.31:0/767595009 learned_addr learned my addr 172.21.15.31:0/767595009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb3a3af640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcb200a2e20 0x7fcb200a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb39bae640 1 -- 172.21.15.31:0/767595009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed34f70 msgr2=0x55599ed35350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb39bae640 1 --2- 172.21.15.31:0/767595009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed34f70 0x55599ed35350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb39bae640 1 -- 172.21.15.31:0/767595009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed44640 msgr2=0x55599ed46a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb39bae640 1 --2- 172.21.15.31:0/767595009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed44640 0x55599ed46a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb39bae640 1 -- 172.21.15.31:0/767595009 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55599e9ae210 con 0x55599ed35920 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.162+0000 7fcb393ad640 1 --2- 172.21.15.31:0/767595009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed44640 0x55599ed46a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:16.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.163+0000 7fcb2affd640 1 -- 172.21.15.31:0/354108204 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7fcb2c0cf900 con 0x7fcb34107740 2024-04-25T02:13:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.163+0000 7fcb39bae640 1 --2- 172.21.15.31:0/767595009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 0x55599ed44100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009a20 tx=0x7fcb30031af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba89a4d05b2c1c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.163+0000 7fcb1ffff640 1 -- 172.21.15.31:0/767595009 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30035040 con 0x55599ed35920 2024-04-25T02:13:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.163+0000 7fcb1ffff640 1 -- 172.21.15.31:0/767595009 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb30031760 con 0x55599ed35920 2024-04-25T02:13:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.163+0000 7fcb1ffff640 1 -- 172.21.15.31:0/767595009 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb300047a0 con 0x55599ed35920 2024-04-25T02:13:16.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.164+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/767595009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 msgr2=0x55599ed44100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:16.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.164+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/767595009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 0x55599ed44100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009a20 tx=0x7fcb30031af0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.164+0000 7fcb3a3af640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcb200a2e20 0x7fcb200a52e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c098810 tx=0x7fcb2c097a40 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:16.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.164+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/767595009 shutdown_connections 2024-04-25T02:13:16.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.164+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/767595009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed44640 0x55599ed46a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.164+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/767595009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 0x55599ed44100 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.164+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/767595009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed34f70 0x55599ed35350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.164+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/767595009 >> 172.21.15.31:0/767595009 conn(0x55599ed31560 msgr2=0x55599ed31940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.164+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/767595009 shutdown_connections 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.164+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/767595009 wait complete. 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.164+0000 7fcb3ebfb740 1 -- start start 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb3ebfb740 1 Processor -- start 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed34f70 0x55599ed41630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed41b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed44640 0x55599ed420b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55599ed473a0 con 0x55599ed35920 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55599ec55b50 con 0x55599ed44640 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55599ed47520 con 0x55599ed34f70 2024-04-25T02:13:16.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb3ebfb740 10 client.0 _ll_register_callbacks cb 0x55599ec90ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb39bae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed41b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb39bae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed41b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48188/0 (socket says 172.21.15.31:48188) 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb39bae640 1 -- 172.21.15.31:0/1196389985 learned_addr learned my addr 172.21.15.31:0/1196389985 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb393ad640 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed34f70 0x55599ed41630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.165+0000 7fcb393ad640 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed44640 0x55599ed420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb39bae640 1 -- 172.21.15.31:0/1196389985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed34f70 msgr2=0x55599ed41630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb39bae640 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed34f70 0x55599ed41630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb39bae640 1 -- 172.21.15.31:0/1196389985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed44640 msgr2=0x55599ed420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb39bae640 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed44640 0x55599ed420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb39bae640 1 -- 172.21.15.31:0/1196389985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55599ea7a620 con 0x55599ed35920 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb393ad640 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed44640 0x55599ed420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb39bae640 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed41b70 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fcb30000c00 tx=0x7fcb30005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2030c89e44fb636 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:13:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1196389985 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30035070 con 0x55599ed35920 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1196389985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb30031760 con 0x55599ed35920 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1196389985 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7fcb30031560 con 0x55599ed35920 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1196389985 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fcb30009520 con 0x55599ed35920 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb3ebfb740 10 client.17640 fetch_fsmap learned FSMap version 49 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.166+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1196389985 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55599ed53630 con 0x55599ed35920 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.167+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1196389985 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb30031760 con 0x55599ed35920 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.167+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1196389985 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fcb30055070 con 0x55599ed35920 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.167+0000 7fcb3ebfb740 10 client.17640 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.167+0000 7fcb3ebfb740 10 client.17640 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.167+0000 7fcb3ebfb740 20 client.17640 populate_metadata read hostname 'smithi031' 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.167+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55599ed54980 0x55599ed56d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.167+0000 7fcb3ebfb740 4 client.17640 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.167+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1196389985 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55599eca71f0 con 0x55599ed54980 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.167+0000 7fcb393ad640 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55599ed54980 0x55599ed56d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.168+0000 7fcb393ad640 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55599ed54980 0x55599ed56d70 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=921805b985e1f585 server_cookie=fe8a5eace275bc3d in_seq=0 out_seq=0 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.168+0000 7fcb1d7fa640 10 client.17640 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1196389985 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55599eca71f0 con 0x55599ed54980 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb1d7fa640 10 client.17640 handle_command_reply: tid=0 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 1 client.17640 shutdown 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1196389985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 msgr2=0x55599ed41b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed41b70 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fcb30000c00 tx=0x7fcb30005760 comp rx=0 tx=0).stop 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1196389985 shutdown_connections 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55599ed54980 0x55599ed56d70 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb1d7fa640 0 client.17640 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed44640 0x55599ed420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed41b70 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1196389985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed34f70 0x55599ed41630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1196389985 >> 172.21.15.31:0/1196389985 conn(0x55599ed31560 msgr2=0x55599ed373d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1196389985 shutdown_connections 2024-04-25T02:13:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1196389985 wait complete. 2024-04-25T02:13:16.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.169+0000 7fcb3ebfb740 20 client.17640 trim_cache size 0 max 16384 2024-04-25T02:13:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.192+0000 7fcb3ebfb740 1 Processor -- start 2024-04-25T02:13:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.192+0000 7fcb3ebfb740 1 -- start start 2024-04-25T02:13:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.192+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed34f70 0x55599ed43d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.192+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed502a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.192+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed44640 0x55599ed303b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.192+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb341aaff0 con 0x55599ed35920 2024-04-25T02:13:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.192+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcb341acb30 con 0x55599ed34f70 2024-04-25T02:13:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.192+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcb341ab6c0 con 0x55599ed44640 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb39bae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb39bae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48200/0 (socket says 172.21.15.31:48200) 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb39bae640 1 -- 172.21.15.31:0/615013711 learned_addr learned my addr 172.21.15.31:0/615013711 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb393ad640 1 --2- 172.21.15.31:0/615013711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed34f70 0x55599ed43d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb393ad640 1 --2- 172.21.15.31:0/615013711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed44640 0x55599ed303b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb39bae640 1 -- 172.21.15.31:0/615013711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed44640 msgr2=0x55599ed303b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb39bae640 1 --2- 172.21.15.31:0/615013711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed44640 0x55599ed303b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb39bae640 1 -- 172.21.15.31:0/615013711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed34f70 msgr2=0x55599ed43d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb39bae640 1 --2- 172.21.15.31:0/615013711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed34f70 0x55599ed43d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb39bae640 1 -- 172.21.15.31:0/615013711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55599ed356c0 con 0x55599ed35920 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb393ad640 1 --2- 172.21.15.31:0/615013711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed44640 0x55599ed303b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb39bae640 1 --2- 172.21.15.31:0/615013711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed502a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fcb3003ca00 tx=0x7fcb300058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bea5ae7de445388 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb1f7fe640 1 -- 172.21.15.31:0/615013711 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3005b070 con 0x55599ed35920 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.193+0000 7fcb1f7fe640 1 -- 172.21.15.31:0/615013711 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb3003ca80 con 0x55599ed35920 2024-04-25T02:13:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.194+0000 7fcb1f7fe640 1 -- 172.21.15.31:0/615013711 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30035040 con 0x55599ed35920 2024-04-25T02:13:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.194+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/615013711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 msgr2=0x55599ed502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.194+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/615013711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed502a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fcb3003ca00 tx=0x7fcb300058f0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.195+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/615013711 shutdown_connections 2024-04-25T02:13:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.195+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/615013711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed44640 0x55599ed303b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.195+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/615013711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed35920 0x55599ed502a0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.195+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/615013711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed34f70 0x55599ed43d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.195+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/615013711 >> 172.21.15.31:0/615013711 conn(0x55599ed31560 msgr2=0x55599ed2fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.195+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/615013711 shutdown_connections 2024-04-25T02:13:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.195+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/615013711 wait complete. 2024-04-25T02:13:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.195+0000 7fcb3ebfb740 1 -- start start 2024-04-25T02:13:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.195+0000 7fcb3ebfb740 1 Processor -- start 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed34f70 0x55599ed4cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 0x55599ed47e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb3ebfb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed44640 0x55599ed483a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb341ac460 con 0x55599ed34f70 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcb341abd90 con 0x55599ed44640 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb3ebfb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcb341a8ff0 con 0x55599ed35920 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb3ebfb740 10 client.0 _ll_register_callbacks cb 0x7fcb24009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb393ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed34f70 0x55599ed4cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb393ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed34f70 0x55599ed4cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48202/0 (socket says 172.21.15.31:48202) 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb393ad640 1 -- 172.21.15.31:0/1451792449 learned_addr learned my addr 172.21.15.31:0/1451792449 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb39bae640 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 0x55599ed47e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb393ad640 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed44640 0x55599ed483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb393ad640 1 -- 172.21.15.31:0/1451792449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 msgr2=0x55599ed47e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb393ad640 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 0x55599ed47e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb393ad640 1 -- 172.21.15.31:0/1451792449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed44640 msgr2=0x55599ed483a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb393ad640 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed44640 0x55599ed483a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.196+0000 7fcb393ad640 1 -- 172.21.15.31:0/1451792449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55599ed48b80 con 0x55599ed34f70 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.197+0000 7fcb393ad640 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed34f70 0x55599ed4cb40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x55599ed572b0 tx=0x7fcb24025040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae6bf3c7a930013 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.197+0000 7fcb1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.197+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1451792449 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb24019040 con 0x55599ed34f70 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.197+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1451792449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb3003ca80 con 0x55599ed34f70 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.197+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1451792449 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7fcb24005020 con 0x55599ed34f70 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.197+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1451792449 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fcb24038050 con 0x55599ed34f70 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.197+0000 7fcb3ebfb740 10 client.17646 fetch_fsmap learned FSMap version 49 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.197+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1451792449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55599edbac20 con 0x55599ed34f70 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.197+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1451792449 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb3003ca80 con 0x55599ed34f70 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.198+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1451792449 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fcb24004bf0 con 0x55599ed34f70 2024-04-25T02:13:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.198+0000 7fcb3ebfb740 10 client.17646 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:13:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.198+0000 7fcb3ebfb740 10 client.17646 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:13:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.198+0000 7fcb3ebfb740 20 client.17646 populate_metadata read hostname 'smithi031' 2024-04-25T02:13:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.198+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55599ed54980 0x55599edbd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.198+0000 7fcb3ebfb740 4 client.17646 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:13:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.198+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1451792449 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55599eca1540 con 0x55599ed54980 2024-04-25T02:13:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.198+0000 7fcb39bae640 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55599ed54980 0x55599edbd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.198+0000 7fcb1d7fa640 10 client.17646 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.198+0000 7fcb39bae640 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55599ed54980 0x55599edbd390 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7293ba71b71df8ba server_cookie=dcd8bbdbbecb549e in_seq=0 out_seq=0 2024-04-25T02:13:16.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:15 smithi031 ceph-mon[29204]: pgmap v591: 97 pgs: 97 active+clean; 937 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 284 op/s 2024-04-25T02:13:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.398+0000 7fcb1d7fa640 1 -- 172.21.15.31:0/1451792449 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55599eca1540 con 0x55599ed54980 2024-04-25T02:13:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.398+0000 7fcb1d7fa640 10 client.17646 handle_command_reply: tid=0 2024-04-25T02:13:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.398+0000 7fcb3ebfb740 1 client.17646 shutdown 2024-04-25T02:13:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1451792449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed34f70 msgr2=0x55599ed4cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed34f70 0x55599ed4cb40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x55599ed572b0 tx=0x7fcb24025040 comp rx=0 tx=0).stop 2024-04-25T02:13:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1451792449 shutdown_connections 2024-04-25T02:13:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55599ed54980 0x55599edbd390 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599ed44640 0x55599ed483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55599ed35920 0x55599ed47e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb3ebfb740 1 --2- 172.21.15.31:0/1451792449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55599ed34f70 0x55599ed4cb40 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1451792449 >> 172.21.15.31:0/1451792449 conn(0x55599ed31560 msgr2=0x55599ed30510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb1d7fa640 0 client.17646 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1451792449 shutdown_connections 2024-04-25T02:13:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb3ebfb740 1 -- 172.21.15.31:0/1451792449 wait complete. 2024-04-25T02:13:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.399+0000 7fcb3ebfb740 20 client.17646 trim_cache size 0 max 16384 2024-04-25T02:13:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:13:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:13:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:13:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:13:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 -- 172.21.15.31:0/354108204 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcb200a2e20 msgr2=0x7fcb200a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcb200a2e20 0x7fcb200a52e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c098810 tx=0x7fcb2c097a40 comp rx=0 tx=0).stop 2024-04-25T02:13:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 -- 172.21.15.31:0/354108204 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb34107740 msgr2=0x7fcb3419c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb34107740 0x7fcb3419c770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c00ec50 tx=0x7fcb2c00c6a0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 -- 172.21.15.31:0/354108204 shutdown_connections 2024-04-25T02:13:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fcb200a2e20 0x7fcb200a52e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb34107740 0x7fcb3419c770 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34100600 0x7fcb34198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 --2- 172.21.15.31:0/354108204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb340ffc40 0x7fcb34197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 -- 172.21.15.31:0/354108204 >> 172.21.15.31:0/354108204 conn(0x7fcb34076b80 msgr2=0x7fcb340fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 -- 172.21.15.31:0/354108204 shutdown_connections 2024-04-25T02:13:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:16.400+0000 7fcb3be12640 1 -- 172.21.15.31:0/354108204 wait complete. 2024-04-25T02:13:16.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:15 smithi040 ceph-mon[32418]: pgmap v591: 97 pgs: 97 active+clean; 937 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 284 op/s 2024-04-25T02:13:16.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:15 smithi134 ceph-mon[33098]: pgmap v591: 97 pgs: 97 active+clean; 937 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 284 op/s 2024-04-25T02:13:17.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 53367 9.01 MB/sec execute 201 sec latency 72.981 ms 2024-04-25T02:13:18.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 53506 8.99 MB/sec execute 202 sec latency 57.824 ms 2024-04-25T02:13:18.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:17 smithi031 ceph-mon[29204]: pgmap v592: 97 pgs: 97 active+clean; 937 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 284 op/s 2024-04-25T02:13:18.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:17 smithi040 ceph-mon[32418]: pgmap v592: 97 pgs: 97 active+clean; 937 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 284 op/s 2024-04-25T02:13:18.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:17 smithi134 ceph-mon[33098]: pgmap v592: 97 pgs: 97 active+clean; 937 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 284 op/s 2024-04-25T02:13:19.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 53637 8.95 MB/sec execute 203 sec latency 56.699 ms 2024-04-25T02:13:20.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 53811 8.95 MB/sec execute 204 sec latency 71.724 ms 2024-04-25T02:13:20.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:19 smithi031 ceph-mon[29204]: pgmap v593: 97 pgs: 97 active+clean; 937 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 284 op/s 2024-04-25T02:13:20.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:19 smithi040 ceph-mon[32418]: pgmap v593: 97 pgs: 97 active+clean; 937 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 284 op/s 2024-04-25T02:13:20.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:19 smithi134 ceph-mon[33098]: pgmap v593: 97 pgs: 97 active+clean; 937 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 284 op/s 2024-04-25T02:13:21.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 53982 8.92 MB/sec execute 205 sec latency 62.539 ms 2024-04-25T02:13:22.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 54144 8.91 MB/sec execute 206 sec latency 61.518 ms 2024-04-25T02:13:22.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:21 smithi031 ceph-mon[29204]: pgmap v594: 97 pgs: 97 active+clean; 947 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 360 op/s 2024-04-25T02:13:22.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:21 smithi040 ceph-mon[32418]: pgmap v594: 97 pgs: 97 active+clean; 947 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 360 op/s 2024-04-25T02:13:22.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:21 smithi134 ceph-mon[33098]: pgmap v594: 97 pgs: 97 active+clean; 947 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 360 op/s 2024-04-25T02:13:23.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 54290 8.90 MB/sec execute 207 sec latency 54.012 ms 2024-04-25T02:13:24.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 54479 8.94 MB/sec execute 208 sec latency 51.743 ms 2024-04-25T02:13:24.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:23 smithi031 ceph-mon[29204]: pgmap v595: 97 pgs: 97 active+clean; 947 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 249 op/s 2024-04-25T02:13:24.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:23 smithi040 ceph-mon[32418]: pgmap v595: 97 pgs: 97 active+clean; 947 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 249 op/s 2024-04-25T02:13:24.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:23 smithi134 ceph-mon[33098]: pgmap v595: 97 pgs: 97 active+clean; 947 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 249 op/s 2024-04-25T02:13:25.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 54678 8.96 MB/sec execute 209 sec latency 59.702 ms 2024-04-25T02:13:26.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 54902 8.98 MB/sec execute 210 sec latency 58.647 ms 2024-04-25T02:13:26.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:25 smithi031 ceph-mon[29204]: pgmap v596: 97 pgs: 97 active+clean; 964 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.2 MiB/s wr, 359 op/s 2024-04-25T02:13:26.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:25 smithi040 ceph-mon[32418]: pgmap v596: 97 pgs: 97 active+clean; 964 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.2 MiB/s wr, 359 op/s 2024-04-25T02:13:26.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:25 smithi134 ceph-mon[33098]: pgmap v596: 97 pgs: 97 active+clean; 964 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.2 MiB/s wr, 359 op/s 2024-04-25T02:13:27.099 INFO:tasks.workunit.client.0.smithi031.stdout: 10 55049 8.97 MB/sec execute 211 sec latency 91.977 ms 2024-04-25T02:13:28.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 55214 8.97 MB/sec execute 212 sec latency 60.059 ms 2024-04-25T02:13:28.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:27 smithi031 ceph-mon[29204]: pgmap v597: 97 pgs: 97 active+clean; 964 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-04-25T02:13:28.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:27 smithi040 ceph-mon[32418]: pgmap v597: 97 pgs: 97 active+clean; 964 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-04-25T02:13:28.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:27 smithi134 ceph-mon[33098]: pgmap v597: 97 pgs: 97 active+clean; 964 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-04-25T02:13:29.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 55382 9.00 MB/sec execute 213 sec latency 149.184 ms 2024-04-25T02:13:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.533+0000 7f0973412640 1 -- 172.21.15.31:0/991611624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c1029d0 msgr2=0x7f096c102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.533+0000 7f0973412640 1 --2- 172.21.15.31:0/991611624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c1029d0 0x7f096c102db0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f0968009900 tx=0x7f096802a510 comp rx=0 tx=0).stop 2024-04-25T02:13:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.533+0000 7f0973412640 1 -- 172.21.15.31:0/991611624 shutdown_connections 2024-04-25T02:13:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.533+0000 7f0973412640 1 --2- 172.21.15.31:0/991611624 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f096c10bd10 0x7f096c10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.533+0000 7f0973412640 1 --2- 172.21.15.31:0/991611624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c1032f0 0x7f096c10b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.533+0000 7f0973412640 1 --2- 172.21.15.31:0/991611624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c1029d0 0x7f096c102db0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.533+0000 7f0973412640 1 -- 172.21.15.31:0/991611624 >> 172.21.15.31:0/991611624 conn(0x7f096c0777c0 msgr2=0x7f096c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.533+0000 7f0973412640 1 -- 172.21.15.31:0/991611624 shutdown_connections 2024-04-25T02:13:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.533+0000 7f0973412640 1 -- 172.21.15.31:0/991611624 wait complete. 2024-04-25T02:13:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f0973412640 1 Processor -- start 2024-04-25T02:13:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f0973412640 1 -- start start 2024-04-25T02:13:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f0973412640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f096c1029d0 0x7f096c100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f0973412640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c1032f0 0x7f096c100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f0973412640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c10bd10 0x7f096c1010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f0973412640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1119c0 con 0x7f096c1032f0 2024-04-25T02:13:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f0973412640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f096c111840 con 0x7f096c10bd10 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f0973412640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f096c111b40 con 0x7f096c1029d0 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f09709ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c1032f0 0x7f096c100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f09711ae640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f096c1029d0 0x7f096c100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f09709ad640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c1032f0 0x7f096c100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43966/0 (socket says 172.21.15.31:43966) 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f09709ad640 1 -- 172.21.15.31:0/2386344423 learned_addr learned my addr 172.21.15.31:0/2386344423 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f09719af640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c10bd10 0x7f096c1010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f09709ad640 1 -- 172.21.15.31:0/2386344423 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f096c1029d0 msgr2=0x7f096c100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f09709ad640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f096c1029d0 0x7f096c100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f09709ad640 1 -- 172.21.15.31:0/2386344423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c10bd10 msgr2=0x7f096c1010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f09709ad640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c10bd10 0x7f096c1010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.534+0000 7f09709ad640 1 -- 172.21.15.31:0/2386344423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096c1a63c0 con 0x7f096c1032f0 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.535+0000 7f09719af640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c10bd10 0x7f096c1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.535+0000 7f09711ae640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f096c1029d0 0x7f096c100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.535+0000 7f09709ad640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c1032f0 0x7f096c100b60 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f095c00ea60 tx=0x7f095c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.535+0000 7f09627fc640 1 -- 172.21.15.31:0/2386344423 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c00cde0 con 0x7f096c1032f0 2024-04-25T02:13:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.535+0000 7f09627fc640 1 -- 172.21.15.31:0/2386344423 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f095c0035f0 con 0x7f096c1032f0 2024-04-25T02:13:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.535+0000 7f0973412640 1 -- 172.21.15.31:0/2386344423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096c1a66b0 con 0x7f096c1032f0 2024-04-25T02:13:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.535+0000 7f09627fc640 1 -- 172.21.15.31:0/2386344423 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c010690 con 0x7f096c1032f0 2024-04-25T02:13:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.535+0000 7f0973412640 1 -- 172.21.15.31:0/2386344423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f096c1a6c50 con 0x7f096c1032f0 2024-04-25T02:13:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.538+0000 7f09627fc640 1 -- 172.21.15.31:0/2386344423 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f095c0037a0 con 0x7f096c1032f0 2024-04-25T02:13:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.538+0000 7f09761f9740 1 Processor -- start 2024-04-25T02:13:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.538+0000 7f09761f9740 1 -- start start 2024-04-25T02:13:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09761f9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3240 0x5586167e3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09761f9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167edcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09761f9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167f0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09761f9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558616747cb0 con 0x5586167ee230 2024-04-25T02:13:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09761f9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5586166f05b0 con 0x5586167e3b60 2024-04-25T02:13:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09761f9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x558616743c70 con 0x5586167e3240 2024-04-25T02:13:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09711ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167edcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09711ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167edcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38510/0 (socket says 172.21.15.31:38510) 2024-04-25T02:13:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09711ae640 1 -- 172.21.15.31:0/3202938462 learned_addr learned my addr 172.21.15.31:0/3202938462 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09709ad640 1 --2- 172.21.15.31:0/3202938462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167f0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09719af640 1 --2- 172.21.15.31:0/3202938462 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3240 0x5586167e3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09627fc640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09580a2e50 0x7f09580a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09711ae640 1 -- 172.21.15.31:0/3202938462 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3240 msgr2=0x5586167e3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09711ae640 1 --2- 172.21.15.31:0/3202938462 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3240 0x5586167e3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09711ae640 1 -- 172.21.15.31:0/3202938462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 msgr2=0x5586167f0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09627fc640 1 -- 172.21.15.31:0/2386344423 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7f095c0cf610 con 0x7f096c1032f0 2024-04-25T02:13:29.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09719af640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09580a2e50 0x7f09580a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09711ae640 1 --2- 172.21.15.31:0/3202938462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167f0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09711ae640 1 -- 172.21.15.31:0/3202938462 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55861645c210 con 0x5586167e3b60 2024-04-25T02:13:29.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.539+0000 7f09709ad640 1 --2- 172.21.15.31:0/3202938462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:29.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.540+0000 7f09711ae640 1 --2- 172.21.15.31:0/3202938462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167edcf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0968009900 tx=0x7f0968005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ded38ec0ab5e767 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:29.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.540+0000 7f09577fe640 1 -- 172.21.15.31:0/3202938462 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0968040070 con 0x5586167e3b60 2024-04-25T02:13:29.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.540+0000 7f09719af640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09580a2e50 0x7f09580a5310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0964005c40 tx=0x7f0964004a40 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:29.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.540+0000 7f09577fe640 1 -- 172.21.15.31:0/3202938462 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0968031860 con 0x5586167e3b60 2024-04-25T02:13:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.540+0000 7f09577fe640 1 -- 172.21.15.31:0/3202938462 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0968035040 con 0x5586167e3b60 2024-04-25T02:13:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.541+0000 7f09761f9740 1 -- 172.21.15.31:0/3202938462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 msgr2=0x5586167edcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.541+0000 7f09761f9740 1 --2- 172.21.15.31:0/3202938462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167edcf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0968009900 tx=0x7f0968005d90 comp rx=0 tx=0).stop 2024-04-25T02:13:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.541+0000 7f09761f9740 1 -- 172.21.15.31:0/3202938462 shutdown_connections 2024-04-25T02:13:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.541+0000 7f09761f9740 1 --2- 172.21.15.31:0/3202938462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.541+0000 7f09761f9740 1 --2- 172.21.15.31:0/3202938462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167edcf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.541+0000 7f09761f9740 1 --2- 172.21.15.31:0/3202938462 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3240 0x5586167e3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.541+0000 7f09761f9740 1 -- 172.21.15.31:0/3202938462 >> 172.21.15.31:0/3202938462 conn(0x5586167df560 msgr2=0x5586167df940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.541+0000 7f09761f9740 1 -- 172.21.15.31:0/3202938462 shutdown_connections 2024-04-25T02:13:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.541+0000 7f09761f9740 1 -- 172.21.15.31:0/3202938462 wait complete. 2024-04-25T02:13:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09761f9740 1 -- start start 2024-04-25T02:13:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09761f9740 1 Processor -- start 2024-04-25T02:13:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09761f9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3240 0x5586167eb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09761f9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09761f9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167ebbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09761f9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5586167f0fc0 con 0x5586167ee230 2024-04-25T02:13:29.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09761f9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558616703b50 con 0x5586167e3b60 2024-04-25T02:13:29.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09761f9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5586167f1140 con 0x5586167e3240 2024-04-25T02:13:29.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09761f9740 10 client.0 _ll_register_callbacks cb 0x55861673eca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:13:29.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09711ae640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3240 0x5586167eb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09709ad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09719af640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167ebbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09709ad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167eb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38520/0 (socket says 172.21.15.31:38520) 2024-04-25T02:13:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.542+0000 7f09709ad640 1 -- 172.21.15.31:0/35953726 learned_addr learned my addr 172.21.15.31:0/35953726 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.543+0000 7f09709ad640 1 -- 172.21.15.31:0/35953726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3240 msgr2=0x5586167eb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.543+0000 7f09709ad640 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3240 0x5586167eb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.543+0000 7f09709ad640 1 -- 172.21.15.31:0/35953726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 msgr2=0x5586167ebbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.543+0000 7f09709ad640 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167ebbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.543+0000 7f09709ad640 1 -- 172.21.15.31:0/35953726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558616528620 con 0x5586167e3b60 2024-04-25T02:13:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.543+0000 7f09719af640 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167ebbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.543+0000 7f09709ad640 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167eb6b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f095c0ccde0 tx=0x7f095c0d3040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee82541ca2baac86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.543+0000 7f0954ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:13:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.543+0000 7f0954ff9640 1 -- 172.21.15.31:0/35953726 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c09b690 con 0x5586167e3b60 2024-04-25T02:13:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.543+0000 7f09761f9740 1 -- 172.21.15.31:0/35953726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0968031860 con 0x5586167e3b60 2024-04-25T02:13:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.543+0000 7f0954ff9640 1 -- 172.21.15.31:0/35953726 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7f095c0a26f0 con 0x5586167e3b60 2024-04-25T02:13:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.544+0000 7f0954ff9640 1 -- 172.21.15.31:0/35953726 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f095c09b4b0 con 0x5586167e3b60 2024-04-25T02:13:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.544+0000 7f09761f9740 10 client.27218 fetch_fsmap learned FSMap version 49 2024-04-25T02:13:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.544+0000 7f09761f9740 1 -- 172.21.15.31:0/35953726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5586167fd1d0 con 0x5586167e3b60 2024-04-25T02:13:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.544+0000 7f0954ff9640 1 -- 172.21.15.31:0/35953726 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0968031860 con 0x5586167e3b60 2024-04-25T02:13:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.544+0000 7f0954ff9640 1 -- 172.21.15.31:0/35953726 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f095c0ba2e0 con 0x5586167e3b60 2024-04-25T02:13:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.544+0000 7f09761f9740 10 client.27218 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:13:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.544+0000 7f09761f9740 10 client.27218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:13:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.544+0000 7f09761f9740 20 client.27218 populate_metadata read hostname 'smithi031' 2024-04-25T02:13:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.544+0000 7f09761f9740 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5586167fe6e0 0x558616800ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.544+0000 7f09761f9740 4 client.27218 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:13:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.544+0000 7f09761f9740 1 -- 172.21.15.31:0/35953726 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586167551f0 con 0x5586167fe6e0 2024-04-25T02:13:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.545+0000 7f09711ae640 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5586167fe6e0 0x558616800ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.545+0000 7f0954ff9640 10 client.27218 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:29.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.546+0000 7f09711ae640 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5586167fe6e0 0x558616800ad0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c05350143a17554 server_cookie=84172aa409869271 in_seq=0 out_seq=0 2024-04-25T02:13:29.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.546+0000 7f0954ff9640 1 -- 172.21.15.31:0/35953726 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5586167551f0 con 0x5586167fe6e0 2024-04-25T02:13:29.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f0954ff9640 10 client.27218 handle_command_reply: tid=0 2024-04-25T02:13:29.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 1 client.27218 shutdown 2024-04-25T02:13:29.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 1 -- 172.21.15.31:0/35953726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 msgr2=0x5586167eb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:29.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167eb6b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f095c0ccde0 tx=0x7f095c0d3040 comp rx=0 tx=0).stop 2024-04-25T02:13:29.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 1 -- 172.21.15.31:0/35953726 shutdown_connections 2024-04-25T02:13:29.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5586167fe6e0 0x558616800ad0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167ebbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f0954ff9640 0 client.27218 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:29.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3b60 0x5586167eb6b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 1 --2- 172.21.15.31:0/35953726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3240 0x5586167eb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 1 -- 172.21.15.31:0/35953726 >> 172.21.15.31:0/35953726 conn(0x5586167df560 msgr2=0x5586167e5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 1 -- 172.21.15.31:0/35953726 shutdown_connections 2024-04-25T02:13:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 1 -- 172.21.15.31:0/35953726 wait complete. 2024-04-25T02:13:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.547+0000 7f09761f9740 20 client.27218 trim_cache size 0 max 16384 2024-04-25T02:13:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.573+0000 7f09761f9740 1 Processor -- start 2024-04-25T02:13:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.573+0000 7f09761f9740 1 -- start start 2024-04-25T02:13:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.573+0000 7f09761f9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.573+0000 7f09761f9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3b60 0x5586167ea2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.573+0000 7f09761f9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167e59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.573+0000 7f09761f9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1af510 con 0x5586167ee230 2024-04-25T02:13:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.573+0000 7f09761f9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1b1050 con 0x5586167e3240 2024-04-25T02:13:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.573+0000 7f09761f9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1afbe0 con 0x5586167e3b60 2024-04-25T02:13:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f09711ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f09719af640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167e59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f09711ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38534/0 (socket says 172.21.15.31:38534) 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f09711ae640 1 -- 172.21.15.31:0/396661737 learned_addr learned my addr 172.21.15.31:0/396661737 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f09709ad640 1 --2- 172.21.15.31:0/396661737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3b60 0x5586167ea2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f09711ae640 1 -- 172.21.15.31:0/396661737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3b60 msgr2=0x5586167ea2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f09711ae640 1 --2- 172.21.15.31:0/396661737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3b60 0x5586167ea2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f09711ae640 1 -- 172.21.15.31:0/396661737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 msgr2=0x5586167e59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f09711ae640 1 --2- 172.21.15.31:0/396661737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167e59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f09711ae640 1 -- 172.21.15.31:0/396661737 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586167e3990 con 0x5586167e3240 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f09711ae640 1 --2- 172.21.15.31:0/396661737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f9e80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0968033000 tx=0x7f0968005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf63c144febf922a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f0956ffd640 1 -- 172.21.15.31:0/396661737 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0968056070 con 0x5586167e3240 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f0956ffd640 1 -- 172.21.15.31:0/396661737 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09680500d0 con 0x5586167e3240 2024-04-25T02:13:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.574+0000 7f0956ffd640 1 -- 172.21.15.31:0/396661737 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0968035040 con 0x5586167e3240 2024-04-25T02:13:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.575+0000 7f09761f9740 1 -- 172.21.15.31:0/396661737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 msgr2=0x5586167f9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.575+0000 7f09761f9740 1 --2- 172.21.15.31:0/396661737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f9e80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0968033000 tx=0x7f0968005e80 comp rx=0 tx=0).stop 2024-04-25T02:13:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 -- 172.21.15.31:0/396661737 shutdown_connections 2024-04-25T02:13:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 --2- 172.21.15.31:0/396661737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167e59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 --2- 172.21.15.31:0/396661737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3b60 0x5586167ea2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 --2- 172.21.15.31:0/396661737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f9e80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 -- 172.21.15.31:0/396661737 >> 172.21.15.31:0/396661737 conn(0x5586167df560 msgr2=0x5586167ddd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 -- 172.21.15.31:0/396661737 shutdown_connections 2024-04-25T02:13:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 -- 172.21.15.31:0/396661737 wait complete. 2024-04-25T02:13:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 -- start start 2024-04-25T02:13:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 Processor -- start 2024-04-25T02:13:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3b60 0x5586167e79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167e7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1b0980 con 0x5586167ee230 2024-04-25T02:13:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1b02b0 con 0x5586167e3240 2024-04-25T02:13:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.576+0000 7f09761f9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1ad510 con 0x5586167e3b60 2024-04-25T02:13:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09711ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09719af640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167e7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09711ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38550/0 (socket says 172.21.15.31:38550) 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09709ad640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3b60 0x5586167e79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09711ae640 1 -- 172.21.15.31:0/2741623643 learned_addr learned my addr 172.21.15.31:0/2741623643 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09761f9740 10 client.0 _ll_register_callbacks cb 0x7f095c00ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09709ad640 1 -- 172.21.15.31:0/2741623643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 msgr2=0x5586167f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09709ad640 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09709ad640 1 -- 172.21.15.31:0/2741623643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 msgr2=0x5586167e7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09709ad640 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167e7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09709ad640 1 -- 172.21.15.31:0/2741623643 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586167e8490 con 0x5586167e3b60 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09719af640 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167e7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09711ae640 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09709ad640 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3b60 0x5586167e79e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f095c00ea30 tx=0x7f095c003e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d8f6ea2311633d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f0954ff9640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f0954ff9640 1 -- 172.21.15.31:0/2741623643 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c09b690 con 0x5586167e3b60 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f0954ff9640 1 -- 172.21.15.31:0/2741623643 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7f095c0d1d90 con 0x5586167e3b60 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.577+0000 7f09761f9740 1 -- 172.21.15.31:0/2741623643 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09680500d0 con 0x5586167e3b60 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f0954ff9640 1 -- 172.21.15.31:0/2741623643 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f095c003b30 con 0x5586167e3b60 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f09761f9740 10 client.27010 fetch_fsmap learned FSMap version 49 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f09761f9740 1 -- 172.21.15.31:0/2741623643 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5586168649c0 con 0x5586167e3b60 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f0954ff9640 1 -- 172.21.15.31:0/2741623643 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f095c09c250 con 0x5586167e3b60 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f0954ff9640 1 -- 172.21.15.31:0/2741623643 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f095c0978a0 con 0x5586167e3b60 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f09761f9740 10 client.27010 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f09761f9740 10 client.27010 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f09761f9740 20 client.27010 populate_metadata read hostname 'smithi031' 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f09761f9740 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558616865120 0x558616867510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f09761f9740 4 client.27010 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:13:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f09761f9740 1 -- 172.21.15.31:0/2741623643 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f09680500d0 con 0x558616865120 2024-04-25T02:13:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.578+0000 7f09711ae640 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558616865120 0x558616867510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.579+0000 7f0954ff9640 10 client.27010 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.579+0000 7f09711ae640 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558616865120 0x558616867510 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96c17fee98686c9f server_cookie=39d71d764c42e662 in_seq=0 out_seq=0 2024-04-25T02:13:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f0954ff9640 1 -- 172.21.15.31:0/2741623643 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f09680500d0 con 0x558616865120 2024-04-25T02:13:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f0954ff9640 10 client.27010 handle_command_reply: tid=0 2024-04-25T02:13:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 1 client.27010 shutdown 2024-04-25T02:13:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 1 -- 172.21.15.31:0/2741623643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3b60 msgr2=0x5586167e79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3b60 0x5586167e79e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f095c00ea30 tx=0x7f095c003e30 comp rx=0 tx=0).stop 2024-04-25T02:13:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 1 -- 172.21.15.31:0/2741623643 shutdown_connections 2024-04-25T02:13:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x558616865120 0x558616867510 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586167ee230 0x5586167e7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5586167e3b60 0x5586167e79e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 1 --2- 172.21.15.31:0/2741623643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5586167e3240 0x5586167f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f0954ff9640 0 client.27010 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 1 -- 172.21.15.31:0/2741623643 >> 172.21.15.31:0/2741623643 conn(0x5586167df560 msgr2=0x5586167de5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 1 -- 172.21.15.31:0/2741623643 shutdown_connections 2024-04-25T02:13:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 1 -- 172.21.15.31:0/2741623643 wait complete. 2024-04-25T02:13:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.647+0000 7f09761f9740 20 client.27010 trim_cache size 0 max 16384 2024-04-25T02:13:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:13:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:13:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:13:29.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:13:29.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 -- 172.21.15.31:0/2386344423 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09580a2e50 msgr2=0x7f09580a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:29.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09580a2e50 0x7f09580a5310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0964005c40 tx=0x7f0964004a40 comp rx=0 tx=0).stop 2024-04-25T02:13:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 -- 172.21.15.31:0/2386344423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c1032f0 msgr2=0x7f096c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c1032f0 0x7f096c100b60 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f095c00ea60 tx=0x7f095c00ef20 comp rx=0 tx=0).stop 2024-04-25T02:13:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 -- 172.21.15.31:0/2386344423 shutdown_connections 2024-04-25T02:13:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f09580a2e50 0x7f09580a5310 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c10bd10 0x7f096c1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c1032f0 0x7f096c100b60 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 --2- 172.21.15.31:0/2386344423 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f096c1029d0 0x7f096c100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 -- 172.21.15.31:0/2386344423 >> 172.21.15.31:0/2386344423 conn(0x7f096c0777c0 msgr2=0x7f096c0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 -- 172.21.15.31:0/2386344423 shutdown_connections 2024-04-25T02:13:29.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:29.648+0000 7f0973412640 1 -- 172.21.15.31:0/2386344423 wait complete. 2024-04-25T02:13:30.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 55530 9.00 MB/sec execute 214 sec latency 81.115 ms 2024-04-25T02:13:30.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:29 smithi031 ceph-mon[29204]: pgmap v598: 97 pgs: 97 active+clean; 964 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-04-25T02:13:30.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:29 smithi040 ceph-mon[32418]: pgmap v598: 97 pgs: 97 active+clean; 964 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-04-25T02:13:30.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:29 smithi134 ceph-mon[33098]: pgmap v598: 97 pgs: 97 active+clean; 964 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-04-25T02:13:31.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 55676 8.99 MB/sec execute 215 sec latency 58.102 ms 2024-04-25T02:13:32.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 55815 8.97 MB/sec execute 216 sec latency 69.956 ms 2024-04-25T02:13:32.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:31 smithi031 ceph-mon[29204]: pgmap v599: 97 pgs: 97 active+clean; 970 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.4 MiB/s wr, 352 op/s 2024-04-25T02:13:32.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:31 smithi040 ceph-mon[32418]: pgmap v599: 97 pgs: 97 active+clean; 970 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.4 MiB/s wr, 352 op/s 2024-04-25T02:13:32.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:31 smithi134 ceph-mon[33098]: pgmap v599: 97 pgs: 97 active+clean; 970 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.4 MiB/s wr, 352 op/s 2024-04-25T02:13:33.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 55986 8.97 MB/sec execute 217 sec latency 65.585 ms 2024-04-25T02:13:34.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 56156 8.96 MB/sec execute 218 sec latency 162.641 ms 2024-04-25T02:13:34.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:33 smithi031 ceph-mon[29204]: pgmap v600: 97 pgs: 97 active+clean; 970 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 232 op/s 2024-04-25T02:13:34.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:33 smithi040 ceph-mon[32418]: pgmap v600: 97 pgs: 97 active+clean; 970 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 232 op/s 2024-04-25T02:13:34.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:33 smithi134 ceph-mon[33098]: pgmap v600: 97 pgs: 97 active+clean; 970 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 232 op/s 2024-04-25T02:13:35.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 56288 8.94 MB/sec execute 219 sec latency 68.120 ms 2024-04-25T02:13:36.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 56436 8.94 MB/sec execute 220 sec latency 101.604 ms 2024-04-25T02:13:36.243 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:35 smithi031 ceph-mon[29204]: pgmap v601: 97 pgs: 97 active+clean; 989 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.5 MiB/s wr, 339 op/s 2024-04-25T02:13:36.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:35 smithi040 ceph-mon[32418]: pgmap v601: 97 pgs: 97 active+clean; 989 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.5 MiB/s wr, 339 op/s 2024-04-25T02:13:36.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:35 smithi134 ceph-mon[33098]: pgmap v601: 97 pgs: 97 active+clean; 989 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.5 MiB/s wr, 339 op/s 2024-04-25T02:13:37.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 56646 8.99 MB/sec execute 221 sec latency 60.802 ms 2024-04-25T02:13:38.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 56836 9.00 MB/sec execute 222 sec latency 53.251 ms 2024-04-25T02:13:38.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:37 smithi040 ceph-mon[32418]: pgmap v602: 97 pgs: 97 active+clean; 989 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-04-25T02:13:38.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:37 smithi134 ceph-mon[33098]: pgmap v602: 97 pgs: 97 active+clean; 989 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-04-25T02:13:38.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:37 smithi031 ceph-mon[29204]: pgmap v602: 97 pgs: 97 active+clean; 989 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-04-25T02:13:39.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 56963 8.98 MB/sec execute 223 sec latency 62.659 ms 2024-04-25T02:13:40.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 57108 8.96 MB/sec execute 224 sec latency 89.882 ms 2024-04-25T02:13:40.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:39 smithi040 ceph-mon[32418]: pgmap v603: 97 pgs: 97 active+clean; 989 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-04-25T02:13:40.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:39 smithi134 ceph-mon[33098]: pgmap v603: 97 pgs: 97 active+clean; 989 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-04-25T02:13:40.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:39 smithi031 ceph-mon[29204]: pgmap v603: 97 pgs: 97 active+clean; 989 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-04-25T02:13:41.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 57259 8.95 MB/sec execute 225 sec latency 62.762 ms 2024-04-25T02:13:42.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 57419 8.92 MB/sec execute 226 sec latency 76.799 ms 2024-04-25T02:13:42.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:41 smithi040 ceph-mon[32418]: pgmap v604: 97 pgs: 97 active+clean; 988 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 357 op/s 2024-04-25T02:13:42.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:41 smithi134 ceph-mon[33098]: pgmap v604: 97 pgs: 97 active+clean; 988 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 357 op/s 2024-04-25T02:13:42.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:41 smithi031 ceph-mon[29204]: pgmap v604: 97 pgs: 97 active+clean; 988 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 357 op/s 2024-04-25T02:13:42.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.778+0000 7f5fe159a640 1 -- 172.21.15.31:0/2494356416 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80035f0 con 0x7f5fe4106630 2024-04-25T02:13:42.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.778+0000 7f5fe8914640 1 -- 172.21.15.31:0/2494356416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fe4106630 msgr2=0x7f5fe4106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.778+0000 7f5fe8914640 1 --2- 172.21.15.31:0/2494356416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fe4106630 0x7f5fe4106a10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009900 tx=0x7f5fd802a440 comp rx=0 tx=0).stop 2024-04-25T02:13:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.779+0000 7f5fe8914640 1 -- 172.21.15.31:0/2494356416 shutdown_connections 2024-04-25T02:13:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.779+0000 7f5fe8914640 1 --2- 172.21.15.31:0/2494356416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fe4101030 0x7f5fe410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.779+0000 7f5fe8914640 1 --2- 172.21.15.31:0/2494356416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe4100670 0x7f5fe4100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.779+0000 7f5fe8914640 1 --2- 172.21.15.31:0/2494356416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fe4106630 0x7f5fe4106a10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.779+0000 7f5fe8914640 1 -- 172.21.15.31:0/2494356416 >> 172.21.15.31:0/2494356416 conn(0x7f5fe40777c0 msgr2=0x7f5fe4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.779+0000 7f5fe8914640 1 -- 172.21.15.31:0/2494356416 shutdown_connections 2024-04-25T02:13:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.779+0000 7f5fe8914640 1 -- 172.21.15.31:0/2494356416 wait complete. 2024-04-25T02:13:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.780+0000 7f5fe8914640 1 Processor -- start 2024-04-25T02:13:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.780+0000 7f5fe8914640 1 -- start start 2024-04-25T02:13:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.780+0000 7f5fe8914640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe4100670 0x7f5fe419b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.780+0000 7f5fe8914640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fe4101030 0x7f5fe419bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.780+0000 7f5fe8914640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fe4106630 0x7f5fe4195a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.780+0000 7f5fe8914640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe410f830 con 0x7f5fe4100670 2024-04-25T02:13:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.780+0000 7f5fe8914640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe410f6b0 con 0x7f5fe4106630 2024-04-25T02:13:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.780+0000 7f5fe8914640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe410f9b0 con 0x7f5fe4101030 2024-04-25T02:13:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.781+0000 7f5fe1d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fe4101030 0x7f5fe419bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.781+0000 7f5fe1d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fe4101030 0x7f5fe419bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:57594/0 (socket says 172.21.15.31:57594) 2024-04-25T02:13:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.781+0000 7f5fe1d9b640 1 -- 172.21.15.31:0/3601407627 learned_addr learned my addr 172.21.15.31:0/3601407627 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.781+0000 7f5fe259c640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe4100670 0x7f5fe419b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.781+0000 7f5fe1d9b640 1 -- 172.21.15.31:0/3601407627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fe4106630 msgr2=0x7f5fe4195a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.781+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fe4106630 0x7f5fe4195a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.781+0000 7f5fe1d9b640 1 -- 172.21.15.31:0/3601407627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe4100670 msgr2=0x7f5fe419b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.781+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe4100670 0x7f5fe419b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.781+0000 7f5fe1d9b640 1 -- 172.21.15.31:0/3601407627 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe41961f0 con 0x7f5fe4101030 2024-04-25T02:13:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.781+0000 7f5fe259c640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe4100670 0x7f5fe419b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:13:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.782+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fe4101030 0x7f5fe419bdb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00b6d0 tx=0x7f5fdc00bb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:42.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.782+0000 7f5fd37fe640 1 -- 172.21.15.31:0/3601407627 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc023070 con 0x7f5fe4101030 2024-04-25T02:13:42.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.782+0000 7f5fe8914640 1 -- 172.21.15.31:0/3601407627 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe41964e0 con 0x7f5fe4101030 2024-04-25T02:13:42.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.782+0000 7f5fe8914640 1 -- 172.21.15.31:0/3601407627 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe41a2730 con 0x7f5fe4101030 2024-04-25T02:13:42.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.783+0000 7f5fd37fe640 1 -- 172.21.15.31:0/3601407627 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fdc008cc0 con 0x7f5fe4101030 2024-04-25T02:13:42.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.783+0000 7f5fd37fe640 1 -- 172.21.15.31:0/3601407627 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc01eb60 con 0x7f5fe4101030 2024-04-25T02:13:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.786+0000 7f5feb626740 1 Processor -- start 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.786+0000 7f5feb626740 1 -- start start 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5feb626740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd1a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5feb626740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd24cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5feb626740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd27620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5feb626740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ca7cc7ecb0 con 0x55ca7cd25230 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5feb626740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ca7cc275b0 con 0x55ca7cd1a240 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5feb626740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ca7cc7ac70 con 0x55ca7cd1ab60 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5fe2d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd1a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5fe2d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd1a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39136/0 (socket says 172.21.15.31:39136) 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/2595122316 learned_addr learned my addr 172.21.15.31:0/2595122316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/2595122316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd27620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/2595122316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 msgr2=0x55ca7cd24cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5fe2d9d640 1 --2- 172.21.15.31:0/2595122316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd24cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.787+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/2595122316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 msgr2=0x55ca7cd27620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.788+0000 7f5fe2d9d640 1 --2- 172.21.15.31:0/2595122316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd27620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.788+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/2595122316 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca7c993210 con 0x55ca7cd1a240 2024-04-25T02:13:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.788+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/2595122316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd27620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.788+0000 7f5fe2d9d640 1 --2- 172.21.15.31:0/2595122316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd1a620 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400ba90 tx=0x7f5fd400bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f797c7b83d16f4f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.789+0000 7f5fd0ff9640 1 -- 172.21.15.31:0/2595122316 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4018030 con 0x55ca7cd1a240 2024-04-25T02:13:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.789+0000 7f5fd0ff9640 1 -- 172.21.15.31:0/2595122316 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fd4005020 con 0x55ca7cd1a240 2024-04-25T02:13:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.789+0000 7f5fd0ff9640 1 -- 172.21.15.31:0/2595122316 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4004d00 con 0x55ca7cd1a240 2024-04-25T02:13:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.790+0000 7f5feb626740 1 -- 172.21.15.31:0/2595122316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 msgr2=0x55ca7cd1a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.790+0000 7f5feb626740 1 --2- 172.21.15.31:0/2595122316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd1a620 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400ba90 tx=0x7f5fd400bf50 comp rx=0 tx=0).stop 2024-04-25T02:13:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.790+0000 7f5fd37fe640 1 -- 172.21.15.31:0/3601407627 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5fdc026070 con 0x7f5fe4101030 2024-04-25T02:13:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.791+0000 7f5feb626740 1 -- 172.21.15.31:0/2595122316 shutdown_connections 2024-04-25T02:13:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.791+0000 7f5feb626740 1 --2- 172.21.15.31:0/2595122316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd27620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.791+0000 7f5feb626740 1 --2- 172.21.15.31:0/2595122316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd24cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.791+0000 7f5feb626740 1 --2- 172.21.15.31:0/2595122316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd1a620 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.791+0000 7f5feb626740 1 -- 172.21.15.31:0/2595122316 >> 172.21.15.31:0/2595122316 conn(0x55ca7cd16560 msgr2=0x55ca7cd16940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.792+0000 7f5fd37fe640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5fcc0a2fd0 0x7f5fcc0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.792+0000 7f5fd37fe640 1 -- 172.21.15.31:0/3601407627 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7f5fdc0de3c0 con 0x7f5fe4101030 2024-04-25T02:13:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.792+0000 7f5feb626740 1 -- 172.21.15.31:0/2595122316 shutdown_connections 2024-04-25T02:13:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.793+0000 7f5feb626740 1 -- 172.21.15.31:0/2595122316 wait complete. 2024-04-25T02:13:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.793+0000 7f5fe259c640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5fcc0a2fd0 0x7f5fcc0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.793+0000 7f5feb626740 1 -- start start 2024-04-25T02:13:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.793+0000 7f5feb626740 1 Processor -- start 2024-04-25T02:13:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.793+0000 7f5feb626740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd22150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.793+0000 7f5feb626740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd22690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5feb626740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd22bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5feb626740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ca7cd39030 con 0x55ca7cd25230 2024-04-25T02:13:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5feb626740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ca7cc3ab50 con 0x55ca7cd1a240 2024-04-25T02:13:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5feb626740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ca7cd391b0 con 0x55ca7cd1ab60 2024-04-25T02:13:42.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5feb626740 10 client.0 _ll_register_callbacks cb 0x55ca7cc75ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:13:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5fe2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd22bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5fe2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd22bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56156/0 (socket says 172.21.15.31:56156) 2024-04-25T02:13:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/917971048 learned_addr learned my addr 172.21.15.31:0/917971048 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd22690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5fe259c640 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd22150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/917971048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 msgr2=0x55ca7cd22690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5fe2d9d640 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd22690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/917971048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 msgr2=0x55ca7cd22150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5fe2d9d640 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd22150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.794+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/917971048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca7ca5f620 con 0x55ca7cd25230 2024-04-25T02:13:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.795+0000 7f5fe259c640 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd22150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:13:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.795+0000 7f5fe2d9d640 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd22bd0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4004700 tx=0x7f5fd4007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db564e6a11165a1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.795+0000 7f5fc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.795+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd22690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.795+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/917971048 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4004030 con 0x55ca7cd25230 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.795+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/917971048 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7f5fd40041d0 con 0x55ca7cd25230 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.795+0000 7f5feb626740 1 -- 172.21.15.31:0/917971048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fd4005020 con 0x55ca7cd25230 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.795+0000 7f5fe259c640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5fcc0a2fd0 0x7f5fcc0a5490 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4104df0 tx=0x7f5fd8004870 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/917971048 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f5fd402e050 con 0x55ca7cd25230 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5feb626740 10 client.17676 fetch_fsmap learned FSMap version 49 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5feb626740 1 -- 172.21.15.31:0/917971048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55ca7cd45220 con 0x55ca7cd25230 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/917971048 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fd4022020 con 0x55ca7cd25230 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/917971048 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f5fd4003ac0 con 0x55ca7cd25230 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5feb626740 10 client.17676 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5feb626740 10 client.17676 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5feb626740 20 client.17676 populate_metadata read hostname 'smithi031' 2024-04-25T02:13:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5feb626740 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ca7cd46630 0x55ca7cd48a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5feb626740 4 client.17676 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:13:42.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5feb626740 1 -- 172.21.15.31:0/917971048 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca7cc8c1f0 con 0x55ca7cd46630 2024-04-25T02:13:42.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.796+0000 7f5fe259c640 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ca7cd46630 0x55ca7cd48a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.797+0000 7f5fe259c640 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ca7cd46630 0x55ca7cd48a20 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8debb0eba1ac03b1 server_cookie=5b451a09be4e091c in_seq=0 out_seq=0 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.797+0000 7f5fc9ffb640 10 client.17676 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.798+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/917971048 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ca7cc8c1f0 con 0x55ca7cd46630 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.798+0000 7f5fc9ffb640 10 client.17676 handle_command_reply: tid=0 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.798+0000 7f5feb626740 1 client.17676 shutdown 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.798+0000 7f5feb626740 1 -- 172.21.15.31:0/917971048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 msgr2=0x55ca7cd22bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.798+0000 7f5feb626740 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd22bd0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4004700 tx=0x7f5fd4007d20 comp rx=0 tx=0).stop 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.798+0000 7f5feb626740 1 -- 172.21.15.31:0/917971048 shutdown_connections 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.798+0000 7f5feb626740 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ca7cd46630 0x55ca7cd48a20 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.798+0000 7f5feb626740 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd22bd0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.798+0000 7f5feb626740 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd22690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.798+0000 7f5feb626740 1 --2- 172.21.15.31:0/917971048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd22150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.798+0000 7f5feb626740 1 -- 172.21.15.31:0/917971048 >> 172.21.15.31:0/917971048 conn(0x55ca7cd16560 msgr2=0x55ca7cd1c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:42.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.799+0000 7f5fc9ffb640 0 client.17676 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:42.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.799+0000 7f5feb626740 1 -- 172.21.15.31:0/917971048 shutdown_connections 2024-04-25T02:13:42.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.799+0000 7f5feb626740 1 -- 172.21.15.31:0/917971048 wait complete. 2024-04-25T02:13:42.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.799+0000 7f5feb626740 20 client.17676 trim_cache size 0 max 16384 2024-04-25T02:13:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.821+0000 7f5feb626740 1 Processor -- start 2024-04-25T02:13:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.821+0000 7f5feb626740 1 -- start start 2024-04-25T02:13:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5feb626740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd41f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5feb626740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd212b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5feb626740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd15440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5feb626740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc400e790 con 0x55ca7cd25230 2024-04-25T02:13:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5feb626740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc40102d0 con 0x55ca7cd1a240 2024-04-25T02:13:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5feb626740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc400ee60 con 0x55ca7cd1ab60 2024-04-25T02:13:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5fe2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd15440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5fe259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd41f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5fe2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd15440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56166/0 (socket says 172.21.15.31:56166) 2024-04-25T02:13:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/430602806 learned_addr learned my addr 172.21.15.31:0/430602806 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/430602806 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd212b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/430602806 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 msgr2=0x55ca7cd212b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5fe2d9d640 1 --2- 172.21.15.31:0/430602806 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd212b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/430602806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 msgr2=0x55ca7cd41f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5fe2d9d640 1 --2- 172.21.15.31:0/430602806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd41f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5fe2d9d640 1 -- 172.21.15.31:0/430602806 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca7cd108b0 con 0x55ca7cd25230 2024-04-25T02:13:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.822+0000 7f5fe2d9d640 1 --2- 172.21.15.31:0/430602806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd15440 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4000c00 tx=0x7f5fd401d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d86cd1a63eefae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.823+0000 7f5fcbfff640 1 -- 172.21.15.31:0/430602806 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4037070 con 0x55ca7cd25230 2024-04-25T02:13:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.823+0000 7f5fcbfff640 1 -- 172.21.15.31:0/430602806 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fd4022050 con 0x55ca7cd25230 2024-04-25T02:13:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.823+0000 7f5fcbfff640 1 -- 172.21.15.31:0/430602806 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4018040 con 0x55ca7cd25230 2024-04-25T02:13:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.823+0000 7f5feb626740 1 -- 172.21.15.31:0/430602806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 msgr2=0x55ca7cd15440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.823+0000 7f5feb626740 1 --2- 172.21.15.31:0/430602806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd15440 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4000c00 tx=0x7f5fd401d5f0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.824+0000 7f5feb626740 1 -- 172.21.15.31:0/430602806 shutdown_connections 2024-04-25T02:13:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.824+0000 7f5feb626740 1 --2- 172.21.15.31:0/430602806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cd15440 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.824+0000 7f5feb626740 1 --2- 172.21.15.31:0/430602806 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd212b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.824+0000 7f5feb626740 1 --2- 172.21.15.31:0/430602806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd41f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.824+0000 7f5feb626740 1 -- 172.21.15.31:0/430602806 >> 172.21.15.31:0/430602806 conn(0x55ca7cd16560 msgr2=0x55ca7cd14d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.824+0000 7f5feb626740 1 -- 172.21.15.31:0/430602806 shutdown_connections 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.824+0000 7f5feb626740 1 -- 172.21.15.31:0/430602806 wait complete. 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.824+0000 7f5feb626740 1 -- start start 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.824+0000 7f5feb626740 1 Processor -- start 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5feb626740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd39750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5feb626740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd39c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5feb626740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cda4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5fe259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd39750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5feb626740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc400fc00 con 0x55ca7cd25230 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5fe259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd39750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:39160/0 (socket says 172.21.15.31:39160) 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5fe259c640 1 -- 172.21.15.31:0/2598691578 learned_addr learned my addr 172.21.15.31:0/2598691578 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd39c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5feb626740 1 -- 172.21.15.31:0/2598691578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc400f530 con 0x55ca7cd1a240 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5feb626740 1 -- 172.21.15.31:0/2598691578 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc400c790 con 0x55ca7cd1ab60 2024-04-25T02:13:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5fe2d9d640 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cda4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.825+0000 7f5feb626740 10 client.0 _ll_register_callbacks cb 0x7f5fdc02b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5fe1d9b640 1 -- 172.21.15.31:0/2598691578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 msgr2=0x55ca7cd39750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd39750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5fe1d9b640 1 -- 172.21.15.31:0/2598691578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 msgr2=0x55ca7cda4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cda4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5fe1d9b640 1 -- 172.21.15.31:0/2598691578 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca7cda4ec0 con 0x55ca7cd1ab60 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5fe2d9d640 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cda4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5fe259c640 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd39750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5fe1d9b640 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd39c90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0dbad0 tx=0x7f5fdc00de80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d5c86f435d03b13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5fc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/2598691578 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc0aa200 con 0x55ca7cd1ab60 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5feb626740 1 -- 172.21.15.31:0/2598691578 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fd4022050 con 0x55ca7cd1ab60 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.826+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/2598691578 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7f5fdc0abdd0 con 0x55ca7cd1ab60 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.827+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/2598691578 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f5fdc0a89e0 con 0x55ca7cd1ab60 2024-04-25T02:13:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.827+0000 7f5feb626740 10 client.27025 fetch_fsmap learned FSMap version 49 2024-04-25T02:13:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.827+0000 7f5feb626740 1 -- 172.21.15.31:0/2598691578 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55ca7cdac2c0 con 0x55ca7cd1ab60 2024-04-25T02:13:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.827+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/2598691578 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fdc0aafa0 con 0x55ca7cd1ab60 2024-04-25T02:13:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.827+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/2598691578 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f5fdc01e410 con 0x55ca7cd1ab60 2024-04-25T02:13:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.827+0000 7f5feb626740 10 client.27025 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:13:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.827+0000 7f5feb626740 10 client.27025 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:13:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.827+0000 7f5feb626740 20 client.27025 populate_metadata read hostname 'smithi031' 2024-04-25T02:13:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.828+0000 7f5feb626740 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ca7cdaca90 0x55ca7cdaeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.828+0000 7f5feb626740 4 client.27025 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:13:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.828+0000 7f5feb626740 1 -- 172.21.15.31:0/2598691578 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5fd4022050 con 0x55ca7cdaca90 2024-04-25T02:13:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.828+0000 7f5fe259c640 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ca7cdaca90 0x55ca7cdaeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.828+0000 7f5fe259c640 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ca7cdaca90 0x55ca7cdaeeb0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7702c468d152929 server_cookie=ae9defbacb56bd5c in_seq=0 out_seq=0 2024-04-25T02:13:42.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.828+0000 7f5fc9ffb640 10 client.27025 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5fc9ffb640 1 -- 172.21.15.31:0/2598691578 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5fd4022050 con 0x55ca7cdaca90 2024-04-25T02:13:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5fc9ffb640 10 client.27025 handle_command_reply: tid=0 2024-04-25T02:13:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5feb626740 1 client.27025 shutdown 2024-04-25T02:13:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5feb626740 1 -- 172.21.15.31:0/2598691578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 msgr2=0x55ca7cd39c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5feb626740 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd39c90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0dbad0 tx=0x7f5fdc00de80 comp rx=0 tx=0).stop 2024-04-25T02:13:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5feb626740 1 -- 172.21.15.31:0/2598691578 shutdown_connections 2024-04-25T02:13:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5feb626740 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ca7cdaca90 0x55ca7cdaeeb0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5feb626740 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca7cd25230 0x55ca7cda4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5fc9ffb640 0 client.27025 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5feb626740 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ca7cd1ab60 0x55ca7cd39c90 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0dbad0 tx=0x7f5fdc00de80 comp rx=0 tx=0).stop 2024-04-25T02:13:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5feb626740 1 --2- 172.21.15.31:0/2598691578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ca7cd1a240 0x55ca7cd39750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5feb626740 1 -- 172.21.15.31:0/2598691578 >> 172.21.15.31:0/2598691578 conn(0x55ca7cd16560 msgr2=0x55ca7cd22a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5feb626740 1 -- 172.21.15.31:0/2598691578 shutdown_connections 2024-04-25T02:13:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.883+0000 7f5feb626740 1 -- 172.21.15.31:0/2598691578 wait complete. 2024-04-25T02:13:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.884+0000 7f5feb626740 20 client.27025 trim_cache size 0 max 16384 2024-04-25T02:13:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:13:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:13:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:13:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:13:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.885+0000 7f5fe8914640 1 -- 172.21.15.31:0/3601407627 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5fcc0a2fd0 msgr2=0x7f5fcc0a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.885+0000 7f5fe8914640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5fcc0a2fd0 0x7f5fcc0a5490 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4104df0 tx=0x7f5fd8004870 comp rx=0 tx=0).stop 2024-04-25T02:13:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.885+0000 7f5fe8914640 1 -- 172.21.15.31:0/3601407627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fe4101030 msgr2=0x7f5fe419bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.885+0000 7f5fe8914640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fe4101030 0x7f5fe419bdb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00b6d0 tx=0x7f5fdc00bb90 comp rx=0 tx=0).stop 2024-04-25T02:13:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.886+0000 7f5fe8914640 1 -- 172.21.15.31:0/3601407627 shutdown_connections 2024-04-25T02:13:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.886+0000 7f5fe8914640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5fcc0a2fd0 0x7f5fcc0a5490 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.886+0000 7f5fe8914640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fe4106630 0x7f5fe4195a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.886+0000 7f5fe8914640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fe4101030 0x7f5fe419bdb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.886+0000 7f5fe8914640 1 --2- 172.21.15.31:0/3601407627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe4100670 0x7f5fe419b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.886+0000 7f5fe8914640 1 -- 172.21.15.31:0/3601407627 >> 172.21.15.31:0/3601407627 conn(0x7f5fe40777c0 msgr2=0x7f5fe41030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.886+0000 7f5fe8914640 1 -- 172.21.15.31:0/3601407627 shutdown_connections 2024-04-25T02:13:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:42.888+0000 7f5fe8914640 1 -- 172.21.15.31:0/3601407627 wait complete. 2024-04-25T02:13:43.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 57587 8.92 MB/sec execute 227 sec latency 121.641 ms 2024-04-25T02:13:44.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 57727 8.90 MB/sec execute 228 sec latency 60.150 ms 2024-04-25T02:13:44.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:43 smithi040 ceph-mon[32418]: pgmap v605: 97 pgs: 97 active+clean; 988 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 234 op/s 2024-04-25T02:13:44.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:43 smithi134 ceph-mon[33098]: pgmap v605: 97 pgs: 97 active+clean; 988 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 234 op/s 2024-04-25T02:13:44.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:43 smithi031 ceph-mon[29204]: pgmap v605: 97 pgs: 97 active+clean; 988 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 234 op/s 2024-04-25T02:13:45.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 57893 8.91 MB/sec execute 229 sec latency 92.597 ms 2024-04-25T02:13:46.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 58111 8.94 MB/sec execute 230 sec latency 70.363 ms 2024-04-25T02:13:46.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:45 smithi040 ceph-mon[32418]: pgmap v606: 97 pgs: 97 active+clean; 995 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 348 op/s 2024-04-25T02:13:46.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:45 smithi134 ceph-mon[33098]: pgmap v606: 97 pgs: 97 active+clean; 995 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 348 op/s 2024-04-25T02:13:46.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:45 smithi031 ceph-mon[29204]: pgmap v606: 97 pgs: 97 active+clean; 995 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 348 op/s 2024-04-25T02:13:47.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 58341 8.96 MB/sec execute 231 sec latency 61.466 ms 2024-04-25T02:13:48.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 58473 8.95 MB/sec execute 232 sec latency 70.219 ms 2024-04-25T02:13:48.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:47 smithi040 ceph-mon[32418]: pgmap v607: 97 pgs: 97 active+clean; 995 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 241 op/s 2024-04-25T02:13:48.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:47 smithi134 ceph-mon[33098]: pgmap v607: 97 pgs: 97 active+clean; 995 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 241 op/s 2024-04-25T02:13:48.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:47 smithi031 ceph-mon[29204]: pgmap v607: 97 pgs: 97 active+clean; 995 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 241 op/s 2024-04-25T02:13:49.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 58649 8.97 MB/sec execute 233 sec latency 63.686 ms 2024-04-25T02:13:50.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 58850 8.99 MB/sec execute 234 sec latency 52.025 ms 2024-04-25T02:13:50.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:50 smithi040 ceph-mon[32418]: pgmap v608: 97 pgs: 97 active+clean; 995 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 241 op/s 2024-04-25T02:13:50.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:50 smithi134 ceph-mon[33098]: pgmap v608: 97 pgs: 97 active+clean; 995 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 241 op/s 2024-04-25T02:13:50.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:50 smithi031 ceph-mon[29204]: pgmap v608: 97 pgs: 97 active+clean; 995 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 241 op/s 2024-04-25T02:13:51.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 59010 8.98 MB/sec execute 235 sec latency 52.209 ms 2024-04-25T02:13:52.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 59171 8.98 MB/sec execute 236 sec latency 54.347 ms 2024-04-25T02:13:52.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:52 smithi040 ceph-mon[32418]: pgmap v609: 97 pgs: 97 active+clean; 1009 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.2 MiB/s wr, 358 op/s 2024-04-25T02:13:52.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:52 smithi134 ceph-mon[33098]: pgmap v609: 97 pgs: 97 active+clean; 1009 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.2 MiB/s wr, 358 op/s 2024-04-25T02:13:52.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:52 smithi031 ceph-mon[29204]: pgmap v609: 97 pgs: 97 active+clean; 1009 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.2 MiB/s wr, 358 op/s 2024-04-25T02:13:53.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 59314 8.95 MB/sec execute 237 sec latency 78.284 ms 2024-04-25T02:13:54.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 59492 8.97 MB/sec execute 238 sec latency 104.207 ms 2024-04-25T02:13:54.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:54 smithi040 ceph-mon[32418]: pgmap v610: 97 pgs: 97 active+clean; 1009 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 230 op/s 2024-04-25T02:13:54.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:54 smithi134 ceph-mon[33098]: pgmap v610: 97 pgs: 97 active+clean; 1009 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 230 op/s 2024-04-25T02:13:54.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:54 smithi031 ceph-mon[29204]: pgmap v610: 97 pgs: 97 active+clean; 1009 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 230 op/s 2024-04-25T02:13:55.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 59676 8.97 MB/sec execute 239 sec latency 95.293 ms 2024-04-25T02:13:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.029+0000 7f94a971d640 1 -- 172.21.15.31:0/1857569609 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94a4102a00 msgr2=0x7f94a4102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.029+0000 7f94a971d640 1 --2- 172.21.15.31:0/1857569609 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94a4102a00 0x7f94a4102de0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9494009900 tx=0x7f949402a510 comp rx=0 tx=0).stop 2024-04-25T02:13:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.029+0000 7f94a971d640 1 -- 172.21.15.31:0/1857569609 shutdown_connections 2024-04-25T02:13:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.029+0000 7f94a971d640 1 --2- 172.21.15.31:0/1857569609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94a410bd40 0x7f94a410e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.029+0000 7f94a971d640 1 --2- 172.21.15.31:0/1857569609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94a4103320 0x7f94a410b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.029+0000 7f94a971d640 1 --2- 172.21.15.31:0/1857569609 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94a4102a00 0x7f94a4102de0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.029+0000 7f94a971d640 1 -- 172.21.15.31:0/1857569609 >> 172.21.15.31:0/1857569609 conn(0x7f94a40769c0 msgr2=0x7f94a4076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.029+0000 7f94a971d640 1 -- 172.21.15.31:0/1857569609 shutdown_connections 2024-04-25T02:13:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.029+0000 7f94a971d640 1 -- 172.21.15.31:0/1857569609 wait complete. 2024-04-25T02:13:56.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.030+0000 7f94a971d640 1 Processor -- start 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.030+0000 7f94a971d640 1 -- start start 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.030+0000 7f94a971d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94a4102a00 0x7f94a419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.030+0000 7f94a971d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94a4103320 0x7f94a419a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.030+0000 7f94a971d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94a410bd40 0x7f94a419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.030+0000 7f94a971d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41119f0 con 0x7f94a4102a00 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.030+0000 7f94a971d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4111870 con 0x7f94a410bd40 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.030+0000 7f94a971d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4111b70 con 0x7f94a4103320 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94a410bd40 0x7f94a419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94a4102a00 0x7f94a419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94a410bd40 0x7f94a419e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58416/0 (socket says 172.21.15.31:58416) 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94a4102a00 0x7f94a419a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34110/0 (socket says 172.21.15.31:34110) 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a37fe640 1 -- 172.21.15.31:0/804264756 learned_addr learned my addr 172.21.15.31:0/804264756 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a27fc640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94a4103320 0x7f94a419a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a37fe640 1 -- 172.21.15.31:0/804264756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94a4103320 msgr2=0x7f94a419a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a37fe640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94a4103320 0x7f94a419a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a37fe640 1 -- 172.21.15.31:0/804264756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94a4102a00 msgr2=0x7f94a419a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a37fe640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94a4102a00 0x7f94a419a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a37fe640 1 -- 172.21.15.31:0/804264756 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a419f060 con 0x7f94a410bd40 2024-04-25T02:13:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f94a37fe640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94a410bd40 0x7f94a419e930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f949c00eb10 tx=0x7f949c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.031+0000 7f948bfff640 1 -- 172.21.15.31:0/804264756 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c004b20 con 0x7f94a410bd40 2024-04-25T02:13:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.032+0000 7f948bfff640 1 -- 172.21.15.31:0/804264756 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f949c004cc0 con 0x7f94a410bd40 2024-04-25T02:13:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.032+0000 7f948bfff640 1 -- 172.21.15.31:0/804264756 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c010690 con 0x7f94a410bd40 2024-04-25T02:13:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.032+0000 7f94a971d640 1 -- 172.21.15.31:0/804264756 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a419f350 con 0x7f94a410bd40 2024-04-25T02:13:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.032+0000 7f94a971d640 1 -- 172.21.15.31:0/804264756 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a41a6be0 con 0x7f94a410bd40 2024-04-25T02:13:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.034+0000 7f948bfff640 1 -- 172.21.15.31:0/804264756 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f949c005020 con 0x7f94a410bd40 2024-04-25T02:13:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94ac422740 1 Processor -- start 2024-04-25T02:13:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94ac422740 1 -- start start 2024-04-25T02:13:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94ac422740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557988400240 0x557988400620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94ac422740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 0x55798840acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94ac422740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55798840b230 0x55798840d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94ac422740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557988364cb0 con 0x55798840b230 2024-04-25T02:13:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94ac422740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55798830d5b0 con 0x557988400b60 2024-04-25T02:13:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94ac422740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557988360c70 con 0x557988400240 2024-04-25T02:13:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94a27fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557988400240 0x557988400620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94a27fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557988400240 0x557988400620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:57464/0 (socket says 172.21.15.31:57464) 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94a27fc640 1 -- 172.21.15.31:0/1106852111 learned_addr learned my addr 172.21.15.31:0/1106852111 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94a27fc640 1 --2- 172.21.15.31:0/1106852111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55798840b230 0x55798840d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f948bfff640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f948c0a2da0 0x7f948c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.035+0000 7f94a2ffd640 1 --2- 172.21.15.31:0/1106852111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 0x55798840acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94a37fe640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f948c0a2da0 0x7f948c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94a27fc640 1 -- 172.21.15.31:0/1106852111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557988400240 msgr2=0x557988400620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94a27fc640 1 --2- 172.21.15.31:0/1106852111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557988400240 0x557988400620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94a27fc640 1 -- 172.21.15.31:0/1106852111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 msgr2=0x55798840acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94a27fc640 1 --2- 172.21.15.31:0/1106852111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 0x55798840acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94a27fc640 1 -- 172.21.15.31:0/1106852111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557988079210 con 0x55798840b230 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94a2ffd640 1 --2- 172.21.15.31:0/1106852111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 0x55798840acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f948bfff640 1 -- 172.21.15.31:0/804264756 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7f949c0cf940 con 0x7f94a410bd40 2024-04-25T02:13:56.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94a27fc640 1 --2- 172.21.15.31:0/1106852111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55798840b230 0x55798840d620 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f94980058c0 tx=0x7f9498005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f14d26903b4b74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:56.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94897fa640 1 -- 172.21.15.31:0/1106852111 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498011e30 con 0x55798840b230 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94a37fe640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f948c0a2da0 0x7f948c0a5260 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f949c0982e0 tx=0x7f949c097a70 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94897fa640 1 -- 172.21.15.31:0/1106852111 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f949800f7c0 con 0x55798840b230 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.036+0000 7f94897fa640 1 -- 172.21.15.31:0/1106852111 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94980038e0 con 0x55798840b230 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.037+0000 7f94ac422740 1 -- 172.21.15.31:0/1106852111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55798840b230 msgr2=0x55798840d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.037+0000 7f94ac422740 1 --2- 172.21.15.31:0/1106852111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55798840b230 0x55798840d620 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f94980058c0 tx=0x7f9498005f90 comp rx=0 tx=0).stop 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.037+0000 7f94ac422740 1 -- 172.21.15.31:0/1106852111 shutdown_connections 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.037+0000 7f94ac422740 1 --2- 172.21.15.31:0/1106852111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55798840b230 0x55798840d620 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.037+0000 7f94ac422740 1 --2- 172.21.15.31:0/1106852111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 0x55798840acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.037+0000 7f94ac422740 1 --2- 172.21.15.31:0/1106852111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557988400240 0x557988400620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.037+0000 7f94ac422740 1 -- 172.21.15.31:0/1106852111 >> 172.21.15.31:0/1106852111 conn(0x5579883fc560 msgr2=0x5579883fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.037+0000 7f94ac422740 1 -- 172.21.15.31:0/1106852111 shutdown_connections 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.037+0000 7f94ac422740 1 -- 172.21.15.31:0/1106852111 wait complete. 2024-04-25T02:13:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94ac422740 1 -- start start 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94ac422740 1 Processor -- start 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94ac422740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400240 0x557988408270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94ac422740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400b60 0x5579884087b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94ac422740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 0x557988408cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94ac422740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55798840e020 con 0x557988400b60 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94ac422740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557988320b50 con 0x557988400240 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94ac422740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55798840e1a0 con 0x55798840b230 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94ac422740 10 client.0 _ll_register_callbacks cb 0x55798835bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94a27fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400240 0x557988408270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94a27fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 0x557988408cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94a2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400b60 0x5579884087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94a27fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 0x557988408cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:57470/0 (socket says 172.21.15.31:57470) 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.038+0000 7f94a27fc640 1 -- 172.21.15.31:0/3479627561 learned_addr learned my addr 172.21.15.31:0/3479627561 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f94a27fc640 1 -- 172.21.15.31:0/3479627561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400240 msgr2=0x557988408270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:56.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f94a27fc640 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400240 0x557988408270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f94a27fc640 1 -- 172.21.15.31:0/3479627561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400b60 msgr2=0x5579884087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f94a27fc640 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400b60 0x5579884087b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f94a27fc640 1 -- 172.21.15.31:0/3479627561 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557988145620 con 0x55798840b230 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f94a2ffd640 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400b60 0x5579884087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f94a27fc640 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 0x557988408cf0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9498009250 tx=0x7f9498005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30dcead22c060c1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f9486ff5640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f9486ff5640 1 -- 172.21.15.31:0/3479627561 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498027040 con 0x55798840b230 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f94ac422740 1 -- 172.21.15.31:0/3479627561 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f949800f7c0 con 0x55798840b230 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f9486ff5640 1 -- 172.21.15.31:0/3479627561 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7f949801f440 con 0x55798840b230 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f9486ff5640 1 -- 172.21.15.31:0/3479627561 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f9498011e30 con 0x55798840b230 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f94ac422740 10 client.27037 fetch_fsmap learned FSMap version 49 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.039+0000 7f94ac422740 1 -- 172.21.15.31:0/3479627561 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55798841a2e0 con 0x55798840b230 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.040+0000 7f9486ff5640 1 -- 172.21.15.31:0/3479627561 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f949800f7c0 con 0x55798840b230 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.040+0000 7f9486ff5640 1 -- 172.21.15.31:0/3479627561 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f9498035020 con 0x55798840b230 2024-04-25T02:13:56.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.040+0000 7f94ac422740 10 client.27037 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:13:56.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.040+0000 7f94ac422740 10 client.27037 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:13:56.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.040+0000 7f94ac422740 20 client.27037 populate_metadata read hostname 'smithi031' 2024-04-25T02:13:56.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.040+0000 7f94ac422740 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55798841b630 0x55798841da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.040+0000 7f94ac422740 4 client.27037 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:13:56.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.040+0000 7f94ac422740 1 -- 172.21.15.31:0/3479627561 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579883721f0 con 0x55798841b630 2024-04-25T02:13:56.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.040+0000 7f94a2ffd640 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55798841b630 0x55798841da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.041+0000 7f94a2ffd640 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55798841b630 0x55798841da20 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=507a365a4d255ecf server_cookie=a66e924b30d56296 in_seq=0 out_seq=0 2024-04-25T02:13:56.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.041+0000 7f9486ff5640 10 client.27037 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:56.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f9486ff5640 1 -- 172.21.15.31:0/3479627561 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5579883721f0 con 0x55798841b630 2024-04-25T02:13:56.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f9486ff5640 10 client.27037 handle_command_reply: tid=0 2024-04-25T02:13:56.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 1 client.27037 shutdown 2024-04-25T02:13:56.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 1 -- 172.21.15.31:0/3479627561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 msgr2=0x557988408cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:56.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 0x557988408cf0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9498009250 tx=0x7f9498005850 comp rx=0 tx=0).stop 2024-04-25T02:13:56.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 1 -- 172.21.15.31:0/3479627561 shutdown_connections 2024-04-25T02:13:56.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55798841b630 0x55798841da20 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 0x557988408cf0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400b60 0x5579884087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f9486ff5640 0 client.27037 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:56.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 1 --2- 172.21.15.31:0/3479627561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400240 0x557988408270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 1 -- 172.21.15.31:0/3479627561 >> 172.21.15.31:0/3479627561 conn(0x5579883fc560 msgr2=0x557988402360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:56.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 1 -- 172.21.15.31:0/3479627561 shutdown_connections 2024-04-25T02:13:56.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 1 -- 172.21.15.31:0/3479627561 wait complete. 2024-04-25T02:13:56.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.042+0000 7f94ac422740 20 client.27037 trim_cache size 0 max 16384 2024-04-25T02:13:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.072+0000 7f94ac422740 1 Processor -- start 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94ac422740 1 -- start start 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94ac422740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x55798840a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94ac422740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 0x557988402a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94ac422740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 0x557988408b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94ac422740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41af4b0 con 0x557988400240 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94ac422740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41b0ff0 con 0x557988400b60 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94ac422740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41afb80 con 0x55798840b230 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94a27fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x55798840a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94a27fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x55798840a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34130/0 (socket says 172.21.15.31:34130) 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94a27fc640 1 -- 172.21.15.31:0/2846921964 learned_addr learned my addr 172.21.15.31:0/2846921964 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94a2ffd640 1 --2- 172.21.15.31:0/2846921964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 0x557988402a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94a27fc640 1 --2- 172.21.15.31:0/2846921964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 0x557988408b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94a27fc640 1 -- 172.21.15.31:0/2846921964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 msgr2=0x557988408b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94a27fc640 1 --2- 172.21.15.31:0/2846921964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 0x557988408b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94a27fc640 1 -- 172.21.15.31:0/2846921964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 msgr2=0x557988402a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94a27fc640 1 --2- 172.21.15.31:0/2846921964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 0x557988402a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94a27fc640 1 -- 172.21.15.31:0/2846921964 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579883fd790 con 0x557988400240 2024-04-25T02:13:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.073+0000 7f94a2ffd640 1 --2- 172.21.15.31:0/2846921964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 0x557988402a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:13:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.074+0000 7f94a27fc640 1 --2- 172.21.15.31:0/2846921964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x55798840a9a0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f9498005fd0 tx=0x7f9498005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3414126d54fc15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.074+0000 7f9488ff9640 1 -- 172.21.15.31:0/2846921964 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498011230 con 0x557988400240 2024-04-25T02:13:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.074+0000 7f9488ff9640 1 -- 172.21.15.31:0/2846921964 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f949800f790 con 0x557988400240 2024-04-25T02:13:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.074+0000 7f9488ff9640 1 -- 172.21.15.31:0/2846921964 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949801f440 con 0x557988400240 2024-04-25T02:13:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.075+0000 7f94ac422740 1 -- 172.21.15.31:0/2846921964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 msgr2=0x55798840a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.075+0000 7f94ac422740 1 --2- 172.21.15.31:0/2846921964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x55798840a9a0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f9498005fd0 tx=0x7f9498005bb0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.075+0000 7f94ac422740 1 -- 172.21.15.31:0/2846921964 shutdown_connections 2024-04-25T02:13:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.075+0000 7f94ac422740 1 --2- 172.21.15.31:0/2846921964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55798840b230 0x557988408b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.075+0000 7f94ac422740 1 --2- 172.21.15.31:0/2846921964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557988400b60 0x557988402a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.075+0000 7f94ac422740 1 --2- 172.21.15.31:0/2846921964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x55798840a9a0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.075+0000 7f94ac422740 1 -- 172.21.15.31:0/2846921964 >> 172.21.15.31:0/2846921964 conn(0x5579883fc560 msgr2=0x5579883fb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.075+0000 7f94ac422740 1 -- 172.21.15.31:0/2846921964 shutdown_connections 2024-04-25T02:13:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.075+0000 7f94ac422740 1 -- 172.21.15.31:0/2846921964 wait complete. 2024-04-25T02:13:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.075+0000 7f94ac422740 1 -- start start 2024-04-25T02:13:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.076+0000 7f94ac422740 1 Processor -- start 2024-04-25T02:13:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.076+0000 7f94ac422740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x5579884138f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.076+0000 7f94ac422740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557988400b60 0x55798840ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.076+0000 7f94ac422740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798840b230 0x55798840f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.076+0000 7f94ac422740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41b0920 con 0x557988400240 2024-04-25T02:13:56.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.076+0000 7f94ac422740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41b0250 con 0x55798840b230 2024-04-25T02:13:56.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.076+0000 7f94ac422740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41ad4b0 con 0x557988400b60 2024-04-25T02:13:56.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.076+0000 7f94ac422740 10 client.0 _ll_register_callbacks cb 0x7f9498005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:13:56.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.076+0000 7f94a27fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x5579884138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f94a27fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x5579884138f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34140/0 (socket says 172.21.15.31:34140) 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f94a27fc640 1 -- 172.21.15.31:0/2770265435 learned_addr learned my addr 172.21.15.31:0/2770265435 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f94a2ffd640 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557988400b60 0x55798840ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f94a27fc640 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798840b230 0x55798840f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f94a27fc640 1 -- 172.21.15.31:0/2770265435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557988400b60 msgr2=0x55798840ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f94a27fc640 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557988400b60 0x55798840ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f94a27fc640 1 -- 172.21.15.31:0/2770265435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798840b230 msgr2=0x55798840f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f94a27fc640 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798840b230 0x55798840f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f94a27fc640 1 -- 172.21.15.31:0/2770265435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557988480a20 con 0x557988400240 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f94a27fc640 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x5579884138f0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f9498009800 tx=0x7f9498005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93256e15b0293531 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f9486ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f9486ff5640 1 -- 172.21.15.31:0/2770265435 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498009c70 con 0x557988400240 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f9486ff5640 1 -- 172.21.15.31:0/2770265435 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8484+0+0 (secure 0 0 0) 0x7f94980052d0 con 0x557988400240 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.077+0000 7f94ac422740 1 -- 172.21.15.31:0/2770265435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f949800f790 con 0x557988400240 2024-04-25T02:13:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f9486ff5640 1 -- 172.21.15.31:0/2770265435 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f9498037460 con 0x557988400240 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f94ac422740 10 client.17700 fetch_fsmap learned FSMap version 49 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f94ac422740 1 -- 172.21.15.31:0/2770265435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x557988481b60 con 0x557988400240 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f9486ff5640 1 -- 172.21.15.31:0/2770265435 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f949800f790 con 0x557988400240 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f9486ff5640 1 -- 172.21.15.31:0/2770265435 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f9498035070 con 0x557988400240 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f94ac422740 10 client.17700 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f94ac422740 10 client.17700 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f94ac422740 20 client.17700 populate_metadata read hostname 'smithi031' 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f94ac422740 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55798841b630 0x557988484220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f94ac422740 4 client.17700 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f94ac422740 1 -- 172.21.15.31:0/2770265435 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55798836c540 con 0x55798841b630 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.078+0000 7f94a2ffd640 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55798841b630 0x557988484220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.079+0000 7f94a2ffd640 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55798841b630 0x557988484220 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc317253b3ced683 server_cookie=5241d984795908c6 in_seq=0 out_seq=0 2024-04-25T02:13:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.079+0000 7f9486ff5640 10 client.17700 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:56.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 59851 8.97 MB/sec execute 240 sec latency 102.745 ms 2024-04-25T02:13:56.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.148+0000 7f9486ff5640 1 -- 172.21.15.31:0/2770265435 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55798836c540 con 0x55798841b630 2024-04-25T02:13:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.148+0000 7f9486ff5640 10 client.17700 handle_command_reply: tid=0 2024-04-25T02:13:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.148+0000 7f94ac422740 1 client.17700 shutdown 2024-04-25T02:13:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f94ac422740 1 -- 172.21.15.31:0/2770265435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 msgr2=0x5579884138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f94ac422740 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x5579884138f0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f9498009800 tx=0x7f9498005850 comp rx=0 tx=0).stop 2024-04-25T02:13:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f94ac422740 1 -- 172.21.15.31:0/2770265435 shutdown_connections 2024-04-25T02:13:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f94ac422740 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55798841b630 0x557988484220 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f94ac422740 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798840b230 0x55798840f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f9486ff5640 0 client.17700 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:13:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f94ac422740 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557988400b60 0x55798840ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f94ac422740 1 --2- 172.21.15.31:0/2770265435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557988400240 0x5579884138f0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f94ac422740 1 -- 172.21.15.31:0/2770265435 >> 172.21.15.31:0/2770265435 conn(0x5579883fc560 msgr2=0x557988409920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f94ac422740 1 -- 172.21.15.31:0/2770265435 shutdown_connections 2024-04-25T02:13:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f94ac422740 1 -- 172.21.15.31:0/2770265435 wait complete. 2024-04-25T02:13:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.149+0000 7f94ac422740 20 client.17700 trim_cache size 0 max 16384 2024-04-25T02:13:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:13:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:13:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:13:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:13:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 -- 172.21.15.31:0/804264756 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f948c0a2da0 msgr2=0x7f948c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f948c0a2da0 0x7f948c0a5260 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f949c0982e0 tx=0x7f949c097a70 comp rx=0 tx=0).stop 2024-04-25T02:13:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 -- 172.21.15.31:0/804264756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94a410bd40 msgr2=0x7f94a419e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:13:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94a410bd40 0x7f94a419e930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f949c00eb10 tx=0x7f949c00c2d0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 -- 172.21.15.31:0/804264756 shutdown_connections 2024-04-25T02:13:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f948c0a2da0 0x7f948c0a5260 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94a410bd40 0x7f94a419e930 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94a4103320 0x7f94a419a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 --2- 172.21.15.31:0/804264756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94a4102a00 0x7f94a419a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:13:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 -- 172.21.15.31:0/804264756 >> 172.21.15.31:0/804264756 conn(0x7f94a40769c0 msgr2=0x7f94a40ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:13:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 -- 172.21.15.31:0/804264756 shutdown_connections 2024-04-25T02:13:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:13:56.150+0000 7f94a971d640 1 -- 172.21.15.31:0/804264756 wait complete. 2024-04-25T02:13:56.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:56 smithi040 ceph-mon[32418]: pgmap v611: 97 pgs: 97 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.4 MiB/s wr, 333 op/s 2024-04-25T02:13:56.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:56 smithi134 ceph-mon[33098]: pgmap v611: 97 pgs: 97 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.4 MiB/s wr, 333 op/s 2024-04-25T02:13:56.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:56 smithi031 ceph-mon[29204]: pgmap v611: 97 pgs: 97 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.4 MiB/s wr, 333 op/s 2024-04-25T02:13:57.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60018 8.98 MB/sec execute 241 sec latency 77.938 ms 2024-04-25T02:13:58.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60242 9.02 MB/sec execute 242 sec latency 50.972 ms 2024-04-25T02:13:58.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:13:58 smithi040 ceph-mon[32418]: pgmap v612: 97 pgs: 97 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-04-25T02:13:58.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:13:58 smithi134 ceph-mon[33098]: pgmap v612: 97 pgs: 97 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-04-25T02:13:58.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:13:58 smithi031 ceph-mon[29204]: pgmap v612: 97 pgs: 97 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-04-25T02:13:59.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60371 9.00 MB/sec execute 243 sec latency 94.767 ms 2024-04-25T02:14:00.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60530 9.00 MB/sec execute 244 sec latency 81.519 ms 2024-04-25T02:14:00.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:00 smithi040 ceph-mon[32418]: pgmap v613: 97 pgs: 97 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-04-25T02:14:00.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:00 smithi134 ceph-mon[33098]: pgmap v613: 97 pgs: 97 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-04-25T02:14:00.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:00 smithi031 ceph-mon[29204]: pgmap v613: 97 pgs: 97 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-04-25T02:14:01.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60667 8.99 MB/sec execute 245 sec latency 158.096 ms 2024-04-25T02:14:02.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60667 8.95 MB/sec execute 246 sec latency 1158.185 ms 2024-04-25T02:14:02.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:02 smithi040 ceph-mon[32418]: pgmap v614: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.8 MiB/s wr, 372 op/s 2024-04-25T02:14:02.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:02 smithi134 ceph-mon[33098]: pgmap v614: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.8 MiB/s wr, 372 op/s 2024-04-25T02:14:02.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:02 smithi031 ceph-mon[29204]: pgmap v614: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.8 MiB/s wr, 372 op/s 2024-04-25T02:14:03.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60667 8.91 MB/sec execute 247 sec latency 2158.245 ms 2024-04-25T02:14:03.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:03 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:14:03.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:03 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:14:03.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:03 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:14:04.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60667 8.88 MB/sec execute 248 sec latency 3158.303 ms 2024-04-25T02:14:04.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:04 smithi040 ceph-mon[32418]: pgmap v615: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.2 MiB/s wr, 255 op/s 2024-04-25T02:14:04.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:04 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:14:04.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:04 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:14:04.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:04 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:14:04.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:04 smithi134 ceph-mon[33098]: pgmap v615: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.2 MiB/s wr, 255 op/s 2024-04-25T02:14:04.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:04 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:14:04.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:04 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:14:04.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:04 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:14:04.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:04 smithi031 ceph-mon[29204]: pgmap v615: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.2 MiB/s wr, 255 op/s 2024-04-25T02:14:04.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:04 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:14:04.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:04 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:14:04.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:04 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:14:05.100 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60794 8.85 MB/sec execute 249 sec latency 3329.140 ms 2024-04-25T02:14:06.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60948 8.83 MB/sec execute 250 sec latency 63.800 ms 2024-04-25T02:14:06.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:06 smithi040 ceph-mon[32418]: pgmap v616: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 320 op/s 2024-04-25T02:14:06.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:06 smithi134 ceph-mon[33098]: pgmap v616: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 320 op/s 2024-04-25T02:14:06.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:06 smithi031 ceph-mon[29204]: pgmap v616: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 320 op/s 2024-04-25T02:14:07.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 61090 8.81 MB/sec execute 251 sec latency 93.796 ms 2024-04-25T02:14:08.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 61237 8.80 MB/sec execute 252 sec latency 112.468 ms 2024-04-25T02:14:08.294 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:08 smithi031 ceph-mon[29204]: pgmap v617: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 217 op/s 2024-04-25T02:14:08.294 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:08 smithi031 ceph-mon[29204]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T02:14:08.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:08 smithi040 ceph-mon[32418]: pgmap v617: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 217 op/s 2024-04-25T02:14:08.415 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:08 smithi040 ceph-mon[32418]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T02:14:08.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:08 smithi134 ceph-mon[33098]: pgmap v617: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 217 op/s 2024-04-25T02:14:08.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:08 smithi134 ceph-mon[33098]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T02:14:09.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 61433 8.82 MB/sec execute 253 sec latency 61.305 ms 2024-04-25T02:14:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.282+0000 7f7d8cd53640 1 -- 172.21.15.31:0/304023415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88101730 msgr2=0x7f7d88101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.282+0000 7f7d8cd53640 1 --2- 172.21.15.31:0/304023415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88101730 0x7f7d88101bb0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009930 tx=0x7f7d7c02a420 comp rx=0 tx=0).stop 2024-04-25T02:14:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.282+0000 7f7d8cd53640 1 -- 172.21.15.31:0/304023415 shutdown_connections 2024-04-25T02:14:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.282+0000 7f7d8cd53640 1 --2- 172.21.15.31:0/304023415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d881020f0 0x7f7d8810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.282+0000 7f7d8cd53640 1 --2- 172.21.15.31:0/304023415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88101730 0x7f7d88101bb0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.282+0000 7f7d8cd53640 1 --2- 172.21.15.31:0/304023415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d88107740 0x7f7d88107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.282+0000 7f7d8cd53640 1 -- 172.21.15.31:0/304023415 >> 172.21.15.31:0/304023415 conn(0x7f7d880fd200 msgr2=0x7f7d880ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.283+0000 7f7d8cd53640 1 -- 172.21.15.31:0/304023415 shutdown_connections 2024-04-25T02:14:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.283+0000 7f7d8cd53640 1 -- 172.21.15.31:0/304023415 wait complete. 2024-04-25T02:14:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8cd53640 1 Processor -- start 2024-04-25T02:14:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8cd53640 1 -- start start 2024-04-25T02:14:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8cd53640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88101730 0x7f7d8810d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8cd53640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d881020f0 0x7f7d8810de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8cd53640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d88107740 0x7f7d881a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8cd53640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d88111a50 con 0x7f7d88101730 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8cd53640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881118d0 con 0x7f7d881020f0 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8cd53640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7d88111bd0 con 0x7f7d88107740 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88101730 0x7f7d8810d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88101730 0x7f7d8810d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40498/0 (socket says 172.21.15.31:40498) 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8659c640 1 -- 172.21.15.31:0/1862802034 learned_addr learned my addr 172.21.15.31:0/1862802034 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d86d9d640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d88107740 0x7f7d881a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d85d9b640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d881020f0 0x7f7d8810de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8659c640 1 -- 172.21.15.31:0/1862802034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d88107740 msgr2=0x7f7d881a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8659c640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d88107740 0x7f7d881a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.284+0000 7f7d8659c640 1 -- 172.21.15.31:0/1862802034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d881020f0 msgr2=0x7f7d8810de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.285+0000 7f7d8659c640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d881020f0 0x7f7d8810de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.285+0000 7f7d8659c640 1 -- 172.21.15.31:0/1862802034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d881a34a0 con 0x7f7d88101730 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.285+0000 7f7d8659c640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88101730 0x7f7d8810d900 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800d950 tx=0x7f7d7800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.285+0000 7f7d777fe640 1 -- 172.21.15.31:0/1862802034 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d78017030 con 0x7f7d88101730 2024-04-25T02:14:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.285+0000 7f7d777fe640 1 -- 172.21.15.31:0/1862802034 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d78004020 con 0x7f7d88101730 2024-04-25T02:14:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.285+0000 7f7d777fe640 1 -- 172.21.15.31:0/1862802034 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d780047a0 con 0x7f7d88101730 2024-04-25T02:14:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.285+0000 7f7d8cd53640 1 -- 172.21.15.31:0/1862802034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d881a3790 con 0x7f7d88101730 2024-04-25T02:14:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.285+0000 7f7d8cd53640 1 -- 172.21.15.31:0/1862802034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d881ab020 con 0x7f7d88101730 2024-04-25T02:14:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.288+0000 7f7d777fe640 1 -- 172.21.15.31:0/1862802034 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7d780041d0 con 0x7f7d88101730 2024-04-25T02:14:09.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.288+0000 7f7d8fa5d740 1 Processor -- start 2024-04-25T02:14:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d8fa5d740 1 -- start start 2024-04-25T02:14:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344240 0x55b070344620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344b60 0x55b07034ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b07034f230 0x55b070351620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d8fa5d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b0702a8cb0 con 0x55b070344b60 2024-04-25T02:14:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d8fa5d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b0702515b0 con 0x55b070344240 2024-04-25T02:14:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d8fa5d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b0702a4c70 con 0x55b07034f230 2024-04-25T02:14:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d86d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344240 0x55b070344620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d86d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344240 0x55b070344620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36706/0 (socket says 172.21.15.31:36706) 2024-04-25T02:14:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d86d9d640 1 -- 172.21.15.31:0/2885872492 learned_addr learned my addr 172.21.15.31:0/2885872492 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d8659c640 1 --2- 172.21.15.31:0/2885872492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344b60 0x55b07034ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.289+0000 7f7d85d9b640 1 --2- 172.21.15.31:0/2885872492 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b07034f230 0x55b070351620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.290+0000 7f7d86d9d640 1 -- 172.21.15.31:0/2885872492 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b07034f230 msgr2=0x55b070351620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.290+0000 7f7d86d9d640 1 --2- 172.21.15.31:0/2885872492 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b07034f230 0x55b070351620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.290+0000 7f7d86d9d640 1 -- 172.21.15.31:0/2885872492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344b60 msgr2=0x55b07034ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.290+0000 7f7d86d9d640 1 --2- 172.21.15.31:0/2885872492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344b60 0x55b07034ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.290+0000 7f7d777fe640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7d700a2e20 0x7f7d700a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.290+0000 7f7d86d9d640 1 -- 172.21.15.31:0/2885872492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b06ffbd210 con 0x55b070344240 2024-04-25T02:14:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.290+0000 7f7d86d9d640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7d700a2e20 0x7f7d700a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.290+0000 7f7d86d9d640 1 --2- 172.21.15.31:0/2885872492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344240 0x55b070344620 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7d80009fd0 tx=0x7f7d800118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63545a45e09556b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.290+0000 7f7d777fe640 1 -- 172.21.15.31:0/1862802034 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8528+0+0 (secure 0 0 0) 0x7f7d780ced10 con 0x7f7d88101730 2024-04-25T02:14:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.290+0000 7f7d74ff9640 1 -- 172.21.15.31:0/2885872492 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80004250 con 0x55b070344240 2024-04-25T02:14:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.290+0000 7f7d74ff9640 1 -- 172.21.15.31:0/2885872492 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d80003db0 con 0x55b070344240 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.291+0000 7f7d74ff9640 1 -- 172.21.15.31:0/2885872492 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8001bcb0 con 0x55b070344240 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.291+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/2885872492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344240 msgr2=0x55b070344620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.291+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/2885872492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344240 0x55b070344620 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7d80009fd0 tx=0x7f7d800118d0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.291+0000 7f7d86d9d640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7d700a2e20 0x7f7d700a52e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7d80020000 tx=0x7f7d80021000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.291+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/2885872492 shutdown_connections 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.291+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/2885872492 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b07034f230 0x55b070351620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.291+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/2885872492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344b60 0x55b07034ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.291+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/2885872492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344240 0x55b070344620 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.291+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/2885872492 >> 172.21.15.31:0/2885872492 conn(0x55b070340560 msgr2=0x55b070340940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.292+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/2885872492 shutdown_connections 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.292+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/2885872492 wait complete. 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.292+0000 7f7d8fa5d740 1 -- start start 2024-04-25T02:14:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.292+0000 7f7d8fa5d740 1 Processor -- start 2024-04-25T02:14:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.292+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.292+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344b60 0x55b07034c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.292+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b07034f230 0x55b07034cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.292+0000 7f7d8659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.292+0000 7f7d8fa5d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b0703520c0 con 0x55b070344240 2024-04-25T02:14:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d8659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40514/0 (socket says 172.21.15.31:40514) 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d8659c640 1 -- 172.21.15.31:0/1856773390 learned_addr learned my addr 172.21.15.31:0/1856773390 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/1856773390 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b070264b50 con 0x55b070344b60 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/1856773390 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b070352240 con 0x55b07034f230 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d85d9b640 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344b60 0x55b07034c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d86d9d640 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b07034f230 0x55b07034cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d8fa5d740 10 client.0 _ll_register_callbacks cb 0x55b07029fca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d8659c640 1 -- 172.21.15.31:0/1856773390 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b07034f230 msgr2=0x55b07034cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d8659c640 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b07034f230 0x55b07034cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d8659c640 1 -- 172.21.15.31:0/1856773390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344b60 msgr2=0x55b07034c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d8659c640 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344b60 0x55b07034c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d8659c640 1 -- 172.21.15.31:0/1856773390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b070089620 con 0x55b070344240 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d85d9b640 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344b60 0x55b07034c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d8659c640 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034c0d0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000c00 tx=0x7f7d780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa491f14a40cf3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d6dffb640 1 -- 172.21.15.31:0/1856773390 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d78014070 con 0x55b070344240 2024-04-25T02:14:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.293+0000 7f7d6dffb640 1 -- 172.21.15.31:0/1856773390 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8528+0+0 (secure 0 0 0) 0x7f7d780cc6e0 con 0x55b070344240 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.294+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/1856773390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d80003db0 con 0x55b070344240 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.294+0000 7f7d6dffb640 1 -- 172.21.15.31:0/1856773390 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f7d7809ad70 con 0x55b070344240 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.294+0000 7f7d8fa5d740 10 client.17712 fetch_fsmap learned FSMap version 49 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.294+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/1856773390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55b07035e240 con 0x55b070344240 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.294+0000 7f7d6dffb640 1 -- 172.21.15.31:0/1856773390 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d80003db0 con 0x55b070344240 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.294+0000 7f7d6dffb640 1 -- 172.21.15.31:0/1856773390 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f7d780bb0c0 con 0x55b070344240 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.294+0000 7f7d8fa5d740 10 client.17712 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.294+0000 7f7d8fa5d740 10 client.17712 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.294+0000 7f7d8fa5d740 20 client.17712 populate_metadata read hostname 'smithi031' 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.295+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b07035fb30 0x55b070361ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.295+0000 7f7d8fa5d740 4 client.17712 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.295+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/1856773390 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0702b61f0 con 0x55b07035fb30 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.295+0000 7f7d85d9b640 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b07035fb30 0x55b070361ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.295+0000 7f7d6dffb640 10 client.17712 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.295+0000 7f7d85d9b640 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b07035fb30 0x55b070361ff0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54c050dae8f8cdd6 server_cookie=a544ad5bea92a5a9 in_seq=0 out_seq=0 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d6dffb640 1 -- 172.21.15.31:0/1856773390 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0702b61f0 con 0x55b07035fb30 2024-04-25T02:14:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d6dffb640 10 client.17712 handle_command_reply: tid=0 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d8fa5d740 1 client.17712 shutdown 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/1856773390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 msgr2=0x55b07034c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034c0d0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000c00 tx=0x7f7d780046c0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/1856773390 shutdown_connections 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b07035fb30 0x55b070361ff0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b07034f230 0x55b07034cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b070344b60 0x55b07034c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/1856773390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034c0d0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/1856773390 >> 172.21.15.31:0/1856773390 conn(0x55b070340560 msgr2=0x55b070346220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d6dffb640 0 client.17712 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.296+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/1856773390 shutdown_connections 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.297+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/1856773390 wait complete. 2024-04-25T02:14:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.297+0000 7f7d8fa5d740 20 client.17712 trim_cache size 0 max 16384 2024-04-25T02:14:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.318+0000 7f7d8fa5d740 1 Processor -- start 2024-04-25T02:14:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.318+0000 7f7d8fa5d740 1 -- start start 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.318+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.318+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b070344b60 0x55b07035afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.318+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b07034f230 0x55b07034b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.318+0000 7f7d8fa5d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881b3a30 con 0x55b070344240 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.318+0000 7f7d8fa5d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881b5570 con 0x55b07034f230 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.318+0000 7f7d8fa5d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881b4100 con 0x55b070344b60 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d8659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d85d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b070344b60 0x55b07035afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d8659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40528/0 (socket says 172.21.15.31:40528) 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d8659c640 1 -- 172.21.15.31:0/2229218734 learned_addr learned my addr 172.21.15.31:0/2229218734 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d86d9d640 1 --2- 172.21.15.31:0/2229218734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b07034f230 0x55b07034b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d8659c640 1 -- 172.21.15.31:0/2229218734 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b070344b60 msgr2=0x55b07035afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d8659c640 1 --2- 172.21.15.31:0/2229218734 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b070344b60 0x55b07035afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d8659c640 1 -- 172.21.15.31:0/2229218734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b07034f230 msgr2=0x55b07034b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d8659c640 1 --2- 172.21.15.31:0/2229218734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b07034f230 0x55b07034b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d8659c640 1 -- 172.21.15.31:0/2229218734 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b07035f8e0 con 0x55b070344240 2024-04-25T02:14:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d8659c640 1 --2- 172.21.15.31:0/2229218734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034e800 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f7d780a3970 tx=0x7f7d780d23d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd73d79dab095df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d6ffff640 1 -- 172.21.15.31:0/2229218734 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d78014030 con 0x55b070344240 2024-04-25T02:14:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.319+0000 7f7d6ffff640 1 -- 172.21.15.31:0/2229218734 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d780d24f0 con 0x55b070344240 2024-04-25T02:14:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.320+0000 7f7d6ffff640 1 -- 172.21.15.31:0/2229218734 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7809ab90 con 0x55b070344240 2024-04-25T02:14:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.320+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/2229218734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 msgr2=0x55b07034e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.320+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/2229218734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034e800 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f7d780a3970 tx=0x7f7d780d23d0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.320+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/2229218734 shutdown_connections 2024-04-25T02:14:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.320+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/2229218734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b07034f230 0x55b07034b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.320+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/2229218734 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b070344b60 0x55b07035afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.320+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/2229218734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b07034e800 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.320+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/2229218734 >> 172.21.15.31:0/2229218734 conn(0x55b070340560 msgr2=0x55b07034a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.320+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/2229218734 shutdown_connections 2024-04-25T02:14:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.320+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/2229218734 wait complete. 2024-04-25T02:14:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d8fa5d740 1 -- start start 2024-04-25T02:14:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d8fa5d740 1 Processor -- start 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b070357a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b070344b60 0x55b070349580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d8fa5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b07034f230 0x55b070349cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d8fa5d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881b4ea0 con 0x55b070344240 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d8fa5d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881b47d0 con 0x55b07034f230 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d8fa5d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881b1a10 con 0x55b070344b60 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d8fa5d740 10 client.0 _ll_register_callbacks cb 0x7f7d780050c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d85d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b070344b60 0x55b070349580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d86d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b07034f230 0x55b070349cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d8659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b070357a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d85d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b070344b60 0x55b070349580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:54958/0 (socket says 172.21.15.31:54958) 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.321+0000 7f7d85d9b640 1 -- 172.21.15.31:0/3179259573 learned_addr learned my addr 172.21.15.31:0/3179259573 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d86d9d640 1 -- 172.21.15.31:0/3179259573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b070344b60 msgr2=0x55b070349580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d86d9d640 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b070344b60 0x55b070349580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d86d9d640 1 -- 172.21.15.31:0/3179259573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 msgr2=0x55b070357a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d86d9d640 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b070357a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d86d9d640 1 -- 172.21.15.31:0/3179259573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0703483c0 con 0x55b07034f230 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d8659c640 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b070357a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d86d9d640 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b07034f230 0x55b070349cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7d80011dd0 tx=0x7f7d80003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d38268295f02c1f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d6dffb640 1 -- 172.21.15.31:0/3179259573 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80029040 con 0x55b07034f230 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/3179259573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d780d24f0 con 0x55b07034f230 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d6dffb640 1 -- 172.21.15.31:0/3179259573 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8528+0+0 (secure 0 0 0) 0x7f7d80004250 con 0x55b07034f230 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d6dffb640 1 -- 172.21.15.31:0/3179259573 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f7d8000fa00 con 0x55b07034f230 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d8fa5d740 10 client.27275 fetch_fsmap learned FSMap version 49 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.322+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/3179259573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55b0703c54a0 con 0x55b07034f230 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.323+0000 7f7d6dffb640 1 -- 172.21.15.31:0/3179259573 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d780d24f0 con 0x55b07034f230 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.323+0000 7f7d6dffb640 1 -- 172.21.15.31:0/3179259573 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f7d80031440 con 0x55b07034f230 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.323+0000 7f7d8fa5d740 10 client.27275 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.323+0000 7f7d8fa5d740 10 client.27275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.323+0000 7f7d8fa5d740 20 client.27275 populate_metadata read hostname 'smithi031' 2024-04-25T02:14:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.323+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b0703c59c0 0x55b0703c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.323+0000 7f7d8fa5d740 4 client.27275 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:14:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.323+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/3179259573 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b0702b0540 con 0x55b0703c59c0 2024-04-25T02:14:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.323+0000 7f7d8659c640 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b0703c59c0 0x55b0703c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.324+0000 7f7d6dffb640 10 client.27275 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.324+0000 7f7d8659c640 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b0703c59c0 0x55b0703c7db0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=545a3edf1043a695 server_cookie=b7d84952d20d74c3 in_seq=0 out_seq=0 2024-04-25T02:14:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d6dffb640 1 -- 172.21.15.31:0/3179259573 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b0702b0540 con 0x55b0703c59c0 2024-04-25T02:14:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d6dffb640 10 client.27275 handle_command_reply: tid=0 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 1 client.27275 shutdown 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/3179259573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b07034f230 msgr2=0x55b070349cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b07034f230 0x55b070349cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7d80011dd0 tx=0x7f7d80003970 comp rx=0 tx=0).stop 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/3179259573 shutdown_connections 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b0703c59c0 0x55b0703c7db0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b07034f230 0x55b070349cc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b070344b60 0x55b070349580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 1 --2- 172.21.15.31:0/3179259573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b070344240 0x55b070357a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/3179259573 >> 172.21.15.31:0/3179259573 conn(0x55b070340560 msgr2=0x55b07034b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d6dffb640 0 client.27275 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/3179259573 shutdown_connections 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 1 -- 172.21.15.31:0/3179259573 wait complete. 2024-04-25T02:14:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.382+0000 7f7d8fa5d740 20 client.27275 trim_cache size 0 max 16384 2024-04-25T02:14:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:14:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:14:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:14:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:14:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 -- 172.21.15.31:0/1862802034 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7d700a2e20 msgr2=0x7f7d700a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7d700a2e20 0x7f7d700a52e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7d80020000 tx=0x7f7d80021000 comp rx=0 tx=0).stop 2024-04-25T02:14:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 -- 172.21.15.31:0/1862802034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88101730 msgr2=0x7f7d8810d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88101730 0x7f7d8810d900 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800d950 tx=0x7f7d7800de10 comp rx=0 tx=0).stop 2024-04-25T02:14:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 -- 172.21.15.31:0/1862802034 shutdown_connections 2024-04-25T02:14:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f7d700a2e20 0x7f7d700a52e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d88107740 0x7f7d881a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d881020f0 0x7f7d8810de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 --2- 172.21.15.31:0/1862802034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d88101730 0x7f7d8810d900 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 -- 172.21.15.31:0/1862802034 >> 172.21.15.31:0/1862802034 conn(0x7f7d880fd200 msgr2=0x7f7d880fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 -- 172.21.15.31:0/1862802034 shutdown_connections 2024-04-25T02:14:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:09.383+0000 7f7d8cd53640 1 -- 172.21.15.31:0/1862802034 wait complete. 2024-04-25T02:14:10.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 61673 8.86 MB/sec execute 254 sec latency 71.000 ms 2024-04-25T02:14:10.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:10 smithi040 ceph-mon[32418]: pgmap v619: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 261 op/s 2024-04-25T02:14:10.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:10 smithi134 ceph-mon[33098]: pgmap v619: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 261 op/s 2024-04-25T02:14:10.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:10 smithi031 ceph-mon[29204]: pgmap v619: 97 pgs: 97 active+clean; 1.0 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 261 op/s 2024-04-25T02:14:11.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 61854 8.87 MB/sec execute 255 sec latency 59.906 ms 2024-04-25T02:14:12.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 61975 8.85 MB/sec execute 256 sec latency 79.818 ms 2024-04-25T02:14:12.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:12 smithi040 ceph-mon[32418]: pgmap v620: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 223 op/s 2024-04-25T02:14:12.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:12 smithi134 ceph-mon[33098]: pgmap v620: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 223 op/s 2024-04-25T02:14:12.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:12 smithi031 ceph-mon[29204]: pgmap v620: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 223 op/s 2024-04-25T02:14:13.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 62146 8.87 MB/sec execute 257 sec latency 61.166 ms 2024-04-25T02:14:14.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 62338 8.88 MB/sec execute 258 sec latency 53.448 ms 2024-04-25T02:14:14.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:14 smithi040 ceph-mon[32418]: pgmap v621: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 223 op/s 2024-04-25T02:14:14.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:14 smithi040 ceph-mon[32418]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T02:14:14.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:14 smithi134 ceph-mon[33098]: pgmap v621: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 223 op/s 2024-04-25T02:14:14.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:14 smithi134 ceph-mon[33098]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T02:14:14.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:14 smithi031 ceph-mon[29204]: pgmap v621: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 223 op/s 2024-04-25T02:14:14.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:14 smithi031 ceph-mon[29204]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T02:14:15.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 62510 8.87 MB/sec execute 259 sec latency 58.447 ms 2024-04-25T02:14:16.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 62644 8.85 MB/sec execute 260 sec latency 51.982 ms 2024-04-25T02:14:16.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:16 smithi040 ceph-mon[32418]: pgmap v623: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.2 MiB/s wr, 344 op/s 2024-04-25T02:14:16.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:16 smithi134 ceph-mon[33098]: pgmap v623: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.2 MiB/s wr, 344 op/s 2024-04-25T02:14:16.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:16 smithi031 ceph-mon[29204]: pgmap v623: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.2 MiB/s wr, 344 op/s 2024-04-25T02:14:17.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 62782 8.83 MB/sec execute 261 sec latency 54.942 ms 2024-04-25T02:14:18.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 62963 8.85 MB/sec execute 262 sec latency 60.898 ms 2024-04-25T02:14:18.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:18 smithi040 ceph-mon[32418]: pgmap v624: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 280 op/s 2024-04-25T02:14:18.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:18 smithi134 ceph-mon[33098]: pgmap v624: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 280 op/s 2024-04-25T02:14:18.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:18 smithi031 ceph-mon[29204]: pgmap v624: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 280 op/s 2024-04-25T02:14:19.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 63166 8.86 MB/sec execute 263 sec latency 58.996 ms 2024-04-25T02:14:20.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 63366 8.88 MB/sec execute 264 sec latency 89.028 ms 2024-04-25T02:14:20.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:20 smithi040 ceph-mon[32418]: pgmap v625: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.8 MiB/s wr, 275 op/s 2024-04-25T02:14:20.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:20 smithi134 ceph-mon[33098]: pgmap v625: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.8 MiB/s wr, 275 op/s 2024-04-25T02:14:20.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:20 smithi031 ceph-mon[29204]: pgmap v625: 97 pgs: 97 active+clean; 1.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.8 MiB/s wr, 275 op/s 2024-04-25T02:14:21.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 63506 8.87 MB/sec execute 265 sec latency 234.090 ms 2024-04-25T02:14:22.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 63660 8.88 MB/sec execute 266 sec latency 60.547 ms 2024-04-25T02:14:22.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:22 smithi040 ceph-mon[32418]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.0 MiB/s wr, 263 op/s 2024-04-25T02:14:22.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:22 smithi134 ceph-mon[33098]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.0 MiB/s wr, 263 op/s 2024-04-25T02:14:22.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:22 smithi031 ceph-mon[29204]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.0 MiB/s wr, 263 op/s 2024-04-25T02:14:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.515+0000 7f3dfff4f640 1 -- 172.21.15.31:0/2610034493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df8101750 msgr2=0x7f3df8101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.515+0000 7f3dfff4f640 1 --2- 172.21.15.31:0/2610034493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df8101750 0x7f3df8101bd0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009890 tx=0x7f3de802a360 comp rx=0 tx=0).stop 2024-04-25T02:14:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.515+0000 7f3dfff4f640 1 -- 172.21.15.31:0/2610034493 shutdown_connections 2024-04-25T02:14:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.515+0000 7f3dfff4f640 1 --2- 172.21.15.31:0/2610034493 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3df8102110 0x7f3df810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.515+0000 7f3dfff4f640 1 --2- 172.21.15.31:0/2610034493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df8101750 0x7f3df8101bd0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.515+0000 7f3dfff4f640 1 --2- 172.21.15.31:0/2610034493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3df8107760 0x7f3df8107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.515+0000 7f3dfff4f640 1 -- 172.21.15.31:0/2610034493 >> 172.21.15.31:0/2610034493 conn(0x7f3df80fd240 msgr2=0x7f3df80ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.515+0000 7f3dfff4f640 1 -- 172.21.15.31:0/2610034493 shutdown_connections 2024-04-25T02:14:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.515+0000 7f3dfff4f640 1 -- 172.21.15.31:0/2610034493 wait complete. 2024-04-25T02:14:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.516+0000 7f3dfff4f640 1 Processor -- start 2024-04-25T02:14:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.516+0000 7f3dfff4f640 1 -- start start 2024-04-25T02:14:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfff4f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3df8101750 0x7f3df8073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfff4f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df8102110 0x7f3df8073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfff4f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3df8107760 0x7f3df80744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfff4f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3df81119e0 con 0x7f3df8102110 2024-04-25T02:14:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfff4f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3df8111860 con 0x7f3df8107760 2024-04-25T02:14:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfff4f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3df8111b60 con 0x7f3df8101750 2024-04-25T02:14:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfd4ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df8102110 0x7f3df8073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfdceb640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3df8101750 0x7f3df8073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfd4ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df8102110 0x7f3df8073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33596/0 (socket says 172.21.15.31:33596) 2024-04-25T02:14:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfe4ec640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3df8107760 0x7f3df80744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfd4ea640 1 -- 172.21.15.31:0/1480336435 learned_addr learned my addr 172.21.15.31:0/1480336435 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfe4ec640 1 -- 172.21.15.31:0/1480336435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3df8101750 msgr2=0x7f3df8073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfe4ec640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3df8101750 0x7f3df8073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfe4ec640 1 -- 172.21.15.31:0/1480336435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df8102110 msgr2=0x7f3df8073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfe4ec640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df8102110 0x7f3df8073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.517+0000 7f3dfe4ec640 1 -- 172.21.15.31:0/1480336435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df81a6380 con 0x7f3df8107760 2024-04-25T02:14:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.518+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df8102110 0x7f3df8073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.518+0000 7f3dfe4ec640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3df8107760 0x7f3df80744c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009fd0 tx=0x7f3df000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.518+0000 7f3deeffd640 1 -- 172.21.15.31:0/1480336435 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df000ce20 con 0x7f3df8107760 2024-04-25T02:14:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.518+0000 7f3deeffd640 1 -- 172.21.15.31:0/1480336435 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3df0005020 con 0x7f3df8107760 2024-04-25T02:14:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.518+0000 7f3dfff4f640 1 -- 172.21.15.31:0/1480336435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df81a6670 con 0x7f3df8107760 2024-04-25T02:14:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.518+0000 7f3deeffd640 1 -- 172.21.15.31:0/1480336435 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df0010690 con 0x7f3df8107760 2024-04-25T02:14:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.518+0000 7f3dfff4f640 1 -- 172.21.15.31:0/1480336435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df81a6bc0 con 0x7f3df8107760 2024-04-25T02:14:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.522+0000 7f3e02c5a740 1 Processor -- start 2024-04-25T02:14:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.522+0000 7f3deeffd640 1 -- 172.21.15.31:0/1480336435 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3df00051d0 con 0x7f3df8107760 2024-04-25T02:14:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.522+0000 7f3e02c5a740 1 -- start start 2024-04-25T02:14:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1d9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1d9b60 0x5631da1e3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1e4230 0x5631da1e6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3e02c5a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5631da13dcb0 con 0x5631da1d9240 2024-04-25T02:14:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3e02c5a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5631da0e65b0 con 0x5631da1e4230 2024-04-25T02:14:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3e02c5a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5631da139c70 con 0x5631da1d9b60 2024-04-25T02:14:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3dfd4ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1d9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3dfd4ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1d9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33598/0 (socket says 172.21.15.31:33598) 2024-04-25T02:14:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3dfd4ea640 1 -- 172.21.15.31:0/801831917 learned_addr learned my addr 172.21.15.31:0/801831917 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3dfdceb640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1d9b60 0x5631da1e3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/801831917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1e4230 0x5631da1e6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.523+0000 7f3deeffd640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3de40a2e50 0x7f3de40a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3dfd4ea640 1 -- 172.21.15.31:0/801831917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1d9b60 msgr2=0x5631da1e3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/801831917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1d9b60 0x5631da1e3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3deeffd640 1 -- 172.21.15.31:0/1480336435 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8484+0+0 (secure 0 0 0) 0x7f3df00cfae0 con 0x7f3df8107760 2024-04-25T02:14:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3dfe4ec640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3de40a2e50 0x7f3de40a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3dfd4ea640 1 -- 172.21.15.31:0/801831917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1e4230 msgr2=0x5631da1e6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/801831917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1e4230 0x5631da1e6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3dfd4ea640 1 -- 172.21.15.31:0/801831917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631d9e52210 con 0x5631da1d9240 2024-04-25T02:14:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3dfdceb640 1 --2- 172.21.15.31:0/801831917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1d9b60 0x5631da1e3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/801831917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1d9620 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f3de8005b60 tx=0x7f3de8005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3228b45a540f2869 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3de3fff640 1 -- 172.21.15.31:0/801831917 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8037040 con 0x5631da1d9240 2024-04-25T02:14:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3de3fff640 1 -- 172.21.15.31:0/801831917 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3de8030340 con 0x5631da1d9240 2024-04-25T02:14:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.524+0000 7f3de3fff640 1 -- 172.21.15.31:0/801831917 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8032b10 con 0x5631da1d9240 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.525+0000 7f3e02c5a740 1 -- 172.21.15.31:0/801831917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 msgr2=0x5631da1d9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.525+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/801831917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1d9620 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f3de8005b60 tx=0x7f3de8005d80 comp rx=0 tx=0).stop 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.525+0000 7f3dfe4ec640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3de40a2e50 0x7f3de40a5310 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3df0098480 tx=0x7f3df00cfde0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.525+0000 7f3e02c5a740 1 -- 172.21.15.31:0/801831917 shutdown_connections 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.525+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/801831917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1e4230 0x5631da1e6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.525+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/801831917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1d9b60 0x5631da1e3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.525+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/801831917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1d9620 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.525+0000 7f3e02c5a740 1 -- 172.21.15.31:0/801831917 >> 172.21.15.31:0/801831917 conn(0x5631da1d5560 msgr2=0x5631da1d5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.526+0000 7f3e02c5a740 1 -- 172.21.15.31:0/801831917 shutdown_connections 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.526+0000 7f3e02c5a740 1 -- 172.21.15.31:0/801831917 wait complete. 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.526+0000 7f3e02c5a740 1 -- start start 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.526+0000 7f3e02c5a740 1 Processor -- start 2024-04-25T02:14:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1e11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1e1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1e1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3e02c5a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5631da1e6f60 con 0x5631da1d9240 2024-04-25T02:14:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3e02c5a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5631da0f9b50 con 0x5631da1d9b60 2024-04-25T02:14:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3dfdceb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1e1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3dfd4ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1e11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3dfdceb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1e1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46932/0 (socket says 172.21.15.31:46932) 2024-04-25T02:14:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3dfdceb640 1 -- 172.21.15.31:0/361157255 learned_addr learned my addr 172.21.15.31:0/361157255 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3e02c5a740 1 -- 172.21.15.31:0/361157255 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5631da1e70e0 con 0x5631da1e4230 2024-04-25T02:14:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3e02c5a740 10 client.0 _ll_register_callbacks cb 0x5631da134ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:14:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3dfdceb640 1 -- 172.21.15.31:0/361157255 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 msgr2=0x5631da1e1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:14:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3dfdceb640 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1e1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3dfdceb640 1 -- 172.21.15.31:0/361157255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 msgr2=0x5631da1e11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3dfdceb640 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1e11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.527+0000 7f3dfdceb640 1 -- 172.21.15.31:0/361157255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631d9f1e620 con 0x5631da1d9b60 2024-04-25T02:14:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.528+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1e11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.528+0000 7f3dfdceb640 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1e1710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009c70 tx=0x7f3df40039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c0ef255e254bf5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.528+0000 7f3de17fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.528+0000 7f3de17fa640 1 -- 172.21.15.31:0/361157255 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df4011650 con 0x5631da1d9b60 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.528+0000 7f3de17fa640 1 -- 172.21.15.31:0/361157255 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8484+0+0 (secure 0 0 0) 0x7f3df40117f0 con 0x5631da1d9b60 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.528+0000 7f3e02c5a740 1 -- 172.21.15.31:0/361157255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3de8030340 con 0x5631da1d9b60 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.528+0000 7f3de17fa640 1 -- 172.21.15.31:0/361157255 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f3df40119f0 con 0x5631da1d9b60 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.528+0000 7f3e02c5a740 10 client.27287 fetch_fsmap learned FSMap version 49 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.528+0000 7f3e02c5a740 1 -- 172.21.15.31:0/361157255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5631da1f3180 con 0x5631da1d9b60 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.529+0000 7f3de17fa640 1 -- 172.21.15.31:0/361157255 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3de8030340 con 0x5631da1d9b60 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.529+0000 7f3de17fa640 1 -- 172.21.15.31:0/361157255 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f3df4022070 con 0x5631da1d9b60 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.529+0000 7f3e02c5a740 10 client.27287 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.529+0000 7f3e02c5a740 10 client.27287 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:14:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.529+0000 7f3e02c5a740 20 client.27287 populate_metadata read hostname 'smithi031' 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.529+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631da1f4700 0x5631da1f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.529+0000 7f3e02c5a740 4 client.27287 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.529+0000 7f3e02c5a740 1 -- 172.21.15.31:0/361157255 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631da14b1f0 con 0x5631da1f4700 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.529+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631da1f4700 0x5631da1f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.529+0000 7f3de17fa640 10 client.27287 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.530+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631da1f4700 0x5631da1f6b50 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb0fcb01a776883e server_cookie=ffb7a534e4a3b8c2 in_seq=0 out_seq=0 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.530+0000 7f3de17fa640 1 -- 172.21.15.31:0/361157255 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631da14b1f0 con 0x5631da1f4700 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.530+0000 7f3de17fa640 10 client.27287 handle_command_reply: tid=0 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.530+0000 7f3e02c5a740 1 client.27287 shutdown 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3e02c5a740 1 -- 172.21.15.31:0/361157255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 msgr2=0x5631da1e1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1e1710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009c70 tx=0x7f3df40039c0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3e02c5a740 1 -- 172.21.15.31:0/361157255 shutdown_connections 2024-04-25T02:14:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631da1f4700 0x5631da1f6b50 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1e1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1e1710 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/361157255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1e11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3e02c5a740 1 -- 172.21.15.31:0/361157255 >> 172.21.15.31:0/361157255 conn(0x5631da1d5560 msgr2=0x5631da1db360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3de17fa640 0 client.27287 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3e02c5a740 1 -- 172.21.15.31:0/361157255 shutdown_connections 2024-04-25T02:14:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3e02c5a740 1 -- 172.21.15.31:0/361157255 wait complete. 2024-04-25T02:14:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.531+0000 7f3e02c5a740 20 client.27287 trim_cache size 0 max 16384 2024-04-25T02:14:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3e02c5a740 1 Processor -- start 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3e02c5a740 1 -- start start 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1e3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1efe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1fc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3e02c5a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3df81af470 con 0x5631da1d9240 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3e02c5a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3df81b0fb0 con 0x5631da1d9b60 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3e02c5a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3df81afb40 con 0x5631da1e4230 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3dfd4ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1e3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3dfdceb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1efe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3dfd4ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1e3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33604/0 (socket says 172.21.15.31:33604) 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3dfd4ea640 1 -- 172.21.15.31:0/583564630 learned_addr learned my addr 172.21.15.31:0/583564630 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.554+0000 7f3dfdceb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1efe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:46934/0 (socket says 172.21.15.31:46934) 2024-04-25T02:14:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.555+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/583564630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1fc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.555+0000 7f3dfdceb640 1 -- 172.21.15.31:0/583564630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 msgr2=0x5631da1fc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.555+0000 7f3dfdceb640 1 --2- 172.21.15.31:0/583564630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1fc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.555+0000 7f3dfdceb640 1 -- 172.21.15.31:0/583564630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 msgr2=0x5631da1e3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.555+0000 7f3dfdceb640 1 --2- 172.21.15.31:0/583564630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1e3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.555+0000 7f3dfdceb640 1 -- 172.21.15.31:0/583564630 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631da1d9040 con 0x5631da1d9b60 2024-04-25T02:14:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.555+0000 7f3dfdceb640 1 --2- 172.21.15.31:0/583564630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1efe20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x5631da1e0a30 tx=0x7f3df4003f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1bb0af7f6b08520 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.555+0000 7f3de37fe640 1 -- 172.21.15.31:0/583564630 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df4028070 con 0x5631da1d9b60 2024-04-25T02:14:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.555+0000 7f3de37fe640 1 -- 172.21.15.31:0/583564630 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3df4010d70 con 0x5631da1d9b60 2024-04-25T02:14:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.555+0000 7f3de37fe640 1 -- 172.21.15.31:0/583564630 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df4011bf0 con 0x5631da1d9b60 2024-04-25T02:14:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.556+0000 7f3e02c5a740 1 -- 172.21.15.31:0/583564630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 msgr2=0x5631da1efe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.556+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/583564630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1efe20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x5631da1e0a30 tx=0x7f3df4003f00 comp rx=0 tx=0).stop 2024-04-25T02:14:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.556+0000 7f3e02c5a740 1 -- 172.21.15.31:0/583564630 shutdown_connections 2024-04-25T02:14:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.556+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/583564630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1fc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.557+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/583564630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1d9b60 0x5631da1efe20 secure :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0x5631da1e0a30 tx=0x7f3df4003f00 comp rx=0 tx=0).stop 2024-04-25T02:14:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.557+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/583564630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1e3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.557+0000 7f3e02c5a740 1 -- 172.21.15.31:0/583564630 >> 172.21.15.31:0/583564630 conn(0x5631da1d5560 msgr2=0x5631da1d5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.557+0000 7f3e02c5a740 1 -- 172.21.15.31:0/583564630 shutdown_connections 2024-04-25T02:14:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.558+0000 7f3e02c5a740 1 -- 172.21.15.31:0/583564630 wait complete. 2024-04-25T02:14:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.558+0000 7f3e02c5a740 1 -- start start 2024-04-25T02:14:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.558+0000 7f3e02c5a740 1 Processor -- start 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.559+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1ec330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.559+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1ec870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.559+0000 7f3e02c5a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1dd440 0x5631da1dd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.559+0000 7f3e02c5a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3df81b08e0 con 0x5631da1d9240 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.559+0000 7f3e02c5a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3df81b0210 con 0x5631da1dd440 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.559+0000 7f3e02c5a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3df81ad470 con 0x5631da1e4230 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.559+0000 7f3e02c5a740 10 client.0 _ll_register_callbacks cb 0x7f3df4004080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.559+0000 7f3dfd4ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1ec330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.559+0000 7f3dfd4ea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1ec330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33614/0 (socket says 172.21.15.31:33614) 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.559+0000 7f3dfd4ea640 1 -- 172.21.15.31:0/3161799133 learned_addr learned my addr 172.21.15.31:0/3161799133 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.559+0000 7f3dfdceb640 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1ec870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1dd440 0x5631da1dd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3dfd4ea640 1 -- 172.21.15.31:0/3161799133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 msgr2=0x5631da1ec870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1ec870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3dfd4ea640 1 -- 172.21.15.31:0/3161799133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1dd440 msgr2=0x5631da1dd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1dd440 0x5631da1dd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3dfd4ea640 1 -- 172.21.15.31:0/3161799133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631da1dde00 con 0x5631da1d9240 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3dfdceb640 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1ec870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3dfd4ea640 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1ec330 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f3de8035040 tx=0x7f3de8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967e338e14def6f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3de17fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3de17fa640 1 -- 172.21.15.31:0/3161799133 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8037040 con 0x5631da1d9240 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3e02c5a740 1 -- 172.21.15.31:0/3161799133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3df4010d70 con 0x5631da1d9240 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3de17fa640 1 -- 172.21.15.31:0/3161799133 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8484+0+0 (secure 0 0 0) 0x7f3de8004240 con 0x5631da1d9240 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3de17fa640 1 -- 172.21.15.31:0/3161799133 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f3de8004440 con 0x5631da1d9240 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3e02c5a740 10 client.17736 fetch_fsmap learned FSMap version 49 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.560+0000 7f3e02c5a740 1 -- 172.21.15.31:0/3161799133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5631da25a940 con 0x5631da1d9240 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.561+0000 7f3de17fa640 1 -- 172.21.15.31:0/3161799133 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3df4010d70 con 0x5631da1d9240 2024-04-25T02:14:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.561+0000 7f3de17fa640 1 -- 172.21.15.31:0/3161799133 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f3de803d070 con 0x5631da1d9240 2024-04-25T02:14:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.561+0000 7f3e02c5a740 10 client.17736 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:14:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.561+0000 7f3e02c5a740 10 client.17736 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:14:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.561+0000 7f3e02c5a740 20 client.17736 populate_metadata read hostname 'smithi031' 2024-04-25T02:14:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.561+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631da25ac30 0x5631da25d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.561+0000 7f3e02c5a740 4 client.17736 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:14:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.561+0000 7f3e02c5a740 1 -- 172.21.15.31:0/3161799133 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5631da145540 con 0x5631da25ac30 2024-04-25T02:14:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.562+0000 7f3dfdceb640 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631da25ac30 0x5631da25d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.562+0000 7f3de17fa640 10 client.17736 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.562+0000 7f3dfdceb640 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631da25ac30 0x5631da25d020 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48c953148fb6d2af server_cookie=54ce1be3fbdb197e in_seq=0 out_seq=0 2024-04-25T02:14:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.630+0000 7f3de17fa640 1 -- 172.21.15.31:0/3161799133 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5631da145540 con 0x5631da25ac30 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.630+0000 7f3de17fa640 10 client.17736 handle_command_reply: tid=0 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.630+0000 7f3e02c5a740 1 client.17736 shutdown 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.630+0000 7f3e02c5a740 1 -- 172.21.15.31:0/3161799133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 msgr2=0x5631da1ec330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.630+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1ec330 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f3de8035040 tx=0x7f3de8000bc0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3e02c5a740 1 -- 172.21.15.31:0/3161799133 shutdown_connections 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5631da25ac30 0x5631da25d020 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631da1dd440 0x5631da1dd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3de17fa640 0 client.17736 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631da1e4230 0x5631da1ec870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3e02c5a740 1 --2- 172.21.15.31:0/3161799133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da1d9240 0x5631da1ec330 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3e02c5a740 1 -- 172.21.15.31:0/3161799133 >> 172.21.15.31:0/3161799133 conn(0x5631da1d5560 msgr2=0x5631da1dc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3e02c5a740 1 -- 172.21.15.31:0/3161799133 shutdown_connections 2024-04-25T02:14:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3e02c5a740 1 -- 172.21.15.31:0/3161799133 wait complete. 2024-04-25T02:14:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3e02c5a740 20 client.17736 trim_cache size 0 max 16384 2024-04-25T02:14:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:14:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:14:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:14:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:14:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3dfff4f640 1 -- 172.21.15.31:0/1480336435 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3de40a2e50 msgr2=0x7f3de40a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3dfff4f640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3de40a2e50 0x7f3de40a5310 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3df0098480 tx=0x7f3df00cfde0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3dfff4f640 1 -- 172.21.15.31:0/1480336435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3df8107760 msgr2=0x7f3df80744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.631+0000 7f3dfff4f640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3df8107760 0x7f3df80744c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009fd0 tx=0x7f3df000ef10 comp rx=0 tx=0).stop 2024-04-25T02:14:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.632+0000 7f3dfff4f640 1 -- 172.21.15.31:0/1480336435 shutdown_connections 2024-04-25T02:14:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.632+0000 7f3dfff4f640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3de40a2e50 0x7f3de40a5310 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.632+0000 7f3dfff4f640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3df8107760 0x7f3df80744c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.632+0000 7f3dfff4f640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3df8102110 0x7f3df8073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.632+0000 7f3dfff4f640 1 --2- 172.21.15.31:0/1480336435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3df8101750 0x7f3df8073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:22.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.632+0000 7f3dfff4f640 1 -- 172.21.15.31:0/1480336435 >> 172.21.15.31:0/1480336435 conn(0x7f3df80fd240 msgr2=0x7f3df80fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:22.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.632+0000 7f3dfff4f640 1 -- 172.21.15.31:0/1480336435 shutdown_connections 2024-04-25T02:14:22.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:22.632+0000 7f3dfff4f640 1 -- 172.21.15.31:0/1480336435 wait complete. 2024-04-25T02:14:23.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 63797 8.89 MB/sec execute 267 sec latency 75.314 ms 2024-04-25T02:14:24.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 63970 8.90 MB/sec execute 268 sec latency 61.464 ms 2024-04-25T02:14:24.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:24 smithi040 ceph-mon[32418]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.0 MiB/s wr, 263 op/s 2024-04-25T02:14:24.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:24 smithi134 ceph-mon[33098]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.0 MiB/s wr, 263 op/s 2024-04-25T02:14:24.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:24 smithi031 ceph-mon[29204]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.0 MiB/s wr, 263 op/s 2024-04-25T02:14:25.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 64119 8.88 MB/sec execute 269 sec latency 58.093 ms 2024-04-25T02:14:26.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 64256 8.86 MB/sec execute 270 sec latency 52.226 ms 2024-04-25T02:14:26.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:26 smithi040 ceph-mon[32418]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 8.0 MiB/s wr, 376 op/s 2024-04-25T02:14:26.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:26 smithi134 ceph-mon[33098]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 8.0 MiB/s wr, 376 op/s 2024-04-25T02:14:26.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:26 smithi031 ceph-mon[29204]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 8.0 MiB/s wr, 376 op/s 2024-04-25T02:14:27.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 64412 8.85 MB/sec execute 271 sec latency 59.982 ms 2024-04-25T02:14:28.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 64542 8.83 MB/sec execute 272 sec latency 71.264 ms 2024-04-25T02:14:28.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:28 smithi040 ceph-mon[32418]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 233 op/s 2024-04-25T02:14:28.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:28 smithi134 ceph-mon[33098]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 233 op/s 2024-04-25T02:14:28.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:28 smithi031 ceph-mon[29204]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 233 op/s 2024-04-25T02:14:29.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 64687 8.81 MB/sec execute 273 sec latency 69.144 ms 2024-04-25T02:14:30.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 64884 8.83 MB/sec execute 274 sec latency 105.971 ms 2024-04-25T02:14:30.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:30 smithi040 ceph-mon[32418]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 233 op/s 2024-04-25T02:14:30.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:30 smithi134 ceph-mon[33098]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 233 op/s 2024-04-25T02:14:30.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:30 smithi031 ceph-mon[29204]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 233 op/s 2024-04-25T02:14:31.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 65119 8.85 MB/sec execute 275 sec latency 59.015 ms 2024-04-25T02:14:32.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 65289 8.85 MB/sec execute 276 sec latency 84.105 ms 2024-04-25T02:14:32.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:32 smithi040 ceph-mon[32418]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 346 op/s 2024-04-25T02:14:32.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:32 smithi134 ceph-mon[33098]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 346 op/s 2024-04-25T02:14:32.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:32 smithi031 ceph-mon[29204]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 346 op/s 2024-04-25T02:14:33.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 65411 8.86 MB/sec execute 277 sec latency 290.737 ms 2024-04-25T02:14:34.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 65567 8.87 MB/sec execute 278 sec latency 93.937 ms 2024-04-25T02:14:34.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:34 smithi040 ceph-mon[32418]: pgmap v632: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 235 op/s 2024-04-25T02:14:34.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:34 smithi134 ceph-mon[33098]: pgmap v632: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 235 op/s 2024-04-25T02:14:34.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:34 smithi031 ceph-mon[29204]: pgmap v632: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 235 op/s 2024-04-25T02:14:35.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 65741 8.87 MB/sec execute 279 sec latency 100.924 ms 2024-04-25T02:14:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.755+0000 7f399a0a5640 1 -- 172.21.15.31:0/371254756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39941029d0 msgr2=0x7f3994102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.755+0000 7f399a0a5640 1 --2- 172.21.15.31:0/371254756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39941029d0 0x7f3994102db0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f3984009900 tx=0x7f398402a480 comp rx=0 tx=0).stop 2024-04-25T02:14:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.755+0000 7f399a0a5640 1 -- 172.21.15.31:0/371254756 shutdown_connections 2024-04-25T02:14:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.755+0000 7f399a0a5640 1 --2- 172.21.15.31:0/371254756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f399410bda0 0x7f399410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.755+0000 7f399a0a5640 1 --2- 172.21.15.31:0/371254756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39941032f0 0x7f399410b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.755+0000 7f399a0a5640 1 --2- 172.21.15.31:0/371254756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39941029d0 0x7f3994102db0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.755+0000 7f399a0a5640 1 -- 172.21.15.31:0/371254756 >> 172.21.15.31:0/371254756 conn(0x7f3994077760 msgr2=0x7f3994077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.755+0000 7f399a0a5640 1 -- 172.21.15.31:0/371254756 shutdown_connections 2024-04-25T02:14:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.755+0000 7f399a0a5640 1 -- 172.21.15.31:0/371254756 wait complete. 2024-04-25T02:14:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f399a0a5640 1 Processor -- start 2024-04-25T02:14:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f399a0a5640 1 -- start start 2024-04-25T02:14:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f399a0a5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39941029d0 0x7f3994197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f399a0a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39941032f0 0x7f39941983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f399a0a5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f399410bda0 0x7f399419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f399a0a5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39941119e0 con 0x7f39941032f0 2024-04-25T02:14:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f399a0a5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3994111860 con 0x7f399410bda0 2024-04-25T02:14:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f399a0a5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3994111b60 con 0x7f39941029d0 2024-04-25T02:14:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f3992ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39941032f0 0x7f39941983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f3992ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39941032f0 0x7f39941983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51486/0 (socket says 172.21.15.31:51486) 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f3992ffd640 1 -- 172.21.15.31:0/4177594299 learned_addr learned my addr 172.21.15.31:0/4177594299 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.756+0000 7f39937fe640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39941029d0 0x7f3994197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f3993fff640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f399410bda0 0x7f399419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f3992ffd640 1 -- 172.21.15.31:0/4177594299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39941029d0 msgr2=0x7f3994197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f3992ffd640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39941029d0 0x7f3994197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f3992ffd640 1 -- 172.21.15.31:0/4177594299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f399410bda0 msgr2=0x7f399419c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f3992ffd640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f399410bda0 0x7f399419c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f3992ffd640 1 -- 172.21.15.31:0/4177594299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399419ce90 con 0x7f39941032f0 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f3992ffd640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39941032f0 0x7f39941983e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f398800d950 tx=0x7f398800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f3990ff9640 1 -- 172.21.15.31:0/4177594299 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988017030 con 0x7f39941032f0 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f3990ff9640 1 -- 172.21.15.31:0/4177594299 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3988004020 con 0x7f39941032f0 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f399a0a5640 1 -- 172.21.15.31:0/4177594299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399419d180 con 0x7f39941032f0 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f3990ff9640 1 -- 172.21.15.31:0/4177594299 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39880047a0 con 0x7f39941032f0 2024-04-25T02:14:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.757+0000 7f399a0a5640 1 -- 172.21.15.31:0/4177594299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3994100550 con 0x7f39941032f0 2024-04-25T02:14:35.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.761+0000 7f3990ff9640 1 -- 172.21.15.31:0/4177594299 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f39880041d0 con 0x7f39941032f0 2024-04-25T02:14:35.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.761+0000 7f399cdb9740 1 Processor -- start 2024-04-25T02:14:35.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.761+0000 7f399cdb9740 1 -- start start 2024-04-25T02:14:35.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 0x55f79633d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796347cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f796348230 0x55f79634a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f399cdb9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f7962a1cb0 con 0x55f79633db60 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f399cdb9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f79624a5b0 con 0x55f79633d240 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f399cdb9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f79629dc70 con 0x55f796348230 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f3990ff9640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f397c0a2e20 0x7f397c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f39937fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796347cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f39937fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796347cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51502/0 (socket says 172.21.15.31:51502) 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f39937fe640 1 -- 172.21.15.31:0/3071597259 learned_addr learned my addr 172.21.15.31:0/3071597259 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f3992ffd640 1 --2- 172.21.15.31:0/3071597259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f796348230 0x55f79634a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f3993fff640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f397c0a2e20 0x7f397c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f3993fff640 1 --2- 172.21.15.31:0/3071597259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 0x55f79633d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f3990ff9640 1 -- 172.21.15.31:0/4177594299 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8484+0+0 (secure 0 0 0) 0x7f39880ced90 con 0x7f39941032f0 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f39937fe640 1 -- 172.21.15.31:0/3071597259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f796348230 msgr2=0x55f79634a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f39937fe640 1 --2- 172.21.15.31:0/3071597259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f796348230 0x55f79634a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f39937fe640 1 -- 172.21.15.31:0/3071597259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 msgr2=0x55f79633d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f39937fe640 1 --2- 172.21.15.31:0/3071597259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 0x55f79633d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f39937fe640 1 -- 172.21.15.31:0/3071597259 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f795fb6210 con 0x55f79633db60 2024-04-25T02:14:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.762+0000 7f3993fff640 1 --2- 172.21.15.31:0/3071597259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 0x55f79633d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.763+0000 7f3993fff640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f397c0a2e20 0x7f397c0a52e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f399406aac0 tx=0x7f398c004a40 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:35.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.763+0000 7f39937fe640 1 --2- 172.21.15.31:0/3071597259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796347cf0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f3984009900 tx=0x7f3984032bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4166b9ac78ed5d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.763+0000 7f3979ffb640 1 -- 172.21.15.31:0/3071597259 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984040070 con 0x55f79633db60 2024-04-25T02:14:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.763+0000 7f3979ffb640 1 -- 172.21.15.31:0/3071597259 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3984004030 con 0x55f79633db60 2024-04-25T02:14:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.763+0000 7f3979ffb640 1 -- 172.21.15.31:0/3071597259 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984035040 con 0x55f79633db60 2024-04-25T02:14:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.764+0000 7f399cdb9740 1 -- 172.21.15.31:0/3071597259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 msgr2=0x55f796347cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.764+0000 7f399cdb9740 1 --2- 172.21.15.31:0/3071597259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796347cf0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f3984009900 tx=0x7f3984032bb0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.764+0000 7f399cdb9740 1 -- 172.21.15.31:0/3071597259 shutdown_connections 2024-04-25T02:14:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.764+0000 7f399cdb9740 1 --2- 172.21.15.31:0/3071597259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f796348230 0x55f79634a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.764+0000 7f399cdb9740 1 --2- 172.21.15.31:0/3071597259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796347cf0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.764+0000 7f399cdb9740 1 --2- 172.21.15.31:0/3071597259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 0x55f79633d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.764+0000 7f399cdb9740 1 -- 172.21.15.31:0/3071597259 >> 172.21.15.31:0/3071597259 conn(0x55f796339560 msgr2=0x55f796339940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.764+0000 7f399cdb9740 1 -- 172.21.15.31:0/3071597259 shutdown_connections 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.764+0000 7f399cdb9740 1 -- 172.21.15.31:0/3071597259 wait complete. 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.764+0000 7f399cdb9740 1 -- start start 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f399cdb9740 1 Processor -- start 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 0x55f796345120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796345660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f796348230 0x55f796345ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f399cdb9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f79634afc0 con 0x55f79633db60 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f399cdb9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f79625db50 con 0x55f79633d240 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f399cdb9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f79634b140 con 0x55f796348230 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f3992ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796345660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f399cdb9740 10 client.0 _ll_register_callbacks cb 0x55f796298ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f3992ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796345660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51504/0 (socket says 172.21.15.31:51504) 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f3992ffd640 1 -- 172.21.15.31:0/61987721 learned_addr learned my addr 172.21.15.31:0/61987721 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f39937fe640 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 0x55f796345120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.765+0000 7f3993fff640 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f796348230 0x55f796345ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f3992ffd640 1 -- 172.21.15.31:0/61987721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f796348230 msgr2=0x55f796345ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f3992ffd640 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f796348230 0x55f796345ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f3992ffd640 1 -- 172.21.15.31:0/61987721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 msgr2=0x55f796345120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f3992ffd640 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 0x55f796345120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f3992ffd640 1 -- 172.21.15.31:0/61987721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f796082620 con 0x55f79633db60 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f39937fe640 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 0x55f796345120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f3992ffd640 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796345660 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f3988097ca0 tx=0x7f398809ae10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9544e7a6e877cfee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f3972ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f3972ffd640 1 -- 172.21.15.31:0/61987721 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398809b2d0 con 0x55f79633db60 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f3972ffd640 1 -- 172.21.15.31:0/61987721 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8484+0+0 (secure 0 0 0) 0x7f3988097940 con 0x55f79633db60 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f399cdb9740 1 -- 172.21.15.31:0/61987721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3984004030 con 0x55f79633db60 2024-04-25T02:14:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f3972ffd640 1 -- 172.21.15.31:0/61987721 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f398809f680 con 0x55f79633db60 2024-04-25T02:14:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f399cdb9740 10 client.17748 fetch_fsmap learned FSMap version 49 2024-04-25T02:14:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.766+0000 7f399cdb9740 1 -- 172.21.15.31:0/61987721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55f796357110 con 0x55f79633db60 2024-04-25T02:14:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.767+0000 7f3972ffd640 1 -- 172.21.15.31:0/61987721 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39880a3f40 con 0x55f79633db60 2024-04-25T02:14:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.767+0000 7f3972ffd640 1 -- 172.21.15.31:0/61987721 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f398809a8b0 con 0x55f79633db60 2024-04-25T02:14:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.767+0000 7f399cdb9740 10 client.17748 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.767+0000 7f399cdb9740 10 client.17748 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.767+0000 7f399cdb9740 20 client.17748 populate_metadata read hostname 'smithi031' 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.767+0000 7f399cdb9740 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f796358620 0x55f79635aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.767+0000 7f399cdb9740 4 client.17748 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.767+0000 7f399cdb9740 1 -- 172.21.15.31:0/61987721 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3984004030 con 0x55f796358620 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.767+0000 7f39937fe640 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f796358620 0x55f79635aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.768+0000 7f3972ffd640 10 client.17748 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.768+0000 7f39937fe640 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f796358620 0x55f79635aa10 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d5a9818180542d1 server_cookie=ef27412e37b977a in_seq=0 out_seq=0 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.768+0000 7f3972ffd640 1 -- 172.21.15.31:0/61987721 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f3984004030 con 0x55f796358620 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.768+0000 7f3972ffd640 10 client.17748 handle_command_reply: tid=0 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.768+0000 7f399cdb9740 1 client.17748 shutdown 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f399cdb9740 1 -- 172.21.15.31:0/61987721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 msgr2=0x55f796345660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f399cdb9740 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796345660 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f3988097ca0 tx=0x7f398809ae10 comp rx=0 tx=0).stop 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f399cdb9740 1 -- 172.21.15.31:0/61987721 shutdown_connections 2024-04-25T02:14:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f399cdb9740 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f796358620 0x55f79635aa10 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f399cdb9740 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f796348230 0x55f796345ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f399cdb9740 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f79633db60 0x55f796345660 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f399cdb9740 1 --2- 172.21.15.31:0/61987721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f79633d240 0x55f796345120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f399cdb9740 1 -- 172.21.15.31:0/61987721 >> 172.21.15.31:0/61987721 conn(0x55f796339560 msgr2=0x55f79633f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f3972ffd640 0 client.17748 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f399cdb9740 1 -- 172.21.15.31:0/61987721 shutdown_connections 2024-04-25T02:14:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f399cdb9740 1 -- 172.21.15.31:0/61987721 wait complete. 2024-04-25T02:14:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.769+0000 7f399cdb9740 20 client.17748 trim_cache size 0 max 16384 2024-04-25T02:14:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f399cdb9740 1 Processor -- start 2024-04-25T02:14:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f399cdb9740 1 -- start start 2024-04-25T02:14:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796353e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f796348230 0x55f79633f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f796340ae0 0x55f7963602e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f399cdb9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39941ab030 con 0x55f796340ae0 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f399cdb9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f39941acb70 con 0x55f796348230 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f399cdb9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f39941ab700 con 0x55f79633d240 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f39937fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796353e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f3992ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f796348230 0x55f79633f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f39937fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796353e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:45574/0 (socket says 172.21.15.31:45574) 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f39937fe640 1 -- 172.21.15.31:0/2405856466 learned_addr learned my addr 172.21.15.31:0/2405856466 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.791+0000 7f3993fff640 1 --2- 172.21.15.31:0/2405856466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f796340ae0 0x55f7963602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.792+0000 7f3992ffd640 1 -- 172.21.15.31:0/2405856466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 msgr2=0x55f796353e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.792+0000 7f3992ffd640 1 --2- 172.21.15.31:0/2405856466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796353e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.792+0000 7f3992ffd640 1 -- 172.21.15.31:0/2405856466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f796340ae0 msgr2=0x55f7963602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.792+0000 7f3992ffd640 1 --2- 172.21.15.31:0/2405856466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f796340ae0 0x55f7963602e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.792+0000 7f3992ffd640 1 -- 172.21.15.31:0/2405856466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f79634a990 con 0x55f796348230 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.792+0000 7f39937fe640 1 --2- 172.21.15.31:0/2405856466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796353e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.792+0000 7f3992ffd640 1 --2- 172.21.15.31:0/2405856466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f796348230 0x55f79633f9a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55f7963461b0 tx=0x7f398809af40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f97acf55f5a5395 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.792+0000 7f39797fa640 1 -- 172.21.15.31:0/2405856466 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398809b2d0 con 0x55f796348230 2024-04-25T02:14:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.792+0000 7f39797fa640 1 -- 172.21.15.31:0/2405856466 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39880a3f40 con 0x55f796348230 2024-04-25T02:14:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.792+0000 7f39797fa640 1 -- 172.21.15.31:0/2405856466 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398809ac30 con 0x55f796348230 2024-04-25T02:14:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.793+0000 7f399cdb9740 1 -- 172.21.15.31:0/2405856466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f796348230 msgr2=0x55f79633f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.793+0000 7f399cdb9740 1 --2- 172.21.15.31:0/2405856466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f796348230 0x55f79633f9a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55f7963461b0 tx=0x7f398809af40 comp rx=0 tx=0).stop 2024-04-25T02:14:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.793+0000 7f399cdb9740 1 -- 172.21.15.31:0/2405856466 shutdown_connections 2024-04-25T02:14:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.793+0000 7f399cdb9740 1 --2- 172.21.15.31:0/2405856466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f796340ae0 0x55f7963602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.793+0000 7f399cdb9740 1 --2- 172.21.15.31:0/2405856466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f796348230 0x55f79633f9a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.793+0000 7f399cdb9740 1 --2- 172.21.15.31:0/2405856466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796353e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.793+0000 7f399cdb9740 1 -- 172.21.15.31:0/2405856466 >> 172.21.15.31:0/2405856466 conn(0x55f796339560 msgr2=0x55f7963385c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.793+0000 7f399cdb9740 1 -- 172.21.15.31:0/2405856466 shutdown_connections 2024-04-25T02:14:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.793+0000 7f399cdb9740 1 -- 172.21.15.31:0/2405856466 wait complete. 2024-04-25T02:14:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.793+0000 7f399cdb9740 1 -- start start 2024-04-25T02:14:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f399cdb9740 1 Processor -- start 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796350390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f796340ae0 0x55f7963508d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f399cdb9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f796348230 0x55f79634d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f399cdb9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39941ac4a0 con 0x55f796348230 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f399cdb9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f39941abdd0 con 0x55f796340ae0 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f399cdb9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f39941a9030 con 0x55f79633d240 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f39937fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796350390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f3993fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f796348230 0x55f79634d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f39937fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796350390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:45590/0 (socket says 172.21.15.31:45590) 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f3992ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f796340ae0 0x55f7963508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f39937fe640 1 -- 172.21.15.31:0/3989373397 learned_addr learned my addr 172.21.15.31:0/3989373397 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.794+0000 7f399cdb9740 10 client.0 _ll_register_callbacks cb 0x7f39880046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f39937fe640 1 -- 172.21.15.31:0/3989373397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f796340ae0 msgr2=0x55f7963508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f39937fe640 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f796340ae0 0x55f7963508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f39937fe640 1 -- 172.21.15.31:0/3989373397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f796348230 msgr2=0x55f79634d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f39937fe640 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f796348230 0x55f79634d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f39937fe640 1 -- 172.21.15.31:0/3989373397 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f79634dbd0 con 0x55f79633d240 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f3993fff640 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f796348230 0x55f79634d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f39937fe640 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796350390 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f398402a9a0 tx=0x7f398402a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b846bff978c4a6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f3972ffd640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f3972ffd640 1 -- 172.21.15.31:0/3989373397 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984035040 con 0x55f79633d240 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f399cdb9740 1 -- 172.21.15.31:0/3989373397 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39880a3f40 con 0x55f79633d240 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f3972ffd640 1 -- 172.21.15.31:0/3989373397 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8484+0+0 (secure 0 0 0) 0x7f39840316a0 con 0x55f79633d240 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f3972ffd640 1 -- 172.21.15.31:0/3989373397 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f3984040040 con 0x55f79633d240 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f399cdb9740 10 client.27094 fetch_fsmap learned FSMap version 49 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.795+0000 7f399cdb9740 1 -- 172.21.15.31:0/3989373397 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55f7963bf710 con 0x55f79633d240 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.796+0000 7f3972ffd640 1 -- 172.21.15.31:0/3989373397 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39880a3f40 con 0x55f79633d240 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.796+0000 7f3972ffd640 1 -- 172.21.15.31:0/3989373397 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f3984060020 con 0x55f79633d240 2024-04-25T02:14:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.796+0000 7f399cdb9740 10 client.27094 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:14:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.796+0000 7f399cdb9740 10 client.27094 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:14:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.796+0000 7f399cdb9740 20 client.27094 populate_metadata read hostname 'smithi031' 2024-04-25T02:14:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.796+0000 7f399cdb9740 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f7963bfaa0 0x55f7963c1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.796+0000 7f399cdb9740 4 client.27094 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:14:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.796+0000 7f399cdb9740 1 -- 172.21.15.31:0/3989373397 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f7962af1f0 con 0x55f7963bfaa0 2024-04-25T02:14:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.796+0000 7f3992ffd640 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f7963bfaa0 0x55f7963c1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.797+0000 7f3992ffd640 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f7963bfaa0 0x55f7963c1e90 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25fbc3ddd001a0c9 server_cookie=26bd48660ec3ca85 in_seq=0 out_seq=0 2024-04-25T02:14:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.797+0000 7f3972ffd640 10 client.27094 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.856+0000 7f3972ffd640 1 -- 172.21.15.31:0/3989373397 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f7962af1f0 con 0x55f7963bfaa0 2024-04-25T02:14:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.856+0000 7f3972ffd640 10 client.27094 handle_command_reply: tid=0 2024-04-25T02:14:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.856+0000 7f399cdb9740 1 client.27094 shutdown 2024-04-25T02:14:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f399cdb9740 1 -- 172.21.15.31:0/3989373397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 msgr2=0x55f796350390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f399cdb9740 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796350390 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f398402a9a0 tx=0x7f398402a480 comp rx=0 tx=0).stop 2024-04-25T02:14:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f399cdb9740 1 -- 172.21.15.31:0/3989373397 shutdown_connections 2024-04-25T02:14:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f399cdb9740 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f7963bfaa0 0x55f7963c1e90 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f399cdb9740 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f796348230 0x55f79634d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f399cdb9740 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f796340ae0 0x55f7963508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f399cdb9740 1 --2- 172.21.15.31:0/3989373397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f79633d240 0x55f796350390 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f3972ffd640 0 client.27094 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f399cdb9740 1 -- 172.21.15.31:0/3989373397 >> 172.21.15.31:0/3989373397 conn(0x55f796339560 msgr2=0x55f7963385c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f399cdb9740 1 -- 172.21.15.31:0/3989373397 shutdown_connections 2024-04-25T02:14:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f399cdb9740 1 -- 172.21.15.31:0/3989373397 wait complete. 2024-04-25T02:14:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.857+0000 7f399cdb9740 20 client.27094 trim_cache size 0 max 16384 2024-04-25T02:14:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:14:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:14:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:14:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:14:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.858+0000 7f399a0a5640 1 -- 172.21.15.31:0/4177594299 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f397c0a2e20 msgr2=0x7f397c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.858+0000 7f399a0a5640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f397c0a2e20 0x7f397c0a52e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f399406aac0 tx=0x7f398c004a40 comp rx=0 tx=0).stop 2024-04-25T02:14:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.858+0000 7f399a0a5640 1 -- 172.21.15.31:0/4177594299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39941032f0 msgr2=0x7f39941983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.858+0000 7f399a0a5640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39941032f0 0x7f39941983e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f398800d950 tx=0x7f398800de10 comp rx=0 tx=0).stop 2024-04-25T02:14:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.858+0000 7f399a0a5640 1 -- 172.21.15.31:0/4177594299 shutdown_connections 2024-04-25T02:14:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.858+0000 7f399a0a5640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f397c0a2e20 0x7f397c0a52e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.858+0000 7f399a0a5640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f399410bda0 0x7f399419c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.858+0000 7f399a0a5640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39941032f0 0x7f39941983e0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.858+0000 7f399a0a5640 1 --2- 172.21.15.31:0/4177594299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39941029d0 0x7f3994197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.858+0000 7f399a0a5640 1 -- 172.21.15.31:0/4177594299 >> 172.21.15.31:0/4177594299 conn(0x7f3994077760 msgr2=0x7f39940fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.859+0000 7f399a0a5640 1 -- 172.21.15.31:0/4177594299 shutdown_connections 2024-04-25T02:14:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:35.859+0000 7f399a0a5640 1 -- 172.21.15.31:0/4177594299 wait complete. 2024-04-25T02:14:36.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 65896 8.86 MB/sec execute 280 sec latency 111.487 ms 2024-04-25T02:14:36.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:36 smithi040 ceph-mon[32418]: pgmap v633: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.2 MiB/s wr, 353 op/s 2024-04-25T02:14:36.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:36 smithi134 ceph-mon[33098]: pgmap v633: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.2 MiB/s wr, 353 op/s 2024-04-25T02:14:36.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:36 smithi031 ceph-mon[29204]: pgmap v633: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.2 MiB/s wr, 353 op/s 2024-04-25T02:14:37.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 66062 8.85 MB/sec execute 281 sec latency 67.871 ms 2024-04-25T02:14:38.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 66217 8.84 MB/sec execute 282 sec latency 155.089 ms 2024-04-25T02:14:38.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:38 smithi040 ceph-mon[32418]: pgmap v634: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:14:38.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:38 smithi134 ceph-mon[33098]: pgmap v634: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:14:38.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:38 smithi031 ceph-mon[29204]: pgmap v634: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:14:39.101 INFO:tasks.workunit.client.0.smithi031.stdout: 10 66360 8.83 MB/sec execute 283 sec latency 78.415 ms 2024-04-25T02:14:40.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 66541 8.83 MB/sec execute 284 sec latency 68.852 ms 2024-04-25T02:14:40.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:40 smithi040 ceph-mon[32418]: pgmap v635: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:14:40.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:40 smithi134 ceph-mon[33098]: pgmap v635: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:14:40.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:40 smithi031 ceph-mon[29204]: pgmap v635: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 231 op/s 2024-04-25T02:14:41.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 66752 8.85 MB/sec execute 285 sec latency 65.091 ms 2024-04-25T02:14:42.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 66944 8.86 MB/sec execute 286 sec latency 54.896 ms 2024-04-25T02:14:42.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:42 smithi040 ceph-mon[32418]: pgmap v636: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 341 op/s 2024-04-25T02:14:42.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:42 smithi134 ceph-mon[33098]: pgmap v636: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 341 op/s 2024-04-25T02:14:42.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:42 smithi031 ceph-mon[29204]: pgmap v636: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 341 op/s 2024-04-25T02:14:43.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 67115 8.88 MB/sec execute 287 sec latency 63.829 ms 2024-04-25T02:14:44.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 67289 8.90 MB/sec execute 288 sec latency 62.723 ms 2024-04-25T02:14:44.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:44 smithi040 ceph-mon[32418]: pgmap v637: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 228 op/s 2024-04-25T02:14:44.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:44 smithi134 ceph-mon[33098]: pgmap v637: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 228 op/s 2024-04-25T02:14:44.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:44 smithi031 ceph-mon[29204]: pgmap v637: 97 pgs: 97 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 228 op/s 2024-04-25T02:14:45.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 67454 8.89 MB/sec execute 289 sec latency 60.378 ms 2024-04-25T02:14:46.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 67604 8.88 MB/sec execute 290 sec latency 59.080 ms 2024-04-25T02:14:46.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:46 smithi040 ceph-mon[32418]: pgmap v638: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.5 MiB/s wr, 353 op/s 2024-04-25T02:14:46.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:46 smithi134 ceph-mon[33098]: pgmap v638: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.5 MiB/s wr, 353 op/s 2024-04-25T02:14:46.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:46 smithi031 ceph-mon[29204]: pgmap v638: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.5 MiB/s wr, 353 op/s 2024-04-25T02:14:47.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 67745 8.86 MB/sec execute 291 sec latency 55.748 ms 2024-04-25T02:14:48.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 67912 8.86 MB/sec execute 292 sec latency 253.203 ms 2024-04-25T02:14:48.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:48 smithi040 ceph-mon[32418]: pgmap v639: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 234 op/s 2024-04-25T02:14:48.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:48 smithi134 ceph-mon[33098]: pgmap v639: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 234 op/s 2024-04-25T02:14:48.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:48 smithi031 ceph-mon[29204]: pgmap v639: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 234 op/s 2024-04-25T02:14:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.997+0000 7fc9dba45640 1 -- 172.21.15.31:0/2617327748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9d4106630 msgr2=0x7fc9d4106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.997+0000 7fc9dba45640 1 --2- 172.21.15.31:0/2617327748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9d4106630 0x7fc9d4106a10 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009930 tx=0x7fc9d002a420 comp rx=0 tx=0).stop 2024-04-25T02:14:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.998+0000 7fc9dba45640 1 -- 172.21.15.31:0/2617327748 shutdown_connections 2024-04-25T02:14:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.998+0000 7fc9dba45640 1 --2- 172.21.15.31:0/2617327748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d4101030 0x7fc9d410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.998+0000 7fc9dba45640 1 --2- 172.21.15.31:0/2617327748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9d4100670 0x7fc9d4100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.998+0000 7fc9dba45640 1 --2- 172.21.15.31:0/2617327748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9d4106630 0x7fc9d4106a10 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.998+0000 7fc9dba45640 1 -- 172.21.15.31:0/2617327748 >> 172.21.15.31:0/2617327748 conn(0x7fc9d40777c0 msgr2=0x7fc9d4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.998+0000 7fc9dba45640 1 -- 172.21.15.31:0/2617327748 shutdown_connections 2024-04-25T02:14:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.998+0000 7fc9dba45640 1 -- 172.21.15.31:0/2617327748 wait complete. 2024-04-25T02:14:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9dba45640 1 Processor -- start 2024-04-25T02:14:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9dba45640 1 -- start start 2024-04-25T02:14:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9dba45640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9d4100670 0x7fc9d410e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9dba45640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d4101030 0x7fc9d410e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9dba45640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9d4106630 0x7fc9d410ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9dba45640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d41119e0 con 0x7fc9d4106630 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9dba45640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d4111860 con 0x7fc9d4100670 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9dba45640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d4111b60 con 0x7fc9d4101030 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9d9fe2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9d4106630 0x7fc9d410ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9d8fe0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d4101030 0x7fc9d410e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9d9fe2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9d4106630 0x7fc9d410ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54252/0 (socket says 172.21.15.31:54252) 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9d97e1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9d4100670 0x7fc9d410e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9d9fe2640 1 -- 172.21.15.31:0/3782765628 learned_addr learned my addr 172.21.15.31:0/3782765628 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:48.999+0000 7fc9d8fe0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d4101030 0x7fc9d410e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:39320/0 (socket says 172.21.15.31:39320) 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9d9fe2640 1 -- 172.21.15.31:0/3782765628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d4101030 msgr2=0x7fc9d410e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:49.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9d9fe2640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d4101030 0x7fc9d410e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9d9fe2640 1 -- 172.21.15.31:0/3782765628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9d4100670 msgr2=0x7fc9d410e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9d9fe2640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9d4100670 0x7fc9d410e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9d9fe2640 1 -- 172.21.15.31:0/3782765628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d41a63a0 con 0x7fc9d4106630 2024-04-25T02:14:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9d97e1640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9d4100670 0x7fc9d410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9d9fe2640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9d4106630 0x7fc9d410ec30 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00eb10 tx=0x7fc9cc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9ca7fc640 1 -- 172.21.15.31:0/3782765628 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc004b20 con 0x7fc9d4106630 2024-04-25T02:14:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9dba45640 1 -- 172.21.15.31:0/3782765628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d41a6690 con 0x7fc9d4106630 2024-04-25T02:14:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9dba45640 1 -- 172.21.15.31:0/3782765628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d41a6c30 con 0x7fc9d4106630 2024-04-25T02:14:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9ca7fc640 1 -- 172.21.15.31:0/3782765628 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9cc004cc0 con 0x7fc9d4106630 2024-04-25T02:14:49.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.000+0000 7fc9ca7fc640 1 -- 172.21.15.31:0/3782765628 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc010690 con 0x7fc9d4106630 2024-04-25T02:14:49.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.003+0000 7fc9ca7fc640 1 -- 172.21.15.31:0/3782765628 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc9cc005020 con 0x7fc9d4106630 2024-04-25T02:14:49.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.003+0000 7fc9de82a740 1 Processor -- start 2024-04-25T02:14:49.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.003+0000 7fc9de82a740 1 -- start start 2024-04-25T02:14:49.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 0x556d6fe16620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe16b60 0x556d6fe20cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe21230 0x556d6fe23620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9de82a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556d6fd7acb0 con 0x556d6fe21230 2024-04-25T02:14:49.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9de82a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556d6fd235b0 con 0x556d6fe16b60 2024-04-25T02:14:49.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9de82a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556d6fd76c70 con 0x556d6fe16240 2024-04-25T02:14:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9d8fe0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 0x556d6fe16620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9d97e1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe16b60 0x556d6fe20cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9d8fe0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 0x556d6fe16620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:39330/0 (socket says 172.21.15.31:39330) 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9d8fe0640 1 -- 172.21.15.31:0/768416596 learned_addr learned my addr 172.21.15.31:0/768416596 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/768416596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe21230 0x556d6fe23620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9ca7fc640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9c00a2ef0 0x7fc9c00a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9d8fe0640 1 -- 172.21.15.31:0/768416596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe16b60 msgr2=0x556d6fe20cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/768416596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe16b60 0x556d6fe20cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9d8fe0640 1 -- 172.21.15.31:0/768416596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe21230 msgr2=0x556d6fe23620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/768416596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe21230 0x556d6fe23620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9d8fe0640 1 -- 172.21.15.31:0/768416596 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d6fa8f210 con 0x556d6fe16240 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.004+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/768416596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 0x556d6fe16620 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4009f90 tx=0x7fc9c4005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cb9e737ef0fe3e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.005+0000 7fc9bf7fe640 1 -- 172.21.15.31:0/768416596 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4004030 con 0x556d6fe16240 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.005+0000 7fc9bf7fe640 1 -- 172.21.15.31:0/768416596 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9c40054d0 con 0x556d6fe16240 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.005+0000 7fc9d9fe2640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9c00a2ef0 0x7fc9c00a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.005+0000 7fc9bf7fe640 1 -- 172.21.15.31:0/768416596 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4004cc0 con 0x556d6fe16240 2024-04-25T02:14:49.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.005+0000 7fc9ca7fc640 1 -- 172.21.15.31:0/3782765628 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8484+0+0 (secure 0 0 0) 0x7fc9cc0cf960 con 0x7fc9d4106630 2024-04-25T02:14:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9de82a740 1 -- 172.21.15.31:0/768416596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 msgr2=0x556d6fe16620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9de82a740 1 --2- 172.21.15.31:0/768416596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 0x556d6fe16620 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4009f90 tx=0x7fc9c4005850 comp rx=0 tx=0).stop 2024-04-25T02:14:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9d9fe2640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9c00a2ef0 0x7fc9c00a53b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0a4580 tx=0x7fc9cc022000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9de82a740 1 -- 172.21.15.31:0/768416596 shutdown_connections 2024-04-25T02:14:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9de82a740 1 --2- 172.21.15.31:0/768416596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe21230 0x556d6fe23620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9de82a740 1 --2- 172.21.15.31:0/768416596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe16b60 0x556d6fe20cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9de82a740 1 --2- 172.21.15.31:0/768416596 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 0x556d6fe16620 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9de82a740 1 -- 172.21.15.31:0/768416596 >> 172.21.15.31:0/768416596 conn(0x556d6fe12560 msgr2=0x556d6fe12940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9de82a740 1 -- 172.21.15.31:0/768416596 shutdown_connections 2024-04-25T02:14:49.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9de82a740 1 -- 172.21.15.31:0/768416596 wait complete. 2024-04-25T02:14:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9de82a740 1 -- start start 2024-04-25T02:14:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.006+0000 7fc9de82a740 1 Processor -- start 2024-04-25T02:14:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe16240 0x556d6fe1e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16b60 0x556d6fe1e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe21230 0x556d6fe1ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9de82a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556d6fe24020 con 0x556d6fe21230 2024-04-25T02:14:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9de82a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556d6fd36b50 con 0x556d6fe16240 2024-04-25T02:14:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9de82a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556d6fe241a0 con 0x556d6fe16b60 2024-04-25T02:14:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9d8fe0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe16240 0x556d6fe1e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9d97e1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16b60 0x556d6fe1e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9d8fe0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe21230 0x556d6fe1ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9d97e1640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16b60 0x556d6fe1e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:39336/0 (socket says 172.21.15.31:39336) 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9d97e1640 1 -- 172.21.15.31:0/3379799098 learned_addr learned my addr 172.21.15.31:0/3379799098 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9de82a740 10 client.0 _ll_register_callbacks cb 0x556d6fd71ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9d97e1640 1 -- 172.21.15.31:0/3379799098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe16240 msgr2=0x556d6fe1e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9d97e1640 1 --2- 172.21.15.31:0/3379799098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe16240 0x556d6fe1e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9d97e1640 1 -- 172.21.15.31:0/3379799098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe21230 msgr2=0x556d6fe1ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9d97e1640 1 --2- 172.21.15.31:0/3379799098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe21230 0x556d6fe1ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.007+0000 7fc9d97e1640 1 -- 172.21.15.31:0/3379799098 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d6fb5b620 con 0x556d6fe16b60 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.008+0000 7fc9d97e1640 1 --2- 172.21.15.31:0/3379799098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16b60 0x556d6fe1e6f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc9d002aa10 tx=0x7fc9d0005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57f3833108875ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.008+0000 7fc9bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.008+0000 7fc9bcff9640 1 -- 172.21.15.31:0/3379799098 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0035040 con 0x556d6fe16b60 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.008+0000 7fc9bcff9640 1 -- 172.21.15.31:0/3379799098 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8484+0+0 (secure 0 0 0) 0x7fc9d0004760 con 0x556d6fe16b60 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.008+0000 7fc9de82a740 1 -- 172.21.15.31:0/3379799098 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9c40054d0 con 0x556d6fe16b60 2024-04-25T02:14:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.008+0000 7fc9bcff9640 1 -- 172.21.15.31:0/3379799098 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fc9d0004960 con 0x556d6fe16b60 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.008+0000 7fc9de82a740 10 client.27106 fetch_fsmap learned FSMap version 49 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.008+0000 7fc9de82a740 1 -- 172.21.15.31:0/3379799098 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x556d6fe30250 con 0x556d6fe16b60 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.009+0000 7fc9bcff9640 1 -- 172.21.15.31:0/3379799098 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9d00035f0 con 0x556d6fe16b60 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.009+0000 7fc9bcff9640 1 -- 172.21.15.31:0/3379799098 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fc9d0044020 con 0x556d6fe16b60 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.009+0000 7fc9de82a740 10 client.27106 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.009+0000 7fc9de82a740 10 client.27106 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.009+0000 7fc9de82a740 20 client.27106 populate_metadata read hostname 'smithi031' 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.009+0000 7fc9de82a740 1 --2- 172.21.15.31:0/3379799098 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556d6fe31760 0x556d6fe33b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.009+0000 7fc9de82a740 4 client.27106 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.009+0000 7fc9de82a740 1 -- 172.21.15.31:0/3379799098 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc9c40054d0 con 0x556d6fe31760 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.009+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/3379799098 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556d6fe31760 0x556d6fe33b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.010+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/3379799098 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556d6fe31760 0x556d6fe33b50 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d263cae7e1d42ae server_cookie=41fe20879e1c07f5 in_seq=0 out_seq=0 2024-04-25T02:14:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.010+0000 7fc9bcff9640 10 client.27106 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9bcff9640 1 -- 172.21.15.31:0/3379799098 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fc9c40054d0 con 0x556d6fe31760 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9bcff9640 10 client.27106 handle_command_reply: tid=0 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 1 client.27106 shutdown 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 1 -- 172.21.15.31:0/3379799098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16b60 msgr2=0x556d6fe1e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 1 --2- 172.21.15.31:0/3379799098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16b60 0x556d6fe1e6f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc9d002aa10 tx=0x7fc9d0005270 comp rx=0 tx=0).stop 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 1 -- 172.21.15.31:0/3379799098 shutdown_connections 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 1 --2- 172.21.15.31:0/3379799098 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556d6fe31760 0x556d6fe33b50 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 1 --2- 172.21.15.31:0/3379799098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe21230 0x556d6fe1ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 1 --2- 172.21.15.31:0/3379799098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16b60 0x556d6fe1e6f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9bcff9640 0 client.27106 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 1 --2- 172.21.15.31:0/3379799098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe16240 0x556d6fe1e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 1 -- 172.21.15.31:0/3379799098 >> 172.21.15.31:0/3379799098 conn(0x556d6fe12560 msgr2=0x556d6fe18260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 1 -- 172.21.15.31:0/3379799098 shutdown_connections 2024-04-25T02:14:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 1 -- 172.21.15.31:0/3379799098 wait complete. 2024-04-25T02:14:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.011+0000 7fc9de82a740 20 client.27106 trim_cache size 0 max 16384 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9de82a740 1 Processor -- start 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9de82a740 1 -- start start 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 0x556d6fe208e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16b60 0x556d6fe189a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe21230 0x556d6fe1ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9de82a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b800e7a0 con 0x556d6fe16b60 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9de82a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b80102e0 con 0x556d6fe21230 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9de82a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b800ee70 con 0x556d6fe16240 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9d97e1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16b60 0x556d6fe189a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9d8fe0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 0x556d6fe208e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9d97e1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16b60 0x556d6fe189a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54292/0 (socket says 172.21.15.31:54292) 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9d97e1640 1 -- 172.21.15.31:0/2955692118 learned_addr learned my addr 172.21.15.31:0/2955692118 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.033+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/2955692118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe21230 0x556d6fe1ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.034+0000 7fc9d97e1640 1 -- 172.21.15.31:0/2955692118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 msgr2=0x556d6fe208e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.034+0000 7fc9d97e1640 1 --2- 172.21.15.31:0/2955692118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 0x556d6fe208e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.034+0000 7fc9d97e1640 1 -- 172.21.15.31:0/2955692118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe21230 msgr2=0x556d6fe1ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.034+0000 7fc9d97e1640 1 --2- 172.21.15.31:0/2955692118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe21230 0x556d6fe1ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.034+0000 7fc9d97e1640 1 -- 172.21.15.31:0/2955692118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d6fe2dcf0 con 0x556d6fe16b60 2024-04-25T02:14:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.034+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/2955692118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 0x556d6fe208e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:14:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.034+0000 7fc9d97e1640 1 --2- 172.21.15.31:0/2955692118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16b60 0x556d6fe189a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0000c00 tx=0x7fc9d0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6be846e30ed090 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.034+0000 7fc9beffd640 1 -- 172.21.15.31:0/2955692118 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0053070 con 0x556d6fe16b60 2024-04-25T02:14:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.034+0000 7fc9beffd640 1 -- 172.21.15.31:0/2955692118 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9d00316e0 con 0x556d6fe16b60 2024-04-25T02:14:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.034+0000 7fc9beffd640 1 -- 172.21.15.31:0/2955692118 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0035030 con 0x556d6fe16b60 2024-04-25T02:14:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.035+0000 7fc9de82a740 1 -- 172.21.15.31:0/2955692118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16b60 msgr2=0x556d6fe189a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.035+0000 7fc9de82a740 1 --2- 172.21.15.31:0/2955692118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16b60 0x556d6fe189a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0000c00 tx=0x7fc9d0005fd0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.035+0000 7fc9de82a740 1 -- 172.21.15.31:0/2955692118 shutdown_connections 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.035+0000 7fc9de82a740 1 --2- 172.21.15.31:0/2955692118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe21230 0x556d6fe1ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.035+0000 7fc9de82a740 1 --2- 172.21.15.31:0/2955692118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16b60 0x556d6fe189a0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.035+0000 7fc9de82a740 1 --2- 172.21.15.31:0/2955692118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16240 0x556d6fe208e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.035+0000 7fc9de82a740 1 -- 172.21.15.31:0/2955692118 >> 172.21.15.31:0/2955692118 conn(0x556d6fe12560 msgr2=0x556d6fe115c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.035+0000 7fc9de82a740 1 -- 172.21.15.31:0/2955692118 shutdown_connections 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.035+0000 7fc9de82a740 1 -- 172.21.15.31:0/2955692118 wait complete. 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.036+0000 7fc9de82a740 1 -- start start 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.036+0000 7fc9de82a740 1 Processor -- start 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.036+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16240 0x556d6fe1ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.036+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16b60 0x556d6fe1b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.036+0000 7fc9de82a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe21230 0x556d6fe1b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9de82a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b800fc10 con 0x556d6fe16240 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9de82a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b800f540 con 0x556d6fe21230 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9de82a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b800c7a0 con 0x556d6fe16b60 2024-04-25T02:14:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9d8fe0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16240 0x556d6fe1ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9de82a740 10 client.0 _ll_register_callbacks cb 0x7fc9d0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9d8fe0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16240 0x556d6fe1ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54296/0 (socket says 172.21.15.31:54296) 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9d8fe0640 1 -- 172.21.15.31:0/1234066410 learned_addr learned my addr 172.21.15.31:0/1234066410 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe21230 0x556d6fe1b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9d8fe0640 1 -- 172.21.15.31:0/1234066410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16b60 msgr2=0x556d6fe1b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16b60 0x556d6fe1b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9d8fe0640 1 -- 172.21.15.31:0/1234066410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe21230 msgr2=0x556d6fe1b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe21230 0x556d6fe1b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9d8fe0640 1 -- 172.21.15.31:0/1234066410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d6fe96c70 con 0x556d6fe16240 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9d8fe0640 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16240 0x556d6fe1ad90 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40058c0 tx=0x7fc9c4013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e49ea5b94799db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.037+0000 7fc9bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.038+0000 7fc9bcff9640 1 -- 172.21.15.31:0/1234066410 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4027040 con 0x556d6fe16240 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.038+0000 7fc9bcff9640 1 -- 172.21.15.31:0/1234066410 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8484+0+0 (secure 0 0 0) 0x7fc9c4011030 con 0x556d6fe16240 2024-04-25T02:14:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.038+0000 7fc9de82a740 1 -- 172.21.15.31:0/1234066410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9d00316e0 con 0x556d6fe16240 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.038+0000 7fc9bcff9640 1 -- 172.21.15.31:0/1234066410 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fc9c4011230 con 0x556d6fe16240 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.038+0000 7fc9de82a740 10 client.17772 fetch_fsmap learned FSMap version 49 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.038+0000 7fc9de82a740 1 -- 172.21.15.31:0/1234066410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x556d6fe97f10 con 0x556d6fe16240 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.038+0000 7fc9bcff9640 1 -- 172.21.15.31:0/1234066410 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9c4031040 con 0x556d6fe16240 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.039+0000 7fc9bcff9640 1 -- 172.21.15.31:0/1234066410 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fc9c4010430 con 0x556d6fe16240 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.039+0000 7fc9de82a740 10 client.17772 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.039+0000 7fc9de82a740 10 client.17772 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.039+0000 7fc9de82a740 20 client.17772 populate_metadata read hostname 'smithi031' 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.039+0000 7fc9de82a740 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556d6fe984e0 0x556d6fe9a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.039+0000 7fc9de82a740 4 client.17772 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.039+0000 7fc9de82a740 1 -- 172.21.15.31:0/1234066410 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fc9d00316e0 con 0x556d6fe984e0 2024-04-25T02:14:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.039+0000 7fc9d97e1640 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556d6fe984e0 0x556d6fe9a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:14:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.039+0000 7fc9d97e1640 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556d6fe984e0 0x556d6fe9a900 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60f9684128731181 server_cookie=326dc1633db0884a in_seq=0 out_seq=0 2024-04-25T02:14:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.039+0000 7fc9bcff9640 10 client.17772 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:49.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 68058 8.84 MB/sec execute 293 sec latency 95.593 ms 2024-04-25T02:14:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9bcff9640 1 -- 172.21.15.31:0/1234066410 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fc9d00316e0 con 0x556d6fe984e0 2024-04-25T02:14:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9bcff9640 10 client.17772 handle_command_reply: tid=0 2024-04-25T02:14:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9de82a740 1 client.17772 shutdown 2024-04-25T02:14:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9de82a740 1 -- 172.21.15.31:0/1234066410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16240 msgr2=0x556d6fe1ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9de82a740 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16240 0x556d6fe1ad90 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40058c0 tx=0x7fc9c4013040 comp rx=0 tx=0).stop 2024-04-25T02:14:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9de82a740 1 -- 172.21.15.31:0/1234066410 shutdown_connections 2024-04-25T02:14:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9de82a740 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556d6fe984e0 0x556d6fe9a900 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9de82a740 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d6fe21230 0x556d6fe1b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9bcff9640 0 client.17772 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:14:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9de82a740 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556d6fe16b60 0x556d6fe1b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9de82a740 1 --2- 172.21.15.31:0/1234066410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d6fe16240 0x556d6fe1ad90 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.106+0000 7fc9de82a740 1 -- 172.21.15.31:0/1234066410 >> 172.21.15.31:0/1234066410 conn(0x556d6fe12560 msgr2=0x556d6fe1f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.107+0000 7fc9de82a740 1 -- 172.21.15.31:0/1234066410 shutdown_connections 2024-04-25T02:14:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.107+0000 7fc9de82a740 1 -- 172.21.15.31:0/1234066410 wait complete. 2024-04-25T02:14:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.107+0000 7fc9de82a740 20 client.17772 trim_cache size 0 max 16384 2024-04-25T02:14:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:14:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:14:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:14:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:14:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.107+0000 7fc9dba45640 1 -- 172.21.15.31:0/3782765628 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9c00a2ef0 msgr2=0x7fc9c00a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.107+0000 7fc9dba45640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9c00a2ef0 0x7fc9c00a53b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0a4580 tx=0x7fc9cc022000 comp rx=0 tx=0).stop 2024-04-25T02:14:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.107+0000 7fc9dba45640 1 -- 172.21.15.31:0/3782765628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9d4106630 msgr2=0x7fc9d410ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:14:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.107+0000 7fc9dba45640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9d4106630 0x7fc9d410ec30 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00eb10 tx=0x7fc9cc00c2d0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.108+0000 7fc9dba45640 1 -- 172.21.15.31:0/3782765628 shutdown_connections 2024-04-25T02:14:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.108+0000 7fc9dba45640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9c00a2ef0 0x7fc9c00a53b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.108+0000 7fc9dba45640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9d4106630 0x7fc9d410ec30 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.108+0000 7fc9dba45640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d4101030 0x7fc9d410e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.108+0000 7fc9dba45640 1 --2- 172.21.15.31:0/3782765628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9d4100670 0x7fc9d410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:14:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.108+0000 7fc9dba45640 1 -- 172.21.15.31:0/3782765628 >> 172.21.15.31:0/3782765628 conn(0x7fc9d40777c0 msgr2=0x7fc9d4103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:14:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.108+0000 7fc9dba45640 1 -- 172.21.15.31:0/3782765628 shutdown_connections 2024-04-25T02:14:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:14:49.108+0000 7fc9dba45640 1 -- 172.21.15.31:0/3782765628 wait complete. 2024-04-25T02:14:50.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 68181 8.83 MB/sec execute 294 sec latency 193.237 ms 2024-04-25T02:14:50.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:50 smithi040 ceph-mon[32418]: pgmap v640: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 234 op/s 2024-04-25T02:14:50.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:50 smithi134 ceph-mon[33098]: pgmap v640: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 234 op/s 2024-04-25T02:14:50.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:50 smithi031 ceph-mon[29204]: pgmap v640: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 234 op/s 2024-04-25T02:14:51.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 68362 8.85 MB/sec execute 295 sec latency 64.867 ms 2024-04-25T02:14:52.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 68568 8.87 MB/sec execute 296 sec latency 71.221 ms 2024-04-25T02:14:52.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:52 smithi040 ceph-mon[32418]: pgmap v641: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 355 op/s 2024-04-25T02:14:52.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:52 smithi134 ceph-mon[33098]: pgmap v641: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 355 op/s 2024-04-25T02:14:52.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:52 smithi031 ceph-mon[29204]: pgmap v641: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.5 MiB/s wr, 355 op/s 2024-04-25T02:14:53.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 68747 8.87 MB/sec execute 297 sec latency 84.656 ms 2024-04-25T02:14:54.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 68869 8.86 MB/sec execute 298 sec latency 70.711 ms 2024-04-25T02:14:54.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:54 smithi040 ceph-mon[32418]: pgmap v642: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 245 op/s 2024-04-25T02:14:54.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:54 smithi134 ceph-mon[33098]: pgmap v642: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 245 op/s 2024-04-25T02:14:54.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:54 smithi031 ceph-mon[29204]: pgmap v642: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 245 op/s 2024-04-25T02:14:55.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 69007 8.86 MB/sec execute 299 sec latency 132.559 ms 2024-04-25T02:14:56.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 69174 8.85 MB/sec execute 300 sec latency 57.731 ms 2024-04-25T02:14:56.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:56 smithi040 ceph-mon[32418]: pgmap v643: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.6 MiB/s wr, 363 op/s 2024-04-25T02:14:56.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:56 smithi134 ceph-mon[33098]: pgmap v643: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.6 MiB/s wr, 363 op/s 2024-04-25T02:14:56.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:56 smithi031 ceph-mon[29204]: pgmap v643: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.6 MiB/s wr, 363 op/s 2024-04-25T02:14:57.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 69330 8.85 MB/sec execute 301 sec latency 69.182 ms 2024-04-25T02:14:58.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 69478 8.84 MB/sec execute 302 sec latency 74.682 ms 2024-04-25T02:14:58.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:14:58 smithi040 ceph-mon[32418]: pgmap v644: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.7 MiB/s wr, 238 op/s 2024-04-25T02:14:58.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:14:58 smithi134 ceph-mon[33098]: pgmap v644: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.7 MiB/s wr, 238 op/s 2024-04-25T02:14:58.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:14:58 smithi031 ceph-mon[29204]: pgmap v644: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.7 MiB/s wr, 238 op/s 2024-04-25T02:14:59.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 69630 8.84 MB/sec execute 303 sec latency 58.111 ms 2024-04-25T02:15:00.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 69761 8.83 MB/sec execute 304 sec latency 81.385 ms 2024-04-25T02:15:00.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:00 smithi040 ceph-mon[32418]: pgmap v645: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.7 MiB/s wr, 238 op/s 2024-04-25T02:15:00.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:00 smithi134 ceph-mon[33098]: pgmap v645: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.7 MiB/s wr, 238 op/s 2024-04-25T02:15:00.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:00 smithi031 ceph-mon[29204]: pgmap v645: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.7 MiB/s wr, 238 op/s 2024-04-25T02:15:01.102 INFO:tasks.workunit.client.0.smithi031.stdout: 10 69947 8.83 MB/sec execute 305 sec latency 119.787 ms 2024-04-25T02:15:02.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 70141 8.84 MB/sec execute 306 sec latency 85.774 ms 2024-04-25T02:15:02.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.265+0000 7ff252e41640 1 -- 172.21.15.31:0/1236122691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff24c10dc10 msgr2=0x7ff24c10dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.265+0000 7ff252e41640 1 --2- 172.21.15.31:0/1236122691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff24c10dc10 0x7ff24c10dff0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7ff248009930 tx=0x7ff24802a420 comp rx=0 tx=0).stop 2024-04-25T02:15:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.265+0000 7ff252e41640 1 -- 172.21.15.31:0/1236122691 shutdown_connections 2024-04-25T02:15:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.265+0000 7ff252e41640 1 --2- 172.21.15.31:0/1236122691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24c108600 0x7ff24c114a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.265+0000 7ff252e41640 1 --2- 172.21.15.31:0/1236122691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff24c107c80 0x7ff24c1080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.265+0000 7ff252e41640 1 --2- 172.21.15.31:0/1236122691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff24c10dc10 0x7ff24c10dff0 secure :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7ff248009930 tx=0x7ff24802a420 comp rx=0 tx=0).stop 2024-04-25T02:15:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.265+0000 7ff252e41640 1 -- 172.21.15.31:0/1236122691 >> 172.21.15.31:0/1236122691 conn(0x7ff24c077520 msgr2=0x7ff24c106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.265+0000 7ff252e41640 1 -- 172.21.15.31:0/1236122691 shutdown_connections 2024-04-25T02:15:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.265+0000 7ff252e41640 1 -- 172.21.15.31:0/1236122691 wait complete. 2024-04-25T02:15:02.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.266+0000 7ff252e41640 1 Processor -- start 2024-04-25T02:15:02.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.266+0000 7ff252e41640 1 -- start start 2024-04-25T02:15:02.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff252e41640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff24c107c80 0x7ff24c19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff252e41640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24c108600 0x7ff24c19eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff252e41640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff24c1a2e20 0x7ff24c1a32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff252640640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff24c1a2e20 0x7ff24c1a32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff251e3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff24c107c80 0x7ff24c19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff252640640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff24c1a2e20 0x7ff24c1a32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44114/0 (socket says 172.21.15.31:44114) 2024-04-25T02:15:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff25163e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24c108600 0x7ff24c19eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff251e3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff24c107c80 0x7ff24c19e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35450/0 (socket says 172.21.15.31:35450) 2024-04-25T02:15:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff25163e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24c108600 0x7ff24c19eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:37108/0 (socket says 172.21.15.31:37108) 2024-04-25T02:15:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff252640640 1 -- 172.21.15.31:0/1443140293 learned_addr learned my addr 172.21.15.31:0/1443140293 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff252e41640 1 -- 172.21.15.31:0/1443140293 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c117e50 con 0x7ff24c107c80 2024-04-25T02:15:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff252e41640 1 -- 172.21.15.31:0/1443140293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c117cd0 con 0x7ff24c1a2e20 2024-04-25T02:15:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.267+0000 7ff252e41640 1 -- 172.21.15.31:0/1443140293 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c117fd0 con 0x7ff24c108600 2024-04-25T02:15:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.268+0000 7ff252640640 1 -- 172.21.15.31:0/1443140293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24c108600 msgr2=0x7ff24c19eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.268+0000 7ff252640640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24c108600 0x7ff24c19eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.268+0000 7ff252640640 1 -- 172.21.15.31:0/1443140293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff24c107c80 msgr2=0x7ff24c19e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.268+0000 7ff252640640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff24c107c80 0x7ff24c19e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.268+0000 7ff252640640 1 -- 172.21.15.31:0/1443140293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24c1a3910 con 0x7ff24c1a2e20 2024-04-25T02:15:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.268+0000 7ff25163e640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24c108600 0x7ff24c19eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:15:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.268+0000 7ff252640640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff24c1a2e20 0x7ff24c1a32d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff24400eb10 tx=0x7ff24400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.268+0000 7ff242ffd640 1 -- 172.21.15.31:0/1443140293 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff244004b20 con 0x7ff24c1a2e20 2024-04-25T02:15:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.269+0000 7ff242ffd640 1 -- 172.21.15.31:0/1443140293 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff244004cc0 con 0x7ff24c1a2e20 2024-04-25T02:15:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.269+0000 7ff242ffd640 1 -- 172.21.15.31:0/1443140293 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff244010720 con 0x7ff24c1a2e20 2024-04-25T02:15:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.269+0000 7ff252e41640 1 -- 172.21.15.31:0/1443140293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24c1a3c00 con 0x7ff24c1a2e20 2024-04-25T02:15:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.269+0000 7ff252e41640 1 -- 172.21.15.31:0/1443140293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff24c1ab340 con 0x7ff24c1a2e20 2024-04-25T02:15:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.271+0000 7ff242ffd640 1 -- 172.21.15.31:0/1443140293 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff244005020 con 0x7ff24c1a2e20 2024-04-25T02:15:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.272+0000 7ff256da8740 1 Processor -- start 2024-04-25T02:15:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.273+0000 7ff242ffd640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff2380a2d40 0x7ff2380a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.273+0000 7ff25163e640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff2380a2d40 0x7ff2380a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.273+0000 7ff242ffd640 1 -- 172.21.15.31:0/1443140293 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8528+0+0 (secure 0 0 0) 0x7ff2440cfa20 con 0x7ff24c1a2e20 2024-04-25T02:15:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.273+0000 7ff256da8740 1 -- start start 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.273+0000 7ff256da8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 0x55b2e25ab790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.273+0000 7ff256da8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25abcd0 0x55b2e25b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.273+0000 7ff256da8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25b63a0 0x55b2e25b8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.273+0000 7ff256da8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b2e250fcb0 con 0x55b2e25ab3b0 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.273+0000 7ff256da8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b2e24b85b0 con 0x55b2e25abcd0 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.273+0000 7ff256da8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b2e250bc70 con 0x55b2e25b63a0 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff252640640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25b63a0 0x55b2e25b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff25163e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25abcd0 0x55b2e25b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff252640640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25b63a0 0x55b2e25b8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:37120/0 (socket says 172.21.15.31:37120) 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff252640640 1 -- 172.21.15.31:0/3265500145 learned_addr learned my addr 172.21.15.31:0/3265500145 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff25163e640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff2380a2d40 0x7ff2380a5200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff24c078050 tx=0x7ff23c005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff251e3f640 1 --2- 172.21.15.31:0/3265500145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 0x55b2e25ab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff252640640 1 -- 172.21.15.31:0/3265500145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25abcd0 msgr2=0x55b2e25b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff252640640 1 --2- 172.21.15.31:0/3265500145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25abcd0 0x55b2e25b5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff252640640 1 -- 172.21.15.31:0/3265500145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 msgr2=0x55b2e25ab790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff252640640 1 --2- 172.21.15.31:0/3265500145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 0x55b2e25ab790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff252640640 1 -- 172.21.15.31:0/3265500145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2e2224210 con 0x55b2e25b63a0 2024-04-25T02:15:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.274+0000 7ff252640640 1 --2- 172.21.15.31:0/3265500145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25b63a0 0x55b2e25b8790 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff244098910 tx=0x7ff24409ba80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3df1e0d7adfa4947 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.275+0000 7ff237fff640 1 -- 172.21.15.31:0/3265500145 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24409bf40 con 0x55b2e25b63a0 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.275+0000 7ff237fff640 1 -- 172.21.15.31:0/3265500145 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2440a02f0 con 0x55b2e25b63a0 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.275+0000 7ff237fff640 1 -- 172.21.15.31:0/3265500145 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24409c1a0 con 0x55b2e25b63a0 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.276+0000 7ff256da8740 1 -- 172.21.15.31:0/3265500145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25b63a0 msgr2=0x55b2e25b8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.276+0000 7ff256da8740 1 --2- 172.21.15.31:0/3265500145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25b63a0 0x55b2e25b8790 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff244098910 tx=0x7ff24409ba80 comp rx=0 tx=0).stop 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.276+0000 7ff256da8740 1 -- 172.21.15.31:0/3265500145 shutdown_connections 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.276+0000 7ff256da8740 1 --2- 172.21.15.31:0/3265500145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25b63a0 0x55b2e25b8790 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.276+0000 7ff256da8740 1 --2- 172.21.15.31:0/3265500145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25abcd0 0x55b2e25b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.276+0000 7ff256da8740 1 --2- 172.21.15.31:0/3265500145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 0x55b2e25ab790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.276+0000 7ff256da8740 1 -- 172.21.15.31:0/3265500145 >> 172.21.15.31:0/3265500145 conn(0x55b2e25a76d0 msgr2=0x55b2e25a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.276+0000 7ff256da8740 1 -- 172.21.15.31:0/3265500145 shutdown_connections 2024-04-25T02:15:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.276+0000 7ff256da8740 1 -- 172.21.15.31:0/3265500145 wait complete. 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.276+0000 7ff256da8740 1 -- start start 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff256da8740 1 Processor -- start 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff256da8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff256da8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff256da8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25b63a0 0x55b2e25b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff256da8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b2e25ca1a0 con 0x55b2e25b63a0 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff256da8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b2e24cbb50 con 0x55b2e25ab3b0 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff256da8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b2e25ca320 con 0x55b2e25abcd0 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff256da8740 10 client.0 _ll_register_callbacks cb 0x55b2e2506ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff25163e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff251e3f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff251e3f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44128/0 (socket says 172.21.15.31:44128) 2024-04-25T02:15:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff252640640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25b63a0 0x55b2e25b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff25163e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:37132/0 (socket says 172.21.15.31:37132) 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.277+0000 7ff251e3f640 1 -- 172.21.15.31:0/3558425057 learned_addr learned my addr 172.21.15.31:0/3558425057 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff251e3f640 1 -- 172.21.15.31:0/3558425057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 msgr2=0x55b2e25b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff251e3f640 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff251e3f640 1 -- 172.21.15.31:0/3558425057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25b63a0 msgr2=0x55b2e25b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff251e3f640 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25b63a0 0x55b2e25b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff251e3f640 1 -- 172.21.15.31:0/3558425057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2e22f0620 con 0x55b2e25ab3b0 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff25163e640 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff251e3f640 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b33d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff24802a990 tx=0x7ff2480043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b27be57efe4317fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff2357fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff2357fa640 1 -- 172.21.15.31:0/3558425057 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248035040 con 0x55b2e25ab3b0 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff2357fa640 1 -- 172.21.15.31:0/3558425057 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8528+0+0 (secure 0 0 0) 0x7ff248005450 con 0x55b2e25ab3b0 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff256da8740 1 -- 172.21.15.31:0/3558425057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2440a02f0 con 0x55b2e25ab3b0 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff2357fa640 1 -- 172.21.15.31:0/3558425057 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff248005650 con 0x55b2e25ab3b0 2024-04-25T02:15:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff256da8740 10 client.27341 fetch_fsmap learned FSMap version 49 2024-04-25T02:15:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.278+0000 7ff256da8740 1 -- 172.21.15.31:0/3558425057 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55b2e25d6440 con 0x55b2e25ab3b0 2024-04-25T02:15:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.279+0000 7ff2357fa640 1 -- 172.21.15.31:0/3558425057 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff248040030 con 0x55b2e25ab3b0 2024-04-25T02:15:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.279+0000 7ff2357fa640 1 -- 172.21.15.31:0/3558425057 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7ff248003760 con 0x55b2e25ab3b0 2024-04-25T02:15:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.279+0000 7ff256da8740 10 client.27341 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:15:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.279+0000 7ff256da8740 10 client.27341 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:15:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.279+0000 7ff256da8740 20 client.27341 populate_metadata read hostname 'smithi031' 2024-04-25T02:15:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.279+0000 7ff256da8740 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b2e25d7950 0x55b2e25d9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.279+0000 7ff256da8740 4 client.27341 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:15:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.279+0000 7ff256da8740 1 -- 172.21.15.31:0/3558425057 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff2440a02f0 con 0x55b2e25d7950 2024-04-25T02:15:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.279+0000 7ff25163e640 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b2e25d7950 0x55b2e25d9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.280+0000 7ff25163e640 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b2e25d7950 0x55b2e25d9d60 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86125b79261fb10b server_cookie=51eaa23cbed97742 in_seq=0 out_seq=0 2024-04-25T02:15:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.280+0000 7ff2357fa640 10 client.27341 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:02.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.280+0000 7ff2357fa640 1 -- 172.21.15.31:0/3558425057 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7ff2440a02f0 con 0x55b2e25d7950 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.280+0000 7ff2357fa640 10 client.27341 handle_command_reply: tid=0 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.280+0000 7ff256da8740 1 client.27341 shutdown 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.280+0000 7ff256da8740 1 -- 172.21.15.31:0/3558425057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25ab3b0 msgr2=0x55b2e25b33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.280+0000 7ff256da8740 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b33d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff24802a990 tx=0x7ff2480043b0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.281+0000 7ff256da8740 1 -- 172.21.15.31:0/3558425057 shutdown_connections 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.281+0000 7ff256da8740 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b2e25d7950 0x55b2e25d9d60 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.281+0000 7ff256da8740 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25b63a0 0x55b2e25b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.281+0000 7ff2357fa640 0 client.27341 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.281+0000 7ff256da8740 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.281+0000 7ff256da8740 1 --2- 172.21.15.31:0/3558425057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b33d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.281+0000 7ff256da8740 1 -- 172.21.15.31:0/3558425057 >> 172.21.15.31:0/3558425057 conn(0x55b2e25a76d0 msgr2=0x55b2e25ad4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.281+0000 7ff256da8740 1 -- 172.21.15.31:0/3558425057 shutdown_connections 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.281+0000 7ff256da8740 1 -- 172.21.15.31:0/3558425057 wait complete. 2024-04-25T02:15:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.281+0000 7ff256da8740 20 client.27341 trim_cache size 0 max 16384 2024-04-25T02:15:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.304+0000 7ff256da8740 1 Processor -- start 2024-04-25T02:15:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.304+0000 7ff256da8740 1 -- start start 2024-04-25T02:15:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.304+0000 7ff256da8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25ab3b0 0x55b2e25d3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.304+0000 7ff256da8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.304+0000 7ff256da8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25b63a0 0x55b2e25df670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.304+0000 7ff256da8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c1ab680 con 0x55b2e25abcd0 2024-04-25T02:15:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.304+0000 7ff256da8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c1ad1c0 con 0x55b2e25b63a0 2024-04-25T02:15:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.304+0000 7ff256da8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c1abd50 con 0x55b2e25ab3b0 2024-04-25T02:15:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff25163e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff251e3f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25ab3b0 0x55b2e25d3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff25163e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35478/0 (socket says 172.21.15.31:35478) 2024-04-25T02:15:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff25163e640 1 -- 172.21.15.31:0/634991987 learned_addr learned my addr 172.21.15.31:0/634991987 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff252640640 1 --2- 172.21.15.31:0/634991987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25b63a0 0x55b2e25df670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff25163e640 1 -- 172.21.15.31:0/634991987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25ab3b0 msgr2=0x55b2e25d3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff25163e640 1 --2- 172.21.15.31:0/634991987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25ab3b0 0x55b2e25d3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff25163e640 1 -- 172.21.15.31:0/634991987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25b63a0 msgr2=0x55b2e25df670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff25163e640 1 --2- 172.21.15.31:0/634991987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25b63a0 0x55b2e25df670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff25163e640 1 -- 172.21.15.31:0/634991987 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2e25a1850 con 0x55b2e25abcd0 2024-04-25T02:15:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff25163e640 1 --2- 172.21.15.31:0/634991987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3c70 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x55b2e25ac970 tx=0x7ff23c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807b2679284b594f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff2377fe640 1 -- 172.21.15.31:0/634991987 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c022a60 con 0x55b2e25abcd0 2024-04-25T02:15:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff2377fe640 1 -- 172.21.15.31:0/634991987 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff23c030070 con 0x55b2e25abcd0 2024-04-25T02:15:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.305+0000 7ff2377fe640 1 -- 172.21.15.31:0/634991987 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c020890 con 0x55b2e25abcd0 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.306+0000 7ff256da8740 1 -- 172.21.15.31:0/634991987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25abcd0 msgr2=0x55b2e25b3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.306+0000 7ff256da8740 1 --2- 172.21.15.31:0/634991987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3c70 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x55b2e25ac970 tx=0x7ff23c014000 comp rx=0 tx=0).stop 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.306+0000 7ff256da8740 1 -- 172.21.15.31:0/634991987 shutdown_connections 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.306+0000 7ff256da8740 1 --2- 172.21.15.31:0/634991987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25b63a0 0x55b2e25df670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.306+0000 7ff256da8740 1 --2- 172.21.15.31:0/634991987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25abcd0 0x55b2e25b3c70 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.306+0000 7ff256da8740 1 --2- 172.21.15.31:0/634991987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25ab3b0 0x55b2e25d3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.306+0000 7ff256da8740 1 -- 172.21.15.31:0/634991987 >> 172.21.15.31:0/634991987 conn(0x55b2e25a76d0 msgr2=0x55b2e25b4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.306+0000 7ff256da8740 1 -- 172.21.15.31:0/634991987 shutdown_connections 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.306+0000 7ff256da8740 1 -- 172.21.15.31:0/634991987 wait complete. 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.307+0000 7ff256da8740 1 -- start start 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.307+0000 7ff256da8740 1 Processor -- start 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.307+0000 7ff256da8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.307+0000 7ff256da8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 0x55b2e25b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.307+0000 7ff251e3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.307+0000 7ff251e3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35490/0 (socket says 172.21.15.31:35490) 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff251e3f640 1 -- 172.21.15.31:0/1436804724 learned_addr learned my addr 172.21.15.31:0/1436804724 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff256da8740 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25b63a0 0x55b2e263cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff256da8740 1 -- 172.21.15.31:0/1436804724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c1acaf0 con 0x55b2e25ab3b0 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff256da8740 1 -- 172.21.15.31:0/1436804724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c1ac420 con 0x55b2e25b63a0 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff256da8740 1 -- 172.21.15.31:0/1436804724 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c0719c0 con 0x55b2e25abcd0 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff25163e640 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 0x55b2e25b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff256da8740 10 client.0 _ll_register_callbacks cb 0x7ff2440093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff252640640 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25b63a0 0x55b2e263cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff251e3f640 1 -- 172.21.15.31:0/1436804724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 msgr2=0x55b2e25b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff251e3f640 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 0x55b2e25b0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff251e3f640 1 -- 172.21.15.31:0/1436804724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25b63a0 msgr2=0x55b2e263cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff251e3f640 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25b63a0 0x55b2e263cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff251e3f640 1 -- 172.21.15.31:0/1436804724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2e263d420 con 0x55b2e25ab3b0 2024-04-25T02:15:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff25163e640 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 0x55b2e25b0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff251e3f640 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b07e0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff248006010 tx=0x7ff248005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8afdb04c02172e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff2357fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff2357fa640 1 -- 172.21.15.31:0/1436804724 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248035040 con 0x55b2e25ab3b0 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.308+0000 7ff2357fa640 1 -- 172.21.15.31:0/1436804724 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8528+0+0 (secure 0 0 0) 0x7ff2480037f0 con 0x55b2e25ab3b0 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff256da8740 1 -- 172.21.15.31:0/1436804724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff23c030070 con 0x55b2e25ab3b0 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff2357fa640 1 -- 172.21.15.31:0/1436804724 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff24805f050 con 0x55b2e25ab3b0 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff256da8740 10 client.17790 fetch_fsmap learned FSMap version 49 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff256da8740 1 -- 172.21.15.31:0/1436804724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55b2e263e6f0 con 0x55b2e25ab3b0 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff2357fa640 1 -- 172.21.15.31:0/1436804724 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff23c030070 con 0x55b2e25ab3b0 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff2357fa640 1 -- 172.21.15.31:0/1436804724 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7ff248057020 con 0x55b2e25ab3b0 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff256da8740 10 client.17790 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff256da8740 10 client.17790 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:15:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff256da8740 20 client.17790 populate_metadata read hostname 'smithi031' 2024-04-25T02:15:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff256da8740 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b2e25d7950 0x55b2e2640e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff256da8740 4 client.17790 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:15:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.309+0000 7ff256da8740 1 -- 172.21.15.31:0/1436804724 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b2e251d1f0 con 0x55b2e25d7950 2024-04-25T02:15:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.310+0000 7ff25163e640 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b2e25d7950 0x55b2e2640e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.310+0000 7ff25163e640 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b2e25d7950 0x55b2e2640e70 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba537ff1969e5c9f server_cookie=add461236d1bfe42 in_seq=0 out_seq=0 2024-04-25T02:15:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.310+0000 7ff2357fa640 10 client.17790 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.364+0000 7ff2357fa640 1 -- 172.21.15.31:0/1436804724 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b2e251d1f0 con 0x55b2e25d7950 2024-04-25T02:15:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.364+0000 7ff2357fa640 10 client.17790 handle_command_reply: tid=0 2024-04-25T02:15:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.364+0000 7ff256da8740 1 client.17790 shutdown 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.364+0000 7ff256da8740 1 -- 172.21.15.31:0/1436804724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 msgr2=0x55b2e25b07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.364+0000 7ff256da8740 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b07e0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff248006010 tx=0x7ff248005b50 comp rx=0 tx=0).stop 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.364+0000 7ff256da8740 1 -- 172.21.15.31:0/1436804724 shutdown_connections 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.364+0000 7ff256da8740 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b2e25d7950 0x55b2e2640e70 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.365+0000 7ff256da8740 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b2e25b63a0 0x55b2e263cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.365+0000 7ff2357fa640 0 client.17790 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.365+0000 7ff256da8740 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b2e25abcd0 0x55b2e25b0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.365+0000 7ff256da8740 1 --2- 172.21.15.31:0/1436804724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b2e25ab3b0 0x55b2e25b07e0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.365+0000 7ff256da8740 1 -- 172.21.15.31:0/1436804724 >> 172.21.15.31:0/1436804724 conn(0x55b2e25a76d0 msgr2=0x55b2e25b34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.365+0000 7ff256da8740 1 -- 172.21.15.31:0/1436804724 shutdown_connections 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.365+0000 7ff256da8740 1 -- 172.21.15.31:0/1436804724 wait complete. 2024-04-25T02:15:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.365+0000 7ff256da8740 20 client.17790 trim_cache size 0 max 16384 2024-04-25T02:15:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:15:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:15:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:15:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:15:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 -- 172.21.15.31:0/1443140293 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff2380a2d40 msgr2=0x7ff2380a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff2380a2d40 0x7ff2380a5200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff24c078050 tx=0x7ff23c005f90 comp rx=0 tx=0).stop 2024-04-25T02:15:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 -- 172.21.15.31:0/1443140293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff24c1a2e20 msgr2=0x7ff24c1a32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff24c1a2e20 0x7ff24c1a32d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff24400eb10 tx=0x7ff24400c2d0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 -- 172.21.15.31:0/1443140293 shutdown_connections 2024-04-25T02:15:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff2380a2d40 0x7ff2380a5200 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff24c1a2e20 0x7ff24c1a32d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff24c108600 0x7ff24c19eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 --2- 172.21.15.31:0/1443140293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff24c107c80 0x7ff24c19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 -- 172.21.15.31:0/1443140293 >> 172.21.15.31:0/1443140293 conn(0x7ff24c077520 msgr2=0x7ff24c078890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 -- 172.21.15.31:0/1443140293 shutdown_connections 2024-04-25T02:15:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:02.366+0000 7ff252e41640 1 -- 172.21.15.31:0/1443140293 wait complete. 2024-04-25T02:15:02.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:02 smithi040 ceph-mon[32418]: pgmap v646: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 341 op/s 2024-04-25T02:15:02.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:02 smithi134 ceph-mon[33098]: pgmap v646: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 341 op/s 2024-04-25T02:15:02.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:02 smithi031 ceph-mon[29204]: pgmap v646: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 341 op/s 2024-04-25T02:15:03.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 70305 8.84 MB/sec execute 307 sec latency 140.662 ms 2024-04-25T02:15:03.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:03 smithi040 ceph-mon[32418]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T02:15:03.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:03 smithi134 ceph-mon[33098]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T02:15:03.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:03 smithi031 ceph-mon[29204]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T02:15:04.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 70437 8.84 MB/sec execute 308 sec latency 70.343 ms 2024-04-25T02:15:04.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:04 smithi040 ceph-mon[32418]: pgmap v648: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 265 op/s 2024-04-25T02:15:04.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:04 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:15:04.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:04 smithi134 ceph-mon[33098]: pgmap v648: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 265 op/s 2024-04-25T02:15:04.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:04 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:15:04.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:04 smithi031 ceph-mon[29204]: pgmap v648: 97 pgs: 97 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 265 op/s 2024-04-25T02:15:04.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:04 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:15:05.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 70594 8.84 MB/sec execute 309 sec latency 64.221 ms 2024-04-25T02:15:06.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 70773 8.87 MB/sec execute 310 sec latency 68.870 ms 2024-04-25T02:15:06.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:06 smithi040 ceph-mon[32418]: pgmap v649: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 289 op/s 2024-04-25T02:15:06.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:06 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:06.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:06 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:06.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:06 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:06.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:06 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:06.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:06 smithi134 ceph-mon[33098]: pgmap v649: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 289 op/s 2024-04-25T02:15:06.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:06 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:06.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:06 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:06.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:06 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:06.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:06 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:06.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:06 smithi031 ceph-mon[29204]: pgmap v649: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 289 op/s 2024-04-25T02:15:06.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:06 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:06.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:06 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:06.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:06 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:06.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:06 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:07.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 70927 8.85 MB/sec execute 311 sec latency 62.025 ms 2024-04-25T02:15:07.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:07.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:07.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:15:07.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:15:07.494 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:07 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:07.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:07.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:15:07.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:07 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:07.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:07.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:07.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.695 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.695 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.695 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:15:07.695 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:15:07.695 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:15:07.695 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:07 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:15:08.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 71075 8.85 MB/sec execute 312 sec latency 59.413 ms 2024-04-25T02:15:08.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:08 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-25T02:15:08.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:08 smithi031 ceph-mon[29204]: Unable to set osd_memory_target on smithi040 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T02:15:08.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:08 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi134 to 176.2M 2024-04-25T02:15:08.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:08 smithi031 ceph-mon[29204]: Unable to set osd_memory_target on smithi134 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T02:15:08.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:08 smithi031 ceph-mon[29204]: pgmap v650: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 289 op/s 2024-04-25T02:15:08.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:08 smithi031 ceph-mon[29204]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T02:15:08.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:08 smithi031 ceph-mon[29204]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T02:15:08.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:08 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-25T02:15:08.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:08 smithi040 ceph-mon[32418]: Unable to set osd_memory_target on smithi040 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T02:15:08.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:08 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi134 to 176.2M 2024-04-25T02:15:08.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:08 smithi040 ceph-mon[32418]: Unable to set osd_memory_target on smithi134 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T02:15:08.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:08 smithi040 ceph-mon[32418]: pgmap v650: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 289 op/s 2024-04-25T02:15:08.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:08 smithi040 ceph-mon[32418]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T02:15:08.665 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:08 smithi040 ceph-mon[32418]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T02:15:08.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:08 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-04-25T02:15:08.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:08 smithi134 ceph-mon[33098]: Unable to set osd_memory_target on smithi040 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T02:15:08.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:08 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi134 to 176.2M 2024-04-25T02:15:08.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:08 smithi134 ceph-mon[33098]: Unable to set osd_memory_target on smithi134 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T02:15:08.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:08 smithi134 ceph-mon[33098]: pgmap v650: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 289 op/s 2024-04-25T02:15:08.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:08 smithi134 ceph-mon[33098]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T02:15:08.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:08 smithi134 ceph-mon[33098]: Unable to set osd_memory_target on smithi031 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T02:15:09.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 71219 8.83 MB/sec execute 313 sec latency 50.623 ms 2024-04-25T02:15:10.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 71401 8.83 MB/sec execute 314 sec latency 73.480 ms 2024-04-25T02:15:10.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:09 smithi031 ceph-mon[29204]: pgmap v651: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 289 op/s 2024-04-25T02:15:10.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:09 smithi031 ceph-mon[29204]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T02:15:10.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:09 smithi040 ceph-mon[32418]: pgmap v651: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 289 op/s 2024-04-25T02:15:10.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:09 smithi040 ceph-mon[32418]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T02:15:10.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:09 smithi134 ceph-mon[33098]: pgmap v651: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 289 op/s 2024-04-25T02:15:10.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:09 smithi134 ceph-mon[33098]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T02:15:11.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 71548 8.82 MB/sec execute 315 sec latency 71.041 ms 2024-04-25T02:15:11.987 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:11 smithi031 ceph-mon[29204]: pgmap v653: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.5 MiB/s wr, 332 op/s 2024-04-25T02:15:12.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 71695 8.81 MB/sec execute 316 sec latency 63.690 ms 2024-04-25T02:15:12.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:11 smithi040 ceph-mon[32418]: pgmap v653: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.5 MiB/s wr, 332 op/s 2024-04-25T02:15:12.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:11 smithi134 ceph-mon[33098]: pgmap v653: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.5 MiB/s wr, 332 op/s 2024-04-25T02:15:13.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 71890 8.84 MB/sec execute 317 sec latency 62.925 ms 2024-04-25T02:15:14.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 72113 8.86 MB/sec execute 318 sec latency 52.983 ms 2024-04-25T02:15:14.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:13 smithi031 ceph-mon[29204]: pgmap v654: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.7 MiB/s wr, 291 op/s 2024-04-25T02:15:14.244 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:13 smithi134 ceph-mon[33098]: pgmap v654: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.7 MiB/s wr, 291 op/s 2024-04-25T02:15:14.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:13 smithi040 ceph-mon[32418]: pgmap v654: 97 pgs: 97 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.7 MiB/s wr, 291 op/s 2024-04-25T02:15:15.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 72285 8.87 MB/sec execute 319 sec latency 54.585 ms 2024-04-25T02:15:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.493+0000 7ff4f7a88640 1 -- 172.21.15.31:0/3130532479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f01032f0 msgr2=0x7ff4f010b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.493+0000 7ff4f7a88640 1 --2- 172.21.15.31:0/3130532479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f01032f0 0x7ff4f010b7d0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009900 tx=0x7ff4e002a510 comp rx=0 tx=0).stop 2024-04-25T02:15:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.493+0000 7ff4f7a88640 1 -- 172.21.15.31:0/3130532479 shutdown_connections 2024-04-25T02:15:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.493+0000 7ff4f7a88640 1 --2- 172.21.15.31:0/3130532479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4f010bd10 0x7ff4f010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.493+0000 7ff4f7a88640 1 --2- 172.21.15.31:0/3130532479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f01032f0 0x7ff4f010b7d0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.493+0000 7ff4f7a88640 1 --2- 172.21.15.31:0/3130532479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f01029d0 0x7ff4f0102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.493+0000 7ff4f7a88640 1 -- 172.21.15.31:0/3130532479 >> 172.21.15.31:0/3130532479 conn(0x7ff4f00777c0 msgr2=0x7ff4f0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.493+0000 7ff4f7a88640 1 -- 172.21.15.31:0/3130532479 shutdown_connections 2024-04-25T02:15:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.493+0000 7ff4f7a88640 1 -- 172.21.15.31:0/3130532479 wait complete. 2024-04-25T02:15:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f7a88640 1 Processor -- start 2024-04-25T02:15:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f7a88640 1 -- start start 2024-04-25T02:15:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f7a88640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f01029d0 0x7ff4f0197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f7a88640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4f01032f0 0x7ff4f01984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f7a88640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f010bd10 0x7ff4f019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f7a88640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f01119c0 con 0x7ff4f01029d0 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f7a88640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f0111840 con 0x7ff4f010bd10 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f7a88640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f0111b40 con 0x7ff4f01032f0 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f5824640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f01029d0 0x7ff4f0197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f5824640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f01029d0 0x7ff4f0197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55490/0 (socket says 172.21.15.31:55490) 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f5824640 1 -- 172.21.15.31:0/220613701 learned_addr learned my addr 172.21.15.31:0/220613701 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f5023640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4f01032f0 0x7ff4f01984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.494+0000 7ff4f6025640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f010bd10 0x7ff4f019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4f5023640 1 -- 172.21.15.31:0/220613701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f010bd10 msgr2=0x7ff4f019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4f5023640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f010bd10 0x7ff4f019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4f5023640 1 -- 172.21.15.31:0/220613701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f01029d0 msgr2=0x7ff4f0197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4f5023640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f01029d0 0x7ff4f0197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4f5023640 1 -- 172.21.15.31:0/220613701 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f019cf50 con 0x7ff4f01032f0 2024-04-25T02:15:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4f5824640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f01029d0 0x7ff4f0197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:15:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4f6025640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f010bd10 0x7ff4f019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:15:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4f5023640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4f01032f0 0x7ff4f01984a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff4e002aa30 tx=0x7ff4e0031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4e6ffd640 1 -- 172.21.15.31:0/220613701 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e003f030 con 0x7ff4f01032f0 2024-04-25T02:15:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4f7a88640 1 -- 172.21.15.31:0/220613701 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f019d1e0 con 0x7ff4f01032f0 2024-04-25T02:15:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4e6ffd640 1 -- 172.21.15.31:0/220613701 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4e0004490 con 0x7ff4f01032f0 2024-04-25T02:15:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4e6ffd640 1 -- 172.21.15.31:0/220613701 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0035070 con 0x7ff4f01032f0 2024-04-25T02:15:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.495+0000 7ff4f7a88640 1 -- 172.21.15.31:0/220613701 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f01a4ad0 con 0x7ff4f01032f0 2024-04-25T02:15:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.498+0000 7ff4e6ffd640 1 -- 172.21.15.31:0/220613701 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff4e0042030 con 0x7ff4f01032f0 2024-04-25T02:15:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.499+0000 7ff4e6ffd640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff4dc0a2e20 0x7ff4dc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.499+0000 7ff4fa793740 1 Processor -- start 2024-04-25T02:15:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.499+0000 7ff4f5824640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff4dc0a2e20 0x7ff4dc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.499+0000 7ff4e6ffd640 1 -- 172.21.15.31:0/220613701 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7ff4e00ef430 con 0x7ff4f01032f0 2024-04-25T02:15:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4fa793740 1 -- start start 2024-04-25T02:15:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653db650 0x55c6653dba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 0x55c6653e6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653e6640 0x55c6653e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4fa793740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c66533bcb0 con 0x55c6653dbf70 2024-04-25T02:15:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4fa793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c6652e45b0 con 0x55c6653db650 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4fa793740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c665337c70 con 0x55c6653e6640 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4f5824640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff4dc0a2e20 0x7ff4dc0a52e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec011550 tx=0x7ff4ec005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4f5824640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653db650 0x55c6653dba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4f5824640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653db650 0x55c6653dba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:35506/0 (socket says 172.21.15.31:35506) 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4f5824640 1 -- 172.21.15.31:0/2518219766 learned_addr learned my addr 172.21.15.31:0/2518219766 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4f6025640 1 --2- 172.21.15.31:0/2518219766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653e6640 0x55c6653e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4f5023640 1 --2- 172.21.15.31:0/2518219766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 0x55c6653e6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4f5824640 1 -- 172.21.15.31:0/2518219766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653e6640 msgr2=0x55c6653e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4f5824640 1 --2- 172.21.15.31:0/2518219766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653e6640 0x55c6653e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4f5824640 1 -- 172.21.15.31:0/2518219766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 msgr2=0x55c6653e6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4f5824640 1 --2- 172.21.15.31:0/2518219766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 0x55c6653e6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.500+0000 7ff4f5824640 1 -- 172.21.15.31:0/2518219766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c665050210 con 0x55c6653db650 2024-04-25T02:15:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.501+0000 7ff4f5824640 1 --2- 172.21.15.31:0/2518219766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653db650 0x55c6653dba30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00f770 tx=0x7ff4ec014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dc93629f2e27c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:15.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.501+0000 7ff4dbfff640 1 -- 172.21.15.31:0/2518219766 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec01c940 con 0x55c6653db650 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.501+0000 7ff4dbfff640 1 -- 172.21.15.31:0/2518219766 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4ec0263f0 con 0x55c6653db650 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.501+0000 7ff4dbfff640 1 -- 172.21.15.31:0/2518219766 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec00e610 con 0x55c6653db650 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.502+0000 7ff4fa793740 1 -- 172.21.15.31:0/2518219766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653db650 msgr2=0x55c6653dba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.502+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2518219766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653db650 0x55c6653dba30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00f770 tx=0x7ff4ec014000 comp rx=0 tx=0).stop 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.502+0000 7ff4fa793740 1 -- 172.21.15.31:0/2518219766 shutdown_connections 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.502+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2518219766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653e6640 0x55c6653e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.502+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2518219766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 0x55c6653e6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.502+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2518219766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653db650 0x55c6653dba30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.502+0000 7ff4fa793740 1 -- 172.21.15.31:0/2518219766 >> 172.21.15.31:0/2518219766 conn(0x55c6653d7970 msgr2=0x55c6653d7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.502+0000 7ff4fa793740 1 -- 172.21.15.31:0/2518219766 shutdown_connections 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.502+0000 7ff4fa793740 1 -- 172.21.15.31:0/2518219766 wait complete. 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.502+0000 7ff4fa793740 1 -- start start 2024-04-25T02:15:15.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.502+0000 7ff4fa793740 1 Processor -- start 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653e5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653dbf70 0x55c6653e34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6653e39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4fa793740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c6653e9430 con 0x55c6653db650 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4fa793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c6652f7b50 con 0x55c6653e6640 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4fa793740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c6653e95b0 con 0x55c6653dbf70 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4f5824640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653e5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4f6025640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6653e39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4f5824640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653e5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55508/0 (socket says 172.21.15.31:55508) 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4f5824640 1 -- 172.21.15.31:0/2341354099 learned_addr learned my addr 172.21.15.31:0/2341354099 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4fa793740 10 client.0 _ll_register_callbacks cb 0x55c665332ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4f5023640 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653dbf70 0x55c6653e34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4f6025640 1 -- 172.21.15.31:0/2341354099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653dbf70 msgr2=0x55c6653e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4f6025640 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653dbf70 0x55c6653e34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4f6025640 1 -- 172.21.15.31:0/2341354099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 msgr2=0x55c6653e5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4f6025640 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653e5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4f6025640 1 -- 172.21.15.31:0/2341354099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c66511c620 con 0x55c6653e6640 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.503+0000 7ff4f5824640 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653e5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.504+0000 7ff4f6025640 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6653e39f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8009fd0 tx=0x7ff4e8012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b90fc892923ba2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.504+0000 7ff4d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.504+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2341354099 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8003970 con 0x55c6653e6640 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.504+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2341354099 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7ff4e8012e00 con 0x55c6653e6640 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.504+0000 7ff4fa793740 1 -- 172.21.15.31:0/2341354099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4ec0263f0 con 0x55c6653e6640 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.504+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2341354099 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff4e8003b10 con 0x55c6653e6640 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.504+0000 7ff4fa793740 10 client.27359 fetch_fsmap learned FSMap version 49 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.504+0000 7ff4fa793740 1 -- 172.21.15.31:0/2341354099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c6653f99d0 con 0x55c6653e6640 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.504+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2341354099 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4ec0263f0 con 0x55c6653e6640 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.505+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2341354099 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7ff4e801c460 con 0x55c6653e6640 2024-04-25T02:15:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.505+0000 7ff4fa793740 10 client.27359 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:15:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.505+0000 7ff4fa793740 10 client.27359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:15:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.505+0000 7ff4fa793740 20 client.27359 populate_metadata read hostname 'smithi031' 2024-04-25T02:15:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.505+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c6653faf00 0x55c6653fd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.505+0000 7ff4fa793740 4 client.27359 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:15:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.505+0000 7ff4fa793740 1 -- 172.21.15.31:0/2341354099 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6653491f0 con 0x55c6653faf00 2024-04-25T02:15:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.505+0000 7ff4f5824640 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c6653faf00 0x55c6653fd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.505+0000 7ff4d97fa640 10 client.27359 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.505+0000 7ff4f5824640 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c6653faf00 0x55c6653fd310 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4363f89a231df63d server_cookie=10e21db8969253cd in_seq=0 out_seq=0 2024-04-25T02:15:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.506+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2341354099 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c6653491f0 con 0x55c6653faf00 2024-04-25T02:15:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.506+0000 7ff4d97fa640 10 client.27359 handle_command_reply: tid=0 2024-04-25T02:15:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.506+0000 7ff4fa793740 1 client.27359 shutdown 2024-04-25T02:15:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.506+0000 7ff4fa793740 1 -- 172.21.15.31:0/2341354099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 msgr2=0x55c6653e39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.506+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6653e39f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8009fd0 tx=0x7ff4e8012820 comp rx=0 tx=0).stop 2024-04-25T02:15:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.507+0000 7ff4fa793740 1 -- 172.21.15.31:0/2341354099 shutdown_connections 2024-04-25T02:15:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.507+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c6653faf00 0x55c6653fd310 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.507+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6653e39f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.507+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653dbf70 0x55c6653e34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.507+0000 7ff4d97fa640 0 client.27359 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.507+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2341354099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653e5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.507+0000 7ff4fa793740 1 -- 172.21.15.31:0/2341354099 >> 172.21.15.31:0/2341354099 conn(0x55c6653d7970 msgr2=0x55c6653dd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.507+0000 7ff4fa793740 1 -- 172.21.15.31:0/2341354099 shutdown_connections 2024-04-25T02:15:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.507+0000 7ff4fa793740 1 -- 172.21.15.31:0/2341354099 wait complete. 2024-04-25T02:15:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.507+0000 7ff4fa793740 20 client.27359 trim_cache size 0 max 16384 2024-04-25T02:15:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.537+0000 7ff4fa793740 1 Processor -- start 2024-04-25T02:15:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.537+0000 7ff4fa793740 1 -- start start 2024-04-25T02:15:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.537+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653db650 0x55c6653dddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.537+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 0x55c6653e5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.537+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6653e3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.537+0000 7ff4fa793740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d400e6e0 con 0x55c6653dbf70 2024-04-25T02:15:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.537+0000 7ff4fa793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d4010220 con 0x55c6653e6640 2024-04-25T02:15:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.537+0000 7ff4fa793740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d400edb0 con 0x55c6653db650 2024-04-25T02:15:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f5023640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 0x55c6653e5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f5023640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 0x55c6653e5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55512/0 (socket says 172.21.15.31:55512) 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f5824640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653db650 0x55c6653dddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f5023640 1 -- 172.21.15.31:0/3799075127 learned_addr learned my addr 172.21.15.31:0/3799075127 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f6025640 1 --2- 172.21.15.31:0/3799075127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6653e3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f5824640 1 -- 172.21.15.31:0/3799075127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 msgr2=0x55c6653e3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f5824640 1 --2- 172.21.15.31:0/3799075127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6653e3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f5824640 1 -- 172.21.15.31:0/3799075127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 msgr2=0x55c6653e5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f5824640 1 --2- 172.21.15.31:0/3799075127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 0x55c6653e5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f5824640 1 -- 172.21.15.31:0/3799075127 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6653d7290 con 0x55c6653db650 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f5023640 1 --2- 172.21.15.31:0/3799075127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 0x55c6653e5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4f5824640 1 --2- 172.21.15.31:0/3799075127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653db650 0x55c6653dddb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00f740 tx=0x7ff4ec018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a388c1dc40da049 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4db7fe640 1 -- 172.21.15.31:0/3799075127 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec039070 con 0x55c6653db650 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4db7fe640 1 -- 172.21.15.31:0/3799075127 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4ec0263f0 con 0x55c6653db650 2024-04-25T02:15:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.538+0000 7ff4db7fe640 1 -- 172.21.15.31:0/3799075127 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec01c9a0 con 0x55c6653db650 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.539+0000 7ff4fa793740 1 -- 172.21.15.31:0/3799075127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653db650 msgr2=0x55c6653dddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.539+0000 7ff4fa793740 1 --2- 172.21.15.31:0/3799075127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653db650 0x55c6653dddb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00f740 tx=0x7ff4ec018000 comp rx=0 tx=0).stop 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.540+0000 7ff4fa793740 1 -- 172.21.15.31:0/3799075127 shutdown_connections 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.540+0000 7ff4fa793740 1 --2- 172.21.15.31:0/3799075127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6653e3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.540+0000 7ff4fa793740 1 --2- 172.21.15.31:0/3799075127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653dbf70 0x55c6653e5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.540+0000 7ff4fa793740 1 --2- 172.21.15.31:0/3799075127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653db650 0x55c6653dddb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.540+0000 7ff4fa793740 1 -- 172.21.15.31:0/3799075127 >> 172.21.15.31:0/3799075127 conn(0x55c6653d7970 msgr2=0x55c6653e2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.540+0000 7ff4fa793740 1 -- 172.21.15.31:0/3799075127 shutdown_connections 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.540+0000 7ff4fa793740 1 -- 172.21.15.31:0/3799075127 wait complete. 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.540+0000 7ff4fa793740 1 -- start start 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.540+0000 7ff4fa793740 1 Processor -- start 2024-04-25T02:15:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653dbf70 0x55c6653f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4fa793740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6654604f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4fa793740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d400fb50 con 0x55c6653db650 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4fa793740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d400f480 con 0x55c6653e6640 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4fa793740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d400c6e0 con 0x55c6653dbf70 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4fa793740 10 client.0 _ll_register_callbacks cb 0x7ff4e800f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4f5824640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4f5824640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653f56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55516/0 (socket says 172.21.15.31:55516) 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4f5824640 1 -- 172.21.15.31:0/2915317488 learned_addr learned my addr 172.21.15.31:0/2915317488 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4f6025640 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6654604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4f5023640 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653dbf70 0x55c6653f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4f5824640 1 -- 172.21.15.31:0/2915317488 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653dbf70 msgr2=0x55c6653f5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4f5824640 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653dbf70 0x55c6653f5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4f5824640 1 -- 172.21.15.31:0/2915317488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 msgr2=0x55c6654604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4f5824640 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6654604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.541+0000 7ff4f5824640 1 -- 172.21.15.31:0/2915317488 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c665460a30 con 0x55c6653db650 2024-04-25T02:15:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.542+0000 7ff4f5824640 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653f56c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec0097c0 tx=0x7ff4ec01c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d507109231892cda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.542+0000 7ff4d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:15:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.542+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2915317488 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec039070 con 0x55c6653db650 2024-04-25T02:15:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.542+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2915317488 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7ff4ec01c920 con 0x55c6653db650 2024-04-25T02:15:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.542+0000 7ff4fa793740 1 -- 172.21.15.31:0/2915317488 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4ec0263f0 con 0x55c6653db650 2024-04-25T02:15:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.542+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2915317488 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff4ec045050 con 0x55c6653db650 2024-04-25T02:15:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.542+0000 7ff4fa793740 10 client.17808 fetch_fsmap learned FSMap version 49 2024-04-25T02:15:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.542+0000 7ff4fa793740 1 -- 172.21.15.31:0/2915317488 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c665461ca0 con 0x55c6653db650 2024-04-25T02:15:15.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.543+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2915317488 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4ec0263f0 con 0x55c6653db650 2024-04-25T02:15:15.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.543+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2915317488 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7ff4ec034170 con 0x55c6653db650 2024-04-25T02:15:15.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.543+0000 7ff4fa793740 10 client.17808 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:15:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.543+0000 7ff4fa793740 10 client.17808 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:15:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.543+0000 7ff4fa793740 20 client.17808 populate_metadata read hostname 'smithi031' 2024-04-25T02:15:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.543+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c6654620e0 0x55c6654644d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.543+0000 7ff4fa793740 4 client.17808 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:15:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.543+0000 7ff4fa793740 1 -- 172.21.15.31:0/2915317488 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c665343540 con 0x55c6654620e0 2024-04-25T02:15:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.543+0000 7ff4f5023640 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c6654620e0 0x55c6654644d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.544+0000 7ff4d97fa640 10 client.17808 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.544+0000 7ff4f5023640 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c6654620e0 0x55c6654644d0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f3bee01921ee8ce server_cookie=27a44e2d89808d9d in_seq=0 out_seq=0 2024-04-25T02:15:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.597+0000 7ff4d97fa640 1 -- 172.21.15.31:0/2915317488 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c665343540 con 0x55c6654620e0 2024-04-25T02:15:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4d97fa640 10 client.17808 handle_command_reply: tid=0 2024-04-25T02:15:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 1 client.17808 shutdown 2024-04-25T02:15:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 1 -- 172.21.15.31:0/2915317488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 msgr2=0x55c6653f56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653f56c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec0097c0 tx=0x7ff4ec01c740 comp rx=0 tx=0).stop 2024-04-25T02:15:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 1 -- 172.21.15.31:0/2915317488 shutdown_connections 2024-04-25T02:15:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c6654620e0 0x55c6654644d0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6653e6640 0x55c6654604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4d97fa640 0 client.17808 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6653dbf70 0x55c6653f5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 1 --2- 172.21.15.31:0/2915317488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6653db650 0x55c6653f56c0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 1 -- 172.21.15.31:0/2915317488 >> 172.21.15.31:0/2915317488 conn(0x55c6653d7970 msgr2=0x55c6653e2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 1 -- 172.21.15.31:0/2915317488 shutdown_connections 2024-04-25T02:15:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 1 -- 172.21.15.31:0/2915317488 wait complete. 2024-04-25T02:15:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.598+0000 7ff4fa793740 20 client.17808 trim_cache size 0 max 16384 2024-04-25T02:15:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:15:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:15:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:15:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:15:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 -- 172.21.15.31:0/220613701 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff4dc0a2e20 msgr2=0x7ff4dc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff4dc0a2e20 0x7ff4dc0a52e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec011550 tx=0x7ff4ec005f90 comp rx=0 tx=0).stop 2024-04-25T02:15:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 -- 172.21.15.31:0/220613701 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4f01032f0 msgr2=0x7ff4f01984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4f01032f0 0x7ff4f01984a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff4e002aa30 tx=0x7ff4e0031a20 comp rx=0 tx=0).stop 2024-04-25T02:15:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 -- 172.21.15.31:0/220613701 shutdown_connections 2024-04-25T02:15:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff4dc0a2e20 0x7ff4dc0a52e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4f010bd10 0x7ff4f019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4f01032f0 0x7ff4f01984a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 --2- 172.21.15.31:0/220613701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4f01029d0 0x7ff4f0197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 -- 172.21.15.31:0/220613701 >> 172.21.15.31:0/220613701 conn(0x7ff4f00777c0 msgr2=0x7ff4f00fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 -- 172.21.15.31:0/220613701 shutdown_connections 2024-04-25T02:15:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:15.599+0000 7ff4f7a88640 1 -- 172.21.15.31:0/220613701 wait complete. 2024-04-25T02:15:16.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 72378 8.84 MB/sec execute 320 sec latency 186.798 ms 2024-04-25T02:15:16.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:15 smithi031 ceph-mon[29204]: pgmap v655: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-04-25T02:15:16.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:15 smithi040 ceph-mon[32418]: pgmap v655: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-04-25T02:15:16.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:15 smithi134 ceph-mon[33098]: pgmap v655: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-04-25T02:15:17.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 72532 8.84 MB/sec execute 321 sec latency 99.973 ms 2024-04-25T02:15:18.105 INFO:tasks.workunit.client.0.smithi031.stdout: 10 72707 8.85 MB/sec execute 322 sec latency 58.421 ms 2024-04-25T02:15:18.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:17 smithi040 ceph-mon[32418]: pgmap v656: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-04-25T02:15:18.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:17 smithi134 ceph-mon[33098]: pgmap v656: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-04-25T02:15:18.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:17 smithi031 ceph-mon[29204]: pgmap v656: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-04-25T02:15:19.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 72863 8.83 MB/sec execute 323 sec latency 121.550 ms 2024-04-25T02:15:20.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 73013 8.83 MB/sec execute 324 sec latency 57.133 ms 2024-04-25T02:15:20.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:19 smithi040 ceph-mon[32418]: pgmap v657: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-04-25T02:15:20.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:19 smithi134 ceph-mon[33098]: pgmap v657: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-04-25T02:15:20.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:19 smithi031 ceph-mon[29204]: pgmap v657: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.3 MiB/s wr, 275 op/s 2024-04-25T02:15:21.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 73143 8.82 MB/sec execute 325 sec latency 137.990 ms 2024-04-25T02:15:21.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:21 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:15:21.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:21 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:15:21.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:21 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:15:22.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 73300 8.82 MB/sec execute 326 sec latency 180.203 ms 2024-04-25T02:15:22.242 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:22 smithi031 ceph-mon[29204]: pgmap v658: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.5 MiB/s wr, 353 op/s 2024-04-25T02:15:22.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:22 smithi040 ceph-mon[32418]: pgmap v658: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.5 MiB/s wr, 353 op/s 2024-04-25T02:15:22.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:22 smithi134 ceph-mon[33098]: pgmap v658: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.5 MiB/s wr, 353 op/s 2024-04-25T02:15:23.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 73511 8.83 MB/sec execute 327 sec latency 75.061 ms 2024-04-25T02:15:24.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 73740 8.85 MB/sec execute 328 sec latency 65.881 ms 2024-04-25T02:15:24.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:24 smithi040 ceph-mon[32418]: pgmap v659: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.2 MiB/s wr, 244 op/s 2024-04-25T02:15:24.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:24 smithi134 ceph-mon[33098]: pgmap v659: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.2 MiB/s wr, 244 op/s 2024-04-25T02:15:24.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:24 smithi031 ceph-mon[29204]: pgmap v659: 97 pgs: 97 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.2 MiB/s wr, 244 op/s 2024-04-25T02:15:25.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 73854 8.83 MB/sec execute 329 sec latency 188.028 ms 2024-04-25T02:15:26.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 74023 8.85 MB/sec execute 330 sec latency 253.726 ms 2024-04-25T02:15:26.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:26 smithi040 ceph-mon[32418]: pgmap v660: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.9 MiB/s wr, 371 op/s 2024-04-25T02:15:26.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:26 smithi134 ceph-mon[33098]: pgmap v660: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.9 MiB/s wr, 371 op/s 2024-04-25T02:15:26.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:26 smithi031 ceph-mon[29204]: pgmap v660: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.9 MiB/s wr, 371 op/s 2024-04-25T02:15:27.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 74204 8.86 MB/sec execute 331 sec latency 54.354 ms 2024-04-25T02:15:28.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 74355 8.86 MB/sec execute 332 sec latency 148.802 ms 2024-04-25T02:15:28.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:28 smithi040 ceph-mon[32418]: pgmap v661: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 247 op/s 2024-04-25T02:15:28.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:28 smithi134 ceph-mon[33098]: pgmap v661: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 247 op/s 2024-04-25T02:15:28.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:28 smithi031 ceph-mon[29204]: pgmap v661: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 247 op/s 2024-04-25T02:15:28.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.732+0000 7efe04f4f640 1 -- 172.21.15.31:0/2736146044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe00107830 msgr2=0x7efe00107c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:28.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.732+0000 7efe04f4f640 1 --2- 172.21.15.31:0/2736146044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe00107830 0x7efe00107c10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009930 tx=0x7efdf002a4d0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.732+0000 7efe04f4f640 1 -- 172.21.15.31:0/2736146044 shutdown_connections 2024-04-25T02:15:28.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.732+0000 7efe04f4f640 1 --2- 172.21.15.31:0/2736146044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe0006a2e0 0x7efe0010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.732+0000 7efe04f4f640 1 --2- 172.21.15.31:0/2736146044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe00069940 0x7efe00069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.732+0000 7efe04f4f640 1 --2- 172.21.15.31:0/2736146044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe00107830 0x7efe00107c10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.732+0000 7efe04f4f640 1 -- 172.21.15.31:0/2736146044 >> 172.21.15.31:0/2736146044 conn(0x7efe00077520 msgr2=0x7efe00077930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:28.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.732+0000 7efe04f4f640 1 -- 172.21.15.31:0/2736146044 shutdown_connections 2024-04-25T02:15:28.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.732+0000 7efe04f4f640 1 -- 172.21.15.31:0/2736146044 wait complete. 2024-04-25T02:15:28.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efe04f4f640 1 Processor -- start 2024-04-25T02:15:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efe04f4f640 1 -- start start 2024-04-25T02:15:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efe04f4f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe00069940 0x7efe00197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efe04f4f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0006a2e0 0x7efe00198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efe04f4f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe00107830 0x7efe0019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efe04f4f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efe00111a70 con 0x7efe00069940 2024-04-25T02:15:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efe04f4f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efe001118f0 con 0x7efe0006a2e0 2024-04-25T02:15:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efe04f4f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efe00111bf0 con 0x7efe00107830 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efdfe59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe00069940 0x7efe00197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efdfe59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe00069940 0x7efe00197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37880/0 (socket says 172.21.15.31:37880) 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efdfe59c640 1 -- 172.21.15.31:0/1750640983 learned_addr learned my addr 172.21.15.31:0/1750640983 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0006a2e0 0x7efe00198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.733+0000 7efdfed9d640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe00107830 0x7efe0019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.734+0000 7efdfdd9b640 1 -- 172.21.15.31:0/1750640983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe00107830 msgr2=0x7efe0019c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.734+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe00107830 0x7efe0019c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.734+0000 7efdfdd9b640 1 -- 172.21.15.31:0/1750640983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe00069940 msgr2=0x7efe00197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.734+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe00069940 0x7efe00197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.734+0000 7efdfdd9b640 1 -- 172.21.15.31:0/1750640983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0019ce20 con 0x7efe0006a2e0 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.734+0000 7efdfe59c640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe00069940 0x7efe00197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.734+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0006a2e0 0x7efe00198370 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efdf400e9f0 tx=0x7efdf400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.734+0000 7efdef7fe640 1 -- 172.21.15.31:0/1750640983 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf400cd80 con 0x7efe0006a2e0 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.734+0000 7efdef7fe640 1 -- 172.21.15.31:0/1750640983 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdf40035f0 con 0x7efe0006a2e0 2024-04-25T02:15:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.734+0000 7efe04f4f640 1 -- 172.21.15.31:0/1750640983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe0019d110 con 0x7efe0006a2e0 2024-04-25T02:15:28.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.734+0000 7efe04f4f640 1 -- 172.21.15.31:0/1750640983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe001a49a0 con 0x7efe0006a2e0 2024-04-25T02:15:28.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.735+0000 7efdef7fe640 1 -- 172.21.15.31:0/1750640983 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf4010670 con 0x7efe0006a2e0 2024-04-25T02:15:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.737+0000 7efe07c5c740 1 Processor -- start 2024-04-25T02:15:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.737+0000 7efdef7fe640 1 -- 172.21.15.31:0/1750640983 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efdf4003fa0 con 0x7efe0006a2e0 2024-04-25T02:15:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.737+0000 7efe07c5c740 1 -- start start 2024-04-25T02:15:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efe07c5c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880240 0x55ab5c880620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efe07c5c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 0x55ab5c88acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efe07c5c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c88b230 0x55ab5c88d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efe07c5c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ab5c7e4cb0 con 0x55ab5c88b230 2024-04-25T02:15:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efe07c5c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ab5c78d5b0 con 0x55ab5c880240 2024-04-25T02:15:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efe07c5c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ab5c7e0c70 con 0x55ab5c880b60 2024-04-25T02:15:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880240 0x55ab5c880620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880240 0x55ab5c880620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47144/0 (socket says 172.21.15.31:47144) 2024-04-25T02:15:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfed9d640 1 -- 172.21.15.31:0/212061570 learned_addr learned my addr 172.21.15.31:0/212061570 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfe59c640 1 --2- 172.21.15.31:0/212061570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 0x55ab5c88acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfed9d640 1 -- 172.21.15.31:0/212061570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 msgr2=0x55ab5c88acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfed9d640 1 --2- 172.21.15.31:0/212061570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 0x55ab5c88acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfed9d640 1 -- 172.21.15.31:0/212061570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c88b230 msgr2=0x55ab5c88d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/212061570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c88b230 0x55ab5c88d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfed9d640 1 --2- 172.21.15.31:0/212061570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c88b230 0x55ab5c88d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfed9d640 1 -- 172.21.15.31:0/212061570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab5c4f9210 con 0x55ab5c880240 2024-04-25T02:15:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/212061570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c88b230 0x55ab5c88d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:15:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.738+0000 7efdfed9d640 1 --2- 172.21.15.31:0/212061570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880240 0x55ab5c880620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009fd0 tx=0x7efdf80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dec614d7c212606 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.739+0000 7efdef7fe640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efde80a2e20 0x7efde80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.739+0000 7efdfe59c640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efde80a2e20 0x7efde80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.739+0000 7efdef7fe640 1 -- 172.21.15.31:0/1750640983 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7efdf40cf4d0 con 0x7efe0006a2e0 2024-04-25T02:15:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.739+0000 7efdecff9640 1 -- 172.21.15.31:0/212061570 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8004d20 con 0x55ab5c880240 2024-04-25T02:15:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.739+0000 7efdecff9640 1 -- 172.21.15.31:0/212061570 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdf801a070 con 0x55ab5c880240 2024-04-25T02:15:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.739+0000 7efdecff9640 1 -- 172.21.15.31:0/212061570 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8016570 con 0x55ab5c880240 2024-04-25T02:15:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.740+0000 7efdfe59c640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efde80a2e20 0x7efde80a52e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efdf0000c00 tx=0x7efdf0005a20 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.740+0000 7efe07c5c740 1 -- 172.21.15.31:0/212061570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880240 msgr2=0x55ab5c880620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.740+0000 7efe07c5c740 1 --2- 172.21.15.31:0/212061570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880240 0x55ab5c880620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009fd0 tx=0x7efdf80118d0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.740+0000 7efe07c5c740 1 -- 172.21.15.31:0/212061570 shutdown_connections 2024-04-25T02:15:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.740+0000 7efe07c5c740 1 --2- 172.21.15.31:0/212061570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c88b230 0x55ab5c88d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.740+0000 7efe07c5c740 1 --2- 172.21.15.31:0/212061570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 0x55ab5c88acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.740+0000 7efe07c5c740 1 --2- 172.21.15.31:0/212061570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880240 0x55ab5c880620 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.740+0000 7efe07c5c740 1 -- 172.21.15.31:0/212061570 >> 172.21.15.31:0/212061570 conn(0x55ab5c87c560 msgr2=0x55ab5c87c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.740+0000 7efe07c5c740 1 -- 172.21.15.31:0/212061570 shutdown_connections 2024-04-25T02:15:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.740+0000 7efe07c5c740 1 -- 172.21.15.31:0/212061570 wait complete. 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efe07c5c740 1 -- start start 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efe07c5c740 1 Processor -- start 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efe07c5c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c888140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efe07c5c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 0x55ab5c888680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efe07c5c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c88b230 0x55ab5c888bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efe07c5c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ab5c88e020 con 0x55ab5c880240 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efe07c5c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ab5c7a0b50 con 0x55ab5c88b230 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efe07c5c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ab5c88e1a0 con 0x55ab5c880b60 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efdfdd9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 0x55ab5c888680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efdfdd9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 0x55ab5c888680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:39304/0 (socket says 172.21.15.31:39304) 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efdfdd9b640 1 -- 172.21.15.31:0/2988014471 learned_addr learned my addr 172.21.15.31:0/2988014471 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efe07c5c740 10 client.0 _ll_register_callbacks cb 0x55ab5c7dbca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.741+0000 7efdfed9d640 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c88b230 0x55ab5c888bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.742+0000 7efdfdd9b640 1 -- 172.21.15.31:0/2988014471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c88b230 msgr2=0x55ab5c888bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.742+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c88b230 0x55ab5c888bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.742+0000 7efdfdd9b640 1 -- 172.21.15.31:0/2988014471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 msgr2=0x55ab5c888140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.742+0000 7efdfe59c640 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c888140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.742+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c888140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.742+0000 7efdfdd9b640 1 -- 172.21.15.31:0/2988014471 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab5c5c5620 con 0x55ab5c880b60 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.742+0000 7efdfed9d640 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c88b230 0x55ab5c888bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.742+0000 7efdfe59c640 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c888140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.742+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 0x55ab5c888680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efdf4000c00 tx=0x7efdf409c0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78e5eb959a055df8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.742+0000 7efde5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.742+0000 7efde5ffb640 1 -- 172.21.15.31:0/2988014471 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf409fd60 con 0x55ab5c880b60 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.743+0000 7efde5ffb640 1 -- 172.21.15.31:0/2988014471 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7efdf4098020 con 0x55ab5c880b60 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.743+0000 7efe07c5c740 1 -- 172.21.15.31:0/2988014471 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdf801a070 con 0x55ab5c880b60 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.743+0000 7efde5ffb640 1 -- 172.21.15.31:0/2988014471 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7efdf40b9050 con 0x55ab5c880b60 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.743+0000 7efe07c5c740 10 client.27154 fetch_fsmap learned FSMap version 50 2024-04-25T02:15:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.743+0000 7efe07c5c740 1 -- 172.21.15.31:0/2988014471 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55ab5c89a180 con 0x55ab5c880b60 2024-04-25T02:15:28.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.743+0000 7efde5ffb640 1 -- 172.21.15.31:0/2988014471 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdf801a070 con 0x55ab5c880b60 2024-04-25T02:15:28.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.743+0000 7efde5ffb640 1 -- 172.21.15.31:0/2988014471 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7efdf409cce0 con 0x55ab5c880b60 2024-04-25T02:15:28.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.743+0000 7efe07c5c740 10 client.27154 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:15:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.744+0000 7efe07c5c740 10 client.27154 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:15:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.744+0000 7efe07c5c740 20 client.27154 populate_metadata read hostname 'smithi031' 2024-04-25T02:15:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.744+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ab5c89ba40 0x55ab5c89dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.744+0000 7efe07c5c740 4 client.27154 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:15:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.744+0000 7efe07c5c740 1 -- 172.21.15.31:0/2988014471 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab5c7f21f0 con 0x55ab5c89ba40 2024-04-25T02:15:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.744+0000 7efdfe59c640 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ab5c89ba40 0x55ab5c89dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.745+0000 7efde5ffb640 10 client.27154 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.745+0000 7efdfe59c640 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ab5c89ba40 0x55ab5c89dee0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d18fae5999c399d1 server_cookie=6fff2079ed5f8c64 in_seq=0 out_seq=0 2024-04-25T02:15:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efde5ffb640 1 -- 172.21.15.31:0/2988014471 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab5c7f21f0 con 0x55ab5c89ba40 2024-04-25T02:15:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efde5ffb640 10 client.27154 handle_command_reply: tid=0 2024-04-25T02:15:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 1 client.27154 shutdown 2024-04-25T02:15:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 1 -- 172.21.15.31:0/2988014471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 msgr2=0x55ab5c888680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 0x55ab5c888680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efdf4000c00 tx=0x7efdf409c0b0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 1 -- 172.21.15.31:0/2988014471 shutdown_connections 2024-04-25T02:15:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ab5c89ba40 0x55ab5c89dee0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c88b230 0x55ab5c888bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c880b60 0x55ab5c888680 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2988014471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c888140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 1 -- 172.21.15.31:0/2988014471 >> 172.21.15.31:0/2988014471 conn(0x55ab5c87c560 msgr2=0x55ab5c882260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efde5ffb640 0 client.27154 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 1 -- 172.21.15.31:0/2988014471 shutdown_connections 2024-04-25T02:15:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 1 -- 172.21.15.31:0/2988014471 wait complete. 2024-04-25T02:15:28.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.747+0000 7efe07c5c740 20 client.27154 trim_cache size 0 max 16384 2024-04-25T02:15:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efe07c5c740 1 Processor -- start 2024-04-25T02:15:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efe07c5c740 1 -- start start 2024-04-25T02:15:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efe07c5c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c88a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efe07c5c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880b60 0x55ab5c896f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efe07c5c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c88b230 0x55ab5c887300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efe07c5c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efde000e690 con 0x55ab5c880240 2024-04-25T02:15:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efe07c5c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efde00101d0 con 0x55ab5c880b60 2024-04-25T02:15:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efe07c5c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efde000ed60 con 0x55ab5c88b230 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efdfe59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c88a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efdfe59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c88a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37886/0 (socket says 172.21.15.31:37886) 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efdfe59c640 1 -- 172.21.15.31:0/3231752992 learned_addr learned my addr 172.21.15.31:0/3231752992 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/3231752992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880b60 0x55ab5c896f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efdfed9d640 1 --2- 172.21.15.31:0/3231752992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c88b230 0x55ab5c887300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efdfe59c640 1 -- 172.21.15.31:0/3231752992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c88b230 msgr2=0x55ab5c887300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efdfe59c640 1 --2- 172.21.15.31:0/3231752992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c88b230 0x55ab5c887300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efdfe59c640 1 -- 172.21.15.31:0/3231752992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880b60 msgr2=0x55ab5c896f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efdfe59c640 1 --2- 172.21.15.31:0/3231752992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880b60 0x55ab5c896f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.770+0000 7efdfe59c640 1 -- 172.21.15.31:0/3231752992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab5c880990 con 0x55ab5c880240 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.771+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/3231752992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880b60 0x55ab5c896f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.771+0000 7efdfe59c640 1 --2- 172.21.15.31:0/3231752992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c88a870 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7efdf0031440 tx=0x7efdf003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd659e425f65932 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.771+0000 7efde7fff640 1 -- 172.21.15.31:0/3231752992 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf0035050 con 0x55ab5c880240 2024-04-25T02:15:28.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.771+0000 7efde7fff640 1 -- 172.21.15.31:0/3231752992 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdf0056070 con 0x55ab5c880240 2024-04-25T02:15:28.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.771+0000 7efde7fff640 1 -- 172.21.15.31:0/3231752992 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf0031700 con 0x55ab5c880240 2024-04-25T02:15:28.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.772+0000 7efe07c5c740 1 -- 172.21.15.31:0/3231752992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 msgr2=0x55ab5c88a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:28.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.772+0000 7efe07c5c740 1 --2- 172.21.15.31:0/3231752992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c88a870 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7efdf0031440 tx=0x7efdf003e000 comp rx=0 tx=0).stop 2024-04-25T02:15:28.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.772+0000 7efe07c5c740 1 -- 172.21.15.31:0/3231752992 shutdown_connections 2024-04-25T02:15:28.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.772+0000 7efe07c5c740 1 --2- 172.21.15.31:0/3231752992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c88b230 0x55ab5c887300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.772+0000 7efe07c5c740 1 --2- 172.21.15.31:0/3231752992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880b60 0x55ab5c896f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.772+0000 7efe07c5c740 1 --2- 172.21.15.31:0/3231752992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c88a870 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.772+0000 7efe07c5c740 1 -- 172.21.15.31:0/3231752992 >> 172.21.15.31:0/3231752992 conn(0x55ab5c87c560 msgr2=0x55ab5c888100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:28.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.773+0000 7efe07c5c740 1 -- 172.21.15.31:0/3231752992 shutdown_connections 2024-04-25T02:15:28.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.773+0000 7efe07c5c740 1 -- 172.21.15.31:0/3231752992 wait complete. 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.773+0000 7efe07c5c740 1 -- start start 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.773+0000 7efe07c5c740 1 Processor -- start 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.773+0000 7efe07c5c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c893640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.774+0000 7efdfe59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c893640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.774+0000 7efdfe59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c893640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37900/0 (socket says 172.21.15.31:37900) 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.774+0000 7efdfe59c640 1 -- 172.21.15.31:0/2665651509 learned_addr learned my addr 172.21.15.31:0/2665651509 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.774+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880b60 0x55ab5c8849d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.774+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c88b230 0x55ab5c884f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.774+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880b60 0x55ab5c8849d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.774+0000 7efe07c5c740 1 -- 172.21.15.31:0/2665651509 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efde000fb00 con 0x55ab5c880240 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.774+0000 7efdfed9d640 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c88b230 0x55ab5c884f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.774+0000 7efe07c5c740 1 -- 172.21.15.31:0/2665651509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efde000f430 con 0x55ab5c880b60 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.774+0000 7efe07c5c740 1 -- 172.21.15.31:0/2665651509 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efde000c690 con 0x55ab5c88b230 2024-04-25T02:15:28.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.775+0000 7efdfe59c640 1 -- 172.21.15.31:0/2665651509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c88b230 msgr2=0x55ab5c884f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.775+0000 7efdfe59c640 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c88b230 0x55ab5c884f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.775+0000 7efdfe59c640 1 -- 172.21.15.31:0/2665651509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880b60 msgr2=0x55ab5c8849d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.775+0000 7efdfe59c640 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880b60 0x55ab5c8849d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.775+0000 7efdfe59c640 1 -- 172.21.15.31:0/2665651509 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab5c885450 con 0x55ab5c880240 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.775+0000 7efe07c5c740 10 client.0 _ll_register_callbacks cb 0x7efdf40044f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.775+0000 7efe07c5c740 1 -- 172.21.15.31:0/2665651509 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdf0056070 con 0x55ab5c880240 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.775+0000 7efdfe59c640 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c893640 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7efdf0050810 tx=0x7efdf0050ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8132873d33d19eb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.775+0000 7efde5ffb640 10 client.17820 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.775+0000 7efde5ffb640 1 -- 172.21.15.31:0/2665651509 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf005e030 con 0x55ab5c880240 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efde5ffb640 1 -- 172.21.15.31:0/2665651509 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7efdf004f040 con 0x55ab5c880240 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efde5ffb640 1 -- 172.21.15.31:0/2665651509 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7efdf0035070 con 0x55ab5c880240 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efe07c5c740 10 client.17820 fetch_fsmap learned FSMap version 50 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efe07c5c740 1 -- 172.21.15.31:0/2665651509 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=50}) -- 0x55ab5c901c80 con 0x55ab5c880240 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efde5ffb640 1 -- 172.21.15.31:0/2665651509 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7efdf00314a0 con 0x55ab5c880240 2024-04-25T02:15:28.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efe07c5c740 10 client.17820 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:15:28.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efe07c5c740 10 client.17820 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:15:28.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efe07c5c740 20 client.17820 populate_metadata read hostname 'smithi031' 2024-04-25T02:15:28.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ab5c9022d0 0x55ab5c9046f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:28.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efe07c5c740 4 client.17820 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:15:28.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efe07c5c740 1 -- 172.21.15.31:0/2665651509 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab5c7ec540 con 0x55ab5c9022d0 2024-04-25T02:15:28.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.776+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ab5c9022d0 0x55ab5c9046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:28.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.777+0000 7efde5ffb640 10 client.17820 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:28.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.777+0000 7efdfdd9b640 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ab5c9022d0 0x55ab5c9046f0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72fbd6c91bb6af4a server_cookie=1bfe0d9a98669e2c in_seq=0 out_seq=0 2024-04-25T02:15:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efde5ffb640 1 -- 172.21.15.31:0/2665651509 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ab5c7ec540 con 0x55ab5c9022d0 2024-04-25T02:15:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efde5ffb640 10 client.17820 handle_command_reply: tid=0 2024-04-25T02:15:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efe07c5c740 1 client.17820 shutdown 2024-04-25T02:15:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efe07c5c740 1 -- 172.21.15.31:0/2665651509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 msgr2=0x55ab5c893640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c893640 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7efdf0050810 tx=0x7efdf0050ae0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efe07c5c740 1 -- 172.21.15.31:0/2665651509 shutdown_connections 2024-04-25T02:15:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55ab5c9022d0 0x55ab5c9046f0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ab5c88b230 0x55ab5c884f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efde5ffb640 0 client.17820 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ab5c880b60 0x55ab5c8849d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efe07c5c740 1 --2- 172.21.15.31:0/2665651509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab5c880240 0x55ab5c893640 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efe07c5c740 1 -- 172.21.15.31:0/2665651509 >> 172.21.15.31:0/2665651509 conn(0x55ab5c87c560 msgr2=0x55ab5c887180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efe07c5c740 1 -- 172.21.15.31:0/2665651509 shutdown_connections 2024-04-25T02:15:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.850+0000 7efe07c5c740 1 -- 172.21.15.31:0/2665651509 wait complete. 2024-04-25T02:15:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe07c5c740 20 client.17820 trim_cache size 0 max 16384 2024-04-25T02:15:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:15:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:15:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:15:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 -- 172.21.15.31:0/1750640983 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efde80a2e20 msgr2=0x7efde80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efde80a2e20 0x7efde80a52e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efdf0000c00 tx=0x7efdf0005a20 comp rx=0 tx=0).stop 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 -- 172.21.15.31:0/1750640983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0006a2e0 msgr2=0x7efe00198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0006a2e0 0x7efe00198370 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efdf400e9f0 tx=0x7efdf400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 -- 172.21.15.31:0/1750640983 shutdown_connections 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7efde80a2e20 0x7efde80a52e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe00107830 0x7efe0019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0006a2e0 0x7efe00198370 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 --2- 172.21.15.31:0/1750640983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe00069940 0x7efe00197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 -- 172.21.15.31:0/1750640983 >> 172.21.15.31:0/1750640983 conn(0x7efe00077520 msgr2=0x7efe000fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 -- 172.21.15.31:0/1750640983 shutdown_connections 2024-04-25T02:15:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:28.851+0000 7efe04f4f640 1 -- 172.21.15.31:0/1750640983 wait complete. 2024-04-25T02:15:29.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 74500 8.84 MB/sec execute 333 sec latency 67.376 ms 2024-04-25T02:15:30.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 74659 8.83 MB/sec execute 334 sec latency 52.971 ms 2024-04-25T02:15:30.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:30 smithi040 ceph-mon[32418]: pgmap v662: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 247 op/s 2024-04-25T02:15:30.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:30 smithi134 ceph-mon[33098]: pgmap v662: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 247 op/s 2024-04-25T02:15:30.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:30 smithi031 ceph-mon[29204]: pgmap v662: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 247 op/s 2024-04-25T02:15:31.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 74801 8.81 MB/sec execute 335 sec latency 64.961 ms 2024-04-25T02:15:32.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 74957 8.80 MB/sec execute 336 sec latency 94.016 ms 2024-04-25T02:15:32.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:32 smithi040 ceph-mon[32418]: pgmap v663: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.2 MiB/s wr, 372 op/s 2024-04-25T02:15:32.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:32 smithi134 ceph-mon[33098]: pgmap v663: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.2 MiB/s wr, 372 op/s 2024-04-25T02:15:32.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:32 smithi031 ceph-mon[29204]: pgmap v663: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.2 MiB/s wr, 372 op/s 2024-04-25T02:15:33.103 INFO:tasks.workunit.client.0.smithi031.stdout: 10 75141 8.80 MB/sec execute 337 sec latency 60.866 ms 2024-04-25T02:15:34.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 75336 8.83 MB/sec execute 338 sec latency 55.239 ms 2024-04-25T02:15:34.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:34 smithi040 ceph-mon[32418]: pgmap v664: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 252 op/s 2024-04-25T02:15:34.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:34 smithi134 ceph-mon[33098]: pgmap v664: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 252 op/s 2024-04-25T02:15:34.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:34 smithi031 ceph-mon[29204]: pgmap v664: 97 pgs: 97 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.0 MiB/s wr, 252 op/s 2024-04-25T02:15:35.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 75577 8.86 MB/sec execute 339 sec latency 50.283 ms 2024-04-25T02:15:36.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 75753 8.87 MB/sec execute 340 sec latency 62.471 ms 2024-04-25T02:15:36.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:36 smithi040 ceph-mon[32418]: pgmap v665: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 377 op/s 2024-04-25T02:15:36.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:36 smithi134 ceph-mon[33098]: pgmap v665: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 377 op/s 2024-04-25T02:15:36.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:36 smithi031 ceph-mon[29204]: pgmap v665: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 377 op/s 2024-04-25T02:15:37.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 75858 8.85 MB/sec execute 341 sec latency 70.038 ms 2024-04-25T02:15:38.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 76017 8.85 MB/sec execute 342 sec latency 84.908 ms 2024-04-25T02:15:38.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:38 smithi031 ceph-mon[29204]: pgmap v666: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.4 MiB/s wr, 250 op/s 2024-04-25T02:15:38.305 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:38 smithi040 ceph-mon[32418]: pgmap v666: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.4 MiB/s wr, 250 op/s 2024-04-25T02:15:38.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:38 smithi134 ceph-mon[33098]: pgmap v666: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.4 MiB/s wr, 250 op/s 2024-04-25T02:15:39.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 76193 8.85 MB/sec execute 343 sec latency 108.244 ms 2024-04-25T02:15:40.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 76353 8.85 MB/sec execute 344 sec latency 69.125 ms 2024-04-25T02:15:40.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:40 smithi040 ceph-mon[32418]: pgmap v667: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.4 MiB/s wr, 250 op/s 2024-04-25T02:15:40.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:40 smithi134 ceph-mon[33098]: pgmap v667: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.4 MiB/s wr, 250 op/s 2024-04-25T02:15:40.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:40 smithi031 ceph-mon[29204]: pgmap v667: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.4 MiB/s wr, 250 op/s 2024-04-25T02:15:41.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 76485 8.84 MB/sec execute 345 sec latency 65.231 ms 2024-04-25T02:15:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.976+0000 7f3a3915a640 1 -- 172.21.15.31:0/1724109760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a34104320 msgr2=0x7f3a34110830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.976+0000 7f3a3915a640 1 --2- 172.21.15.31:0/1724109760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a34104320 0x7f3a34110830 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009cd0 tx=0x7f3a2c02a8f0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.976+0000 7f3a3915a640 1 -- 172.21.15.31:0/1724109760 shutdown_connections 2024-04-25T02:15:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.976+0000 7f3a3915a640 1 --2- 172.21.15.31:0/1724109760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a34104320 0x7f3a34110830 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.976+0000 7f3a3915a640 1 --2- 172.21.15.31:0/1724109760 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a34103960 0x7f3a34103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.976+0000 7f3a3915a640 1 --2- 172.21.15.31:0/1724109760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a34109970 0x7f3a34109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.976+0000 7f3a3915a640 1 -- 172.21.15.31:0/1724109760 >> 172.21.15.31:0/1724109760 conn(0x7f3a340ff450 msgr2=0x7f3a34101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.976+0000 7f3a3915a640 1 -- 172.21.15.31:0/1724109760 shutdown_connections 2024-04-25T02:15:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.976+0000 7f3a3915a640 1 -- 172.21.15.31:0/1724109760 wait complete. 2024-04-25T02:15:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.977+0000 7f3a3915a640 1 Processor -- start 2024-04-25T02:15:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.977+0000 7f3a3915a640 1 -- start start 2024-04-25T02:15:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3915a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a34103960 0x7f3a34199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3915a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a34104320 0x7f3a3419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3915a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a34109970 0x7f3a3419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3915a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3a34113c80 con 0x7f3a34109970 2024-04-25T02:15:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3915a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3a34113b00 con 0x7f3a34104320 2024-04-25T02:15:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3915a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3a34113e00 con 0x7f3a34103960 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a34109970 0x7f3a3419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a32d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a34103960 0x7f3a34199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a32d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a34103960 0x7f3a34199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36206/0 (socket says 172.21.15.31:36206) 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a32d9d640 1 -- 172.21.15.31:0/498766800 learned_addr learned my addr 172.21.15.31:0/498766800 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3259c640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a34104320 0x7f3a3419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3359e640 1 -- 172.21.15.31:0/498766800 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a34103960 msgr2=0x7f3a34199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3359e640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a34103960 0x7f3a34199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3359e640 1 -- 172.21.15.31:0/498766800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a34104320 msgr2=0x7f3a3419a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3359e640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a34104320 0x7f3a3419a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3359e640 1 -- 172.21.15.31:0/498766800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3419ee30 con 0x7f3a34109970 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.978+0000 7f3a3259c640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a34104320 0x7f3a3419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.979+0000 7f3a3359e640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a34109970 0x7f3a3419e700 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009bf0 tx=0x7f3a2c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.979+0000 7f3a1bfff640 1 -- 172.21.15.31:0/498766800 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c004240 con 0x7f3a34109970 2024-04-25T02:15:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.979+0000 7f3a1bfff640 1 -- 172.21.15.31:0/498766800 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a2c040070 con 0x7f3a34109970 2024-04-25T02:15:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.979+0000 7f3a3915a640 1 -- 172.21.15.31:0/498766800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3419f0c0 con 0x7f3a34109970 2024-04-25T02:15:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.979+0000 7f3a3915a640 1 -- 172.21.15.31:0/498766800 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a341a69b0 con 0x7f3a34109970 2024-04-25T02:15:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.979+0000 7f3a1bfff640 1 -- 172.21.15.31:0/498766800 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c045c70 con 0x7f3a34109970 2024-04-25T02:15:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a3be64740 1 Processor -- start 2024-04-25T02:15:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a1bfff640 1 -- 172.21.15.31:0/498766800 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3a2c031440 con 0x7f3a34109970 2024-04-25T02:15:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a3be64740 1 -- start start 2024-04-25T02:15:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 0x557985e82620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 0x557985e8ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e8d230 0x557985e8f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a3be64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557985de6cb0 con 0x557985e82240 2024-04-25T02:15:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a3be64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557985d8f5b0 con 0x557985e82b60 2024-04-25T02:15:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a3be64740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557985de2c70 con 0x557985e8d230 2024-04-25T02:15:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a3259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 0x557985e82620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a32d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 0x557985e8ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a3259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 0x557985e82620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50646/0 (socket says 172.21.15.31:50646) 2024-04-25T02:15:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.983+0000 7f3a3259c640 1 -- 172.21.15.31:0/1524720687 learned_addr learned my addr 172.21.15.31:0/1524720687 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.984+0000 7f3a3259c640 1 -- 172.21.15.31:0/1524720687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e8d230 msgr2=0x557985e8f620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:15:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.984+0000 7f3a3259c640 1 --2- 172.21.15.31:0/1524720687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e8d230 0x557985e8f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.984+0000 7f3a3259c640 1 -- 172.21.15.31:0/1524720687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 msgr2=0x557985e8ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.984+0000 7f3a3259c640 1 --2- 172.21.15.31:0/1524720687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 0x557985e8ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.984+0000 7f3a3259c640 1 -- 172.21.15.31:0/1524720687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557985afb210 con 0x557985e82240 2024-04-25T02:15:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.984+0000 7f3a1bfff640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3a1c0a2e20 0x7f3a1c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.984+0000 7f3a32d9d640 1 --2- 172.21.15.31:0/1524720687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 0x557985e8ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:15:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.984+0000 7f3a3259c640 1 --2- 172.21.15.31:0/1524720687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 0x557985e82620 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f3a28011540 tx=0x7f3a28011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0014b5209ed7d6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.985+0000 7f3a1bfff640 1 -- 172.21.15.31:0/498766800 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7f3a2c0f03a0 con 0x7f3a34109970 2024-04-25T02:15:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.985+0000 7f3a32d9d640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3a1c0a2e20 0x7f3a1c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.985+0000 7f3a197fa640 1 -- 172.21.15.31:0/1524720687 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2801e070 con 0x557985e82240 2024-04-25T02:15:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.985+0000 7f3a197fa640 1 -- 172.21.15.31:0/1524720687 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a2800fa70 con 0x557985e82240 2024-04-25T02:15:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.985+0000 7f3a197fa640 1 -- 172.21.15.31:0/1524720687 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28019b40 con 0x557985e82240 2024-04-25T02:15:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.986+0000 7f3a3be64740 1 -- 172.21.15.31:0/1524720687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 msgr2=0x557985e82620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.986+0000 7f3a3be64740 1 --2- 172.21.15.31:0/1524720687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 0x557985e82620 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f3a28011540 tx=0x7f3a28011a00 comp rx=0 tx=0).stop 2024-04-25T02:15:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.986+0000 7f3a32d9d640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3a1c0a2e20 0x7f3a1c0a52e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009620 tx=0x7f3a24005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.986+0000 7f3a3be64740 1 -- 172.21.15.31:0/1524720687 shutdown_connections 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.986+0000 7f3a3be64740 1 --2- 172.21.15.31:0/1524720687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e8d230 0x557985e8f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.986+0000 7f3a3be64740 1 --2- 172.21.15.31:0/1524720687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 0x557985e8ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.986+0000 7f3a3be64740 1 --2- 172.21.15.31:0/1524720687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 0x557985e82620 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.986+0000 7f3a3be64740 1 -- 172.21.15.31:0/1524720687 >> 172.21.15.31:0/1524720687 conn(0x557985e7e560 msgr2=0x557985e7e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.986+0000 7f3a3be64740 1 -- 172.21.15.31:0/1524720687 shutdown_connections 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.986+0000 7f3a3be64740 1 -- 172.21.15.31:0/1524720687 wait complete. 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3be64740 1 -- start start 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3be64740 1 Processor -- start 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82240 0x557985e8a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82b60 0x557985e8a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e8d230 0x557985e8ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3be64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557985e90020 con 0x557985e8d230 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3be64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557985da2b50 con 0x557985e82240 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3be64740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557985e901a0 con 0x557985e82b60 2024-04-25T02:15:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3be64740 10 client.0 _ll_register_callbacks cb 0x557985dddca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a32d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82240 0x557985e8a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82b60 0x557985e8a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e8d230 0x557985e8ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a32d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82240 0x557985e8a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51622/0 (socket says 172.21.15.31:51622) 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a32d9d640 1 -- 172.21.15.31:0/3631934826 learned_addr learned my addr 172.21.15.31:0/3631934826 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.987+0000 7f3a3259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82b60 0x557985e8a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36238/0 (socket says 172.21.15.31:36238) 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a3359e640 1 -- 172.21.15.31:0/3631934826 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82b60 msgr2=0x557985e8a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a3359e640 1 --2- 172.21.15.31:0/3631934826 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82b60 0x557985e8a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a3359e640 1 -- 172.21.15.31:0/3631934826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82240 msgr2=0x557985e8a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a3359e640 1 --2- 172.21.15.31:0/3631934826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82240 0x557985e8a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a3359e640 1 -- 172.21.15.31:0/3631934826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557985bc7620 con 0x557985e8d230 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a3359e640 1 --2- 172.21.15.31:0/3631934826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e8d230 0x557985e8ac30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c0edc20 tx=0x7f3a2c0bc350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c38ca93d8530a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a16ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a16ff5640 1 -- 172.21.15.31:0/3631934826 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c0bc9f0 con 0x557985e8d230 2024-04-25T02:15:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a16ff5640 1 -- 172.21.15.31:0/3631934826 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7f3a2c0bcc50 con 0x557985e8d230 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a3be64740 1 -- 172.21.15.31:0/3631934826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a2800fa70 con 0x557985e8d230 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a16ff5640 1 -- 172.21.15.31:0/3631934826 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f3a2c0c0da0 con 0x557985e8d230 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a3be64740 10 client.17832 fetch_fsmap learned FSMap version 50 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.988+0000 7f3a3be64740 1 -- 172.21.15.31:0/3631934826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x557985ea4a40 con 0x557985e8d230 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.989+0000 7f3a16ff5640 1 -- 172.21.15.31:0/3631934826 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a2800fa70 con 0x557985e8d230 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.989+0000 7f3a16ff5640 1 -- 172.21.15.31:0/3631934826 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f3a2c04d030 con 0x557985e8d230 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.989+0000 7f3a3be64740 10 client.17832 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.989+0000 7f3a3be64740 10 client.17832 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.989+0000 7f3a3be64740 20 client.17832 populate_metadata read hostname 'smithi031' 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.989+0000 7f3a3be64740 1 --2- 172.21.15.31:0/3631934826 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557985ea5e30 0x557985ea8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.989+0000 7f3a3be64740 4 client.17832 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.989+0000 7f3a3be64740 1 -- 172.21.15.31:0/3631934826 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557985df41f0 con 0x557985ea5e30 2024-04-25T02:15:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.989+0000 7f3a32d9d640 1 --2- 172.21.15.31:0/3631934826 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557985ea5e30 0x557985ea8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.990+0000 7f3a32d9d640 1 --2- 172.21.15.31:0/3631934826 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557985ea5e30 0x557985ea8260 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fe371ba9af1f36b server_cookie=acd380822fe970b in_seq=0 out_seq=0 2024-04-25T02:15:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.990+0000 7f3a16ff5640 10 client.17832 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a16ff5640 1 -- 172.21.15.31:0/3631934826 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557985df41f0 con 0x557985ea5e30 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a16ff5640 10 client.17832 handle_command_reply: tid=0 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 1 client.17832 shutdown 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 1 -- 172.21.15.31:0/3631934826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e8d230 msgr2=0x557985e8ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 1 --2- 172.21.15.31:0/3631934826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e8d230 0x557985e8ac30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c0edc20 tx=0x7f3a2c0bc350 comp rx=0 tx=0).stop 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 1 -- 172.21.15.31:0/3631934826 shutdown_connections 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 1 --2- 172.21.15.31:0/3631934826 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557985ea5e30 0x557985ea8260 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 1 --2- 172.21.15.31:0/3631934826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e8d230 0x557985e8ac30 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 1 --2- 172.21.15.31:0/3631934826 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82b60 0x557985e8a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 1 --2- 172.21.15.31:0/3631934826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82240 0x557985e8a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a16ff5640 0 client.17832 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 1 -- 172.21.15.31:0/3631934826 >> 172.21.15.31:0/3631934826 conn(0x557985e7e560 msgr2=0x557985e84260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 1 -- 172.21.15.31:0/3631934826 shutdown_connections 2024-04-25T02:15:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 1 -- 172.21.15.31:0/3631934826 wait complete. 2024-04-25T02:15:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:41.991+0000 7f3a3be64740 20 client.17832 trim_cache size 0 max 16384 2024-04-25T02:15:42.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.016+0000 7f3a3be64740 1 Processor -- start 2024-04-25T02:15:42.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.016+0000 7f3a3be64740 1 -- start start 2024-04-25T02:15:42.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.016+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82240 0x557985e849a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:42.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82b60 0x557985e7d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:42.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e8d230 0x557985eadb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:42.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a3be64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3a341aacf0 con 0x557985e82b60 2024-04-25T02:15:42.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a3be64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3a341ac830 con 0x557985e8d230 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a3be64740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3a341ab3c0 con 0x557985e82240 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a32d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82240 0x557985e849a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a32d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82240 0x557985e849a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36252/0 (socket says 172.21.15.31:36252) 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a32d9d640 1 -- 172.21.15.31:0/3107257680 learned_addr learned my addr 172.21.15.31:0/3107257680 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a3259c640 1 --2- 172.21.15.31:0/3107257680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82b60 0x557985e7d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a3359e640 1 --2- 172.21.15.31:0/3107257680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e8d230 0x557985eadb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a32d9d640 1 -- 172.21.15.31:0/3107257680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e8d230 msgr2=0x557985eadb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a32d9d640 1 --2- 172.21.15.31:0/3107257680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e8d230 0x557985eadb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a32d9d640 1 -- 172.21.15.31:0/3107257680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82b60 msgr2=0x557985e7d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a32d9d640 1 --2- 172.21.15.31:0/3107257680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82b60 0x557985e7d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.017+0000 7f3a32d9d640 1 -- 172.21.15.31:0/3107257680 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557985ea88e0 con 0x557985e82240 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.018+0000 7f3a32d9d640 1 --2- 172.21.15.31:0/3107257680 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82240 0x557985e849a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3a24004330 tx=0x7f3a2401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbeafb8130a317f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.018+0000 7f3a18ff9640 1 -- 172.21.15.31:0/3107257680 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2402b420 con 0x557985e82240 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.018+0000 7f3a18ff9640 1 -- 172.21.15.31:0/3107257680 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a24005020 con 0x557985e82240 2024-04-25T02:15:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.018+0000 7f3a18ff9640 1 -- 172.21.15.31:0/3107257680 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24004920 con 0x557985e82240 2024-04-25T02:15:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.019+0000 7f3a3be64740 1 -- 172.21.15.31:0/3107257680 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82240 msgr2=0x557985e849a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.019+0000 7f3a3be64740 1 --2- 172.21.15.31:0/3107257680 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82240 0x557985e849a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3a24004330 tx=0x7f3a2401b000 comp rx=0 tx=0).stop 2024-04-25T02:15:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.019+0000 7f3a3be64740 1 -- 172.21.15.31:0/3107257680 shutdown_connections 2024-04-25T02:15:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.019+0000 7f3a3be64740 1 --2- 172.21.15.31:0/3107257680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e8d230 0x557985eadb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.019+0000 7f3a3be64740 1 --2- 172.21.15.31:0/3107257680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82b60 0x557985e7d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.019+0000 7f3a3be64740 1 --2- 172.21.15.31:0/3107257680 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e82240 0x557985e849a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.019+0000 7f3a3be64740 1 -- 172.21.15.31:0/3107257680 >> 172.21.15.31:0/3107257680 conn(0x557985e7e560 msgr2=0x557985e7cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.019+0000 7f3a3be64740 1 -- 172.21.15.31:0/3107257680 shutdown_connections 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.019+0000 7f3a3be64740 1 -- 172.21.15.31:0/3107257680 wait complete. 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.019+0000 7f3a3be64740 1 -- start start 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.019+0000 7f3a3be64740 1 Processor -- start 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 0x557985e872b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 0x557985e877f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3be64740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e8d230 0x557985f0b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3be64740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3a341ac160 con 0x557985e82240 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3be64740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3a341aba90 con 0x557985e82b60 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3be64740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3a341a8d30 con 0x557985e8d230 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3be64740 10 client.0 _ll_register_callbacks cb 0x7f3a2c03ffc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a32d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 0x557985e872b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 0x557985e877f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 0x557985e877f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:51640/0 (socket says 172.21.15.31:51640) 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3259c640 1 -- 172.21.15.31:0/1942920180 learned_addr learned my addr 172.21.15.31:0/1942920180 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3359e640 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e8d230 0x557985f0b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a32d9d640 1 -- 172.21.15.31:0/1942920180 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e8d230 msgr2=0x557985f0b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a32d9d640 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e8d230 0x557985f0b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a32d9d640 1 -- 172.21.15.31:0/1942920180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 msgr2=0x557985e877f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a32d9d640 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 0x557985e877f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a32d9d640 1 -- 172.21.15.31:0/1942920180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557985f0b920 con 0x557985e82240 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.020+0000 7f3a3259c640 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 0x557985e877f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.021+0000 7f3a32d9d640 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 0x557985e872b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3a240041f0 tx=0x7f3a24005320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e38eef3d6bc11a2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.021+0000 7f3a16ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.021+0000 7f3a16ff5640 1 -- 172.21.15.31:0/1942920180 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24038030 con 0x557985e82240 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.021+0000 7f3a16ff5640 1 -- 172.21.15.31:0/1942920180 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7f3a24030050 con 0x557985e82240 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.021+0000 7f3a3be64740 1 -- 172.21.15.31:0/1942920180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a24005020 con 0x557985e82240 2024-04-25T02:15:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.021+0000 7f3a16ff5640 1 -- 172.21.15.31:0/1942920180 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f3a240054d0 con 0x557985e82240 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.021+0000 7f3a3be64740 10 client.17838 fetch_fsmap learned FSMap version 50 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.021+0000 7f3a3be64740 1 -- 172.21.15.31:0/1942920180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x557985f0cad0 con 0x557985e82240 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.022+0000 7f3a16ff5640 1 -- 172.21.15.31:0/1942920180 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a24005020 con 0x557985e82240 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.022+0000 7f3a16ff5640 1 -- 172.21.15.31:0/1942920180 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f3a24044020 con 0x557985e82240 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.022+0000 7f3a3be64740 10 client.17838 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.022+0000 7f3a3be64740 10 client.17838 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.022+0000 7f3a3be64740 20 client.17838 populate_metadata read hostname 'smithi031' 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.022+0000 7f3a3be64740 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557985f0d130 0x557985f0f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.022+0000 7f3a3be64740 4 client.17838 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.022+0000 7f3a3be64740 1 -- 172.21.15.31:0/1942920180 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557985dee540 con 0x557985f0d130 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.022+0000 7f3a3259c640 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557985f0d130 0x557985f0f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.023+0000 7f3a3259c640 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557985f0d130 0x557985f0f550 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2f4cdb260083fcb server_cookie=fd71f36a9283b27 in_seq=0 out_seq=0 2024-04-25T02:15:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.023+0000 7f3a16ff5640 10 client.17838 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:42.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 76631 8.84 MB/sec execute 346 sec latency 75.355 ms 2024-04-25T02:15:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.113+0000 7f3a16ff5640 1 -- 172.21.15.31:0/1942920180 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557985dee540 con 0x557985f0d130 2024-04-25T02:15:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.113+0000 7f3a16ff5640 10 client.17838 handle_command_reply: tid=0 2024-04-25T02:15:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.113+0000 7f3a3be64740 1 client.17838 shutdown 2024-04-25T02:15:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a3be64740 1 -- 172.21.15.31:0/1942920180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 msgr2=0x557985e872b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a3be64740 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 0x557985e872b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3a240041f0 tx=0x7f3a24005320 comp rx=0 tx=0).stop 2024-04-25T02:15:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a3be64740 1 -- 172.21.15.31:0/1942920180 shutdown_connections 2024-04-25T02:15:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a3be64740 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x557985f0d130 0x557985f0f550 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a3be64740 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557985e8d230 0x557985f0b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a3be64740 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557985e82b60 0x557985e877f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a16ff5640 0 client.17838 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a3be64740 1 --2- 172.21.15.31:0/1942920180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557985e82240 0x557985e872b0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a3be64740 1 -- 172.21.15.31:0/1942920180 >> 172.21.15.31:0/1942920180 conn(0x557985e7e560 msgr2=0x557985e8aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a3be64740 1 -- 172.21.15.31:0/1942920180 shutdown_connections 2024-04-25T02:15:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a3be64740 1 -- 172.21.15.31:0/1942920180 wait complete. 2024-04-25T02:15:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.114+0000 7f3a3be64740 20 client.17838 trim_cache size 0 max 16384 2024-04-25T02:15:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:15:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:15:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:15:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:15:42.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 -- 172.21.15.31:0/498766800 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3a1c0a2e20 msgr2=0x7f3a1c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:42.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3a1c0a2e20 0x7f3a1c0a52e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009620 tx=0x7f3a24005f90 comp rx=0 tx=0).stop 2024-04-25T02:15:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 -- 172.21.15.31:0/498766800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a34109970 msgr2=0x7f3a3419e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a34109970 0x7f3a3419e700 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009bf0 tx=0x7f3a2c003ba0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 -- 172.21.15.31:0/498766800 shutdown_connections 2024-04-25T02:15:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f3a1c0a2e20 0x7f3a1c0a52e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a34109970 0x7f3a3419e700 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a34104320 0x7f3a3419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 --2- 172.21.15.31:0/498766800 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a34103960 0x7f3a34199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 -- 172.21.15.31:0/498766800 >> 172.21.15.31:0/498766800 conn(0x7f3a340ff450 msgr2=0x7f3a340ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 -- 172.21.15.31:0/498766800 shutdown_connections 2024-04-25T02:15:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:42.115+0000 7f3a3915a640 1 -- 172.21.15.31:0/498766800 wait complete. 2024-04-25T02:15:42.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:42 smithi040 ceph-mon[32418]: pgmap v668: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.4 MiB/s wr, 352 op/s 2024-04-25T02:15:42.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:42 smithi134 ceph-mon[33098]: pgmap v668: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.4 MiB/s wr, 352 op/s 2024-04-25T02:15:42.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:42 smithi031 ceph-mon[29204]: pgmap v668: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.4 MiB/s wr, 352 op/s 2024-04-25T02:15:43.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 76797 8.84 MB/sec execute 347 sec latency 76.999 ms 2024-04-25T02:15:44.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 76995 8.84 MB/sec execute 348 sec latency 99.573 ms 2024-04-25T02:15:44.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:44 smithi040 ceph-mon[32418]: pgmap v669: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 227 op/s 2024-04-25T02:15:44.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:44 smithi134 ceph-mon[33098]: pgmap v669: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 227 op/s 2024-04-25T02:15:44.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:44 smithi031 ceph-mon[29204]: pgmap v669: 97 pgs: 97 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 227 op/s 2024-04-25T02:15:45.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 77166 8.84 MB/sec execute 349 sec latency 101.111 ms 2024-04-25T02:15:46.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 77319 8.84 MB/sec execute 350 sec latency 79.544 ms 2024-04-25T02:15:46.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:46 smithi040 ceph-mon[32418]: pgmap v670: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.8 MiB/s wr, 354 op/s 2024-04-25T02:15:46.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:46 smithi134 ceph-mon[33098]: pgmap v670: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.8 MiB/s wr, 354 op/s 2024-04-25T02:15:46.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:46 smithi031 ceph-mon[29204]: pgmap v670: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.8 MiB/s wr, 354 op/s 2024-04-25T02:15:47.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 77492 8.85 MB/sec execute 351 sec latency 83.932 ms 2024-04-25T02:15:48.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 77654 8.86 MB/sec execute 352 sec latency 78.284 ms 2024-04-25T02:15:48.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:48 smithi040 ceph-mon[32418]: pgmap v671: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 229 op/s 2024-04-25T02:15:48.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:48 smithi134 ceph-mon[33098]: pgmap v671: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 229 op/s 2024-04-25T02:15:48.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:48 smithi031 ceph-mon[29204]: pgmap v671: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 229 op/s 2024-04-25T02:15:49.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 77808 8.86 MB/sec execute 353 sec latency 68.286 ms 2024-04-25T02:15:50.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 77948 8.84 MB/sec execute 354 sec latency 53.787 ms 2024-04-25T02:15:50.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:50 smithi040 ceph-mon[32418]: pgmap v672: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 229 op/s 2024-04-25T02:15:50.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:50 smithi134 ceph-mon[33098]: pgmap v672: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 229 op/s 2024-04-25T02:15:50.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:50 smithi031 ceph-mon[29204]: pgmap v672: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 229 op/s 2024-04-25T02:15:51.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 78087 8.83 MB/sec execute 355 sec latency 129.554 ms 2024-04-25T02:15:52.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 78256 8.82 MB/sec execute 356 sec latency 74.507 ms 2024-04-25T02:15:52.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:52 smithi040 ceph-mon[32418]: pgmap v673: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.3 MiB/s wr, 349 op/s 2024-04-25T02:15:52.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:52 smithi134 ceph-mon[33098]: pgmap v673: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.3 MiB/s wr, 349 op/s 2024-04-25T02:15:52.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:52 smithi031 ceph-mon[29204]: pgmap v673: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.3 MiB/s wr, 349 op/s 2024-04-25T02:15:53.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 78429 8.81 MB/sec execute 357 sec latency 62.123 ms 2024-04-25T02:15:54.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 78577 8.80 MB/sec execute 358 sec latency 55.804 ms 2024-04-25T02:15:54.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:54 smithi040 ceph-mon[32418]: pgmap v674: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 247 op/s 2024-04-25T02:15:54.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:54 smithi134 ceph-mon[33098]: pgmap v674: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 247 op/s 2024-04-25T02:15:54.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:54 smithi031 ceph-mon[29204]: pgmap v674: 97 pgs: 97 active+clean; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.3 MiB/s wr, 247 op/s 2024-04-25T02:15:55.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 78763 8.81 MB/sec execute 359 sec latency 58.086 ms 2024-04-25T02:15:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.242+0000 7f90eb909640 1 -- 172.21.15.31:0/1902308541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90e4106630 msgr2=0x7f90e4106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.242+0000 7f90eb909640 1 --2- 172.21.15.31:0/1902308541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90e4106630 0x7f90e4106a10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009860 tx=0x7f90e002a340 comp rx=0 tx=0).stop 2024-04-25T02:15:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.242+0000 7f90eb909640 1 -- 172.21.15.31:0/1902308541 shutdown_connections 2024-04-25T02:15:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.242+0000 7f90eb909640 1 --2- 172.21.15.31:0/1902308541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e4101030 0x7f90e410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.242+0000 7f90eb909640 1 --2- 172.21.15.31:0/1902308541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90e4100670 0x7f90e4100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.242+0000 7f90eb909640 1 --2- 172.21.15.31:0/1902308541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90e4106630 0x7f90e4106a10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.242+0000 7f90eb909640 1 -- 172.21.15.31:0/1902308541 >> 172.21.15.31:0/1902308541 conn(0x7f90e40777c0 msgr2=0x7f90e4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.243+0000 7f90eb909640 1 -- 172.21.15.31:0/1902308541 shutdown_connections 2024-04-25T02:15:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.243+0000 7f90eb909640 1 -- 172.21.15.31:0/1902308541 wait complete. 2024-04-25T02:15:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.243+0000 7f90eb909640 1 Processor -- start 2024-04-25T02:15:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.243+0000 7f90eb909640 1 -- start start 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90eb909640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e4100670 0x7f90e40fd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90eb909640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90e4101030 0x7f90e40fdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90eb909640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90e4106630 0x7f90e40fe080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90eb909640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f90e41108b0 con 0x7f90e4100670 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90eb909640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90e4110730 con 0x7f90e4106630 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90eb909640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f90e4110a30 con 0x7f90e4101030 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90e96a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e4100670 0x7f90e40fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90e9ea6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90e4106630 0x7f90e40fe080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90e96a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e4100670 0x7f90e40fd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49994/0 (socket says 172.21.15.31:49994) 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90e96a5640 1 -- 172.21.15.31:0/3179597263 learned_addr learned my addr 172.21.15.31:0/3179597263 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90e4101030 0x7f90e40fdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90e9ea6640 1 -- 172.21.15.31:0/3179597263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90e4101030 msgr2=0x7f90e40fdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90e9ea6640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90e4101030 0x7f90e40fdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90e9ea6640 1 -- 172.21.15.31:0/3179597263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e4100670 msgr2=0x7f90e40fd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90e9ea6640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e4100670 0x7f90e40fd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90e9ea6640 1 -- 172.21.15.31:0/3179597263 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e41a6420 con 0x7f90e4106630 2024-04-25T02:15:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.244+0000 7f90e9ea6640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90e4106630 0x7f90e40fe080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00efc0 tx=0x7f90dc00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.245+0000 7f90da7fc640 1 -- 172.21.15.31:0/3179597263 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc003970 con 0x7f90e4106630 2024-04-25T02:15:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.245+0000 7f90da7fc640 1 -- 172.21.15.31:0/3179597263 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90dc004020 con 0x7f90e4106630 2024-04-25T02:15:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.245+0000 7f90da7fc640 1 -- 172.21.15.31:0/3179597263 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc00b940 con 0x7f90e4106630 2024-04-25T02:15:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.245+0000 7f90eb909640 1 -- 172.21.15.31:0/3179597263 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e41a6710 con 0x7f90e4106630 2024-04-25T02:15:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.245+0000 7f90eb909640 1 -- 172.21.15.31:0/3179597263 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e41a6c10 con 0x7f90e4106630 2024-04-25T02:15:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.247+0000 7f90da7fc640 1 -- 172.21.15.31:0/3179597263 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f90dc01d020 con 0x7f90e4106630 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90ee612740 1 Processor -- start 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90ee612740 1 -- start start 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90ee612740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 0x556e81065620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90ee612740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 0x556e8106fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90ee612740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81070230 0x556e81072620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90ee612740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556e80fc9cb0 con 0x556e81065b60 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90ee612740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556e80f725b0 con 0x556e81065240 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90ee612740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556e80fc5c70 con 0x556e81070230 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90da7fc640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f90d00a2e50 0x7f90d00a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90e8ea4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 0x556e81065620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90da7fc640 1 -- 172.21.15.31:0/3179597263 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7f90dc0cf1f0 con 0x7f90e4106630 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90e8ea4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 0x556e81065620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:40698/0 (socket says 172.21.15.31:40698) 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90e8ea4640 1 -- 172.21.15.31:0/4125772257 learned_addr learned my addr 172.21.15.31:0/4125772257 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.248+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/4125772257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81070230 0x556e81072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.249+0000 7f90e9ea6640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f90d00a2e50 0x7f90d00a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.249+0000 7f90e8ea4640 1 -- 172.21.15.31:0/4125772257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81070230 msgr2=0x556e81072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:55.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.249+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/4125772257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81070230 0x556e81072620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.249+0000 7f90e8ea4640 1 -- 172.21.15.31:0/4125772257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 msgr2=0x556e8106fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:55.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.249+0000 7f90e96a5640 1 --2- 172.21.15.31:0/4125772257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 0x556e8106fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.249+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/4125772257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 0x556e8106fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.249+0000 7f90e9ea6640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f90d00a2e50 0x7f90d00a5310 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f90dc009830 tx=0x7f90dc0667a0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.249+0000 7f90e8ea4640 1 -- 172.21.15.31:0/4125772257 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e80cde210 con 0x556e81065240 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.249+0000 7f90e96a5640 1 --2- 172.21.15.31:0/4125772257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 0x556e8106fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.249+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/4125772257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 0x556e81065620 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f90d40137e0 tx=0x7f90d4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8d0ce2085d9a5c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.250+0000 7f90cf7fe640 1 -- 172.21.15.31:0/4125772257 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d4013d60 con 0x556e81065240 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.250+0000 7f90cf7fe640 1 -- 172.21.15.31:0/4125772257 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90d4005020 con 0x556e81065240 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.250+0000 7f90cf7fe640 1 -- 172.21.15.31:0/4125772257 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d40049e0 con 0x556e81065240 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.251+0000 7f90ee612740 1 -- 172.21.15.31:0/4125772257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 msgr2=0x556e81065620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.251+0000 7f90ee612740 1 --2- 172.21.15.31:0/4125772257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 0x556e81065620 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f90d40137e0 tx=0x7f90d4005850 comp rx=0 tx=0).stop 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.251+0000 7f90ee612740 1 -- 172.21.15.31:0/4125772257 shutdown_connections 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.251+0000 7f90ee612740 1 --2- 172.21.15.31:0/4125772257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81070230 0x556e81072620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.251+0000 7f90ee612740 1 --2- 172.21.15.31:0/4125772257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 0x556e8106fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.251+0000 7f90ee612740 1 --2- 172.21.15.31:0/4125772257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 0x556e81065620 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.251+0000 7f90ee612740 1 -- 172.21.15.31:0/4125772257 >> 172.21.15.31:0/4125772257 conn(0x556e81061560 msgr2=0x556e81061940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.251+0000 7f90ee612740 1 -- 172.21.15.31:0/4125772257 shutdown_connections 2024-04-25T02:15:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.251+0000 7f90ee612740 1 -- 172.21.15.31:0/4125772257 wait complete. 2024-04-25T02:15:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.251+0000 7f90ee612740 1 -- start start 2024-04-25T02:15:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.251+0000 7f90ee612740 1 Processor -- start 2024-04-25T02:15:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90ee612740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065240 0x556e8106d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90ee612740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 0x556e8106d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90ee612740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81070230 0x556e8106dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90ee612740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556e81073020 con 0x556e81065b60 2024-04-25T02:15:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90ee612740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556e80f85b50 con 0x556e81070230 2024-04-25T02:15:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90ee612740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556e810731a0 con 0x556e81065240 2024-04-25T02:15:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90ee612740 10 client.0 _ll_register_callbacks cb 0x556e80fc0ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:15:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90e96a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 0x556e8106d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90e8ea4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065240 0x556e8106d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90e8ea4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065240 0x556e8106d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:56180/0 (socket says 172.21.15.31:56180) 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90e8ea4640 1 -- 172.21.15.31:0/2339955977 learned_addr learned my addr 172.21.15.31:0/2339955977 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81070230 0x556e8106dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90e96a5640 1 -- 172.21.15.31:0/2339955977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065240 msgr2=0x556e8106d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90e96a5640 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065240 0x556e8106d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90e96a5640 1 -- 172.21.15.31:0/2339955977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81070230 msgr2=0x556e8106dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90e96a5640 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81070230 0x556e8106dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90e96a5640 1 -- 172.21.15.31:0/2339955977 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e80daa620 con 0x556e81065b60 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.252+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065240 0x556e8106d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.253+0000 7f90e96a5640 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 0x556e8106d6b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f90e00057c0 tx=0x7f90e00306b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0a163184499379 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.253+0000 7f90ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.253+0000 7f90ccff9640 1 -- 172.21.15.31:0/2339955977 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e0037040 con 0x556e81065b60 2024-04-25T02:15:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.253+0000 7f90ee612740 1 -- 172.21.15.31:0/2339955977 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90d4005020 con 0x556e81065b60 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.253+0000 7f90ccff9640 1 -- 172.21.15.31:0/2339955977 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7f90e0004250 con 0x556e81065b60 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.253+0000 7f90ccff9640 1 -- 172.21.15.31:0/2339955977 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f90e0004450 con 0x556e81065b60 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.253+0000 7f90ee612740 10 client.17847 fetch_fsmap learned FSMap version 50 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.253+0000 7f90ee612740 1 -- 172.21.15.31:0/2339955977 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x556e8107f150 con 0x556e81065b60 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.253+0000 7f90ccff9640 1 -- 172.21.15.31:0/2339955977 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90d4005020 con 0x556e81065b60 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.254+0000 7f90ccff9640 1 -- 172.21.15.31:0/2339955977 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f90e0032540 con 0x556e81065b60 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.254+0000 7f90ee612740 10 client.17847 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.254+0000 7f90ee612740 10 client.17847 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.254+0000 7f90ee612740 20 client.17847 populate_metadata read hostname 'smithi031' 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.254+0000 7f90ee612740 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556e810806b0 0x556e81082aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.254+0000 7f90ee612740 4 client.17847 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.254+0000 7f90ee612740 1 -- 172.21.15.31:0/2339955977 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e80fd71f0 con 0x556e810806b0 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.254+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556e810806b0 0x556e81082aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.254+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556e810806b0 0x556e81082aa0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f797155be1c33fc server_cookie=f29b7bf3b16339e0 in_seq=0 out_seq=0 2024-04-25T02:15:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.254+0000 7f90ccff9640 10 client.17847 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.255+0000 7f90ccff9640 1 -- 172.21.15.31:0/2339955977 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e80fd71f0 con 0x556e810806b0 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.255+0000 7f90ccff9640 10 client.17847 handle_command_reply: tid=0 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.255+0000 7f90ee612740 1 client.17847 shutdown 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.255+0000 7f90ee612740 1 -- 172.21.15.31:0/2339955977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 msgr2=0x556e8106d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.255+0000 7f90ee612740 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 0x556e8106d6b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f90e00057c0 tx=0x7f90e00306b0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.256+0000 7f90ee612740 1 -- 172.21.15.31:0/2339955977 shutdown_connections 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.256+0000 7f90ee612740 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556e810806b0 0x556e81082aa0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.256+0000 7f90ee612740 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81070230 0x556e8106dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.256+0000 7f90ee612740 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065b60 0x556e8106d6b0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.256+0000 7f90ccff9640 0 client.17847 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.256+0000 7f90ee612740 1 --2- 172.21.15.31:0/2339955977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065240 0x556e8106d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.256+0000 7f90ee612740 1 -- 172.21.15.31:0/2339955977 >> 172.21.15.31:0/2339955977 conn(0x556e81061560 msgr2=0x556e81067240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.256+0000 7f90ee612740 1 -- 172.21.15.31:0/2339955977 shutdown_connections 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.256+0000 7f90ee612740 1 -- 172.21.15.31:0/2339955977 wait complete. 2024-04-25T02:15:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.256+0000 7f90ee612740 20 client.17847 trim_cache size 0 max 16384 2024-04-25T02:15:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.279+0000 7f90ee612740 1 Processor -- start 2024-04-25T02:15:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90ee612740 1 -- start start 2024-04-25T02:15:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90ee612740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 0x556e8107bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90ee612740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065b60 0x556e8106c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90ee612740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81070230 0x556e8106da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90ee612740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f90e41af420 con 0x556e81070230 2024-04-25T02:15:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90ee612740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90e41b0f60 con 0x556e81065240 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90ee612740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f90e41afaf0 con 0x556e81065b60 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90e8ea4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81070230 0x556e8106da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90e8ea4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81070230 0x556e8106da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50020/0 (socket says 172.21.15.31:50020) 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90e8ea4640 1 -- 172.21.15.31:0/2887945147 learned_addr learned my addr 172.21.15.31:0/2887945147 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/2887945147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 0x556e8107bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90e96a5640 1 --2- 172.21.15.31:0/2887945147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065b60 0x556e8106c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90e8ea4640 1 -- 172.21.15.31:0/2887945147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065b60 msgr2=0x556e8106c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/2887945147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065b60 0x556e8106c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90e8ea4640 1 -- 172.21.15.31:0/2887945147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 msgr2=0x556e8107bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/2887945147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 0x556e8107bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.280+0000 7f90e8ea4640 1 -- 172.21.15.31:0/2887945147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e81072990 con 0x556e81070230 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.281+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/2887945147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81070230 0x556e8106da30 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f90d40058c0 tx=0x7f90d4015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de12c7fdcdeca7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.281+0000 7f90ceffd640 1 -- 172.21.15.31:0/2887945147 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d40055f0 con 0x556e81070230 2024-04-25T02:15:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.281+0000 7f90ceffd640 1 -- 172.21.15.31:0/2887945147 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90d4032450 con 0x556e81070230 2024-04-25T02:15:55.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.281+0000 7f90ceffd640 1 -- 172.21.15.31:0/2887945147 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d40036e0 con 0x556e81070230 2024-04-25T02:15:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.282+0000 7f90ee612740 1 -- 172.21.15.31:0/2887945147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81070230 msgr2=0x556e8106da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.282+0000 7f90ee612740 1 --2- 172.21.15.31:0/2887945147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81070230 0x556e8106da30 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f90d40058c0 tx=0x7f90d4015040 comp rx=0 tx=0).stop 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.282+0000 7f90ee612740 1 -- 172.21.15.31:0/2887945147 shutdown_connections 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.282+0000 7f90ee612740 1 --2- 172.21.15.31:0/2887945147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81070230 0x556e8106da30 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.282+0000 7f90ee612740 1 --2- 172.21.15.31:0/2887945147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81065b60 0x556e8106c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.282+0000 7f90ee612740 1 --2- 172.21.15.31:0/2887945147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065240 0x556e8107bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.282+0000 7f90ee612740 1 -- 172.21.15.31:0/2887945147 >> 172.21.15.31:0/2887945147 conn(0x556e81061560 msgr2=0x556e8105fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.282+0000 7f90ee612740 1 -- 172.21.15.31:0/2887945147 shutdown_connections 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.282+0000 7f90ee612740 1 -- 172.21.15.31:0/2887945147 wait complete. 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90ee612740 1 -- start start 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90ee612740 1 Processor -- start 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90ee612740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065240 0x556e8106a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90ee612740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065b60 0x556e8106ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90ee612740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81070230 0x556e81069440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90ee612740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f90e41b0890 con 0x556e81065240 2024-04-25T02:15:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90ee612740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90e41b01c0 con 0x556e81065b60 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90ee612740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f90e41ad420 con 0x556e81070230 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90ee612740 10 client.0 _ll_register_callbacks cb 0x7f90dc009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90e8ea4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065240 0x556e8106a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90e96a5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065b60 0x556e8106ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90e8ea4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065240 0x556e8106a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50030/0 (socket says 172.21.15.31:50030) 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90e8ea4640 1 -- 172.21.15.31:0/3655109704 learned_addr learned my addr 172.21.15.31:0/3655109704 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.283+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81070230 0x556e81069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90e96a5640 1 -- 172.21.15.31:0/3655109704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81070230 msgr2=0x556e81069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90e96a5640 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81070230 0x556e81069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90e96a5640 1 -- 172.21.15.31:0/3655109704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065240 msgr2=0x556e8106a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90e96a5640 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065240 0x556e8106a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90e96a5640 1 -- 172.21.15.31:0/3655109704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e81078ca0 con 0x556e81065b60 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90e96a5640 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065b60 0x556e8106ad70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f90e0032a80 tx=0x7f90e0030950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f45e5148e6fc1de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:15:55.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90ccff9640 1 -- 172.21.15.31:0/3655109704 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e0037040 con 0x556e81065b60 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90ccff9640 1 -- 172.21.15.31:0/3655109704 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8484+0+0 (secure 0 0 0) 0x7f90e0032650 con 0x556e81065b60 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90ee612740 1 -- 172.21.15.31:0/3655109704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90d4032450 con 0x556e81065b60 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90ccff9640 1 -- 172.21.15.31:0/3655109704 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f90e0000e60 con 0x556e81065b60 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90ee612740 10 client.27419 fetch_fsmap learned FSMap version 50 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.284+0000 7f90ee612740 1 -- 172.21.15.31:0/3655109704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x556e810e6160 con 0x556e81065b60 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.285+0000 7f90ccff9640 1 -- 172.21.15.31:0/3655109704 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90d4032450 con 0x556e81065b60 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.285+0000 7f90ccff9640 1 -- 172.21.15.31:0/3655109704 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f90e005b070 con 0x556e81065b60 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.285+0000 7f90ee612740 10 client.27419 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.285+0000 7f90ee612740 10 client.27419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.285+0000 7f90ee612740 20 client.27419 populate_metadata read hostname 'smithi031' 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.285+0000 7f90ee612740 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556e810e68d0 0x556e810e8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.285+0000 7f90ee612740 4 client.27419 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.285+0000 7f90ee612740 1 -- 172.21.15.31:0/3655109704 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556e80fd1540 con 0x556e810e68d0 2024-04-25T02:15:55.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.285+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556e810e68d0 0x556e810e8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:15:55.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.286+0000 7f90ccff9640 10 client.27419 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:55.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.286+0000 7f90e8ea4640 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556e810e68d0 0x556e810e8cf0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e77977da1174f296 server_cookie=d36720ede1acb822 in_seq=0 out_seq=0 2024-04-25T02:15:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ccff9640 1 -- 172.21.15.31:0/3655109704 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556e80fd1540 con 0x556e810e68d0 2024-04-25T02:15:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ccff9640 10 client.27419 handle_command_reply: tid=0 2024-04-25T02:15:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 1 client.27419 shutdown 2024-04-25T02:15:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 1 -- 172.21.15.31:0/3655109704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065b60 msgr2=0x556e8106ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065b60 0x556e8106ad70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f90e0032a80 tx=0x7f90e0030950 comp rx=0 tx=0).stop 2024-04-25T02:15:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 1 -- 172.21.15.31:0/3655109704 shutdown_connections 2024-04-25T02:15:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556e810e68d0 0x556e810e8cf0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556e81070230 0x556e81069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ccff9640 0 client.27419 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:15:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e81065b60 0x556e8106ad70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 1 --2- 172.21.15.31:0/3655109704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556e81065240 0x556e8106a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 1 -- 172.21.15.31:0/3655109704 >> 172.21.15.31:0/3655109704 conn(0x556e81061560 msgr2=0x556e810605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 1 -- 172.21.15.31:0/3655109704 shutdown_connections 2024-04-25T02:15:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 1 -- 172.21.15.31:0/3655109704 wait complete. 2024-04-25T02:15:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.360+0000 7f90ee612740 20 client.27419 trim_cache size 0 max 16384 2024-04-25T02:15:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:15:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:15:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:15:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:15:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 -- 172.21.15.31:0/3179597263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f90d00a2e50 msgr2=0x7f90d00a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f90d00a2e50 0x7f90d00a5310 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f90dc009830 tx=0x7f90dc0667a0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 -- 172.21.15.31:0/3179597263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90e4106630 msgr2=0x7f90e40fe080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:15:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90e4106630 0x7f90e40fe080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00efc0 tx=0x7f90dc00c800 comp rx=0 tx=0).stop 2024-04-25T02:15:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 -- 172.21.15.31:0/3179597263 shutdown_connections 2024-04-25T02:15:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f90d00a2e50 0x7f90d00a5310 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90e4106630 0x7f90e40fe080 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90e4101030 0x7f90e40fdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 --2- 172.21.15.31:0/3179597263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e4100670 0x7f90e40fd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:15:55.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 -- 172.21.15.31:0/3179597263 >> 172.21.15.31:0/3179597263 conn(0x7f90e40777c0 msgr2=0x7f90e4069270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:15:55.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 -- 172.21.15.31:0/3179597263 shutdown_connections 2024-04-25T02:15:55.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:15:55.361+0000 7f90eb909640 1 -- 172.21.15.31:0/3179597263 wait complete. 2024-04-25T02:15:56.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 78985 8.84 MB/sec execute 360 sec latency 92.291 ms 2024-04-25T02:15:56.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:56 smithi040 ceph-mon[32418]: pgmap v675: 97 pgs: 97 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.5 MiB/s wr, 353 op/s 2024-04-25T02:15:56.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:56 smithi134 ceph-mon[33098]: pgmap v675: 97 pgs: 97 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.5 MiB/s wr, 353 op/s 2024-04-25T02:15:56.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:56 smithi031 ceph-mon[29204]: pgmap v675: 97 pgs: 97 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.5 MiB/s wr, 353 op/s 2024-04-25T02:15:57.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 79164 8.85 MB/sec execute 361 sec latency 63.265 ms 2024-04-25T02:15:58.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 79293 8.84 MB/sec execute 362 sec latency 64.578 ms 2024-04-25T02:15:58.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:15:58 smithi040 ceph-mon[32418]: pgmap v676: 97 pgs: 97 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 226 op/s 2024-04-25T02:15:58.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:15:58 smithi134 ceph-mon[33098]: pgmap v676: 97 pgs: 97 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 226 op/s 2024-04-25T02:15:58.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:15:58 smithi031 ceph-mon[29204]: pgmap v676: 97 pgs: 97 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 226 op/s 2024-04-25T02:15:59.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 79447 8.85 MB/sec execute 363 sec latency 78.028 ms 2024-04-25T02:16:00.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 79621 8.85 MB/sec execute 364 sec latency 58.147 ms 2024-04-25T02:16:00.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:00 smithi040 ceph-mon[32418]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 226 op/s 2024-04-25T02:16:00.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:00 smithi134 ceph-mon[33098]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 226 op/s 2024-04-25T02:16:00.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:00 smithi031 ceph-mon[29204]: pgmap v677: 97 pgs: 97 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 226 op/s 2024-04-25T02:16:01.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 79754 8.84 MB/sec execute 365 sec latency 118.920 ms 2024-04-25T02:16:02.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 79887 8.83 MB/sec execute 366 sec latency 68.548 ms 2024-04-25T02:16:02.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:02 smithi040 ceph-mon[32418]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.4 MiB/s wr, 350 op/s 2024-04-25T02:16:02.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:02 smithi134 ceph-mon[33098]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.4 MiB/s wr, 350 op/s 2024-04-25T02:16:02.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:02 smithi031 ceph-mon[29204]: pgmap v678: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.4 MiB/s wr, 350 op/s 2024-04-25T02:16:03.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 80032 8.83 MB/sec execute 367 sec latency 104.117 ms 2024-04-25T02:16:03.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:03 smithi040 ceph-mon[32418]: osdmap e120: 12 total, 12 up, 12 in 2024-04-25T02:16:03.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:03 smithi134 ceph-mon[33098]: osdmap e120: 12 total, 12 up, 12 in 2024-04-25T02:16:03.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:03 smithi031 ceph-mon[29204]: osdmap e120: 12 total, 12 up, 12 in 2024-04-25T02:16:04.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 80202 8.83 MB/sec execute 368 sec latency 97.703 ms 2024-04-25T02:16:04.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:04 smithi040 ceph-mon[32418]: pgmap v680: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 276 op/s 2024-04-25T02:16:04.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:04 smithi134 ceph-mon[33098]: pgmap v680: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 276 op/s 2024-04-25T02:16:04.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:04 smithi031 ceph-mon[29204]: pgmap v680: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 276 op/s 2024-04-25T02:16:05.104 INFO:tasks.workunit.client.0.smithi031.stdout: 10 80392 8.84 MB/sec execute 369 sec latency 219.183 ms 2024-04-25T02:16:06.105 INFO:tasks.workunit.client.0.smithi031.stdout: 10 80514 8.83 MB/sec execute 370 sec latency 131.441 ms 2024-04-25T02:16:06.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:06 smithi040 ceph-mon[32418]: pgmap v681: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 287 op/s 2024-04-25T02:16:06.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:06 smithi134 ceph-mon[33098]: pgmap v681: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 287 op/s 2024-04-25T02:16:06.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:06 smithi031 ceph-mon[29204]: pgmap v681: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 287 op/s 2024-04-25T02:16:07.105 INFO:tasks.workunit.client.0.smithi031.stdout: 10 80673 8.83 MB/sec execute 371 sec latency 76.726 ms 2024-04-25T02:16:08.105 INFO:tasks.workunit.client.0.smithi031.stdout: 10 80816 8.83 MB/sec execute 372 sec latency 99.266 ms 2024-04-25T02:16:08.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:08 smithi040 ceph-mon[32418]: pgmap v682: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 287 op/s 2024-04-25T02:16:08.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:08 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:16:08.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:08 smithi134 ceph-mon[33098]: pgmap v682: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 287 op/s 2024-04-25T02:16:08.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:08 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:16:08.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:08 smithi031 ceph-mon[29204]: pgmap v682: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 287 op/s 2024-04-25T02:16:08.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:08 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:16:08.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.496+0000 7fa845d9b640 1 -- 172.21.15.31:0/872739333 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa840034020 con 0x7fa848114890 2024-04-25T02:16:08.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.496+0000 7fa84d0d4640 1 -- 172.21.15.31:0/872739333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa848114890 msgr2=0x7fa848116c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:08.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.496+0000 7fa84d0d4640 1 --2- 172.21.15.31:0/872739333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa848114890 0x7fa848116c80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa840009cd0 tx=0x7fa84002a970 comp rx=0 tx=0).stop 2024-04-25T02:16:08.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.497+0000 7fa84d0d4640 1 -- 172.21.15.31:0/872739333 shutdown_connections 2024-04-25T02:16:08.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.497+0000 7fa84d0d4640 1 --2- 172.21.15.31:0/872739333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa848114890 0x7fa848116c80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.497+0000 7fa84d0d4640 1 --2- 172.21.15.31:0/872739333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa848109060 0x7fa848114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.497+0000 7fa84d0d4640 1 --2- 172.21.15.31:0/872739333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848108740 0x7fa848108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.497+0000 7fa84d0d4640 1 -- 172.21.15.31:0/872739333 >> 172.21.15.31:0/872739333 conn(0x7fa84806c810 msgr2=0x7fa84806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:08.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.497+0000 7fa84d0d4640 1 -- 172.21.15.31:0/872739333 shutdown_connections 2024-04-25T02:16:08.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.497+0000 7fa84d0d4640 1 -- 172.21.15.31:0/872739333 wait complete. 2024-04-25T02:16:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.498+0000 7fa84d0d4640 1 Processor -- start 2024-04-25T02:16:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.498+0000 7fa84d0d4640 1 -- start start 2024-04-25T02:16:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.498+0000 7fa84d0d4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa848108740 0x7fa84810a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.498+0000 7fa84d0d4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848109060 0x7fa84810a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.498+0000 7fa84d0d4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa848114890 0x7fa84810ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa84d0d4640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa84811a1f0 con 0x7fa848109060 2024-04-25T02:16:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa846d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa848108740 0x7fa84810a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa84d0d4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa84811a070 con 0x7fa848108740 2024-04-25T02:16:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa84d0d4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa84811a370 con 0x7fa848114890 2024-04-25T02:16:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa846d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa848108740 0x7fa84810a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45958/0 (socket says 172.21.15.31:45958) 2024-04-25T02:16:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa846d9d640 1 -- 172.21.15.31:0/2845993267 learned_addr learned my addr 172.21.15.31:0/2845993267 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa84759e640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa848114890 0x7fa84810ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa84659c640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848109060 0x7fa84810a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa846d9d640 1 -- 172.21.15.31:0/2845993267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa848114890 msgr2=0x7fa84810ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa846d9d640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa848114890 0x7fa84810ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa846d9d640 1 -- 172.21.15.31:0/2845993267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848109060 msgr2=0x7fa84810a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa846d9d640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848109060 0x7fa84810a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa846d9d640 1 -- 172.21.15.31:0/2845993267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8481aebb0 con 0x7fa848108740 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa84759e640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa848114890 0x7fa84810ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa84659c640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848109060 0x7fa84810a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.499+0000 7fa846d9d640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa848108740 0x7fa84810a1b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00d950 tx=0x7fa83c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.500+0000 7fa82ffff640 1 -- 172.21.15.31:0/2845993267 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c017030 con 0x7fa848108740 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.500+0000 7fa82ffff640 1 -- 172.21.15.31:0/2845993267 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa83c004020 con 0x7fa848108740 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.500+0000 7fa82ffff640 1 -- 172.21.15.31:0/2845993267 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0047a0 con 0x7fa848108740 2024-04-25T02:16:08.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.500+0000 7fa84d0d4640 1 -- 172.21.15.31:0/2845993267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8481aeea0 con 0x7fa848108740 2024-04-25T02:16:08.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.500+0000 7fa84d0d4640 1 -- 172.21.15.31:0/2845993267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8481af310 con 0x7fa848108740 2024-04-25T02:16:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.502+0000 7fa82ffff640 1 -- 172.21.15.31:0/2845993267 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa83c0041d0 con 0x7fa848108740 2024-04-25T02:16:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.502+0000 7fa84fde7740 1 Processor -- start 2024-04-25T02:16:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa84fde7740 1 -- start start 2024-04-25T02:16:08.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce3a50 0x55f2abce3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 0x55f2abcee500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 0x55f2abcf0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa84fde7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f2abc3fcb0 con 0x55f2abce4370 2024-04-25T02:16:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa84fde7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f2abbe85b0 con 0x55f2abce3a50 2024-04-25T02:16:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa84fde7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f2abc3bc70 con 0x55f2abceea40 2024-04-25T02:16:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa84759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce3a50 0x55f2abce3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa846d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 0x55f2abcee500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa84659c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 0x55f2abcf0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa82ffff640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8300a2e20 0x7fa8300a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa846d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 0x55f2abcee500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45554/0 (socket says 172.21.15.31:45554) 2024-04-25T02:16:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa846d9d640 1 -- 172.21.15.31:0/2360617838 learned_addr learned my addr 172.21.15.31:0/2360617838 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.503+0000 7fa84659c640 1 --2- 172.21.15.31:0/2360617838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 0x55f2abcf0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:51626/0 (socket says 172.21.15.31:51626) 2024-04-25T02:16:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa84659c640 1 -- 172.21.15.31:0/2360617838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce3a50 msgr2=0x55f2abce3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa84659c640 1 --2- 172.21.15.31:0/2360617838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce3a50 0x55f2abce3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa84659c640 1 -- 172.21.15.31:0/2360617838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 msgr2=0x55f2abcee500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa84659c640 1 --2- 172.21.15.31:0/2360617838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 0x55f2abcee500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa82ffff640 1 -- 172.21.15.31:0/2845993267 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8528+0+0 (secure 0 0 0) 0x7fa83c0cee40 con 0x7fa848108740 2024-04-25T02:16:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa84659c640 1 -- 172.21.15.31:0/2360617838 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2ab954210 con 0x55f2abceea40 2024-04-25T02:16:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa84759e640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8300a2e20 0x7fa8300a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa84659c640 1 --2- 172.21.15.31:0/2360617838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 0x55f2abcf0e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa838009b80 tx=0x7fa838009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7046872e8bc7749 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa82d7fa640 1 -- 172.21.15.31:0/2360617838 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa838011da0 con 0x55f2abceea40 2024-04-25T02:16:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa82d7fa640 1 -- 172.21.15.31:0/2360617838 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8380054d0 con 0x55f2abceea40 2024-04-25T02:16:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa84759e640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8300a2e20 0x7fa8300a52e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa840004af0 tx=0x7fa84002a970 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.504+0000 7fa82d7fa640 1 -- 172.21.15.31:0/2360617838 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8380126b0 con 0x55f2abceea40 2024-04-25T02:16:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.505+0000 7fa84fde7740 1 -- 172.21.15.31:0/2360617838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 msgr2=0x55f2abcf0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.505+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2360617838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 0x55f2abcf0e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa838009b80 tx=0x7fa838009f80 comp rx=0 tx=0).stop 2024-04-25T02:16:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.505+0000 7fa84fde7740 1 -- 172.21.15.31:0/2360617838 shutdown_connections 2024-04-25T02:16:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.505+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2360617838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 0x55f2abcf0e30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.505+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2360617838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 0x55f2abcee500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.505+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2360617838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce3a50 0x55f2abce3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.505+0000 7fa84fde7740 1 -- 172.21.15.31:0/2360617838 >> 172.21.15.31:0/2360617838 conn(0x55f2abcdfd70 msgr2=0x55f2abce0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.505+0000 7fa84fde7740 1 -- 172.21.15.31:0/2360617838 shutdown_connections 2024-04-25T02:16:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.506+0000 7fa84fde7740 1 -- 172.21.15.31:0/2360617838 wait complete. 2024-04-25T02:16:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.506+0000 7fa84fde7740 1 -- start start 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.506+0000 7fa84fde7740 1 Processor -- start 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.506+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce3a50 0x55f2abceb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.506+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce4370 0x55f2abcebe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.506+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 0x55f2abcec3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.506+0000 7fa84fde7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f2abcf1830 con 0x55f2abce3a50 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.506+0000 7fa84fde7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f2abbfbb50 con 0x55f2abce4370 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.506+0000 7fa84fde7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f2abcf19b0 con 0x55f2abceea40 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa84fde7740 10 client.0 _ll_register_callbacks cb 0x55f2abc36ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa84659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce4370 0x55f2abcebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa846d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce3a50 0x55f2abceb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa846d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce3a50 0x55f2abceb920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45556/0 (socket says 172.21.15.31:45556) 2024-04-25T02:16:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa846d9d640 1 -- 172.21.15.31:0/312757504 learned_addr learned my addr 172.21.15.31:0/312757504 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa84759e640 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 0x55f2abcec3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa84659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce4370 0x55f2abcebe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:45988/0 (socket says 172.21.15.31:45988) 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa846d9d640 1 -- 172.21.15.31:0/312757504 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 msgr2=0x55f2abcec3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa846d9d640 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 0x55f2abcec3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa846d9d640 1 -- 172.21.15.31:0/312757504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce4370 msgr2=0x55f2abcebe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa846d9d640 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce4370 0x55f2abcebe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa846d9d640 1 -- 172.21.15.31:0/312757504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2aba20620 con 0x55f2abce3a50 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa84759e640 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 0x55f2abcec3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa846d9d640 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce3a50 0x55f2abceb920 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0977c0 tx=0x7fa83c09acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2679863eced37454 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.507+0000 7fa82aff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa82aff5640 1 -- 172.21.15.31:0/312757504 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0cca10 con 0x55f2abce3a50 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa82aff5640 1 -- 172.21.15.31:0/312757504 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8528+0+0 (secure 0 0 0) 0x7fa83c09a7b0 con 0x55f2abce3a50 2024-04-25T02:16:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa84fde7740 1 -- 172.21.15.31:0/312757504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8380054d0 con 0x55f2abce3a50 2024-04-25T02:16:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa82aff5640 1 -- 172.21.15.31:0/312757504 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fa83c09f710 con 0x55f2abce3a50 2024-04-25T02:16:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa84fde7740 10 client.17865 fetch_fsmap learned FSMap version 50 2024-04-25T02:16:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa84fde7740 1 -- 172.21.15.31:0/312757504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55f2abcfda60 con 0x55f2abce3a50 2024-04-25T02:16:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa82aff5640 1 -- 172.21.15.31:0/312757504 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa83c0a3fd0 con 0x55f2abce3a50 2024-04-25T02:16:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa82aff5640 1 -- 172.21.15.31:0/312757504 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7fa83c0ccce0 con 0x55f2abce3a50 2024-04-25T02:16:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa84fde7740 10 client.17865 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:16:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa84fde7740 10 client.17865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:16:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa84fde7740 20 client.17865 populate_metadata read hostname 'smithi031' 2024-04-25T02:16:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa84fde7740 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f2abcfef20 0x55f2abd01310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa84fde7740 4 client.17865 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:16:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.508+0000 7fa84fde7740 1 -- 172.21.15.31:0/312757504 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa8380054d0 con 0x55f2abcfef20 2024-04-25T02:16:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.509+0000 7fa84659c640 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f2abcfef20 0x55f2abd01310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.509+0000 7fa84659c640 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f2abcfef20 0x55f2abd01310 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=892d06b6b175ac81 server_cookie=33fb9a8446cd306c in_seq=0 out_seq=0 2024-04-25T02:16:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.509+0000 7fa82aff5640 10 client.17865 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.510+0000 7fa82aff5640 1 -- 172.21.15.31:0/312757504 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fa8380054d0 con 0x55f2abcfef20 2024-04-25T02:16:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.510+0000 7fa82aff5640 10 client.17865 handle_command_reply: tid=0 2024-04-25T02:16:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.510+0000 7fa84fde7740 1 client.17865 shutdown 2024-04-25T02:16:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.510+0000 7fa84fde7740 1 -- 172.21.15.31:0/312757504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce3a50 msgr2=0x55f2abceb920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.510+0000 7fa84fde7740 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce3a50 0x55f2abceb920 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0977c0 tx=0x7fa83c09acc0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.510+0000 7fa84fde7740 1 -- 172.21.15.31:0/312757504 shutdown_connections 2024-04-25T02:16:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.511+0000 7fa84fde7740 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f2abcfef20 0x55f2abd01310 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.511+0000 7fa84fde7740 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abceea40 0x55f2abcec3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.511+0000 7fa82aff5640 0 client.17865 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.511+0000 7fa84fde7740 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce4370 0x55f2abcebe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.511+0000 7fa84fde7740 1 --2- 172.21.15.31:0/312757504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce3a50 0x55f2abceb920 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.511+0000 7fa84fde7740 1 -- 172.21.15.31:0/312757504 >> 172.21.15.31:0/312757504 conn(0x55f2abcdfd70 msgr2=0x55f2abcd65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.511+0000 7fa84fde7740 1 -- 172.21.15.31:0/312757504 shutdown_connections 2024-04-25T02:16:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.511+0000 7fa84fde7740 1 -- 172.21.15.31:0/312757504 wait complete. 2024-04-25T02:16:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.511+0000 7fa84fde7740 20 client.17865 trim_cache size 0 max 16384 2024-04-25T02:16:08.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.539+0000 7fa84fde7740 1 Processor -- start 2024-04-25T02:16:08.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.539+0000 7fa84fde7740 1 -- start start 2024-04-25T02:16:08.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce3a50 0x55f2abcedfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abce4370 0x55f2abceaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abceea40 0x55f2abce61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84fde7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa8481b3980 con 0x55f2abceea40 2024-04-25T02:16:08.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84fde7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8481b54c0 con 0x55f2abce3a50 2024-04-25T02:16:08.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84fde7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa8481b4050 con 0x55f2abce4370 2024-04-25T02:16:08.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abceea40 0x55f2abce61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abceea40 0x55f2abce61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45568/0 (socket says 172.21.15.31:45568) 2024-04-25T02:16:08.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84759e640 1 -- 172.21.15.31:0/2554221434 learned_addr learned my addr 172.21.15.31:0/2554221434 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:08.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84659c640 1 --2- 172.21.15.31:0/2554221434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abce4370 0x55f2abceaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa846d9d640 1 --2- 172.21.15.31:0/2554221434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce3a50 0x55f2abcedfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84759e640 1 -- 172.21.15.31:0/2554221434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abce4370 msgr2=0x55f2abceaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:08.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84759e640 1 --2- 172.21.15.31:0/2554221434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abce4370 0x55f2abceaaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84759e640 1 -- 172.21.15.31:0/2554221434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce3a50 msgr2=0x55f2abcedfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:08.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84759e640 1 --2- 172.21.15.31:0/2554221434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce3a50 0x55f2abcedfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.540+0000 7fa84759e640 1 -- 172.21.15.31:0/2554221434 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2abce9e90 con 0x55f2abceea40 2024-04-25T02:16:08.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.541+0000 7fa84759e640 1 --2- 172.21.15.31:0/2554221434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abceea40 0x55f2abce61b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fa840035000 tx=0x7fa840005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e129cf8380884 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:08.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.541+0000 7fa82cff9640 1 -- 172.21.15.31:0/2554221434 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa840034040 con 0x55f2abceea40 2024-04-25T02:16:08.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.541+0000 7fa82cff9640 1 -- 172.21.15.31:0/2554221434 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa840003b70 con 0x55f2abceea40 2024-04-25T02:16:08.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.541+0000 7fa82cff9640 1 -- 172.21.15.31:0/2554221434 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa840043040 con 0x55f2abceea40 2024-04-25T02:16:08.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.542+0000 7fa84fde7740 1 -- 172.21.15.31:0/2554221434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abceea40 msgr2=0x55f2abce61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:08.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.542+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2554221434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abceea40 0x55f2abce61b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fa840035000 tx=0x7fa840005d80 comp rx=0 tx=0).stop 2024-04-25T02:16:08.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.542+0000 7fa84fde7740 1 -- 172.21.15.31:0/2554221434 shutdown_connections 2024-04-25T02:16:08.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.542+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2554221434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abceea40 0x55f2abce61b0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.542+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2554221434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abce4370 0x55f2abceaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.542+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2554221434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abce3a50 0x55f2abcedfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.542+0000 7fa84fde7740 1 -- 172.21.15.31:0/2554221434 >> 172.21.15.31:0/2554221434 conn(0x55f2abcdfd70 msgr2=0x55f2abcd5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:08.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.542+0000 7fa84fde7740 1 -- 172.21.15.31:0/2554221434 shutdown_connections 2024-04-25T02:16:08.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.542+0000 7fa84fde7740 1 -- 172.21.15.31:0/2554221434 wait complete. 2024-04-25T02:16:08.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.543+0000 7fa84fde7740 1 -- start start 2024-04-25T02:16:08.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.543+0000 7fa84fde7740 1 Processor -- start 2024-04-25T02:16:08.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.543+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abce3a50 0x55f2abcf6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.543+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 0x55f2abcf7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.543+0000 7fa84fde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abceea40 0x55f2abce7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.543+0000 7fa84659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 0x55f2abcf7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.543+0000 7fa84659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 0x55f2abcf7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45580/0 (socket says 172.21.15.31:45580) 2024-04-25T02:16:08.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.543+0000 7fa846d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abce3a50 0x55f2abcf6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa846d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abce3a50 0x55f2abcf6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:51654/0 (socket says 172.21.15.31:51654) 2024-04-25T02:16:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84fde7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa8481b4df0 con 0x55f2abce4370 2024-04-25T02:16:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abceea40 0x55f2abce7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84659c640 1 -- 172.21.15.31:0/2852317495 learned_addr learned my addr 172.21.15.31:0/2852317495 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84fde7740 1 -- 172.21.15.31:0/2852317495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa8481b4720 con 0x55f2abceea40 2024-04-25T02:16:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84fde7740 1 -- 172.21.15.31:0/2852317495 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa8481b1960 con 0x55f2abce3a50 2024-04-25T02:16:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84fde7740 10 client.0 _ll_register_callbacks cb 0x7fa83c004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:16:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84759e640 1 -- 172.21.15.31:0/2852317495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abce3a50 msgr2=0x55f2abcf6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84759e640 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abce3a50 0x55f2abcf6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84759e640 1 -- 172.21.15.31:0/2852317495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 msgr2=0x55f2abcf7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84759e640 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 0x55f2abcf7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84759e640 1 -- 172.21.15.31:0/2852317495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2abce8730 con 0x55f2abceea40 2024-04-25T02:16:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.544+0000 7fa84659c640 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 0x55f2abcf7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:16:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.545+0000 7fa84759e640 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abceea40 0x55f2abce7f20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa84002ae90 tx=0x7fa840005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5020ffa60c5d30c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.545+0000 7fa82aff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:16:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.545+0000 7fa82aff5640 1 -- 172.21.15.31:0/2852317495 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa840034070 con 0x55f2abceea40 2024-04-25T02:16:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.546+0000 7fa82aff5640 1 -- 172.21.15.31:0/2852317495 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8528+0+0 (secure 0 0 0) 0x7fa840030a40 con 0x55f2abceea40 2024-04-25T02:16:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.546+0000 7fa84fde7740 1 -- 172.21.15.31:0/2852317495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa840003b70 con 0x55f2abceea40 2024-04-25T02:16:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.546+0000 7fa82aff5640 1 -- 172.21.15.31:0/2852317495 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fa840043040 con 0x55f2abceea40 2024-04-25T02:16:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.546+0000 7fa84fde7740 10 client.27437 fetch_fsmap learned FSMap version 50 2024-04-25T02:16:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.546+0000 7fa84fde7740 1 -- 172.21.15.31:0/2852317495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55f2abd64eb0 con 0x55f2abceea40 2024-04-25T02:16:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.546+0000 7fa82aff5640 1 -- 172.21.15.31:0/2852317495 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa840059070 con 0x55f2abceea40 2024-04-25T02:16:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.547+0000 7fa82aff5640 1 -- 172.21.15.31:0/2852317495 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7fa84005f4d0 con 0x55f2abceea40 2024-04-25T02:16:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.547+0000 7fa84fde7740 10 client.27437 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:16:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.547+0000 7fa84fde7740 10 client.27437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:16:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.547+0000 7fa84fde7740 20 client.27437 populate_metadata read hostname 'smithi031' 2024-04-25T02:16:08.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.547+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f2abcfef20 0x55f2abd67530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.547+0000 7fa84fde7740 4 client.27437 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:16:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.547+0000 7fa84fde7740 1 -- 172.21.15.31:0/2852317495 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fa840003b70 con 0x55f2abcfef20 2024-04-25T02:16:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.547+0000 7fa846d9d640 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f2abcfef20 0x55f2abd67530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.548+0000 7fa82aff5640 10 client.27437 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.548+0000 7fa846d9d640 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f2abcfef20 0x55f2abd67530 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56610a6aa062e17c server_cookie=e6794132ab6d6ea4 in_seq=0 out_seq=0 2024-04-25T02:16:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.628+0000 7fa82aff5640 1 -- 172.21.15.31:0/2852317495 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fa840003b70 con 0x55f2abcfef20 2024-04-25T02:16:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.628+0000 7fa82aff5640 10 client.27437 handle_command_reply: tid=0 2024-04-25T02:16:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.628+0000 7fa84fde7740 1 client.27437 shutdown 2024-04-25T02:16:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.628+0000 7fa84fde7740 1 -- 172.21.15.31:0/2852317495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abceea40 msgr2=0x55f2abce7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.628+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abceea40 0x55f2abce7f20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa84002ae90 tx=0x7fa840005bf0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.628+0000 7fa84fde7740 1 -- 172.21.15.31:0/2852317495 shutdown_connections 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.629+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f2abcfef20 0x55f2abd67530 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.629+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f2abceea40 0x55f2abce7f20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.629+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2abce4370 0x55f2abcf7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.629+0000 7fa82aff5640 0 client.27437 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.629+0000 7fa84fde7740 1 --2- 172.21.15.31:0/2852317495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f2abce3a50 0x55f2abcf6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.629+0000 7fa84fde7740 1 -- 172.21.15.31:0/2852317495 >> 172.21.15.31:0/2852317495 conn(0x55f2abcdfd70 msgr2=0x55f2abcd65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.629+0000 7fa84fde7740 1 -- 172.21.15.31:0/2852317495 shutdown_connections 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.629+0000 7fa84fde7740 1 -- 172.21.15.31:0/2852317495 wait complete. 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.629+0000 7fa84fde7740 20 client.27437 trim_cache size 0 max 16384 2024-04-25T02:16:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:16:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:16:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:16:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:16:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 -- 172.21.15.31:0/2845993267 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8300a2e20 msgr2=0x7fa8300a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8300a2e20 0x7fa8300a52e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa840004af0 tx=0x7fa84002a970 comp rx=0 tx=0).stop 2024-04-25T02:16:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 -- 172.21.15.31:0/2845993267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa848108740 msgr2=0x7fa84810a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa848108740 0x7fa84810a1b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00d950 tx=0x7fa83c00de10 comp rx=0 tx=0).stop 2024-04-25T02:16:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 -- 172.21.15.31:0/2845993267 shutdown_connections 2024-04-25T02:16:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa8300a2e20 0x7fa8300a52e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa848114890 0x7fa84810ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848109060 0x7fa84810a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 --2- 172.21.15.31:0/2845993267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa848108740 0x7fa84810a1b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 -- 172.21.15.31:0/2845993267 >> 172.21.15.31:0/2845993267 conn(0x7fa84806c810 msgr2=0x7fa8480703c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 -- 172.21.15.31:0/2845993267 shutdown_connections 2024-04-25T02:16:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:08.630+0000 7fa84d0d4640 1 -- 172.21.15.31:0/2845993267 wait complete. 2024-04-25T02:16:09.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 80959 8.83 MB/sec execute 373 sec latency 112.465 ms 2024-04-25T02:16:09.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:09 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:16:09.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:09 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:16:09.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:09 smithi031 ceph-mon[29204]: osdmap e121: 12 total, 12 up, 12 in 2024-04-25T02:16:09.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:09 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:16:09.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:09 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:16:09.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:09 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:16:09.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:09 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:16:09.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:09 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:16:09.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:09 smithi040 ceph-mon[32418]: osdmap e121: 12 total, 12 up, 12 in 2024-04-25T02:16:09.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:09 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:16:09.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:09 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:16:09.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:09 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:16:09.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:09 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:16:09.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:09 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:16:09.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:09 smithi134 ceph-mon[33098]: osdmap e121: 12 total, 12 up, 12 in 2024-04-25T02:16:09.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:09 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:16:09.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:09 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:16:09.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:09 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:16:10.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 81130 8.84 MB/sec execute 374 sec latency 85.325 ms 2024-04-25T02:16:10.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:10 smithi040 ceph-mon[32418]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 287 op/s 2024-04-25T02:16:10.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:10 smithi134 ceph-mon[33098]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 287 op/s 2024-04-25T02:16:10.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:10 smithi031 ceph-mon[29204]: pgmap v683: 97 pgs: 97 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 287 op/s 2024-04-25T02:16:11.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 81272 8.82 MB/sec execute 375 sec latency 74.821 ms 2024-04-25T02:16:12.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 81415 8.81 MB/sec execute 376 sec latency 63.510 ms 2024-04-25T02:16:12.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:12 smithi040 ceph-mon[32418]: pgmap v685: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.3 MiB/s wr, 339 op/s 2024-04-25T02:16:12.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:12 smithi134 ceph-mon[33098]: pgmap v685: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.3 MiB/s wr, 339 op/s 2024-04-25T02:16:12.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:12 smithi031 ceph-mon[29204]: pgmap v685: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.3 MiB/s wr, 339 op/s 2024-04-25T02:16:13.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 81580 8.80 MB/sec execute 377 sec latency 84.586 ms 2024-04-25T02:16:14.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 81745 8.79 MB/sec execute 378 sec latency 57.619 ms 2024-04-25T02:16:14.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:14 smithi040 ceph-mon[32418]: pgmap v686: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.4 MiB/s wr, 299 op/s 2024-04-25T02:16:14.837 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:14 smithi031 ceph-mon[29204]: pgmap v686: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.4 MiB/s wr, 299 op/s 2024-04-25T02:16:14.837 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:14 smithi134 ceph-mon[33098]: pgmap v686: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.4 MiB/s wr, 299 op/s 2024-04-25T02:16:15.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 81928 8.81 MB/sec execute 379 sec latency 57.073 ms 2024-04-25T02:16:16.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 82109 8.81 MB/sec execute 380 sec latency 59.643 ms 2024-04-25T02:16:16.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:16 smithi040 ceph-mon[32418]: pgmap v687: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.8 MiB/s wr, 298 op/s 2024-04-25T02:16:16.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:16 smithi134 ceph-mon[33098]: pgmap v687: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.8 MiB/s wr, 298 op/s 2024-04-25T02:16:16.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:16 smithi031 ceph-mon[29204]: pgmap v687: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.8 MiB/s wr, 298 op/s 2024-04-25T02:16:17.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 82278 8.82 MB/sec execute 381 sec latency 154.403 ms 2024-04-25T02:16:18.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 82488 8.83 MB/sec execute 382 sec latency 60.495 ms 2024-04-25T02:16:18.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:18 smithi040 ceph-mon[32418]: pgmap v688: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.8 MiB/s wr, 298 op/s 2024-04-25T02:16:18.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:18 smithi134 ceph-mon[33098]: pgmap v688: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.8 MiB/s wr, 298 op/s 2024-04-25T02:16:18.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:18 smithi031 ceph-mon[29204]: pgmap v688: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.8 MiB/s wr, 298 op/s 2024-04-25T02:16:19.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 82677 8.83 MB/sec execute 383 sec latency 60.343 ms 2024-04-25T02:16:20.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 82816 8.83 MB/sec execute 384 sec latency 57.073 ms 2024-04-25T02:16:20.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:20 smithi040 ceph-mon[32418]: pgmap v689: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.8 MiB/s wr, 298 op/s 2024-04-25T02:16:20.707 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:20 smithi134 ceph-mon[33098]: pgmap v689: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.8 MiB/s wr, 298 op/s 2024-04-25T02:16:20.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:20 smithi031 ceph-mon[29204]: pgmap v689: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.8 MiB/s wr, 298 op/s 2024-04-25T02:16:21.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 82968 8.82 MB/sec execute 385 sec latency 64.636 ms 2024-04-25T02:16:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.769+0000 7f5499789640 1 -- 172.21.15.31:0/2323612226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5494109960 msgr2=0x7f5494109d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.769+0000 7f5499789640 1 --2- 172.21.15.31:0/2323612226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5494109960 0x7f5494109d40 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5484009930 tx=0x7f548402a420 comp rx=0 tx=0).stop 2024-04-25T02:16:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.769+0000 7f5499789640 1 -- 172.21.15.31:0/2323612226 shutdown_connections 2024-04-25T02:16:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.769+0000 7f5499789640 1 --2- 172.21.15.31:0/2323612226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5494104310 0x7f5494110820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.769+0000 7f5499789640 1 --2- 172.21.15.31:0/2323612226 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5494103950 0x7f5494103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.769+0000 7f5499789640 1 --2- 172.21.15.31:0/2323612226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5494109960 0x7f5494109d40 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.769+0000 7f5499789640 1 -- 172.21.15.31:0/2323612226 >> 172.21.15.31:0/2323612226 conn(0x7f54940ff400 msgr2=0x7f5494101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.769+0000 7f5499789640 1 -- 172.21.15.31:0/2323612226 shutdown_connections 2024-04-25T02:16:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.769+0000 7f5499789640 1 -- 172.21.15.31:0/2323612226 wait complete. 2024-04-25T02:16:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f5499789640 1 Processor -- start 2024-04-25T02:16:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f5499789640 1 -- start start 2024-04-25T02:16:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f5499789640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5494103950 0x7f549419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f5499789640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5494104310 0x7f549419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f5499789640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5494109960 0x7f549419e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f5499789640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5494113c20 con 0x7f5494104310 2024-04-25T02:16:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f5499789640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5494113aa0 con 0x7f5494109960 2024-04-25T02:16:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f5499789640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5494113da0 con 0x7f5494103950 2024-04-25T02:16:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f548bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5494104310 0x7f549419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f548bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5494104310 0x7f549419a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47436/0 (socket says 172.21.15.31:47436) 2024-04-25T02:16:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f548bfff640 1 -- 172.21.15.31:0/1012865911 learned_addr learned my addr 172.21.15.31:0/1012865911 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f5492ffd640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5494103950 0x7f549419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.770+0000 7f54937fe640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5494109960 0x7f549419e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.771+0000 7f548bfff640 1 -- 172.21.15.31:0/1012865911 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5494103950 msgr2=0x7f549419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.771+0000 7f548bfff640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5494103950 0x7f549419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.771+0000 7f548bfff640 1 -- 172.21.15.31:0/1012865911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5494109960 msgr2=0x7f549419e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.771+0000 7f548bfff640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5494109960 0x7f549419e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.771+0000 7f548bfff640 1 -- 172.21.15.31:0/1012865911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549419f0f0 con 0x7f5494104310 2024-04-25T02:16:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.771+0000 7f54937fe640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5494109960 0x7f549419e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:16:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.771+0000 7f548bfff640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5494104310 0x7f549419a640 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f547c00e9f0 tx=0x7f547c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.771+0000 7f5490ff9640 1 -- 172.21.15.31:0/1012865911 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547c00cd80 con 0x7f5494104310 2024-04-25T02:16:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.771+0000 7f5490ff9640 1 -- 172.21.15.31:0/1012865911 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f547c0054d0 con 0x7f5494104310 2024-04-25T02:16:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.771+0000 7f5490ff9640 1 -- 172.21.15.31:0/1012865911 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547c010490 con 0x7f5494104310 2024-04-25T02:16:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.771+0000 7f5499789640 1 -- 172.21.15.31:0/1012865911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549419f3e0 con 0x7f5494104310 2024-04-25T02:16:21.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.772+0000 7f5499789640 1 -- 172.21.15.31:0/1012865911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54941a6c70 con 0x7f5494104310 2024-04-25T02:16:21.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.775+0000 7f5490ff9640 1 -- 172.21.15.31:0/1012865911 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f547c0106b0 con 0x7f5494104310 2024-04-25T02:16:21.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.775+0000 7f549c573740 1 Processor -- start 2024-04-25T02:16:21.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.775+0000 7f549c573740 1 -- start start 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f549c573740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a2c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f549c573740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a3f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f549c573740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a3fa40 0x5603e4a41e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f549c573740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5603e4990cb0 con 0x5603e4a3fa40 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f549c573740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5603e49395b0 con 0x5603e4a2c940 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f549c573740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5603e498cc70 con 0x5603e4a2bf90 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f54937fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a2c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f548bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a3fa40 0x5603e4a41e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f54937fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a2c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:41324/0 (socket says 172.21.15.31:41324) 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f54937fe640 1 -- 172.21.15.31:0/2321434128 learned_addr learned my addr 172.21.15.31:0/2321434128 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f5492ffd640 1 --2- 172.21.15.31:0/2321434128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a3f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f54937fe640 1 -- 172.21.15.31:0/2321434128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 msgr2=0x5603e4a3f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:21.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f54937fe640 1 --2- 172.21.15.31:0/2321434128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a3f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f54937fe640 1 -- 172.21.15.31:0/2321434128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a3fa40 msgr2=0x5603e4a41e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:21.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f54937fe640 1 --2- 172.21.15.31:0/2321434128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a3fa40 0x5603e4a41e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f5490ff9640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54800a2e20 0x7f54800a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.776+0000 7f54937fe640 1 -- 172.21.15.31:0/2321434128 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e46a5210 con 0x5603e4a2bf90 2024-04-25T02:16:21.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.777+0000 7f5490ff9640 1 -- 172.21.15.31:0/1012865911 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7f547c0cf730 con 0x7f5494104310 2024-04-25T02:16:21.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.777+0000 7f54937fe640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54800a2e20 0x7f54800a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.777+0000 7f54937fe640 1 --2- 172.21.15.31:0/2321434128 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a2c370 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f548c011790 tx=0x7f548c011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=499db9defdb62992 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.777+0000 7f54897fa640 1 -- 172.21.15.31:0/2321434128 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548c00fe30 con 0x5603e4a2bf90 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.777+0000 7f54937fe640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54800a2e20 0x7f54800a52e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f548c020000 tx=0x7f548c021000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.777+0000 7f54897fa640 1 -- 172.21.15.31:0/2321434128 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f548c01c3f0 con 0x5603e4a2bf90 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.777+0000 7f54897fa640 1 -- 172.21.15.31:0/2321434128 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548c025630 con 0x5603e4a2bf90 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.778+0000 7f549c573740 1 -- 172.21.15.31:0/2321434128 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 msgr2=0x5603e4a2c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.778+0000 7f549c573740 1 --2- 172.21.15.31:0/2321434128 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a2c370 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f548c011790 tx=0x7f548c011c50 comp rx=0 tx=0).stop 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.778+0000 7f549c573740 1 -- 172.21.15.31:0/2321434128 shutdown_connections 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.778+0000 7f549c573740 1 --2- 172.21.15.31:0/2321434128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a3fa40 0x5603e4a41e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.778+0000 7f549c573740 1 --2- 172.21.15.31:0/2321434128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a3f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.778+0000 7f549c573740 1 --2- 172.21.15.31:0/2321434128 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a2c370 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.778+0000 7f549c573740 1 -- 172.21.15.31:0/2321434128 >> 172.21.15.31:0/2321434128 conn(0x5603e4a28560 msgr2=0x5603e4a28940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.778+0000 7f549c573740 1 -- 172.21.15.31:0/2321434128 shutdown_connections 2024-04-25T02:16:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.778+0000 7f549c573740 1 -- 172.21.15.31:0/2321434128 wait complete. 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f549c573740 1 -- start start 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f549c573740 1 Processor -- start 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f549c573740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a3c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f549c573740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a3cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f549c573740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a3fa40 0x5603e4a3d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f549c573740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5603e4a428a0 con 0x5603e4a3fa40 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f549c573740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5603e494cb50 con 0x5603e4a2c940 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f549c573740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5603e4a42a20 con 0x5603e4a2bf90 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f549c573740 10 client.0 _ll_register_callbacks cb 0x5603e4987ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f548bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a3cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f548bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a3cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58910/0 (socket says 172.21.15.31:58910) 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f548bfff640 1 -- 172.21.15.31:0/2046012372 learned_addr learned my addr 172.21.15.31:0/2046012372 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.779+0000 7f5492ffd640 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a3c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.780+0000 7f54937fe640 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a3fa40 0x5603e4a3d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.780+0000 7f548bfff640 1 -- 172.21.15.31:0/2046012372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 msgr2=0x5603e4a3c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.780+0000 7f548bfff640 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a3c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.780+0000 7f548bfff640 1 -- 172.21.15.31:0/2046012372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a3fa40 msgr2=0x5603e4a3d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.780+0000 7f548bfff640 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a3fa40 0x5603e4a3d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.780+0000 7f548bfff640 1 -- 172.21.15.31:0/2046012372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e4771620 con 0x5603e4a2c940 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.780+0000 7f548bfff640 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a3cf20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f547c00c6a0 tx=0x7f547c009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64cc138f110183f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.780+0000 7f547a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.780+0000 7f547a7fc640 1 -- 172.21.15.31:0/2046012372 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547c09b7b0 con 0x5603e4a2c940 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.780+0000 7f547a7fc640 1 -- 172.21.15.31:0/2046012372 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7f547c0cfbe0 con 0x5603e4a2c940 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.780+0000 7f549c573740 1 -- 172.21.15.31:0/2046012372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f548c01c3f0 con 0x5603e4a2c940 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.781+0000 7f547a7fc640 1 -- 172.21.15.31:0/2046012372 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f547c09b5d0 con 0x5603e4a2c940 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.781+0000 7f549c573740 10 client.27449 fetch_fsmap learned FSMap version 50 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.781+0000 7f549c573740 1 -- 172.21.15.31:0/2046012372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5603e4a4eae0 con 0x5603e4a2c940 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.781+0000 7f547a7fc640 1 -- 172.21.15.31:0/2046012372 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f548c01c3f0 con 0x5603e4a2c940 2024-04-25T02:16:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.781+0000 7f547a7fc640 1 -- 172.21.15.31:0/2046012372 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f547c0a4450 con 0x5603e4a2c940 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.781+0000 7f549c573740 10 client.27449 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.781+0000 7f549c573740 10 client.27449 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.781+0000 7f549c573740 20 client.27449 populate_metadata read hostname 'smithi031' 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.781+0000 7f549c573740 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5603e4a504a0 0x5603e4a528e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.782+0000 7f549c573740 4 client.27449 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.782+0000 7f549c573740 1 -- 172.21.15.31:0/2046012372 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603e499e1f0 con 0x5603e4a504a0 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.782+0000 7f5492ffd640 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5603e4a504a0 0x5603e4a528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.782+0000 7f5492ffd640 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5603e4a504a0 0x5603e4a528e0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8af0968746d02ee server_cookie=61c791f563623c5c in_seq=0 out_seq=0 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.782+0000 7f547a7fc640 10 client.27449 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f547a7fc640 1 -- 172.21.15.31:0/2046012372 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5603e499e1f0 con 0x5603e4a504a0 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f547a7fc640 10 client.27449 handle_command_reply: tid=0 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f549c573740 1 client.27449 shutdown 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f549c573740 1 -- 172.21.15.31:0/2046012372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 msgr2=0x5603e4a3cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f549c573740 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a3cf20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f547c00c6a0 tx=0x7f547c009720 comp rx=0 tx=0).stop 2024-04-25T02:16:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f549c573740 1 -- 172.21.15.31:0/2046012372 shutdown_connections 2024-04-25T02:16:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f549c573740 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5603e4a504a0 0x5603e4a528e0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f549c573740 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a3fa40 0x5603e4a3d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f549c573740 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a3cf20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f549c573740 1 --2- 172.21.15.31:0/2046012372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a3c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f547a7fc640 0 client.27449 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.783+0000 7f549c573740 1 -- 172.21.15.31:0/2046012372 >> 172.21.15.31:0/2046012372 conn(0x5603e4a28560 msgr2=0x5603e4a2e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.784+0000 7f549c573740 1 -- 172.21.15.31:0/2046012372 shutdown_connections 2024-04-25T02:16:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.784+0000 7f549c573740 1 -- 172.21.15.31:0/2046012372 wait complete. 2024-04-25T02:16:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.784+0000 7f549c573740 20 client.27449 trim_cache size 0 max 16384 2024-04-25T02:16:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.806+0000 7f549c573740 1 Processor -- start 2024-04-25T02:16:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.806+0000 7f549c573740 1 -- start start 2024-04-25T02:16:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f549c573740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a4b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f549c573740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2c940 0x5603e4a3f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f549c573740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a3fa40 0x5603e4a27440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f549c573740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54941ab180 con 0x5603e4a2c940 2024-04-25T02:16:21.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f549c573740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f54941accc0 con 0x5603e4a3fa40 2024-04-25T02:16:21.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f549c573740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f54941ab850 con 0x5603e4a2bf90 2024-04-25T02:16:21.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f548bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2c940 0x5603e4a3f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f548bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2c940 0x5603e4a3f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47472/0 (socket says 172.21.15.31:47472) 2024-04-25T02:16:21.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f548bfff640 1 -- 172.21.15.31:0/2680973643 learned_addr learned my addr 172.21.15.31:0/2680973643 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:21.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f5492ffd640 1 --2- 172.21.15.31:0/2680973643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a4b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f54937fe640 1 --2- 172.21.15.31:0/2680973643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a3fa40 0x5603e4a27440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f548bfff640 1 -- 172.21.15.31:0/2680973643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 msgr2=0x5603e4a4b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:21.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f548bfff640 1 --2- 172.21.15.31:0/2680973643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a4b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f548bfff640 1 -- 172.21.15.31:0/2680973643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a3fa40 msgr2=0x5603e4a27440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:21.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f548bfff640 1 --2- 172.21.15.31:0/2680973643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a3fa40 0x5603e4a27440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f548bfff640 1 -- 172.21.15.31:0/2680973643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e4a50180 con 0x5603e4a2c940 2024-04-25T02:16:21.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.807+0000 7f54937fe640 1 --2- 172.21.15.31:0/2680973643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a3fa40 0x5603e4a27440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:16:21.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.808+0000 7f548bfff640 1 --2- 172.21.15.31:0/2680973643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2c940 0x5603e4a3f0a0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f547c0980d0 tx=0x7f547c0ccf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd943ac626f06720 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:21.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.808+0000 7f5488ff9640 1 -- 172.21.15.31:0/2680973643 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547c09b7b0 con 0x5603e4a2c940 2024-04-25T02:16:21.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.808+0000 7f5488ff9640 1 -- 172.21.15.31:0/2680973643 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f547c09c370 con 0x5603e4a2c940 2024-04-25T02:16:21.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.808+0000 7f5488ff9640 1 -- 172.21.15.31:0/2680973643 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547c0982e0 con 0x5603e4a2c940 2024-04-25T02:16:21.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.809+0000 7f549c573740 1 -- 172.21.15.31:0/2680973643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2c940 msgr2=0x5603e4a3f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:21.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.809+0000 7f549c573740 1 --2- 172.21.15.31:0/2680973643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2c940 0x5603e4a3f0a0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f547c0980d0 tx=0x7f547c0ccf90 comp rx=0 tx=0).stop 2024-04-25T02:16:21.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.809+0000 7f549c573740 1 -- 172.21.15.31:0/2680973643 shutdown_connections 2024-04-25T02:16:21.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.809+0000 7f549c573740 1 --2- 172.21.15.31:0/2680973643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a3fa40 0x5603e4a27440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.809+0000 7f549c573740 1 --2- 172.21.15.31:0/2680973643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2c940 0x5603e4a3f0a0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.809+0000 7f549c573740 1 --2- 172.21.15.31:0/2680973643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a2bf90 0x5603e4a4b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.809+0000 7f549c573740 1 -- 172.21.15.31:0/2680973643 >> 172.21.15.31:0/2680973643 conn(0x5603e4a28560 msgr2=0x5603e4a26d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:21.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.809+0000 7f549c573740 1 -- 172.21.15.31:0/2680973643 shutdown_connections 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.809+0000 7f549c573740 1 -- 172.21.15.31:0/2680973643 wait complete. 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f549c573740 1 -- start start 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f549c573740 1 Processor -- start 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f549c573740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2bf90 0x5603e4a438b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f549c573740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a43df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f549c573740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a3fa40 0x5603e4a44330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f549c573740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54941ac5f0 con 0x5603e4a2bf90 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f549c573740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f54941abf20 con 0x5603e4a2c940 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f549c573740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f54941a9180 con 0x5603e4a3fa40 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f54937fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a3fa40 0x5603e4a44330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f548bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a43df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f54937fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a3fa40 0x5603e4a44330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:41352/0 (socket says 172.21.15.31:41352) 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f548bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a43df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:58928/0 (socket says 172.21.15.31:58928) 2024-04-25T02:16:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f54937fe640 1 -- 172.21.15.31:0/1678849152 learned_addr learned my addr 172.21.15.31:0/1678849152 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.810+0000 7f549c573740 10 client.0 _ll_register_callbacks cb 0x7f547c0d5240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.811+0000 7f5492ffd640 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2bf90 0x5603e4a438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.811+0000 7f548bfff640 1 -- 172.21.15.31:0/1678849152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a3fa40 msgr2=0x5603e4a44330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.811+0000 7f548bfff640 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a3fa40 0x5603e4a44330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.811+0000 7f548bfff640 1 -- 172.21.15.31:0/1678849152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2bf90 msgr2=0x5603e4a438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.812+0000 7f548bfff640 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2bf90 0x5603e4a438b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.812+0000 7f548bfff640 1 -- 172.21.15.31:0/1678849152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e4ab5a00 con 0x5603e4a2c940 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.812+0000 7f5492ffd640 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2bf90 0x5603e4a438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.812+0000 7f548bfff640 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a43df0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f547c0cfdb0 tx=0x7f547c0097a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbc4d8e041e50450 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.813+0000 7f547a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.813+0000 7f547a7fc640 1 -- 172.21.15.31:0/1678849152 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547c09b7b0 con 0x5603e4a2c940 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.813+0000 7f547a7fc640 1 -- 172.21.15.31:0/1678849152 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7f547c0050f0 con 0x5603e4a2c940 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.813+0000 7f549c573740 1 -- 172.21.15.31:0/1678849152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f547c09c370 con 0x5603e4a2c940 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.813+0000 7f547a7fc640 1 -- 172.21.15.31:0/1678849152 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f547c005330 con 0x5603e4a2c940 2024-04-25T02:16:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.813+0000 7f549c573740 10 client.27455 fetch_fsmap learned FSMap version 50 2024-04-25T02:16:21.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.813+0000 7f549c573740 1 -- 172.21.15.31:0/1678849152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5603e4ab6cd0 con 0x5603e4a2c940 2024-04-25T02:16:21.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.813+0000 7f547a7fc640 1 -- 172.21.15.31:0/1678849152 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f547c09c370 con 0x5603e4a2c940 2024-04-25T02:16:21.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.814+0000 7f547a7fc640 1 -- 172.21.15.31:0/1678849152 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f547c097a50 con 0x5603e4a2c940 2024-04-25T02:16:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.814+0000 7f549c573740 10 client.27455 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:16:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.814+0000 7f549c573740 10 client.27455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:16:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.814+0000 7f549c573740 20 client.27455 populate_metadata read hostname 'smithi031' 2024-04-25T02:16:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.814+0000 7f549c573740 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5603e4ab70b0 0x5603e4ab94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.814+0000 7f549c573740 4 client.27455 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:16:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.814+0000 7f549c573740 1 -- 172.21.15.31:0/1678849152 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5603e4998540 con 0x5603e4ab70b0 2024-04-25T02:16:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.814+0000 7f5492ffd640 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5603e4ab70b0 0x5603e4ab94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.815+0000 7f5492ffd640 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5603e4ab70b0 0x5603e4ab94a0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdcea53d3c4d5e3a server_cookie=6a0b819f0aa989fb in_seq=0 out_seq=0 2024-04-25T02:16:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.815+0000 7f547a7fc640 10 client.27455 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.880+0000 7f547a7fc640 1 -- 172.21.15.31:0/1678849152 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5603e4998540 con 0x5603e4ab70b0 2024-04-25T02:16:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.880+0000 7f547a7fc640 10 client.27455 handle_command_reply: tid=0 2024-04-25T02:16:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.880+0000 7f549c573740 1 client.27455 shutdown 2024-04-25T02:16:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.881+0000 7f549c573740 1 -- 172.21.15.31:0/1678849152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 msgr2=0x5603e4a43df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.881+0000 7f549c573740 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a43df0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f547c0cfdb0 tx=0x7f547c0097a0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.881+0000 7f549c573740 1 -- 172.21.15.31:0/1678849152 shutdown_connections 2024-04-25T02:16:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.881+0000 7f549c573740 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5603e4ab70b0 0x5603e4ab94a0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.881+0000 7f549c573740 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5603e4a3fa40 0x5603e4a44330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.881+0000 7f549c573740 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5603e4a2c940 0x5603e4a43df0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.881+0000 7f549c573740 1 --2- 172.21.15.31:0/1678849152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603e4a2bf90 0x5603e4a438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.881+0000 7f549c573740 1 -- 172.21.15.31:0/1678849152 >> 172.21.15.31:0/1678849152 conn(0x5603e4a28560 msgr2=0x5603e4a275c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.881+0000 7f547a7fc640 0 client.27455 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.881+0000 7f549c573740 1 -- 172.21.15.31:0/1678849152 shutdown_connections 2024-04-25T02:16:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.881+0000 7f549c573740 1 -- 172.21.15.31:0/1678849152 wait complete. 2024-04-25T02:16:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.882+0000 7f549c573740 20 client.27455 trim_cache size 0 max 16384 2024-04-25T02:16:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:16:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:16:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:16:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 -- 172.21.15.31:0/1012865911 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54800a2e20 msgr2=0x7f54800a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54800a2e20 0x7f54800a52e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f548c020000 tx=0x7f548c021000 comp rx=0 tx=0).stop 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 -- 172.21.15.31:0/1012865911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5494104310 msgr2=0x7f549419a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5494104310 0x7f549419a640 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f547c00e9f0 tx=0x7f547c00edf0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 -- 172.21.15.31:0/1012865911 shutdown_connections 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f54800a2e20 0x7f54800a52e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5494109960 0x7f549419e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5494104310 0x7f549419a640 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 --2- 172.21.15.31:0/1012865911 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5494103950 0x7f549419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 -- 172.21.15.31:0/1012865911 >> 172.21.15.31:0/1012865911 conn(0x7f54940ff400 msgr2=0x7f54940ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 -- 172.21.15.31:0/1012865911 shutdown_connections 2024-04-25T02:16:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:21.883+0000 7f5499789640 1 -- 172.21.15.31:0/1012865911 wait complete. 2024-04-25T02:16:22.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 83132 8.84 MB/sec execute 386 sec latency 94.491 ms 2024-04-25T02:16:22.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:22 smithi040 ceph-mon[32418]: pgmap v690: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.5 MiB/s wr, 365 op/s 2024-04-25T02:16:22.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:22 smithi134 ceph-mon[33098]: pgmap v690: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.5 MiB/s wr, 365 op/s 2024-04-25T02:16:22.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:22 smithi031 ceph-mon[29204]: pgmap v690: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.5 MiB/s wr, 365 op/s 2024-04-25T02:16:23.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 83266 8.82 MB/sec execute 387 sec latency 55.895 ms 2024-04-25T02:16:24.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 83429 8.83 MB/sec execute 388 sec latency 88.075 ms 2024-04-25T02:16:24.664 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:24 smithi040 ceph-mon[32418]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:16:24.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:24 smithi134 ceph-mon[33098]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:16:24.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:24 smithi031 ceph-mon[29204]: pgmap v691: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 228 op/s 2024-04-25T02:16:25.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 83589 8.83 MB/sec execute 389 sec latency 168.923 ms 2024-04-25T02:16:26.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 83772 8.82 MB/sec execute 390 sec latency 88.361 ms 2024-04-25T02:16:26.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:26 smithi040 ceph-mon[32418]: pgmap v692: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.7 MiB/s wr, 343 op/s 2024-04-25T02:16:26.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:26 smithi134 ceph-mon[33098]: pgmap v692: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.7 MiB/s wr, 343 op/s 2024-04-25T02:16:26.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:26 smithi031 ceph-mon[29204]: pgmap v692: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.7 MiB/s wr, 343 op/s 2024-04-25T02:16:27.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 83917 8.82 MB/sec execute 391 sec latency 71.308 ms 2024-04-25T02:16:28.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 84088 8.82 MB/sec execute 392 sec latency 67.566 ms 2024-04-25T02:16:28.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:28 smithi040 ceph-mon[32418]: pgmap v693: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 228 op/s 2024-04-25T02:16:28.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:28 smithi134 ceph-mon[33098]: pgmap v693: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 228 op/s 2024-04-25T02:16:28.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:28 smithi031 ceph-mon[29204]: pgmap v693: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 228 op/s 2024-04-25T02:16:29.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 84246 8.83 MB/sec execute 393 sec latency 80.611 ms 2024-04-25T02:16:30.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 84395 8.83 MB/sec execute 394 sec latency 89.948 ms 2024-04-25T02:16:30.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:30 smithi040 ceph-mon[32418]: pgmap v694: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 228 op/s 2024-04-25T02:16:30.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:30 smithi134 ceph-mon[33098]: pgmap v694: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 228 op/s 2024-04-25T02:16:30.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:30 smithi031 ceph-mon[29204]: pgmap v694: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.5 MiB/s wr, 228 op/s 2024-04-25T02:16:31.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 84552 8.85 MB/sec execute 395 sec latency 106.229 ms 2024-04-25T02:16:32.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 84693 8.83 MB/sec execute 396 sec latency 140.334 ms 2024-04-25T02:16:32.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:32 smithi040 ceph-mon[32418]: pgmap v695: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.2 MiB/s wr, 347 op/s 2024-04-25T02:16:32.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:32 smithi134 ceph-mon[33098]: pgmap v695: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.2 MiB/s wr, 347 op/s 2024-04-25T02:16:32.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:32 smithi031 ceph-mon[29204]: pgmap v695: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.2 MiB/s wr, 347 op/s 2024-04-25T02:16:33.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 84821 8.82 MB/sec execute 397 sec latency 140.763 ms 2024-04-25T02:16:34.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 84966 8.80 MB/sec execute 398 sec latency 73.304 ms 2024-04-25T02:16:34.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:34 smithi040 ceph-mon[32418]: pgmap v696: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 234 op/s 2024-04-25T02:16:34.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:34 smithi134 ceph-mon[33098]: pgmap v696: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 234 op/s 2024-04-25T02:16:34.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:34 smithi031 ceph-mon[29204]: pgmap v696: 97 pgs: 97 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 234 op/s 2024-04-25T02:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.021+0000 7f15a0990640 1 -- 172.21.15.31:0/181961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f159c100fe0 msgr2=0x7f159c10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.021+0000 7f15a0990640 1 --2- 172.21.15.31:0/181961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f159c100fe0 0x7f159c10d4f0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f1594009cd0 tx=0x7f159402a8e0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.021+0000 7f15a0990640 1 -- 172.21.15.31:0/181961 shutdown_connections 2024-04-25T02:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.021+0000 7f15a0990640 1 --2- 172.21.15.31:0/181961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f159c100fe0 0x7f159c10d4f0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.021+0000 7f15a0990640 1 --2- 172.21.15.31:0/181961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f159c100620 0x7f159c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.021+0000 7f15a0990640 1 --2- 172.21.15.31:0/181961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f159c106630 0x7f159c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.021+0000 7f15a0990640 1 -- 172.21.15.31:0/181961 >> 172.21.15.31:0/181961 conn(0x7f159c0fc110 msgr2=0x7f159c0fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.021+0000 7f15a0990640 1 -- 172.21.15.31:0/181961 shutdown_connections 2024-04-25T02:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.021+0000 7f15a0990640 1 -- 172.21.15.31:0/181961 wait complete. 2024-04-25T02:16:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.022+0000 7f15a0990640 1 Processor -- start 2024-04-25T02:16:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.022+0000 7f15a0990640 1 -- start start 2024-04-25T02:16:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f15a0990640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f159c100620 0x7f159c197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f15a0990640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f159c100fe0 0x7f159c198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f15a0990640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f159c106630 0x7f159c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f15a0990640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f159c111980 con 0x7f159c100fe0 2024-04-25T02:16:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f15a0990640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f159c111800 con 0x7f159c100620 2024-04-25T02:16:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f15a0990640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f159c111b00 con 0x7f159c106630 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f159a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f159c100620 0x7f159c197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f1599d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f159c100fe0 0x7f159c198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f159a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f159c100620 0x7f159c197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49352/0 (socket says 172.21.15.31:49352) 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f159a59c640 1 -- 172.21.15.31:0/2028140708 learned_addr learned my addr 172.21.15.31:0/2028140708 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f159ad9d640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f159c106630 0x7f159c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f1599d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f159c100fe0 0x7f159c198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45258/0 (socket says 172.21.15.31:45258) 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f1599d9b640 1 -- 172.21.15.31:0/2028140708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f159c106630 msgr2=0x7f159c19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f1599d9b640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f159c106630 0x7f159c19c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f1599d9b640 1 -- 172.21.15.31:0/2028140708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f159c100620 msgr2=0x7f159c197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f1599d9b640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f159c100620 0x7f159c197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.023+0000 7f1599d9b640 1 -- 172.21.15.31:0/2028140708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159c19ce30 con 0x7f159c100fe0 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.024+0000 7f1599d9b640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f159c100fe0 0x7f159c198380 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f159000e9f0 tx=0x7f159000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.024+0000 7f158b7fe640 1 -- 172.21.15.31:0/2028140708 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159000ccc0 con 0x7f159c100fe0 2024-04-25T02:16:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.024+0000 7f15a0990640 1 -- 172.21.15.31:0/2028140708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f159c19d120 con 0x7f159c100fe0 2024-04-25T02:16:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.024+0000 7f158b7fe640 1 -- 172.21.15.31:0/2028140708 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15900108e0 con 0x7f159c100fe0 2024-04-25T02:16:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.024+0000 7f158b7fe640 1 -- 172.21.15.31:0/2028140708 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15900188e0 con 0x7f159c100fe0 2024-04-25T02:16:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.024+0000 7f15a0990640 1 -- 172.21.15.31:0/2028140708 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f159c10e0e0 con 0x7f159c100fe0 2024-04-25T02:16:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.027+0000 7f15a377e740 1 Processor -- start 2024-04-25T02:16:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f15a377e740 1 -- start start 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f15a377e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270e240 0x556f7270e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f15a377e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270eb60 0x556f72718cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f15a377e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f72719230 0x556f7271b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f15a377e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556f72672cb0 con 0x556f7270eb60 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f15a377e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556f7261b5b0 con 0x556f72719230 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f15a377e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556f7266ec70 con 0x556f7270e240 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f158b7fe640 1 -- 172.21.15.31:0/2028140708 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1590018a80 con 0x7f159c100fe0 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f159a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270eb60 0x556f72718cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f159ad9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270e240 0x556f7270e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f159a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270eb60 0x556f72718cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45260/0 (socket says 172.21.15.31:45260) 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f159a59c640 1 -- 172.21.15.31:0/518773295 learned_addr learned my addr 172.21.15.31:0/518773295 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.028+0000 7f1599d9b640 1 --2- 172.21.15.31:0/518773295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f72719230 0x556f7271b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.029+0000 7f159a59c640 1 -- 172.21.15.31:0/518773295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270e240 msgr2=0x556f7270e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:35.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.029+0000 7f159a59c640 1 --2- 172.21.15.31:0/518773295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270e240 0x556f7270e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.029+0000 7f159a59c640 1 -- 172.21.15.31:0/518773295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f72719230 msgr2=0x556f7271b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:35.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.029+0000 7f159a59c640 1 --2- 172.21.15.31:0/518773295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f72719230 0x556f7271b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.029+0000 7f159a59c640 1 -- 172.21.15.31:0/518773295 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f72387210 con 0x556f7270eb60 2024-04-25T02:16:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.029+0000 7f159a59c640 1 --2- 172.21.15.31:0/518773295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270eb60 0x556f72718cf0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f158c009d40 tx=0x7f158c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea07c681a0de130 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.029+0000 7f158b7fe640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f15840a2e20 0x7f15840a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.030+0000 7f158b7fe640 1 -- 172.21.15.31:0/2028140708 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7f15900cf7c0 con 0x7f159c100fe0 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.030+0000 7f1588ff9640 1 -- 172.21.15.31:0/518773295 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c004220 con 0x556f7270eb60 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.030+0000 7f159ad9d640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f15840a2e20 0x7f15840a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.030+0000 7f1588ff9640 1 -- 172.21.15.31:0/518773295 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f158c0043c0 con 0x556f7270eb60 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.030+0000 7f1588ff9640 1 -- 172.21.15.31:0/518773295 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c004e50 con 0x556f7270eb60 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.031+0000 7f15a377e740 1 -- 172.21.15.31:0/518773295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270eb60 msgr2=0x556f72718cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.031+0000 7f159ad9d640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f15840a2e20 0x7f15840a52e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1594005e10 tx=0x7f159403d040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.031+0000 7f15a377e740 1 --2- 172.21.15.31:0/518773295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270eb60 0x556f72718cf0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f158c009d40 tx=0x7f158c011810 comp rx=0 tx=0).stop 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.031+0000 7f15a377e740 1 -- 172.21.15.31:0/518773295 shutdown_connections 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.031+0000 7f15a377e740 1 --2- 172.21.15.31:0/518773295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f72719230 0x556f7271b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.031+0000 7f15a377e740 1 --2- 172.21.15.31:0/518773295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270eb60 0x556f72718cf0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.031+0000 7f15a377e740 1 --2- 172.21.15.31:0/518773295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270e240 0x556f7270e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.031+0000 7f15a377e740 1 -- 172.21.15.31:0/518773295 >> 172.21.15.31:0/518773295 conn(0x556f7270a560 msgr2=0x556f7270a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.031+0000 7f15a377e740 1 -- 172.21.15.31:0/518773295 shutdown_connections 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.031+0000 7f15a377e740 1 -- 172.21.15.31:0/518773295 wait complete. 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f15a377e740 1 -- start start 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f15a377e740 1 Processor -- start 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f15a377e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 0x556f727161c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f15a377e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f72716700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f15a377e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 0x556f72716c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f159a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 0x556f727161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f159a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 0x556f727161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49374/0 (socket says 172.21.15.31:49374) 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f15a377e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556f7271c070 con 0x556f72719230 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f159a59c640 1 -- 172.21.15.31:0/388741977 learned_addr learned my addr 172.21.15.31:0/388741977 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f15a377e740 1 -- 172.21.15.31:0/388741977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556f7262eb50 con 0x556f7270e240 2024-04-25T02:16:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f15a377e740 1 -- 172.21.15.31:0/388741977 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556f7271c1f0 con 0x556f7270eb60 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f159ad9d640 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 0x556f72716c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f15a377e740 10 client.0 _ll_register_callbacks cb 0x556f72669ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.032+0000 7f1599d9b640 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f72716700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f159a59c640 1 -- 172.21.15.31:0/388741977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 msgr2=0x556f72716700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f159a59c640 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f72716700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f159a59c640 1 -- 172.21.15.31:0/388741977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 msgr2=0x556f72716c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f159a59c640 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 0x556f72716c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f159a59c640 1 -- 172.21.15.31:0/388741977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f72453620 con 0x556f7270e240 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f159ad9d640 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 0x556f72716c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f1599d9b640 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f72716700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f159a59c640 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 0x556f727161c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f158c019ce0 tx=0x7f158c019d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=172f638673a0f1d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f1581ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:16:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f1581ffb640 1 -- 172.21.15.31:0/388741977 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c028020 con 0x556f7270e240 2024-04-25T02:16:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f1581ffb640 1 -- 172.21.15.31:0/388741977 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7f158c024480 con 0x556f7270e240 2024-04-25T02:16:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f15a377e740 1 -- 172.21.15.31:0/388741977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f158c0043c0 con 0x556f7270e240 2024-04-25T02:16:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f1581ffb640 1 -- 172.21.15.31:0/388741977 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f158c033050 con 0x556f7270e240 2024-04-25T02:16:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f15a377e740 10 client.27467 fetch_fsmap learned FSMap version 50 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.033+0000 7f15a377e740 1 -- 172.21.15.31:0/388741977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x556f72728070 con 0x556f7270e240 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.034+0000 7f1581ffb640 1 -- 172.21.15.31:0/388741977 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f158c0043c0 con 0x556f7270e240 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.034+0000 7f1581ffb640 1 -- 172.21.15.31:0/388741977 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f158c023670 con 0x556f7270e240 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.034+0000 7f15a377e740 10 client.27467 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.034+0000 7f15a377e740 10 client.27467 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.034+0000 7f15a377e740 20 client.27467 populate_metadata read hostname 'smithi031' 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.034+0000 7f15a377e740 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f72729530 0x556f7272b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.034+0000 7f15a377e740 4 client.27467 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.034+0000 7f15a377e740 1 -- 172.21.15.31:0/388741977 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f726801f0 con 0x556f72729530 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.034+0000 7f1599d9b640 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f72729530 0x556f7272b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.035+0000 7f1599d9b640 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f72729530 0x556f7272b920 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b5a10e0dead57ee server_cookie=e96cb0be97e05423 in_seq=0 out_seq=0 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.035+0000 7f1581ffb640 10 client.27467 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.036+0000 7f1581ffb640 1 -- 172.21.15.31:0/388741977 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556f726801f0 con 0x556f72729530 2024-04-25T02:16:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.036+0000 7f1581ffb640 10 client.27467 handle_command_reply: tid=0 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.036+0000 7f15a377e740 1 client.27467 shutdown 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f15a377e740 1 -- 172.21.15.31:0/388741977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 msgr2=0x556f727161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f15a377e740 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 0x556f727161c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f158c019ce0 tx=0x7f158c019d10 comp rx=0 tx=0).stop 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f15a377e740 1 -- 172.21.15.31:0/388741977 shutdown_connections 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f15a377e740 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f72729530 0x556f7272b920 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f1581ffb640 0 client.27467 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f15a377e740 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 0x556f72716c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f15a377e740 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f72716700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f15a377e740 1 --2- 172.21.15.31:0/388741977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 0x556f727161c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f15a377e740 1 -- 172.21.15.31:0/388741977 >> 172.21.15.31:0/388741977 conn(0x556f7270a560 msgr2=0x556f72710260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f15a377e740 1 -- 172.21.15.31:0/388741977 shutdown_connections 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f15a377e740 1 -- 172.21.15.31:0/388741977 wait complete. 2024-04-25T02:16:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.037+0000 7f15a377e740 20 client.27467 trim_cache size 0 max 16384 2024-04-25T02:16:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.062+0000 7f15a377e740 1 Processor -- start 2024-04-25T02:16:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.062+0000 7f15a377e740 1 -- start start 2024-04-25T02:16:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.062+0000 7f15a377e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270e240 0x556f727188f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.062+0000 7f15a377e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f72724f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.062+0000 7f15a377e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f72719230 0x556f727310f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.062+0000 7f15a377e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1af550 con 0x556f7270e240 2024-04-25T02:16:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f15a377e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1b1090 con 0x556f72719230 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f15a377e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1afc20 con 0x556f7270eb60 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f159a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270e240 0x556f727188f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f1599d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f72724f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f159a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270e240 0x556f727188f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45274/0 (socket says 172.21.15.31:45274) 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f159ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f72719230 0x556f727310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f159a59c640 1 -- 172.21.15.31:0/2680461738 learned_addr learned my addr 172.21.15.31:0/2680461738 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f159a59c640 1 -- 172.21.15.31:0/2680461738 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 msgr2=0x556f72724f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f159a59c640 1 --2- 172.21.15.31:0/2680461738 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f72724f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f159a59c640 1 -- 172.21.15.31:0/2680461738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f72719230 msgr2=0x556f727310f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f159a59c640 1 --2- 172.21.15.31:0/2680461738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f72719230 0x556f727310f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f159a59c640 1 -- 172.21.15.31:0/2680461738 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f72714710 con 0x556f7270e240 2024-04-25T02:16:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.063+0000 7f159a59c640 1 --2- 172.21.15.31:0/2680461738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270e240 0x556f727188f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f158c000c00 tx=0x7f158c011d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff6cf9ae997b99d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.064+0000 7f1583fff640 1 -- 172.21.15.31:0/2680461738 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c028070 con 0x556f7270e240 2024-04-25T02:16:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.064+0000 7f1583fff640 1 -- 172.21.15.31:0/2680461738 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f158c01e040 con 0x556f7270e240 2024-04-25T02:16:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.064+0000 7f1583fff640 1 -- 172.21.15.31:0/2680461738 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c01d070 con 0x556f7270e240 2024-04-25T02:16:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.064+0000 7f15a377e740 1 -- 172.21.15.31:0/2680461738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270e240 msgr2=0x556f727188f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.064+0000 7f15a377e740 1 --2- 172.21.15.31:0/2680461738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270e240 0x556f727188f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f158c000c00 tx=0x7f158c011d50 comp rx=0 tx=0).stop 2024-04-25T02:16:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.065+0000 7f15a377e740 1 -- 172.21.15.31:0/2680461738 shutdown_connections 2024-04-25T02:16:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.065+0000 7f15a377e740 1 --2- 172.21.15.31:0/2680461738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f72719230 0x556f727310f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.065+0000 7f15a377e740 1 --2- 172.21.15.31:0/2680461738 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f72724f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.065+0000 7f15a377e740 1 --2- 172.21.15.31:0/2680461738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f7270e240 0x556f727188f0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.065+0000 7f15a377e740 1 -- 172.21.15.31:0/2680461738 >> 172.21.15.31:0/2680461738 conn(0x556f7270a560 msgr2=0x556f7270a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.065+0000 7f15a377e740 1 -- 172.21.15.31:0/2680461738 shutdown_connections 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.065+0000 7f15a377e740 1 -- 172.21.15.31:0/2680461738 wait complete. 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.065+0000 7f15a377e740 1 -- start start 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.065+0000 7f15a377e740 1 Processor -- start 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f15a377e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 0x556f72721840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f15a377e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f7271cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f15a377e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 0x556f7271d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f15a377e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1b09c0 con 0x556f72719230 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f15a377e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1b02f0 con 0x556f7270e240 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f15a377e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f159c1ad550 con 0x556f7270eb60 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f15a377e740 10 client.0 _ll_register_callbacks cb 0x7f158c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f1599d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f7271cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f159a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 0x556f72721840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f159ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 0x556f7271d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f1599d9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f7271cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:53160/0 (socket says 172.21.15.31:53160) 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f159a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 0x556f72721840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:49388/0 (socket says 172.21.15.31:49388) 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f1599d9b640 1 -- 172.21.15.31:0/1421071739 learned_addr learned my addr 172.21.15.31:0/1421071739 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f1599d9b640 1 -- 172.21.15.31:0/1421071739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 msgr2=0x556f72721840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f1599d9b640 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 0x556f72721840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f1599d9b640 1 -- 172.21.15.31:0/1421071739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 msgr2=0x556f7271d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f1599d9b640 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 0x556f7271d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f1599d9b640 1 -- 172.21.15.31:0/1421071739 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f7271d8d0 con 0x556f7270eb60 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.066+0000 7f159ad9d640 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 0x556f7271d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.067+0000 7f1599d9b640 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f7271cbb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f15900981b0 tx=0x7f159009b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fae896870c466b87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.067+0000 7f1581ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.067+0000 7f1581ffb640 1 -- 172.21.15.31:0/1421071739 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15900a0050 con 0x556f7270eb60 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.067+0000 7f1581ffb640 1 -- 172.21.15.31:0/1421071739 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7f1590010a90 con 0x556f7270eb60 2024-04-25T02:16:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.067+0000 7f15a377e740 1 -- 172.21.15.31:0/1421071739 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f158c01e040 con 0x556f7270eb60 2024-04-25T02:16:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.067+0000 7f1581ffb640 1 -- 172.21.15.31:0/1421071739 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f15900a4910 con 0x556f7270eb60 2024-04-25T02:16:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.067+0000 7f15a377e740 10 client.27244 fetch_fsmap learned FSMap version 50 2024-04-25T02:16:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.067+0000 7f15a377e740 1 -- 172.21.15.31:0/1421071739 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x556f7278f980 con 0x556f7270eb60 2024-04-25T02:16:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.067+0000 7f1581ffb640 1 -- 172.21.15.31:0/1421071739 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f158c01e040 con 0x556f7270eb60 2024-04-25T02:16:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.067+0000 7f1581ffb640 1 -- 172.21.15.31:0/1421071739 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f15900cfb70 con 0x556f7270eb60 2024-04-25T02:16:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.068+0000 7f15a377e740 10 client.27244 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:16:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.068+0000 7f15a377e740 10 client.27244 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:16:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.068+0000 7f15a377e740 20 client.27244 populate_metadata read hostname 'smithi031' 2024-04-25T02:16:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.068+0000 7f15a377e740 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f727900b0 0x556f727924d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.068+0000 7f15a377e740 4 client.27244 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:16:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.068+0000 7f15a377e740 1 -- 172.21.15.31:0/1421071739 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556f7267a540 con 0x556f727900b0 2024-04-25T02:16:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.068+0000 7f159a59c640 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f727900b0 0x556f727924d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.068+0000 7f159a59c640 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f727900b0 0x556f727924d0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a439060526930216 server_cookie=3061ff51a1fb36fc in_seq=0 out_seq=0 2024-04-25T02:16:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.068+0000 7f1581ffb640 10 client.27244 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:35.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 85100 8.79 MB/sec execute 399 sec latency 197.043 ms 2024-04-25T02:16:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f1581ffb640 1 -- 172.21.15.31:0/1421071739 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556f7267a540 con 0x556f727900b0 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f1581ffb640 10 client.27244 handle_command_reply: tid=0 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f15a377e740 1 client.27244 shutdown 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f15a377e740 1 -- 172.21.15.31:0/1421071739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 msgr2=0x556f7271cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f15a377e740 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f7271cbb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f15900981b0 tx=0x7f159009b9c0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f15a377e740 1 -- 172.21.15.31:0/1421071739 shutdown_connections 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f15a377e740 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556f727900b0 0x556f727924d0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f15a377e740 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f72719230 0x556f7271d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f15a377e740 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556f7270eb60 0x556f7271cbb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f15a377e740 1 --2- 172.21.15.31:0/1421071739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f7270e240 0x556f72721840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f15a377e740 1 -- 172.21.15.31:0/1421071739 >> 172.21.15.31:0/1421071739 conn(0x556f7270a560 msgr2=0x556f7270f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f1581ffb640 0 client.27244 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f15a377e740 1 -- 172.21.15.31:0/1421071739 shutdown_connections 2024-04-25T02:16:35.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.132+0000 7f15a377e740 1 -- 172.21.15.31:0/1421071739 wait complete. 2024-04-25T02:16:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.133+0000 7f15a377e740 20 client.27244 trim_cache size 0 max 16384 2024-04-25T02:16:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:16:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:16:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:16:35.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.133+0000 7f15a0990640 1 -- 172.21.15.31:0/2028140708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f15840a2e20 msgr2=0x7f15840a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.133+0000 7f15a0990640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f15840a2e20 0x7f15840a52e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1594005e10 tx=0x7f159403d040 comp rx=0 tx=0).stop 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.133+0000 7f15a0990640 1 -- 172.21.15.31:0/2028140708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f159c100fe0 msgr2=0x7f159c198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.133+0000 7f15a0990640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f159c100fe0 0x7f159c198380 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f159000e9f0 tx=0x7f159000eeb0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.134+0000 7f15a0990640 1 -- 172.21.15.31:0/2028140708 shutdown_connections 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.134+0000 7f15a0990640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f15840a2e20 0x7f15840a52e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.134+0000 7f15a0990640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f159c106630 0x7f159c19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.134+0000 7f15a0990640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f159c100fe0 0x7f159c198380 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.134+0000 7f15a0990640 1 --2- 172.21.15.31:0/2028140708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f159c100620 0x7f159c197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.134+0000 7f15a0990640 1 -- 172.21.15.31:0/2028140708 >> 172.21.15.31:0/2028140708 conn(0x7f159c0fc110 msgr2=0x7f159c0fc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.134+0000 7f15a0990640 1 -- 172.21.15.31:0/2028140708 shutdown_connections 2024-04-25T02:16:35.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:35.134+0000 7f15a0990640 1 -- 172.21.15.31:0/2028140708 wait complete. 2024-04-25T02:16:36.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 85254 8.78 MB/sec execute 400 sec latency 82.389 ms 2024-04-25T02:16:36.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:36 smithi040 ceph-mon[32418]: pgmap v697: 97 pgs: 97 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.4 MiB/s wr, 336 op/s 2024-04-25T02:16:36.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:36 smithi134 ceph-mon[33098]: pgmap v697: 97 pgs: 97 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.4 MiB/s wr, 336 op/s 2024-04-25T02:16:36.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:36 smithi031 ceph-mon[29204]: pgmap v697: 97 pgs: 97 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.4 MiB/s wr, 336 op/s 2024-04-25T02:16:37.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 85420 8.78 MB/sec execute 401 sec latency 116.246 ms 2024-04-25T02:16:38.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 85630 8.81 MB/sec execute 402 sec latency 55.494 ms 2024-04-25T02:16:38.543 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:38 smithi031 ceph-mon[29204]: pgmap v698: 97 pgs: 97 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 MiB/s wr, 221 op/s 2024-04-25T02:16:38.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:38 smithi040 ceph-mon[32418]: pgmap v698: 97 pgs: 97 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 MiB/s wr, 221 op/s 2024-04-25T02:16:38.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:38 smithi134 ceph-mon[33098]: pgmap v698: 97 pgs: 97 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 MiB/s wr, 221 op/s 2024-04-25T02:16:39.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 85833 8.81 MB/sec execute 403 sec latency 51.730 ms 2024-04-25T02:16:40.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 86004 8.82 MB/sec execute 404 sec latency 56.207 ms 2024-04-25T02:16:40.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:40 smithi040 ceph-mon[32418]: pgmap v699: 97 pgs: 97 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 MiB/s wr, 221 op/s 2024-04-25T02:16:40.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:40 smithi134 ceph-mon[33098]: pgmap v699: 97 pgs: 97 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 MiB/s wr, 221 op/s 2024-04-25T02:16:40.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:40 smithi031 ceph-mon[29204]: pgmap v699: 97 pgs: 97 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 MiB/s wr, 221 op/s 2024-04-25T02:16:41.106 INFO:tasks.workunit.client.0.smithi031.stdout: 10 86171 8.82 MB/sec execute 405 sec latency 62.557 ms 2024-04-25T02:16:42.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 86320 8.82 MB/sec execute 406 sec latency 115.579 ms 2024-04-25T02:16:42.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:42 smithi040 ceph-mon[32418]: pgmap v700: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.4 MiB/s wr, 352 op/s 2024-04-25T02:16:42.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:42 smithi134 ceph-mon[33098]: pgmap v700: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.4 MiB/s wr, 352 op/s 2024-04-25T02:16:42.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:42 smithi031 ceph-mon[29204]: pgmap v700: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.4 MiB/s wr, 352 op/s 2024-04-25T02:16:43.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 86462 8.81 MB/sec execute 407 sec latency 148.593 ms 2024-04-25T02:16:44.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 86623 8.82 MB/sec execute 408 sec latency 88.404 ms 2024-04-25T02:16:44.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:44 smithi040 ceph-mon[32418]: pgmap v701: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 234 op/s 2024-04-25T02:16:44.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:44 smithi134 ceph-mon[33098]: pgmap v701: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 234 op/s 2024-04-25T02:16:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:44 smithi031 ceph-mon[29204]: pgmap v701: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.8 MiB/s wr, 234 op/s 2024-04-25T02:16:45.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 86786 8.82 MB/sec execute 409 sec latency 53.354 ms 2024-04-25T02:16:46.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 86958 8.82 MB/sec execute 410 sec latency 52.964 ms 2024-04-25T02:16:46.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:46 smithi040 ceph-mon[32418]: pgmap v702: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.3 MiB/s wr, 340 op/s 2024-04-25T02:16:46.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:46 smithi134 ceph-mon[33098]: pgmap v702: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.3 MiB/s wr, 340 op/s 2024-04-25T02:16:46.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:46 smithi031 ceph-mon[29204]: pgmap v702: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.3 MiB/s wr, 340 op/s 2024-04-25T02:16:47.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 87128 8.82 MB/sec execute 411 sec latency 84.479 ms 2024-04-25T02:16:48.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 87264 8.81 MB/sec execute 412 sec latency 94.963 ms 2024-04-25T02:16:48.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.262+0000 7fe3c3093640 1 -- 172.21.15.31:0/2677890355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3bc101030 msgr2=0x7fe3bc10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:48.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.262+0000 7fe3c3093640 1 --2- 172.21.15.31:0/2677890355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3bc101030 0x7fe3bc10d4f0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009cd0 tx=0x7fe3b402a8e0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.262+0000 7fe3c3093640 1 -- 172.21.15.31:0/2677890355 shutdown_connections 2024-04-25T02:16:48.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.262+0000 7fe3c3093640 1 --2- 172.21.15.31:0/2677890355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3bc101030 0x7fe3bc10d4f0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.262+0000 7fe3c3093640 1 --2- 172.21.15.31:0/2677890355 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3bc100670 0x7fe3bc100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.262+0000 7fe3c3093640 1 --2- 172.21.15.31:0/2677890355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3bc106630 0x7fe3bc106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.262+0000 7fe3c3093640 1 -- 172.21.15.31:0/2677890355 >> 172.21.15.31:0/2677890355 conn(0x7fe3bc0777c0 msgr2=0x7fe3bc077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.262+0000 7fe3c3093640 1 -- 172.21.15.31:0/2677890355 shutdown_connections 2024-04-25T02:16:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.262+0000 7fe3c3093640 1 -- 172.21.15.31:0/2677890355 wait complete. 2024-04-25T02:16:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.262+0000 7fe3c3093640 1 Processor -- start 2024-04-25T02:16:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3c3093640 1 -- start start 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3c3093640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3bc100670 0x7fe3bc19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3c3093640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3bc101030 0x7fe3bc19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3c3093640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3bc106630 0x7fe3bc19e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3c3093640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc10f7e0 con 0x7fe3bc100670 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3c3093640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc10f660 con 0x7fe3bc106630 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3c3093640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc10f960 con 0x7fe3bc101030 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3bbfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3bc101030 0x7fe3bc19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3bbfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3bc101030 0x7fe3bc19a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:50358/0 (socket says 172.21.15.31:50358) 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3bbfff640 1 -- 172.21.15.31:0/144529503 learned_addr learned my addr 172.21.15.31:0/144529503 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3c1630640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3bc106630 0x7fe3bc19e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3c0e2f640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3bc100670 0x7fe3bc19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3bbfff640 1 -- 172.21.15.31:0/144529503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3bc106630 msgr2=0x7fe3bc19e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3bbfff640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3bc106630 0x7fe3bc19e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3bbfff640 1 -- 172.21.15.31:0/144529503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3bc100670 msgr2=0x7fe3bc19a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3bbfff640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3bc100670 0x7fe3bc19a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.263+0000 7fe3bbfff640 1 -- 172.21.15.31:0/144529503 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3bc19f040 con 0x7fe3bc101030 2024-04-25T02:16:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.264+0000 7fe3bbfff640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3bc101030 0x7fe3bc19a590 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80097d0 tx=0x7fe3a8009c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.264+0000 7fe3b9ffb640 1 -- 172.21.15.31:0/144529503 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a800eab0 con 0x7fe3bc101030 2024-04-25T02:16:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.264+0000 7fe3c3093640 1 -- 172.21.15.31:0/144529503 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3bc19f330 con 0x7fe3bc101030 2024-04-25T02:16:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.264+0000 7fe3b9ffb640 1 -- 172.21.15.31:0/144529503 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3a8019070 con 0x7fe3bc101030 2024-04-25T02:16:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.264+0000 7fe3b9ffb640 1 -- 172.21.15.31:0/144529503 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8013360 con 0x7fe3bc101030 2024-04-25T02:16:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.264+0000 7fe3c3093640 1 -- 172.21.15.31:0/144529503 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3bc1a6bc0 con 0x7fe3bc101030 2024-04-25T02:16:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.267+0000 7fe3b9ffb640 1 -- 172.21.15.31:0/144529503 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe3a8013070 con 0x7fe3bc101030 2024-04-25T02:16:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.267+0000 7fe3c5d9f740 1 Processor -- start 2024-04-25T02:16:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3c5d9f740 1 -- start start 2024-04-25T02:16:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 0x556777ec0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0b60 0x556777ecacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ecd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556777e24cb0 con 0x556777ec0240 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556777dcd5b0 con 0x556777ec0b60 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556777e20c70 con 0x556777ecb230 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3c1630640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 0x556777ec0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3c1630640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 0x556777ec0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37422/0 (socket says 172.21.15.31:37422) 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3c1630640 1 -- 172.21.15.31:0/2342161552 learned_addr learned my addr 172.21.15.31:0/2342161552 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3c0e2f640 1 --2- 172.21.15.31:0/2342161552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0b60 0x556777ecacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.268+0000 7fe3bbfff640 1 --2- 172.21.15.31:0/2342161552 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ecd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.269+0000 7fe3b9ffb640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe3ac0a2e20 0x7fe3ac0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.269+0000 7fe3c1630640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe3ac0a2e20 0x7fe3ac0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.269+0000 7fe3c1630640 1 -- 172.21.15.31:0/2342161552 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 msgr2=0x556777ecd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.269+0000 7fe3c1630640 1 --2- 172.21.15.31:0/2342161552 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ecd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.269+0000 7fe3c1630640 1 -- 172.21.15.31:0/2342161552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0b60 msgr2=0x556777ecacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.269+0000 7fe3c1630640 1 --2- 172.21.15.31:0/2342161552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0b60 0x556777ecacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.269+0000 7fe3c1630640 1 -- 172.21.15.31:0/2342161552 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556777b39210 con 0x556777ec0240 2024-04-25T02:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.269+0000 7fe3bbfff640 1 --2- 172.21.15.31:0/2342161552 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ecd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.269+0000 7fe3b9ffb640 1 -- 172.21.15.31:0/144529503 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7fe3a80cef20 con 0x7fe3bc101030 2024-04-25T02:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.270+0000 7fe3c1630640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe3ac0a2e20 0x7fe3ac0a52e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009430 tx=0x7fe3b4049000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.270+0000 7fe3c1630640 1 --2- 172.21.15.31:0/2342161552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 0x556777ec0620 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4005e10 tx=0x7fe3b40318d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff3189341da5037 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.270+0000 7fe3a6ffd640 1 -- 172.21.15.31:0/2342161552 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4045040 con 0x556777ec0240 2024-04-25T02:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.270+0000 7fe3a6ffd640 1 -- 172.21.15.31:0/2342161552 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3b4036040 con 0x556777ec0240 2024-04-25T02:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.270+0000 7fe3a6ffd640 1 -- 172.21.15.31:0/2342161552 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4030800 con 0x556777ec0240 2024-04-25T02:16:48.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.271+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2342161552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 msgr2=0x556777ec0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:48.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.271+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/2342161552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 0x556777ec0620 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4005e10 tx=0x7fe3b40318d0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.271+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2342161552 shutdown_connections 2024-04-25T02:16:48.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.271+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/2342161552 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ecd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.271+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/2342161552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0b60 0x556777ecacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.271+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/2342161552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 0x556777ec0620 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.271+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2342161552 >> 172.21.15.31:0/2342161552 conn(0x556777ebc560 msgr2=0x556777ebc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.271+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2342161552 shutdown_connections 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.271+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2342161552 wait complete. 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.271+0000 7fe3c5d9f740 1 -- start start 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3c5d9f740 1 Processor -- start 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 0x556777ec81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0b60 0x556777ec8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ec8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556777ecdfc0 con 0x556777ec0b60 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556777de0b50 con 0x556777ec0240 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556777ece140 con 0x556777ecb230 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3c5d9f740 10 client.0 _ll_register_callbacks cb 0x556777e1bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3bbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0b60 0x556777ec8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3c1630640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ec8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3bbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0b60 0x556777ec8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37430/0 (socket says 172.21.15.31:37430) 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.272+0000 7fe3bbfff640 1 -- 172.21.15.31:0/1713046078 learned_addr learned my addr 172.21.15.31:0/1713046078 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.273+0000 7fe3c0e2f640 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 0x556777ec81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.273+0000 7fe3c1630640 1 -- 172.21.15.31:0/1713046078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 msgr2=0x556777ec81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.273+0000 7fe3c1630640 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 0x556777ec81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.273+0000 7fe3c1630640 1 -- 172.21.15.31:0/1713046078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0b60 msgr2=0x556777ec8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.273+0000 7fe3c1630640 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0b60 0x556777ec8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.273+0000 7fe3c1630640 1 -- 172.21.15.31:0/1713046078 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556777c05620 con 0x556777ecb230 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.273+0000 7fe3c1630640 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ec8c60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe3b40049d0 tx=0x7fe3b4030ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d3d50fe342cb93f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.273+0000 7fe39bfff640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.273+0000 7fe39bfff640 1 -- 172.21.15.31:0/1713046078 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4058020 con 0x556777ecb230 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.274+0000 7fe39bfff640 1 -- 172.21.15.31:0/1713046078 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7fe3b4003e00 con 0x556777ecb230 2024-04-25T02:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.274+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/1713046078 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3b4036040 con 0x556777ecb230 2024-04-25T02:16:48.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.274+0000 7fe39bfff640 1 -- 172.21.15.31:0/1713046078 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fe3b4063050 con 0x556777ecb230 2024-04-25T02:16:48.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.274+0000 7fe3c5d9f740 10 client.27256 fetch_fsmap learned FSMap version 50 2024-04-25T02:16:48.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.274+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/1713046078 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x556777ece010 con 0x556777ecb230 2024-04-25T02:16:48.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.274+0000 7fe39bfff640 1 -- 172.21.15.31:0/1713046078 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3b4036040 con 0x556777ecb230 2024-04-25T02:16:48.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.275+0000 7fe39bfff640 1 -- 172.21.15.31:0/1713046078 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7fe3b4048040 con 0x556777ecb230 2024-04-25T02:16:48.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.275+0000 7fe3c5d9f740 10 client.27256 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.275+0000 7fe3c5d9f740 10 client.27256 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.275+0000 7fe3c5d9f740 20 client.27256 populate_metadata read hostname 'smithi031' 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.275+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556777edb410 0x556777edd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.275+0000 7fe3c5d9f740 4 client.27256 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.275+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/1713046078 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556777e321f0 con 0x556777edb410 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.275+0000 7fe3c0e2f640 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556777edb410 0x556777edd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.276+0000 7fe39bfff640 10 client.27256 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.276+0000 7fe3c0e2f640 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556777edb410 0x556777edd800 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3534075800cce14f server_cookie=b32547d7dc8d1163 in_seq=0 out_seq=0 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe39bfff640 1 -- 172.21.15.31:0/1713046078 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556777e321f0 con 0x556777edb410 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe39bfff640 10 client.27256 handle_command_reply: tid=0 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe3c5d9f740 1 client.27256 shutdown 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/1713046078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 msgr2=0x556777ec8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ec8c60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe3b40049d0 tx=0x7fe3b4030ac0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/1713046078 shutdown_connections 2024-04-25T02:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556777edb410 0x556777edd800 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ec8c60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0b60 0x556777ec8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/1713046078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 0x556777ec81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/1713046078 >> 172.21.15.31:0/1713046078 conn(0x556777ebc560 msgr2=0x556777ec2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.277+0000 7fe39bfff640 0 client.27256 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.278+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/1713046078 shutdown_connections 2024-04-25T02:16:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.278+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/1713046078 wait complete. 2024-04-25T02:16:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.278+0000 7fe3c5d9f740 20 client.27256 trim_cache size 0 max 16384 2024-04-25T02:16:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c5d9f740 1 Processor -- start 2024-04-25T02:16:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c5d9f740 1 -- start start 2024-04-25T02:16:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 0x556777ed6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0b60 0x556777ec8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ee2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc1af470 con 0x556777ec0240 2024-04-25T02:16:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc1b0fb0 con 0x556777ec0b60 2024-04-25T02:16:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc1afb40 con 0x556777ecb230 2024-04-25T02:16:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c1630640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ee2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c0e2f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 0x556777ed6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c1630640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ee2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:50398/0 (socket says 172.21.15.31:50398) 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c1630640 1 -- 172.21.15.31:0/838526819 learned_addr learned my addr 172.21.15.31:0/838526819 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3c0e2f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 0x556777ed6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37436/0 (socket says 172.21.15.31:37436) 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.303+0000 7fe3bbfff640 1 --2- 172.21.15.31:0/838526819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0b60 0x556777ec8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.304+0000 7fe3c1630640 1 -- 172.21.15.31:0/838526819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0b60 msgr2=0x556777ec8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.304+0000 7fe3c1630640 1 --2- 172.21.15.31:0/838526819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0b60 0x556777ec8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.304+0000 7fe3c1630640 1 -- 172.21.15.31:0/838526819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 msgr2=0x556777ed6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.304+0000 7fe3c1630640 1 --2- 172.21.15.31:0/838526819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 0x556777ed6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.304+0000 7fe3c1630640 1 -- 172.21.15.31:0/838526819 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556777eddcd0 con 0x556777ecb230 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.304+0000 7fe3c1630640 1 --2- 172.21.15.31:0/838526819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ee2f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4002230 tx=0x7fe3b4054b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=341ec625504b301a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.304+0000 7fe3a67fc640 1 -- 172.21.15.31:0/838526819 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4058040 con 0x556777ecb230 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.304+0000 7fe3a67fc640 1 -- 172.21.15.31:0/838526819 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3b4009b20 con 0x556777ecb230 2024-04-25T02:16:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.304+0000 7fe3a67fc640 1 -- 172.21.15.31:0/838526819 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4045070 con 0x556777ecb230 2024-04-25T02:16:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.305+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/838526819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 msgr2=0x556777ee2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.305+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/838526819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ee2f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4002230 tx=0x7fe3b4054b40 comp rx=0 tx=0).stop 2024-04-25T02:16:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.305+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/838526819 shutdown_connections 2024-04-25T02:16:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.305+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/838526819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ecb230 0x556777ee2f70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.305+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/838526819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0b60 0x556777ec8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.305+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/838526819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ec0240 0x556777ed6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.305+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/838526819 >> 172.21.15.31:0/838526819 conn(0x556777ebc560 msgr2=0x556777ebb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.306+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/838526819 shutdown_connections 2024-04-25T02:16:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.306+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/838526819 wait complete. 2024-04-25T02:16:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.306+0000 7fe3c5d9f740 1 -- start start 2024-04-25T02:16:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.306+0000 7fe3c5d9f740 1 Processor -- start 2024-04-25T02:16:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 0x556777ed3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ec0b60 0x556777ec49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c5d9f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ecb230 0x556777ec4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc1b08e0 con 0x556777ecb230 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc1b0210 con 0x556777ec0240 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c5d9f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe3bc1ad470 con 0x556777ec0b60 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c5d9f740 10 client.0 _ll_register_callbacks cb 0x7fe3b0009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c1630640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ecb230 0x556777ec4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c0e2f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 0x556777ed3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c1630640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ecb230 0x556777ec4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37442/0 (socket says 172.21.15.31:37442) 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c0e2f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 0x556777ed3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:52148/0 (socket says 172.21.15.31:52148) 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3bbfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ec0b60 0x556777ec49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.307+0000 7fe3c1630640 1 -- 172.21.15.31:0/2089190955 learned_addr learned my addr 172.21.15.31:0/2089190955 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe3c1630640 1 -- 172.21.15.31:0/2089190955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ec0b60 msgr2=0x556777ec49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe3c1630640 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ec0b60 0x556777ec49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe3c1630640 1 -- 172.21.15.31:0/2089190955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 msgr2=0x556777ed3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:16:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe3c1630640 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 0x556777ed3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe3c1630640 1 -- 172.21.15.31:0/2089190955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556777ec5430 con 0x556777ecb230 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe3c0e2f640 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 0x556777ed3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe3c1630640 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ecb230 0x556777ec4ef0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4054b10 tx=0x7fe3b4004620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5264f26d7aacf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe39bfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe39bfff640 1 -- 172.21.15.31:0/2089190955 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4045070 con 0x556777ecb230 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2089190955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3b4009b20 con 0x556777ecb230 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe39bfff640 1 -- 172.21.15.31:0/2089190955 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7fe3b4003e00 con 0x556777ecb230 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe39bfff640 1 -- 172.21.15.31:0/2089190955 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fe3b4058070 con 0x556777ecb230 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe3c5d9f740 10 client.17922 fetch_fsmap learned FSMap version 50 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.308+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2089190955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x556777f49d70 con 0x556777ecb230 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.309+0000 7fe39bfff640 1 -- 172.21.15.31:0/2089190955 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3b4009b20 con 0x556777ecb230 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.309+0000 7fe39bfff640 1 -- 172.21.15.31:0/2089190955 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7fe3b4048040 con 0x556777ecb230 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.309+0000 7fe3c5d9f740 10 client.17922 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.309+0000 7fe3c5d9f740 10 client.17922 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.309+0000 7fe3c5d9f740 20 client.17922 populate_metadata read hostname 'smithi031' 2024-04-25T02:16:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.309+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556777f4a3f0 0x556777f4c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:16:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.309+0000 7fe3c5d9f740 4 client.17922 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:16:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.309+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2089190955 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556777e2c540 con 0x556777f4a3f0 2024-04-25T02:16:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.309+0000 7fe3c0e2f640 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556777f4a3f0 0x556777f4c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:16:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.310+0000 7fe39bfff640 10 client.17922 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.310+0000 7fe3c0e2f640 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556777f4a3f0 0x556777f4c810 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a4e840308417791 server_cookie=c82c0980659c26c in_seq=0 out_seq=0 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.376+0000 7fe39bfff640 1 -- 172.21.15.31:0/2089190955 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556777e2c540 con 0x556777f4a3f0 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.376+0000 7fe39bfff640 10 client.17922 handle_command_reply: tid=0 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.377+0000 7fe3c5d9f740 1 client.17922 shutdown 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.377+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2089190955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ecb230 msgr2=0x556777ec4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.377+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ecb230 0x556777ec4ef0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4054b10 tx=0x7fe3b4004620 comp rx=0 tx=0).stop 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.377+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2089190955 shutdown_connections 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.377+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556777f4a3f0 0x556777f4c810 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.377+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556777ecb230 0x556777ec4ef0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.377+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556777ec0b60 0x556777ec49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.377+0000 7fe3c5d9f740 1 --2- 172.21.15.31:0/2089190955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556777ec0240 0x556777ed3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.377+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2089190955 >> 172.21.15.31:0/2089190955 conn(0x556777ebc560 msgr2=0x556777ec9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.378+0000 7fe39bfff640 0 client.17922 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:16:48.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.378+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2089190955 shutdown_connections 2024-04-25T02:16:48.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.378+0000 7fe3c5d9f740 1 -- 172.21.15.31:0/2089190955 wait complete. 2024-04-25T02:16:48.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.378+0000 7fe3c5d9f740 20 client.17922 trim_cache size 0 max 16384 2024-04-25T02:16:48.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:16:48.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:16:48.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:16:48.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:16:48.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.379+0000 7fe3c3093640 1 -- 172.21.15.31:0/144529503 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe3ac0a2e20 msgr2=0x7fe3ac0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:48.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.379+0000 7fe3c3093640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe3ac0a2e20 0x7fe3ac0a52e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009430 tx=0x7fe3b4049000 comp rx=0 tx=0).stop 2024-04-25T02:16:48.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.379+0000 7fe3c3093640 1 -- 172.21.15.31:0/144529503 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3bc101030 msgr2=0x7fe3bc19a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:16:48.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.379+0000 7fe3c3093640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3bc101030 0x7fe3bc19a590 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80097d0 tx=0x7fe3a8009c90 comp rx=0 tx=0).stop 2024-04-25T02:16:48.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.379+0000 7fe3c3093640 1 -- 172.21.15.31:0/144529503 shutdown_connections 2024-04-25T02:16:48.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.379+0000 7fe3c3093640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe3ac0a2e20 0x7fe3ac0a52e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.379+0000 7fe3c3093640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3bc106630 0x7fe3bc19e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.380+0000 7fe3c3093640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3bc101030 0x7fe3bc19a590 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.380+0000 7fe3c3093640 1 --2- 172.21.15.31:0/144529503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3bc100670 0x7fe3bc19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:16:48.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.380+0000 7fe3c3093640 1 -- 172.21.15.31:0/144529503 >> 172.21.15.31:0/144529503 conn(0x7fe3bc0777c0 msgr2=0x7fe3bc103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:16:48.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.380+0000 7fe3c3093640 1 -- 172.21.15.31:0/144529503 shutdown_connections 2024-04-25T02:16:48.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:16:48.380+0000 7fe3c3093640 1 -- 172.21.15.31:0/144529503 wait complete. 2024-04-25T02:16:48.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:48 smithi040 ceph-mon[32418]: pgmap v703: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 237 op/s 2024-04-25T02:16:48.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:48 smithi134 ceph-mon[33098]: pgmap v703: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 237 op/s 2024-04-25T02:16:48.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:48 smithi031 ceph-mon[29204]: pgmap v703: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 237 op/s 2024-04-25T02:16:49.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 87419 8.80 MB/sec execute 413 sec latency 79.268 ms 2024-04-25T02:16:50.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 87618 8.81 MB/sec execute 414 sec latency 66.716 ms 2024-04-25T02:16:50.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:50 smithi040 ceph-mon[32418]: pgmap v704: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 237 op/s 2024-04-25T02:16:50.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:50 smithi134 ceph-mon[33098]: pgmap v704: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 237 op/s 2024-04-25T02:16:50.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:50 smithi031 ceph-mon[29204]: pgmap v704: 97 pgs: 97 active+clean; 1.4 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 237 op/s 2024-04-25T02:16:51.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 87800 8.83 MB/sec execute 415 sec latency 65.847 ms 2024-04-25T02:16:52.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 87921 8.82 MB/sec execute 416 sec latency 97.895 ms 2024-04-25T02:16:52.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:52 smithi040 ceph-mon[32418]: pgmap v705: 97 pgs: 97 active+clean; 1.4 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.4 MiB/s wr, 367 op/s 2024-04-25T02:16:52.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:52 smithi134 ceph-mon[33098]: pgmap v705: 97 pgs: 97 active+clean; 1.4 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.4 MiB/s wr, 367 op/s 2024-04-25T02:16:52.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:52 smithi031 ceph-mon[29204]: pgmap v705: 97 pgs: 97 active+clean; 1.4 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.4 MiB/s wr, 367 op/s 2024-04-25T02:16:53.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 88079 8.82 MB/sec execute 417 sec latency 87.938 ms 2024-04-25T02:16:54.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 88251 8.82 MB/sec execute 418 sec latency 85.444 ms 2024-04-25T02:16:54.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:54 smithi040 ceph-mon[32418]: pgmap v706: 97 pgs: 97 active+clean; 1.4 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 235 op/s 2024-04-25T02:16:54.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:54 smithi134 ceph-mon[33098]: pgmap v706: 97 pgs: 97 active+clean; 1.4 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 235 op/s 2024-04-25T02:16:54.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:54 smithi031 ceph-mon[29204]: pgmap v706: 97 pgs: 97 active+clean; 1.4 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 235 op/s 2024-04-25T02:16:55.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 88424 8.81 MB/sec execute 419 sec latency 101.311 ms 2024-04-25T02:16:56.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 88570 8.80 MB/sec execute 420 sec latency 63.974 ms 2024-04-25T02:16:56.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:56 smithi040 ceph-mon[32418]: pgmap v707: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.8 MiB/s wr, 348 op/s 2024-04-25T02:16:56.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:56 smithi134 ceph-mon[33098]: pgmap v707: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.8 MiB/s wr, 348 op/s 2024-04-25T02:16:56.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:56 smithi031 ceph-mon[29204]: pgmap v707: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.8 MiB/s wr, 348 op/s 2024-04-25T02:16:57.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 88717 8.79 MB/sec execute 421 sec latency 70.216 ms 2024-04-25T02:16:58.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 88894 8.80 MB/sec execute 422 sec latency 88.573 ms 2024-04-25T02:16:58.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:16:58 smithi040 ceph-mon[32418]: pgmap v708: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 241 op/s 2024-04-25T02:16:58.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:16:58 smithi134 ceph-mon[33098]: pgmap v708: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 241 op/s 2024-04-25T02:16:58.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:16:58 smithi031 ceph-mon[29204]: pgmap v708: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 241 op/s 2024-04-25T02:16:59.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 89045 8.80 MB/sec execute 423 sec latency 106.877 ms 2024-04-25T02:17:00.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 89212 8.80 MB/sec execute 424 sec latency 54.530 ms 2024-04-25T02:17:00.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:00 smithi040 ceph-mon[32418]: pgmap v709: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 241 op/s 2024-04-25T02:17:00.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:00 smithi134 ceph-mon[33098]: pgmap v709: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 241 op/s 2024-04-25T02:17:00.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:00 smithi031 ceph-mon[29204]: pgmap v709: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 241 op/s 2024-04-25T02:17:01.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 89435 8.82 MB/sec execute 425 sec latency 81.356 ms 2024-04-25T02:17:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.510+0000 7f0058c15640 1 -- 172.21.15.31:0/3577609015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0054106630 msgr2=0x7f0054106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.510+0000 7f0058c15640 1 --2- 172.21.15.31:0/3577609015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0054106630 0x7f0054106a10 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0044009930 tx=0x7f004402a420 comp rx=0 tx=0).stop 2024-04-25T02:17:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.510+0000 7f0058c15640 1 -- 172.21.15.31:0/3577609015 shutdown_connections 2024-04-25T02:17:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.510+0000 7f0058c15640 1 --2- 172.21.15.31:0/3577609015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054101030 0x7f005410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.510+0000 7f0058c15640 1 --2- 172.21.15.31:0/3577609015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0054100670 0x7f0054100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.510+0000 7f0058c15640 1 --2- 172.21.15.31:0/3577609015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0054106630 0x7f0054106a10 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.510+0000 7f0058c15640 1 -- 172.21.15.31:0/3577609015 >> 172.21.15.31:0/3577609015 conn(0x7f00540777c0 msgr2=0x7f0054077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.510+0000 7f0058c15640 1 -- 172.21.15.31:0/3577609015 shutdown_connections 2024-04-25T02:17:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.510+0000 7f0058c15640 1 -- 172.21.15.31:0/3577609015 wait complete. 2024-04-25T02:17:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.511+0000 7f0058c15640 1 Processor -- start 2024-04-25T02:17:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.511+0000 7f0058c15640 1 -- start start 2024-04-25T02:17:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0058c15640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0054100670 0x7f0054197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0058c15640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054101030 0x7f0054198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0058c15640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0054106630 0x7f005419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0058c15640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0054069ba0 con 0x7f0054100670 2024-04-25T02:17:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0058c15640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0054069a20 con 0x7f0054101030 2024-04-25T02:17:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0058c15640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0054069d20 con 0x7f0054106630 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f005259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0054100670 0x7f0054197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f005259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0054100670 0x7f0054197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55470/0 (socket says 172.21.15.31:55470) 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0051d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054101030 0x7f0054198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f005259c640 1 -- 172.21.15.31:0/2529076908 learned_addr learned my addr 172.21.15.31:0/2529076908 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0052d9d640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0054106630 0x7f005419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0051d9b640 1 -- 172.21.15.31:0/2529076908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0054106630 msgr2=0x7f005419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0051d9b640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0054106630 0x7f005419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0051d9b640 1 -- 172.21.15.31:0/2529076908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0054100670 msgr2=0x7f0054197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0051d9b640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0054100670 0x7f0054197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0051d9b640 1 -- 172.21.15.31:0/2529076908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005419ceb0 con 0x7f0054101030 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.512+0000 7f0051d9b640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054101030 0x7f0054198400 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f004800e9f0 tx=0x7f004800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.513+0000 7f00437fe640 1 -- 172.21.15.31:0/2529076908 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004800ccc0 con 0x7f0054101030 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.513+0000 7f0058c15640 1 -- 172.21.15.31:0/2529076908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005419d1a0 con 0x7f0054101030 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.513+0000 7f00437fe640 1 -- 172.21.15.31:0/2529076908 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0048005020 con 0x7f0054101030 2024-04-25T02:17:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.513+0000 7f00437fe640 1 -- 172.21.15.31:0/2529076908 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0048004a00 con 0x7f0054101030 2024-04-25T02:17:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.513+0000 7f0058c15640 1 -- 172.21.15.31:0/2529076908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00541a4a30 con 0x7f0054101030 2024-04-25T02:17:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.515+0000 7f00437fe640 1 -- 172.21.15.31:0/2529076908 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0048004ba0 con 0x7f0054101030 2024-04-25T02:17:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.516+0000 7f005b9ff740 1 Processor -- start 2024-04-25T02:17:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.516+0000 7f005b9ff740 1 -- start start 2024-04-25T02:17:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.516+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017cc240 0x55c9017cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.516+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017d6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.516+0000 7f00437fe640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f003c0a2da0 0x7f003c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017d7230 0x55c9017d9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005b9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c901730cb0 con 0x55c9017ccb60 2024-04-25T02:17:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005b9ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c9016d95b0 con 0x55c9017cc240 2024-04-25T02:17:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005b9ff740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c90172cc70 con 0x55c9017d7230 2024-04-25T02:17:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017d6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017d6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55484/0 (socket says 172.21.15.31:55484) 2024-04-25T02:17:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005259c640 1 -- 172.21.15.31:0/3585637855 learned_addr learned my addr 172.21.15.31:0/3585637855 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f0052d9d640 1 --2- 172.21.15.31:0/3585637855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017cc240 0x55c9017cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f0051d9b640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f003c0a2da0 0x7f003c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f0052d9d640 1 --2- 172.21.15.31:0/3585637855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017d7230 0x55c9017d9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f00437fe640 1 -- 172.21.15.31:0/2529076908 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7f00480cf390 con 0x7f0054101030 2024-04-25T02:17:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005259c640 1 -- 172.21.15.31:0/3585637855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017d7230 msgr2=0x55c9017d9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005259c640 1 --2- 172.21.15.31:0/3585637855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017d7230 0x55c9017d9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005259c640 1 -- 172.21.15.31:0/3585637855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017cc240 msgr2=0x55c9017cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005259c640 1 --2- 172.21.15.31:0/3585637855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017cc240 0x55c9017cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005259c640 1 -- 172.21.15.31:0/3585637855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c901445210 con 0x55c9017ccb60 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f0051d9b640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f003c0a2da0 0x7f003c0a5260 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f00480ccbb0 tx=0x7f0048066500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.517+0000 7f005259c640 1 --2- 172.21.15.31:0/3585637855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017d6cf0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f0044009900 tx=0x7f0044004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ccdb92e393bc925 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.518+0000 7f0040ff9640 1 -- 172.21.15.31:0/3585637855 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0044040070 con 0x55c9017ccb60 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.518+0000 7f0040ff9640 1 -- 172.21.15.31:0/3585637855 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00440054d0 con 0x55c9017ccb60 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.518+0000 7f0040ff9640 1 -- 172.21.15.31:0/3585637855 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0044035040 con 0x55c9017ccb60 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.518+0000 7f005b9ff740 1 -- 172.21.15.31:0/3585637855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 msgr2=0x55c9017d6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.518+0000 7f005b9ff740 1 --2- 172.21.15.31:0/3585637855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017d6cf0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f0044009900 tx=0x7f0044004040 comp rx=0 tx=0).stop 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.518+0000 7f005b9ff740 1 -- 172.21.15.31:0/3585637855 shutdown_connections 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.518+0000 7f005b9ff740 1 --2- 172.21.15.31:0/3585637855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017d7230 0x55c9017d9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.518+0000 7f005b9ff740 1 --2- 172.21.15.31:0/3585637855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017d6cf0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.518+0000 7f005b9ff740 1 --2- 172.21.15.31:0/3585637855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017cc240 0x55c9017cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.518+0000 7f005b9ff740 1 -- 172.21.15.31:0/3585637855 >> 172.21.15.31:0/3585637855 conn(0x55c9017c8560 msgr2=0x55c9017c8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005b9ff740 1 -- 172.21.15.31:0/3585637855 shutdown_connections 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005b9ff740 1 -- 172.21.15.31:0/3585637855 wait complete. 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005b9ff740 1 -- start start 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005b9ff740 1 Processor -- start 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017cc240 0x55c9017d4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017ccb60 0x55c9017d4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017d7230 0x55c9017d4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005b9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c9017d9fc0 con 0x55c9017d7230 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005b9ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c9016ecb50 con 0x55c9017cc240 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005b9ff740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c9017da140 con 0x55c9017ccb60 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005b9ff740 10 client.0 _ll_register_callbacks cb 0x55c901727ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017ccb60 0x55c9017d4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005259c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017ccb60 0x55c9017d4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:38306/0 (socket says 172.21.15.31:38306) 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.519+0000 7f005259c640 1 -- 172.21.15.31:0/3510532262 learned_addr learned my addr 172.21.15.31:0/3510532262 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f0052d9d640 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017d7230 0x55c9017d4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f005259c640 1 -- 172.21.15.31:0/3510532262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017cc240 msgr2=0x55c9017d4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f0052d9d640 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017cc240 0x55c9017d4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f005259c640 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017cc240 0x55c9017d4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f005259c640 1 -- 172.21.15.31:0/3510532262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017d7230 msgr2=0x55c9017d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f005259c640 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017d7230 0x55c9017d4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f005259c640 1 -- 172.21.15.31:0/3510532262 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c901511620 con 0x55c9017ccb60 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f0052d9d640 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017d7230 0x55c9017d4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f005259c640 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017ccb60 0x55c9017d4660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0044005df0 tx=0x7f00440048b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad61cefeab34d01a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f0039ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f0039ffb640 1 -- 172.21.15.31:0/3510532262 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0044035070 con 0x55c9017ccb60 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f0039ffb640 1 -- 172.21.15.31:0/3510532262 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7f0044005190 con 0x55c9017ccb60 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f005b9ff740 1 -- 172.21.15.31:0/3510532262 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00440054d0 con 0x55c9017ccb60 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f0039ffb640 1 -- 172.21.15.31:0/3510532262 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f0044040040 con 0x55c9017ccb60 2024-04-25T02:17:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.520+0000 7f005b9ff740 10 client.27274 fetch_fsmap learned FSMap version 50 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.521+0000 7f005b9ff740 1 -- 172.21.15.31:0/3510532262 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55c9017d9fc0 con 0x55c9017ccb60 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.521+0000 7f0039ffb640 1 -- 172.21.15.31:0/3510532262 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00440054d0 con 0x55c9017ccb60 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.521+0000 7f0039ffb640 1 -- 172.21.15.31:0/3510532262 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f004405b020 con 0x55c9017ccb60 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.521+0000 7f005b9ff740 10 client.27274 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.521+0000 7f005b9ff740 10 client.27274 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.521+0000 7f005b9ff740 20 client.27274 populate_metadata read hostname 'smithi031' 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.521+0000 7f005b9ff740 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c9017e72c0 0x55c9017e96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.521+0000 7f005b9ff740 4 client.27274 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.521+0000 7f005b9ff740 1 -- 172.21.15.31:0/3510532262 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c90173e1f0 con 0x55c9017e72c0 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.521+0000 7f0052d9d640 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c9017e72c0 0x55c9017e96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.522+0000 7f0052d9d640 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c9017e72c0 0x55c9017e96b0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec8f4f5f370e533f server_cookie=dab33b846f47729f in_seq=0 out_seq=0 2024-04-25T02:17:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.522+0000 7f0039ffb640 10 client.27274 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.522+0000 7f0039ffb640 1 -- 172.21.15.31:0/3510532262 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c90173e1f0 con 0x55c9017e72c0 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.522+0000 7f0039ffb640 10 client.27274 handle_command_reply: tid=0 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.522+0000 7f005b9ff740 1 client.27274 shutdown 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f005b9ff740 1 -- 172.21.15.31:0/3510532262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017ccb60 msgr2=0x55c9017d4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f005b9ff740 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017ccb60 0x55c9017d4660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0044005df0 tx=0x7f00440048b0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f005b9ff740 1 -- 172.21.15.31:0/3510532262 shutdown_connections 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f005b9ff740 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c9017e72c0 0x55c9017e96b0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f0039ffb640 0 client.27274 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f005b9ff740 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017d7230 0x55c9017d4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f005b9ff740 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017ccb60 0x55c9017d4660 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f005b9ff740 1 --2- 172.21.15.31:0/3510532262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017cc240 0x55c9017d4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f005b9ff740 1 -- 172.21.15.31:0/3510532262 >> 172.21.15.31:0/3510532262 conn(0x55c9017c8560 msgr2=0x55c9017ce260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f005b9ff740 1 -- 172.21.15.31:0/3510532262 shutdown_connections 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f005b9ff740 1 -- 172.21.15.31:0/3510532262 wait complete. 2024-04-25T02:17:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.523+0000 7f005b9ff740 20 client.27274 trim_cache size 0 max 16384 2024-04-25T02:17:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f005b9ff740 1 Processor -- start 2024-04-25T02:17:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f005b9ff740 1 -- start start 2024-04-25T02:17:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017e2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017d49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017d7230 0x55c9017eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f005b9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f00541ab080 con 0x55c9017ccb60 2024-04-25T02:17:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f005b9ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00541acbc0 con 0x55c9017d7230 2024-04-25T02:17:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f005b9ff740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f00541ab750 con 0x55c9017cc240 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f005259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017d49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f0052d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017e2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f0052d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017e2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:38320/0 (socket says 172.21.15.31:38320) 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f0052d9d640 1 -- 172.21.15.31:0/172925863 learned_addr learned my addr 172.21.15.31:0/172925863 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f005259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017d49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55508/0 (socket says 172.21.15.31:55508) 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.545+0000 7f0052d9d640 1 --2- 172.21.15.31:0/172925863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017d7230 0x55c9017eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.546+0000 7f0052d9d640 1 -- 172.21.15.31:0/172925863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017d7230 msgr2=0x55c9017eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.546+0000 7f0052d9d640 1 --2- 172.21.15.31:0/172925863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017d7230 0x55c9017eedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.546+0000 7f0052d9d640 1 -- 172.21.15.31:0/172925863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 msgr2=0x55c9017d49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.546+0000 7f0052d9d640 1 --2- 172.21.15.31:0/172925863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017d49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.546+0000 7f0052d9d640 1 -- 172.21.15.31:0/172925863 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9017cc980 con 0x55c9017cc240 2024-04-25T02:17:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.546+0000 7f0052d9d640 1 --2- 172.21.15.31:0/172925863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017e2eb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f004c011310 tx=0x7f004c013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76166eb29d231878 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.546+0000 7f003bfff640 1 -- 172.21.15.31:0/172925863 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c004b20 con 0x55c9017cc240 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.546+0000 7f003bfff640 1 -- 172.21.15.31:0/172925863 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f004c004cc0 con 0x55c9017cc240 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.546+0000 7f003bfff640 1 -- 172.21.15.31:0/172925863 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c010430 con 0x55c9017cc240 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.547+0000 7f005b9ff740 1 -- 172.21.15.31:0/172925863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 msgr2=0x55c9017e2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.547+0000 7f005b9ff740 1 --2- 172.21.15.31:0/172925863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017e2eb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f004c011310 tx=0x7f004c013040 comp rx=0 tx=0).stop 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.547+0000 7f005b9ff740 1 -- 172.21.15.31:0/172925863 shutdown_connections 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.547+0000 7f005b9ff740 1 --2- 172.21.15.31:0/172925863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017d7230 0x55c9017eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.547+0000 7f005b9ff740 1 --2- 172.21.15.31:0/172925863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017d49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.547+0000 7f005b9ff740 1 --2- 172.21.15.31:0/172925863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017e2eb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.547+0000 7f005b9ff740 1 -- 172.21.15.31:0/172925863 >> 172.21.15.31:0/172925863 conn(0x55c9017c8560 msgr2=0x55c9017c75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.547+0000 7f005b9ff740 1 -- 172.21.15.31:0/172925863 shutdown_connections 2024-04-25T02:17:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.547+0000 7f005b9ff740 1 -- 172.21.15.31:0/172925863 wait complete. 2024-04-25T02:17:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.547+0000 7f005b9ff740 1 -- start start 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.547+0000 7f005b9ff740 1 Processor -- start 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017df3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017df900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005b9ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017d7230 0x55c9017dc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005b9ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f00541ac4f0 con 0x55c9017ccb60 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005b9ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00541abe20 con 0x55c9017d7230 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005b9ff740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f00541a9060 con 0x55c9017cc240 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005b9ff740 10 client.0 _ll_register_callbacks cb 0x7f004c009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017df900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f0052d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017df3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f0052d9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017df3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:38326/0 (socket says 172.21.15.31:38326) 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017df900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55514/0 (socket says 172.21.15.31:55514) 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f0052d9d640 1 -- 172.21.15.31:0/4116207611 learned_addr learned my addr 172.21.15.31:0/4116207611 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:01.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f0052d9d640 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017d7230 0x55c9017dc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005259c640 1 -- 172.21.15.31:0/4116207611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 msgr2=0x55c9017df3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005259c640 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017df3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005259c640 1 -- 172.21.15.31:0/4116207611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017d7230 msgr2=0x55c9017dc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005259c640 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017d7230 0x55c9017dc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f005259c640 1 -- 172.21.15.31:0/4116207611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9017dca30 con 0x55c9017ccb60 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.548+0000 7f0052d9d640 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017df3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.549+0000 7f005259c640 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017df900 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f0044031440 tx=0x7f0044005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c141eaa69bc6fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.549+0000 7f0039ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.549+0000 7f0039ffb640 1 -- 172.21.15.31:0/4116207611 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0044035070 con 0x55c9017ccb60 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.549+0000 7f0039ffb640 1 -- 172.21.15.31:0/4116207611 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8484+0+0 (secure 0 0 0) 0x7f004404b8a0 con 0x55c9017ccb60 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.549+0000 7f005b9ff740 1 -- 172.21.15.31:0/4116207611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f004c004cc0 con 0x55c9017ccb60 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.549+0000 7f0039ffb640 1 -- 172.21.15.31:0/4116207611 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f0044009520 con 0x55c9017ccb60 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.549+0000 7f005b9ff740 10 client.17940 fetch_fsmap learned FSMap version 50 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.549+0000 7f005b9ff740 1 -- 172.21.15.31:0/4116207611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55c90184dfe0 con 0x55c9017ccb60 2024-04-25T02:17:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.549+0000 7f0039ffb640 1 -- 172.21.15.31:0/4116207611 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f004c004cc0 con 0x55c9017ccb60 2024-04-25T02:17:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.550+0000 7f0039ffb640 1 -- 172.21.15.31:0/4116207611 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f004405b030 con 0x55c9017ccb60 2024-04-25T02:17:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.550+0000 7f005b9ff740 10 client.17940 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:17:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.550+0000 7f005b9ff740 10 client.17940 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:17:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.550+0000 7f005b9ff740 20 client.17940 populate_metadata read hostname 'smithi031' 2024-04-25T02:17:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.550+0000 7f005b9ff740 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c9017e72c0 0x55c901850700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.550+0000 7f005b9ff740 4 client.17940 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:17:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.550+0000 7f005b9ff740 1 -- 172.21.15.31:0/4116207611 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c901738540 con 0x55c9017e72c0 2024-04-25T02:17:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.550+0000 7f0052d9d640 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c9017e72c0 0x55c901850700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.550+0000 7f0039ffb640 10 client.17940 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.550+0000 7f0052d9d640 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c9017e72c0 0x55c901850700 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba38a0ebceba146d server_cookie=ce6930157e064aee in_seq=0 out_seq=0 2024-04-25T02:17:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.657+0000 7f0039ffb640 1 -- 172.21.15.31:0/4116207611 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c901738540 con 0x55c9017e72c0 2024-04-25T02:17:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.657+0000 7f0039ffb640 10 client.17940 handle_command_reply: tid=0 2024-04-25T02:17:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.657+0000 7f005b9ff740 1 client.17940 shutdown 2024-04-25T02:17:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f005b9ff740 1 -- 172.21.15.31:0/4116207611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 msgr2=0x55c9017df900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f005b9ff740 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017df900 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f0044031440 tx=0x7f0044005fd0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f005b9ff740 1 -- 172.21.15.31:0/4116207611 shutdown_connections 2024-04-25T02:17:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f005b9ff740 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c9017e72c0 0x55c901850700 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f005b9ff740 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c9017d7230 0x55c9017dc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f005b9ff740 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c9017ccb60 0x55c9017df900 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f005b9ff740 1 --2- 172.21.15.31:0/4116207611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c9017cc240 0x55c9017df3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f005b9ff740 1 -- 172.21.15.31:0/4116207611 >> 172.21.15.31:0/4116207611 conn(0x55c9017c8560 msgr2=0x55c9017d57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f0039ffb640 0 client.17940 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f005b9ff740 1 -- 172.21.15.31:0/4116207611 shutdown_connections 2024-04-25T02:17:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f005b9ff740 1 -- 172.21.15.31:0/4116207611 wait complete. 2024-04-25T02:17:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.658+0000 7f005b9ff740 20 client.17940 trim_cache size 0 max 16384 2024-04-25T02:17:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:17:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:17:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:17:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:17:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 -- 172.21.15.31:0/2529076908 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f003c0a2da0 msgr2=0x7f003c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f003c0a2da0 0x7f003c0a5260 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f00480ccbb0 tx=0x7f0048066500 comp rx=0 tx=0).stop 2024-04-25T02:17:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 -- 172.21.15.31:0/2529076908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054101030 msgr2=0x7f0054198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:01.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054101030 0x7f0054198400 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f004800e9f0 tx=0x7f004800eeb0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 -- 172.21.15.31:0/2529076908 shutdown_connections 2024-04-25T02:17:01.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f003c0a2da0 0x7f003c0a5260 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0054106630 0x7f005419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054101030 0x7f0054198400 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 --2- 172.21.15.31:0/2529076908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0054100670 0x7f0054197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:01.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 -- 172.21.15.31:0/2529076908 >> 172.21.15.31:0/2529076908 conn(0x7f00540777c0 msgr2=0x7f00541031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:01.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 -- 172.21.15.31:0/2529076908 shutdown_connections 2024-04-25T02:17:01.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:01.659+0000 7f0058c15640 1 -- 172.21.15.31:0/2529076908 wait complete. 2024-04-25T02:17:02.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 89614 8.82 MB/sec execute 426 sec latency 93.035 ms 2024-04-25T02:17:02.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:02 smithi040 ceph-mon[32418]: pgmap v710: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.9 MiB/s wr, 362 op/s 2024-04-25T02:17:02.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:02 smithi134 ceph-mon[33098]: pgmap v710: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.9 MiB/s wr, 362 op/s 2024-04-25T02:17:02.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:02 smithi031 ceph-mon[29204]: pgmap v710: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.9 MiB/s wr, 362 op/s 2024-04-25T02:17:03.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 89758 8.82 MB/sec execute 427 sec latency 109.342 ms 2024-04-25T02:17:03.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:03 smithi040 ceph-mon[32418]: osdmap e122: 12 total, 12 up, 12 in 2024-04-25T02:17:03.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:03 smithi134 ceph-mon[33098]: osdmap e122: 12 total, 12 up, 12 in 2024-04-25T02:17:03.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:03 smithi031 ceph-mon[29204]: osdmap e122: 12 total, 12 up, 12 in 2024-04-25T02:17:04.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 89886 8.81 MB/sec execute 428 sec latency 92.839 ms 2024-04-25T02:17:04.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:04 smithi040 ceph-mon[32418]: pgmap v712: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.0 MiB/s wr, 279 op/s 2024-04-25T02:17:04.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:04 smithi134 ceph-mon[33098]: pgmap v712: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.0 MiB/s wr, 279 op/s 2024-04-25T02:17:04.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:04 smithi031 ceph-mon[29204]: pgmap v712: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.0 MiB/s wr, 279 op/s 2024-04-25T02:17:05.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 90063 8.83 MB/sec execute 429 sec latency 68.463 ms 2024-04-25T02:17:06.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 90243 8.84 MB/sec execute 430 sec latency 56.419 ms 2024-04-25T02:17:06.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:06 smithi040 ceph-mon[32418]: pgmap v713: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 315 op/s 2024-04-25T02:17:06.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:06 smithi134 ceph-mon[33098]: pgmap v713: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 315 op/s 2024-04-25T02:17:06.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:06 smithi031 ceph-mon[29204]: pgmap v713: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 315 op/s 2024-04-25T02:17:07.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 90409 8.83 MB/sec execute 431 sec latency 51.660 ms 2024-04-25T02:17:08.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 90552 8.82 MB/sec execute 432 sec latency 56.538 ms 2024-04-25T02:17:08.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:08 smithi040 ceph-mon[32418]: pgmap v714: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 315 op/s 2024-04-25T02:17:08.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:08 smithi134 ceph-mon[33098]: pgmap v714: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 315 op/s 2024-04-25T02:17:08.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:08 smithi031 ceph-mon[29204]: pgmap v714: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 315 op/s 2024-04-25T02:17:09.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 90676 8.81 MB/sec execute 433 sec latency 74.498 ms 2024-04-25T02:17:10.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 90812 8.80 MB/sec execute 434 sec latency 248.655 ms 2024-04-25T02:17:10.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:10 smithi040 ceph-mon[32418]: pgmap v715: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 315 op/s 2024-04-25T02:17:10.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:10 smithi040 ceph-mon[32418]: osdmap e123: 12 total, 12 up, 12 in 2024-04-25T02:17:10.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:10 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:17:10.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:10 smithi134 ceph-mon[33098]: pgmap v715: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 315 op/s 2024-04-25T02:17:10.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:10 smithi134 ceph-mon[33098]: osdmap e123: 12 total, 12 up, 12 in 2024-04-25T02:17:10.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:10 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:17:10.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:10 smithi031 ceph-mon[29204]: pgmap v715: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.4 MiB/s wr, 315 op/s 2024-04-25T02:17:10.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:10 smithi031 ceph-mon[29204]: osdmap e123: 12 total, 12 up, 12 in 2024-04-25T02:17:10.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:10 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:17:11.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 90980 8.81 MB/sec execute 435 sec latency 83.178 ms 2024-04-25T02:17:11.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:11 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:11 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:11 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:11 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:11 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:17:11.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:11 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:17:11.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:11 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:11 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:11 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:11 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:11 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:17:11.444 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:11 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:17:11.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:11 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:11 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:11 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:11 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:11.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:11 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:17:11.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:11 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:17:12.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 91138 8.80 MB/sec execute 436 sec latency 85.847 ms 2024-04-25T02:17:12.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:12 smithi040 ceph-mon[32418]: pgmap v717: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.3 MiB/s wr, 335 op/s 2024-04-25T02:17:12.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:12 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:12.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:12 smithi134 ceph-mon[33098]: pgmap v717: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.3 MiB/s wr, 335 op/s 2024-04-25T02:17:12.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:12 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:12.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:12 smithi031 ceph-mon[29204]: pgmap v717: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.3 MiB/s wr, 335 op/s 2024-04-25T02:17:12.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:12 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:17:13.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 91285 8.80 MB/sec execute 437 sec latency 67.504 ms 2024-04-25T02:17:14.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 91448 8.80 MB/sec execute 438 sec latency 62.499 ms 2024-04-25T02:17:14.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:14 smithi040 ceph-mon[32418]: pgmap v718: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.2 MiB/s wr, 288 op/s 2024-04-25T02:17:14.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:14 smithi134 ceph-mon[33098]: pgmap v718: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.2 MiB/s wr, 288 op/s 2024-04-25T02:17:14.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:14 smithi031 ceph-mon[29204]: pgmap v718: 97 pgs: 97 active+clean; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.2 MiB/s wr, 288 op/s 2024-04-25T02:17:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.788+0000 7f016e990640 1 -- 172.21.15.31:0/102950490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0168101710 msgr2=0x7f016810d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.788+0000 7f016e990640 1 --2- 172.21.15.31:0/102950490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0168101710 0x7f016810d700 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0160009cd0 tx=0x7f016002a8e0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.789+0000 7f016e990640 1 -- 172.21.15.31:0/102950490 shutdown_connections 2024-04-25T02:17:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.789+0000 7f016e990640 1 --2- 172.21.15.31:0/102950490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0168101710 0x7f016810d700 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.789+0000 7f016e990640 1 --2- 172.21.15.31:0/102950490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0168100d50 0x7f01681011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.789+0000 7f016e990640 1 --2- 172.21.15.31:0/102950490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0168106840 0x7f0168106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.789+0000 7f016e990640 1 -- 172.21.15.31:0/102950490 >> 172.21.15.31:0/102950490 conn(0x7f0168076bd0 msgr2=0x7f01680fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.789+0000 7f016e990640 1 -- 172.21.15.31:0/102950490 shutdown_connections 2024-04-25T02:17:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.789+0000 7f016e990640 1 -- 172.21.15.31:0/102950490 wait complete. 2024-04-25T02:17:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.789+0000 7f016e990640 1 Processor -- start 2024-04-25T02:17:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.789+0000 7f016e990640 1 -- start start 2024-04-25T02:17:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f016e990640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0168100d50 0x7f01681980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f016e990640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0168101710 0x7f0168198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f016e990640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0168106840 0x7f016819c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f016e990640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0168110b00 con 0x7f0168100d50 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f016e990640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0168110980 con 0x7f0168106840 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f016e990640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0168110c80 con 0x7f0168101710 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f0167fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0168100d50 0x7f01681980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f0167fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0168100d50 0x7f01681980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49656/0 (socket says 172.21.15.31:49656) 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f0167fff640 1 -- 172.21.15.31:0/3458593988 learned_addr learned my addr 172.21.15.31:0/3458593988 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f01677fe640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0168101710 0x7f0168198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f016cf2d640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0168106840 0x7f016819c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f0167fff640 1 -- 172.21.15.31:0/3458593988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0168101710 msgr2=0x7f0168198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f0167fff640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0168101710 0x7f0168198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f0167fff640 1 -- 172.21.15.31:0/3458593988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0168106840 msgr2=0x7f016819c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f0167fff640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0168106840 0x7f016819c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f0167fff640 1 -- 172.21.15.31:0/3458593988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f016819d0d0 con 0x7f0168100d50 2024-04-25T02:17:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.790+0000 7f0167fff640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0168100d50 0x7f01681980e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f015800e9f0 tx=0x7f015800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.791+0000 7f01657fa640 1 -- 172.21.15.31:0/3458593988 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015800cd80 con 0x7f0168100d50 2024-04-25T02:17:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.791+0000 7f01657fa640 1 -- 172.21.15.31:0/3458593988 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01580054d0 con 0x7f0168100d50 2024-04-25T02:17:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.791+0000 7f01657fa640 1 -- 172.21.15.31:0/3458593988 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158010490 con 0x7f0168100d50 2024-04-25T02:17:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.791+0000 7f016e990640 1 -- 172.21.15.31:0/3458593988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f016819d3c0 con 0x7f0168100d50 2024-04-25T02:17:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.791+0000 7f016e990640 1 -- 172.21.15.31:0/3458593988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01680fbf40 con 0x7f0168100d50 2024-04-25T02:17:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.794+0000 7f01657fa640 1 -- 172.21.15.31:0/3458593988 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0158005020 con 0x7f0168100d50 2024-04-25T02:17:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.794+0000 7f017177c740 1 Processor -- start 2024-04-25T02:17:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.794+0000 7f017177c740 1 -- start start 2024-04-25T02:17:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.794+0000 7f017177c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bb6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.794+0000 7f017177c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bb6b60 0x561f49bc0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.794+0000 7f017177c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bc1230 0x561f49bc3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.794+0000 7f017177c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561f49b1acb0 con 0x561f49bb6b60 2024-04-25T02:17:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.794+0000 7f017177c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561f49ac35b0 con 0x561f49bb6240 2024-04-25T02:17:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.794+0000 7f017177c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561f49b16c70 con 0x561f49bc1230 2024-04-25T02:17:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f016cf2d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bb6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f0167fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bb6b60 0x561f49bc0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f016cf2d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bb6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47128/0 (socket says 172.21.15.31:47128) 2024-04-25T02:17:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f016cf2d640 1 -- 172.21.15.31:0/3606793980 learned_addr learned my addr 172.21.15.31:0/3606793980 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f01677fe640 1 --2- 172.21.15.31:0/3606793980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bc1230 0x561f49bc3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f01657fa640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f01500a2e20 0x7f01500a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f016cf2d640 1 -- 172.21.15.31:0/3606793980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bc1230 msgr2=0x561f49bc3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f016cf2d640 1 --2- 172.21.15.31:0/3606793980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bc1230 0x561f49bc3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f016cf2d640 1 -- 172.21.15.31:0/3606793980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bb6b60 msgr2=0x561f49bc0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f016cf2d640 1 --2- 172.21.15.31:0/3606793980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bb6b60 0x561f49bc0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f016cf2d640 1 -- 172.21.15.31:0/3606793980 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f4982f210 con 0x561f49bb6240 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f0167fff640 1 --2- 172.21.15.31:0/3606793980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bb6b60 0x561f49bc0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.795+0000 7f01657fa640 1 -- 172.21.15.31:0/3458593988 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f01580cf6c0 con 0x7f0168100d50 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.796+0000 7f016cf2d640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f01500a2e20 0x7f01500a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.796+0000 7f016cf2d640 1 --2- 172.21.15.31:0/3606793980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bb6620 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0160009cd0 tx=0x7f0160009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa2039de2af00fde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.796+0000 7f014e7fc640 1 -- 172.21.15.31:0/3606793980 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160031850 con 0x561f49bb6240 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.796+0000 7f014e7fc640 1 -- 172.21.15.31:0/3606793980 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0160036040 con 0x561f49bb6240 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.796+0000 7f014e7fc640 1 -- 172.21.15.31:0/3606793980 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01600307a0 con 0x561f49bb6240 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.797+0000 7f017177c740 1 -- 172.21.15.31:0/3606793980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 msgr2=0x561f49bb6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.797+0000 7f016cf2d640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f01500a2e20 0x7f01500a52e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f01600317f0 tx=0x7f016004b000 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.797+0000 7f017177c740 1 --2- 172.21.15.31:0/3606793980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bb6620 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0160009cd0 tx=0x7f0160009a20 comp rx=0 tx=0).stop 2024-04-25T02:17:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.797+0000 7f017177c740 1 -- 172.21.15.31:0/3606793980 shutdown_connections 2024-04-25T02:17:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.797+0000 7f017177c740 1 --2- 172.21.15.31:0/3606793980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bc1230 0x561f49bc3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.797+0000 7f017177c740 1 --2- 172.21.15.31:0/3606793980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bb6b60 0x561f49bc0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.797+0000 7f017177c740 1 --2- 172.21.15.31:0/3606793980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bb6620 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.797+0000 7f017177c740 1 -- 172.21.15.31:0/3606793980 >> 172.21.15.31:0/3606793980 conn(0x561f49bb2560 msgr2=0x561f49bb2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.797+0000 7f017177c740 1 -- 172.21.15.31:0/3606793980 shutdown_connections 2024-04-25T02:17:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.797+0000 7f017177c740 1 -- 172.21.15.31:0/3606793980 wait complete. 2024-04-25T02:17:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.797+0000 7f017177c740 1 -- start start 2024-04-25T02:17:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f017177c740 1 Processor -- start 2024-04-25T02:17:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f017177c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bbe110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f017177c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6b60 0x561f49bbe650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f017177c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bc1230 0x561f49bbeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f017177c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561f49bc3fc0 con 0x561f49bc1230 2024-04-25T02:17:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f017177c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561f49ad6b50 con 0x561f49bb6240 2024-04-25T02:17:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f017177c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561f49bc4140 con 0x561f49bb6b60 2024-04-25T02:17:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f0167fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bbe110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f016cf2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bc1230 0x561f49bbeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f0167fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bbe110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47142/0 (socket says 172.21.15.31:47142) 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f0167fff640 1 -- 172.21.15.31:0/1654704281 learned_addr learned my addr 172.21.15.31:0/1654704281 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f01677fe640 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6b60 0x561f49bbe650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f017177c740 10 client.0 _ll_register_callbacks cb 0x561f49b11ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f016cf2d640 1 -- 172.21.15.31:0/1654704281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6b60 msgr2=0x561f49bbe650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f016cf2d640 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6b60 0x561f49bbe650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f016cf2d640 1 -- 172.21.15.31:0/1654704281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 msgr2=0x561f49bbe110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f016cf2d640 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bbe110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.798+0000 7f016cf2d640 1 -- 172.21.15.31:0/1654704281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f498fb620 con 0x561f49bc1230 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.799+0000 7f0167fff640 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bbe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.799+0000 7f016cf2d640 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bc1230 0x561f49bbeb90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f0160009ca0 tx=0x7f0160004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9303d580755edafd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.799+0000 7f01477fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.799+0000 7f01477fe640 1 -- 172.21.15.31:0/1654704281 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016005c060 con 0x561f49bc1230 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.799+0000 7f017177c740 1 -- 172.21.15.31:0/1654704281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0160036040 con 0x561f49bc1230 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.799+0000 7f01477fe640 1 -- 172.21.15.31:0/1654704281 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f0160003fc0 con 0x561f49bc1230 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.799+0000 7f01477fe640 1 -- 172.21.15.31:0/1654704281 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f0160053040 con 0x561f49bc1230 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.799+0000 7f017177c740 10 client.17952 fetch_fsmap learned FSMap version 50 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.799+0000 7f017177c740 1 -- 172.21.15.31:0/1654704281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x561f49bc3fc0 con 0x561f49bc1230 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.800+0000 7f01477fe640 1 -- 172.21.15.31:0/1654704281 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0160036040 con 0x561f49bc1230 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.800+0000 7f01477fe640 1 -- 172.21.15.31:0/1654704281 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f0160030070 con 0x561f49bc1230 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.800+0000 7f017177c740 10 client.17952 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.800+0000 7f017177c740 10 client.17952 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.800+0000 7f017177c740 20 client.17952 populate_metadata read hostname 'smithi031' 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.800+0000 7f017177c740 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561f49bd9aa0 0x561f49bdbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.800+0000 7f017177c740 4 client.17952 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:17:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.800+0000 7f017177c740 1 -- 172.21.15.31:0/1654704281 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f49b281f0 con 0x561f49bd9aa0 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.800+0000 7f0167fff640 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561f49bd9aa0 0x561f49bdbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.801+0000 7f01477fe640 10 client.17952 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.801+0000 7f0167fff640 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561f49bd9aa0 0x561f49bdbe90 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f612942ca339835 server_cookie=926429a51245da66 in_seq=0 out_seq=0 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f01477fe640 1 -- 172.21.15.31:0/1654704281 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f49b281f0 con 0x561f49bd9aa0 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f01477fe640 10 client.17952 handle_command_reply: tid=0 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f017177c740 1 client.17952 shutdown 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f017177c740 1 -- 172.21.15.31:0/1654704281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bc1230 msgr2=0x561f49bbeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f017177c740 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bc1230 0x561f49bbeb90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f0160009ca0 tx=0x7f0160004730 comp rx=0 tx=0).stop 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f017177c740 1 -- 172.21.15.31:0/1654704281 shutdown_connections 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f017177c740 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561f49bd9aa0 0x561f49bdbe90 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f017177c740 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bc1230 0x561f49bbeb90 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f017177c740 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6b60 0x561f49bbe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f017177c740 1 --2- 172.21.15.31:0/1654704281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bbe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f017177c740 1 -- 172.21.15.31:0/1654704281 >> 172.21.15.31:0/1654704281 conn(0x561f49bb2560 msgr2=0x561f49bb8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f01477fe640 0 client.17952 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.802+0000 7f017177c740 1 -- 172.21.15.31:0/1654704281 shutdown_connections 2024-04-25T02:17:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.803+0000 7f017177c740 1 -- 172.21.15.31:0/1654704281 wait complete. 2024-04-25T02:17:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.803+0000 7f017177c740 20 client.17952 trim_cache size 0 max 16384 2024-04-25T02:17:14.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.829+0000 7f017177c740 1 Processor -- start 2024-04-25T02:17:14.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.829+0000 7f017177c740 1 -- start start 2024-04-25T02:17:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.830+0000 7f017177c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bc0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.830+0000 7f017177c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bb6b60 0x561f49bbd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.830+0000 7f017177c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bc1230 0x561f49be1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.830+0000 7f017177c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f01681ab100 con 0x561f49bb6b60 2024-04-25T02:17:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.830+0000 7f017177c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01681acc40 con 0x561f49bb6240 2024-04-25T02:17:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.830+0000 7f017177c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f01681ab7d0 con 0x561f49bc1230 2024-04-25T02:17:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.830+0000 7f01677fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bb6b60 0x561f49bbd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.830+0000 7f0167fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bc0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.830+0000 7f0167fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bc0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47158/0 (socket says 172.21.15.31:47158) 2024-04-25T02:17:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.830+0000 7f0167fff640 1 -- 172.21.15.31:0/2251477625 learned_addr learned my addr 172.21.15.31:0/2251477625 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.830+0000 7f016cf2d640 1 --2- 172.21.15.31:0/2251477625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bc1230 0x561f49be1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.831+0000 7f0167fff640 1 -- 172.21.15.31:0/2251477625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bc1230 msgr2=0x561f49be1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.831+0000 7f0167fff640 1 --2- 172.21.15.31:0/2251477625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bc1230 0x561f49be1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.831+0000 7f0167fff640 1 -- 172.21.15.31:0/2251477625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bb6b60 msgr2=0x561f49bbd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.831+0000 7f0167fff640 1 --2- 172.21.15.31:0/2251477625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bb6b60 0x561f49bbd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.831+0000 7f0167fff640 1 -- 172.21.15.31:0/2251477625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f49bc3990 con 0x561f49bb6240 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.831+0000 7f016cf2d640 1 --2- 172.21.15.31:0/2251477625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bc1230 0x561f49be1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.831+0000 7f0167fff640 1 --2- 172.21.15.31:0/2251477625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bc0840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f015809b900 tx=0x7f01580d5550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=217fa6f20154bd9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.831+0000 7f014dffb640 1 -- 172.21.15.31:0/2251477625 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015809b740 con 0x561f49bb6240 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.831+0000 7f014dffb640 1 -- 172.21.15.31:0/2251477625 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0158005310 con 0x561f49bb6240 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.831+0000 7f014dffb640 1 -- 172.21.15.31:0/2251477625 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015809b560 con 0x561f49bb6240 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.832+0000 7f017177c740 1 -- 172.21.15.31:0/2251477625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 msgr2=0x561f49bc0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.832+0000 7f017177c740 1 --2- 172.21.15.31:0/2251477625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bc0840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f015809b900 tx=0x7f01580d5550 comp rx=0 tx=0).stop 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.832+0000 7f017177c740 1 -- 172.21.15.31:0/2251477625 shutdown_connections 2024-04-25T02:17:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.832+0000 7f017177c740 1 --2- 172.21.15.31:0/2251477625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bc1230 0x561f49be1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.832+0000 7f017177c740 1 --2- 172.21.15.31:0/2251477625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bb6b60 0x561f49bbd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.832+0000 7f017177c740 1 --2- 172.21.15.31:0/2251477625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6240 0x561f49bc0840 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.832+0000 7f017177c740 1 -- 172.21.15.31:0/2251477625 >> 172.21.15.31:0/2251477625 conn(0x561f49bb2560 msgr2=0x561f49bbc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.832+0000 7f017177c740 1 -- 172.21.15.31:0/2251477625 shutdown_connections 2024-04-25T02:17:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.832+0000 7f017177c740 1 -- 172.21.15.31:0/2251477625 wait complete. 2024-04-25T02:17:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f017177c740 1 -- start start 2024-04-25T02:17:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f017177c740 1 Processor -- start 2024-04-25T02:17:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f017177c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6240 0x561f49bcced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f017177c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6b60 0x561f49bcd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f017177c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bc1230 0x561f49c37ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f017177c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f01681ac570 con 0x561f49bc1230 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f017177c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01681abea0 con 0x561f49bb6b60 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f017177c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f01681a9100 con 0x561f49bb6240 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f017177c740 10 client.0 _ll_register_callbacks cb 0x7f01580d5420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f0167fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6240 0x561f49bcced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f016cf2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bc1230 0x561f49c37ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f0167fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6240 0x561f49bcced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:40964/0 (socket says 172.21.15.31:40964) 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f0167fff640 1 -- 172.21.15.31:0/3644236757 learned_addr learned my addr 172.21.15.31:0/3644236757 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.833+0000 7f01677fe640 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6b60 0x561f49bcd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f0167fff640 1 -- 172.21.15.31:0/3644236757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6b60 msgr2=0x561f49bcd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f0167fff640 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6b60 0x561f49bcd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f0167fff640 1 -- 172.21.15.31:0/3644236757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bc1230 msgr2=0x561f49c37ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f0167fff640 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bc1230 0x561f49c37ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f0167fff640 1 -- 172.21.15.31:0/3644236757 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f49c38480 con 0x561f49bb6240 2024-04-25T02:17:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f0167fff640 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6240 0x561f49bcced0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f01580cce90 tx=0x7f015809ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a2e5432467d9e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f01477fe640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f01477fe640 1 -- 172.21.15.31:0/3644236757 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015809b560 con 0x561f49bb6240 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f01477fe640 1 -- 172.21.15.31:0/3644236757 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f01580046c0 con 0x561f49bb6240 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f017177c740 1 -- 172.21.15.31:0/3644236757 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561f49b22540 con 0x561f49bb6240 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f01477fe640 1 -- 172.21.15.31:0/3644236757 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f015809b740 con 0x561f49bb6240 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f017177c740 10 client.27298 fetch_fsmap learned FSMap version 50 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f017177c740 1 -- 172.21.15.31:0/3644236757 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x561f49c3f840 con 0x561f49bb6240 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.834+0000 7f01477fe640 1 -- 172.21.15.31:0/3644236757 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561f49b22540 con 0x561f49bb6240 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.835+0000 7f01477fe640 1 -- 172.21.15.31:0/3644236757 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f0158098270 con 0x561f49bb6240 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.835+0000 7f017177c740 10 client.27298 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.835+0000 7f017177c740 10 client.27298 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.835+0000 7f017177c740 20 client.27298 populate_metadata read hostname 'smithi031' 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.835+0000 7f017177c740 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561f49c3fe50 0x561f49c42270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.835+0000 7f017177c740 4 client.27298 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:17:14.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.835+0000 7f017177c740 1 -- 172.21.15.31:0/3644236757 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f49aa08e0 con 0x561f49c3fe50 2024-04-25T02:17:14.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.835+0000 7f01677fe640 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561f49c3fe50 0x561f49c42270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:14.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.836+0000 7f01677fe640 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561f49c3fe50 0x561f49c42270 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eeb5ba6efef27814 server_cookie=c8f535ef917e845a in_seq=0 out_seq=0 2024-04-25T02:17:14.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.836+0000 7f01477fe640 10 client.27298 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.909+0000 7f01477fe640 1 -- 172.21.15.31:0/3644236757 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f49aa08e0 con 0x561f49c3fe50 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.909+0000 7f01477fe640 10 client.27298 handle_command_reply: tid=0 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.909+0000 7f017177c740 1 client.27298 shutdown 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f017177c740 1 -- 172.21.15.31:0/3644236757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6240 msgr2=0x561f49bcced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f017177c740 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6240 0x561f49bcced0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f01580cce90 tx=0x7f015809ba90 comp rx=0 tx=0).stop 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f017177c740 1 -- 172.21.15.31:0/3644236757 shutdown_connections 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f017177c740 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x561f49c3fe50 0x561f49c42270 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f017177c740 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f49bc1230 0x561f49c37ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f017177c740 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f49bb6b60 0x561f49bcd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f017177c740 1 --2- 172.21.15.31:0/3644236757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561f49bb6240 0x561f49bcced0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f01477fe640 0 client.27298 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f017177c740 1 -- 172.21.15.31:0/3644236757 >> 172.21.15.31:0/3644236757 conn(0x561f49bb2560 msgr2=0x561f49bb7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f017177c740 1 -- 172.21.15.31:0/3644236757 shutdown_connections 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f017177c740 1 -- 172.21.15.31:0/3644236757 wait complete. 2024-04-25T02:17:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f017177c740 20 client.27298 trim_cache size 0 max 16384 2024-04-25T02:17:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:17:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:17:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:17:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:17:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f016e990640 1 -- 172.21.15.31:0/3458593988 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f01500a2e20 msgr2=0x7f01500a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f016e990640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f01500a2e20 0x7f01500a52e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f01600317f0 tx=0x7f016004b000 comp rx=0 tx=0).stop 2024-04-25T02:17:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f016e990640 1 -- 172.21.15.31:0/3458593988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0168100d50 msgr2=0x7f01681980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.910+0000 7f016e990640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0168100d50 0x7f01681980e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f015800e9f0 tx=0x7f015800edf0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.911+0000 7f016e990640 1 -- 172.21.15.31:0/3458593988 shutdown_connections 2024-04-25T02:17:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.911+0000 7f016e990640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f01500a2e20 0x7f01500a52e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.911+0000 7f016e990640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0168106840 0x7f016819c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.911+0000 7f016e990640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0168101710 0x7f0168198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.911+0000 7f016e990640 1 --2- 172.21.15.31:0/3458593988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0168100d50 0x7f01681980e0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.911+0000 7f016e990640 1 -- 172.21.15.31:0/3458593988 >> 172.21.15.31:0/3458593988 conn(0x7f0168076bd0 msgr2=0x7f01680fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.911+0000 7f016e990640 1 -- 172.21.15.31:0/3458593988 shutdown_connections 2024-04-25T02:17:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:14.911+0000 7f016e990640 1 -- 172.21.15.31:0/3458593988 wait complete. 2024-04-25T02:17:15.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 91587 8.80 MB/sec execute 439 sec latency 109.852 ms 2024-04-25T02:17:16.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 91754 8.80 MB/sec execute 440 sec latency 91.926 ms 2024-04-25T02:17:16.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:16 smithi040 ceph-mon[32418]: pgmap v719: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 266 op/s 2024-04-25T02:17:16.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:16 smithi134 ceph-mon[33098]: pgmap v719: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 266 op/s 2024-04-25T02:17:16.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:16 smithi031 ceph-mon[29204]: pgmap v719: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 266 op/s 2024-04-25T02:17:17.107 INFO:tasks.workunit.client.0.smithi031.stdout: 10 91921 8.79 MB/sec execute 441 sec latency 81.683 ms 2024-04-25T02:17:18.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 92063 8.79 MB/sec execute 442 sec latency 79.113 ms 2024-04-25T02:17:18.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:18 smithi040 ceph-mon[32418]: pgmap v720: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 266 op/s 2024-04-25T02:17:18.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:18 smithi134 ceph-mon[33098]: pgmap v720: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 266 op/s 2024-04-25T02:17:18.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:18 smithi031 ceph-mon[29204]: pgmap v720: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 266 op/s 2024-04-25T02:17:19.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 92244 8.79 MB/sec execute 443 sec latency 91.841 ms 2024-04-25T02:17:20.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 92396 8.79 MB/sec execute 444 sec latency 167.668 ms 2024-04-25T02:17:20.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:20 smithi040 ceph-mon[32418]: pgmap v721: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 266 op/s 2024-04-25T02:17:20.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:20 smithi134 ceph-mon[33098]: pgmap v721: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 266 op/s 2024-04-25T02:17:20.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:20 smithi031 ceph-mon[29204]: pgmap v721: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 266 op/s 2024-04-25T02:17:21.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 92571 8.80 MB/sec execute 445 sec latency 227.230 ms 2024-04-25T02:17:22.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 92747 8.79 MB/sec execute 446 sec latency 84.248 ms 2024-04-25T02:17:22.414 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:22 smithi040 ceph-mon[32418]: pgmap v722: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.6 MiB/s wr, 345 op/s 2024-04-25T02:17:22.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:22 smithi134 ceph-mon[33098]: pgmap v722: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.6 MiB/s wr, 345 op/s 2024-04-25T02:17:22.493 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:22 smithi031 ceph-mon[29204]: pgmap v722: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.6 MiB/s wr, 345 op/s 2024-04-25T02:17:23.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 92922 8.80 MB/sec execute 447 sec latency 59.711 ms 2024-04-25T02:17:24.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 93086 8.80 MB/sec execute 448 sec latency 67.188 ms 2024-04-25T02:17:24.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:24 smithi040 ceph-mon[32418]: pgmap v723: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.0 MiB/s wr, 244 op/s 2024-04-25T02:17:24.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:24 smithi134 ceph-mon[33098]: pgmap v723: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.0 MiB/s wr, 244 op/s 2024-04-25T02:17:24.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:24 smithi031 ceph-mon[29204]: pgmap v723: 97 pgs: 97 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.0 MiB/s wr, 244 op/s 2024-04-25T02:17:25.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 93245 8.81 MB/sec execute 449 sec latency 58.014 ms 2024-04-25T02:17:26.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 93381 8.80 MB/sec execute 450 sec latency 88.250 ms 2024-04-25T02:17:26.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:26 smithi040 ceph-mon[32418]: pgmap v724: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.8 MiB/s wr, 351 op/s 2024-04-25T02:17:26.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:26 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:17:26.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:26 smithi134 ceph-mon[33098]: pgmap v724: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.8 MiB/s wr, 351 op/s 2024-04-25T02:17:26.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:26 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:17:26.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:26 smithi031 ceph-mon[29204]: pgmap v724: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.8 MiB/s wr, 351 op/s 2024-04-25T02:17:26.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:26 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:17:27.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 93569 8.82 MB/sec execute 451 sec latency 62.484 ms 2024-04-25T02:17:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.036+0000 7f637c79e640 1 -- 172.21.15.31:0/4103384250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6374100670 msgr2=0x7f6374100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.036+0000 7f637c79e640 1 --2- 172.21.15.31:0/4103384250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6374100670 0x7f6374100af0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f6364009890 tx=0x7f636402a360 comp rx=0 tx=0).stop 2024-04-25T02:17:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.037+0000 7f637c79e640 1 -- 172.21.15.31:0/4103384250 shutdown_connections 2024-04-25T02:17:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.037+0000 7f637c79e640 1 --2- 172.21.15.31:0/4103384250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6374101030 0x7f637410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.037+0000 7f637c79e640 1 --2- 172.21.15.31:0/4103384250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6374100670 0x7f6374100af0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.037+0000 7f637c79e640 1 --2- 172.21.15.31:0/4103384250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6374106630 0x7f6374106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.037+0000 7f637c79e640 1 -- 172.21.15.31:0/4103384250 >> 172.21.15.31:0/4103384250 conn(0x7f63740777c0 msgr2=0x7f6374077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.037+0000 7f637c79e640 1 -- 172.21.15.31:0/4103384250 shutdown_connections 2024-04-25T02:17:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.037+0000 7f637c79e640 1 -- 172.21.15.31:0/4103384250 wait complete. 2024-04-25T02:17:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f637c79e640 1 Processor -- start 2024-04-25T02:17:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f637c79e640 1 -- start start 2024-04-25T02:17:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f637c79e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6374100670 0x7f637419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f637c79e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6374101030 0x7f637419cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f637c79e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6374106630 0x7f6374196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f6379d39640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6374101030 0x7f637419cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f6379d39640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6374101030 0x7f637419cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34364/0 (socket says 172.21.15.31:34364) 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f6379d39640 1 -- 172.21.15.31:0/3255902498 learned_addr learned my addr 172.21.15.31:0/3255902498 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f637c79e640 1 -- 172.21.15.31:0/3255902498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6374110900 con 0x7f6374101030 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f637c79e640 1 -- 172.21.15.31:0/3255902498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6374110780 con 0x7f6374100670 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f637c79e640 1 -- 172.21.15.31:0/3255902498 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6374110a80 con 0x7f6374106630 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.038+0000 7f637a53a640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6374100670 0x7f637419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.039+0000 7f6379d39640 1 -- 172.21.15.31:0/3255902498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6374106630 msgr2=0x7f6374196a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.039+0000 7f6379d39640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6374106630 0x7f6374196a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.039+0000 7f6379d39640 1 -- 172.21.15.31:0/3255902498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6374100670 msgr2=0x7f637419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.039+0000 7f6379d39640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6374100670 0x7f637419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.039+0000 7f6379d39640 1 -- 172.21.15.31:0/3255902498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6374197200 con 0x7f6374101030 2024-04-25T02:17:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.039+0000 7f6379d39640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6374101030 0x7f637419cdc0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6364005d00 tx=0x7f6364030b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.040+0000 7f636b7fe640 1 -- 172.21.15.31:0/3255902498 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6364040020 con 0x7f6374101030 2024-04-25T02:17:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.040+0000 7f636b7fe640 1 -- 172.21.15.31:0/3255902498 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6364030340 con 0x7f6374101030 2024-04-25T02:17:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.040+0000 7f636b7fe640 1 -- 172.21.15.31:0/3255902498 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6364037040 con 0x7f6374101030 2024-04-25T02:17:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.040+0000 7f637c79e640 1 -- 172.21.15.31:0/3255902498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6374197490 con 0x7f6374101030 2024-04-25T02:17:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.040+0000 7f637c79e640 1 -- 172.21.15.31:0/3255902498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6374197870 con 0x7f6374101030 2024-04-25T02:17:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.044+0000 7f636b7fe640 1 -- 172.21.15.31:0/3255902498 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6364043020 con 0x7f6374101030 2024-04-25T02:17:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.044+0000 7f637f584740 1 Processor -- start 2024-04-25T02:17:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.044+0000 7f637f584740 1 -- start start 2024-04-25T02:17:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637f584740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035613a50 0x556035613e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637f584740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035614370 0x55603561e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637f584740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55603561ea40 0x556035620e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637f584740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55603556fcb0 con 0x556035614370 2024-04-25T02:17:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637f584740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5560355185b0 con 0x556035613a50 2024-04-25T02:17:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637f584740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55603556bc70 con 0x55603561ea40 2024-04-25T02:17:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637ad3b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035613a50 0x556035613e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637ad3b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035613a50 0x556035613e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47526/0 (socket says 172.21.15.31:47526) 2024-04-25T02:17:28.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637ad3b640 1 -- 172.21.15.31:0/2457513325 learned_addr learned my addr 172.21.15.31:0/2457513325 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637a53a640 1 --2- 172.21.15.31:0/2457513325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035614370 0x55603561e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f6379d39640 1 --2- 172.21.15.31:0/2457513325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55603561ea40 0x556035620e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637ad3b640 1 -- 172.21.15.31:0/2457513325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55603561ea40 msgr2=0x556035620e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637ad3b640 1 --2- 172.21.15.31:0/2457513325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55603561ea40 0x556035620e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637ad3b640 1 -- 172.21.15.31:0/2457513325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035614370 msgr2=0x55603561e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637ad3b640 1 --2- 172.21.15.31:0/2457513325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035614370 0x55603561e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.045+0000 7f637ad3b640 1 -- 172.21.15.31:0/2457513325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556035284210 con 0x556035613a50 2024-04-25T02:17:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.046+0000 7f637ad3b640 1 --2- 172.21.15.31:0/2457513325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035613a50 0x556035613e30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6370009bd0 tx=0x7f637000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41d7091811d13312 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.046+0000 7f636b7fe640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f63600a2f50 0x7f63600a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.046+0000 7f6368ff9640 1 -- 172.21.15.31:0/2457513325 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6370005020 con 0x556035613a50 2024-04-25T02:17:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.046+0000 7f636b7fe640 1 -- 172.21.15.31:0/3255902498 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f63640ef4a0 con 0x7f6374101030 2024-04-25T02:17:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.046+0000 7f6368ff9640 1 -- 172.21.15.31:0/2457513325 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63700047a0 con 0x556035613a50 2024-04-25T02:17:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.046+0000 7f637a53a640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f63600a2f50 0x7f63600a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.047+0000 7f6368ff9640 1 -- 172.21.15.31:0/2457513325 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6370010690 con 0x556035613a50 2024-04-25T02:17:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.047+0000 7f637f584740 1 -- 172.21.15.31:0/2457513325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035613a50 msgr2=0x556035613e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.047+0000 7f637a53a640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f63600a2f50 0x7f63600a5410 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x556035615010 tx=0x7f636c011040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.047+0000 7f637f584740 1 --2- 172.21.15.31:0/2457513325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035613a50 0x556035613e30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6370009bd0 tx=0x7f637000e720 comp rx=0 tx=0).stop 2024-04-25T02:17:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.047+0000 7f637f584740 1 -- 172.21.15.31:0/2457513325 shutdown_connections 2024-04-25T02:17:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.047+0000 7f637f584740 1 --2- 172.21.15.31:0/2457513325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55603561ea40 0x556035620e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.047+0000 7f637f584740 1 --2- 172.21.15.31:0/2457513325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035614370 0x55603561e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.047+0000 7f637f584740 1 --2- 172.21.15.31:0/2457513325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035613a50 0x556035613e30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.047+0000 7f637f584740 1 -- 172.21.15.31:0/2457513325 >> 172.21.15.31:0/2457513325 conn(0x55603560fd70 msgr2=0x556035610150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.047+0000 7f637f584740 1 -- 172.21.15.31:0/2457513325 shutdown_connections 2024-04-25T02:17:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.048+0000 7f637f584740 1 -- 172.21.15.31:0/2457513325 wait complete. 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.048+0000 7f637f584740 1 -- start start 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.048+0000 7f637f584740 1 Processor -- start 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.048+0000 7f637f584740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 0x55603561b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f637f584740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 0x55603561be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f637f584740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55603561ea40 0x55603561c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f637f584740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5560356218d0 con 0x55603561ea40 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f637f584740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55603552bb50 con 0x556035614370 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f637f584740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556035621a50 con 0x556035613a50 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f637f584740 10 client.0 _ll_register_callbacks cb 0x556035566ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f637a53a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 0x55603561b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f6379d39640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 0x55603561be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f6379d39640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 0x55603561be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:47540/0 (socket says 172.21.15.31:47540) 2024-04-25T02:17:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f6379d39640 1 -- 172.21.15.31:0/861425652 learned_addr learned my addr 172.21.15.31:0/861425652 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f637a53a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 0x55603561b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:54974/0 (socket says 172.21.15.31:54974) 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f637ad3b640 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55603561ea40 0x55603561c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f6379d39640 1 -- 172.21.15.31:0/861425652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 msgr2=0x55603561b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f6379d39640 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 0x55603561b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f6379d39640 1 -- 172.21.15.31:0/861425652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55603561ea40 msgr2=0x55603561c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f6379d39640 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55603561ea40 0x55603561c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f6379d39640 1 -- 172.21.15.31:0/861425652 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556035350620 con 0x556035614370 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f635dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.049+0000 7f6379d39640 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 0x55603561be80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6364042fd0 tx=0x7f636402a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0eec4a6f3a96204 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.050+0000 7f635dffb640 1 -- 172.21.15.31:0/861425652 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63640bb5d0 con 0x556035614370 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.050+0000 7f637f584740 1 -- 172.21.15.31:0/861425652 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63700047a0 con 0x556035614370 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.050+0000 7f635dffb640 1 -- 172.21.15.31:0/861425652 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f63640307a0 con 0x556035614370 2024-04-25T02:17:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.050+0000 7f635dffb640 1 -- 172.21.15.31:0/861425652 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f63640bb3f0 con 0x556035614370 2024-04-25T02:17:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.050+0000 7f637f584740 10 client.27536 fetch_fsmap learned FSMap version 51 2024-04-25T02:17:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.050+0000 7f637f584740 1 -- 172.21.15.31:0/861425652 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55603562d880 con 0x556035614370 2024-04-25T02:17:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.050+0000 7f635dffb640 1 -- 172.21.15.31:0/861425652 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63700047a0 con 0x556035614370 2024-04-25T02:17:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.050+0000 7f635dffb640 1 -- 172.21.15.31:0/861425652 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f63640ece30 con 0x556035614370 2024-04-25T02:17:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.051+0000 7f637f584740 10 client.27536 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:17:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.051+0000 7f637f584740 10 client.27536 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:17:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.051+0000 7f637f584740 20 client.27536 populate_metadata read hostname 'smithi031' 2024-04-25T02:17:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.051+0000 7f637f584740 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55603562ece0 0x5560356310d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.051+0000 7f637f584740 4 client.27536 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:17:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.051+0000 7f637f584740 1 -- 172.21.15.31:0/861425652 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55603557d1f0 con 0x55603562ece0 2024-04-25T02:17:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.051+0000 7f637a53a640 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55603562ece0 0x5560356310d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.052+0000 7f637a53a640 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55603562ece0 0x5560356310d0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa987ea5f8e7612b server_cookie=86f40048deaf846d in_seq=0 out_seq=0 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.052+0000 7f635dffb640 10 client.27536 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.053+0000 7f635dffb640 1 -- 172.21.15.31:0/861425652 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55603557d1f0 con 0x55603562ece0 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.053+0000 7f635dffb640 10 client.27536 handle_command_reply: tid=0 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.053+0000 7f637f584740 1 client.27536 shutdown 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.053+0000 7f637f584740 1 -- 172.21.15.31:0/861425652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 msgr2=0x55603561be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.053+0000 7f637f584740 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 0x55603561be80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6364042fd0 tx=0x7f636402a360 comp rx=0 tx=0).stop 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.054+0000 7f637f584740 1 -- 172.21.15.31:0/861425652 shutdown_connections 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.054+0000 7f637f584740 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55603562ece0 0x5560356310d0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.054+0000 7f637f584740 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55603561ea40 0x55603561c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.054+0000 7f637f584740 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 0x55603561be80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.054+0000 7f635dffb640 0 client.27536 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.054+0000 7f637f584740 1 --2- 172.21.15.31:0/861425652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 0x55603561b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.054+0000 7f637f584740 1 -- 172.21.15.31:0/861425652 >> 172.21.15.31:0/861425652 conn(0x55603560fd70 msgr2=0x556035615a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.054+0000 7f637f584740 1 -- 172.21.15.31:0/861425652 shutdown_connections 2024-04-25T02:17:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.054+0000 7f637f584740 1 -- 172.21.15.31:0/861425652 wait complete. 2024-04-25T02:17:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.054+0000 7f637f584740 20 client.27536 trim_cache size 0 max 16384 2024-04-25T02:17:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.085+0000 7f637f584740 1 Processor -- start 2024-04-25T02:17:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.085+0000 7f637f584740 1 -- start start 2024-04-25T02:17:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637f584740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035613a50 0x55603561e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637f584740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 0x55603562a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637f584740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55603561ea40 0x55603561aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637f584740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f63741aadd0 con 0x556035613a50 2024-04-25T02:17:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637f584740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63741ac910 con 0x556035614370 2024-04-25T02:17:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637f584740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f63741ab4a0 con 0x55603561ea40 2024-04-25T02:17:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637a53a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035613a50 0x55603561e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637a53a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035613a50 0x55603561e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34402/0 (socket says 172.21.15.31:34402) 2024-04-25T02:17:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637a53a640 1 -- 172.21.15.31:0/2992036019 learned_addr learned my addr 172.21.15.31:0/2992036019 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f6379d39640 1 --2- 172.21.15.31:0/2992036019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 0x55603562a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637ad3b640 1 --2- 172.21.15.31:0/2992036019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55603561ea40 0x55603561aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637a53a640 1 -- 172.21.15.31:0/2992036019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55603561ea40 msgr2=0x55603561aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637a53a640 1 --2- 172.21.15.31:0/2992036019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55603561ea40 0x55603561aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637a53a640 1 -- 172.21.15.31:0/2992036019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 msgr2=0x55603562a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637a53a640 1 --2- 172.21.15.31:0/2992036019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 0x55603562a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637a53a640 1 -- 172.21.15.31:0/2992036019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556035631730 con 0x556035613a50 2024-04-25T02:17:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.086+0000 7f637a53a640 1 --2- 172.21.15.31:0/2992036019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035613a50 0x55603561e070 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f636c00ffd0 tx=0x7f636c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e4c828528ff1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.087+0000 7f635ffff640 1 -- 172.21.15.31:0/2992036019 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c02d560 con 0x556035613a50 2024-04-25T02:17:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.087+0000 7f635ffff640 1 -- 172.21.15.31:0/2992036019 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f636c033070 con 0x556035613a50 2024-04-25T02:17:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.087+0000 7f635ffff640 1 -- 172.21.15.31:0/2992036019 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c028440 con 0x556035613a50 2024-04-25T02:17:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.088+0000 7f637f584740 1 -- 172.21.15.31:0/2992036019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035613a50 msgr2=0x55603561e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.088+0000 7f637f584740 1 --2- 172.21.15.31:0/2992036019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035613a50 0x55603561e070 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f636c00ffd0 tx=0x7f636c017000 comp rx=0 tx=0).stop 2024-04-25T02:17:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.088+0000 7f637f584740 1 -- 172.21.15.31:0/2992036019 shutdown_connections 2024-04-25T02:17:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.088+0000 7f637f584740 1 --2- 172.21.15.31:0/2992036019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55603561ea40 0x55603561aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.088+0000 7f637f584740 1 --2- 172.21.15.31:0/2992036019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556035614370 0x55603562a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.088+0000 7f637f584740 1 --2- 172.21.15.31:0/2992036019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035613a50 0x55603561e070 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.088+0000 7f637f584740 1 -- 172.21.15.31:0/2992036019 >> 172.21.15.31:0/2992036019 conn(0x55603560fd70 msgr2=0x55603561b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.088+0000 7f637f584740 1 -- 172.21.15.31:0/2992036019 shutdown_connections 2024-04-25T02:17:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.088+0000 7f637f584740 1 -- 172.21.15.31:0/2992036019 wait complete. 2024-04-25T02:17:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.088+0000 7f637f584740 1 -- start start 2024-04-25T02:17:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637f584740 1 Processor -- start 2024-04-25T02:17:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637f584740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 0x556035618870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637f584740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035614370 0x556035618db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637f584740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55603561ea40 0x5560356192f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637f584740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f63741ac240 con 0x556035614370 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637f584740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63741abb70 con 0x55603561ea40 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637f584740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f63741a8df0 con 0x556035613a50 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637f584740 10 client.0 _ll_register_callbacks cb 0x7f63640f2c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637a53a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 0x556035618870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637ad3b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55603561ea40 0x5560356192f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637a53a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 0x556035618870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:55006/0 (socket says 172.21.15.31:55006) 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f6379d39640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035614370 0x556035618db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.089+0000 7f637a53a640 1 -- 172.21.15.31:0/3959919078 learned_addr learned my addr 172.21.15.31:0/3959919078 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.090+0000 7f637ad3b640 1 -- 172.21.15.31:0/3959919078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 msgr2=0x556035618870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.090+0000 7f637ad3b640 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 0x556035618870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.090+0000 7f637ad3b640 1 -- 172.21.15.31:0/3959919078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035614370 msgr2=0x556035618db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.090+0000 7f637ad3b640 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035614370 0x556035618db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.090+0000 7f637ad3b640 1 -- 172.21.15.31:0/3959919078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560356166d0 con 0x55603561ea40 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.090+0000 7f6379d39640 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035614370 0x556035618db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.090+0000 7f635dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.090+0000 7f637ad3b640 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55603561ea40 0x5560356192f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x556035616330 tx=0x7f6370004870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6090c85c0062a9e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.090+0000 7f635dffb640 1 -- 172.21.15.31:0/3959919078 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6370027030 con 0x55603561ea40 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.090+0000 7f635dffb640 1 -- 172.21.15.31:0/3959919078 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f6370004020 con 0x55603561ea40 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.090+0000 7f637f584740 1 -- 172.21.15.31:0/3959919078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f636c033070 con 0x55603561ea40 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f635dffb640 1 -- 172.21.15.31:0/3959919078 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f6370031050 con 0x55603561ea40 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f637f584740 10 client.27542 fetch_fsmap learned FSMap version 51 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f637f584740 1 -- 172.21.15.31:0/3959919078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x5560356955f0 con 0x55603561ea40 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f635dffb640 1 -- 172.21.15.31:0/3959919078 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f636c033070 con 0x55603561ea40 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f635dffb640 1 -- 172.21.15.31:0/3959919078 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f63700035f0 con 0x55603561ea40 2024-04-25T02:17:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f637f584740 10 client.27542 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:17:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f637f584740 10 client.27542 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:17:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f637f584740 20 client.27542 populate_metadata read hostname 'smithi031' 2024-04-25T02:17:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f637f584740 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560356959e0 0x556035697dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f637f584740 4 client.27542 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:17:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f637f584740 1 -- 172.21.15.31:0/3959919078 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556035577540 con 0x5560356959e0 2024-04-25T02:17:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.091+0000 7f637a53a640 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560356959e0 0x556035697dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.092+0000 7f637a53a640 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560356959e0 0x556035697dd0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c68e1c151d76393a server_cookie=b153f7b5ba0a3e0c in_seq=0 out_seq=0 2024-04-25T02:17:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.092+0000 7f635dffb640 10 client.27542 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:28.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 93732 8.81 MB/sec execute 452 sec latency 75.154 ms 2024-04-25T02:17:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f635dffb640 1 -- 172.21.15.31:0/3959919078 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556035577540 con 0x5560356959e0 2024-04-25T02:17:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f635dffb640 10 client.27542 handle_command_reply: tid=0 2024-04-25T02:17:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f637f584740 1 client.27542 shutdown 2024-04-25T02:17:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f637f584740 1 -- 172.21.15.31:0/3959919078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55603561ea40 msgr2=0x5560356192f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f637f584740 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55603561ea40 0x5560356192f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x556035616330 tx=0x7f6370004870 comp rx=0 tx=0).stop 2024-04-25T02:17:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f637f584740 1 -- 172.21.15.31:0/3959919078 shutdown_connections 2024-04-25T02:17:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f637f584740 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x5560356959e0 0x556035697dd0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f635dffb640 0 client.27542 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f637f584740 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55603561ea40 0x5560356192f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f637f584740 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556035614370 0x556035618db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f637f584740 1 --2- 172.21.15.31:0/3959919078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556035613a50 0x556035618870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.201+0000 7f637f584740 1 -- 172.21.15.31:0/3959919078 >> 172.21.15.31:0/3959919078 conn(0x55603560fd70 msgr2=0x556035615a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.202+0000 7f637f584740 1 -- 172.21.15.31:0/3959919078 shutdown_connections 2024-04-25T02:17:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.202+0000 7f637f584740 1 -- 172.21.15.31:0/3959919078 wait complete. 2024-04-25T02:17:28.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.203+0000 7f637f584740 20 client.27542 trim_cache size 0 max 16384 2024-04-25T02:17:28.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:17:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:17:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:17:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:17:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.210+0000 7f637c79e640 1 -- 172.21.15.31:0/3255902498 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f63600a2f50 msgr2=0x7f63600a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.210+0000 7f637c79e640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f63600a2f50 0x7f63600a5410 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x556035615010 tx=0x7f636c011040 comp rx=0 tx=0).stop 2024-04-25T02:17:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.210+0000 7f637c79e640 1 -- 172.21.15.31:0/3255902498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6374101030 msgr2=0x7f637419cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.210+0000 7f637c79e640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6374101030 0x7f637419cdc0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6364005d00 tx=0x7f6364030b20 comp rx=0 tx=0).stop 2024-04-25T02:17:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.210+0000 7f637c79e640 1 -- 172.21.15.31:0/3255902498 shutdown_connections 2024-04-25T02:17:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.210+0000 7f637c79e640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f63600a2f50 0x7f63600a5410 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.210+0000 7f637c79e640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6374106630 0x7f6374196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.210+0000 7f637c79e640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6374101030 0x7f637419cdc0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.210+0000 7f637c79e640 1 --2- 172.21.15.31:0/3255902498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6374100670 0x7f637419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.210+0000 7f637c79e640 1 -- 172.21.15.31:0/3255902498 >> 172.21.15.31:0/3255902498 conn(0x7f63740777c0 msgr2=0x7f6374103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.210+0000 7f637c79e640 1 -- 172.21.15.31:0/3255902498 shutdown_connections 2024-04-25T02:17:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:28.211+0000 7f637c79e640 1 -- 172.21.15.31:0/3255902498 wait complete. 2024-04-25T02:17:28.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:28 smithi040 ceph-mon[32418]: pgmap v725: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 227 op/s 2024-04-25T02:17:28.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:28 smithi134 ceph-mon[33098]: pgmap v725: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 227 op/s 2024-04-25T02:17:28.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:28 smithi031 ceph-mon[29204]: pgmap v725: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 227 op/s 2024-04-25T02:17:29.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 93884 8.82 MB/sec execute 453 sec latency 101.645 ms 2024-04-25T02:17:30.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 93998 8.80 MB/sec execute 454 sec latency 80.157 ms 2024-04-25T02:17:30.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:30 smithi040 ceph-mon[32418]: pgmap v726: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 227 op/s 2024-04-25T02:17:30.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:30 smithi134 ceph-mon[33098]: pgmap v726: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 227 op/s 2024-04-25T02:17:30.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:30 smithi031 ceph-mon[29204]: pgmap v726: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 227 op/s 2024-04-25T02:17:31.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 94169 8.81 MB/sec execute 455 sec latency 56.490 ms 2024-04-25T02:17:32.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 94335 8.81 MB/sec execute 456 sec latency 76.241 ms 2024-04-25T02:17:32.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:32 smithi040 ceph-mon[32418]: pgmap v727: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.8 MiB/s wr, 354 op/s 2024-04-25T02:17:32.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:32 smithi134 ceph-mon[33098]: pgmap v727: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.8 MiB/s wr, 354 op/s 2024-04-25T02:17:32.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:32 smithi031 ceph-mon[29204]: pgmap v727: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.8 MiB/s wr, 354 op/s 2024-04-25T02:17:33.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 94498 8.81 MB/sec execute 457 sec latency 64.101 ms 2024-04-25T02:17:34.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 94638 8.79 MB/sec execute 458 sec latency 73.032 ms 2024-04-25T02:17:34.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:34 smithi040 ceph-mon[32418]: pgmap v728: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.7 MiB/s wr, 233 op/s 2024-04-25T02:17:34.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:34 smithi134 ceph-mon[33098]: pgmap v728: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.7 MiB/s wr, 233 op/s 2024-04-25T02:17:34.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:34 smithi031 ceph-mon[29204]: pgmap v728: 97 pgs: 97 active+clean; 1.5 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.7 MiB/s wr, 233 op/s 2024-04-25T02:17:35.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 94785 8.79 MB/sec execute 459 sec latency 95.825 ms 2024-04-25T02:17:36.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 94952 8.79 MB/sec execute 460 sec latency 69.638 ms 2024-04-25T02:17:36.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:36 smithi040 ceph-mon[32418]: pgmap v729: 97 pgs: 97 active+clean; 1.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.4 MiB/s wr, 349 op/s 2024-04-25T02:17:36.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:36 smithi134 ceph-mon[33098]: pgmap v729: 97 pgs: 97 active+clean; 1.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.4 MiB/s wr, 349 op/s 2024-04-25T02:17:36.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:36 smithi031 ceph-mon[29204]: pgmap v729: 97 pgs: 97 active+clean; 1.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.4 MiB/s wr, 349 op/s 2024-04-25T02:17:37.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 95117 8.79 MB/sec execute 461 sec latency 54.909 ms 2024-04-25T02:17:38.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 95262 8.78 MB/sec execute 462 sec latency 50.972 ms 2024-04-25T02:17:38.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:38 smithi040 ceph-mon[32418]: pgmap v730: 97 pgs: 97 active+clean; 1.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-04-25T02:17:38.443 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:38 smithi134 ceph-mon[33098]: pgmap v730: 97 pgs: 97 active+clean; 1.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-04-25T02:17:38.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:38 smithi031 ceph-mon[29204]: pgmap v730: 97 pgs: 97 active+clean; 1.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-04-25T02:17:39.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 95432 8.77 MB/sec execute 463 sec latency 71.980 ms 2024-04-25T02:17:40.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 95635 8.79 MB/sec execute 464 sec latency 77.042 ms 2024-04-25T02:17:40.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:40 smithi040 ceph-mon[32418]: pgmap v731: 97 pgs: 97 active+clean; 1.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-04-25T02:17:40.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:40 smithi134 ceph-mon[33098]: pgmap v731: 97 pgs: 97 active+clean; 1.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-04-25T02:17:40.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:40 smithi031 ceph-mon[29204]: pgmap v731: 97 pgs: 97 active+clean; 1.5 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-04-25T02:17:41.108 INFO:tasks.workunit.client.0.smithi031.stdout: 10 95788 8.78 MB/sec execute 465 sec latency 82.733 ms 2024-04-25T02:17:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.334+0000 7f1717bdc640 1 -- 172.21.15.31:0/1656733291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17101029d0 msgr2=0x7f1710102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.334+0000 7f1717bdc640 1 --2- 172.21.15.31:0/1656733291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17101029d0 0x7f1710102db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f170c009900 tx=0x7f170c02a510 comp rx=0 tx=0).stop 2024-04-25T02:17:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.334+0000 7f1717bdc640 1 -- 172.21.15.31:0/1656733291 shutdown_connections 2024-04-25T02:17:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.334+0000 7f1717bdc640 1 --2- 172.21.15.31:0/1656733291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f171010bd10 0x7f171010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.334+0000 7f1717bdc640 1 --2- 172.21.15.31:0/1656733291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17101032f0 0x7f171010b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.334+0000 7f1717bdc640 1 --2- 172.21.15.31:0/1656733291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17101029d0 0x7f1710102db0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.334+0000 7f1717bdc640 1 -- 172.21.15.31:0/1656733291 >> 172.21.15.31:0/1656733291 conn(0x7f17100777c0 msgr2=0x7f1710077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.334+0000 7f1717bdc640 1 -- 172.21.15.31:0/1656733291 shutdown_connections 2024-04-25T02:17:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.334+0000 7f1717bdc640 1 -- 172.21.15.31:0/1656733291 wait complete. 2024-04-25T02:17:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.335+0000 7f1717bdc640 1 Processor -- start 2024-04-25T02:17:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.335+0000 7f1717bdc640 1 -- start start 2024-04-25T02:17:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.335+0000 7f1717bdc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17101029d0 0x7f1710197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.335+0000 7f1717bdc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17101032f0 0x7f1710198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.335+0000 7f1717bdc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f171010bd10 0x7f171019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.335+0000 7f1717bdc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f17101119c0 con 0x7f17101032f0 2024-04-25T02:17:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.335+0000 7f1717bdc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1710111840 con 0x7f171010bd10 2024-04-25T02:17:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.335+0000 7f1715978640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17101029d0 0x7f1710197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1717bdc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1710111b40 con 0x7f17101029d0 2024-04-25T02:17:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1715177640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17101032f0 0x7f1710198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1715978640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17101029d0 0x7f1710197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36242/0 (socket says 172.21.15.31:36242) 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1716179640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f171010bd10 0x7f171019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1715978640 1 -- 172.21.15.31:0/3968668949 learned_addr learned my addr 172.21.15.31:0/3968668949 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1716179640 1 -- 172.21.15.31:0/3968668949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17101029d0 msgr2=0x7f1710197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1716179640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17101029d0 0x7f1710197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1716179640 1 -- 172.21.15.31:0/3968668949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17101032f0 msgr2=0x7f1710198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1716179640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17101032f0 0x7f1710198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1716179640 1 -- 172.21.15.31:0/3968668949 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171019ceb0 con 0x7f171010bd10 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1715978640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17101029d0 0x7f1710197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.336+0000 7f1716179640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f171010bd10 0x7f171019c780 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f170800ec50 tx=0x7f170800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.337+0000 7f1706ffd640 1 -- 172.21.15.31:0/3968668949 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1708003b70 con 0x7f171010bd10 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.337+0000 7f1706ffd640 1 -- 172.21.15.31:0/3968668949 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1708003d10 con 0x7f171010bd10 2024-04-25T02:17:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.337+0000 7f1717bdc640 1 -- 172.21.15.31:0/3968668949 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171019d1a0 con 0x7f171010bd10 2024-04-25T02:17:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.337+0000 7f1706ffd640 1 -- 172.21.15.31:0/3968668949 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1708010760 con 0x7f171010bd10 2024-04-25T02:17:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.337+0000 7f1717bdc640 1 -- 172.21.15.31:0/3968668949 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17101a4a30 con 0x7f171010bd10 2024-04-25T02:17:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.340+0000 7f171a9c7740 1 Processor -- start 2024-04-25T02:17:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.340+0000 7f1706ffd640 1 -- 172.21.15.31:0/3968668949 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1708004020 con 0x7f171010bd10 2024-04-25T02:17:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.340+0000 7f171a9c7740 1 -- start start 2024-04-25T02:17:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.340+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3240 0x55b3d94f3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.340+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 0x55b3d94fdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.340+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d9500620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.340+0000 7f171a9c7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b3d9457cb0 con 0x55b3d94f3b60 2024-04-25T02:17:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.340+0000 7f171a9c7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b3d94005b0 con 0x55b3d94fe230 2024-04-25T02:17:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.340+0000 7f171a9c7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b3d9453c70 con 0x55b3d94f3240 2024-04-25T02:17:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1715177640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3240 0x55b3d94f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1715978640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 0x55b3d94fdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1715177640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3240 0x55b3d94f3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36248/0 (socket says 172.21.15.31:36248) 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1715177640 1 -- 172.21.15.31:0/832473380 learned_addr learned my addr 172.21.15.31:0/832473380 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1715177640 1 -- 172.21.15.31:0/832473380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 msgr2=0x55b3d9500620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1715177640 1 --2- 172.21.15.31:0/832473380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d9500620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1715177640 1 -- 172.21.15.31:0/832473380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 msgr2=0x55b3d94fdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1715177640 1 --2- 172.21.15.31:0/832473380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 0x55b3d94fdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1715177640 1 -- 172.21.15.31:0/832473380 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3d916c210 con 0x55b3d94f3240 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1706ffd640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f16fc0a2da0 0x7f16fc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1706ffd640 1 -- 172.21.15.31:0/3968668949 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f17080cf9d0 con 0x7f171010bd10 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1715978640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f16fc0a2da0 0x7f16fc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.341+0000 7f1715177640 1 --2- 172.21.15.31:0/832473380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3240 0x55b3d94f3620 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f17000058c0 tx=0x7f1700011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47569a7a7016a6af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.342+0000 7f16fbfff640 1 -- 172.21.15.31:0/832473380 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170001e070 con 0x55b3d94f3240 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.342+0000 7f16fbfff640 1 -- 172.21.15.31:0/832473380 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1700004250 con 0x55b3d94f3240 2024-04-25T02:17:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.342+0000 7f16fbfff640 1 -- 172.21.15.31:0/832473380 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1700019650 con 0x55b3d94f3240 2024-04-25T02:17:41.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.343+0000 7f171a9c7740 1 -- 172.21.15.31:0/832473380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3240 msgr2=0x55b3d94f3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.343+0000 7f171a9c7740 1 --2- 172.21.15.31:0/832473380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3240 0x55b3d94f3620 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f17000058c0 tx=0x7f1700011810 comp rx=0 tx=0).stop 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.343+0000 7f1715978640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f16fc0a2da0 0x7f16fc0a5260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f170c031440 tx=0x7f170c0058d0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.343+0000 7f171a9c7740 1 -- 172.21.15.31:0/832473380 shutdown_connections 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.343+0000 7f171a9c7740 1 --2- 172.21.15.31:0/832473380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d9500620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.343+0000 7f171a9c7740 1 --2- 172.21.15.31:0/832473380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 0x55b3d94fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.343+0000 7f171a9c7740 1 --2- 172.21.15.31:0/832473380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3240 0x55b3d94f3620 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.343+0000 7f171a9c7740 1 -- 172.21.15.31:0/832473380 >> 172.21.15.31:0/832473380 conn(0x55b3d94ef560 msgr2=0x55b3d94ef940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.343+0000 7f171a9c7740 1 -- 172.21.15.31:0/832473380 shutdown_connections 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.343+0000 7f171a9c7740 1 -- 172.21.15.31:0/832473380 wait complete. 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.343+0000 7f171a9c7740 1 -- start start 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f171a9c7740 1 Processor -- start 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3240 0x55b3d94fb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3b60 0x55b3d94fb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d94fbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f171a9c7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b3d9501020 con 0x55b3d94f3240 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f1715978640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3240 0x55b3d94fb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f171a9c7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b3d9413b50 con 0x55b3d94fe230 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f171a9c7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b3d95011a0 con 0x55b3d94f3b60 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f1715978640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3240 0x55b3d94fb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51452/0 (socket says 172.21.15.31:51452) 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f1715978640 1 -- 172.21.15.31:0/4252138736 learned_addr learned my addr 172.21.15.31:0/4252138736 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f1715177640 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3b60 0x55b3d94fb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f171a9c7740 10 client.0 _ll_register_callbacks cb 0x55b3d944eca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.344+0000 7f1716179640 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d94fbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.345+0000 7f1715978640 1 -- 172.21.15.31:0/4252138736 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3b60 msgr2=0x55b3d94fb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.345+0000 7f1715978640 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3b60 0x55b3d94fb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.345+0000 7f1715978640 1 -- 172.21.15.31:0/4252138736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 msgr2=0x55b3d94fbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.345+0000 7f1715978640 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d94fbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.345+0000 7f1715978640 1 -- 172.21.15.31:0/4252138736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3d9238620 con 0x55b3d94f3240 2024-04-25T02:17:41.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.345+0000 7f1715978640 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3240 0x55b3d94fb1f0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f170c0317a0 tx=0x7f170c02a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1812e5ef21206014 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:41.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.346+0000 7f16f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:17:41.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.346+0000 7f16f97fa640 1 -- 172.21.15.31:0/4252138736 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170c035040 con 0x55b3d94f3240 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.346+0000 7f171a9c7740 1 -- 172.21.15.31:0/4252138736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1700004250 con 0x55b3d94f3240 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.346+0000 7f16f97fa640 1 -- 172.21.15.31:0/4252138736 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f170c03d970 con 0x55b3d94f3240 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.347+0000 7f16f97fa640 1 -- 172.21.15.31:0/4252138736 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f170c03db70 con 0x55b3d94f3240 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.347+0000 7f171a9c7740 10 client.17988 fetch_fsmap learned FSMap version 51 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.347+0000 7f171a9c7740 1 -- 172.21.15.31:0/4252138736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55b3d950d010 con 0x55b3d94f3240 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.347+0000 7f16f97fa640 1 -- 172.21.15.31:0/4252138736 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1700004250 con 0x55b3d94f3240 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.348+0000 7f16f97fa640 1 -- 172.21.15.31:0/4252138736 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f170c051070 con 0x55b3d94f3240 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.348+0000 7f171a9c7740 10 client.17988 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.348+0000 7f171a9c7740 10 client.17988 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.348+0000 7f171a9c7740 20 client.17988 populate_metadata read hostname 'smithi031' 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.348+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3d950e4d0 0x55b3d95108e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.348+0000 7f171a9c7740 4 client.17988 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.348+0000 7f171a9c7740 1 -- 172.21.15.31:0/4252138736 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3d94651f0 con 0x55b3d950e4d0 2024-04-25T02:17:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.348+0000 7f1715177640 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3d950e4d0 0x55b3d95108e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.349+0000 7f1715177640 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3d950e4d0 0x55b3d95108e0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c827fe4ed2b50bfd server_cookie=21b462ac0ca32f97 in_seq=0 out_seq=0 2024-04-25T02:17:41.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.349+0000 7f16f97fa640 10 client.17988 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f16f97fa640 1 -- 172.21.15.31:0/4252138736 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3d94651f0 con 0x55b3d950e4d0 2024-04-25T02:17:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f16f97fa640 10 client.17988 handle_command_reply: tid=0 2024-04-25T02:17:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f171a9c7740 1 client.17988 shutdown 2024-04-25T02:17:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f171a9c7740 1 -- 172.21.15.31:0/4252138736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3240 msgr2=0x55b3d94fb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3240 0x55b3d94fb1f0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f170c0317a0 tx=0x7f170c02a510 comp rx=0 tx=0).stop 2024-04-25T02:17:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f171a9c7740 1 -- 172.21.15.31:0/4252138736 shutdown_connections 2024-04-25T02:17:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3d950e4d0 0x55b3d95108e0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d94fbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f16f97fa640 0 client.17988 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94f3b60 0x55b3d94fb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4252138736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3240 0x55b3d94fb1f0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f171a9c7740 1 -- 172.21.15.31:0/4252138736 >> 172.21.15.31:0/4252138736 conn(0x55b3d94ef560 msgr2=0x55b3d94f5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f171a9c7740 1 -- 172.21.15.31:0/4252138736 shutdown_connections 2024-04-25T02:17:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.350+0000 7f171a9c7740 1 -- 172.21.15.31:0/4252138736 wait complete. 2024-04-25T02:17:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.351+0000 7f171a9c7740 20 client.17988 trim_cache size 0 max 16384 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f171a9c7740 1 Processor -- start 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f171a9c7740 1 -- start start 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 0x55b3d95108c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d9509ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d950e4d0 0x55b3d94ef940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f171a9c7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f17101ab1b0 con 0x55b3d94f3b60 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f171a9c7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f17101accf0 con 0x55b3d94fe230 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f171a9c7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f17101ab880 con 0x55b3d950e4d0 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f1715978640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 0x55b3d95108c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f1715177640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d9509ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f1715978640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 0x55b3d95108c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51468/0 (socket says 172.21.15.31:51468) 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f1715978640 1 -- 172.21.15.31:0/1311769007 learned_addr learned my addr 172.21.15.31:0/1311769007 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.375+0000 7f1716179640 1 --2- 172.21.15.31:0/1311769007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d950e4d0 0x55b3d94ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.376+0000 7f1715978640 1 -- 172.21.15.31:0/1311769007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d950e4d0 msgr2=0x55b3d94ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.376+0000 7f1715978640 1 --2- 172.21.15.31:0/1311769007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d950e4d0 0x55b3d94ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.376+0000 7f1715978640 1 -- 172.21.15.31:0/1311769007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 msgr2=0x55b3d9509ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.376+0000 7f1715978640 1 --2- 172.21.15.31:0/1311769007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d9509ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.376+0000 7f1715978640 1 -- 172.21.15.31:0/1311769007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3d94fa1c0 con 0x55b3d94f3b60 2024-04-25T02:17:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.376+0000 7f1716179640 1 --2- 172.21.15.31:0/1311769007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d950e4d0 0x55b3d94ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:17:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.376+0000 7f1715177640 1 --2- 172.21.15.31:0/1311769007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d9509ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:17:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.376+0000 7f1715978640 1 --2- 172.21.15.31:0/1311769007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 0x55b3d95108c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f170c006010 tx=0x7f170c03db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f776fd496c06271 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.376+0000 7f16fb7fe640 1 -- 172.21.15.31:0/1311769007 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170c004770 con 0x55b3d94f3b60 2024-04-25T02:17:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.376+0000 7f16fb7fe640 1 -- 172.21.15.31:0/1311769007 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f170c060020 con 0x55b3d94f3b60 2024-04-25T02:17:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.376+0000 7f16fb7fe640 1 -- 172.21.15.31:0/1311769007 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170c003d90 con 0x55b3d94f3b60 2024-04-25T02:17:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.377+0000 7f171a9c7740 1 -- 172.21.15.31:0/1311769007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 msgr2=0x55b3d95108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.377+0000 7f171a9c7740 1 --2- 172.21.15.31:0/1311769007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 0x55b3d95108c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f170c006010 tx=0x7f170c03db00 comp rx=0 tx=0).stop 2024-04-25T02:17:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.377+0000 7f171a9c7740 1 -- 172.21.15.31:0/1311769007 shutdown_connections 2024-04-25T02:17:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.377+0000 7f171a9c7740 1 --2- 172.21.15.31:0/1311769007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d950e4d0 0x55b3d94ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.377+0000 7f171a9c7740 1 --2- 172.21.15.31:0/1311769007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d94fe230 0x55b3d9509ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.377+0000 7f171a9c7740 1 --2- 172.21.15.31:0/1311769007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f3b60 0x55b3d95108c0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.377+0000 7f171a9c7740 1 -- 172.21.15.31:0/1311769007 >> 172.21.15.31:0/1311769007 conn(0x55b3d94ef560 msgr2=0x55b3d94ee5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.377+0000 7f171a9c7740 1 -- 172.21.15.31:0/1311769007 shutdown_connections 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.377+0000 7f171a9c7740 1 -- 172.21.15.31:0/1311769007 wait complete. 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.378+0000 7f171a9c7740 1 -- start start 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.378+0000 7f171a9c7740 1 Processor -- start 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.378+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94fe230 0x55b3d94f8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.378+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d950e4d0 0x55b3d94f8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.378+0000 7f171a9c7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f8ad0 0x55b3d9573a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f171a9c7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f17101ac620 con 0x55b3d94f8ad0 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f171a9c7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f17101abf50 con 0x55b3d950e4d0 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f171a9c7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f17101a91b0 con 0x55b3d94fe230 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f171a9c7740 10 client.0 _ll_register_callbacks cb 0x7f170c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1716179640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f8ad0 0x55b3d9573a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1716179640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f8ad0 0x55b3d9573a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51484/0 (socket says 172.21.15.31:51484) 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1716179640 1 -- 172.21.15.31:0/4107873875 learned_addr learned my addr 172.21.15.31:0/4107873875 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1715978640 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94fe230 0x55b3d94f8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1715177640 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d950e4d0 0x55b3d94f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1715177640 1 -- 172.21.15.31:0/4107873875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94fe230 msgr2=0x55b3d94f8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1715177640 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94fe230 0x55b3d94f8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1715177640 1 -- 172.21.15.31:0/4107873875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f8ad0 msgr2=0x55b3d9573a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1715177640 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f8ad0 0x55b3d9573a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1715177640 1 -- 172.21.15.31:0/4107873875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3d9574070 con 0x55b3d950e4d0 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1716179640 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f8ad0 0x55b3d9573a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f1715177640 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d950e4d0 0x55b3d94f8590 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f170001c040 tx=0x7f1700011e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7862890c389ec6d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.379+0000 7f16f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f16f97fa640 1 -- 172.21.15.31:0/4107873875 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170001e040 con 0x55b3d950e4d0 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f16f97fa640 1 -- 172.21.15.31:0/4107873875 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f1700004330 con 0x55b3d950e4d0 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f171a9c7740 1 -- 172.21.15.31:0/4107873875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f170c060020 con 0x55b3d950e4d0 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f16f97fa640 1 -- 172.21.15.31:0/4107873875 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f1700005050 con 0x55b3d950e4d0 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f171a9c7740 10 client.27557 fetch_fsmap learned FSMap version 51 2024-04-25T02:17:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f171a9c7740 1 -- 172.21.15.31:0/4107873875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55b3d9575380 con 0x55b3d950e4d0 2024-04-25T02:17:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f16f97fa640 1 -- 172.21.15.31:0/4107873875 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17000035f0 con 0x55b3d950e4d0 2024-04-25T02:17:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f16f97fa640 1 -- 172.21.15.31:0/4107873875 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f1700036070 con 0x55b3d950e4d0 2024-04-25T02:17:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f171a9c7740 10 client.27557 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:17:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f171a9c7740 10 client.27557 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:17:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f171a9c7740 20 client.27557 populate_metadata read hostname 'smithi031' 2024-04-25T02:17:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.380+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3d9575910 0x55b3d9577d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.381+0000 7f171a9c7740 4 client.27557 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:17:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.381+0000 7f171a9c7740 1 -- 172.21.15.31:0/4107873875 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f170c060020 con 0x55b3d9575910 2024-04-25T02:17:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.381+0000 7f1715978640 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3d9575910 0x55b3d9577d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.381+0000 7f1715978640 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3d9575910 0x55b3d9577d30 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e12a65ffc87fab8 server_cookie=c7568808e6a34a90 in_seq=0 out_seq=0 2024-04-25T02:17:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.381+0000 7f16f97fa640 10 client.27557 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:41.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f16f97fa640 1 -- 172.21.15.31:0/4107873875 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f170c060020 con 0x55b3d9575910 2024-04-25T02:17:41.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f16f97fa640 10 client.27557 handle_command_reply: tid=0 2024-04-25T02:17:41.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 1 client.27557 shutdown 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 1 -- 172.21.15.31:0/4107873875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d950e4d0 msgr2=0x55b3d94f8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d950e4d0 0x55b3d94f8590 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f170001c040 tx=0x7f1700011e50 comp rx=0 tx=0).stop 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 1 -- 172.21.15.31:0/4107873875 shutdown_connections 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b3d9575910 0x55b3d9577d30 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b3d94f8ad0 0x55b3d9573a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f16f97fa640 0 client.27557 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3d950e4d0 0x55b3d94f8590 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 1 --2- 172.21.15.31:0/4107873875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3d94fe230 0x55b3d94f8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 1 -- 172.21.15.31:0/4107873875 >> 172.21.15.31:0/4107873875 conn(0x55b3d94ef560 msgr2=0x55b3d94ee5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 1 -- 172.21.15.31:0/4107873875 shutdown_connections 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 1 -- 172.21.15.31:0/4107873875 wait complete. 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.445+0000 7f171a9c7740 20 client.27557 trim_cache size 0 max 16384 2024-04-25T02:17:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:17:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:17:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:17:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:17:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 -- 172.21.15.31:0/3968668949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f16fc0a2da0 msgr2=0x7f16fc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f16fc0a2da0 0x7f16fc0a5260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f170c031440 tx=0x7f170c0058d0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 -- 172.21.15.31:0/3968668949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f171010bd10 msgr2=0x7f171019c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f171010bd10 0x7f171019c780 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f170800ec50 tx=0x7f170800c6a0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 -- 172.21.15.31:0/3968668949 shutdown_connections 2024-04-25T02:17:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f16fc0a2da0 0x7f16fc0a5260 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f171010bd10 0x7f171019c780 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17101032f0 0x7f1710198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 --2- 172.21.15.31:0/3968668949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17101029d0 0x7f1710197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 -- 172.21.15.31:0/3968668949 >> 172.21.15.31:0/3968668949 conn(0x7f17100777c0 msgr2=0x7f17100fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 -- 172.21.15.31:0/3968668949 shutdown_connections 2024-04-25T02:17:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:41.446+0000 7f1717bdc640 1 -- 172.21.15.31:0/3968668949 wait complete. 2024-04-25T02:17:42.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 95982 8.79 MB/sec execute 466 sec latency 118.745 ms 2024-04-25T02:17:42.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:42 smithi040 ceph-mon[32418]: pgmap v732: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 7.8 MiB/s wr, 363 op/s 2024-04-25T02:17:42.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:42 smithi134 ceph-mon[33098]: pgmap v732: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 7.8 MiB/s wr, 363 op/s 2024-04-25T02:17:42.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:42 smithi031 ceph-mon[29204]: pgmap v732: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 7.8 MiB/s wr, 363 op/s 2024-04-25T02:17:43.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 96146 8.78 MB/sec execute 467 sec latency 69.231 ms 2024-04-25T02:17:44.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 96339 8.79 MB/sec execute 468 sec latency 70.656 ms 2024-04-25T02:17:44.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:44 smithi040 ceph-mon[32418]: pgmap v733: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.9 MiB/s wr, 237 op/s 2024-04-25T02:17:44.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:44 smithi134 ceph-mon[33098]: pgmap v733: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.9 MiB/s wr, 237 op/s 2024-04-25T02:17:44.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:44 smithi031 ceph-mon[29204]: pgmap v733: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.9 MiB/s wr, 237 op/s 2024-04-25T02:17:45.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 96526 8.80 MB/sec execute 469 sec latency 90.631 ms 2024-04-25T02:17:46.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 96708 8.81 MB/sec execute 470 sec latency 204.952 ms 2024-04-25T02:17:46.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:46 smithi040 ceph-mon[32418]: pgmap v734: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.7 MiB/s wr, 355 op/s 2024-04-25T02:17:46.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:46 smithi134 ceph-mon[33098]: pgmap v734: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.7 MiB/s wr, 355 op/s 2024-04-25T02:17:46.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:46 smithi031 ceph-mon[29204]: pgmap v734: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.7 MiB/s wr, 355 op/s 2024-04-25T02:17:47.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 96867 8.82 MB/sec execute 471 sec latency 224.924 ms 2024-04-25T02:17:48.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 97060 8.83 MB/sec execute 472 sec latency 120.484 ms 2024-04-25T02:17:48.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:48 smithi134 ceph-mon[33098]: pgmap v735: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-04-25T02:17:48.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:48 smithi031 ceph-mon[29204]: pgmap v735: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-04-25T02:17:48.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:48 smithi040 ceph-mon[32418]: pgmap v735: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-04-25T02:17:49.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 97218 8.83 MB/sec execute 473 sec latency 50.870 ms 2024-04-25T02:17:50.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 97320 8.82 MB/sec execute 474 sec latency 144.854 ms 2024-04-25T02:17:50.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:50 smithi134 ceph-mon[33098]: pgmap v736: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-04-25T02:17:50.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:50 smithi031 ceph-mon[29204]: pgmap v736: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-04-25T02:17:50.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:50 smithi040 ceph-mon[32418]: pgmap v736: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-04-25T02:17:51.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 97465 8.81 MB/sec execute 475 sec latency 52.049 ms 2024-04-25T02:17:52.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 97625 8.81 MB/sec execute 476 sec latency 55.265 ms 2024-04-25T02:17:52.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:52 smithi134 ceph-mon[33098]: pgmap v737: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 341 op/s 2024-04-25T02:17:52.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:52 smithi031 ceph-mon[29204]: pgmap v737: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 341 op/s 2024-04-25T02:17:52.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:52 smithi040 ceph-mon[32418]: pgmap v737: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 341 op/s 2024-04-25T02:17:53.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 97786 8.81 MB/sec execute 477 sec latency 64.950 ms 2024-04-25T02:17:54.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 97951 8.81 MB/sec execute 478 sec latency 80.784 ms 2024-04-25T02:17:54.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:54 smithi134 ceph-mon[33098]: pgmap v738: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.8 MiB/s wr, 220 op/s 2024-04-25T02:17:54.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:54 smithi031 ceph-mon[29204]: pgmap v738: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.8 MiB/s wr, 220 op/s 2024-04-25T02:17:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.584+0000 7f041b891640 1 -- 172.21.15.31:0/253221570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f041406a4b0 msgr2=0x7f041406a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.584+0000 7f041b891640 1 --2- 172.21.15.31:0/253221570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f041406a4b0 0x7f041406a930 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f0404009900 tx=0x7f040402a410 comp rx=0 tx=0).stop 2024-04-25T02:17:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.584+0000 7f041b891640 1 -- 172.21.15.31:0/253221570 shutdown_connections 2024-04-25T02:17:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.584+0000 7f041b891640 1 --2- 172.21.15.31:0/253221570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f041410e090 0x7f0414110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.584+0000 7f041b891640 1 --2- 172.21.15.31:0/253221570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f041406a4b0 0x7f041406a930 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.584+0000 7f041b891640 1 --2- 172.21.15.31:0/253221570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04141026f0 0x7f0414069f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.584+0000 7f041b891640 1 -- 172.21.15.31:0/253221570 >> 172.21.15.31:0/253221570 conn(0x7f0414076440 msgr2=0x7f0414076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.584+0000 7f041b891640 1 -- 172.21.15.31:0/253221570 shutdown_connections 2024-04-25T02:17:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.584+0000 7f041b891640 1 -- 172.21.15.31:0/253221570 wait complete. 2024-04-25T02:17:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.584+0000 7f041b891640 1 Processor -- start 2024-04-25T02:17:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f041b891640 1 -- start start 2024-04-25T02:17:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f041b891640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f041406a4b0 0x7f041419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f041b891640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04141026f0 0x7f041419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f041b891640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f041410e090 0x7f041419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f041b891640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0414113ce0 con 0x7f041410e090 2024-04-25T02:17:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f041b891640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0414113b60 con 0x7f04141026f0 2024-04-25T02:17:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f041b891640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0414113e60 con 0x7f041406a4b0 2024-04-25T02:17:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f0418e2c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04141026f0 0x7f041419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f0418e2c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04141026f0 0x7f041419a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44382/0 (socket says 172.21.15.31:44382) 2024-04-25T02:17:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f0418e2c640 1 -- 172.21.15.31:0/2955148265 learned_addr learned my addr 172.21.15.31:0/2955148265 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f041962d640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f041406a4b0 0x7f041419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.585+0000 7f0419e2e640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f041410e090 0x7f041419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.586+0000 7f0418e2c640 1 -- 172.21.15.31:0/2955148265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f041406a4b0 msgr2=0x7f041419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.586+0000 7f0418e2c640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f041406a4b0 0x7f041419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.586+0000 7f0418e2c640 1 -- 172.21.15.31:0/2955148265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f041410e090 msgr2=0x7f041419e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.586+0000 7f0418e2c640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f041410e090 0x7f041419e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.586+0000 7f0418e2c640 1 -- 172.21.15.31:0/2955148265 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f041419f0d0 con 0x7f04141026f0 2024-04-25T02:17:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.586+0000 7f0418e2c640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04141026f0 0x7f041419a620 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0404005ab0 tx=0x7f0404003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.587+0000 7f040a7fc640 1 -- 172.21.15.31:0/2955148265 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040403f070 con 0x7f04141026f0 2024-04-25T02:17:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.587+0000 7f041b891640 1 -- 172.21.15.31:0/2955148265 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f041419f360 con 0x7f04141026f0 2024-04-25T02:17:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.587+0000 7f040a7fc640 1 -- 172.21.15.31:0/2955148265 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0404031b60 con 0x7f04141026f0 2024-04-25T02:17:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.587+0000 7f040a7fc640 1 -- 172.21.15.31:0/2955148265 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0404035030 con 0x7f04141026f0 2024-04-25T02:17:54.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.587+0000 7f041b891640 1 -- 172.21.15.31:0/2955148265 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04141a6c50 con 0x7f04141026f0 2024-04-25T02:17:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.590+0000 7f040a7fc640 1 -- 172.21.15.31:0/2955148265 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0404042070 con 0x7f04141026f0 2024-04-25T02:17:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.591+0000 7f041e595740 1 Processor -- start 2024-04-25T02:17:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.591+0000 7f041e595740 1 -- start start 2024-04-25T02:17:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.591+0000 7f040a7fc640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f04000a2e20 0x7f04000a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.591+0000 7f041e595740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbb4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.591+0000 7f041e595740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbbecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.591+0000 7f041e595740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbbf230 0x55c36fbc1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.591+0000 7f041e595740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c36fb18cb0 con 0x55c36fbbf230 2024-04-25T02:17:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.591+0000 7f041e595740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c36fac15b0 con 0x55c36fbb4b60 2024-04-25T02:17:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.591+0000 7f041e595740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c36fb14c70 con 0x55c36fbb4240 2024-04-25T02:17:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.591+0000 7f0419e2e640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f04000a2e20 0x7f04000a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.591+0000 7f040a7fc640 1 -- 172.21.15.31:0/2955148265 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f04040ef7e0 con 0x7f04141026f0 2024-04-25T02:17:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f041962d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbb4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f0419e2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbbf230 0x55c36fbc1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f041962d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbb4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:35346/0 (socket says 172.21.15.31:35346) 2024-04-25T02:17:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f041962d640 1 -- 172.21.15.31:0/2774744651 learned_addr learned my addr 172.21.15.31:0/2774744651 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f0418e2c640 1 --2- 172.21.15.31:0/2774744651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbbecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f041962d640 1 -- 172.21.15.31:0/2774744651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 msgr2=0x55c36fbbecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f041962d640 1 --2- 172.21.15.31:0/2774744651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbbecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f041962d640 1 -- 172.21.15.31:0/2774744651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbbf230 msgr2=0x55c36fbc1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f041962d640 1 --2- 172.21.15.31:0/2774744651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbbf230 0x55c36fbc1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f041962d640 1 -- 172.21.15.31:0/2774744651 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c36f82d210 con 0x55c36fbb4240 2024-04-25T02:17:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f0418e2c640 1 --2- 172.21.15.31:0/2774744651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbbecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:17:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f0419e2e640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f04000a2e20 0x7f04000a52e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f040c005fd0 tx=0x7f040c013b60 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.592+0000 7f041962d640 1 --2- 172.21.15.31:0/2774744651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbb4620 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0410009cf0 tx=0x7f0410011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a939e0068860d027 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.593+0000 7f03ff7fe640 1 -- 172.21.15.31:0/2774744651 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041001e070 con 0x55c36fbb4240 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.593+0000 7f03ff7fe640 1 -- 172.21.15.31:0/2774744651 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0410009de0 con 0x55c36fbb4240 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.593+0000 7f03ff7fe640 1 -- 172.21.15.31:0/2774744651 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410019450 con 0x55c36fbb4240 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.594+0000 7f041e595740 1 -- 172.21.15.31:0/2774744651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 msgr2=0x55c36fbb4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.594+0000 7f041e595740 1 --2- 172.21.15.31:0/2774744651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbb4620 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0410009cf0 tx=0x7f0410011810 comp rx=0 tx=0).stop 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.594+0000 7f041e595740 1 -- 172.21.15.31:0/2774744651 shutdown_connections 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.594+0000 7f041e595740 1 --2- 172.21.15.31:0/2774744651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbbf230 0x55c36fbc1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.594+0000 7f041e595740 1 --2- 172.21.15.31:0/2774744651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbbecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.594+0000 7f041e595740 1 --2- 172.21.15.31:0/2774744651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbb4620 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.594+0000 7f041e595740 1 -- 172.21.15.31:0/2774744651 >> 172.21.15.31:0/2774744651 conn(0x55c36fbb0560 msgr2=0x55c36fbb0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.594+0000 7f041e595740 1 -- 172.21.15.31:0/2774744651 shutdown_connections 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.594+0000 7f041e595740 1 -- 172.21.15.31:0/2774744651 wait complete. 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.594+0000 7f041e595740 1 -- start start 2024-04-25T02:17:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.594+0000 7f041e595740 1 Processor -- start 2024-04-25T02:17:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f041e595740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f041e595740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4b60 0x55c36fbbc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f041e595740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbbf230 0x55c36fbbcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f041962d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f041962d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52668/0 (socket says 172.21.15.31:52668) 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f041962d640 1 -- 172.21.15.31:0/11550752 learned_addr learned my addr 172.21.15.31:0/11550752 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f041e595740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c36fbc2020 con 0x55c36fbb4240 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f041e595740 1 -- 172.21.15.31:0/11550752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c36fad4b50 con 0x55c36fbbf230 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f0419e2e640 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbbf230 0x55c36fbbcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f0418e2c640 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4b60 0x55c36fbbc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f041e595740 1 -- 172.21.15.31:0/11550752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c36fbc21a0 con 0x55c36fbb4b60 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.595+0000 7f041e595740 10 client.0 _ll_register_callbacks cb 0x55c36fb0fca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.596+0000 7f041962d640 1 -- 172.21.15.31:0/11550752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4b60 msgr2=0x55c36fbbc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.596+0000 7f041962d640 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4b60 0x55c36fbbc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.596+0000 7f041962d640 1 -- 172.21.15.31:0/11550752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbbf230 msgr2=0x55c36fbbcbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.596+0000 7f041962d640 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbbf230 0x55c36fbbcbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.596+0000 7f041962d640 1 -- 172.21.15.31:0/11550752 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c36f8f9620 con 0x55c36fbb4240 2024-04-25T02:17:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.596+0000 7f0419e2e640 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbbf230 0x55c36fbbcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.596+0000 7f041962d640 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbc150 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f04100099d0 tx=0x7f0410009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1392da32ce1b8179 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.596+0000 7f03fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.596+0000 7f03fcff9640 1 -- 172.21.15.31:0/11550752 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041001e040 con 0x55c36fbb4240 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.596+0000 7f03fcff9640 1 -- 172.21.15.31:0/11550752 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f04100051c0 con 0x55c36fbb4240 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.596+0000 7f041e595740 1 -- 172.21.15.31:0/11550752 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0410009de0 con 0x55c36fbb4240 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.597+0000 7f03fcff9640 1 -- 172.21.15.31:0/11550752 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f0410030050 con 0x55c36fbb4240 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.597+0000 7f041e595740 10 client.18006 fetch_fsmap learned FSMap version 51 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.597+0000 7f041e595740 1 -- 172.21.15.31:0/11550752 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55c36fbce170 con 0x55c36fbb4240 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.597+0000 7f03fcff9640 1 -- 172.21.15.31:0/11550752 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0410009de0 con 0x55c36fbb4240 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.597+0000 7f03fcff9640 1 -- 172.21.15.31:0/11550752 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f0410021070 con 0x55c36fbb4240 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.597+0000 7f041e595740 10 client.18006 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.597+0000 7f041e595740 10 client.18006 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.597+0000 7f041e595740 20 client.18006 populate_metadata read hostname 'smithi031' 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.597+0000 7f041e595740 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c36fbcf960 0x55c36fbd1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.597+0000 7f041e595740 4 client.18006 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.597+0000 7f041e595740 1 -- 172.21.15.31:0/11550752 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c36fb261f0 con 0x55c36fbcf960 2024-04-25T02:17:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.598+0000 7f0418e2c640 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c36fbcf960 0x55c36fbd1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.598+0000 7f03fcff9640 10 client.18006 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.598+0000 7f0418e2c640 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c36fbcf960 0x55c36fbd1df0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f245368af2eeafd3 server_cookie=76214602ac738749 in_seq=0 out_seq=0 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f03fcff9640 1 -- 172.21.15.31:0/11550752 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c36fb261f0 con 0x55c36fbcf960 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f03fcff9640 10 client.18006 handle_command_reply: tid=0 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 1 client.18006 shutdown 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 1 -- 172.21.15.31:0/11550752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 msgr2=0x55c36fbbc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbc150 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f04100099d0 tx=0x7f0410009a00 comp rx=0 tx=0).stop 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 1 -- 172.21.15.31:0/11550752 shutdown_connections 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c36fbcf960 0x55c36fbd1df0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbbf230 0x55c36fbbcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f03fcff9640 0 client.18006 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4b60 0x55c36fbbc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 1 --2- 172.21.15.31:0/11550752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbc150 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 1 -- 172.21.15.31:0/11550752 >> 172.21.15.31:0/11550752 conn(0x55c36fbb0560 msgr2=0x55c36fbb6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:54.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 1 -- 172.21.15.31:0/11550752 shutdown_connections 2024-04-25T02:17:54.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 1 -- 172.21.15.31:0/11550752 wait complete. 2024-04-25T02:17:54.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.599+0000 7f041e595740 20 client.18006 trim_cache size 0 max 16384 2024-04-25T02:17:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f041e595740 1 Processor -- start 2024-04-25T02:17:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f041e595740 1 -- start start 2024-04-25T02:17:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f041e595740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbe880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f041e595740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbcaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f041e595740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbbf230 0x55c36fbbb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f041e595740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f04141af410 con 0x55c36fbb4240 2024-04-25T02:17:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f041e595740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f04141b0f50 con 0x55c36fbb4b60 2024-04-25T02:17:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f041e595740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f04141afae0 con 0x55c36fbbf230 2024-04-25T02:17:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f041962d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbe880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f041962d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbe880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52674/0 (socket says 172.21.15.31:52674) 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f041962d640 1 -- 172.21.15.31:0/2524032623 learned_addr learned my addr 172.21.15.31:0/2524032623 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f0418e2c640 1 --2- 172.21.15.31:0/2524032623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbcaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.624+0000 7f0419e2e640 1 --2- 172.21.15.31:0/2524032623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbbf230 0x55c36fbbb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.625+0000 7f041962d640 1 -- 172.21.15.31:0/2524032623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbbf230 msgr2=0x55c36fbbb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.625+0000 7f041962d640 1 --2- 172.21.15.31:0/2524032623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbbf230 0x55c36fbbb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.625+0000 7f041962d640 1 -- 172.21.15.31:0/2524032623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 msgr2=0x55c36fbcaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.625+0000 7f041962d640 1 --2- 172.21.15.31:0/2524032623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbcaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.625+0000 7f041962d640 1 -- 172.21.15.31:0/2524032623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c36fbb54d0 con 0x55c36fbb4240 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.625+0000 7f0418e2c640 1 --2- 172.21.15.31:0/2524032623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbcaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.625+0000 7f041962d640 1 --2- 172.21.15.31:0/2524032623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbe880 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0410009560 tx=0x7f0410009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc087f477399d35b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.625+0000 7f03feffd640 1 -- 172.21.15.31:0/2524032623 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041001e030 con 0x55c36fbb4240 2024-04-25T02:17:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.625+0000 7f03feffd640 1 -- 172.21.15.31:0/2524032623 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0410027070 con 0x55c36fbb4240 2024-04-25T02:17:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.625+0000 7f03feffd640 1 -- 172.21.15.31:0/2524032623 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410003ef0 con 0x55c36fbb4240 2024-04-25T02:17:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.626+0000 7f041e595740 1 -- 172.21.15.31:0/2524032623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 msgr2=0x55c36fbbe880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.626+0000 7f041e595740 1 --2- 172.21.15.31:0/2524032623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbe880 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0410009560 tx=0x7f0410009de0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.626+0000 7f041e595740 1 -- 172.21.15.31:0/2524032623 shutdown_connections 2024-04-25T02:17:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.626+0000 7f041e595740 1 --2- 172.21.15.31:0/2524032623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbbf230 0x55c36fbbb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.626+0000 7f041e595740 1 --2- 172.21.15.31:0/2524032623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbcaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.626+0000 7f041e595740 1 --2- 172.21.15.31:0/2524032623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbb4240 0x55c36fbbe880 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.626+0000 7f041e595740 1 -- 172.21.15.31:0/2524032623 >> 172.21.15.31:0/2524032623 conn(0x55c36fbb0560 msgr2=0x55c36fbaed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.626+0000 7f041e595740 1 -- 172.21.15.31:0/2524032623 shutdown_connections 2024-04-25T02:17:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.626+0000 7f041e595740 1 -- 172.21.15.31:0/2524032623 wait complete. 2024-04-25T02:17:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.626+0000 7f041e595740 1 -- start start 2024-04-25T02:17:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041e595740 1 Processor -- start 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041e595740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbc73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041e595740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbc7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041e595740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbbf230 0x55c36fbc2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041e595740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f04141b0880 con 0x55c36fbbf230 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041e595740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f04141b01b0 con 0x55c36fbb4b60 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041e595740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f04141ad410 con 0x55c36fbb4240 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041962d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbc73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041e595740 10 client.0 _ll_register_callbacks cb 0x7f041001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041962d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbc73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:35378/0 (socket says 172.21.15.31:35378) 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041962d640 1 -- 172.21.15.31:0/3911596776 learned_addr learned my addr 172.21.15.31:0/3911596776 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f0419e2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbbf230 0x55c36fbc2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f0418e2c640 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbc7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041962d640 1 -- 172.21.15.31:0/3911596776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 msgr2=0x55c36fbc7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041962d640 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbc7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041962d640 1 -- 172.21.15.31:0/3911596776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbbf230 msgr2=0x55c36fbc2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041962d640 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbbf230 0x55c36fbc2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f041962d640 1 -- 172.21.15.31:0/3911596776 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c36fbc3110 con 0x55c36fbb4240 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.627+0000 7f0418e2c640 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbc7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.628+0000 7f041962d640 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbc73f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0410005720 tx=0x7f0410009a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4b5ec49f48385a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.628+0000 7f03fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.628+0000 7f03fcff9640 1 -- 172.21.15.31:0/3911596776 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410041060 con 0x55c36fbb4240 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.628+0000 7f041e595740 1 -- 172.21.15.31:0/3911596776 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0410027070 con 0x55c36fbb4240 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.628+0000 7f03fcff9640 1 -- 172.21.15.31:0/3911596776 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8484+0+0 (secure 0 0 0) 0x7f0410003de0 con 0x55c36fbb4240 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.628+0000 7f03fcff9640 1 -- 172.21.15.31:0/3911596776 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f041001e070 con 0x55c36fbb4240 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.628+0000 7f041e595740 10 client.27349 fetch_fsmap learned FSMap version 51 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.628+0000 7f041e595740 1 -- 172.21.15.31:0/3911596776 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55c36fc3e3d0 con 0x55c36fbb4240 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.629+0000 7f03fcff9640 1 -- 172.21.15.31:0/3911596776 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0410027070 con 0x55c36fbb4240 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.629+0000 7f03fcff9640 1 -- 172.21.15.31:0/3911596776 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f0410021040 con 0x55c36fbb4240 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.629+0000 7f041e595740 10 client.27349 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:17:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.629+0000 7f041e595740 10 client.27349 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:17:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.629+0000 7f041e595740 20 client.27349 populate_metadata read hostname 'smithi031' 2024-04-25T02:17:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.629+0000 7f041e595740 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c36fbcf960 0x55c36fc40a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:17:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.629+0000 7f041e595740 4 client.27349 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:17:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.629+0000 7f041e595740 1 -- 172.21.15.31:0/3911596776 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c36fb20540 con 0x55c36fbcf960 2024-04-25T02:17:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.629+0000 7f0418e2c640 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c36fbcf960 0x55c36fc40a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:17:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.629+0000 7f03fcff9640 10 client.27349 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.629+0000 7f0418e2c640 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c36fbcf960 0x55c36fc40a50 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40f8cb2897e1bc73 server_cookie=f471043dfddac0cb in_seq=0 out_seq=0 2024-04-25T02:17:54.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:54 smithi040 ceph-mon[32418]: pgmap v738: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.8 MiB/s wr, 220 op/s 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f03fcff9640 1 -- 172.21.15.31:0/3911596776 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c36fb20540 con 0x55c36fbcf960 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f03fcff9640 10 client.27349 handle_command_reply: tid=0 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 1 client.27349 shutdown 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 1 -- 172.21.15.31:0/3911596776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 msgr2=0x55c36fbc73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbc73f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0410005720 tx=0x7f0410009a50 comp rx=0 tx=0).stop 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 1 -- 172.21.15.31:0/3911596776 shutdown_connections 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55c36fbcf960 0x55c36fc40a50 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c36fbbf230 0x55c36fbc2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f03fcff9640 0 client.27349 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c36fbb4b60 0x55c36fbc7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 1 --2- 172.21.15.31:0/3911596776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c36fbb4240 0x55c36fbc73f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 1 -- 172.21.15.31:0/3911596776 >> 172.21.15.31:0/3911596776 conn(0x55c36fbb0560 msgr2=0x55c36fbaed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 1 -- 172.21.15.31:0/3911596776 shutdown_connections 2024-04-25T02:17:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 1 -- 172.21.15.31:0/3911596776 wait complete. 2024-04-25T02:17:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.696+0000 7f041e595740 20 client.27349 trim_cache size 0 max 16384 2024-04-25T02:17:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:17:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:17:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:17:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:17:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.697+0000 7f041b891640 1 -- 172.21.15.31:0/2955148265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f04000a2e20 msgr2=0x7f04000a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.697+0000 7f041b891640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f04000a2e20 0x7f04000a52e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f040c005fd0 tx=0x7f040c013b60 comp rx=0 tx=0).stop 2024-04-25T02:17:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.697+0000 7f041b891640 1 -- 172.21.15.31:0/2955148265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04141026f0 msgr2=0x7f041419a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:17:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.697+0000 7f041b891640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04141026f0 0x7f041419a620 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0404005ab0 tx=0x7f0404003c10 comp rx=0 tx=0).stop 2024-04-25T02:17:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.697+0000 7f041b891640 1 -- 172.21.15.31:0/2955148265 shutdown_connections 2024-04-25T02:17:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.697+0000 7f041b891640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f04000a2e20 0x7f04000a52e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.697+0000 7f041b891640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f041410e090 0x7f041419e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.697+0000 7f041b891640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04141026f0 0x7f041419a620 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.697+0000 7f041b891640 1 --2- 172.21.15.31:0/2955148265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f041406a4b0 0x7f041419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:17:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.697+0000 7f041b891640 1 -- 172.21.15.31:0/2955148265 >> 172.21.15.31:0/2955148265 conn(0x7f0414076440 msgr2=0x7f04140ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:17:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.697+0000 7f041b891640 1 -- 172.21.15.31:0/2955148265 shutdown_connections 2024-04-25T02:17:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:17:54.698+0000 7f041b891640 1 -- 172.21.15.31:0/2955148265 wait complete. 2024-04-25T02:17:55.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 98094 8.81 MB/sec execute 479 sec latency 88.511 ms 2024-04-25T02:17:56.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 98222 8.80 MB/sec execute 480 sec latency 60.093 ms 2024-04-25T02:17:56.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:56 smithi134 ceph-mon[33098]: pgmap v739: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.9 MiB/s wr, 354 op/s 2024-04-25T02:17:56.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:56 smithi031 ceph-mon[29204]: pgmap v739: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.9 MiB/s wr, 354 op/s 2024-04-25T02:17:56.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:56 smithi040 ceph-mon[32418]: pgmap v739: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.9 MiB/s wr, 354 op/s 2024-04-25T02:17:57.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 98391 8.79 MB/sec execute 481 sec latency 89.898 ms 2024-04-25T02:17:58.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 98564 8.79 MB/sec execute 482 sec latency 67.579 ms 2024-04-25T02:17:58.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:17:58 smithi134 ceph-mon[33098]: pgmap v740: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 236 op/s 2024-04-25T02:17:58.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:17:58 smithi031 ceph-mon[29204]: pgmap v740: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 236 op/s 2024-04-25T02:17:58.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:17:58 smithi040 ceph-mon[32418]: pgmap v740: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 236 op/s 2024-04-25T02:17:59.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 98726 8.78 MB/sec execute 483 sec latency 58.948 ms 2024-04-25T02:18:00.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 98894 8.78 MB/sec execute 484 sec latency 78.734 ms 2024-04-25T02:18:00.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:00 smithi134 ceph-mon[33098]: pgmap v741: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 236 op/s 2024-04-25T02:18:00.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:00 smithi031 ceph-mon[29204]: pgmap v741: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 236 op/s 2024-04-25T02:18:00.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:00 smithi040 ceph-mon[32418]: pgmap v741: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 236 op/s 2024-04-25T02:18:01.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 99035 8.77 MB/sec execute 485 sec latency 129.257 ms 2024-04-25T02:18:02.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 99199 8.77 MB/sec execute 486 sec latency 99.870 ms 2024-04-25T02:18:02.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:02 smithi134 ceph-mon[33098]: pgmap v742: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 8.1 MiB/s wr, 362 op/s 2024-04-25T02:18:02.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:02 smithi031 ceph-mon[29204]: pgmap v742: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 8.1 MiB/s wr, 362 op/s 2024-04-25T02:18:02.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:02 smithi040 ceph-mon[32418]: pgmap v742: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 8.1 MiB/s wr, 362 op/s 2024-04-25T02:18:03.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 99368 8.78 MB/sec execute 487 sec latency 153.985 ms 2024-04-25T02:18:03.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:03 smithi134 ceph-mon[33098]: osdmap e124: 12 total, 12 up, 12 in 2024-04-25T02:18:03.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:03 smithi031 ceph-mon[29204]: osdmap e124: 12 total, 12 up, 12 in 2024-04-25T02:18:03.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:03 smithi040 ceph-mon[32418]: osdmap e124: 12 total, 12 up, 12 in 2024-04-25T02:18:04.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 99543 8.78 MB/sec execute 488 sec latency 100.122 ms 2024-04-25T02:18:04.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:04 smithi031 ceph-mon[29204]: pgmap v744: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.1 MiB/s wr, 312 op/s 2024-04-25T02:18:04.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:04 smithi040 ceph-mon[32418]: pgmap v744: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.1 MiB/s wr, 312 op/s 2024-04-25T02:18:04.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:04 smithi134 ceph-mon[33098]: pgmap v744: 97 pgs: 97 active+clean; 1.5 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.1 MiB/s wr, 312 op/s 2024-04-25T02:18:05.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 99755 8.78 MB/sec execute 489 sec latency 95.642 ms 2024-04-25T02:18:06.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 99915 8.79 MB/sec execute 490 sec latency 120.019 ms 2024-04-25T02:18:06.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:06 smithi031 ceph-mon[29204]: pgmap v745: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T02:18:06.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:06 smithi040 ceph-mon[32418]: pgmap v745: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T02:18:06.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:06 smithi134 ceph-mon[33098]: pgmap v745: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T02:18:07.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 100104 8.80 MB/sec execute 491 sec latency 84.707 ms 2024-04-25T02:18:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.839+0000 7fde3b47b640 1 -- 172.21.15.31:0/184323597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde341087f0 msgr2=0x7fde34108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.839+0000 7fde3b47b640 1 --2- 172.21.15.31:0/184323597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde341087f0 0x7fde34108bd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fde30009860 tx=0x7fde3002a340 comp rx=0 tx=0).stop 2024-04-25T02:18:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.839+0000 7fde3b47b640 1 -- 172.21.15.31:0/184323597 shutdown_connections 2024-04-25T02:18:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.839+0000 7fde3b47b640 1 --2- 172.21.15.31:0/184323597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde3410d0a0 0x7fde34114c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.839+0000 7fde3b47b640 1 --2- 172.21.15.31:0/184323597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde341091a0 0x7fde3410c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.839+0000 7fde3b47b640 1 --2- 172.21.15.31:0/184323597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde341087f0 0x7fde34108bd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.839+0000 7fde3b47b640 1 -- 172.21.15.31:0/184323597 >> 172.21.15.31:0/184323597 conn(0x7fde340774a0 msgr2=0x7fde340798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.839+0000 7fde3b47b640 1 -- 172.21.15.31:0/184323597 shutdown_connections 2024-04-25T02:18:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.839+0000 7fde3b47b640 1 -- 172.21.15.31:0/184323597 wait complete. 2024-04-25T02:18:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.840+0000 7fde3b47b640 1 Processor -- start 2024-04-25T02:18:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.840+0000 7fde3b47b640 1 -- start start 2024-04-25T02:18:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.840+0000 7fde3b47b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde341087f0 0x7fde3419e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.840+0000 7fde3b47b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde3410d0a0 0x7fde3419ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.840+0000 7fde3b47b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde341a2e00 0x7fde341a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.840+0000 7fde3b47b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fde34118000 con 0x7fde341a2e00 2024-04-25T02:18:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.840+0000 7fde3b47b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde34117e80 con 0x7fde341087f0 2024-04-25T02:18:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.840+0000 7fde3b47b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fde34118180 con 0x7fde3410d0a0 2024-04-25T02:18:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.840+0000 7fde39a18640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde341a2e00 0x7fde341a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.840+0000 7fde39a18640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde341a2e00 0x7fde341a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35672/0 (socket says 172.21.15.31:35672) 2024-04-25T02:18:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.840+0000 7fde39a18640 1 -- 172.21.15.31:0/2932819559 learned_addr learned my addr 172.21.15.31:0/2932819559 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.841+0000 7fde39217640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde341087f0 0x7fde3419e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.841+0000 7fde38a16640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde3410d0a0 0x7fde3419ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.841+0000 7fde39a18640 1 -- 172.21.15.31:0/2932819559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde3410d0a0 msgr2=0x7fde3419ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.841+0000 7fde39a18640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde3410d0a0 0x7fde3419ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.841+0000 7fde39a18640 1 -- 172.21.15.31:0/2932819559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde341087f0 msgr2=0x7fde3419e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.841+0000 7fde39a18640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde341087f0 0x7fde3419e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.841+0000 7fde39a18640 1 -- 172.21.15.31:0/2932819559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde341a38f0 con 0x7fde341a2e00 2024-04-25T02:18:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.841+0000 7fde39a18640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde341a2e00 0x7fde341a32b0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009fd0 tx=0x7fde2c00def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.842+0000 7fde2a7fc640 1 -- 172.21.15.31:0/2932819559 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c017070 con 0x7fde341a2e00 2024-04-25T02:18:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.842+0000 7fde2a7fc640 1 -- 172.21.15.31:0/2932819559 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde2c003e20 con 0x7fde341a2e00 2024-04-25T02:18:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.842+0000 7fde3b47b640 1 -- 172.21.15.31:0/2932819559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde341a3be0 con 0x7fde341a2e00 2024-04-25T02:18:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.842+0000 7fde3b47b640 1 -- 172.21.15.31:0/2932819559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde341ab3d0 con 0x7fde341a2e00 2024-04-25T02:18:07.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.842+0000 7fde2a7fc640 1 -- 172.21.15.31:0/2932819559 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c010460 con 0x7fde341a2e00 2024-04-25T02:18:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.846+0000 7fde2a7fc640 1 -- 172.21.15.31:0/2932819559 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fde2c003fd0 con 0x7fde341a2e00 2024-04-25T02:18:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde3e181740 1 Processor -- start 2024-04-25T02:18:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde3e181740 1 -- start start 2024-04-25T02:18:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde3e181740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9ca620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde3e181740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9cab60 0x55e43b9d4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde3e181740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9d5230 0x55e43b9d7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde3e181740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e43b92ecb0 con 0x55e43b9cab60 2024-04-25T02:18:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde3e181740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e43b8d75b0 con 0x55e43b9d5230 2024-04-25T02:18:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde3e181740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e43b92ac70 con 0x55e43b9ca240 2024-04-25T02:18:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde38a16640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde38a16640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9ca620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:39692/0 (socket says 172.21.15.31:39692) 2024-04-25T02:18:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde38a16640 1 -- 172.21.15.31:0/3130542419 learned_addr learned my addr 172.21.15.31:0/3130542419 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.847+0000 7fde38a16640 1 --2- 172.21.15.31:0/3130542419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9d5230 0x55e43b9d7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde38a16640 1 -- 172.21.15.31:0/3130542419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9d5230 msgr2=0x55e43b9d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde38a16640 1 --2- 172.21.15.31:0/3130542419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9d5230 0x55e43b9d7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde38a16640 1 -- 172.21.15.31:0/3130542419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9cab60 msgr2=0x55e43b9d4cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:18:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde38a16640 1 --2- 172.21.15.31:0/3130542419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9cab60 0x55e43b9d4cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde38a16640 1 -- 172.21.15.31:0/3130542419 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e43b643210 con 0x55e43b9ca240 2024-04-25T02:18:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde2a7fc640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fde200a2f00 0x7fde200a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde38a16640 1 --2- 172.21.15.31:0/3130542419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9ca620 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde240058c0 tx=0x7fde24005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c1ea97a7198640f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde1f7fe640 1 -- 172.21.15.31:0/3130542419 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24005020 con 0x55e43b9ca240 2024-04-25T02:18:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde2a7fc640 1 -- 172.21.15.31:0/2932819559 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8528+0+0 (secure 0 0 0) 0x7fde2c0cec30 con 0x7fde341a2e00 2024-04-25T02:18:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde39217640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fde200a2f00 0x7fde200a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde1f7fe640 1 -- 172.21.15.31:0/3130542419 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde240047a0 con 0x55e43b9ca240 2024-04-25T02:18:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.848+0000 7fde1f7fe640 1 -- 172.21.15.31:0/3130542419 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2400fa20 con 0x55e43b9ca240 2024-04-25T02:18:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.849+0000 7fde3e181740 1 -- 172.21.15.31:0/3130542419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 msgr2=0x55e43b9ca620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.849+0000 7fde39217640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fde200a2f00 0x7fde200a53c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fde3002a930 tx=0x7fde300058d0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.849+0000 7fde3e181740 1 --2- 172.21.15.31:0/3130542419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9ca620 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde240058c0 tx=0x7fde24005850 comp rx=0 tx=0).stop 2024-04-25T02:18:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.849+0000 7fde3e181740 1 -- 172.21.15.31:0/3130542419 shutdown_connections 2024-04-25T02:18:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.849+0000 7fde3e181740 1 --2- 172.21.15.31:0/3130542419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9d5230 0x55e43b9d7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.849+0000 7fde3e181740 1 --2- 172.21.15.31:0/3130542419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9cab60 0x55e43b9d4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.849+0000 7fde3e181740 1 --2- 172.21.15.31:0/3130542419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9ca620 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.849+0000 7fde3e181740 1 -- 172.21.15.31:0/3130542419 >> 172.21.15.31:0/3130542419 conn(0x55e43b9c6560 msgr2=0x55e43b9c6940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.850+0000 7fde3e181740 1 -- 172.21.15.31:0/3130542419 shutdown_connections 2024-04-25T02:18:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.850+0000 7fde3e181740 1 -- 172.21.15.31:0/3130542419 wait complete. 2024-04-25T02:18:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.850+0000 7fde3e181740 1 -- start start 2024-04-25T02:18:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.851+0000 7fde3e181740 1 Processor -- start 2024-04-25T02:18:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.851+0000 7fde3e181740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9d2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde3e181740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9cab60 0x55e43b9d26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde3e181740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9d5230 0x55e43b9d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde3e181740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e43b9d8020 con 0x55e43b9d5230 2024-04-25T02:18:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde3e181740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e43b8eab50 con 0x55e43b9cab60 2024-04-25T02:18:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde3e181740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e43b9d81a0 con 0x55e43b9ca240 2024-04-25T02:18:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde39217640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9d2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde39217640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9d2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:39694/0 (socket says 172.21.15.31:39694) 2024-04-25T02:18:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde39217640 1 -- 172.21.15.31:0/1808299702 learned_addr learned my addr 172.21.15.31:0/1808299702 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde38a16640 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9cab60 0x55e43b9d26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde3e181740 10 client.0 _ll_register_callbacks cb 0x55e43b925ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:18:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde39217640 1 -- 172.21.15.31:0/1808299702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9cab60 msgr2=0x55e43b9d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde39217640 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9cab60 0x55e43b9d26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde39217640 1 -- 172.21.15.31:0/1808299702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9d5230 msgr2=0x55e43b9d2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde39217640 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9d5230 0x55e43b9d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.852+0000 7fde39217640 1 -- 172.21.15.31:0/1808299702 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e43b70f620 con 0x55e43b9ca240 2024-04-25T02:18:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.853+0000 7fde39217640 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9d2170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fde300095f0 tx=0x7fde30005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17536fc2f04f70f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.853+0000 7fde1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:18:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.853+0000 7fde1cff9640 1 -- 172.21.15.31:0/1808299702 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30037040 con 0x55e43b9ca240 2024-04-25T02:18:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.853+0000 7fde3e181740 1 -- 172.21.15.31:0/1808299702 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde240047a0 con 0x55e43b9ca240 2024-04-25T02:18:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.853+0000 7fde1cff9640 1 -- 172.21.15.31:0/1808299702 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8528+0+0 (secure 0 0 0) 0x7fde300037f0 con 0x55e43b9ca240 2024-04-25T02:18:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.854+0000 7fde1cff9640 1 -- 172.21.15.31:0/1808299702 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fde30030e40 con 0x55e43b9ca240 2024-04-25T02:18:07.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.855+0000 7fde3e181740 10 client.27361 fetch_fsmap learned FSMap version 51 2024-04-25T02:18:07.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.855+0000 7fde3e181740 1 -- 172.21.15.31:0/1808299702 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55e43b9e4220 con 0x55e43b9ca240 2024-04-25T02:18:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.855+0000 7fde1cff9640 1 -- 172.21.15.31:0/1808299702 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde240047a0 con 0x55e43b9ca240 2024-04-25T02:18:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.855+0000 7fde1cff9640 1 -- 172.21.15.31:0/1808299702 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7fde3003ead0 con 0x55e43b9ca240 2024-04-25T02:18:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.856+0000 7fde3e181740 10 client.27361 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:18:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.856+0000 7fde3e181740 10 client.27361 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:18:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.856+0000 7fde3e181740 20 client.27361 populate_metadata read hostname 'smithi031' 2024-04-25T02:18:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.856+0000 7fde3e181740 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e43b9e5740 0x55e43b9e7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.856+0000 7fde3e181740 4 client.27361 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:18:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.856+0000 7fde3e181740 1 -- 172.21.15.31:0/1808299702 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e43b93c1f0 con 0x55e43b9e5740 2024-04-25T02:18:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.856+0000 7fde38a16640 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e43b9e5740 0x55e43b9e7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.857+0000 7fde38a16640 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e43b9e5740 0x55e43b9e7c00 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19eacbe92437ca3c server_cookie=b79dfe106cb0abce in_seq=0 out_seq=0 2024-04-25T02:18:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.857+0000 7fde1cff9640 10 client.27361 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde1cff9640 1 -- 172.21.15.31:0/1808299702 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e43b93c1f0 con 0x55e43b9e5740 2024-04-25T02:18:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde1cff9640 10 client.27361 handle_command_reply: tid=0 2024-04-25T02:18:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde3e181740 1 client.27361 shutdown 2024-04-25T02:18:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde3e181740 1 -- 172.21.15.31:0/1808299702 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 msgr2=0x55e43b9d2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde3e181740 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9d2170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fde300095f0 tx=0x7fde30005f90 comp rx=0 tx=0).stop 2024-04-25T02:18:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde3e181740 1 -- 172.21.15.31:0/1808299702 shutdown_connections 2024-04-25T02:18:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde3e181740 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e43b9e5740 0x55e43b9e7c00 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde3e181740 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9d5230 0x55e43b9d2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde3e181740 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9cab60 0x55e43b9d26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde1cff9640 0 client.27361 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde3e181740 1 --2- 172.21.15.31:0/1808299702 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9ca240 0x55e43b9d2170 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde3e181740 1 -- 172.21.15.31:0/1808299702 >> 172.21.15.31:0/1808299702 conn(0x55e43b9c6560 msgr2=0x55e43b9cc260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde3e181740 1 -- 172.21.15.31:0/1808299702 shutdown_connections 2024-04-25T02:18:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.858+0000 7fde3e181740 1 -- 172.21.15.31:0/1808299702 wait complete. 2024-04-25T02:18:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.859+0000 7fde3e181740 20 client.27361 trim_cache size 0 max 16384 2024-04-25T02:18:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde3e181740 1 Processor -- start 2024-04-25T02:18:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde3e181740 1 -- start start 2024-04-25T02:18:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde3e181740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9ca240 0x55e43b9d48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde3e181740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 0x55e43b9e0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde3e181740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9d5230 0x55e43b9ed510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde3e181740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fde341b3950 con 0x55e43b9ca240 2024-04-25T02:18:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde3e181740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde341b5490 con 0x55e43b9d5230 2024-04-25T02:18:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde3e181740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fde341b4020 con 0x55e43b9cab60 2024-04-25T02:18:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde39217640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9ca240 0x55e43b9d48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde38a16640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 0x55e43b9e0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde39217640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9ca240 0x55e43b9d48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35684/0 (socket says 172.21.15.31:35684) 2024-04-25T02:18:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde39217640 1 -- 172.21.15.31:0/904650332 learned_addr learned my addr 172.21.15.31:0/904650332 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde38a16640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 0x55e43b9e0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:39700/0 (socket says 172.21.15.31:39700) 2024-04-25T02:18:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.883+0000 7fde39a18640 1 --2- 172.21.15.31:0/904650332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9d5230 0x55e43b9ed510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.884+0000 7fde38a16640 1 -- 172.21.15.31:0/904650332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9d5230 msgr2=0x55e43b9ed510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.884+0000 7fde38a16640 1 --2- 172.21.15.31:0/904650332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9d5230 0x55e43b9ed510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.884+0000 7fde38a16640 1 -- 172.21.15.31:0/904650332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9ca240 msgr2=0x55e43b9d48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.884+0000 7fde38a16640 1 --2- 172.21.15.31:0/904650332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9ca240 0x55e43b9d48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.884+0000 7fde38a16640 1 -- 172.21.15.31:0/904650332 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e43b9ca990 con 0x55e43b9cab60 2024-04-25T02:18:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.884+0000 7fde38a16640 1 --2- 172.21.15.31:0/904650332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 0x55e43b9e0f10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55e43b9d3740 tx=0x7fde24005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e0bad418909f016 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.884+0000 7fde1effd640 1 -- 172.21.15.31:0/904650332 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde240047a0 con 0x55e43b9cab60 2024-04-25T02:18:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.884+0000 7fde1effd640 1 -- 172.21.15.31:0/904650332 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde240044f0 con 0x55e43b9cab60 2024-04-25T02:18:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.884+0000 7fde1effd640 1 -- 172.21.15.31:0/904650332 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2400fe40 con 0x55e43b9cab60 2024-04-25T02:18:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.885+0000 7fde3e181740 1 -- 172.21.15.31:0/904650332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 msgr2=0x55e43b9e0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.885+0000 7fde3e181740 1 --2- 172.21.15.31:0/904650332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 0x55e43b9e0f10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55e43b9d3740 tx=0x7fde24005fd0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.885+0000 7fde3e181740 1 -- 172.21.15.31:0/904650332 shutdown_connections 2024-04-25T02:18:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.885+0000 7fde3e181740 1 --2- 172.21.15.31:0/904650332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9d5230 0x55e43b9ed510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.885+0000 7fde3e181740 1 --2- 172.21.15.31:0/904650332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 0x55e43b9e0f10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.885+0000 7fde3e181740 1 --2- 172.21.15.31:0/904650332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9ca240 0x55e43b9d48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.885+0000 7fde3e181740 1 -- 172.21.15.31:0/904650332 >> 172.21.15.31:0/904650332 conn(0x55e43b9c6560 msgr2=0x55e43b9c55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.886+0000 7fde3e181740 1 -- 172.21.15.31:0/904650332 shutdown_connections 2024-04-25T02:18:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.886+0000 7fde3e181740 1 -- 172.21.15.31:0/904650332 wait complete. 2024-04-25T02:18:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.886+0000 7fde3e181740 1 -- start start 2024-04-25T02:18:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.886+0000 7fde3e181740 1 Processor -- start 2024-04-25T02:18:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde3e181740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9ca240 0x55e43b9ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde3e181740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 0x55e43b9cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde3e181740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9d5230 0x55e43b9cf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde3e181740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fde341b4dc0 con 0x55e43b9d5230 2024-04-25T02:18:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde3e181740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde341b46f0 con 0x55e43b9ca240 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde3e181740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fde341b1930 con 0x55e43b9cab60 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde39217640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9ca240 0x55e43b9ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde39217640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9ca240 0x55e43b9ceda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36164/0 (socket says 172.21.15.31:36164) 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde39217640 1 -- 172.21.15.31:0/157825312 learned_addr learned my addr 172.21.15.31:0/157825312 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde38a16640 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 0x55e43b9cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde39a18640 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9d5230 0x55e43b9cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde3e181740 10 client.0 _ll_register_callbacks cb 0x7fde24005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde39217640 1 -- 172.21.15.31:0/157825312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 msgr2=0x55e43b9cf2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde39217640 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 0x55e43b9cf2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde39217640 1 -- 172.21.15.31:0/157825312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9d5230 msgr2=0x55e43b9cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde39217640 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9d5230 0x55e43b9cf820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.887+0000 7fde39217640 1 -- 172.21.15.31:0/157825312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e43ba4ad80 con 0x55e43b9ca240 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.888+0000 7fde39217640 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9ca240 0x55e43b9ceda0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fde300057c0 tx=0x7fde30004970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fc604c343ae2fba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.888+0000 7fde1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:18:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.888+0000 7fde1cff9640 1 -- 172.21.15.31:0/157825312 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30061060 con 0x55e43b9ca240 2024-04-25T02:18:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.888+0000 7fde3e181740 1 -- 172.21.15.31:0/157825312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde240044f0 con 0x55e43b9ca240 2024-04-25T02:18:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.888+0000 7fde1cff9640 1 -- 172.21.15.31:0/157825312 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8528+0+0 (secure 0 0 0) 0x7fde30004d70 con 0x55e43b9ca240 2024-04-25T02:18:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.889+0000 7fde1cff9640 1 -- 172.21.15.31:0/157825312 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fde30037040 con 0x55e43b9ca240 2024-04-25T02:18:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.889+0000 7fde3e181740 10 client.27590 fetch_fsmap learned FSMap version 51 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.889+0000 7fde3e181740 1 -- 172.21.15.31:0/157825312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55e43ba4bf90 con 0x55e43b9ca240 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.889+0000 7fde1cff9640 1 -- 172.21.15.31:0/157825312 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde240044f0 con 0x55e43b9ca240 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.890+0000 7fde1cff9640 1 -- 172.21.15.31:0/157825312 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7fde30003710 con 0x55e43b9ca240 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.890+0000 7fde3e181740 10 client.27590 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.890+0000 7fde3e181740 10 client.27590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.890+0000 7fde3e181740 20 client.27590 populate_metadata read hostname 'smithi031' 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.890+0000 7fde3e181740 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e43b9e5740 0x55e43ba4e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.890+0000 7fde3e181740 4 client.27590 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.890+0000 7fde3e181740 1 -- 172.21.15.31:0/157825312 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e43b936540 con 0x55e43b9e5740 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.890+0000 7fde38a16640 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e43b9e5740 0x55e43ba4e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.890+0000 7fde1cff9640 10 client.27590 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.890+0000 7fde38a16640 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e43b9e5740 0x55e43ba4e780 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=780a47daf5bf99ae server_cookie=e7079fbeaef667ae in_seq=0 out_seq=0 2024-04-25T02:18:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde1cff9640 1 -- 172.21.15.31:0/157825312 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e43b936540 con 0x55e43b9e5740 2024-04-25T02:18:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde1cff9640 10 client.27590 handle_command_reply: tid=0 2024-04-25T02:18:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 1 client.27590 shutdown 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 1 -- 172.21.15.31:0/157825312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9ca240 msgr2=0x55e43b9ceda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9ca240 0x55e43b9ceda0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fde300057c0 tx=0x7fde30004970 comp rx=0 tx=0).stop 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 1 -- 172.21.15.31:0/157825312 shutdown_connections 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55e43b9e5740 0x55e43ba4e780 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e43b9d5230 0x55e43b9cf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde1cff9640 0 client.27590 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e43b9cab60 0x55e43b9cf2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 1 --2- 172.21.15.31:0/157825312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e43b9ca240 0x55e43b9ceda0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 1 -- 172.21.15.31:0/157825312 >> 172.21.15.31:0/157825312 conn(0x55e43b9c6560 msgr2=0x55e43b9d1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 1 -- 172.21.15.31:0/157825312 shutdown_connections 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 1 -- 172.21.15.31:0/157825312 wait complete. 2024-04-25T02:18:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.965+0000 7fde3e181740 20 client.27590 trim_cache size 0 max 16384 2024-04-25T02:18:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:18:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:18:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:18:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 -- 172.21.15.31:0/2932819559 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fde200a2f00 msgr2=0x7fde200a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fde200a2f00 0x7fde200a53c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fde3002a930 tx=0x7fde300058d0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 -- 172.21.15.31:0/2932819559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde341a2e00 msgr2=0x7fde341a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde341a2e00 0x7fde341a32b0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009fd0 tx=0x7fde2c00def0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 -- 172.21.15.31:0/2932819559 shutdown_connections 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fde200a2f00 0x7fde200a53c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde341a2e00 0x7fde341a32b0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde3410d0a0 0x7fde3419ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 --2- 172.21.15.31:0/2932819559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde341087f0 0x7fde3419e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 -- 172.21.15.31:0/2932819559 >> 172.21.15.31:0/2932819559 conn(0x7fde340774a0 msgr2=0x7fde34079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 -- 172.21.15.31:0/2932819559 shutdown_connections 2024-04-25T02:18:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:07.966+0000 7fde3b47b640 1 -- 172.21.15.31:0/2932819559 wait complete. 2024-04-25T02:18:08.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 100264 8.81 MB/sec execute 492 sec latency 95.741 ms 2024-04-25T02:18:08.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:08 smithi031 ceph-mon[29204]: pgmap v746: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T02:18:08.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:08 smithi040 ceph-mon[32418]: pgmap v746: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T02:18:08.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:08 smithi134 ceph-mon[33098]: pgmap v746: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T02:18:09.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 100423 8.81 MB/sec execute 493 sec latency 85.668 ms 2024-04-25T02:18:10.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 100588 8.83 MB/sec execute 494 sec latency 73.234 ms 2024-04-25T02:18:10.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:10 smithi040 ceph-mon[32418]: pgmap v747: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T02:18:10.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:10 smithi040 ceph-mon[32418]: osdmap e125: 12 total, 12 up, 12 in 2024-04-25T02:18:10.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:10 smithi134 ceph-mon[33098]: pgmap v747: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T02:18:10.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:10 smithi134 ceph-mon[33098]: osdmap e125: 12 total, 12 up, 12 in 2024-04-25T02:18:10.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:10 smithi031 ceph-mon[29204]: pgmap v747: 97 pgs: 97 active+clean; 1.5 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T02:18:10.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:10 smithi031 ceph-mon[29204]: osdmap e125: 12 total, 12 up, 12 in 2024-04-25T02:18:11.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 100715 8.81 MB/sec execute 495 sec latency 114.649 ms 2024-04-25T02:18:12.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 100856 8.82 MB/sec execute 496 sec latency 75.288 ms 2024-04-25T02:18:12.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:12 smithi040 ceph-mon[32418]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.9 MiB/s wr, 320 op/s 2024-04-25T02:18:12.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:12 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:18:12.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:12 smithi134 ceph-mon[33098]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.9 MiB/s wr, 320 op/s 2024-04-25T02:18:12.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:12 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:18:12.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:12 smithi031 ceph-mon[29204]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.9 MiB/s wr, 320 op/s 2024-04-25T02:18:12.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:12 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:18:13.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 100982 8.81 MB/sec execute 497 sec latency 61.961 ms 2024-04-25T02:18:13.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:13 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:18:13.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:13 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:18:13.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:13 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:18:13.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:13 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:18:13.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:13 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:18:13.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:13 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:18:13.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:13 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:18:13.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:13 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:18:13.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:13 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:18:14.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 101120 8.80 MB/sec execute 498 sec latency 54.411 ms 2024-04-25T02:18:14.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:14 smithi040 ceph-mon[32418]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 280 op/s 2024-04-25T02:18:14.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:14 smithi134 ceph-mon[33098]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 280 op/s 2024-04-25T02:18:14.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:14 smithi031 ceph-mon[29204]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 280 op/s 2024-04-25T02:18:15.109 INFO:tasks.workunit.client.0.smithi031.stdout: 10 101295 8.81 MB/sec execute 499 sec latency 91.105 ms 2024-04-25T02:18:16.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 101452 8.80 MB/sec execute 500 sec latency 56.291 ms 2024-04-25T02:18:16.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:16 smithi040 ceph-mon[32418]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 256 op/s 2024-04-25T02:18:16.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:16 smithi134 ceph-mon[33098]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 256 op/s 2024-04-25T02:18:16.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:16 smithi031 ceph-mon[29204]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 256 op/s 2024-04-25T02:18:17.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 101604 8.80 MB/sec execute 501 sec latency 56.826 ms 2024-04-25T02:18:18.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 101752 8.79 MB/sec execute 502 sec latency 60.960 ms 2024-04-25T02:18:18.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:18 smithi040 ceph-mon[32418]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 256 op/s 2024-04-25T02:18:18.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:18 smithi134 ceph-mon[33098]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 256 op/s 2024-04-25T02:18:18.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:18 smithi031 ceph-mon[29204]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 256 op/s 2024-04-25T02:18:19.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 101922 8.80 MB/sec execute 503 sec latency 60.468 ms 2024-04-25T02:18:20.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 102073 8.79 MB/sec execute 504 sec latency 80.715 ms 2024-04-25T02:18:20.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:20 smithi040 ceph-mon[32418]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 256 op/s 2024-04-25T02:18:20.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:20 smithi134 ceph-mon[33098]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 256 op/s 2024-04-25T02:18:20.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:20 smithi031 ceph-mon[29204]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 256 op/s 2024-04-25T02:18:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.091+0000 7fa3b6629640 1 -- 172.21.15.31:0/4151097981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3b0106630 msgr2=0x7fa3b0106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.091+0000 7fa3b6629640 1 --2- 172.21.15.31:0/4151097981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3b0106630 0x7fa3b0106a10 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009c10 tx=0x7fa3a802a6e0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.091+0000 7fa3b6629640 1 -- 172.21.15.31:0/4151097981 shutdown_connections 2024-04-25T02:18:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.091+0000 7fa3b6629640 1 --2- 172.21.15.31:0/4151097981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3b0101030 0x7fa3b010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.091+0000 7fa3b6629640 1 --2- 172.21.15.31:0/4151097981 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3b0100670 0x7fa3b0100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.091+0000 7fa3b6629640 1 --2- 172.21.15.31:0/4151097981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3b0106630 0x7fa3b0106a10 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.091+0000 7fa3b6629640 1 -- 172.21.15.31:0/4151097981 >> 172.21.15.31:0/4151097981 conn(0x7fa3b00777c0 msgr2=0x7fa3b0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.091+0000 7fa3b6629640 1 -- 172.21.15.31:0/4151097981 shutdown_connections 2024-04-25T02:18:21.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.091+0000 7fa3b6629640 1 -- 172.21.15.31:0/4151097981 wait complete. 2024-04-25T02:18:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.092+0000 7fa3b6629640 1 Processor -- start 2024-04-25T02:18:21.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.092+0000 7fa3b6629640 1 -- start start 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.092+0000 7fa3b6629640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3b0100670 0x7fa3b019fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.092+0000 7fa3b6629640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3b0101030 0x7fa3b01a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.092+0000 7fa3b6629640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3b0106630 0x7fa3b0199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.092+0000 7fa3b6629640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b010f7c0 con 0x7fa3b0101030 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.092+0000 7fa3b6629640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b010f640 con 0x7fa3b0106630 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.092+0000 7fa3b6629640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b010f940 con 0x7fa3b0100670 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.092+0000 7fa3af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3b0101030 0x7fa3b01a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3b0101030 0x7fa3b01a0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50236/0 (socket says 172.21.15.31:50236) 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3af7fe640 1 -- 172.21.15.31:0/2951148550 learned_addr learned my addr 172.21.15.31:0/2951148550 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3b4bc6640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3b0106630 0x7fa3b0199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3affff640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3b0100670 0x7fa3b019fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3af7fe640 1 -- 172.21.15.31:0/2951148550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3b0100670 msgr2=0x7fa3b019fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3b0100670 0x7fa3b019fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3af7fe640 1 -- 172.21.15.31:0/2951148550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3b0106630 msgr2=0x7fa3b0199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3b0106630 0x7fa3b0199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3af7fe640 1 -- 172.21.15.31:0/2951148550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b019a820 con 0x7fa3b0101030 2024-04-25T02:18:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3affff640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3b0100670 0x7fa3b019fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:18:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3b0101030 0x7fa3b01a0350 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fa39800e9f0 tx=0x7fa39800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3ad7fa640 1 -- 172.21.15.31:0/2951148550 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39800cd80 con 0x7fa3b0101030 2024-04-25T02:18:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3b6629640 1 -- 172.21.15.31:0/2951148550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b019ab10 con 0x7fa3b0101030 2024-04-25T02:18:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.093+0000 7fa3ad7fa640 1 -- 172.21.15.31:0/2951148550 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3980108e0 con 0x7fa3b0101030 2024-04-25T02:18:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.094+0000 7fa3ad7fa640 1 -- 172.21.15.31:0/2951148550 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398018920 con 0x7fa3b0101030 2024-04-25T02:18:21.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.094+0000 7fa3b6629640 1 -- 172.21.15.31:0/2951148550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b01a6d90 con 0x7fa3b0101030 2024-04-25T02:18:21.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.096+0000 7fa3ad7fa640 1 -- 172.21.15.31:0/2951148550 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa398010a90 con 0x7fa3b0101030 2024-04-25T02:18:21.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.097+0000 7fa3b940b740 1 Processor -- start 2024-04-25T02:18:21.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.097+0000 7fa3b940b740 1 -- start start 2024-04-25T02:18:21.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.097+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e23370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.097+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e23940 0x556179e36500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.097+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e36a40 0x556179e38e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.097+0000 7fa3b940b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556179d87cb0 con 0x556179e22f90 2024-04-25T02:18:21.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.097+0000 7fa3b940b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556179d305b0 con 0x556179e36a40 2024-04-25T02:18:21.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.097+0000 7fa3b940b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556179d83c70 con 0x556179e23940 2024-04-25T02:18:21.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.097+0000 7fa3affff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e23940 0x556179e36500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.097+0000 7fa3b4bc6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e23370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.098+0000 7fa3affff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e23940 0x556179e36500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:58216/0 (socket says 172.21.15.31:58216) 2024-04-25T02:18:21.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.098+0000 7fa3affff640 1 -- 172.21.15.31:0/3268600276 learned_addr learned my addr 172.21.15.31:0/3268600276 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:21.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.098+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/3268600276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e36a40 0x556179e38e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.098+0000 7fa3affff640 1 -- 172.21.15.31:0/3268600276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e36a40 msgr2=0x556179e38e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:21.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.098+0000 7fa3affff640 1 --2- 172.21.15.31:0/3268600276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e36a40 0x556179e38e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.098+0000 7fa3affff640 1 -- 172.21.15.31:0/3268600276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 msgr2=0x556179e23370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:21.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.098+0000 7fa3affff640 1 --2- 172.21.15.31:0/3268600276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e23370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.098+0000 7fa3affff640 1 -- 172.21.15.31:0/3268600276 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556179a9c210 con 0x556179e23940 2024-04-25T02:18:21.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.098+0000 7fa3ad7fa640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa3a40a2da0 0x7fa3a40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.098+0000 7fa3affff640 1 --2- 172.21.15.31:0/3268600276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e23940 0x556179e36500 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8000f80 tx=0x7fa3a8030640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89b5b58c4a6f7744 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:21.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.099+0000 7fa3b4bc6640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa3a40a2da0 0x7fa3a40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.099+0000 7fa3ad7fa640 1 -- 172.21.15.31:0/2951148550 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7fa3980cf7b0 con 0x7fa3b0101030 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.099+0000 7fa39e7fc640 1 -- 172.21.15.31:0/3268600276 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8041020 con 0x556179e23940 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.099+0000 7fa39e7fc640 1 -- 172.21.15.31:0/3268600276 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3a8037040 con 0x556179e23940 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.099+0000 7fa39e7fc640 1 -- 172.21.15.31:0/3268600276 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8036070 con 0x556179e23940 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b4bc6640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa3a40a2da0 0x7fa3a40a5260 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009530 tx=0x7fa3a0009260 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b940b740 1 -- 172.21.15.31:0/3268600276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e23940 msgr2=0x556179e36500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b940b740 1 --2- 172.21.15.31:0/3268600276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e23940 0x556179e36500 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8000f80 tx=0x7fa3a8030640 comp rx=0 tx=0).stop 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b940b740 1 -- 172.21.15.31:0/3268600276 shutdown_connections 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b940b740 1 --2- 172.21.15.31:0/3268600276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e36a40 0x556179e38e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b940b740 1 --2- 172.21.15.31:0/3268600276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e23940 0x556179e36500 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b940b740 1 --2- 172.21.15.31:0/3268600276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e23370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b940b740 1 -- 172.21.15.31:0/3268600276 >> 172.21.15.31:0/3268600276 conn(0x556179e1f560 msgr2=0x556179e1f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b940b740 1 -- 172.21.15.31:0/3268600276 shutdown_connections 2024-04-25T02:18:21.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b940b740 1 -- 172.21.15.31:0/3268600276 wait complete. 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b940b740 1 -- start start 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.100+0000 7fa3b940b740 1 Processor -- start 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e33990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 0x556179e33ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e36a40 0x556179e34410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3b940b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556179e397a0 con 0x556179e22f90 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3b940b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556179d43b50 con 0x556179e23940 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3b940b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556179e39920 con 0x556179e36a40 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3b940b740 10 client.0 _ll_register_callbacks cb 0x556179d7eca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e33990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e33990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50252/0 (socket says 172.21.15.31:50252) 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3affff640 1 -- 172.21.15.31:0/349060343 learned_addr learned my addr 172.21.15.31:0/349060343 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 0x556179e33ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3b4bc6640 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e36a40 0x556179e34410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3affff640 1 -- 172.21.15.31:0/349060343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e36a40 msgr2=0x556179e34410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3affff640 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e36a40 0x556179e34410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3affff640 1 -- 172.21.15.31:0/349060343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 msgr2=0x556179e33ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3affff640 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 0x556179e33ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.101+0000 7fa3affff640 1 -- 172.21.15.31:0/349060343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556179b68620 con 0x556179e22f90 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.102+0000 7fa3affff640 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e33990 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fa3a802ac00 tx=0x7fa3a8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1862fae9b336af0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.102+0000 7fa38f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.102+0000 7fa38f7fe640 1 -- 172.21.15.31:0/349060343 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8036070 con 0x556179e22f90 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.102+0000 7fa38f7fe640 1 -- 172.21.15.31:0/349060343 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7fa3a8030d30 con 0x556179e22f90 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.102+0000 7fa3b940b740 1 -- 172.21.15.31:0/349060343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3a8037040 con 0x556179e22f90 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.102+0000 7fa38f7fe640 1 -- 172.21.15.31:0/349060343 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fa3a8041050 con 0x556179e22f90 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.102+0000 7fa3b940b740 10 client.18036 fetch_fsmap learned FSMap version 51 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.102+0000 7fa3b940b740 1 -- 172.21.15.31:0/349060343 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x556179e459a0 con 0x556179e22f90 2024-04-25T02:18:21.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.103+0000 7fa38f7fe640 1 -- 172.21.15.31:0/349060343 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3a8037040 con 0x556179e22f90 2024-04-25T02:18:21.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.103+0000 7fa38f7fe640 1 -- 172.21.15.31:0/349060343 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7fa3a8056020 con 0x556179e22f90 2024-04-25T02:18:21.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.104+0000 7fa3b940b740 10 client.18036 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:18:21.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.104+0000 7fa3b940b740 10 client.18036 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:18:21.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.104+0000 7fa3b940b740 20 client.18036 populate_metadata read hostname 'smithi031' 2024-04-25T02:18:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.104+0000 7fa3b940b740 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556179e46cf0 0x556179e490e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.104+0000 7fa3b940b740 4 client.18036 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:18:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.104+0000 7fa3b940b740 1 -- 172.21.15.31:0/349060343 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556179d951f0 con 0x556179e46cf0 2024-04-25T02:18:21.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.104+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556179e46cf0 0x556179e490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 102227 8.78 MB/sec execute 505 sec latency 99.977 ms 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.106+0000 7fa38f7fe640 10 client.18036 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.106+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556179e46cf0 0x556179e490e0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dd53690da8eab3a server_cookie=987b59d3b2a1db3c in_seq=0 out_seq=0 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.108+0000 7fa38f7fe640 1 -- 172.21.15.31:0/349060343 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556179d951f0 con 0x556179e46cf0 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.108+0000 7fa38f7fe640 10 client.18036 handle_command_reply: tid=0 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.108+0000 7fa3b940b740 1 client.18036 shutdown 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.108+0000 7fa3b940b740 1 -- 172.21.15.31:0/349060343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 msgr2=0x556179e33990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.108+0000 7fa3b940b740 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e33990 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fa3a802ac00 tx=0x7fa3a8003ba0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.109+0000 7fa3b940b740 1 -- 172.21.15.31:0/349060343 shutdown_connections 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.109+0000 7fa3b940b740 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556179e46cf0 0x556179e490e0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.109+0000 7fa3b940b740 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e36a40 0x556179e34410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.109+0000 7fa38f7fe640 0 client.18036 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:21.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.109+0000 7fa3b940b740 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 0x556179e33ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.109+0000 7fa3b940b740 1 --2- 172.21.15.31:0/349060343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e33990 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.109+0000 7fa3b940b740 1 -- 172.21.15.31:0/349060343 >> 172.21.15.31:0/349060343 conn(0x556179e1f560 msgr2=0x556179e25360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.109+0000 7fa3b940b740 1 -- 172.21.15.31:0/349060343 shutdown_connections 2024-04-25T02:18:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.109+0000 7fa3b940b740 1 -- 172.21.15.31:0/349060343 wait complete. 2024-04-25T02:18:21.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.110+0000 7fa3b940b740 20 client.18036 trim_cache size 0 max 16384 2024-04-25T02:18:21.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3b940b740 1 Processor -- start 2024-04-25T02:18:21.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3b940b740 1 -- start start 2024-04-25T02:18:21.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e33140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 0x556179e422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e36a40 0x556179e4c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3b940b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01ab1d0 con 0x556179e22f90 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3b940b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01acd10 con 0x556179e23940 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3b940b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01ab8a0 con 0x556179e36a40 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e33140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3af7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 0x556179e422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3af7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 0x556179e422c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44930/0 (socket says 172.21.15.31:44930) 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e33140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50254/0 (socket says 172.21.15.31:50254) 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3af7fe640 1 -- 172.21.15.31:0/916574298 learned_addr learned my addr 172.21.15.31:0/916574298 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.137+0000 7fa3b4bc6640 1 --2- 172.21.15.31:0/916574298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e36a40 0x556179e4c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.138+0000 7fa3af7fe640 1 -- 172.21.15.31:0/916574298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e36a40 msgr2=0x556179e4c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.138+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/916574298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e36a40 0x556179e4c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.138+0000 7fa3af7fe640 1 -- 172.21.15.31:0/916574298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 msgr2=0x556179e33140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:21.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.138+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/916574298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e33140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.138+0000 7fa3af7fe640 1 -- 172.21.15.31:0/916574298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556179e39840 con 0x556179e23940 2024-04-25T02:18:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.138+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/916574298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 0x556179e422c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa3980ccf80 tx=0x7fa3980d1cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f42e8a86665aaafc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.138+0000 7fa39dffb640 1 -- 172.21.15.31:0/916574298 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39809b650 con 0x556179e23940 2024-04-25T02:18:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.138+0000 7fa39dffb640 1 -- 172.21.15.31:0/916574298 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa398010de0 con 0x556179e23940 2024-04-25T02:18:21.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.138+0000 7fa39dffb640 1 -- 172.21.15.31:0/916574298 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398098360 con 0x556179e23940 2024-04-25T02:18:21.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.139+0000 7fa3b940b740 1 -- 172.21.15.31:0/916574298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 msgr2=0x556179e422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:21.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.139+0000 7fa3b940b740 1 --2- 172.21.15.31:0/916574298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 0x556179e422c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa3980ccf80 tx=0x7fa3980d1cc0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.139+0000 7fa3b940b740 1 -- 172.21.15.31:0/916574298 shutdown_connections 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.139+0000 7fa3b940b740 1 --2- 172.21.15.31:0/916574298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e36a40 0x556179e4c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.139+0000 7fa3b940b740 1 --2- 172.21.15.31:0/916574298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e23940 0x556179e422c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.139+0000 7fa3b940b740 1 --2- 172.21.15.31:0/916574298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e22f90 0x556179e33140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.139+0000 7fa3b940b740 1 -- 172.21.15.31:0/916574298 >> 172.21.15.31:0/916574298 conn(0x556179e1f560 msgr2=0x556179e41bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.139+0000 7fa3b940b740 1 -- 172.21.15.31:0/916574298 shutdown_connections 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.139+0000 7fa3b940b740 1 -- 172.21.15.31:0/916574298 wait complete. 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.139+0000 7fa3b940b740 1 -- start start 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3b940b740 1 Processor -- start 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e22f90 0x556179e4c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e23940 0x556179e49980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3b940b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e36a40 0x556179e49f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3b940b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01ac640 con 0x556179e23940 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3b940b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01abf70 con 0x556179e36a40 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3b940b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01a9230 con 0x556179e22f90 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3b940b740 10 client.0 _ll_register_callbacks cb 0x7fa3a0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3affff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e22f90 0x556179e4c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3affff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e22f90 0x556179e4c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:58234/0 (socket says 172.21.15.31:58234) 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3affff640 1 -- 172.21.15.31:0/525630466 learned_addr learned my addr 172.21.15.31:0/525630466 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e23940 0x556179e49980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.140+0000 7fa3b4bc6640 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e36a40 0x556179e49f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa3affff640 1 -- 172.21.15.31:0/525630466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e36a40 msgr2=0x556179e49f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa3affff640 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e36a40 0x556179e49f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa3affff640 1 -- 172.21.15.31:0/525630466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e23940 msgr2=0x556179e49980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:21.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa3affff640 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e23940 0x556179e49980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa3affff640 1 -- 172.21.15.31:0/525630466 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556179e4a7c0 con 0x556179e22f90 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e23940 0x556179e49980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa3affff640 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e22f90 0x556179e4c5f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa3a802ac00 tx=0x7fa3a8005c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13236e9c29d82260 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa38f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa38f7fe640 1 -- 172.21.15.31:0/525630466 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8036070 con 0x556179e22f90 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa3b940b740 1 -- 172.21.15.31:0/525630466 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556179d8f540 con 0x556179e22f90 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa38f7fe640 1 -- 172.21.15.31:0/525630466 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7fa3a80516f0 con 0x556179e22f90 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa38f7fe640 1 -- 172.21.15.31:0/525630466 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fa3a802ad20 con 0x556179e22f90 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa3b940b740 10 client.27382 fetch_fsmap learned FSMap version 51 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.141+0000 7fa3b940b740 1 -- 172.21.15.31:0/525630466 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x556179ead010 con 0x556179e22f90 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.142+0000 7fa38f7fe640 1 -- 172.21.15.31:0/525630466 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x556179d8f540 con 0x556179e22f90 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.142+0000 7fa38f7fe640 1 -- 172.21.15.31:0/525630466 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7fa3a8056040 con 0x556179e22f90 2024-04-25T02:18:21.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.142+0000 7fa3b940b740 10 client.27382 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:18:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.142+0000 7fa3b940b740 10 client.27382 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:18:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.142+0000 7fa3b940b740 20 client.27382 populate_metadata read hostname 'smithi031' 2024-04-25T02:18:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.142+0000 7fa3b940b740 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556179e46cf0 0x556179eaf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.142+0000 7fa3b940b740 4 client.27382 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:18:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.142+0000 7fa3b940b740 1 -- 172.21.15.31:0/525630466 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556179d0d8e0 con 0x556179e46cf0 2024-04-25T02:18:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.142+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556179e46cf0 0x556179eaf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.143+0000 7fa3af7fe640 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556179e46cf0 0x556179eaf700 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=860c964eb08ff33c server_cookie=9d2fc34be0c0c4bf in_seq=0 out_seq=0 2024-04-25T02:18:21.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.143+0000 7fa38f7fe640 10 client.27382 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa38f7fe640 1 -- 172.21.15.31:0/525630466 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556179d0d8e0 con 0x556179e46cf0 2024-04-25T02:18:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa38f7fe640 10 client.27382 handle_command_reply: tid=0 2024-04-25T02:18:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa3b940b740 1 client.27382 shutdown 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa3b940b740 1 -- 172.21.15.31:0/525630466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e22f90 msgr2=0x556179e4c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa3b940b740 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e22f90 0x556179e4c5f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa3a802ac00 tx=0x7fa3a8005c80 comp rx=0 tx=0).stop 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa3b940b740 1 -- 172.21.15.31:0/525630466 shutdown_connections 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa3b940b740 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x556179e46cf0 0x556179eaf700 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa3b940b740 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556179e36a40 0x556179e49f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa38f7fe640 0 client.27382 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa3b940b740 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556179e23940 0x556179e49980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa3b940b740 1 --2- 172.21.15.31:0/525630466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556179e22f90 0x556179e4c5f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.205+0000 7fa3b940b740 1 -- 172.21.15.31:0/525630466 >> 172.21.15.31:0/525630466 conn(0x556179e1f560 msgr2=0x556179e42440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.206+0000 7fa3b940b740 1 -- 172.21.15.31:0/525630466 shutdown_connections 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.206+0000 7fa3b940b740 1 -- 172.21.15.31:0/525630466 wait complete. 2024-04-25T02:18:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.206+0000 7fa3b940b740 20 client.27382 trim_cache size 0 max 16384 2024-04-25T02:18:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:18:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:18:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:18:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:18:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.208+0000 7fa3b6629640 1 -- 172.21.15.31:0/2951148550 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa3a40a2da0 msgr2=0x7fa3a40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.208+0000 7fa3b6629640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa3a40a2da0 0x7fa3a40a5260 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009530 tx=0x7fa3a0009260 comp rx=0 tx=0).stop 2024-04-25T02:18:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.209+0000 7fa3b6629640 1 -- 172.21.15.31:0/2951148550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3b0101030 msgr2=0x7fa3b01a0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.209+0000 7fa3b6629640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3b0101030 0x7fa3b01a0350 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fa39800e9f0 tx=0x7fa39800eeb0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.211+0000 7fa3b6629640 1 -- 172.21.15.31:0/2951148550 shutdown_connections 2024-04-25T02:18:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.211+0000 7fa3b6629640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa3a40a2da0 0x7fa3a40a5260 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.211+0000 7fa3b6629640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3b0106630 0x7fa3b0199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.211+0000 7fa3b6629640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3b0101030 0x7fa3b01a0350 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.211+0000 7fa3b6629640 1 --2- 172.21.15.31:0/2951148550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3b0100670 0x7fa3b019fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.211+0000 7fa3b6629640 1 -- 172.21.15.31:0/2951148550 >> 172.21.15.31:0/2951148550 conn(0x7fa3b00777c0 msgr2=0x7fa3b00ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.211+0000 7fa3b6629640 1 -- 172.21.15.31:0/2951148550 shutdown_connections 2024-04-25T02:18:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:21.211+0000 7fa3b6629640 1 -- 172.21.15.31:0/2951148550 wait complete. 2024-04-25T02:18:22.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 102376 8.78 MB/sec execute 506 sec latency 181.109 ms 2024-04-25T02:18:22.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:22 smithi040 ceph-mon[32418]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 8.6 MiB/s wr, 350 op/s 2024-04-25T02:18:22.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:22 smithi134 ceph-mon[33098]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 8.6 MiB/s wr, 350 op/s 2024-04-25T02:18:22.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:22 smithi031 ceph-mon[29204]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 8.6 MiB/s wr, 350 op/s 2024-04-25T02:18:23.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 102537 8.77 MB/sec execute 507 sec latency 59.640 ms 2024-04-25T02:18:24.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 102699 8.76 MB/sec execute 508 sec latency 49.871 ms 2024-04-25T02:18:24.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:24 smithi040 ceph-mon[32418]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.4 MiB/s wr, 240 op/s 2024-04-25T02:18:24.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:24 smithi134 ceph-mon[33098]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.4 MiB/s wr, 240 op/s 2024-04-25T02:18:24.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:24 smithi031 ceph-mon[29204]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.4 MiB/s wr, 240 op/s 2024-04-25T02:18:25.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 102905 8.77 MB/sec execute 509 sec latency 64.173 ms 2024-04-25T02:18:26.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 103105 8.77 MB/sec execute 510 sec latency 76.240 ms 2024-04-25T02:18:26.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:26 smithi040 ceph-mon[32418]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.5 MiB/s wr, 362 op/s 2024-04-25T02:18:26.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:26 smithi134 ceph-mon[33098]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.5 MiB/s wr, 362 op/s 2024-04-25T02:18:26.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:26 smithi031 ceph-mon[29204]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 7.5 MiB/s wr, 362 op/s 2024-04-25T02:18:27.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 103283 8.78 MB/sec execute 511 sec latency 109.273 ms 2024-04-25T02:18:28.110 INFO:tasks.workunit.client.0.smithi031.stdout: 10 103469 8.79 MB/sec execute 512 sec latency 94.278 ms 2024-04-25T02:18:28.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:28 smithi040 ceph-mon[32418]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.7 MiB/s wr, 256 op/s 2024-04-25T02:18:28.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:28 smithi134 ceph-mon[33098]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.7 MiB/s wr, 256 op/s 2024-04-25T02:18:28.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:28 smithi031 ceph-mon[29204]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.7 MiB/s wr, 256 op/s 2024-04-25T02:18:29.111 INFO:tasks.workunit.client.0.smithi031.stdout: 10 103686 8.81 MB/sec execute 513 sec latency 75.045 ms 2024-04-25T02:18:30.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 103832 8.81 MB/sec execute 514 sec latency 198.471 ms 2024-04-25T02:18:30.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:30 smithi040 ceph-mon[32418]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.7 MiB/s wr, 256 op/s 2024-04-25T02:18:30.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:30 smithi134 ceph-mon[33098]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.7 MiB/s wr, 256 op/s 2024-04-25T02:18:30.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:30 smithi031 ceph-mon[29204]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.7 MiB/s wr, 256 op/s 2024-04-25T02:18:31.111 INFO:tasks.workunit.client.0.smithi031.stdout: 10 104008 8.82 MB/sec execute 515 sec latency 59.234 ms 2024-04-25T02:18:32.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 104161 8.81 MB/sec execute 516 sec latency 68.153 ms 2024-04-25T02:18:32.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:32 smithi040 ceph-mon[32418]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.0 MiB/s wr, 365 op/s 2024-04-25T02:18:32.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:32 smithi134 ceph-mon[33098]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.0 MiB/s wr, 365 op/s 2024-04-25T02:18:32.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:32 smithi031 ceph-mon[29204]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.0 MiB/s wr, 365 op/s 2024-04-25T02:18:33.111 INFO:tasks.workunit.client.0.smithi031.stdout: 10 104314 8.82 MB/sec execute 517 sec latency 77.150 ms 2024-04-25T02:18:34.111 INFO:tasks.workunit.client.0.smithi031.stdout: 10 104442 8.81 MB/sec execute 518 sec latency 106.918 ms 2024-04-25T02:18:34.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.348+0000 7f5b9c691640 1 -- 172.21.15.31:0/1344839618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b94103930 msgr2=0x7f5b94103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.348+0000 7f5b9c691640 1 --2- 172.21.15.31:0/1344839618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b94103930 0x7f5b94103db0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5b84009930 tx=0x7f5b8402a420 comp rx=0 tx=0).stop 2024-04-25T02:18:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.348+0000 7f5b9c691640 1 -- 172.21.15.31:0/1344839618 shutdown_connections 2024-04-25T02:18:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.349+0000 7f5b9c691640 1 --2- 172.21.15.31:0/1344839618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b941042f0 0x7f5b94110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.349+0000 7f5b9c691640 1 --2- 172.21.15.31:0/1344839618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b94103930 0x7f5b94103db0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.349+0000 7f5b9c691640 1 --2- 172.21.15.31:0/1344839618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b94109940 0x7f5b94109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.349+0000 7f5b9c691640 1 -- 172.21.15.31:0/1344839618 >> 172.21.15.31:0/1344839618 conn(0x7f5b940ff440 msgr2=0x7f5b94101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.349+0000 7f5b9c691640 1 -- 172.21.15.31:0/1344839618 shutdown_connections 2024-04-25T02:18:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.349+0000 7f5b9c691640 1 -- 172.21.15.31:0/1344839618 wait complete. 2024-04-25T02:18:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.349+0000 7f5b9c691640 1 Processor -- start 2024-04-25T02:18:34.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.349+0000 7f5b9c691640 1 -- start start 2024-04-25T02:18:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b9c691640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b94103930 0x7f5b9419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b9c691640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b941042f0 0x7f5b9419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b9c691640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b94109940 0x7f5b9419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b9c691640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b94113c50 con 0x7f5b94103930 2024-04-25T02:18:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b9c691640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b94113ad0 con 0x7f5b941042f0 2024-04-25T02:18:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b9c691640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5b94113dd0 con 0x7f5b94109940 2024-04-25T02:18:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b99c2c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b941042f0 0x7f5b9419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b9a42d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b94103930 0x7f5b9419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b99c2c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b941042f0 0x7f5b9419a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:34804/0 (socket says 172.21.15.31:34804) 2024-04-25T02:18:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b99c2c640 1 -- 172.21.15.31:0/651723039 learned_addr learned my addr 172.21.15.31:0/651723039 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b9ac2e640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b94109940 0x7f5b9419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b99c2c640 1 -- 172.21.15.31:0/651723039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b94109940 msgr2=0x7f5b9419e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b94109940 0x7f5b9419e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b99c2c640 1 -- 172.21.15.31:0/651723039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b94103930 msgr2=0x7f5b9419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b94103930 0x7f5b9419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.350+0000 7f5b99c2c640 1 -- 172.21.15.31:0/651723039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9419f0d0 con 0x7f5b941042f0 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.351+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b941042f0 0x7f5b9419a670 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5b84005ab0 tx=0x7f5b84003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.351+0000 7f5b8b7fe640 1 -- 172.21.15.31:0/651723039 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8403f070 con 0x7f5b941042f0 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.351+0000 7f5b8b7fe640 1 -- 172.21.15.31:0/651723039 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b84031b50 con 0x7f5b941042f0 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.351+0000 7f5b8b7fe640 1 -- 172.21.15.31:0/651723039 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84035030 con 0x7f5b941042f0 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.351+0000 7f5b9c691640 1 -- 172.21.15.31:0/651723039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b9419f360 con 0x7f5b941042f0 2024-04-25T02:18:34.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.351+0000 7f5b9c691640 1 -- 172.21.15.31:0/651723039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b941a6c50 con 0x7f5b941042f0 2024-04-25T02:18:34.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.354+0000 7f5b8b7fe640 1 -- 172.21.15.31:0/651723039 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5b84042070 con 0x7f5b941042f0 2024-04-25T02:18:34.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.354+0000 7f5b9f395740 1 Processor -- start 2024-04-25T02:18:34.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.354+0000 7f5b9f395740 1 -- start start 2024-04-25T02:18:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9f395740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602be2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9f395740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602be2b60 0x563602beccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9f395740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 0x563602bef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9f395740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563602b46cb0 con 0x563602bed230 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b8b7fe640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5b800a2e20 0x7f5b800a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9f395740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563602aef5b0 con 0x563602be2b60 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9f395740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563602b42c70 con 0x563602be2240 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9ac2e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602be2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9ac2e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602be2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:41436/0 (socket says 172.21.15.31:41436) 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9ac2e640 1 -- 172.21.15.31:0/4122623235 learned_addr learned my addr 172.21.15.31:0/4122623235 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9ac2e640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5b800a2e20 0x7f5b800a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/4122623235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602be2b60 0x563602beccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/4122623235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 0x563602bef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b8b7fe640 1 -- 172.21.15.31:0/651723039 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7f5b840efa10 con 0x7f5b941042f0 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9a42d640 1 -- 172.21.15.31:0/4122623235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 msgr2=0x563602be2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/4122623235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602be2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9a42d640 1 -- 172.21.15.31:0/4122623235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 msgr2=0x563602bef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:34.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/4122623235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 0x563602bef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b9a42d640 1 -- 172.21.15.31:0/4122623235 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56360285b210 con 0x563602be2b60 2024-04-25T02:18:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.355+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/4122623235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 0x563602bef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:18:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.356+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/4122623235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602be2b60 0x563602beccf0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5b90009c20 tx=0x7f5b90011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e097d86f1fa91599 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.356+0000 7f5b9ac2e640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5b800a2e20 0x7f5b800a52e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009270 tx=0x7f5b8c009ba0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.356+0000 7f5b88ff9640 1 -- 172.21.15.31:0/4122623235 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b90019420 con 0x563602be2b60 2024-04-25T02:18:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.356+0000 7f5b88ff9640 1 -- 172.21.15.31:0/4122623235 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b90019a70 con 0x563602be2b60 2024-04-25T02:18:34.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.356+0000 7f5b88ff9640 1 -- 172.21.15.31:0/4122623235 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b90018ae0 con 0x563602be2b60 2024-04-25T02:18:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.357+0000 7f5b9f395740 1 -- 172.21.15.31:0/4122623235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602be2b60 msgr2=0x563602beccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.357+0000 7f5b9f395740 1 --2- 172.21.15.31:0/4122623235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602be2b60 0x563602beccf0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5b90009c20 tx=0x7f5b90011810 comp rx=0 tx=0).stop 2024-04-25T02:18:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.357+0000 7f5b9f395740 1 -- 172.21.15.31:0/4122623235 shutdown_connections 2024-04-25T02:18:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.357+0000 7f5b9f395740 1 --2- 172.21.15.31:0/4122623235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 0x563602bef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.357+0000 7f5b9f395740 1 --2- 172.21.15.31:0/4122623235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602be2b60 0x563602beccf0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.357+0000 7f5b9f395740 1 --2- 172.21.15.31:0/4122623235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602be2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.357+0000 7f5b9f395740 1 -- 172.21.15.31:0/4122623235 >> 172.21.15.31:0/4122623235 conn(0x563602bde560 msgr2=0x563602bde940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.357+0000 7f5b9f395740 1 -- 172.21.15.31:0/4122623235 shutdown_connections 2024-04-25T02:18:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.357+0000 7f5b9f395740 1 -- 172.21.15.31:0/4122623235 wait complete. 2024-04-25T02:18:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.357+0000 7f5b9f395740 1 -- start start 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.357+0000 7f5b9f395740 1 Processor -- start 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b9f395740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602bea0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b9f395740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2b60 0x563602bea630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b9a42d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602bea0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b9a42d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602bea0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:41450/0 (socket says 172.21.15.31:41450) 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b99c2c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2b60 0x563602bea630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b99c2c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2b60 0x563602bea630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41370/0 (socket says 172.21.15.31:41370) 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b9a42d640 1 -- 172.21.15.31:0/3133708304 learned_addr learned my addr 172.21.15.31:0/3133708304 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602bed230 0x563602beab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b9f395740 1 -- 172.21.15.31:0/3133708304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563602bf00c0 con 0x563602be2b60 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b9f395740 1 -- 172.21.15.31:0/3133708304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563602b02b50 con 0x563602bed230 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b9f395740 1 -- 172.21.15.31:0/3133708304 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563602bf0240 con 0x563602be2240 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.358+0000 7f5b9ac2e640 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602bed230 0x563602beab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b9f395740 10 client.0 _ll_register_callbacks cb 0x563602b3dca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b99c2c640 1 -- 172.21.15.31:0/3133708304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 msgr2=0x563602bea0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602bea0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b99c2c640 1 -- 172.21.15.31:0/3133708304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602bed230 msgr2=0x563602beab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602bed230 0x563602beab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b99c2c640 1 -- 172.21.15.31:0/3133708304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563602927620 con 0x563602be2b60 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b9f395740 1 -- 172.21.15.31:0/3133708304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b90019a70 con 0x563602be2b60 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2b60 0x563602bea630 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5b840ed180 tx=0x7f5b840dc180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13440bece3c60eec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b7dffb640 10 client.18054 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b7dffb640 1 -- 172.21.15.31:0/3133708304 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b840d4070 con 0x563602be2b60 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b7dffb640 1 -- 172.21.15.31:0/3133708304 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7f5b840b8670 con 0x563602be2b60 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b7dffb640 1 -- 172.21.15.31:0/3133708304 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f5b84049040 con 0x563602be2b60 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b9f395740 10 client.18054 fetch_fsmap learned FSMap version 51 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.359+0000 7f5b9f395740 1 -- 172.21.15.31:0/3133708304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=51}) -- 0x563602c04880 con 0x563602be2b60 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.360+0000 7f5b7dffb640 1 -- 172.21.15.31:0/3133708304 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f5b8403b460 con 0x563602be2b60 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.360+0000 7f5b9f395740 10 client.18054 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:18:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.360+0000 7f5b9f395740 10 client.18054 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:18:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.360+0000 7f5b9f395740 20 client.18054 populate_metadata read hostname 'smithi031' 2024-04-25T02:18:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.360+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563602c05d10 0x563602c08160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.360+0000 7f5b9f395740 4 client.18054 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:18:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.360+0000 7f5b9f395740 1 -- 172.21.15.31:0/3133708304 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563602b541f0 con 0x563602c05d10 2024-04-25T02:18:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.360+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563602c05d10 0x563602c08160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.361+0000 7f5b7dffb640 10 client.18054 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.361+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563602c05d10 0x563602c08160 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6dc8e57589b069e server_cookie=b94fd9666d571667 in_seq=0 out_seq=0 2024-04-25T02:18:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.361+0000 7f5b7dffb640 1 -- 172.21.15.31:0/3133708304 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563602b541f0 con 0x563602c05d10 2024-04-25T02:18:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.361+0000 7f5b7dffb640 10 client.18054 handle_command_reply: tid=0 2024-04-25T02:18:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.361+0000 7f5b9f395740 1 client.18054 shutdown 2024-04-25T02:18:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b9f395740 1 -- 172.21.15.31:0/3133708304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2b60 msgr2=0x563602bea630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2b60 0x563602bea630 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5b840ed180 tx=0x7f5b840dc180 comp rx=0 tx=0).stop 2024-04-25T02:18:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b9f395740 1 -- 172.21.15.31:0/3133708304 shutdown_connections 2024-04-25T02:18:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563602c05d10 0x563602c08160 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602bed230 0x563602beab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b7dffb640 0 client.18054 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2b60 0x563602bea630 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3133708304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602bea0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b9f395740 1 -- 172.21.15.31:0/3133708304 >> 172.21.15.31:0/3133708304 conn(0x563602bde560 msgr2=0x563602be4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b9f395740 1 -- 172.21.15.31:0/3133708304 shutdown_connections 2024-04-25T02:18:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b9f395740 1 -- 172.21.15.31:0/3133708304 wait complete. 2024-04-25T02:18:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.362+0000 7f5b9f395740 20 client.18054 trim_cache size 0 max 16384 2024-04-25T02:18:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9f395740 1 Processor -- start 2024-04-25T02:18:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9f395740 1 -- start start 2024-04-25T02:18:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9f395740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602bec820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9f395740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602be2b60 0x563602be49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9f395740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 0x563602bdd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9f395740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941ab0a0 con 0x563602bed230 2024-04-25T02:18:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9f395740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941acbe0 con 0x563602be2b60 2024-04-25T02:18:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9f395740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941ab770 con 0x563602be2240 2024-04-25T02:18:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9ac2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 0x563602bdd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9a42d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602bec820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9ac2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 0x563602bdd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41380/0 (socket says 172.21.15.31:41380) 2024-04-25T02:18:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b9ac2e640 1 -- 172.21.15.31:0/3574118368 learned_addr learned my addr 172.21.15.31:0/3574118368 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.388+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/3574118368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602be2b60 0x563602be49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.389+0000 7f5b9a42d640 1 -- 172.21.15.31:0/3574118368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602be2b60 msgr2=0x563602be49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.389+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/3574118368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602be2b60 0x563602be49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.389+0000 7f5b9a42d640 1 -- 172.21.15.31:0/3574118368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 msgr2=0x563602bdd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.389+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/3574118368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 0x563602bdd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.389+0000 7f5b9a42d640 1 -- 172.21.15.31:0/3574118368 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563602be34d0 con 0x563602be2240 2024-04-25T02:18:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.389+0000 7f5b9ac2e640 1 --2- 172.21.15.31:0/3574118368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 0x563602bdd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:18:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.389+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/3574118368 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602bec820 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b90000c00 tx=0x7f5b9001ae70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97228106d8024798 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.389+0000 7f5b7ffff640 1 -- 172.21.15.31:0/3574118368 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9002f070 con 0x563602be2240 2024-04-25T02:18:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.390+0000 7f5b7ffff640 1 -- 172.21.15.31:0/3574118368 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b900052c0 con 0x563602be2240 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.390+0000 7f5b7ffff640 1 -- 172.21.15.31:0/3574118368 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b90003850 con 0x563602be2240 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 -- 172.21.15.31:0/3574118368 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 msgr2=0x563602bec820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3574118368 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602bec820 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b90000c00 tx=0x7f5b9001ae70 comp rx=0 tx=0).stop 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 -- 172.21.15.31:0/3574118368 shutdown_connections 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3574118368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602bed230 0x563602bdd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3574118368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602be2b60 0x563602be49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3574118368 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2240 0x563602bec820 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 -- 172.21.15.31:0/3574118368 >> 172.21.15.31:0/3574118368 conn(0x563602bde560 msgr2=0x563602bdcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 -- 172.21.15.31:0/3574118368 shutdown_connections 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 -- 172.21.15.31:0/3574118368 wait complete. 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 -- start start 2024-04-25T02:18:34.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 Processor -- start 2024-04-25T02:18:34.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.391+0000 7f5b9f395740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2240 0x563602be6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9f395740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2b60 0x563602be73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9f395740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602bed230 0x563602be7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9f395740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941ac510 con 0x563602be2240 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9f395740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941abe40 con 0x563602bed230 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9f395740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941a90c0 con 0x563602be2b60 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9a42d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2240 0x563602be6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9f395740 10 client.0 _ll_register_callbacks cb 0x7f5b840b7c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9a42d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2240 0x563602be6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41388/0 (socket says 172.21.15.31:41388) 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9a42d640 1 -- 172.21.15.31:0/3611523128 learned_addr learned my addr 172.21.15.31:0/3611523128 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2b60 0x563602be73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9ac2e640 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602bed230 0x563602be7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9a42d640 1 -- 172.21.15.31:0/3611523128 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2b60 msgr2=0x563602be73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2b60 0x563602be73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9a42d640 1 -- 172.21.15.31:0/3611523128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602bed230 msgr2=0x563602be7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:34.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602bed230 0x563602be7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9a42d640 1 -- 172.21.15.31:0/3611523128 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563602c6b360 con 0x563602be2240 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9ac2e640 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602bed230 0x563602be7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b9a42d640 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2240 0x563602be6e80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f5b90009460 tx=0x7f5b90009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e2b767b5649dd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.392+0000 7f5b7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.393+0000 7f5b7dffb640 1 -- 172.21.15.31:0/3611523128 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9002f040 con 0x563602be2240 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.393+0000 7f5b9f395740 1 -- 172.21.15.31:0/3611523128 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b900052c0 con 0x563602be2240 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.393+0000 7f5b7dffb640 1 -- 172.21.15.31:0/3611523128 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7f5b90019420 con 0x563602be2240 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.393+0000 7f5b7dffb640 1 -- 172.21.15.31:0/3611523128 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f5b90005b90 con 0x563602be2240 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.393+0000 7f5b9f395740 10 client.18060 fetch_fsmap learned FSMap version 51 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.393+0000 7f5b9f395740 1 -- 172.21.15.31:0/3611523128 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x563602c6c650 con 0x563602be2240 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.393+0000 7f5b7dffb640 1 -- 172.21.15.31:0/3611523128 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b90019620 con 0x563602be2240 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.394+0000 7f5b7dffb640 1 -- 172.21.15.31:0/3611523128 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f5b9002a030 con 0x563602be2240 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.394+0000 7f5b9f395740 10 client.18060 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.394+0000 7f5b9f395740 10 client.18060 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.394+0000 7f5b9f395740 20 client.18060 populate_metadata read hostname 'smithi031' 2024-04-25T02:18:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.394+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563602c05d10 0x563602c6eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.394+0000 7f5b9f395740 4 client.18060 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:18:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.394+0000 7f5b9f395740 1 -- 172.21.15.31:0/3611523128 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5b900052c0 con 0x563602c05d10 2024-04-25T02:18:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.394+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563602c05d10 0x563602c6eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.394+0000 7f5b99c2c640 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563602c05d10 0x563602c6eb80 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31868bf6aa844eda server_cookie=21ea26bc9f824e13 in_seq=0 out_seq=0 2024-04-25T02:18:34.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.394+0000 7f5b7dffb640 10 client.18060 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.491+0000 7f5b7dffb640 1 -- 172.21.15.31:0/3611523128 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5b900052c0 con 0x563602c05d10 2024-04-25T02:18:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.491+0000 7f5b7dffb640 10 client.18060 handle_command_reply: tid=0 2024-04-25T02:18:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.491+0000 7f5b9f395740 1 client.18060 shutdown 2024-04-25T02:18:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.491+0000 7f5b9f395740 1 -- 172.21.15.31:0/3611523128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2240 msgr2=0x563602be6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.491+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2240 0x563602be6e80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f5b90009460 tx=0x7f5b90009e60 comp rx=0 tx=0).stop 2024-04-25T02:18:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.491+0000 7f5b9f395740 1 -- 172.21.15.31:0/3611523128 shutdown_connections 2024-04-25T02:18:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.491+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x563602c05d10 0x563602c6eb80 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.492+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563602bed230 0x563602be7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.492+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563602be2b60 0x563602be73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.492+0000 7f5b9f395740 1 --2- 172.21.15.31:0/3611523128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563602be2240 0x563602be6e80 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.492+0000 7f5b9f395740 1 -- 172.21.15.31:0/3611523128 >> 172.21.15.31:0/3611523128 conn(0x563602bde560 msgr2=0x563602bdd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.492+0000 7f5b7dffb640 0 client.18060 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.492+0000 7f5b9f395740 1 -- 172.21.15.31:0/3611523128 shutdown_connections 2024-04-25T02:18:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.492+0000 7f5b9f395740 1 -- 172.21.15.31:0/3611523128 wait complete. 2024-04-25T02:18:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.492+0000 7f5b9f395740 20 client.18060 trim_cache size 0 max 16384 2024-04-25T02:18:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:18:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:18:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:18:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:18:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 -- 172.21.15.31:0/651723039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5b800a2e20 msgr2=0x7f5b800a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5b800a2e20 0x7f5b800a52e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009270 tx=0x7f5b8c009ba0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 -- 172.21.15.31:0/651723039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b941042f0 msgr2=0x7f5b9419a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b941042f0 0x7f5b9419a670 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5b84005ab0 tx=0x7f5b84003b90 comp rx=0 tx=0).stop 2024-04-25T02:18:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 -- 172.21.15.31:0/651723039 shutdown_connections 2024-04-25T02:18:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5b800a2e20 0x7f5b800a52e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b94109940 0x7f5b9419e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b941042f0 0x7f5b9419a670 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 --2- 172.21.15.31:0/651723039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b94103930 0x7f5b9419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 -- 172.21.15.31:0/651723039 >> 172.21.15.31:0/651723039 conn(0x7f5b940ff440 msgr2=0x7f5b940ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 -- 172.21.15.31:0/651723039 shutdown_connections 2024-04-25T02:18:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:34.493+0000 7f5b9c691640 1 -- 172.21.15.31:0/651723039 wait complete. 2024-04-25T02:18:34.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:34 smithi040 ceph-mon[32418]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.4 MiB/s wr, 231 op/s 2024-04-25T02:18:34.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:34 smithi134 ceph-mon[33098]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.4 MiB/s wr, 231 op/s 2024-04-25T02:18:34.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:34 smithi031 ceph-mon[29204]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.4 MiB/s wr, 231 op/s 2024-04-25T02:18:35.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 104597 8.81 MB/sec execute 519 sec latency 77.334 ms 2024-04-25T02:18:36.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 104745 8.81 MB/sec execute 520 sec latency 420.322 ms 2024-04-25T02:18:36.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:36 smithi040 ceph-mon[32418]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.6 MiB/s wr, 349 op/s 2024-04-25T02:18:36.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:36 smithi134 ceph-mon[33098]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.6 MiB/s wr, 349 op/s 2024-04-25T02:18:36.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:36 smithi031 ceph-mon[29204]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.6 MiB/s wr, 349 op/s 2024-04-25T02:18:37.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 104892 8.80 MB/sec execute 521 sec latency 98.160 ms 2024-04-25T02:18:38.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 105062 8.80 MB/sec execute 522 sec latency 65.418 ms 2024-04-25T02:18:38.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:38 smithi040 ceph-mon[32418]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 227 op/s 2024-04-25T02:18:38.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:38 smithi134 ceph-mon[33098]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 227 op/s 2024-04-25T02:18:38.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:38 smithi031 ceph-mon[29204]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 227 op/s 2024-04-25T02:18:39.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 105203 8.80 MB/sec execute 523 sec latency 54.202 ms 2024-04-25T02:18:40.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 105363 8.80 MB/sec execute 524 sec latency 63.769 ms 2024-04-25T02:18:40.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:40 smithi040 ceph-mon[32418]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 227 op/s 2024-04-25T02:18:40.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:40 smithi134 ceph-mon[33098]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 227 op/s 2024-04-25T02:18:40.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:40 smithi031 ceph-mon[29204]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 227 op/s 2024-04-25T02:18:41.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 105530 8.80 MB/sec execute 525 sec latency 57.098 ms 2024-04-25T02:18:42.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 105682 8.79 MB/sec execute 526 sec latency 54.685 ms 2024-04-25T02:18:42.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:42 smithi040 ceph-mon[32418]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 9.3 MiB/s wr, 365 op/s 2024-04-25T02:18:42.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:42 smithi134 ceph-mon[33098]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 9.3 MiB/s wr, 365 op/s 2024-04-25T02:18:42.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:42 smithi031 ceph-mon[29204]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 9.3 MiB/s wr, 365 op/s 2024-04-25T02:18:43.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 105847 8.78 MB/sec execute 527 sec latency 66.852 ms 2024-04-25T02:18:44.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 106003 8.77 MB/sec execute 528 sec latency 54.824 ms 2024-04-25T02:18:44.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:44 smithi040 ceph-mon[32418]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 256 op/s 2024-04-25T02:18:44.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:44 smithi134 ceph-mon[33098]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 256 op/s 2024-04-25T02:18:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:44 smithi031 ceph-mon[29204]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 256 op/s 2024-04-25T02:18:45.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 106173 8.76 MB/sec execute 529 sec latency 71.575 ms 2024-04-25T02:18:46.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 106374 8.77 MB/sec execute 530 sec latency 64.927 ms 2024-04-25T02:18:46.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:46 smithi040 ceph-mon[32418]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.0 MiB/s wr, 372 op/s 2024-04-25T02:18:46.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:46 smithi134 ceph-mon[33098]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.0 MiB/s wr, 372 op/s 2024-04-25T02:18:46.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:46 smithi031 ceph-mon[29204]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.0 MiB/s wr, 372 op/s 2024-04-25T02:18:47.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 106569 8.78 MB/sec execute 531 sec latency 54.765 ms 2024-04-25T02:18:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.623+0000 7f77585b0640 1 -- 172.21.15.31:0/3606024440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7750103330 msgr2=0x7f775010b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.623+0000 7f77585b0640 1 --2- 172.21.15.31:0/3606024440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7750103330 0x7f775010b810 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f7740009900 tx=0x7f774002a480 comp rx=0 tx=0).stop 2024-04-25T02:18:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.623+0000 7f77585b0640 1 -- 172.21.15.31:0/3606024440 shutdown_connections 2024-04-25T02:18:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.623+0000 7f77585b0640 1 --2- 172.21.15.31:0/3606024440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f775010bd50 0x7f775010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.623+0000 7f77585b0640 1 --2- 172.21.15.31:0/3606024440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7750103330 0x7f775010b810 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.623+0000 7f77585b0640 1 --2- 172.21.15.31:0/3606024440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7750102a10 0x7f7750102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.623+0000 7f77585b0640 1 -- 172.21.15.31:0/3606024440 >> 172.21.15.31:0/3606024440 conn(0x7f7750077790 msgr2=0x7f7750077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.623+0000 7f77585b0640 1 -- 172.21.15.31:0/3606024440 shutdown_connections 2024-04-25T02:18:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.623+0000 7f77585b0640 1 -- 172.21.15.31:0/3606024440 wait complete. 2024-04-25T02:18:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f77585b0640 1 Processor -- start 2024-04-25T02:18:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f77585b0640 1 -- start start 2024-04-25T02:18:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f77585b0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7750102a10 0x7f775019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f77585b0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7750103330 0x7f775019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f77585b0640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f775010bd50 0x7f775019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f77585b0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7750101490 con 0x7f7750103330 2024-04-25T02:18:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f77585b0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7750101310 con 0x7f7750102a10 2024-04-25T02:18:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f77585b0640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7750101610 con 0x7f775010bd50 2024-04-25T02:18:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f7755b4b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7750103330 0x7f775019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f775634c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7750102a10 0x7f775019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f7755b4b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7750103330 0x7f775019a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53996/0 (socket says 172.21.15.31:53996) 2024-04-25T02:18:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f7755b4b640 1 -- 172.21.15.31:0/3211098875 learned_addr learned my addr 172.21.15.31:0/3211098875 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.624+0000 7f7756b4d640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f775010bd50 0x7f775019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f7755b4b640 1 -- 172.21.15.31:0/3211098875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f775010bd50 msgr2=0x7f775019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f7755b4b640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f775010bd50 0x7f775019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f7755b4b640 1 -- 172.21.15.31:0/3211098875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7750102a10 msgr2=0x7f775019a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f7755b4b640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7750102a10 0x7f775019a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f7755b4b640 1 -- 172.21.15.31:0/3211098875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f775019f050 con 0x7f7750103330 2024-04-25T02:18:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f775634c640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7750102a10 0x7f775019a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:18:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f7755b4b640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7750103330 0x7f775019a5b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7740005ab0 tx=0x7f7740003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f77477fe640 1 -- 172.21.15.31:0/3211098875 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774003f070 con 0x7f7750103330 2024-04-25T02:18:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f77585b0640 1 -- 172.21.15.31:0/3211098875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f775019f2e0 con 0x7f7750103330 2024-04-25T02:18:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f77477fe640 1 -- 172.21.15.31:0/3211098875 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7740031d60 con 0x7f7750103330 2024-04-25T02:18:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f77477fe640 1 -- 172.21.15.31:0/3211098875 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7740035030 con 0x7f7750103330 2024-04-25T02:18:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.625+0000 7f77585b0640 1 -- 172.21.15.31:0/3211098875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77501a6bd0 con 0x7f7750103330 2024-04-25T02:18:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f775b39d740 1 Processor -- start 2024-04-25T02:18:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f775b39d740 1 -- start start 2024-04-25T02:18:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f77477fe640 1 -- 172.21.15.31:0/3211098875 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7740042070 con 0x7f7750103330 2024-04-25T02:18:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f775b39d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fb8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f775b39d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fb8b60 0x55d540fc2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f775b39d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fc3230 0x55d540fc5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f775b39d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d540f1ccb0 con 0x55d540fb8b60 2024-04-25T02:18:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f775b39d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d540ec55b0 con 0x55d540fb8240 2024-04-25T02:18:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f775b39d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d540f18c70 con 0x55d540fc3230 2024-04-25T02:18:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f7756b4d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fb8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f7756b4d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fb8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48028/0 (socket says 172.21.15.31:48028) 2024-04-25T02:18:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f7756b4d640 1 -- 172.21.15.31:0/2249642041 learned_addr learned my addr 172.21.15.31:0/2249642041 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f775634c640 1 --2- 172.21.15.31:0/2249642041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fb8b60 0x55d540fc2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.629+0000 7f7755b4b640 1 --2- 172.21.15.31:0/2249642041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fc3230 0x55d540fc5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f77477fe640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f773c0a2da0 0x7f773c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f775634c640 1 -- 172.21.15.31:0/2249642041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fc3230 msgr2=0x55d540fc5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f775634c640 1 --2- 172.21.15.31:0/2249642041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fc3230 0x55d540fc5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f775634c640 1 -- 172.21.15.31:0/2249642041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 msgr2=0x55d540fb8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f775634c640 1 --2- 172.21.15.31:0/2249642041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fb8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f775634c640 1 -- 172.21.15.31:0/2249642041 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d540c31210 con 0x55d540fb8b60 2024-04-25T02:18:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f7756b4d640 1 --2- 172.21.15.31:0/2249642041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fb8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:18:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f77477fe640 1 -- 172.21.15.31:0/3211098875 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7f77400efb90 con 0x7f7750103330 2024-04-25T02:18:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f7756b4d640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f773c0a2da0 0x7f773c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f775634c640 1 --2- 172.21.15.31:0/2249642041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fb8b60 0x55d540fc2cf0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f774c0103d0 tx=0x7f774c010890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e7a382d1a154fa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:47.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f7744bfe640 1 -- 172.21.15.31:0/2249642041 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774c004220 con 0x55d540fb8b60 2024-04-25T02:18:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f7744bfe640 1 -- 172.21.15.31:0/2249642041 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f774c0043c0 con 0x55d540fb8b60 2024-04-25T02:18:47.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.630+0000 7f7744bfe640 1 -- 172.21.15.31:0/2249642041 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774c0119f0 con 0x55d540fb8b60 2024-04-25T02:18:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.631+0000 7f775b39d740 1 -- 172.21.15.31:0/2249642041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fb8b60 msgr2=0x55d540fc2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.631+0000 7f775b39d740 1 --2- 172.21.15.31:0/2249642041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fb8b60 0x55d540fc2cf0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f774c0103d0 tx=0x7f774c010890 comp rx=0 tx=0).stop 2024-04-25T02:18:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.631+0000 7f7756b4d640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f773c0a2da0 0x7f773c0a5260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f77480107d0 tx=0x7f7748004a40 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.631+0000 7f775b39d740 1 -- 172.21.15.31:0/2249642041 shutdown_connections 2024-04-25T02:18:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.631+0000 7f775b39d740 1 --2- 172.21.15.31:0/2249642041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fc3230 0x55d540fc5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.631+0000 7f775b39d740 1 --2- 172.21.15.31:0/2249642041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fb8b60 0x55d540fc2cf0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.631+0000 7f775b39d740 1 --2- 172.21.15.31:0/2249642041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fb8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.631+0000 7f775b39d740 1 -- 172.21.15.31:0/2249642041 >> 172.21.15.31:0/2249642041 conn(0x55d540fb4560 msgr2=0x55d540fb4940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.632+0000 7f775b39d740 1 -- 172.21.15.31:0/2249642041 shutdown_connections 2024-04-25T02:18:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.632+0000 7f775b39d740 1 -- 172.21.15.31:0/2249642041 wait complete. 2024-04-25T02:18:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.632+0000 7f775b39d740 1 -- start start 2024-04-25T02:18:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.632+0000 7f775b39d740 1 Processor -- start 2024-04-25T02:18:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.632+0000 7f775b39d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fc0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.632+0000 7f775b39d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 0x55d540fc0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.632+0000 7f775b39d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fc3230 0x55d540fc0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.632+0000 7f775b39d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d540fc6020 con 0x55d540fc3230 2024-04-25T02:18:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.632+0000 7f775b39d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d540ed8b50 con 0x55d540fb8240 2024-04-25T02:18:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.632+0000 7f775b39d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d540fc61a0 con 0x55d540fb8b60 2024-04-25T02:18:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f775b39d740 10 client.0 _ll_register_callbacks cb 0x55d540f13ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:18:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7755b4b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 0x55d540fc0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7756b4d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fc3230 0x55d540fc0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7755b4b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 0x55d540fc0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:33312/0 (socket says 172.21.15.31:33312) 2024-04-25T02:18:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7755b4b640 1 -- 172.21.15.31:0/2758033551 learned_addr learned my addr 172.21.15.31:0/2758033551 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f775634c640 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fc0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7755b4b640 1 -- 172.21.15.31:0/2758033551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 msgr2=0x55d540fc0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7755b4b640 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fc0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7755b4b640 1 -- 172.21.15.31:0/2758033551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fc3230 msgr2=0x55d540fc0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7755b4b640 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fc3230 0x55d540fc0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7755b4b640 1 -- 172.21.15.31:0/2758033551 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d540cfd620 con 0x55d540fb8b60 2024-04-25T02:18:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7756b4d640 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fc3230 0x55d540fc0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:18:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7755b4b640 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 0x55d540fc0660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7740041000 tx=0x7f774002a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8afd3d7d43ad173e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7739ffb640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:18:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7739ffb640 1 -- 172.21.15.31:0/2758033551 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7740049050 con 0x55d540fb8b60 2024-04-25T02:18:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f7739ffb640 1 -- 172.21.15.31:0/2758033551 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7f77400c01a0 con 0x55d540fb8b60 2024-04-25T02:18:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.633+0000 7f775b39d740 1 -- 172.21.15.31:0/2758033551 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f774c0043c0 con 0x55d540fb8b60 2024-04-25T02:18:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f7739ffb640 1 -- 172.21.15.31:0/2758033551 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f77400bbcc0 con 0x55d540fb8b60 2024-04-25T02:18:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f775b39d740 10 client.27412 fetch_fsmap learned FSMap version 51 2024-04-25T02:18:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f775b39d740 1 -- 172.21.15.31:0/2758033551 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55d540fc6080 con 0x55d540fb8b60 2024-04-25T02:18:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f7739ffb640 1 -- 172.21.15.31:0/2758033551 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f774c0043c0 con 0x55d540fb8b60 2024-04-25T02:18:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f7739ffb640 1 -- 172.21.15.31:0/2758033551 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f77400f8040 con 0x55d540fb8b60 2024-04-25T02:18:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f775b39d740 10 client.27412 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:18:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f775b39d740 10 client.27412 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:18:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f775b39d740 20 client.27412 populate_metadata read hostname 'smithi031' 2024-04-25T02:18:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f775b39d740 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d540fd3650 0x55d540fd5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f775b39d740 4 client.27412 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:18:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f775b39d740 1 -- 172.21.15.31:0/2758033551 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d540f2a1f0 con 0x55d540fd3650 2024-04-25T02:18:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.634+0000 7f775634c640 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d540fd3650 0x55d540fd5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.635+0000 7f7739ffb640 10 client.27412 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.635+0000 7f775634c640 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d540fd3650 0x55d540fd5a90 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=902647ae41c16359 server_cookie=ecf0338a67d28ab6 in_seq=0 out_seq=0 2024-04-25T02:18:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.638+0000 7f7739ffb640 1 -- 172.21.15.31:0/2758033551 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d540f2a1f0 con 0x55d540fd3650 2024-04-25T02:18:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.638+0000 7f7739ffb640 10 client.27412 handle_command_reply: tid=0 2024-04-25T02:18:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.638+0000 7f775b39d740 1 client.27412 shutdown 2024-04-25T02:18:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.638+0000 7f775b39d740 1 -- 172.21.15.31:0/2758033551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 msgr2=0x55d540fc0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.638+0000 7f775b39d740 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 0x55d540fc0660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7740041000 tx=0x7f774002a480 comp rx=0 tx=0).stop 2024-04-25T02:18:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.639+0000 7f775b39d740 1 -- 172.21.15.31:0/2758033551 shutdown_connections 2024-04-25T02:18:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.639+0000 7f775b39d740 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d540fd3650 0x55d540fd5a90 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.639+0000 7f7739ffb640 0 client.27412 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.639+0000 7f775b39d740 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fc3230 0x55d540fc0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.639+0000 7f775b39d740 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 0x55d540fc0660 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.639+0000 7f775b39d740 1 --2- 172.21.15.31:0/2758033551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fc0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.639+0000 7f775b39d740 1 -- 172.21.15.31:0/2758033551 >> 172.21.15.31:0/2758033551 conn(0x55d540fb4560 msgr2=0x55d540fba260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.639+0000 7f775b39d740 1 -- 172.21.15.31:0/2758033551 shutdown_connections 2024-04-25T02:18:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.639+0000 7f775b39d740 1 -- 172.21.15.31:0/2758033551 wait complete. 2024-04-25T02:18:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.639+0000 7f775b39d740 20 client.27412 trim_cache size 0 max 16384 2024-04-25T02:18:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.668+0000 7f775b39d740 1 Processor -- start 2024-04-25T02:18:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.668+0000 7f775b39d740 1 -- start start 2024-04-25T02:18:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.668+0000 7f775b39d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fd5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f775b39d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 0x55d540fceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f775b39d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fc3230 0x55d540fba9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f775b39d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f77501af450 con 0x55d540fc3230 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f775b39d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f77501b0f90 con 0x55d540fb8240 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f775b39d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f77501afb20 con 0x55d540fb8b60 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f7755b4b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 0x55d540fceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f775634c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fd5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f7756b4d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fc3230 0x55d540fba9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f775634c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fd5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:48046/0 (socket says 172.21.15.31:48046) 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f775634c640 1 -- 172.21.15.31:0/3996043465 learned_addr learned my addr 172.21.15.31:0/3996043465 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f7755b4b640 1 -- 172.21.15.31:0/3996043465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 msgr2=0x55d540fd5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f7755b4b640 1 --2- 172.21.15.31:0/3996043465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fd5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f7755b4b640 1 -- 172.21.15.31:0/3996043465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fc3230 msgr2=0x55d540fba9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f7755b4b640 1 --2- 172.21.15.31:0/3996043465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fc3230 0x55d540fba9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.669+0000 7f7755b4b640 1 -- 172.21.15.31:0/3996043465 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d540fb8990 con 0x55d540fb8b60 2024-04-25T02:18:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.670+0000 7f7755b4b640 1 --2- 172.21.15.31:0/3996043465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 0x55d540fceee0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7740005ec0 tx=0x7f77400ba690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26dd09f657fe6e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.670+0000 7f773bfff640 1 -- 172.21.15.31:0/3996043465 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7740049070 con 0x55d540fb8b60 2024-04-25T02:18:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.670+0000 7f773bfff640 1 -- 172.21.15.31:0/3996043465 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77400f3020 con 0x55d540fb8b60 2024-04-25T02:18:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.670+0000 7f773bfff640 1 -- 172.21.15.31:0/3996043465 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77400bbae0 con 0x55d540fb8b60 2024-04-25T02:18:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.671+0000 7f775b39d740 1 -- 172.21.15.31:0/3996043465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 msgr2=0x55d540fceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.671+0000 7f775b39d740 1 --2- 172.21.15.31:0/3996043465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 0x55d540fceee0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7740005ec0 tx=0x7f77400ba690 comp rx=0 tx=0).stop 2024-04-25T02:18:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.671+0000 7f775b39d740 1 -- 172.21.15.31:0/3996043465 shutdown_connections 2024-04-25T02:18:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.671+0000 7f775b39d740 1 --2- 172.21.15.31:0/3996043465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fc3230 0x55d540fba9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.671+0000 7f775b39d740 1 --2- 172.21.15.31:0/3996043465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8b60 0x55d540fceee0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.671+0000 7f775b39d740 1 --2- 172.21.15.31:0/3996043465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fb8240 0x55d540fd5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.671+0000 7f775b39d740 1 -- 172.21.15.31:0/3996043465 >> 172.21.15.31:0/3996043465 conn(0x55d540fb4560 msgr2=0x55d540fb2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:47.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.671+0000 7f775b39d740 1 -- 172.21.15.31:0/3996043465 shutdown_connections 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.671+0000 7f775b39d740 1 -- 172.21.15.31:0/3996043465 wait complete. 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.671+0000 7f775b39d740 1 -- start start 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.671+0000 7f775b39d740 1 Processor -- start 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f775b39d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8240 0x55d540fcb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f775b39d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fb8b60 0x55d540fbcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f775b39d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fc3230 0x55d540fbd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f775b39d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f77501b08c0 con 0x55d540fb8b60 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f775b39d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f77501b01f0 con 0x55d540fc3230 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f775b39d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f77501ad450 con 0x55d540fb8240 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f775b39d740 10 client.0 _ll_register_callbacks cb 0x7f77400bc040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f775634c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8240 0x55d540fcb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f775634c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8240 0x55d540fcb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:33330/0 (socket says 172.21.15.31:33330) 2024-04-25T02:18:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f7755b4b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fb8b60 0x55d540fbcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f775634c640 1 -- 172.21.15.31:0/3360862258 learned_addr learned my addr 172.21.15.31:0/3360862258 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f7756b4d640 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fc3230 0x55d540fbd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f7756b4d640 1 -- 172.21.15.31:0/3360862258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8240 msgr2=0x55d540fcb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f7756b4d640 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8240 0x55d540fcb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f7756b4d640 1 -- 172.21.15.31:0/3360862258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fb8b60 msgr2=0x55d540fbcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f7756b4d640 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fb8b60 0x55d540fbcb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f7756b4d640 1 -- 172.21.15.31:0/3360862258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d540fbd870 con 0x55d540fc3230 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f775634c640 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8240 0x55d540fcb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f7756b4d640 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fc3230 0x55d540fbd040 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f774800f790 tx=0x7f7748004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa128b9f26d1749a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.672+0000 7f7739ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f7739ffb640 1 -- 172.21.15.31:0/3360862258 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774802f060 con 0x55d540fc3230 2024-04-25T02:18:47.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f7739ffb640 1 -- 172.21.15.31:0/3360862258 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7f774801d9b0 con 0x55d540fc3230 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f775b39d740 1 -- 172.21.15.31:0/3360862258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77400f3020 con 0x55d540fc3230 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f7739ffb640 1 -- 172.21.15.31:0/3360862258 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f774801dbb0 con 0x55d540fc3230 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f775b39d740 10 client.27644 fetch_fsmap learned FSMap version 51 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f775b39d740 1 -- 172.21.15.31:0/3360862258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55d541039c20 con 0x55d540fc3230 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f7739ffb640 1 -- 172.21.15.31:0/3360862258 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77400f3020 con 0x55d540fc3230 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f7739ffb640 1 -- 172.21.15.31:0/3360862258 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f774801c420 con 0x55d540fc3230 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f775b39d740 10 client.27644 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f775b39d740 10 client.27644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f775b39d740 20 client.27644 populate_metadata read hostname 'smithi031' 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f775b39d740 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d540fd3650 0x55d54103c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f775b39d740 4 client.27644 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:18:47.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.673+0000 7f775b39d740 1 -- 172.21.15.31:0/3360862258 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d540f24540 con 0x55d540fd3650 2024-04-25T02:18:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.674+0000 7f775634c640 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d540fd3650 0x55d54103c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:18:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.674+0000 7f7739ffb640 10 client.27644 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:47.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.674+0000 7f775634c640 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d540fd3650 0x55d54103c410 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd1007a05a8d2b9c server_cookie=a3a1c224131d3f71 in_seq=0 out_seq=0 2024-04-25T02:18:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.785+0000 7f7739ffb640 1 -- 172.21.15.31:0/3360862258 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d540f24540 con 0x55d540fd3650 2024-04-25T02:18:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.785+0000 7f7739ffb640 10 client.27644 handle_command_reply: tid=0 2024-04-25T02:18:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.785+0000 7f775b39d740 1 client.27644 shutdown 2024-04-25T02:18:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f775b39d740 1 -- 172.21.15.31:0/3360862258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fc3230 msgr2=0x55d540fbd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f775b39d740 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fc3230 0x55d540fbd040 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f774800f790 tx=0x7f7748004ab0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f775b39d740 1 -- 172.21.15.31:0/3360862258 shutdown_connections 2024-04-25T02:18:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f775b39d740 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55d540fd3650 0x55d54103c410 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f775b39d740 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d540fc3230 0x55d540fbd040 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f775b39d740 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d540fb8b60 0x55d540fbcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f7739ffb640 0 client.27644 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:18:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f775b39d740 1 --2- 172.21.15.31:0/3360862258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d540fb8240 0x55d540fcb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f775b39d740 1 -- 172.21.15.31:0/3360862258 >> 172.21.15.31:0/3360862258 conn(0x55d540fb4560 msgr2=0x55d540fb35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f775b39d740 1 -- 172.21.15.31:0/3360862258 shutdown_connections 2024-04-25T02:18:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f775b39d740 1 -- 172.21.15.31:0/3360862258 wait complete. 2024-04-25T02:18:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.786+0000 7f775b39d740 20 client.27644 trim_cache size 0 max 16384 2024-04-25T02:18:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:18:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:18:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:18:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:18:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 -- 172.21.15.31:0/3211098875 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f773c0a2da0 msgr2=0x7f773c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f773c0a2da0 0x7f773c0a5260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f77480107d0 tx=0x7f7748004a40 comp rx=0 tx=0).stop 2024-04-25T02:18:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 -- 172.21.15.31:0/3211098875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7750103330 msgr2=0x7f775019a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:18:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7750103330 0x7f775019a5b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7740005ab0 tx=0x7f7740003c80 comp rx=0 tx=0).stop 2024-04-25T02:18:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 -- 172.21.15.31:0/3211098875 shutdown_connections 2024-04-25T02:18:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f773c0a2da0 0x7f773c0a5260 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f775010bd50 0x7f775019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7750103330 0x7f775019a5b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 --2- 172.21.15.31:0/3211098875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7750102a10 0x7f775019a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 -- 172.21.15.31:0/3211098875 >> 172.21.15.31:0/3211098875 conn(0x7f7750077790 msgr2=0x7f775010dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 -- 172.21.15.31:0/3211098875 shutdown_connections 2024-04-25T02:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:18:47.787+0000 7f77585b0640 1 -- 172.21.15.31:0/3211098875 wait complete. 2024-04-25T02:18:48.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 106746 8.80 MB/sec execute 532 sec latency 112.392 ms 2024-04-25T02:18:48.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:48 smithi040 ceph-mon[32418]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 255 op/s 2024-04-25T02:18:48.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:48 smithi134 ceph-mon[33098]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 255 op/s 2024-04-25T02:18:48.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:48 smithi031 ceph-mon[29204]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 255 op/s 2024-04-25T02:18:49.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 106933 8.80 MB/sec execute 533 sec latency 74.036 ms 2024-04-25T02:18:50.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 107156 8.82 MB/sec execute 534 sec latency 82.356 ms 2024-04-25T02:18:50.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:50 smithi040 ceph-mon[32418]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 255 op/s 2024-04-25T02:18:50.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:50 smithi134 ceph-mon[33098]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 255 op/s 2024-04-25T02:18:50.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:50 smithi031 ceph-mon[29204]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 255 op/s 2024-04-25T02:18:51.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 107294 8.81 MB/sec execute 535 sec latency 82.118 ms 2024-04-25T02:18:52.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 107406 8.80 MB/sec execute 536 sec latency 101.175 ms 2024-04-25T02:18:52.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:52 smithi040 ceph-mon[32418]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.1 MiB/s wr, 376 op/s 2024-04-25T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:52 smithi134 ceph-mon[33098]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.1 MiB/s wr, 376 op/s 2024-04-25T02:18:52.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:52 smithi031 ceph-mon[29204]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.1 MiB/s wr, 376 op/s 2024-04-25T02:18:53.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 107561 8.81 MB/sec execute 537 sec latency 115.396 ms 2024-04-25T02:18:54.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 107718 8.81 MB/sec execute 538 sec latency 72.147 ms 2024-04-25T02:18:54.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:54 smithi040 ceph-mon[32418]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.3 MiB/s wr, 238 op/s 2024-04-25T02:18:54.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:54 smithi134 ceph-mon[33098]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.3 MiB/s wr, 238 op/s 2024-04-25T02:18:54.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:54 smithi031 ceph-mon[29204]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.3 MiB/s wr, 238 op/s 2024-04-25T02:18:55.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 107845 8.80 MB/sec execute 539 sec latency 60.131 ms 2024-04-25T02:18:56.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 107976 8.80 MB/sec execute 540 sec latency 139.205 ms 2024-04-25T02:18:56.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:56 smithi040 ceph-mon[32418]: pgmap v771: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.0 MiB/s wr, 329 op/s 2024-04-25T02:18:56.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:56 smithi134 ceph-mon[33098]: pgmap v771: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.0 MiB/s wr, 329 op/s 2024-04-25T02:18:56.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:56 smithi031 ceph-mon[29204]: pgmap v771: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.0 MiB/s wr, 329 op/s 2024-04-25T02:18:57.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 108126 8.80 MB/sec execute 541 sec latency 114.431 ms 2024-04-25T02:18:58.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 108298 8.79 MB/sec execute 542 sec latency 119.548 ms 2024-04-25T02:18:58.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:18:58 smithi040 ceph-mon[32418]: pgmap v772: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 212 op/s 2024-04-25T02:18:58.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:18:58 smithi134 ceph-mon[33098]: pgmap v772: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 212 op/s 2024-04-25T02:18:58.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:18:58 smithi031 ceph-mon[29204]: pgmap v772: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 212 op/s 2024-04-25T02:18:59.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 108480 8.80 MB/sec execute 543 sec latency 126.042 ms 2024-04-25T02:19:00.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 108614 8.79 MB/sec execute 544 sec latency 70.447 ms 2024-04-25T02:19:00.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:00 smithi040 ceph-mon[32418]: pgmap v773: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 212 op/s 2024-04-25T02:19:00.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:00 smithi134 ceph-mon[33098]: pgmap v773: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 212 op/s 2024-04-25T02:19:00.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:00 smithi031 ceph-mon[29204]: pgmap v773: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 212 op/s 2024-04-25T02:19:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.913+0000 7fd9a204d640 1 -- 172.21.15.31:0/1339335365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd99c102110 msgr2=0x7fd99c10e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.913+0000 7fd9a204d640 1 --2- 172.21.15.31:0/1339335365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd99c102110 0x7fd99c10e620 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd994009cd0 tx=0x7fd99402a8e0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.913+0000 7fd9a204d640 1 -- 172.21.15.31:0/1339335365 shutdown_connections 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.913+0000 7fd9a204d640 1 --2- 172.21.15.31:0/1339335365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd99c102110 0x7fd99c10e620 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.913+0000 7fd9a204d640 1 --2- 172.21.15.31:0/1339335365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd99c101750 0x7fd99c101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.913+0000 7fd9a204d640 1 --2- 172.21.15.31:0/1339335365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c107760 0x7fd99c107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.913+0000 7fd9a204d640 1 -- 172.21.15.31:0/1339335365 >> 172.21.15.31:0/1339335365 conn(0x7fd99c0fd240 msgr2=0x7fd99c0ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.913+0000 7fd9a204d640 1 -- 172.21.15.31:0/1339335365 shutdown_connections 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.913+0000 7fd9a204d640 1 -- 172.21.15.31:0/1339335365 wait complete. 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd9a204d640 1 Processor -- start 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd9a204d640 1 -- start start 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd9a204d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c101750 0x7fd99c073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd9a204d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd99c102110 0x7fd99c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd9a204d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd99c107760 0x7fd99c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd9a204d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1119e0 con 0x7fd99c107760 2024-04-25T02:19:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd9a204d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c111860 con 0x7fd99c101750 2024-04-25T02:19:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd9a204d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c111b60 con 0x7fd99c102110 2024-04-25T02:19:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd99c107760 0x7fd99c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd99c107760 0x7fd99c0744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53700/0 (socket says 172.21.15.31:53700) 2024-04-25T02:19:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd99bfff640 1 -- 172.21.15.31:0/949132388 learned_addr learned my addr 172.21.15.31:0/949132388 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd99affd640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd99c102110 0x7fd99c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.914+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c101750 0x7fd99c073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.915+0000 7fd99bfff640 1 -- 172.21.15.31:0/949132388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd99c102110 msgr2=0x7fd99c073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.915+0000 7fd99bfff640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd99c102110 0x7fd99c073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.915+0000 7fd99bfff640 1 -- 172.21.15.31:0/949132388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c101750 msgr2=0x7fd99c073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.915+0000 7fd99bfff640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c101750 0x7fd99c073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.915+0000 7fd99bfff640 1 -- 172.21.15.31:0/949132388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99c1a6340 con 0x7fd99c107760 2024-04-25T02:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.915+0000 7fd99bfff640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd99c107760 0x7fd99c0744c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fd994033040 tx=0x7fd994009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.916+0000 7fd998ff9640 1 -- 172.21.15.31:0/949132388 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994004040 con 0x7fd99c107760 2024-04-25T02:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.916+0000 7fd998ff9640 1 -- 172.21.15.31:0/949132388 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd994004690 con 0x7fd99c107760 2024-04-25T02:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.916+0000 7fd998ff9640 1 -- 172.21.15.31:0/949132388 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994030da0 con 0x7fd99c107760 2024-04-25T02:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.916+0000 7fd9a204d640 1 -- 172.21.15.31:0/949132388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99c1a65d0 con 0x7fd99c107760 2024-04-25T02:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.917+0000 7fd9a204d640 1 -- 172.21.15.31:0/949132388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99c1a6b20 con 0x7fd99c107760 2024-04-25T02:19:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.920+0000 7fd998ff9640 1 -- 172.21.15.31:0/949132388 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd9940041e0 con 0x7fd99c107760 2024-04-25T02:19:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.920+0000 7fd9a4e36740 1 Processor -- start 2024-04-25T02:19:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.920+0000 7fd9a4e36740 1 -- start start 2024-04-25T02:19:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb2a50 0x56352ecb2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb3370 0x56352ecbd500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecbfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56352ec0ecb0 con 0x56352ecb3370 2024-04-25T02:19:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56352ebb75b0 con 0x56352ecb2a50 2024-04-25T02:19:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56352ec0ac70 con 0x56352ecbda40 2024-04-25T02:19:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb3370 0x56352ecbd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb3370 0x56352ecbd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53712/0 (socket says 172.21.15.31:53712) 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd99b7fe640 1 -- 172.21.15.31:0/3593040276 learned_addr learned my addr 172.21.15.31:0/3593040276 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd99affd640 1 --2- 172.21.15.31:0/3593040276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb2a50 0x56352ecb2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd99affd640 1 --2- 172.21.15.31:0/3593040276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecbfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd99b7fe640 1 -- 172.21.15.31:0/3593040276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 msgr2=0x56352ecbfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/3593040276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecbfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd99b7fe640 1 -- 172.21.15.31:0/3593040276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb2a50 msgr2=0x56352ecb2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/3593040276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb2a50 0x56352ecb2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.921+0000 7fd99b7fe640 1 -- 172.21.15.31:0/3593040276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56352e923210 con 0x56352ecb3370 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.922+0000 7fd998ff9640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd9840a2f00 0x7fd9840a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.922+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/3593040276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb3370 0x56352ecbd500 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009c70 tx=0x7fd98c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e014f6878042032 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.922+0000 7fd99affd640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd9840a2f00 0x7fd9840a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.922+0000 7fd998ff9640 1 -- 172.21.15.31:0/949132388 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7fd99402e030 con 0x7fd99c107760 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.922+0000 7fd981ffb640 1 -- 172.21.15.31:0/3593040276 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c004230 con 0x56352ecb3370 2024-04-25T02:19:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.922+0000 7fd981ffb640 1 -- 172.21.15.31:0/3593040276 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd98c0043d0 con 0x56352ecb3370 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.922+0000 7fd981ffb640 1 -- 172.21.15.31:0/3593040276 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c0197a0 con 0x56352ecb3370 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.923+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3593040276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb3370 msgr2=0x56352ecbd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.923+0000 7fd99affd640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd9840a2f00 0x7fd9840a53c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x56352ecaff00 tx=0x7fd990015040 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.923+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3593040276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb3370 0x56352ecbd500 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009c70 tx=0x7fd98c011810 comp rx=0 tx=0).stop 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.923+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3593040276 shutdown_connections 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.923+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3593040276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecbfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.923+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3593040276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb3370 0x56352ecbd500 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.923+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3593040276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb2a50 0x56352ecb2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.923+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3593040276 >> 172.21.15.31:0/3593040276 conn(0x56352ecaed70 msgr2=0x56352ecaf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.923+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3593040276 shutdown_connections 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.923+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3593040276 wait complete. 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.924+0000 7fd9a4e36740 1 -- start start 2024-04-25T02:19:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.924+0000 7fd9a4e36740 1 Processor -- start 2024-04-25T02:19:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.924+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 0x56352ecba9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.924+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecbaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.924+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecbb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.924+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56352ecc0830 con 0x56352ecb2a50 2024-04-25T02:19:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.924+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56352ebcab50 con 0x56352ecb3370 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.924+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56352ecc09b0 con 0x56352ecbda40 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.924+0000 7fd9a4e36740 10 client.0 _ll_register_callbacks cb 0x56352ec05ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 0x56352ecba9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecbb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecbb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36792/0 (socket says 172.21.15.31:36792) 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99bfff640 1 -- 172.21.15.31:0/825256118 learned_addr learned my addr 172.21.15.31:0/825256118 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99affd640 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecbaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99b7fe640 1 -- 172.21.15.31:0/825256118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 msgr2=0x56352ecbb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecbb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99b7fe640 1 -- 172.21.15.31:0/825256118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 msgr2=0x56352ecbaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecbaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99b7fe640 1 -- 172.21.15.31:0/825256118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56352e9ef620 con 0x56352ecb2a50 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99bfff640 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecbb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 0x56352ecba9e0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009c70 tx=0x7fd98c005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf31a17729fba45a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.925+0000 7fd97affd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.926+0000 7fd97affd640 1 -- 172.21.15.31:0/825256118 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c027070 con 0x56352ecb2a50 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.926+0000 7fd97affd640 1 -- 172.21.15.31:0/825256118 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7fd98c003800 con 0x56352ecb2a50 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.926+0000 7fd9a4e36740 1 -- 172.21.15.31:0/825256118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd98c0043d0 con 0x56352ecb2a50 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.926+0000 7fd97affd640 1 -- 172.21.15.31:0/825256118 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fd98c033050 con 0x56352ecb2a50 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.926+0000 7fd9a4e36740 10 client.18090 fetch_fsmap learned FSMap version 51 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.926+0000 7fd9a4e36740 1 -- 172.21.15.31:0/825256118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x56352eccca80 con 0x56352ecb2a50 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.927+0000 7fd97affd640 1 -- 172.21.15.31:0/825256118 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd98c0043d0 con 0x56352ecb2a50 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.927+0000 7fd97affd640 1 -- 172.21.15.31:0/825256118 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7fd98c018070 con 0x56352ecb2a50 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.927+0000 7fd9a4e36740 10 client.18090 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.927+0000 7fd9a4e36740 10 client.18090 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.927+0000 7fd9a4e36740 20 client.18090 populate_metadata read hostname 'smithi031' 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.927+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56352eccde90 0x56352ecd0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.927+0000 7fd9a4e36740 4 client.18090 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:19:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.927+0000 7fd9a4e36740 1 -- 172.21.15.31:0/825256118 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56352ec1c1f0 con 0x56352eccde90 2024-04-25T02:19:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.927+0000 7fd99affd640 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56352eccde90 0x56352ecd0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.928+0000 7fd97affd640 10 client.18090 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.928+0000 7fd99affd640 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56352eccde90 0x56352ecd0280 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a0d02078fcec5d1 server_cookie=68cc93bdfabea2aa in_seq=0 out_seq=0 2024-04-25T02:19:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.930+0000 7fd97affd640 1 -- 172.21.15.31:0/825256118 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56352ec1c1f0 con 0x56352eccde90 2024-04-25T02:19:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.930+0000 7fd97affd640 10 client.18090 handle_command_reply: tid=0 2024-04-25T02:19:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.930+0000 7fd9a4e36740 1 client.18090 shutdown 2024-04-25T02:19:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd9a4e36740 1 -- 172.21.15.31:0/825256118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 msgr2=0x56352ecba9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 0x56352ecba9e0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009c70 tx=0x7fd98c005760 comp rx=0 tx=0).stop 2024-04-25T02:19:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd9a4e36740 1 -- 172.21.15.31:0/825256118 shutdown_connections 2024-04-25T02:19:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56352eccde90 0x56352ecd0280 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecbb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecbaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/825256118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 0x56352ecba9e0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd97affd640 0 client.18090 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd9a4e36740 1 -- 172.21.15.31:0/825256118 >> 172.21.15.31:0/825256118 conn(0x56352ecaed70 msgr2=0x56352eca55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd9a4e36740 1 -- 172.21.15.31:0/825256118 shutdown_connections 2024-04-25T02:19:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd9a4e36740 1 -- 172.21.15.31:0/825256118 wait complete. 2024-04-25T02:19:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.931+0000 7fd9a4e36740 20 client.18090 trim_cache size 0 max 16384 2024-04-25T02:19:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.960+0000 7fd9a4e36740 1 Processor -- start 2024-04-25T02:19:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.960+0000 7fd9a4e36740 1 -- start start 2024-04-25T02:19:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecb2a50 0x56352ecbd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecc96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecbda40 0x56352ecd5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1aafe0 con 0x56352ecbda40 2024-04-25T02:19:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1acb20 con 0x56352ecb3370 2024-04-25T02:19:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1ab6b0 con 0x56352ecb2a50 2024-04-25T02:19:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecbda40 0x56352ecd5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecb2a50 0x56352ecbd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd99affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecc96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecb2a50 0x56352ecbd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36798/0 (socket says 172.21.15.31:36798) 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd99b7fe640 1 -- 172.21.15.31:0/3009012056 learned_addr learned my addr 172.21.15.31:0/3009012056 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd99b7fe640 1 -- 172.21.15.31:0/3009012056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 msgr2=0x56352ecc96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/3009012056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecc96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd99b7fe640 1 -- 172.21.15.31:0/3009012056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecbda40 msgr2=0x56352ecd5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/3009012056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecbda40 0x56352ecd5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd99b7fe640 1 -- 172.21.15.31:0/3009012056 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56352ecc0160 con 0x56352ecb2a50 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.961+0000 7fd99affd640 1 --2- 172.21.15.31:0/3009012056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecc96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.962+0000 7fd99bfff640 1 --2- 172.21.15.31:0/3009012056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecbda40 0x56352ecd5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.962+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/3009012056 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecb2a50 0x56352ecbd110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009b70 tx=0x7fd98c011d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d70b6be78188147d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.962+0000 7fd9817fa640 1 -- 172.21.15.31:0/3009012056 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c027070 con 0x56352ecb2a50 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.962+0000 7fd9817fa640 1 -- 172.21.15.31:0/3009012056 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd98c004330 con 0x56352ecb2a50 2024-04-25T02:19:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.962+0000 7fd9817fa640 1 -- 172.21.15.31:0/3009012056 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c003800 con 0x56352ecb2a50 2024-04-25T02:19:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.963+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3009012056 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecb2a50 msgr2=0x56352ecbd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.963+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3009012056 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecb2a50 0x56352ecbd110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009b70 tx=0x7fd98c011d10 comp rx=0 tx=0).stop 2024-04-25T02:19:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.963+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3009012056 shutdown_connections 2024-04-25T02:19:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.963+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3009012056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecbda40 0x56352ecd5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.963+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3009012056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecc96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.963+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3009012056 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecb2a50 0x56352ecbd110 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.963+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3009012056 >> 172.21.15.31:0/3009012056 conn(0x56352ecaed70 msgr2=0x56352ecaf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.963+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3009012056 shutdown_connections 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.963+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3009012056 wait complete. 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd9a4e36740 1 -- start start 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd9a4e36740 1 Processor -- start 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 0x56352ecc5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecc6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd9a4e36740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecb7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1ac450 con 0x56352ecb2a50 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1abd80 con 0x56352ecb3370 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd9a4e36740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1a9000 con 0x56352ecbda40 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd9a4e36740 10 client.0 _ll_register_callbacks cb 0x7fd9900096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 0x56352ecc5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd99affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecc6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 0x56352ecc5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53736/0 (socket says 172.21.15.31:53736) 2024-04-25T02:19:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd99b7fe640 1 -- 172.21.15.31:0/3856458436 learned_addr learned my addr 172.21.15.31:0/3856458436 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.964+0000 7fd99bfff640 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecb7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.965+0000 7fd99affd640 1 -- 172.21.15.31:0/3856458436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 msgr2=0x56352ecb7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.965+0000 7fd99affd640 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecb7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.965+0000 7fd99affd640 1 -- 172.21.15.31:0/3856458436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 msgr2=0x56352ecc5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.965+0000 7fd99affd640 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 0x56352ecc5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.965+0000 7fd99affd640 1 -- 172.21.15.31:0/3856458436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56352ecb7a40 con 0x56352ecb3370 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.965+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 0x56352ecc5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.965+0000 7fd99affd640 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecc6140 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x56352ecd0a50 tx=0x7fd990005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44ff92e9ba0f415e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.965+0000 7fd97affd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.965+0000 7fd97affd640 1 -- 172.21.15.31:0/3856458436 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99002a430 con 0x56352ecb3370 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.966+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3856458436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd98c004330 con 0x56352ecb3370 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.966+0000 7fd97affd640 1 -- 172.21.15.31:0/3856458436 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8484+0+0 (secure 0 0 0) 0x7fd99002a5d0 con 0x56352ecb3370 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.966+0000 7fd97affd640 1 -- 172.21.15.31:0/3856458436 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fd99002ae20 con 0x56352ecb3370 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.966+0000 7fd9a4e36740 10 client.27662 fetch_fsmap learned FSMap version 51 2024-04-25T02:19:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.966+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3856458436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x56352ed33ff0 con 0x56352ecb3370 2024-04-25T02:19:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.966+0000 7fd97affd640 1 -- 172.21.15.31:0/3856458436 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd98c004330 con 0x56352ecb3370 2024-04-25T02:19:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.967+0000 7fd97affd640 1 -- 172.21.15.31:0/3856458436 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7fd990039020 con 0x56352ecb3370 2024-04-25T02:19:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.967+0000 7fd9a4e36740 10 client.27662 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:19:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.967+0000 7fd9a4e36740 10 client.27662 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:19:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.967+0000 7fd9a4e36740 20 client.27662 populate_metadata read hostname 'smithi031' 2024-04-25T02:19:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.967+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56352ed34720 0x56352ed36b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.967+0000 7fd9a4e36740 4 client.27662 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:19:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.967+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3856458436 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56352ec16540 con 0x56352ed34720 2024-04-25T02:19:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.967+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56352ed34720 0x56352ed36b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.968+0000 7fd97affd640 10 client.27662 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:00.968+0000 7fd99b7fe640 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56352ed34720 0x56352ed36b40 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f74b97832a65631 server_cookie=a679e4f7ff69cfa4 in_seq=0 out_seq=0 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd97affd640 1 -- 172.21.15.31:0/3856458436 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56352ec16540 con 0x56352ed34720 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd97affd640 10 client.27662 handle_command_reply: tid=0 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 1 client.27662 shutdown 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3856458436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 msgr2=0x56352ecc6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecc6140 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x56352ecd0a50 tx=0x7fd990005f90 comp rx=0 tx=0).stop 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3856458436 shutdown_connections 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x56352ed34720 0x56352ed36b40 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56352ecbda40 0x56352ecb7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd97affd640 0 client.27662 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352ecb3370 0x56352ecc6140 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 1 --2- 172.21.15.31:0/3856458436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56352ecb2a50 0x56352ecc5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3856458436 >> 172.21.15.31:0/3856458436 conn(0x56352ecaed70 msgr2=0x56352eca4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3856458436 shutdown_connections 2024-04-25T02:19:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 1 -- 172.21.15.31:0/3856458436 wait complete. 2024-04-25T02:19:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.056+0000 7fd9a4e36740 20 client.27662 trim_cache size 0 max 16384 2024-04-25T02:19:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:19:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:19:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:19:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:19:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.057+0000 7fd9a204d640 1 -- 172.21.15.31:0/949132388 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd9840a2f00 msgr2=0x7fd9840a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.057+0000 7fd9a204d640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd9840a2f00 0x7fd9840a53c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x56352ecaff00 tx=0x7fd990015040 comp rx=0 tx=0).stop 2024-04-25T02:19:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.057+0000 7fd9a204d640 1 -- 172.21.15.31:0/949132388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd99c107760 msgr2=0x7fd99c0744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.057+0000 7fd9a204d640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd99c107760 0x7fd99c0744c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fd994033040 tx=0x7fd994009360 comp rx=0 tx=0).stop 2024-04-25T02:19:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.057+0000 7fd9a204d640 1 -- 172.21.15.31:0/949132388 shutdown_connections 2024-04-25T02:19:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.057+0000 7fd9a204d640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fd9840a2f00 0x7fd9840a53c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.057+0000 7fd9a204d640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd99c107760 0x7fd99c0744c0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.057+0000 7fd9a204d640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd99c102110 0x7fd99c073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.057+0000 7fd9a204d640 1 --2- 172.21.15.31:0/949132388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd99c101750 0x7fd99c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.057+0000 7fd9a204d640 1 -- 172.21.15.31:0/949132388 >> 172.21.15.31:0/949132388 conn(0x7fd99c0fd240 msgr2=0x7fd99c0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.058+0000 7fd9a204d640 1 -- 172.21.15.31:0/949132388 shutdown_connections 2024-04-25T02:19:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:01.058+0000 7fd9a204d640 1 -- 172.21.15.31:0/949132388 wait complete. 2024-04-25T02:19:01.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 108764 8.79 MB/sec execute 545 sec latency 103.112 ms 2024-04-25T02:19:02.112 INFO:tasks.workunit.client.0.smithi031.stdout: 10 108952 8.80 MB/sec execute 546 sec latency 172.375 ms 2024-04-25T02:19:02.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:02 smithi040 ceph-mon[32418]: pgmap v774: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.8 MiB/s wr, 330 op/s 2024-04-25T02:19:02.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:02 smithi134 ceph-mon[33098]: pgmap v774: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.8 MiB/s wr, 330 op/s 2024-04-25T02:19:02.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:02 smithi031 ceph-mon[29204]: pgmap v774: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.8 MiB/s wr, 330 op/s 2024-04-25T02:19:03.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 109104 8.80 MB/sec execute 547 sec latency 146.103 ms 2024-04-25T02:19:03.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:03 smithi040 ceph-mon[32418]: osdmap e126: 12 total, 12 up, 12 in 2024-04-25T02:19:03.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:03 smithi134 ceph-mon[33098]: osdmap e126: 12 total, 12 up, 12 in 2024-04-25T02:19:03.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:03 smithi031 ceph-mon[29204]: osdmap e126: 12 total, 12 up, 12 in 2024-04-25T02:19:04.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 109257 8.79 MB/sec execute 548 sec latency 56.648 ms 2024-04-25T02:19:04.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:04 smithi040 ceph-mon[32418]: pgmap v776: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.5 MiB/s wr, 251 op/s 2024-04-25T02:19:04.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:04 smithi134 ceph-mon[33098]: pgmap v776: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.5 MiB/s wr, 251 op/s 2024-04-25T02:19:04.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:04 smithi031 ceph-mon[29204]: pgmap v776: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.5 MiB/s wr, 251 op/s 2024-04-25T02:19:05.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 109401 8.78 MB/sec execute 549 sec latency 53.280 ms 2024-04-25T02:19:06.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 109544 8.77 MB/sec execute 550 sec latency 126.368 ms 2024-04-25T02:19:06.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:06 smithi040 ceph-mon[32418]: pgmap v777: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.4 MiB/s wr, 320 op/s 2024-04-25T02:19:06.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:06 smithi134 ceph-mon[33098]: pgmap v777: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.4 MiB/s wr, 320 op/s 2024-04-25T02:19:06.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:06 smithi031 ceph-mon[29204]: pgmap v777: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.4 MiB/s wr, 320 op/s 2024-04-25T02:19:07.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 109727 8.77 MB/sec execute 551 sec latency 73.579 ms 2024-04-25T02:19:08.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 109925 8.77 MB/sec execute 552 sec latency 78.880 ms 2024-04-25T02:19:08.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:08 smithi040 ceph-mon[32418]: pgmap v778: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.4 MiB/s wr, 320 op/s 2024-04-25T02:19:08.693 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:08 smithi134 ceph-mon[33098]: pgmap v778: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.4 MiB/s wr, 320 op/s 2024-04-25T02:19:08.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:08 smithi031 ceph-mon[29204]: pgmap v778: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.4 MiB/s wr, 320 op/s 2024-04-25T02:19:09.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 110085 8.78 MB/sec execute 553 sec latency 55.282 ms 2024-04-25T02:19:10.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 110265 8.79 MB/sec execute 554 sec latency 91.781 ms 2024-04-25T02:19:10.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:10 smithi040 ceph-mon[32418]: pgmap v779: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.4 MiB/s wr, 320 op/s 2024-04-25T02:19:10.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:10 smithi040 ceph-mon[32418]: osdmap e127: 12 total, 12 up, 12 in 2024-04-25T02:19:10.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:10 smithi134 ceph-mon[33098]: pgmap v779: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.4 MiB/s wr, 320 op/s 2024-04-25T02:19:10.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:10 smithi134 ceph-mon[33098]: osdmap e127: 12 total, 12 up, 12 in 2024-04-25T02:19:10.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:10 smithi031 ceph-mon[29204]: pgmap v779: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.4 MiB/s wr, 320 op/s 2024-04-25T02:19:10.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:10 smithi031 ceph-mon[29204]: osdmap e127: 12 total, 12 up, 12 in 2024-04-25T02:19:11.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 110496 8.81 MB/sec execute 555 sec latency 53.983 ms 2024-04-25T02:19:12.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 110651 8.81 MB/sec execute 556 sec latency 179.603 ms 2024-04-25T02:19:12.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:12 smithi040 ceph-mon[32418]: pgmap v781: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 368 op/s 2024-04-25T02:19:12.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:12 smithi134 ceph-mon[33098]: pgmap v781: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 368 op/s 2024-04-25T02:19:12.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:12 smithi031 ceph-mon[29204]: pgmap v781: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 368 op/s 2024-04-25T02:19:13.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 110776 8.81 MB/sec execute 557 sec latency 84.507 ms 2024-04-25T02:19:13.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:13 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:19:13.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:13 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:19:13.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:13 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:19:14.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 110944 8.81 MB/sec execute 558 sec latency 121.698 ms 2024-04-25T02:19:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.182+0000 7f289501c640 1 -- 172.21.15.31:0/2961411209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2890101030 msgr2=0x7f289010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.182+0000 7f289501c640 1 --2- 172.21.15.31:0/2961411209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2890101030 0x7f289010d4f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2888009be0 tx=0x7f288802a6c0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.182+0000 7f289501c640 1 -- 172.21.15.31:0/2961411209 shutdown_connections 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.182+0000 7f289501c640 1 --2- 172.21.15.31:0/2961411209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2890101030 0x7f289010d4f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.182+0000 7f289501c640 1 --2- 172.21.15.31:0/2961411209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2890100670 0x7f2890100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.182+0000 7f289501c640 1 --2- 172.21.15.31:0/2961411209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2890106630 0x7f2890106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.182+0000 7f289501c640 1 -- 172.21.15.31:0/2961411209 >> 172.21.15.31:0/2961411209 conn(0x7f28900777c0 msgr2=0x7f2890077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.182+0000 7f289501c640 1 -- 172.21.15.31:0/2961411209 shutdown_connections 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.182+0000 7f289501c640 1 -- 172.21.15.31:0/2961411209 wait complete. 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.183+0000 7f289501c640 1 Processor -- start 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.183+0000 7f289501c640 1 -- start start 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f289501c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2890100670 0x7f289010df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f289501c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2890101030 0x7f289010e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f289501c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2890106630 0x7f289010ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f289501c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28901119f0 con 0x7f2890106630 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f289501c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2890111870 con 0x7f2890101030 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f289501c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2890111b70 con 0x7f2890100670 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2890100670 0x7f289010df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2890100670 0x7f289010df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:50866/0 (socket says 172.21.15.31:50866) 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f288ed9d640 1 -- 172.21.15.31:0/2469506065 learned_addr learned my addr 172.21.15.31:0/2469506065 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f288f59e640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2890106630 0x7f289010ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f288e59c640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2890101030 0x7f289010e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.184+0000 7f288ed9d640 1 -- 172.21.15.31:0/2469506065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2890101030 msgr2=0x7f289010e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.185+0000 7f288ed9d640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2890101030 0x7f289010e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.185+0000 7f288ed9d640 1 -- 172.21.15.31:0/2469506065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2890106630 msgr2=0x7f289010ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.185+0000 7f288ed9d640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2890106630 0x7f289010ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.185+0000 7f288ed9d640 1 -- 172.21.15.31:0/2469506065 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28901a6330 con 0x7f2890100670 2024-04-25T02:19:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.185+0000 7f288f59e640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2890106630 0x7f289010ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.185+0000 7f288ed9d640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2890100670 0x7f289010df80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f288000e9f0 tx=0x7f288000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.185+0000 7f2877fff640 1 -- 172.21.15.31:0/2469506065 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288000cd80 con 0x7f2890100670 2024-04-25T02:19:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.185+0000 7f2877fff640 1 -- 172.21.15.31:0/2469506065 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28800108e0 con 0x7f2890100670 2024-04-25T02:19:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.185+0000 7f289501c640 1 -- 172.21.15.31:0/2469506065 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28901a6620 con 0x7f2890100670 2024-04-25T02:19:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.185+0000 7f2877fff640 1 -- 172.21.15.31:0/2469506065 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2880018920 con 0x7f2890100670 2024-04-25T02:19:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.185+0000 7f289501c640 1 -- 172.21.15.31:0/2469506065 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28901a6b70 con 0x7f2890100670 2024-04-25T02:19:14.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.188+0000 7f2877fff640 1 -- 172.21.15.31:0/2469506065 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2880010a90 con 0x7f2890100670 2024-04-25T02:19:14.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.188+0000 7f2897e09740 1 Processor -- start 2024-04-25T02:19:14.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.188+0000 7f2897e09740 1 -- start start 2024-04-25T02:19:14.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f2897e09740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 0x55a5be92c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f2897e09740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be936cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f2897e09740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 0x55a5be939620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f2897e09740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a5be890cb0 con 0x55a5be937230 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f2897e09740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a5be8395b0 con 0x55a5be92c240 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f2897e09740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a5be88cc70 con 0x55a5be92cb60 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be936cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f288f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 0x55a5be92c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be936cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:50876/0 (socket says 172.21.15.31:50876) 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f288ed9d640 1 -- 172.21.15.31:0/2597133411 learned_addr learned my addr 172.21.15.31:0/2597133411 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f288e59c640 1 --2- 172.21.15.31:0/2597133411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 0x55a5be939620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f2877fff640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f28780a2da0 0x7f28780a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f288ed9d640 1 -- 172.21.15.31:0/2597133411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 msgr2=0x55a5be92c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f288ed9d640 1 --2- 172.21.15.31:0/2597133411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 0x55a5be92c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.189+0000 7f288ed9d640 1 -- 172.21.15.31:0/2597133411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 msgr2=0x55a5be939620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.190+0000 7f288ed9d640 1 --2- 172.21.15.31:0/2597133411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 0x55a5be939620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.190+0000 7f288ed9d640 1 -- 172.21.15.31:0/2597133411 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5be5a5210 con 0x55a5be92cb60 2024-04-25T02:19:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.190+0000 7f288e59c640 1 --2- 172.21.15.31:0/2597133411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 0x55a5be939620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.190+0000 7f288f59e640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f28780a2da0 0x7f28780a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.190+0000 7f2877fff640 1 -- 172.21.15.31:0/2469506065 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f28800cf770 con 0x7f2890100670 2024-04-25T02:19:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.190+0000 7f288ed9d640 1 --2- 172.21.15.31:0/2597133411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be936cf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f28800cce50 tx=0x7f28800669a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d05d00bbd255b6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.190+0000 7f288f59e640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f28780a2da0 0x7f28780a5260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2888009370 tx=0x7f2888009500 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.190+0000 7f28757fa640 1 -- 172.21.15.31:0/2597133411 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2880026030 con 0x55a5be92cb60 2024-04-25T02:19:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.190+0000 7f28757fa640 1 -- 172.21.15.31:0/2597133411 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2880010de0 con 0x55a5be92cb60 2024-04-25T02:19:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.190+0000 7f28757fa640 1 -- 172.21.15.31:0/2597133411 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288009b100 con 0x55a5be92cb60 2024-04-25T02:19:14.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 -- 172.21.15.31:0/2597133411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 msgr2=0x55a5be936cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:14.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 --2- 172.21.15.31:0/2597133411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be936cf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f28800cce50 tx=0x7f28800669a0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 -- 172.21.15.31:0/2597133411 shutdown_connections 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 --2- 172.21.15.31:0/2597133411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 0x55a5be939620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 --2- 172.21.15.31:0/2597133411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be936cf0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 --2- 172.21.15.31:0/2597133411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 0x55a5be92c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 -- 172.21.15.31:0/2597133411 >> 172.21.15.31:0/2597133411 conn(0x55a5be928560 msgr2=0x55a5be928940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 -- 172.21.15.31:0/2597133411 shutdown_connections 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 -- 172.21.15.31:0/2597133411 wait complete. 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 -- start start 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 Processor -- start 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be934190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92cb60 0x55a5be9346d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.192+0000 7f2897e09740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be937230 0x55a5be934c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f2897e09740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a5be939fc0 con 0x55a5be92c240 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f2897e09740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a5be84cb50 con 0x55a5be92cb60 2024-04-25T02:19:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f2897e09740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a5be93a140 con 0x55a5be937230 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f2897e09740 10 client.0 _ll_register_callbacks cb 0x55a5be887ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be934190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be934190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34112/0 (socket says 172.21.15.31:34112) 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f288ed9d640 1 -- 172.21.15.31:0/1578135887 learned_addr learned my addr 172.21.15.31:0/1578135887 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f288f59e640 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be937230 0x55a5be934c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f288e59c640 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92cb60 0x55a5be9346d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f288ed9d640 1 -- 172.21.15.31:0/1578135887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be937230 msgr2=0x55a5be934c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f288ed9d640 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be937230 0x55a5be934c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f288ed9d640 1 -- 172.21.15.31:0/1578135887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92cb60 msgr2=0x55a5be9346d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f288ed9d640 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92cb60 0x55a5be9346d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f288ed9d640 1 -- 172.21.15.31:0/1578135887 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5be671620 con 0x55a5be92c240 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f288ed9d640 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be934190 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f288009b780 tx=0x7f288009bba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae46ef88ad290c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.193+0000 7f2872ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.194+0000 7f2872ff5640 1 -- 172.21.15.31:0/1578135887 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288009c3b0 con 0x55a5be92c240 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.194+0000 7f2872ff5640 1 -- 172.21.15.31:0/1578135887 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f288009b100 con 0x55a5be92c240 2024-04-25T02:19:14.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.194+0000 7f2897e09740 1 -- 172.21.15.31:0/1578135887 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a5be89e1f0 con 0x55a5be92c240 2024-04-25T02:19:14.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.194+0000 7f2872ff5640 1 -- 172.21.15.31:0/1578135887 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f288006f2e0 con 0x55a5be92c240 2024-04-25T02:19:14.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.194+0000 7f2897e09740 10 client.18108 fetch_fsmap learned FSMap version 51 2024-04-25T02:19:14.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.194+0000 7f2897e09740 1 -- 172.21.15.31:0/1578135887 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55a5be946180 con 0x55a5be92c240 2024-04-25T02:19:14.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.194+0000 7f2872ff5640 1 -- 172.21.15.31:0/1578135887 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a5be89e1f0 con 0x55a5be92c240 2024-04-25T02:19:14.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.195+0000 7f2872ff5640 1 -- 172.21.15.31:0/1578135887 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f28800cd160 con 0x55a5be92c240 2024-04-25T02:19:14.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.195+0000 7f2897e09740 10 client.18108 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:19:14.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.195+0000 7f2897e09740 10 client.18108 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:19:14.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.195+0000 7f2897e09740 20 client.18108 populate_metadata read hostname 'smithi031' 2024-04-25T02:19:14.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.195+0000 7f2897e09740 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a5be9474d0 0x55a5be9498c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.195+0000 7f2897e09740 4 client.18108 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.195+0000 7f2897e09740 1 -- 172.21.15.31:0/1578135887 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5be898540 con 0x55a5be9474d0 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.195+0000 7f288e59c640 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a5be9474d0 0x55a5be9498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.195+0000 7f288e59c640 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a5be9474d0 0x55a5be9498c0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26d2319ac8251147 server_cookie=711ab869de1948e7 in_seq=0 out_seq=0 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.195+0000 7f2872ff5640 10 client.18108 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.197+0000 7f2872ff5640 1 -- 172.21.15.31:0/1578135887 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a5be898540 con 0x55a5be9474d0 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.197+0000 7f2872ff5640 10 client.18108 handle_command_reply: tid=0 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.197+0000 7f2897e09740 1 client.18108 shutdown 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.197+0000 7f2897e09740 1 -- 172.21.15.31:0/1578135887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 msgr2=0x55a5be934190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.197+0000 7f2897e09740 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be934190 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f288009b780 tx=0x7f288009bba0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.197+0000 7f2897e09740 1 -- 172.21.15.31:0/1578135887 shutdown_connections 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.197+0000 7f2897e09740 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a5be9474d0 0x55a5be9498c0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.197+0000 7f2897e09740 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be937230 0x55a5be934c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.198+0000 7f2897e09740 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92cb60 0x55a5be9346d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.198+0000 7f2897e09740 1 --2- 172.21.15.31:0/1578135887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be934190 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.198+0000 7f2872ff5640 0 client.18108 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.198+0000 7f2897e09740 1 -- 172.21.15.31:0/1578135887 >> 172.21.15.31:0/1578135887 conn(0x55a5be928560 msgr2=0x55a5be92e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.198+0000 7f2897e09740 1 -- 172.21.15.31:0/1578135887 shutdown_connections 2024-04-25T02:19:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.198+0000 7f2897e09740 1 -- 172.21.15.31:0/1578135887 wait complete. 2024-04-25T02:19:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.198+0000 7f2897e09740 20 client.18108 trim_cache size 0 max 16384 2024-04-25T02:19:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.219+0000 7f2897e09740 1 Processor -- start 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f2897e09740 1 -- start start 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f2897e09740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be942e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f2897e09740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be927440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f2897e09740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be937230 0x55a5be94ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f2897e09740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28901ab090 con 0x55a5be92c240 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f2897e09740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f28901acbd0 con 0x55a5be937230 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f2897e09740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f28901ab760 con 0x55a5be92cb60 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be942e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be942e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34128/0 (socket says 172.21.15.31:34128) 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f288f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be937230 0x55a5be94ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f288ed9d640 1 -- 172.21.15.31:0/1783587432 learned_addr learned my addr 172.21.15.31:0/1783587432 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f288ed9d640 1 -- 172.21.15.31:0/1783587432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 msgr2=0x55a5be927440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f288ed9d640 1 --2- 172.21.15.31:0/1783587432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be927440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f288ed9d640 1 -- 172.21.15.31:0/1783587432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be937230 msgr2=0x55a5be94ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f288ed9d640 1 --2- 172.21.15.31:0/1783587432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be937230 0x55a5be94ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.220+0000 7f288ed9d640 1 -- 172.21.15.31:0/1783587432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5be92c990 con 0x55a5be92c240 2024-04-25T02:19:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.221+0000 7f288ed9d640 1 --2- 172.21.15.31:0/1783587432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be942e80 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f288000e9c0 tx=0x7f2880097f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51808f7058cb987 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.221+0000 7f2874ff9640 1 -- 172.21.15.31:0/1783587432 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288009c3b0 con 0x55a5be92c240 2024-04-25T02:19:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.221+0000 7f2874ff9640 1 -- 172.21.15.31:0/1783587432 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28800a4920 con 0x55a5be92c240 2024-04-25T02:19:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.221+0000 7f2874ff9640 1 -- 172.21.15.31:0/1783587432 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288009b100 con 0x55a5be92c240 2024-04-25T02:19:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.222+0000 7f2897e09740 1 -- 172.21.15.31:0/1783587432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 msgr2=0x55a5be942e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.222+0000 7f2897e09740 1 --2- 172.21.15.31:0/1783587432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be942e80 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f288000e9c0 tx=0x7f2880097f50 comp rx=0 tx=0).stop 2024-04-25T02:19:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.222+0000 7f2897e09740 1 -- 172.21.15.31:0/1783587432 shutdown_connections 2024-04-25T02:19:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.222+0000 7f2897e09740 1 --2- 172.21.15.31:0/1783587432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be937230 0x55a5be94ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.222+0000 7f2897e09740 1 --2- 172.21.15.31:0/1783587432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be927440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.222+0000 7f2897e09740 1 --2- 172.21.15.31:0/1783587432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be92c240 0x55a5be942e80 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.222+0000 7f2897e09740 1 -- 172.21.15.31:0/1783587432 >> 172.21.15.31:0/1783587432 conn(0x55a5be928560 msgr2=0x55a5be926d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.222+0000 7f2897e09740 1 -- 172.21.15.31:0/1783587432 shutdown_connections 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.222+0000 7f2897e09740 1 -- 172.21.15.31:0/1783587432 wait complete. 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.222+0000 7f2897e09740 1 -- start start 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.222+0000 7f2897e09740 1 Processor -- start 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f2897e09740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 0x55a5be93f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f2897e09740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be93aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f2897e09740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 0x55a5be93afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f2897e09740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28901ac500 con 0x55a5be937230 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f2897e09740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f28901abe30 con 0x55a5be92c240 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f2897e09740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f28901a90b0 con 0x55a5be92cb60 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288f59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 0x55a5be93afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 0x55a5be93f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288f59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 0x55a5be93afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34142/0 (socket says 172.21.15.31:34142) 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 0x55a5be93f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:36986/0 (socket says 172.21.15.31:36986) 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288f59e640 1 -- 172.21.15.31:0/4164583411 learned_addr learned my addr 172.21.15.31:0/4164583411 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f2897e09740 10 client.0 _ll_register_callbacks cb 0x7f2884009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288f59e640 1 -- 172.21.15.31:0/4164583411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 msgr2=0x55a5be93aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288e59c640 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be93aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288f59e640 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be93aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288f59e640 1 -- 172.21.15.31:0/4164583411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 msgr2=0x55a5be93f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288f59e640 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 0x55a5be93f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288f59e640 1 -- 172.21.15.31:0/4164583411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5be93b770 con 0x55a5be937230 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288e59c640 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be93aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.223+0000 7f288ed9d640 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 0x55a5be93f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.224+0000 7f288f59e640 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 0x55a5be93afa0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f2888009be0 tx=0x7f2888004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ab4a393681f8ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.224+0000 7f2872ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.224+0000 7f2872ff5640 1 -- 172.21.15.31:0/4164583411 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2888036040 con 0x55a5be937230 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.224+0000 7f2872ff5640 1 -- 172.21.15.31:0/4164583411 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f2888004720 con 0x55a5be937230 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.224+0000 7f2897e09740 1 -- 172.21.15.31:0/4164583411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28800a4920 con 0x55a5be937230 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.224+0000 7f2872ff5640 1 -- 172.21.15.31:0/4164583411 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f288804c440 con 0x55a5be937230 2024-04-25T02:19:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.224+0000 7f2897e09740 10 client.18114 fetch_fsmap learned FSMap version 51 2024-04-25T02:19:14.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.224+0000 7f2897e09740 1 -- 172.21.15.31:0/4164583411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55a5be9ad6b0 con 0x55a5be937230 2024-04-25T02:19:14.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.224+0000 7f2872ff5640 1 -- 172.21.15.31:0/4164583411 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28800a4920 con 0x55a5be937230 2024-04-25T02:19:14.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.225+0000 7f2872ff5640 1 -- 172.21.15.31:0/4164583411 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f288803c030 con 0x55a5be937230 2024-04-25T02:19:14.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.225+0000 7f2897e09740 10 client.18114 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:19:14.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.225+0000 7f2897e09740 10 client.18114 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:19:14.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.225+0000 7f2897e09740 20 client.18114 populate_metadata read hostname 'smithi031' 2024-04-25T02:19:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.225+0000 7f2897e09740 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a5be9ade60 0x55a5be9b0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.225+0000 7f2897e09740 4 client.18114 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:19:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.225+0000 7f2897e09740 1 -- 172.21.15.31:0/4164583411 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a5be8168e0 con 0x55a5be9ade60 2024-04-25T02:19:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.226+0000 7f288ed9d640 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a5be9ade60 0x55a5be9b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.226+0000 7f2872ff5640 10 client.18114 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.226+0000 7f288ed9d640 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a5be9ade60 0x55a5be9b0280 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=388290ed25e82a6b server_cookie=2e951138bee796e9 in_seq=0 out_seq=0 2024-04-25T02:19:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2872ff5640 1 -- 172.21.15.31:0/4164583411 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a5be8168e0 con 0x55a5be9ade60 2024-04-25T02:19:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2872ff5640 10 client.18114 handle_command_reply: tid=0 2024-04-25T02:19:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2897e09740 1 client.18114 shutdown 2024-04-25T02:19:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2897e09740 1 -- 172.21.15.31:0/4164583411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 msgr2=0x55a5be93afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2897e09740 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 0x55a5be93afa0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f2888009be0 tx=0x7f2888004a40 comp rx=0 tx=0).stop 2024-04-25T02:19:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2897e09740 1 -- 172.21.15.31:0/4164583411 shutdown_connections 2024-04-25T02:19:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2897e09740 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55a5be9ade60 0x55a5be9b0280 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2897e09740 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5be937230 0x55a5be93afa0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2897e09740 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a5be92cb60 0x55a5be93aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2897e09740 1 --2- 172.21.15.31:0/4164583411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5be92c240 0x55a5be93f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2897e09740 1 -- 172.21.15.31:0/4164583411 >> 172.21.15.31:0/4164583411 conn(0x55a5be928560 msgr2=0x55a5be9275c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2872ff5640 0 client.18114 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.302+0000 7f2897e09740 1 -- 172.21.15.31:0/4164583411 shutdown_connections 2024-04-25T02:19:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.303+0000 7f2897e09740 1 -- 172.21.15.31:0/4164583411 wait complete. 2024-04-25T02:19:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.303+0000 7f2897e09740 20 client.18114 trim_cache size 0 max 16384 2024-04-25T02:19:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:19:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:19:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:19:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:19:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.303+0000 7f289501c640 1 -- 172.21.15.31:0/2469506065 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f28780a2da0 msgr2=0x7f28780a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.303+0000 7f289501c640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f28780a2da0 0x7f28780a5260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2888009370 tx=0x7f2888009500 comp rx=0 tx=0).stop 2024-04-25T02:19:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.303+0000 7f289501c640 1 -- 172.21.15.31:0/2469506065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2890100670 msgr2=0x7f289010df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.303+0000 7f289501c640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2890100670 0x7f289010df80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f288000e9f0 tx=0x7f288000eeb0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.303+0000 7f289501c640 1 -- 172.21.15.31:0/2469506065 shutdown_connections 2024-04-25T02:19:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.303+0000 7f289501c640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f28780a2da0 0x7f28780a5260 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.303+0000 7f289501c640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2890106630 0x7f289010ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.303+0000 7f289501c640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2890101030 0x7f289010e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.303+0000 7f289501c640 1 --2- 172.21.15.31:0/2469506065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2890100670 0x7f289010df80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.304+0000 7f289501c640 1 -- 172.21.15.31:0/2469506065 >> 172.21.15.31:0/2469506065 conn(0x7f28900777c0 msgr2=0x7f2890103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.304+0000 7f289501c640 1 -- 172.21.15.31:0/2469506065 shutdown_connections 2024-04-25T02:19:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:14.304+0000 7f289501c640 1 -- 172.21.15.31:0/2469506065 wait complete. 2024-04-25T02:19:14.343 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:14 smithi134 ceph-mon[33098]: pgmap v782: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.5 MiB/s wr, 315 op/s 2024-04-25T02:19:14.343 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:14 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:19:14.343 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:14 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:19:14.343 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:14 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:19:14.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:14 smithi040 ceph-mon[32418]: pgmap v782: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.5 MiB/s wr, 315 op/s 2024-04-25T02:19:14.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:14 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:19:14.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:14 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:19:14.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:14 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:19:14.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:14 smithi031 ceph-mon[29204]: pgmap v782: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.5 MiB/s wr, 315 op/s 2024-04-25T02:19:14.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:14 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:19:14.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:14 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:19:14.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:14 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:19:15.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 111072 8.80 MB/sec execute 559 sec latency 110.660 ms 2024-04-25T02:19:16.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 111229 8.80 MB/sec execute 560 sec latency 103.718 ms 2024-04-25T02:19:16.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:16 smithi040 ceph-mon[32418]: pgmap v783: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 270 op/s 2024-04-25T02:19:16.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:16 smithi134 ceph-mon[33098]: pgmap v783: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 270 op/s 2024-04-25T02:19:16.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:16 smithi031 ceph-mon[29204]: pgmap v783: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 270 op/s 2024-04-25T02:19:17.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 111378 8.79 MB/sec execute 561 sec latency 67.047 ms 2024-04-25T02:19:18.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 111520 8.78 MB/sec execute 562 sec latency 87.927 ms 2024-04-25T02:19:18.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:18 smithi040 ceph-mon[32418]: pgmap v784: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 270 op/s 2024-04-25T02:19:18.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:18 smithi134 ceph-mon[33098]: pgmap v784: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 270 op/s 2024-04-25T02:19:18.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:18 smithi031 ceph-mon[29204]: pgmap v784: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 270 op/s 2024-04-25T02:19:19.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 111678 8.78 MB/sec execute 563 sec latency 109.671 ms 2024-04-25T02:19:20.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 111863 8.79 MB/sec execute 564 sec latency 61.215 ms 2024-04-25T02:19:20.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:20 smithi040 ceph-mon[32418]: pgmap v785: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 270 op/s 2024-04-25T02:19:20.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:20 smithi134 ceph-mon[33098]: pgmap v785: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 270 op/s 2024-04-25T02:19:20.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:20 smithi031 ceph-mon[29204]: pgmap v785: 97 pgs: 97 active+clean; 1.7 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.0 MiB/s wr, 270 op/s 2024-04-25T02:19:21.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 112025 8.79 MB/sec execute 565 sec latency 60.402 ms 2024-04-25T02:19:22.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 112206 8.79 MB/sec execute 566 sec latency 73.835 ms 2024-04-25T02:19:22.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:22 smithi040 ceph-mon[32418]: pgmap v786: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.9 MiB/s wr, 339 op/s 2024-04-25T02:19:22.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:22 smithi134 ceph-mon[33098]: pgmap v786: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.9 MiB/s wr, 339 op/s 2024-04-25T02:19:22.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:22 smithi031 ceph-mon[29204]: pgmap v786: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.9 MiB/s wr, 339 op/s 2024-04-25T02:19:23.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 112392 8.79 MB/sec execute 567 sec latency 56.201 ms 2024-04-25T02:19:24.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 112557 8.79 MB/sec execute 568 sec latency 53.612 ms 2024-04-25T02:19:24.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:24 smithi040 ceph-mon[32418]: pgmap v787: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 222 op/s 2024-04-25T02:19:24.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:24 smithi134 ceph-mon[33098]: pgmap v787: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 222 op/s 2024-04-25T02:19:24.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:24 smithi031 ceph-mon[29204]: pgmap v787: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.0 MiB/s wr, 222 op/s 2024-04-25T02:19:25.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 112719 8.79 MB/sec execute 569 sec latency 56.622 ms 2024-04-25T02:19:26.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 112879 8.79 MB/sec execute 570 sec latency 66.114 ms 2024-04-25T02:19:26.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:26 smithi040 ceph-mon[32418]: pgmap v788: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.5 MiB/s wr, 349 op/s 2024-04-25T02:19:26.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:26 smithi134 ceph-mon[33098]: pgmap v788: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.5 MiB/s wr, 349 op/s 2024-04-25T02:19:26.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:26 smithi031 ceph-mon[29204]: pgmap v788: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.5 MiB/s wr, 349 op/s 2024-04-25T02:19:27.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 113030 8.78 MB/sec execute 571 sec latency 93.823 ms 2024-04-25T02:19:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.433+0000 7f5472e58640 1 -- 172.21.15.31:0/4154984410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f546c101720 msgr2=0x7f546c101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.433+0000 7f5472e58640 1 --2- 172.21.15.31:0/4154984410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f546c101720 0x7f546c101ba0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f5458009890 tx=0x7f545802a360 comp rx=0 tx=0).stop 2024-04-25T02:19:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.433+0000 7f5472e58640 1 -- 172.21.15.31:0/4154984410 shutdown_connections 2024-04-25T02:19:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.433+0000 7f5472e58640 1 --2- 172.21.15.31:0/4154984410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f546c1020e0 0x7f546c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.433+0000 7f5472e58640 1 --2- 172.21.15.31:0/4154984410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f546c101720 0x7f546c101ba0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.433+0000 7f5472e58640 1 --2- 172.21.15.31:0/4154984410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546c107730 0x7f546c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.433+0000 7f5472e58640 1 -- 172.21.15.31:0/4154984410 >> 172.21.15.31:0/4154984410 conn(0x7f546c0fd1f0 msgr2=0x7f546c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.433+0000 7f5472e58640 1 -- 172.21.15.31:0/4154984410 shutdown_connections 2024-04-25T02:19:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.433+0000 7f5472e58640 1 -- 172.21.15.31:0/4154984410 wait complete. 2024-04-25T02:19:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.433+0000 7f5472e58640 1 Processor -- start 2024-04-25T02:19:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5472e58640 1 -- start start 2024-04-25T02:19:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5472e58640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f546c101720 0x7f546c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5472e58640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f546c1020e0 0x7f546c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5472e58640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546c107730 0x7f546c19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5472e58640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f546c1119b0 con 0x7f546c101720 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5472e58640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f546c111830 con 0x7f546c107730 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5472e58640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f546c111b30 con 0x7f546c1020e0 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5470bf4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f546c101720 0x7f546c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5470bf4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f546c101720 0x7f546c197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44096/0 (socket says 172.21.15.31:44096) 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f546bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f546c1020e0 0x7f546c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5470bf4640 1 -- 172.21.15.31:0/176214046 learned_addr learned my addr 172.21.15.31:0/176214046 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f54713f5640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546c107730 0x7f546c19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5470bf4640 1 -- 172.21.15.31:0/176214046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f546c1020e0 msgr2=0x7f546c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5470bf4640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f546c1020e0 0x7f546c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5470bf4640 1 -- 172.21.15.31:0/176214046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546c107730 msgr2=0x7f546c19c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5470bf4640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546c107730 0x7f546c19c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f5470bf4640 1 -- 172.21.15.31:0/176214046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546c19cf00 con 0x7f546c101720 2024-04-25T02:19:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.434+0000 7f546bfff640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f546c1020e0 0x7f546c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.435+0000 7f5470bf4640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f546c101720 0x7f546c197f10 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f546000e9c0 tx=0x7f546000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.435+0000 7f5469ffb640 1 -- 172.21.15.31:0/176214046 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546000cde0 con 0x7f546c101720 2024-04-25T02:19:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.435+0000 7f5469ffb640 1 -- 172.21.15.31:0/176214046 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54600035f0 con 0x7f546c101720 2024-04-25T02:19:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.435+0000 7f5472e58640 1 -- 172.21.15.31:0/176214046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546c19d1f0 con 0x7f546c101720 2024-04-25T02:19:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.435+0000 7f5469ffb640 1 -- 172.21.15.31:0/176214046 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460010690 con 0x7f546c101720 2024-04-25T02:19:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.435+0000 7f5472e58640 1 -- 172.21.15.31:0/176214046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f546c1a4a80 con 0x7f546c101720 2024-04-25T02:19:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.438+0000 7f5475b63740 1 Processor -- start 2024-04-25T02:19:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.438+0000 7f5475b63740 1 -- start start 2024-04-25T02:19:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.438+0000 7f5469ffb640 1 -- 172.21.15.31:0/176214046 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f54600037a0 con 0x7f546c101720 2024-04-25T02:19:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f5475b63740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d782f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f5475b63740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 0x55b8f6d8b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f5475b63740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d8be10 0x55b8f6d8e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f5475b63740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b8f6cdccb0 con 0x55b8f6d78830 2024-04-25T02:19:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f5475b63740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b8f6c855b0 con 0x55b8f6d8be10 2024-04-25T02:19:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f5475b63740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b8f6cd8c70 con 0x55b8f6d77f10 2024-04-25T02:19:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f54713f5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d782f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f54713f5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d782f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:36988/0 (socket says 172.21.15.31:36988) 2024-04-25T02:19:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f54713f5640 1 -- 172.21.15.31:0/3681948574 learned_addr learned my addr 172.21.15.31:0/3681948574 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f5470bf4640 1 --2- 172.21.15.31:0/3681948574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 0x55b8f6d8b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f546bfff640 1 --2- 172.21.15.31:0/3681948574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d8be10 0x55b8f6d8e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f54713f5640 1 -- 172.21.15.31:0/3681948574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d8be10 msgr2=0x55b8f6d8e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f54713f5640 1 --2- 172.21.15.31:0/3681948574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d8be10 0x55b8f6d8e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f54713f5640 1 -- 172.21.15.31:0/3681948574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 msgr2=0x55b8f6d8b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f54713f5640 1 --2- 172.21.15.31:0/3681948574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 0x55b8f6d8b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f54713f5640 1 -- 172.21.15.31:0/3681948574 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8f69f1210 con 0x55b8f6d77f10 2024-04-25T02:19:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.439+0000 7f5470bf4640 1 --2- 172.21.15.31:0/3681948574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 0x55b8f6d8b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:19:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.440+0000 7f54713f5640 1 --2- 172.21.15.31:0/3681948574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d782f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5464009fd0 tx=0x7f54640118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad32c6bc0c626531 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.440+0000 7f5456ffd640 1 -- 172.21.15.31:0/3681948574 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464004d20 con 0x55b8f6d77f10 2024-04-25T02:19:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.440+0000 7f5456ffd640 1 -- 172.21.15.31:0/3681948574 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5464004050 con 0x55b8f6d77f10 2024-04-25T02:19:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.440+0000 7f5456ffd640 1 -- 172.21.15.31:0/3681948574 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464005260 con 0x55b8f6d77f10 2024-04-25T02:19:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.440+0000 7f5469ffb640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f545c0a2ea0 0x7f545c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.440+0000 7f5469ffb640 1 -- 172.21.15.31:0/176214046 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f54600cf690 con 0x7f546c101720 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.440+0000 7f5470bf4640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f545c0a2ea0 0x7f545c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.441+0000 7f5475b63740 1 -- 172.21.15.31:0/3681948574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 msgr2=0x55b8f6d782f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.441+0000 7f5475b63740 1 --2- 172.21.15.31:0/3681948574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d782f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5464009fd0 tx=0x7f54640118d0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.441+0000 7f5470bf4640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f545c0a2ea0 0x7f545c0a5360 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x55b8f6d7ac40 tx=0x7f5460003e10 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.441+0000 7f5475b63740 1 -- 172.21.15.31:0/3681948574 shutdown_connections 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.441+0000 7f5475b63740 1 --2- 172.21.15.31:0/3681948574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d8be10 0x55b8f6d8e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.441+0000 7f5475b63740 1 --2- 172.21.15.31:0/3681948574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 0x55b8f6d8b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.441+0000 7f5475b63740 1 --2- 172.21.15.31:0/3681948574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d782f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.441+0000 7f5475b63740 1 -- 172.21.15.31:0/3681948574 >> 172.21.15.31:0/3681948574 conn(0x55b8f6d74560 msgr2=0x55b8f6d74940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.441+0000 7f5475b63740 1 -- 172.21.15.31:0/3681948574 shutdown_connections 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.442+0000 7f5475b63740 1 -- 172.21.15.31:0/3681948574 wait complete. 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.442+0000 7f5475b63740 1 -- start start 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.442+0000 7f5475b63740 1 Processor -- start 2024-04-25T02:19:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.442+0000 7f5475b63740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d77f10 0x55b8f6d88d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.442+0000 7f5475b63740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d78830 0x55b8f6d89280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.442+0000 7f5475b63740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d8be10 0x55b8f6d897c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.442+0000 7f5475b63740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b8f6d8e830 con 0x55b8f6d77f10 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.442+0000 7f5475b63740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b8f6c98b50 con 0x55b8f6d8be10 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.442+0000 7f5475b63740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b8f6d8e9b0 con 0x55b8f6d78830 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.442+0000 7f5475b63740 10 client.0 _ll_register_callbacks cb 0x55b8f6cd3ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f54713f5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d77f10 0x55b8f6d88d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f546bfff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d78830 0x55b8f6d89280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f54713f5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d77f10 0x55b8f6d88d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44118/0 (socket says 172.21.15.31:44118) 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f54713f5640 1 -- 172.21.15.31:0/3211667625 learned_addr learned my addr 172.21.15.31:0/3211667625 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f54713f5640 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d8be10 0x55b8f6d897c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f546bfff640 1 -- 172.21.15.31:0/3211667625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d8be10 msgr2=0x55b8f6d897c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f546bfff640 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d8be10 0x55b8f6d897c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f546bfff640 1 -- 172.21.15.31:0/3211667625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d77f10 msgr2=0x55b8f6d88d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:27.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f546bfff640 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d77f10 0x55b8f6d88d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f546bfff640 1 -- 172.21.15.31:0/3211667625 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8f6abd620 con 0x55b8f6d78830 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f546bfff640 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d78830 0x55b8f6d89280 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f545802a950 tx=0x7f5458030bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11dde15eb476d4cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f544bfff640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f544bfff640 1 -- 172.21.15.31:0/3211667625 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458037040 con 0x55b8f6d78830 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.443+0000 7f5475b63740 1 -- 172.21.15.31:0/3211667625 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5464004050 con 0x55b8f6d78830 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f544bfff640 1 -- 172.21.15.31:0/3211667625 <== mon.1 v2:172.21.15.134:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f5458004240 con 0x55b8f6d78830 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f544bfff640 1 -- 172.21.15.31:0/3211667625 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f545804d050 con 0x55b8f6d78830 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f5475b63740 10 client.27460 fetch_fsmap learned FSMap version 51 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f5475b63740 1 -- 172.21.15.31:0/3211667625 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55b8f6d9aa60 con 0x55b8f6d78830 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f544bfff640 1 -- 172.21.15.31:0/3211667625 <== mon.1 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5464004050 con 0x55b8f6d78830 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f544bfff640 1 -- 172.21.15.31:0/3211667625 <== mon.1 v2:172.21.15.134:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f5458032b80 con 0x55b8f6d78830 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f5475b63740 10 client.27460 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f5475b63740 10 client.27460 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:19:27.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f5475b63740 20 client.27460 populate_metadata read hostname 'smithi031' 2024-04-25T02:19:27.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f5475b63740 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b8f6d9bed0 0x55b8f6d9e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f5475b63740 4 client.27460 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:19:27.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f5475b63740 1 -- 172.21.15.31:0/3211667625 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8f6cea1f0 con 0x55b8f6d9bed0 2024-04-25T02:19:27.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.444+0000 7f54713f5640 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b8f6d9bed0 0x55b8f6d9e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.445+0000 7f544bfff640 10 client.27460 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:27.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.445+0000 7f54713f5640 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b8f6d9bed0 0x55b8f6d9e2c0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84ae701e1dce1650 server_cookie=a73b60a9b9a36639 in_seq=0 out_seq=0 2024-04-25T02:19:27.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f544bfff640 1 -- 172.21.15.31:0/3211667625 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b8f6cea1f0 con 0x55b8f6d9bed0 2024-04-25T02:19:27.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f544bfff640 10 client.27460 handle_command_reply: tid=0 2024-04-25T02:19:27.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 1 client.27460 shutdown 2024-04-25T02:19:27.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 1 -- 172.21.15.31:0/3211667625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d78830 msgr2=0x55b8f6d89280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:27.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d78830 0x55b8f6d89280 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f545802a950 tx=0x7f5458030bb0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 1 -- 172.21.15.31:0/3211667625 shutdown_connections 2024-04-25T02:19:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b8f6d9bed0 0x55b8f6d9e2c0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f544bfff640 0 client.27460 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d8be10 0x55b8f6d897c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d78830 0x55b8f6d89280 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 1 --2- 172.21.15.31:0/3211667625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d77f10 0x55b8f6d88d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 1 -- 172.21.15.31:0/3211667625 >> 172.21.15.31:0/3211667625 conn(0x55b8f6d74560 msgr2=0x55b8f6d7a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 1 -- 172.21.15.31:0/3211667625 shutdown_connections 2024-04-25T02:19:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 1 -- 172.21.15.31:0/3211667625 wait complete. 2024-04-25T02:19:27.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.446+0000 7f5475b63740 20 client.27460 trim_cache size 0 max 16384 2024-04-25T02:19:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f5475b63740 1 Processor -- start 2024-04-25T02:19:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f5475b63740 1 -- start start 2024-04-25T02:19:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f5475b63740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d8b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f5475b63740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d78830 0x55b8f6d97730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f5475b63740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d8be10 0x55b8f6d7a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f5475b63740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f546c1ab240 con 0x55b8f6d8be10 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f5475b63740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f546c1acd80 con 0x55b8f6d78830 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f5475b63740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f546c1ab910 con 0x55b8f6d77f10 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f54713f5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d8b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f54713f5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d8b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:37004/0 (socket says 172.21.15.31:37004) 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f54713f5640 1 -- 172.21.15.31:0/2657301247 learned_addr learned my addr 172.21.15.31:0/2657301247 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f546bfff640 1 --2- 172.21.15.31:0/2657301247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d78830 0x55b8f6d97730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f54713f5640 1 --2- 172.21.15.31:0/2657301247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d8be10 0x55b8f6d7a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f54713f5640 1 -- 172.21.15.31:0/2657301247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d78830 msgr2=0x55b8f6d97730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f54713f5640 1 --2- 172.21.15.31:0/2657301247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d78830 0x55b8f6d97730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f54713f5640 1 -- 172.21.15.31:0/2657301247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d8be10 msgr2=0x55b8f6d7a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f54713f5640 1 --2- 172.21.15.31:0/2657301247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d8be10 0x55b8f6d7a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.470+0000 7f54713f5640 1 -- 172.21.15.31:0/2657301247 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8f6d7afc0 con 0x55b8f6d77f10 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.471+0000 7f54713f5640 1 --2- 172.21.15.31:0/2657301247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d8b470 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5464004950 tx=0x7f5464004280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcb41c1ef67770d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.471+0000 7f54567fc640 1 -- 172.21.15.31:0/2657301247 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464003a80 con 0x55b8f6d77f10 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.471+0000 7f54567fc640 1 -- 172.21.15.31:0/2657301247 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f546400f4e0 con 0x55b8f6d77f10 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.471+0000 7f54567fc640 1 -- 172.21.15.31:0/2657301247 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464019690 con 0x55b8f6d77f10 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.472+0000 7f5475b63740 1 -- 172.21.15.31:0/2657301247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 msgr2=0x55b8f6d8b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.472+0000 7f5475b63740 1 --2- 172.21.15.31:0/2657301247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d8b470 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5464004950 tx=0x7f5464004280 comp rx=0 tx=0).stop 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.472+0000 7f5475b63740 1 -- 172.21.15.31:0/2657301247 shutdown_connections 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.472+0000 7f5475b63740 1 --2- 172.21.15.31:0/2657301247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d8be10 0x55b8f6d7a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.472+0000 7f5475b63740 1 --2- 172.21.15.31:0/2657301247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d78830 0x55b8f6d97730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.472+0000 7f5475b63740 1 --2- 172.21.15.31:0/2657301247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d77f10 0x55b8f6d8b470 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.472+0000 7f5475b63740 1 -- 172.21.15.31:0/2657301247 >> 172.21.15.31:0/2657301247 conn(0x55b8f6d74560 msgr2=0x55b8f6d79ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.472+0000 7f5475b63740 1 -- 172.21.15.31:0/2657301247 shutdown_connections 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.472+0000 7f5475b63740 1 -- 172.21.15.31:0/2657301247 wait complete. 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.472+0000 7f5475b63740 1 -- start start 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f5475b63740 1 Processor -- start 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f5475b63740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d77f10 0x55b8f6d93fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f5475b63740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 0x55b8f6d8f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f5475b63740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d8be10 0x55b8f6d8f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f5475b63740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f546c1ac6b0 con 0x55b8f6d78830 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f5475b63740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f546c1abfe0 con 0x55b8f6d77f10 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f5475b63740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f546c1a9240 con 0x55b8f6d8be10 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f5475b63740 10 client.0 _ll_register_callbacks cb 0x7f5460005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f546bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 0x55b8f6d8f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f54713f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d77f10 0x55b8f6d93fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f54713f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d77f10 0x55b8f6d93fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:44888/0 (socket says 172.21.15.31:44888) 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f54713f5640 1 -- 172.21.15.31:0/3525916047 learned_addr learned my addr 172.21.15.31:0/3525916047 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f546bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 0x55b8f6d8f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44146/0 (socket says 172.21.15.31:44146) 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.473+0000 7f54713f5640 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d8be10 0x55b8f6d8f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f54713f5640 1 -- 172.21.15.31:0/3525916047 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d8be10 msgr2=0x55b8f6d8f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f54713f5640 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d8be10 0x55b8f6d8f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f54713f5640 1 -- 172.21.15.31:0/3525916047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 msgr2=0x55b8f6d8f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f54713f5640 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 0x55b8f6d8f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f54713f5640 1 -- 172.21.15.31:0/3525916047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8f6d90090 con 0x55b8f6d77f10 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f546bfff640 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 0x55b8f6d8f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f54713f5640 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d77f10 0x55b8f6d93fb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f546400fe70 tx=0x7f5464013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f61fe363769c119 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f544bfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f544bfff640 1 -- 172.21.15.31:0/3525916047 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546400f750 con 0x55b8f6d77f10 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f544bfff640 1 -- 172.21.15.31:0/3525916047 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f546400f8f0 con 0x55b8f6d77f10 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f5475b63740 1 -- 172.21.15.31:0/3525916047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f546400f4e0 con 0x55b8f6d77f10 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f544bfff640 1 -- 172.21.15.31:0/3525916047 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f546400faf0 con 0x55b8f6d77f10 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f5475b63740 10 client.27698 fetch_fsmap learned FSMap version 51 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.474+0000 7f5475b63740 1 -- 172.21.15.31:0/3525916047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55b8f6e02420 con 0x55b8f6d77f10 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.475+0000 7f544bfff640 1 -- 172.21.15.31:0/3525916047 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f546400f4e0 con 0x55b8f6d77f10 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.475+0000 7f544bfff640 1 -- 172.21.15.31:0/3525916047 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f5464025040 con 0x55b8f6d77f10 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.475+0000 7f5475b63740 10 client.27698 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.475+0000 7f5475b63740 10 client.27698 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.475+0000 7f5475b63740 20 client.27698 populate_metadata read hostname 'smithi031' 2024-04-25T02:19:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.475+0000 7f5475b63740 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b8f6d9bed0 0x55b8f6e048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.475+0000 7f5475b63740 4 client.27698 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:19:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.475+0000 7f5475b63740 1 -- 172.21.15.31:0/3525916047 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b8f6ce4540 con 0x55b8f6d9bed0 2024-04-25T02:19:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.475+0000 7f546bfff640 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b8f6d9bed0 0x55b8f6e048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.476+0000 7f546bfff640 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b8f6d9bed0 0x55b8f6e048f0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cc2721930991b8f server_cookie=90ac86479a97c71c in_seq=0 out_seq=0 2024-04-25T02:19:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.476+0000 7f544bfff640 10 client.27698 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f544bfff640 1 -- 172.21.15.31:0/3525916047 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b8f6ce4540 con 0x55b8f6d9bed0 2024-04-25T02:19:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f544bfff640 10 client.27698 handle_command_reply: tid=0 2024-04-25T02:19:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f5475b63740 1 client.27698 shutdown 2024-04-25T02:19:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f5475b63740 1 -- 172.21.15.31:0/3525916047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d77f10 msgr2=0x55b8f6d93fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f5475b63740 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d77f10 0x55b8f6d93fb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f546400fe70 tx=0x7f5464013040 comp rx=0 tx=0).stop 2024-04-25T02:19:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f5475b63740 1 -- 172.21.15.31:0/3525916047 shutdown_connections 2024-04-25T02:19:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f5475b63740 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55b8f6d9bed0 0x55b8f6e048f0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f5475b63740 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b8f6d8be10 0x55b8f6d8f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f5475b63740 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b8f6d78830 0x55b8f6d8f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f544bfff640 0 client.27698 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f5475b63740 1 --2- 172.21.15.31:0/3525916047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b8f6d77f10 0x55b8f6d93fb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f5475b63740 1 -- 172.21.15.31:0/3525916047 >> 172.21.15.31:0/3525916047 conn(0x55b8f6d74560 msgr2=0x55b8f6d735c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f5475b63740 1 -- 172.21.15.31:0/3525916047 shutdown_connections 2024-04-25T02:19:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.560+0000 7f5475b63740 1 -- 172.21.15.31:0/3525916047 wait complete. 2024-04-25T02:19:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.561+0000 7f5475b63740 20 client.27698 trim_cache size 0 max 16384 2024-04-25T02:19:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:19:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:19:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:19:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:19:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.561+0000 7f5472e58640 1 -- 172.21.15.31:0/176214046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f545c0a2ea0 msgr2=0x7f545c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.561+0000 7f5472e58640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f545c0a2ea0 0x7f545c0a5360 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x55b8f6d7ac40 tx=0x7f5460003e10 comp rx=0 tx=0).stop 2024-04-25T02:19:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.561+0000 7f5472e58640 1 -- 172.21.15.31:0/176214046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f546c101720 msgr2=0x7f546c197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.561+0000 7f5472e58640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f546c101720 0x7f546c197f10 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f546000e9c0 tx=0x7f546000ee80 comp rx=0 tx=0).stop 2024-04-25T02:19:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.561+0000 7f5472e58640 1 -- 172.21.15.31:0/176214046 shutdown_connections 2024-04-25T02:19:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.561+0000 7f5472e58640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f545c0a2ea0 0x7f545c0a5360 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.561+0000 7f5472e58640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546c107730 0x7f546c19c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.561+0000 7f5472e58640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f546c1020e0 0x7f546c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.561+0000 7f5472e58640 1 --2- 172.21.15.31:0/176214046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f546c101720 0x7f546c197f10 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.561+0000 7f5472e58640 1 -- 172.21.15.31:0/176214046 >> 172.21.15.31:0/176214046 conn(0x7f546c0fd1f0 msgr2=0x7f546c0fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.562+0000 7f5472e58640 1 -- 172.21.15.31:0/176214046 shutdown_connections 2024-04-25T02:19:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:27.562+0000 7f5472e58640 1 -- 172.21.15.31:0/176214046 wait complete. 2024-04-25T02:19:28.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 113204 8.78 MB/sec execute 572 sec latency 178.458 ms 2024-04-25T02:19:28.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:28 smithi040 ceph-mon[32418]: pgmap v789: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.5 MiB/s wr, 239 op/s 2024-04-25T02:19:28.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:28 smithi134 ceph-mon[33098]: pgmap v789: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.5 MiB/s wr, 239 op/s 2024-04-25T02:19:28.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:28 smithi031 ceph-mon[29204]: pgmap v789: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.5 MiB/s wr, 239 op/s 2024-04-25T02:19:29.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 113392 8.79 MB/sec execute 573 sec latency 239.697 ms 2024-04-25T02:19:30.113 INFO:tasks.workunit.client.0.smithi031.stdout: 10 113545 8.78 MB/sec execute 574 sec latency 69.307 ms 2024-04-25T02:19:30.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:30 smithi040 ceph-mon[32418]: pgmap v790: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.5 MiB/s wr, 239 op/s 2024-04-25T02:19:30.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:30 smithi134 ceph-mon[33098]: pgmap v790: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.5 MiB/s wr, 239 op/s 2024-04-25T02:19:30.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:30 smithi031 ceph-mon[29204]: pgmap v790: 97 pgs: 97 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.5 MiB/s wr, 239 op/s 2024-04-25T02:19:31.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 113696 8.79 MB/sec execute 575 sec latency 98.236 ms 2024-04-25T02:19:32.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 113889 8.80 MB/sec execute 576 sec latency 313.401 ms 2024-04-25T02:19:32.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:32 smithi040 ceph-mon[32418]: pgmap v791: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.4 MiB/s wr, 360 op/s 2024-04-25T02:19:32.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:32 smithi134 ceph-mon[33098]: pgmap v791: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.4 MiB/s wr, 360 op/s 2024-04-25T02:19:32.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:32 smithi031 ceph-mon[29204]: pgmap v791: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.4 MiB/s wr, 360 op/s 2024-04-25T02:19:33.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 114059 8.81 MB/sec execute 577 sec latency 320.933 ms 2024-04-25T02:19:34.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 114185 8.80 MB/sec execute 578 sec latency 86.804 ms 2024-04-25T02:19:34.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:34 smithi040 ceph-mon[32418]: pgmap v792: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.4 MiB/s wr, 248 op/s 2024-04-25T02:19:34.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:34 smithi134 ceph-mon[33098]: pgmap v792: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.4 MiB/s wr, 248 op/s 2024-04-25T02:19:34.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:34 smithi031 ceph-mon[29204]: pgmap v792: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.4 MiB/s wr, 248 op/s 2024-04-25T02:19:35.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 114339 8.80 MB/sec execute 579 sec latency 64.808 ms 2024-04-25T02:19:36.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 114507 8.81 MB/sec execute 580 sec latency 96.322 ms 2024-04-25T02:19:36.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:36 smithi040 ceph-mon[32418]: pgmap v793: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.4 MiB/s wr, 357 op/s 2024-04-25T02:19:36.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:36 smithi134 ceph-mon[33098]: pgmap v793: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.4 MiB/s wr, 357 op/s 2024-04-25T02:19:36.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:36 smithi031 ceph-mon[29204]: pgmap v793: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 8.4 MiB/s wr, 357 op/s 2024-04-25T02:19:37.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 114644 8.80 MB/sec execute 581 sec latency 83.241 ms 2024-04-25T02:19:38.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 114792 8.79 MB/sec execute 582 sec latency 88.424 ms 2024-04-25T02:19:38.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:38 smithi040 ceph-mon[32418]: pgmap v794: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 230 op/s 2024-04-25T02:19:38.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:38 smithi134 ceph-mon[33098]: pgmap v794: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 230 op/s 2024-04-25T02:19:38.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:38 smithi031 ceph-mon[29204]: pgmap v794: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 230 op/s 2024-04-25T02:19:39.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 114936 8.79 MB/sec execute 583 sec latency 73.345 ms 2024-04-25T02:19:40.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 115103 8.79 MB/sec execute 584 sec latency 62.774 ms 2024-04-25T02:19:40.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:40 smithi040 ceph-mon[32418]: pgmap v795: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 230 op/s 2024-04-25T02:19:40.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:40 smithi134 ceph-mon[33098]: pgmap v795: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 230 op/s 2024-04-25T02:19:40.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:40 smithi031 ceph-mon[29204]: pgmap v795: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 230 op/s 2024-04-25T02:19:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.689+0000 7f2f65c7d640 1 -- 172.21.15.31:0/579332241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f60106630 msgr2=0x7f2f60106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.689+0000 7f2f65c7d640 1 --2- 172.21.15.31:0/579332241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f60106630 0x7f2f60106a10 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009930 tx=0x7f2f5002a420 comp rx=0 tx=0).stop 2024-04-25T02:19:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.689+0000 7f2f65c7d640 1 -- 172.21.15.31:0/579332241 shutdown_connections 2024-04-25T02:19:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.689+0000 7f2f65c7d640 1 --2- 172.21.15.31:0/579332241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f60101030 0x7f2f6010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.689+0000 7f2f65c7d640 1 --2- 172.21.15.31:0/579332241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f60100670 0x7f2f60100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.689+0000 7f2f65c7d640 1 --2- 172.21.15.31:0/579332241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f60106630 0x7f2f60106a10 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.689+0000 7f2f65c7d640 1 -- 172.21.15.31:0/579332241 >> 172.21.15.31:0/579332241 conn(0x7f2f600777c0 msgr2=0x7f2f60077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.689+0000 7f2f65c7d640 1 -- 172.21.15.31:0/579332241 shutdown_connections 2024-04-25T02:19:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.689+0000 7f2f65c7d640 1 -- 172.21.15.31:0/579332241 wait complete. 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f65c7d640 1 Processor -- start 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f65c7d640 1 -- start start 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f65c7d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f60100670 0x7f2f60197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f65c7d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f60101030 0x7f2f601983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f65c7d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f60106630 0x7f2f6019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f65c7d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2f601108f0 con 0x7f2f60106630 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f65c7d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f60110770 con 0x7f2f60100670 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f65c7d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2f60110a70 con 0x7f2f60101030 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f5f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f60100670 0x7f2f60197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f5effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f60101030 0x7f2f601983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f5f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f60100670 0x7f2f60197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38270/0 (socket says 172.21.15.31:38270) 2024-04-25T02:19:40.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f5f7fe640 1 -- 172.21.15.31:0/3856679748 learned_addr learned my addr 172.21.15.31:0/3856679748 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.690+0000 7f2f5ffff640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f60106630 0x7f2f6019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.691+0000 7f2f5effd640 1 -- 172.21.15.31:0/3856679748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f60100670 msgr2=0x7f2f60197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.691+0000 7f2f5effd640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f60100670 0x7f2f60197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.691+0000 7f2f5effd640 1 -- 172.21.15.31:0/3856679748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f60106630 msgr2=0x7f2f6019c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.691+0000 7f2f5effd640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f60106630 0x7f2f6019c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.691+0000 7f2f5effd640 1 -- 172.21.15.31:0/3856679748 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f6019ce60 con 0x7f2f60101030 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.691+0000 7f2f5ffff640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f60106630 0x7f2f6019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.691+0000 7f2f5effd640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f60101030 0x7f2f601983b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400e9f0 tx=0x7f2f5400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.691+0000 7f2f5cff9640 1 -- 172.21.15.31:0/3856679748 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5400ccc0 con 0x7f2f60101030 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.691+0000 7f2f5cff9640 1 -- 172.21.15.31:0/3856679748 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f540108e0 con 0x7f2f60101030 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.691+0000 7f2f65c7d640 1 -- 172.21.15.31:0/3856679748 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f6019d150 con 0x7f2f60101030 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.691+0000 7f2f5cff9640 1 -- 172.21.15.31:0/3856679748 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f54018880 con 0x7f2f60101030 2024-04-25T02:19:40.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.692+0000 7f2f65c7d640 1 -- 172.21.15.31:0/3856679748 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f601a49e0 con 0x7f2f60101030 2024-04-25T02:19:40.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f6898c740 1 Processor -- start 2024-04-25T02:19:40.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f6898c740 1 -- start start 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f5cff9640 1 -- 172.21.15.31:0/3856679748 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2f54010a90 con 0x7f2f60101030 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cabaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cac5640 0x55f50cac7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f6898c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f50ca1acb0 con 0x55f50cabaf70 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f6898c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f50c9c35b0 con 0x55f50caba650 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f6898c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f50ca16c70 con 0x55f50cac5640 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f5ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cabaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f5effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cac5640 0x55f50cac7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f5f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f5ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cabaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38282/0 (socket says 172.21.15.31:38282) 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f5ffff640 1 -- 172.21.15.31:0/2295384960 learned_addr learned my addr 172.21.15.31:0/2295384960 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:40.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.695+0000 7f2f5ffff640 1 -- 172.21.15.31:0/2295384960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cac5640 msgr2=0x55f50cac7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f5ffff640 1 --2- 172.21.15.31:0/2295384960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cac5640 0x55f50cac7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f5ffff640 1 -- 172.21.15.31:0/2295384960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 msgr2=0x55f50cac5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f5ffff640 1 --2- 172.21.15.31:0/2295384960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f5ffff640 1 -- 172.21.15.31:0/2295384960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f50c72f210 con 0x55f50caba650 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f5effd640 1 --2- 172.21.15.31:0/2295384960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cac5640 0x55f50cac7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f5f7fe640 1 --2- 172.21.15.31:0/2295384960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f5cff9640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2f480a2e20 0x7f2f480a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f5ffff640 1 --2- 172.21.15.31:0/2295384960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cabaa30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2f58011790 tx=0x7f2f58011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6067a9f0ea22a93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f5f7fe640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2f480a2e20 0x7f2f480a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f5cff9640 1 -- 172.21.15.31:0/3856679748 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f2f540cf6c0 con 0x7f2f60101030 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f45ffb640 1 -- 172.21.15.31:0/2295384960 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58003c30 con 0x55f50caba650 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f45ffb640 1 -- 172.21.15.31:0/2295384960 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f5800fe50 con 0x55f50caba650 2024-04-25T02:19:40.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.696+0000 7f2f45ffb640 1 -- 172.21.15.31:0/2295384960 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f580193e0 con 0x55f50caba650 2024-04-25T02:19:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.697+0000 7f2f6898c740 1 -- 172.21.15.31:0/2295384960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 msgr2=0x55f50cabaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.697+0000 7f2f5f7fe640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2f480a2e20 0x7f2f480a52e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2f50000c00 tx=0x7f2f500058d0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.697+0000 7f2f6898c740 1 --2- 172.21.15.31:0/2295384960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cabaa30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2f58011790 tx=0x7f2f58011c50 comp rx=0 tx=0).stop 2024-04-25T02:19:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.697+0000 7f2f6898c740 1 -- 172.21.15.31:0/2295384960 shutdown_connections 2024-04-25T02:19:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.697+0000 7f2f6898c740 1 --2- 172.21.15.31:0/2295384960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cac5640 0x55f50cac7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.697+0000 7f2f6898c740 1 --2- 172.21.15.31:0/2295384960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.697+0000 7f2f6898c740 1 --2- 172.21.15.31:0/2295384960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cabaa30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.697+0000 7f2f6898c740 1 -- 172.21.15.31:0/2295384960 >> 172.21.15.31:0/2295384960 conn(0x55f50cab6970 msgr2=0x55f50cab6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.697+0000 7f2f6898c740 1 -- 172.21.15.31:0/2295384960 shutdown_connections 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.697+0000 7f2f6898c740 1 -- 172.21.15.31:0/2295384960 wait complete. 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f6898c740 1 -- start start 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f6898c740 1 Processor -- start 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50caba650 0x55f50cac2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50cac5640 0x55f50cac3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f6898c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f50cac8430 con 0x55f50cabaf70 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f6898c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f50c9d6b50 con 0x55f50cac5640 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f6898c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f50cac85b0 con 0x55f50caba650 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f6898c740 10 client.0 _ll_register_callbacks cb 0x55f50ca11ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f5effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f5ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50cac5640 0x55f50cac3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f5effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35414/0 (socket says 172.21.15.31:35414) 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f5effd640 1 -- 172.21.15.31:0/4177143447 learned_addr learned my addr 172.21.15.31:0/4177143447 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:40.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.698+0000 7f2f5f7fe640 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50caba650 0x55f50cac2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f5ffff640 1 -- 172.21.15.31:0/4177143447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50caba650 msgr2=0x55f50cac2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:40.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f5ffff640 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50caba650 0x55f50cac2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f5ffff640 1 -- 172.21.15.31:0/4177143447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 msgr2=0x55f50cac2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:40.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f5ffff640 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f5ffff640 1 -- 172.21.15.31:0/4177143447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f50c7fb620 con 0x55f50cac5640 2024-04-25T02:19:40.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f5f7fe640 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50caba650 0x55f50cac2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:40.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f5effd640 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:40.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f5ffff640 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50cac5640 0x55f50cac3110 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2f58006010 tx=0x7f2f58004350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4b37db51538bbdd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:40.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f3effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:19:40.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f3effd640 1 -- 172.21.15.31:0/4177143447 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5802a060 con 0x55f50cac5640 2024-04-25T02:19:40.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f6898c740 1 -- 172.21.15.31:0/4177143447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f5800fe50 con 0x55f50cac5640 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f3effd640 1 -- 172.21.15.31:0/4177143447 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f2f58011040 con 0x55f50cac5640 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f3effd640 1 -- 172.21.15.31:0/4177143447 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f2f5801f040 con 0x55f50cac5640 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f6898c740 10 client.27710 fetch_fsmap learned FSMap version 51 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.699+0000 7f2f6898c740 1 -- 172.21.15.31:0/4177143447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55f50cad4740 con 0x55f50cac5640 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.700+0000 7f2f3effd640 1 -- 172.21.15.31:0/4177143447 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f5800fe50 con 0x55f50cac5640 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.700+0000 7f2f3effd640 1 -- 172.21.15.31:0/4177143447 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f2f58018070 con 0x55f50cac5640 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.700+0000 7f2f6898c740 10 client.27710 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.700+0000 7f2f6898c740 10 client.27710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.700+0000 7f2f6898c740 20 client.27710 populate_metadata read hostname 'smithi031' 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.700+0000 7f2f6898c740 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f50cad5ba0 0x55f50cad7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.700+0000 7f2f6898c740 4 client.27710 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.700+0000 7f2f6898c740 1 -- 172.21.15.31:0/4177143447 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f50ca281f0 con 0x55f50cad5ba0 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.700+0000 7f2f5f7fe640 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f50cad5ba0 0x55f50cad7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.701+0000 7f2f5f7fe640 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f50cad5ba0 0x55f50cad7f90 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6134126bbeb365a server_cookie=5966384e265dd47e in_seq=0 out_seq=0 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.701+0000 7f2f3effd640 10 client.27710 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:40.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.701+0000 7f2f3effd640 1 -- 172.21.15.31:0/4177143447 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f50ca281f0 con 0x55f50cad5ba0 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.701+0000 7f2f3effd640 10 client.27710 handle_command_reply: tid=0 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.701+0000 7f2f6898c740 1 client.27710 shutdown 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.701+0000 7f2f6898c740 1 -- 172.21.15.31:0/4177143447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50cac5640 msgr2=0x55f50cac3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.701+0000 7f2f6898c740 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50cac5640 0x55f50cac3110 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2f58006010 tx=0x7f2f58004350 comp rx=0 tx=0).stop 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.702+0000 7f2f6898c740 1 -- 172.21.15.31:0/4177143447 shutdown_connections 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.702+0000 7f2f6898c740 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f50cad5ba0 0x55f50cad7f90 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.702+0000 7f2f6898c740 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50cac5640 0x55f50cac3110 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.702+0000 7f2f6898c740 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.702+0000 7f2f6898c740 1 --2- 172.21.15.31:0/4177143447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50caba650 0x55f50cac2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.702+0000 7f2f6898c740 1 -- 172.21.15.31:0/4177143447 >> 172.21.15.31:0/4177143447 conn(0x55f50cab6970 msgr2=0x55f50cabc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.702+0000 7f2f3effd640 0 client.27710 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.702+0000 7f2f6898c740 1 -- 172.21.15.31:0/4177143447 shutdown_connections 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.702+0000 7f2f6898c740 1 -- 172.21.15.31:0/4177143447 wait complete. 2024-04-25T02:19:40.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.702+0000 7f2f6898c740 20 client.27710 trim_cache size 0 max 16384 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f6898c740 1 Processor -- start 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f6898c740 1 -- start start 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50caba650 0x55f50cad12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50cac5640 0x55f50cadd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f6898c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2f601aafc0 con 0x55f50cabaf70 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f6898c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f601acb00 con 0x55f50cac5640 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f6898c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2f601ab690 con 0x55f50caba650 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f5effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f5effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35424/0 (socket says 172.21.15.31:35424) 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f5effd640 1 -- 172.21.15.31:0/1699146773 learned_addr learned my addr 172.21.15.31:0/1699146773 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f5ffff640 1 --2- 172.21.15.31:0/1699146773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50cac5640 0x55f50cadd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.728+0000 7f2f5f7fe640 1 --2- 172.21.15.31:0/1699146773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50caba650 0x55f50cad12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.729+0000 7f2f5effd640 1 -- 172.21.15.31:0/1699146773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50caba650 msgr2=0x55f50cad12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.729+0000 7f2f5effd640 1 --2- 172.21.15.31:0/1699146773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50caba650 0x55f50cad12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.729+0000 7f2f5effd640 1 -- 172.21.15.31:0/1699146773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50cac5640 msgr2=0x55f50cadd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.729+0000 7f2f5effd640 1 --2- 172.21.15.31:0/1699146773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50cac5640 0x55f50cadd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.729+0000 7f2f5effd640 1 -- 172.21.15.31:0/1699146773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f50cac8310 con 0x55f50cabaf70 2024-04-25T02:19:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.729+0000 7f2f5effd640 1 --2- 172.21.15.31:0/1699146773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2f30 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f2f540d52f0 tx=0x7f2f54009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc7cc9919778753 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.729+0000 7f2f457fa640 1 -- 172.21.15.31:0/1699146773 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5409b7f0 con 0x55f50cabaf70 2024-04-25T02:19:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.729+0000 7f2f457fa640 1 -- 172.21.15.31:0/1699146773 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f54010e00 con 0x55f50cabaf70 2024-04-25T02:19:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.729+0000 7f2f457fa640 1 -- 172.21.15.31:0/1699146773 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5409b610 con 0x55f50cabaf70 2024-04-25T02:19:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.730+0000 7f2f6898c740 1 -- 172.21.15.31:0/1699146773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 msgr2=0x55f50cac2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.730+0000 7f2f6898c740 1 --2- 172.21.15.31:0/1699146773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2f30 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f2f540d52f0 tx=0x7f2f54009700 comp rx=0 tx=0).stop 2024-04-25T02:19:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.730+0000 7f2f6898c740 1 -- 172.21.15.31:0/1699146773 shutdown_connections 2024-04-25T02:19:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.730+0000 7f2f6898c740 1 --2- 172.21.15.31:0/1699146773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50cac5640 0x55f50cadd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.730+0000 7f2f6898c740 1 --2- 172.21.15.31:0/1699146773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cabaf70 0x55f50cac2f30 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.730+0000 7f2f6898c740 1 --2- 172.21.15.31:0/1699146773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50caba650 0x55f50cad12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.730+0000 7f2f6898c740 1 -- 172.21.15.31:0/1699146773 >> 172.21.15.31:0/1699146773 conn(0x55f50cab6970 msgr2=0x55f50cabc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.730+0000 7f2f6898c740 1 -- 172.21.15.31:0/1699146773 shutdown_connections 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.730+0000 7f2f6898c740 1 -- 172.21.15.31:0/1699146773 wait complete. 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.730+0000 7f2f6898c740 1 -- start start 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f6898c740 1 Processor -- start 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cacda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cabaf70 0x55f50cabeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f6898c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cac5640 0x55f50cabf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f6898c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2f601ac430 con 0x55f50cac5640 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f6898c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f601abd60 con 0x55f50caba650 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f6898c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2f601a8fc0 con 0x55f50cabaf70 2024-04-25T02:19:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f5f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cacda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f5effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cabaf70 0x55f50cabeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f6898c740 10 client.0 _ll_register_callbacks cb 0x7f2f580093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f5f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cacda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:38328/0 (socket says 172.21.15.31:38328) 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f5effd640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cabaf70 0x55f50cabeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:58796/0 (socket says 172.21.15.31:58796) 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f5f7fe640 1 -- 172.21.15.31:0/464474017 learned_addr learned my addr 172.21.15.31:0/464474017 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f5ffff640 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cac5640 0x55f50cabf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f5f7fe640 1 -- 172.21.15.31:0/464474017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cabaf70 msgr2=0x55f50cabeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f5f7fe640 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cabaf70 0x55f50cabeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f5f7fe640 1 -- 172.21.15.31:0/464474017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cac5640 msgr2=0x55f50cabf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f5f7fe640 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cac5640 0x55f50cabf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.731+0000 7f2f5f7fe640 1 -- 172.21.15.31:0/464474017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f50cabfa90 con 0x55f50caba650 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.732+0000 7f2f5effd640 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cabaf70 0x55f50cabeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.732+0000 7f2f5f7fe640 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cacda60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009a50 tx=0x7f2f50009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3fc5705c77c519c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.732+0000 7f2f3effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-25T02:19:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.732+0000 7f2f3effd640 1 -- 172.21.15.31:0/464474017 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f50035040 con 0x55f50caba650 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.732+0000 7f2f3effd640 1 -- 172.21.15.31:0/464474017 <== mon.2 v2:172.21.15.40:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f2f50054500 con 0x55f50caba650 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.732+0000 7f2f6898c740 1 -- 172.21.15.31:0/464474017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f50ca22540 con 0x55f50caba650 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.732+0000 7f2f3effd640 1 -- 172.21.15.31:0/464474017 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f2f50054700 con 0x55f50caba650 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.732+0000 7f2f6898c740 10 client.27716 fetch_fsmap learned FSMap version 51 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.732+0000 7f2f6898c740 1 -- 172.21.15.31:0/464474017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55f50cb3bdf0 con 0x55f50caba650 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.733+0000 7f2f3effd640 1 -- 172.21.15.31:0/464474017 <== mon.2 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f50ca22540 con 0x55f50caba650 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.733+0000 7f2f3effd640 1 -- 172.21.15.31:0/464474017 <== mon.2 v2:172.21.15.40:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f2f5003d4c0 con 0x55f50caba650 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.733+0000 7f2f6898c740 10 client.27716 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.733+0000 7f2f6898c740 10 client.27716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.733+0000 7f2f6898c740 20 client.27716 populate_metadata read hostname 'smithi031' 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.733+0000 7f2f6898c740 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f50cb3c520 0x55f50cb3e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.733+0000 7f2f6898c740 4 client.27716 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.733+0000 7f2f6898c740 1 -- 172.21.15.31:0/464474017 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f50c9a08e0 con 0x55f50cb3c520 2024-04-25T02:19:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.733+0000 7f2f5effd640 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f50cb3c520 0x55f50cb3e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.733+0000 7f2f5effd640 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f50cb3c520 0x55f50cb3e940 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fa28e1933b63d1a server_cookie=d133ab91e9beb548 in_seq=0 out_seq=0 2024-04-25T02:19:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.733+0000 7f2f3effd640 10 client.27716 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.840+0000 7f2f3effd640 1 -- 172.21.15.31:0/464474017 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f50c9a08e0 con 0x55f50cb3c520 2024-04-25T02:19:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.840+0000 7f2f3effd640 10 client.27716 handle_command_reply: tid=0 2024-04-25T02:19:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.840+0000 7f2f6898c740 1 client.27716 shutdown 2024-04-25T02:19:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f6898c740 1 -- 172.21.15.31:0/464474017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 msgr2=0x55f50cacda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f6898c740 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cacda60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009a50 tx=0x7f2f50009260 comp rx=0 tx=0).stop 2024-04-25T02:19:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f6898c740 1 -- 172.21.15.31:0/464474017 shutdown_connections 2024-04-25T02:19:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f6898c740 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x55f50cb3c520 0x55f50cb3e940 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f3effd640 0 client.27716 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f6898c740 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f50cac5640 0x55f50cabf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f6898c740 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f50cabaf70 0x55f50cabeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f6898c740 1 --2- 172.21.15.31:0/464474017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f50caba650 0x55f50cacda60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f6898c740 1 -- 172.21.15.31:0/464474017 >> 172.21.15.31:0/464474017 conn(0x55f50cab6970 msgr2=0x55f50cabbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f6898c740 1 -- 172.21.15.31:0/464474017 shutdown_connections 2024-04-25T02:19:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f6898c740 1 -- 172.21.15.31:0/464474017 wait complete. 2024-04-25T02:19:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.841+0000 7f2f6898c740 20 client.27716 trim_cache size 0 max 16384 2024-04-25T02:19:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:19:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:19:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:19:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 -- 172.21.15.31:0/3856679748 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2f480a2e20 msgr2=0x7f2f480a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2f480a2e20 0x7f2f480a52e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2f50000c00 tx=0x7f2f500058d0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 -- 172.21.15.31:0/3856679748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f60101030 msgr2=0x7f2f601983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f60101030 0x7f2f601983b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400e9f0 tx=0x7f2f5400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 -- 172.21.15.31:0/3856679748 shutdown_connections 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f2f480a2e20 0x7f2f480a52e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f60106630 0x7f2f6019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f60101030 0x7f2f601983b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 --2- 172.21.15.31:0/3856679748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f60100670 0x7f2f60197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 -- 172.21.15.31:0/3856679748 >> 172.21.15.31:0/3856679748 conn(0x7f2f600777c0 msgr2=0x7f2f600690d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 -- 172.21.15.31:0/3856679748 shutdown_connections 2024-04-25T02:19:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:40.842+0000 7f2f65c7d640 1 -- 172.21.15.31:0/3856679748 wait complete. 2024-04-25T02:19:41.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 115285 8.79 MB/sec execute 585 sec latency 115.280 ms 2024-04-25T02:19:42.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 115445 8.79 MB/sec execute 586 sec latency 177.948 ms 2024-04-25T02:19:42.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:42 smithi040 ceph-mon[32418]: pgmap v796: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.8 MiB/s wr, 349 op/s 2024-04-25T02:19:42.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:42 smithi134 ceph-mon[33098]: pgmap v796: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.8 MiB/s wr, 349 op/s 2024-04-25T02:19:42.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:42 smithi031 ceph-mon[29204]: pgmap v796: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 8.8 MiB/s wr, 349 op/s 2024-04-25T02:19:43.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 115583 8.78 MB/sec execute 587 sec latency 110.793 ms 2024-04-25T02:19:44.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 115763 8.78 MB/sec execute 588 sec latency 71.093 ms 2024-04-25T02:19:44.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:44 smithi040 ceph-mon[32418]: pgmap v797: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.9 MiB/s wr, 227 op/s 2024-04-25T02:19:44.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:44 smithi134 ceph-mon[33098]: pgmap v797: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.9 MiB/s wr, 227 op/s 2024-04-25T02:19:44.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:44 smithi031 ceph-mon[29204]: pgmap v797: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.9 MiB/s wr, 227 op/s 2024-04-25T02:19:45.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 115943 8.78 MB/sec execute 589 sec latency 83.405 ms 2024-04-25T02:19:46.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 116123 8.78 MB/sec execute 590 sec latency 56.298 ms 2024-04-25T02:19:46.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:46 smithi040 ceph-mon[32418]: pgmap v798: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.3 MiB/s wr, 362 op/s 2024-04-25T02:19:46.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:46 smithi134 ceph-mon[33098]: pgmap v798: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.3 MiB/s wr, 362 op/s 2024-04-25T02:19:46.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:46 smithi031 ceph-mon[29204]: pgmap v798: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 8.3 MiB/s wr, 362 op/s 2024-04-25T02:19:47.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 116277 8.78 MB/sec execute 591 sec latency 57.799 ms 2024-04-25T02:19:48.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 116421 8.77 MB/sec execute 592 sec latency 75.958 ms 2024-04-25T02:19:48.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:48 smithi040 ceph-mon[32418]: pgmap v799: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.3 MiB/s wr, 252 op/s 2024-04-25T02:19:48.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:48 smithi134 ceph-mon[33098]: pgmap v799: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.3 MiB/s wr, 252 op/s 2024-04-25T02:19:48.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:48 smithi031 ceph-mon[29204]: pgmap v799: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.3 MiB/s wr, 252 op/s 2024-04-25T02:19:49.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 116627 8.79 MB/sec execute 593 sec latency 105.936 ms 2024-04-25T02:19:50.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 116824 8.79 MB/sec execute 594 sec latency 69.488 ms 2024-04-25T02:19:50.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:50 smithi040 ceph-mon[32418]: pgmap v800: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.3 MiB/s wr, 252 op/s 2024-04-25T02:19:50.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:50 smithi134 ceph-mon[33098]: pgmap v800: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.3 MiB/s wr, 252 op/s 2024-04-25T02:19:50.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:50 smithi031 ceph-mon[29204]: pgmap v800: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.3 MiB/s wr, 252 op/s 2024-04-25T02:19:51.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 116980 8.79 MB/sec execute 595 sec latency 58.734 ms 2024-04-25T02:19:52.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 117106 8.79 MB/sec execute 596 sec latency 207.891 ms 2024-04-25T02:19:52.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:52 smithi040 ceph-mon[32418]: pgmap v801: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 8.1 MiB/s wr, 363 op/s 2024-04-25T02:19:52.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:52 smithi134 ceph-mon[33098]: pgmap v801: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 8.1 MiB/s wr, 363 op/s 2024-04-25T02:19:52.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:52 smithi031 ceph-mon[29204]: pgmap v801: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 8.1 MiB/s wr, 363 op/s 2024-04-25T02:19:53.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 117313 8.81 MB/sec execute 597 sec latency 90.968 ms 2024-04-25T02:19:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.976+0000 7f775cf63640 1 -- 172.21.15.31:0/105439869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7758073ae0 msgr2=0x7f7758073ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.976+0000 7f775cf63640 1 --2- 172.21.15.31:0/105439869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7758073ae0 0x7f7758073ec0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f7748009860 tx=0x7f774802a3d0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.976+0000 7f775cf63640 1 -- 172.21.15.31:0/105439869 shutdown_connections 2024-04-25T02:19:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.976+0000 7f775cf63640 1 --2- 172.21.15.31:0/105439869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7758112740 0x7f7758114b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.976+0000 7f775cf63640 1 --2- 172.21.15.31:0/105439869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758074400 0x7f7758112200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.976+0000 7f775cf63640 1 --2- 172.21.15.31:0/105439869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7758073ae0 0x7f7758073ec0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.976+0000 7f775cf63640 1 -- 172.21.15.31:0/105439869 >> 172.21.15.31:0/105439869 conn(0x7f77580fd200 msgr2=0x7f77580ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.976+0000 7f775cf63640 1 -- 172.21.15.31:0/105439869 shutdown_connections 2024-04-25T02:19:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.976+0000 7f775cf63640 1 -- 172.21.15.31:0/105439869 wait complete. 2024-04-25T02:19:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f775cf63640 1 Processor -- start 2024-04-25T02:19:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f775cf63640 1 -- start start 2024-04-25T02:19:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f775cf63640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7758073ae0 0x7f775819e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f775cf63640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7758074400 0x7f775819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f775cf63640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758112740 0x7f77581a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f775cf63640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7758117fc0 con 0x7f7758074400 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f775cf63640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7758117e40 con 0x7f7758112740 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f775cf63640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7758118140 con 0x7f7758073ae0 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f7755d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7758074400 0x7f775819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f775659c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7758073ae0 0x7f775819e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f7756d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758112740 0x7f77581a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f775659c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7758073ae0 0x7f775819e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.31:59690/0 (socket says 172.21.15.31:59690) 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.977+0000 7f775659c640 1 -- 172.21.15.31:0/2931482489 learned_addr learned my addr 172.21.15.31:0/2931482489 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f7755d9b640 1 -- 172.21.15.31:0/2931482489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7758073ae0 msgr2=0x7f775819e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f7755d9b640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7758073ae0 0x7f775819e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f7755d9b640 1 -- 172.21.15.31:0/2931482489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758112740 msgr2=0x7f77581a2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f7755d9b640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758112740 0x7f77581a2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f7755d9b640 1 -- 172.21.15.31:0/2931482489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77581a3520 con 0x7f7758074400 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f7756d9d640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758112740 0x7f77581a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f7755d9b640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7758074400 0x7f775819eac0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f774c00e790 tx=0x7f774c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f77477fe640 1 -- 172.21.15.31:0/2931482489 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774c00cab0 con 0x7f7758074400 2024-04-25T02:19:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f775cf63640 1 -- 172.21.15.31:0/2931482489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77581a3810 con 0x7f7758074400 2024-04-25T02:19:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f77477fe640 1 -- 172.21.15.31:0/2931482489 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f774c00cc50 con 0x7f7758074400 2024-04-25T02:19:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f775cf63640 1 -- 172.21.15.31:0/2931482489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77581ab0a0 con 0x7f7758074400 2024-04-25T02:19:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.978+0000 7f77477fe640 1 -- 172.21.15.31:0/2931482489 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774c018680 con 0x7f7758074400 2024-04-25T02:19:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.981+0000 7f775fc73740 1 Processor -- start 2024-04-25T02:19:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f775fc73740 1 -- start start 2024-04-25T02:19:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f775fc73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e232620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f775fc73740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e23ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f775fc73740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e23f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f775fc73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559c6e196cb0 con 0x559c6e232240 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f775fc73740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559c6e13f5b0 con 0x559c6e232b60 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f775fc73740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559c6e192c70 con 0x559c6e23d230 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f77477fe640 1 -- 172.21.15.31:0/2931482489 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f774c010430 con 0x7f7758074400 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f7756d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e232620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f7756d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e232620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33262/0 (socket says 172.21.15.31:33262) 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f7756d9d640 1 -- 172.21.15.31:0/3043222263 learned_addr learned my addr 172.21.15.31:0/3043222263 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f7755d9b640 1 --2- 172.21.15.31:0/3043222263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e23f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f775659c640 1 --2- 172.21.15.31:0/3043222263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e23ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f7756d9d640 1 -- 172.21.15.31:0/3043222263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 msgr2=0x559c6e23f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f7756d9d640 1 --2- 172.21.15.31:0/3043222263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e23f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f7756d9d640 1 -- 172.21.15.31:0/3043222263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 msgr2=0x559c6e23ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f7756d9d640 1 --2- 172.21.15.31:0/3043222263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e23ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.982+0000 7f7756d9d640 1 -- 172.21.15.31:0/3043222263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c6deab210 con 0x559c6e232240 2024-04-25T02:19:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.983+0000 7f7756d9d640 1 --2- 172.21.15.31:0/3043222263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e232620 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f7750011790 tx=0x7f7750011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe45670d03d9480 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.983+0000 7f7744ff9640 1 -- 172.21.15.31:0/3043222263 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775000fc60 con 0x559c6e232240 2024-04-25T02:19:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.983+0000 7f7744ff9640 1 -- 172.21.15.31:0/3043222263 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7750003e20 con 0x559c6e232240 2024-04-25T02:19:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.983+0000 7f77477fe640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f77400a2fa0 0x7f77400a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.983+0000 7f775659c640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f77400a2fa0 0x7f77400a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.983+0000 7f77477fe640 1 -- 172.21.15.31:0/2931482489 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f774c018420 con 0x7f7758074400 2024-04-25T02:19:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.984+0000 7f7744ff9640 1 -- 172.21.15.31:0/3043222263 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7750019430 con 0x559c6e232240 2024-04-25T02:19:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.984+0000 7f775659c640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f77400a2fa0 0x7f77400a5460 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7748032ff0 tx=0x7f77480095b0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.984+0000 7f775fc73740 1 -- 172.21.15.31:0/3043222263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 msgr2=0x559c6e232620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.984+0000 7f775fc73740 1 --2- 172.21.15.31:0/3043222263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e232620 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f7750011790 tx=0x7f7750011c50 comp rx=0 tx=0).stop 2024-04-25T02:19:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.984+0000 7f775fc73740 1 -- 172.21.15.31:0/3043222263 shutdown_connections 2024-04-25T02:19:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.984+0000 7f775fc73740 1 --2- 172.21.15.31:0/3043222263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e23f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.984+0000 7f775fc73740 1 --2- 172.21.15.31:0/3043222263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e23ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.984+0000 7f775fc73740 1 --2- 172.21.15.31:0/3043222263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e232620 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.984+0000 7f775fc73740 1 -- 172.21.15.31:0/3043222263 >> 172.21.15.31:0/3043222263 conn(0x559c6e22e560 msgr2=0x559c6e22e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.984+0000 7f775fc73740 1 -- 172.21.15.31:0/3043222263 shutdown_connections 2024-04-25T02:19:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.984+0000 7f775fc73740 1 -- 172.21.15.31:0/3043222263 wait complete. 2024-04-25T02:19:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.985+0000 7f775fc73740 1 -- start start 2024-04-25T02:19:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.985+0000 7f775fc73740 1 Processor -- start 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775fc73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e23a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775fc73740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e23a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775fc73740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e23ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775fc73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559c6e2400c0 con 0x559c6e232240 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775fc73740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559c6e152b50 con 0x559c6e232b60 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775fc73740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559c6e240240 con 0x559c6e23d230 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e23a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f7755d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e23a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e23a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33270/0 (socket says 172.21.15.31:33270) 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775659c640 1 -- 172.21.15.31:0/4223539066 learned_addr learned my addr 172.21.15.31:0/4223539066 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f7756d9d640 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e23ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775fc73740 10 client.0 _ll_register_callbacks cb 0x559c6e18dca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775659c640 1 -- 172.21.15.31:0/4223539066 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 msgr2=0x559c6e23ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775659c640 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e23ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775659c640 1 -- 172.21.15.31:0/4223539066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 msgr2=0x559c6e23a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775659c640 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e23a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.986+0000 7f775659c640 1 -- 172.21.15.31:0/4223539066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c6df77620 con 0x559c6e232240 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.987+0000 7f775659c640 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e23a0c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7748009980 tx=0x7f774802a3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca91dc5cb1f06883 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.987+0000 7f773dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.987+0000 7f773dffb640 1 -- 172.21.15.31:0/4223539066 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7748005040 con 0x559c6e232240 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.987+0000 7f773dffb640 1 -- 172.21.15.31:0/4223539066 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f77480051e0 con 0x559c6e232240 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.987+0000 7f775fc73740 1 -- 172.21.15.31:0/4223539066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7750003e20 con 0x559c6e232240 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.987+0000 7f773dffb640 1 -- 172.21.15.31:0/4223539066 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f77480053e0 con 0x559c6e232240 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.987+0000 7f775fc73740 10 client.18162 fetch_fsmap learned FSMap version 51 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.987+0000 7f775fc73740 1 -- 172.21.15.31:0/4223539066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x559c6e24c250 con 0x559c6e232240 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.987+0000 7f773dffb640 1 -- 172.21.15.31:0/4223539066 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7750003e20 con 0x559c6e232240 2024-04-25T02:19:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.988+0000 7f773dffb640 1 -- 172.21.15.31:0/4223539066 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f774805b470 con 0x559c6e232240 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.988+0000 7f775fc73740 10 client.18162 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.988+0000 7f775fc73740 10 client.18162 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.988+0000 7f775fc73740 20 client.18162 populate_metadata read hostname 'smithi031' 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.988+0000 7f775fc73740 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c6e24dad0 0x559c6e24ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.988+0000 7f775fc73740 4 client.18162 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.988+0000 7f775fc73740 1 -- 172.21.15.31:0/4223539066 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c6e1a41f0 con 0x559c6e24dad0 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.988+0000 7f7755d9b640 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c6e24dad0 0x559c6e24ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.988+0000 7f7755d9b640 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c6e24dad0 0x559c6e24ff90 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7d961300195055c server_cookie=b319da0b800faeba in_seq=0 out_seq=0 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.988+0000 7f773dffb640 10 client.18162 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.989+0000 7f773dffb640 1 -- 172.21.15.31:0/4223539066 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559c6e1a41f0 con 0x559c6e24dad0 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.989+0000 7f773dffb640 10 client.18162 handle_command_reply: tid=0 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.989+0000 7f775fc73740 1 client.18162 shutdown 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.989+0000 7f775fc73740 1 -- 172.21.15.31:0/4223539066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 msgr2=0x559c6e23a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.989+0000 7f775fc73740 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e23a0c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7748009980 tx=0x7f774802a3d0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.990+0000 7f775fc73740 1 -- 172.21.15.31:0/4223539066 shutdown_connections 2024-04-25T02:19:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.990+0000 7f775fc73740 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c6e24dad0 0x559c6e24ff90 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.990+0000 7f775fc73740 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e23ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.990+0000 7f775fc73740 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e23a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.990+0000 7f775fc73740 1 --2- 172.21.15.31:0/4223539066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e23a0c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.990+0000 7f773dffb640 0 client.18162 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.990+0000 7f775fc73740 1 -- 172.21.15.31:0/4223539066 >> 172.21.15.31:0/4223539066 conn(0x559c6e22e560 msgr2=0x559c6e234260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.990+0000 7f775fc73740 1 -- 172.21.15.31:0/4223539066 shutdown_connections 2024-04-25T02:19:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.990+0000 7f775fc73740 1 -- 172.21.15.31:0/4223539066 wait complete. 2024-04-25T02:19:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:53.990+0000 7f775fc73740 20 client.18162 trim_cache size 0 max 16384 2024-04-25T02:19:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f775fc73740 1 Processor -- start 2024-04-25T02:19:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f775fc73740 1 -- start start 2024-04-25T02:19:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f775fc73740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232240 0x559c6e249030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f775fc73740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e232b60 0x559c6e239260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f775fc73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e23d230 0x559c6e2558a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f775fc73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f77581b3a10 con 0x559c6e23d230 2024-04-25T02:19:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f775fc73740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f77581b5550 con 0x559c6e232240 2024-04-25T02:19:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f775fc73740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f77581b40e0 con 0x559c6e232b60 2024-04-25T02:19:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f7756d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e23d230 0x559c6e2558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f7756d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e23d230 0x559c6e2558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33276/0 (socket says 172.21.15.31:33276) 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f7756d9d640 1 -- 172.21.15.31:0/3127984241 learned_addr learned my addr 172.21.15.31:0/3127984241 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f7755d9b640 1 --2- 172.21.15.31:0/3127984241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e232b60 0x559c6e239260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f775659c640 1 --2- 172.21.15.31:0/3127984241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232240 0x559c6e249030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f7756d9d640 1 -- 172.21.15.31:0/3127984241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e232b60 msgr2=0x559c6e239260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f7756d9d640 1 --2- 172.21.15.31:0/3127984241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e232b60 0x559c6e239260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f7756d9d640 1 -- 172.21.15.31:0/3127984241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232240 msgr2=0x559c6e249030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f7756d9d640 1 --2- 172.21.15.31:0/3127984241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232240 0x559c6e249030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.010+0000 7f7756d9d640 1 -- 172.21.15.31:0/3127984241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c6e232950 con 0x559c6e23d230 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.011+0000 7f7756d9d640 1 --2- 172.21.15.31:0/3127984241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e23d230 0x559c6e2558a0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7750005c40 tx=0x7f77500111d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b33b42cd22786c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.011+0000 7f773ffff640 1 -- 172.21.15.31:0/3127984241 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7750029070 con 0x559c6e23d230 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.011+0000 7f773ffff640 1 -- 172.21.15.31:0/3127984241 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77500113c0 con 0x559c6e23d230 2024-04-25T02:19:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.011+0000 7f773ffff640 1 -- 172.21.15.31:0/3127984241 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77500053a0 con 0x559c6e23d230 2024-04-25T02:19:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.012+0000 7f775fc73740 1 -- 172.21.15.31:0/3127984241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e23d230 msgr2=0x559c6e2558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.012+0000 7f775fc73740 1 --2- 172.21.15.31:0/3127984241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e23d230 0x559c6e2558a0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7750005c40 tx=0x7f77500111d0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.012+0000 7f775fc73740 1 -- 172.21.15.31:0/3127984241 shutdown_connections 2024-04-25T02:19:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.012+0000 7f775fc73740 1 --2- 172.21.15.31:0/3127984241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e23d230 0x559c6e2558a0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.012+0000 7f775fc73740 1 --2- 172.21.15.31:0/3127984241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e232b60 0x559c6e239260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.012+0000 7f775fc73740 1 --2- 172.21.15.31:0/3127984241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232240 0x559c6e249030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.012+0000 7f775fc73740 1 -- 172.21.15.31:0/3127984241 >> 172.21.15.31:0/3127984241 conn(0x559c6e22e560 msgr2=0x559c6e232f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.012+0000 7f775fc73740 1 -- 172.21.15.31:0/3127984241 shutdown_connections 2024-04-25T02:19:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.012+0000 7f775fc73740 1 -- 172.21.15.31:0/3127984241 wait complete. 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.012+0000 7f775fc73740 1 -- start start 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775fc73740 1 Processor -- start 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775fc73740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e245730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775fc73740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e236970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775fc73740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e236eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775fc73740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f77581b4e80 con 0x559c6e232240 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775fc73740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f77581b47b0 con 0x559c6e232b60 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775fc73740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f77581b1a10 con 0x559c6e23d230 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e245730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775fc73740 10 client.0 _ll_register_callbacks cb 0x7f77480096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e245730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33282/0 (socket says 172.21.15.31:33282) 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775659c640 1 -- 172.21.15.31:0/4096779533 learned_addr learned my addr 172.21.15.31:0/4096779533 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f7755d9b640 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e236970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f7756d9d640 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e236eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775659c640 1 -- 172.21.15.31:0/4096779533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 msgr2=0x559c6e236eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775659c640 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e236eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775659c640 1 -- 172.21.15.31:0/4096779533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 msgr2=0x559c6e236970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775659c640 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e236970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.013+0000 7f775659c640 1 -- 172.21.15.31:0/4096779533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c6e237470 con 0x559c6e232240 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.014+0000 7f775659c640 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e245730 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7748009830 tx=0x7f7748030ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9efcacc1438a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.014+0000 7f773dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.014+0000 7f773dffb640 1 -- 172.21.15.31:0/4096779533 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774805bc80 con 0x559c6e232240 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.014+0000 7f773dffb640 1 -- 172.21.15.31:0/4096779533 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f7748064450 con 0x559c6e232240 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.014+0000 7f775fc73740 1 -- 172.21.15.31:0/4096779533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77500113c0 con 0x559c6e232240 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.014+0000 7f773dffb640 1 -- 172.21.15.31:0/4096779533 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f7748064650 con 0x559c6e232240 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.014+0000 7f775fc73740 10 client.18168 fetch_fsmap learned FSMap version 51 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.014+0000 7f775fc73740 1 -- 172.21.15.31:0/4096779533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x559c6e2b3ed0 con 0x559c6e232240 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.014+0000 7f773dffb640 1 -- 172.21.15.31:0/4096779533 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77500113c0 con 0x559c6e232240 2024-04-25T02:19:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.015+0000 7f773dffb640 1 -- 172.21.15.31:0/4096779533 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f7748051810 con 0x559c6e232240 2024-04-25T02:19:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.015+0000 7f775fc73740 10 client.18168 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:19:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.015+0000 7f775fc73740 10 client.18168 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:19:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.015+0000 7f775fc73740 20 client.18168 populate_metadata read hostname 'smithi031' 2024-04-25T02:19:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.015+0000 7f775fc73740 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c6e2b4650 0x559c6e2b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:19:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.015+0000 7f775fc73740 4 client.18168 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:19:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.015+0000 7f775fc73740 1 -- 172.21.15.31:0/4096779533 --> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559c6e240270 con 0x559c6e2b4650 2024-04-25T02:19:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.015+0000 7f7755d9b640 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c6e2b4650 0x559c6e2b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:19:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.015+0000 7f773dffb640 10 client.18168 ms_handle_connect on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.015+0000 7f7755d9b640 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c6e2b4650 0x559c6e2b6a70 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86fbae46cef0c9e1 server_cookie=67d5257d15cd7fe2 in_seq=0 out_seq=0 2024-04-25T02:19:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f773dffb640 1 -- 172.21.15.31:0/4096779533 <== mds.0 v2:172.21.15.40:6838/3236110559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559c6e240270 con 0x559c6e2b4650 2024-04-25T02:19:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f773dffb640 10 client.18168 handle_command_reply: tid=0 2024-04-25T02:19:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f775fc73740 1 client.18168 shutdown 2024-04-25T02:19:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f775fc73740 1 -- 172.21.15.31:0/4096779533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 msgr2=0x559c6e245730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f775fc73740 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e245730 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7748009830 tx=0x7f7748030ef0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f775fc73740 1 -- 172.21.15.31:0/4096779533 shutdown_connections 2024-04-25T02:19:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f775fc73740 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.40:6838/3236110559,v1:172.21.15.40:6839/3236110559] conn(0x559c6e2b4650 0x559c6e2b6a70 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f775fc73740 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559c6e23d230 0x559c6e236eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f775fc73740 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c6e232b60 0x559c6e236970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f773dffb640 0 client.18168 ms_handle_reset on v2:172.21.15.40:6838/3236110559 2024-04-25T02:19:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f775fc73740 1 --2- 172.21.15.31:0/4096779533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6e232240 0x559c6e245730 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.091+0000 7f775fc73740 1 -- 172.21.15.31:0/4096779533 >> 172.21.15.31:0/4096779533 conn(0x559c6e22e560 msgr2=0x559c6e23a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775fc73740 1 -- 172.21.15.31:0/4096779533 shutdown_connections 2024-04-25T02:19:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775fc73740 1 -- 172.21.15.31:0/4096779533 wait complete. 2024-04-25T02:19:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775fc73740 20 client.18168 trim_cache size 0 max 16384 2024-04-25T02:19:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T02:19:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T02:19:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T02:19:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T02:19:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775cf63640 1 -- 172.21.15.31:0/2931482489 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f77400a2fa0 msgr2=0x7f77400a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775cf63640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f77400a2fa0 0x7f77400a5460 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7748032ff0 tx=0x7f77480095b0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775cf63640 1 -- 172.21.15.31:0/2931482489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7758074400 msgr2=0x7f775819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:19:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775cf63640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7758074400 0x7f775819eac0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f774c00e790 tx=0x7f774c00eb90 comp rx=0 tx=0).stop 2024-04-25T02:19:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775cf63640 1 -- 172.21.15.31:0/2931482489 shutdown_connections 2024-04-25T02:19:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775cf63640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f77400a2fa0 0x7f77400a5460 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775cf63640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758112740 0x7f77581a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775cf63640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7758074400 0x7f775819eac0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775cf63640 1 --2- 172.21.15.31:0/2931482489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7758073ae0 0x7f775819e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:19:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.092+0000 7f775cf63640 1 -- 172.21.15.31:0/2931482489 >> 172.21.15.31:0/2931482489 conn(0x7f77580fd200 msgr2=0x7f77580feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:19:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.093+0000 7f775cf63640 1 -- 172.21.15.31:0/2931482489 shutdown_connections 2024-04-25T02:19:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T02:19:54.093+0000 7f775cf63640 1 -- 172.21.15.31:0/2931482489 wait complete. 2024-04-25T02:19:54.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 117477 8.81 MB/sec execute 598 sec latency 143.224 ms 2024-04-25T02:19:54.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:54 smithi040 ceph-mon[32418]: pgmap v802: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.2 MiB/s wr, 244 op/s 2024-04-25T02:19:54.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:54 smithi134 ceph-mon[33098]: pgmap v802: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.2 MiB/s wr, 244 op/s 2024-04-25T02:19:54.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:54 smithi031 ceph-mon[29204]: pgmap v802: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.2 MiB/s wr, 244 op/s 2024-04-25T02:19:55.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 117624 8.81 MB/sec execute 599 sec latency 67.833 ms 2024-04-25T02:19:56.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 cleanup 600 sec 2024-04-25T02:19:56.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:56 smithi040 ceph-mon[32418]: pgmap v803: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.4 MiB/s wr, 361 op/s 2024-04-25T02:19:56.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:56 smithi134 ceph-mon[33098]: pgmap v803: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.4 MiB/s wr, 361 op/s 2024-04-25T02:19:56.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:56 smithi031 ceph-mon[29204]: pgmap v803: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 8.4 MiB/s wr, 361 op/s 2024-04-25T02:19:57.114 INFO:tasks.workunit.client.0.smithi031.stdout: 10 cleanup 601 sec 2024-04-25T02:19:58.018 INFO:tasks.workunit.client.0.smithi031.stdout: 0 cleanup 602 sec 2024-04-25T02:19:58.018 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: Operation Count AvgLat MaxLat 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: ---------------------------------------- 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: NTCreateX 169290 10.452 3304.428 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: Close 124256 0.196 37.449 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: Rename 7168 25.189 420.316 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: Unlink 34216 8.798 3276.117 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: Qpathinfo 153399 6.856 3325.897 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: Qfileinfo 26758 0.374 82.035 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: Qfsinfo 28172 0.952 31.619 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: Sfileinfo 13723 3.070 84.802 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: Find 59327 41.134 3329.133 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: WriteX 83611 0.656 64.573 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: ReadX 265860 0.008 10.019 2024-04-25T02:19:58.019 INFO:tasks.workunit.client.0.smithi031.stdout: LockX 556 1.888 7.076 2024-04-25T02:19:58.020 INFO:tasks.workunit.client.0.smithi031.stdout: UnlockX 556 1.826 6.720 2024-04-25T02:19:58.020 INFO:tasks.workunit.client.0.smithi031.stdout: Flush 11850 7.716 320.929 2024-04-25T02:19:58.020 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-04-25T02:19:58.020 INFO:tasks.workunit.client.0.smithi031.stdout:Throughput 8.80867 MB/sec 10 clients 10 procs max_latency=3329.140 ms 2024-04-25T02:19:58.020 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T02:19:58.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T02:19:58.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:58 smithi040 ceph-mon[32418]: pgmap v804: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 227 op/s 2024-04-25T02:19:58.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:58 smithi134 ceph-mon[33098]: pgmap v804: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 227 op/s 2024-04-25T02:19:58.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:58 smithi031 ceph-mon[29204]: pgmap v804: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 227 op/s 2024-04-25T02:19:59.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:19:59 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:19:59.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:19:59 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:19:59.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:19:59 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:19:59.772 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-25T02:19:59.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T02:20:00.042 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a[29200]: 2024-04-25T02:19:59.999+0000 7fc418dd9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:20:00.043 DEBUG:teuthology.parallel:result is None 2024-04-25T02:20:00.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T02:20:00.093 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T02:20:00.093 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-25T02:20:00.105 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T02:20:00.114 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-25T02:20:00.152 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-25T02:20:00.152 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T02:20:00.152 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T02:20:00.152 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-25T02:20:00.195 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-25T02:20:00.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T02:20:00.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:00 smithi031 ceph-mon[29204]: pgmap v805: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 227 op/s 2024-04-25T02:20:00.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:00 smithi031 ceph-mon[29204]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:20:00.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.382+0000 7f6add037640 1 -- 172.21.15.31:0/2569743532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad8102100 msgr2=0x7f6ad810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:00.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.382+0000 7f6add037640 1 --2- 172.21.15.31:0/2569743532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad8102100 0x7f6ad810e610 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009930 tx=0x7f6ad002a420 comp rx=0 tx=0).stop 2024-04-25T02:20:00.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.382+0000 7f6add037640 1 -- 172.21.15.31:0/2569743532 shutdown_connections 2024-04-25T02:20:00.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.382+0000 7f6add037640 1 --2- 172.21.15.31:0/2569743532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad8102100 0x7f6ad810e610 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.382+0000 7f6add037640 1 --2- 172.21.15.31:0/2569743532 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ad8101740 0x7f6ad8101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.382+0000 7f6add037640 1 --2- 172.21.15.31:0/2569743532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad8107750 0x7f6ad8107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.382+0000 7f6add037640 1 -- 172.21.15.31:0/2569743532 >> 172.21.15.31:0/2569743532 conn(0x7f6ad80fd210 msgr2=0x7f6ad80ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:00.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.382+0000 7f6add037640 1 -- 172.21.15.31:0/2569743532 shutdown_connections 2024-04-25T02:20:00.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.382+0000 7f6add037640 1 -- 172.21.15.31:0/2569743532 wait complete. 2024-04-25T02:20:00.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6add037640 1 Processor -- start 2024-04-25T02:20:00.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6add037640 1 -- start start 2024-04-25T02:20:00.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6add037640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad8101740 0x7f6ad8073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:00.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6add037640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad8102100 0x7f6ad8073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6add037640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ad8107750 0x7f6ad80744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6add037640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad8111a10 con 0x7f6ad8101740 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6add037640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad8111890 con 0x7f6ad8102100 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6add037640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad8111b90 con 0x7f6ad8107750 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6ad6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad8101740 0x7f6ad8073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6ad6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad8101740 0x7f6ad8073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33300/0 (socket says 172.21.15.31:33300) 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6ad6d9d640 1 -- 172.21.15.31:0/1979616453 learned_addr learned my addr 172.21.15.31:0/1979616453 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6ad659c640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad8102100 0x7f6ad8073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.383+0000 7f6ad759e640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ad8107750 0x7f6ad80744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6ad6d9d640 1 -- 172.21.15.31:0/1979616453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ad8107750 msgr2=0x7f6ad80744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6ad6d9d640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ad8107750 0x7f6ad80744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6ad6d9d640 1 -- 172.21.15.31:0/1979616453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad8102100 msgr2=0x7f6ad8073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:00.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6ad6d9d640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad8102100 0x7f6ad8073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6ad6d9d640 1 -- 172.21.15.31:0/1979616453 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad81a6390 con 0x7f6ad8101740 2024-04-25T02:20:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6ad659c640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad8102100 0x7f6ad8073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:20:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6ad6d9d640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad8101740 0x7f6ad8073a40 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800ebc0 tx=0x7f6ac800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6abffff640 1 -- 172.21.15.31:0/1979616453 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8010630 con 0x7f6ad8101740 2024-04-25T02:20:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6abffff640 1 -- 172.21.15.31:0/1979616453 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ac8010c80 con 0x7f6ad8101740 2024-04-25T02:20:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6add037640 1 -- 172.21.15.31:0/1979616453 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad81a6680 con 0x7f6ad8101740 2024-04-25T02:20:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6abffff640 1 -- 172.21.15.31:0/1979616453 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8018b30 con 0x7f6ad8101740 2024-04-25T02:20:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.384+0000 7f6add037640 1 -- 172.21.15.31:0/1979616453 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ad81a6bd0 con 0x7f6ad8101740 2024-04-25T02:20:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.385+0000 7f6add037640 1 -- 172.21.15.31:0/1979616453 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ad804f500 con 0x7f6ad8101740 2024-04-25T02:20:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.386+0000 7f6abffff640 1 -- 172.21.15.31:0/1979616453 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6ac8018cd0 con 0x7f6ad8101740 2024-04-25T02:20:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.387+0000 7f6abffff640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6ac00a2da0 0x7f6ac00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.387+0000 7f6ad659c640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6ac00a2da0 0x7f6ac00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.387+0000 7f6abffff640 1 -- 172.21.15.31:0/1979616453 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f6ac80cfec0 con 0x7f6ad8101740 2024-04-25T02:20:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.387+0000 7f6ad659c640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6ac00a2da0 0x7f6ac00a5260 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80754d0 tx=0x7f6acc005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.390+0000 7f6abffff640 1 -- 172.21.15.31:0/1979616453 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6ac809cb00 con 0x7f6ad8101740 2024-04-25T02:20:00.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:00 smithi040 ceph-mon[32418]: pgmap v805: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 227 op/s 2024-04-25T02:20:00.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:00 smithi040 ceph-mon[32418]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:20:00.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:00 smithi134 ceph-mon[33098]: pgmap v805: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 227 op/s 2024-04-25T02:20:00.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:00 smithi134 ceph-mon[33098]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:20:00.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.600+0000 7f6add037640 1 -- 172.21.15.31:0/1979616453 --> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f6ad8061b80 con 0x7f6ac00a2da0 2024-04-25T02:20:00.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.610+0000 7f6abffff640 1 -- 172.21.15.31:0/1979616453 <== mgr.15351 v2:172.21.15.31:6800/1834919992 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f6ad8061b80 con 0x7f6ac00a2da0 2024-04-25T02:20:00.614 INFO:teuthology.orchestra.run.smithi031.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T01:54:36", "first": "2024-04-25T01:55:00", "last": "2024-04-25T02:20:00", "last_pruned": "2024-04-25T02:20:00", "created_count": 26, "pruned_count": 19, "active": true} 2024-04-25T02:20:00.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 -- 172.21.15.31:0/1979616453 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6ac00a2da0 msgr2=0x7f6ac00a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:00.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6ac00a2da0 0x7f6ac00a5260 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80754d0 tx=0x7f6acc005f90 comp rx=0 tx=0).stop 2024-04-25T02:20:00.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 -- 172.21.15.31:0/1979616453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad8101740 msgr2=0x7f6ad8073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:00.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad8101740 0x7f6ad8073a40 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800ebc0 tx=0x7f6ac800c6a0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 -- 172.21.15.31:0/1979616453 shutdown_connections 2024-04-25T02:20:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6ac00a2da0 0x7f6ac00a5260 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ad8107750 0x7f6ad80744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ad8102100 0x7f6ad8073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 --2- 172.21.15.31:0/1979616453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ad8101740 0x7f6ad8073a40 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 -- 172.21.15.31:0/1979616453 >> 172.21.15.31:0/1979616453 conn(0x7f6ad80fd210 msgr2=0x7f6ad80fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 -- 172.21.15.31:0/1979616453 shutdown_connections 2024-04-25T02:20:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.612+0000 7f6add037640 1 -- 172.21.15.31:0/1979616453 wait complete. 2024-04-25T02:20:00.627 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T02:20:00.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.767+0000 7fc28ccda640 1 -- 172.21.15.31:0/1197739390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288102100 msgr2=0x7fc28810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:00.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.767+0000 7fc28ccda640 1 --2- 172.21.15.31:0/1197739390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288102100 0x7fc28810e610 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fc280009cd0 tx=0x7fc28002a8e0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.767+0000 7fc28ccda640 1 -- 172.21.15.31:0/1197739390 shutdown_connections 2024-04-25T02:20:00.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.767+0000 7fc28ccda640 1 --2- 172.21.15.31:0/1197739390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288102100 0x7fc28810e610 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.767+0000 7fc28ccda640 1 --2- 172.21.15.31:0/1197739390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc288101740 0x7fc288101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.767+0000 7fc28ccda640 1 --2- 172.21.15.31:0/1197739390 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc288107750 0x7fc288107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.767+0000 7fc28ccda640 1 -- 172.21.15.31:0/1197739390 >> 172.21.15.31:0/1197739390 conn(0x7fc2880fd230 msgr2=0x7fc2880ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.767+0000 7fc28ccda640 1 -- 172.21.15.31:0/1197739390 shutdown_connections 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.767+0000 7fc28ccda640 1 -- 172.21.15.31:0/1197739390 wait complete. 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28ccda640 1 Processor -- start 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28ccda640 1 -- start start 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28ccda640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288101740 0x7fc288073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28ccda640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc288102100 0x7fc288073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28ccda640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc288107750 0x7fc2880744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28ccda640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc288111a60 con 0x7fc288101740 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28ccda640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2881118e0 con 0x7fc288102100 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28ccda640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc288111be0 con 0x7fc288107750 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288101740 0x7fc288073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:00.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288101740 0x7fc288073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33314/0 (socket says 172.21.15.31:33314) 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28659c640 1 -- 172.21.15.31:0/3000883570 learned_addr learned my addr 172.21.15.31:0/3000883570 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc285d9b640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc288102100 0x7fc288073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc286d9d640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc288107750 0x7fc2880744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28659c640 1 -- 172.21.15.31:0/3000883570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc288107750 msgr2=0x7fc2880744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28659c640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc288107750 0x7fc2880744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28659c640 1 -- 172.21.15.31:0/3000883570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc288102100 msgr2=0x7fc288073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28659c640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc288102100 0x7fc288073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.768+0000 7fc28659c640 1 -- 172.21.15.31:0/3000883570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2881a63e0 con 0x7fc288101740 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.769+0000 7fc28659c640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288101740 0x7fc288073a40 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc27800c920 tx=0x7fc27800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.769+0000 7fc2777fe640 1 -- 172.21.15.31:0/3000883570 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278007c70 con 0x7fc288101740 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.769+0000 7fc2777fe640 1 -- 172.21.15.31:0/3000883570 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc278007e10 con 0x7fc288101740 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.769+0000 7fc2777fe640 1 -- 172.21.15.31:0/3000883570 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278018820 con 0x7fc288101740 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.769+0000 7fc28ccda640 1 -- 172.21.15.31:0/3000883570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2881a66d0 con 0x7fc288101740 2024-04-25T02:20:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.769+0000 7fc28ccda640 1 -- 172.21.15.31:0/3000883570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2881a6c70 con 0x7fc288101740 2024-04-25T02:20:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.770+0000 7fc28ccda640 1 -- 172.21.15.31:0/3000883570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc28804f500 con 0x7fc288101740 2024-04-25T02:20:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.771+0000 7fc2777fe640 1 -- 172.21.15.31:0/3000883570 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc2780189c0 con 0x7fc288101740 2024-04-25T02:20:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.772+0000 7fc2777fe640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2700a2e20 0x7fc2700a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.772+0000 7fc285d9b640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2700a2e20 0x7fc2700a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.772+0000 7fc2777fe640 1 -- 172.21.15.31:0/3000883570 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7fc2780cf690 con 0x7fc288101740 2024-04-25T02:20:00.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.772+0000 7fc285d9b640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2700a2e20 0x7fc2700a52e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0095f0 tx=0x7fc27c005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.775+0000 7fc2777fe640 1 -- 172.21.15.31:0/3000883570 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc278014030 con 0x7fc288101740 2024-04-25T02:20:00.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.982+0000 7fc28ccda640 1 -- 172.21.15.31:0/3000883570 --> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fc288061b80 con 0x7fc2700a2e20 2024-04-25T02:20:00.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.992+0000 7fc2777fe640 1 -- 172.21.15.31:0/3000883570 <== mgr.15351 v2:172.21.15.31:6800/1834919992 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fc288061b80 con 0x7fc2700a2e20 2024-04-25T02:20:00.997 INFO:teuthology.orchestra.run.smithi031.stdout:/ 1m 6m3h 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.994+0000 7fc28ccda640 1 -- 172.21.15.31:0/3000883570 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2700a2e20 msgr2=0x7fc2700a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.994+0000 7fc28ccda640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2700a2e20 0x7fc2700a52e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0095f0 tx=0x7fc27c005f90 comp rx=0 tx=0).stop 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.994+0000 7fc28ccda640 1 -- 172.21.15.31:0/3000883570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288101740 msgr2=0x7fc288073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.994+0000 7fc28ccda640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288101740 0x7fc288073a40 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc27800c920 tx=0x7fc27800cde0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.994+0000 7fc28ccda640 1 -- 172.21.15.31:0/3000883570 shutdown_connections 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.994+0000 7fc28ccda640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc2700a2e20 0x7fc2700a52e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.994+0000 7fc28ccda640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc288107750 0x7fc2880744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.994+0000 7fc28ccda640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc288102100 0x7fc288073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.994+0000 7fc28ccda640 1 --2- 172.21.15.31:0/3000883570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc288101740 0x7fc288073a40 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.995+0000 7fc28ccda640 1 -- 172.21.15.31:0/3000883570 >> 172.21.15.31:0/3000883570 conn(0x7fc2880fd230 msgr2=0x7fc2880fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.995+0000 7fc28ccda640 1 -- 172.21.15.31:0/3000883570 shutdown_connections 2024-04-25T02:20:00.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:00.995+0000 7fc28ccda640 1 -- 172.21.15.31:0/3000883570 wait complete. 2024-04-25T02:20:01.009 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-25T02:20:01.019 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-25T02:20:01.019 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-04-25T02:20:01.020 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T02:20:01.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T02:20:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.203+0000 7f30ec3d5640 1 -- 172.21.15.31:0/994628970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4106630 msgr2=0x7f30e4106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.203+0000 7f30ec3d5640 1 --2- 172.21.15.31:0/994628970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4106630 0x7f30e4106a10 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009930 tx=0x7f30e002a420 comp rx=0 tx=0).stop 2024-04-25T02:20:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.203+0000 7f30ec3d5640 1 -- 172.21.15.31:0/994628970 shutdown_connections 2024-04-25T02:20:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.203+0000 7f30ec3d5640 1 --2- 172.21.15.31:0/994628970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4100fe0 0x7f30e410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.203+0000 7f30ec3d5640 1 --2- 172.21.15.31:0/994628970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e4100620 0x7f30e4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.203+0000 7f30ec3d5640 1 --2- 172.21.15.31:0/994628970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4106630 0x7f30e4106a10 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.203+0000 7f30ec3d5640 1 -- 172.21.15.31:0/994628970 >> 172.21.15.31:0/994628970 conn(0x7f30e4077740 msgr2=0x7f30e4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.203+0000 7f30ec3d5640 1 -- 172.21.15.31:0/994628970 shutdown_connections 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.203+0000 7f30ec3d5640 1 -- 172.21.15.31:0/994628970 wait complete. 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ec3d5640 1 Processor -- start 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ec3d5640 1 -- start start 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ec3d5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e4100620 0x7f30e419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ec3d5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4100fe0 0x7f30e419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ec3d5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4106630 0x7f30e419e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ec3d5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f30e4110940 con 0x7f30e4106630 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ec3d5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f30e41107c0 con 0x7f30e4100fe0 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ec3d5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f30e4110ac0 con 0x7f30e4100620 2024-04-25T02:20:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ea972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4106630 0x7f30e419e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ea972640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4106630 0x7f30e419e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33338/0 (socket says 172.21.15.31:33338) 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ea972640 1 -- 172.21.15.31:0/1686442138 learned_addr learned my addr 172.21.15.31:0/1686442138 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30e9970640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4100fe0 0x7f30e419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.204+0000 7f30ea171640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e4100620 0x7f30e419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30ea972640 1 -- 172.21.15.31:0/1686442138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e4100620 msgr2=0x7f30e419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30ea972640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e4100620 0x7f30e419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30ea972640 1 -- 172.21.15.31:0/1686442138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4100fe0 msgr2=0x7f30e419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30ea972640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4100fe0 0x7f30e419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30ea972640 1 -- 172.21.15.31:0/1686442138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e419f120 con 0x7f30e4106630 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30ea171640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e4100620 0x7f30e419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30ea972640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4106630 0x7f30e419e9f0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00eb10 tx=0x7f30dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30db7fe640 1 -- 172.21.15.31:0/1686442138 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc004b20 con 0x7f30e4106630 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30db7fe640 1 -- 172.21.15.31:0/1686442138 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30dc004cc0 con 0x7f30e4106630 2024-04-25T02:20:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30db7fe640 1 -- 172.21.15.31:0/1686442138 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc010690 con 0x7f30e4106630 2024-04-25T02:20:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30ec3d5640 1 -- 172.21.15.31:0/1686442138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e419f410 con 0x7f30e4106630 2024-04-25T02:20:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.205+0000 7f30ec3d5640 1 -- 172.21.15.31:0/1686442138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e41a6ca0 con 0x7f30e4106630 2024-04-25T02:20:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.206+0000 7f30ec3d5640 1 -- 172.21.15.31:0/1686442138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e4068560 con 0x7f30e4106630 2024-04-25T02:20:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.207+0000 7f30db7fe640 1 -- 172.21.15.31:0/1686442138 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f30dc005000 con 0x7f30e4106630 2024-04-25T02:20:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.208+0000 7f30db7fe640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f30d00a2d80 0x7f30d00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.208+0000 7f30ea171640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f30d00a2d80 0x7f30d00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.208+0000 7f30db7fe640 1 -- 172.21.15.31:0/1686442138 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f30dc0cfb30 con 0x7f30e4106630 2024-04-25T02:20:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.209+0000 7f30ea171640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f30d00a2d80 0x7f30d00a5240 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f30e002a940 tx=0x7f30e0005fd0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:01.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.212+0000 7f30db7fe640 1 -- 172.21.15.31:0/1686442138 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f30dc053740 con 0x7f30e4106630 2024-04-25T02:20:01.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.398+0000 7f30ec3d5640 1 -- 172.21.15.31:0/1686442138 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f30e41a6fa0 con 0x7f30e4106630 2024-04-25T02:20:01.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.400+0000 7f30db7fe640 1 -- 172.21.15.31:0/1686442138 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+22949 (secure 0 0 0) 0x7f30dc09c770 con 0x7f30e4106630 2024-04-25T02:20:01.405 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T02:20:01.405 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T02:19:09.026489+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":44,"snap_epoch":126,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3045817976":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6800/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1358173279":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/686937010":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/3913527371":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":31,"length":1},{"begin":44,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T02:20:01.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 -- 172.21.15.31:0/1686442138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f30d00a2d80 msgr2=0x7f30d00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f30d00a2d80 0x7f30d00a5240 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f30e002a940 tx=0x7f30e0005fd0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 -- 172.21.15.31:0/1686442138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4106630 msgr2=0x7f30e419e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4106630 0x7f30e419e9f0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00eb10 tx=0x7f30dc00c2d0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 -- 172.21.15.31:0/1686442138 shutdown_connections 2024-04-25T02:20:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f30d00a2d80 0x7f30d00a5240 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30e4106630 0x7f30e419e9f0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4100fe0 0x7f30e419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 --2- 172.21.15.31:0/1686442138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30e4100620 0x7f30e419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 -- 172.21.15.31:0/1686442138 >> 172.21.15.31:0/1686442138 conn(0x7f30e4077740 msgr2=0x7f30e410c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 -- 172.21.15.31:0/1686442138 shutdown_connections 2024-04-25T02:20:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.402+0000 7f30ec3d5640 1 -- 172.21.15.31:0/1686442138 wait complete. 2024-04-25T02:20:01.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:01 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:01.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:01 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:01.418 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:23.432566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:24.255373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 44, 'snap_epoch': 126, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T02:20:01.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T02:20:01.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:01 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:01.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.519+0000 7f3138129640 1 -- 172.21.15.31:0/2776527365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3130100d70 msgr2=0x7f31301011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.519+0000 7f3138129640 1 --2- 172.21.15.31:0/2776527365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3130100d70 0x7f31301011f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f3120009900 tx=0x7f312002a480 comp rx=0 tx=0).stop 2024-04-25T02:20:01.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.519+0000 7f3138129640 1 -- 172.21.15.31:0/2776527365 shutdown_connections 2024-04-25T02:20:01.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.519+0000 7f3138129640 1 --2- 172.21.15.31:0/2776527365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130101730 0x7f313010d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.519+0000 7f3138129640 1 --2- 172.21.15.31:0/2776527365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3130100d70 0x7f31301011f0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.519+0000 7f3138129640 1 --2- 172.21.15.31:0/2776527365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3130106660 0x7f3130106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.519+0000 7f3138129640 1 -- 172.21.15.31:0/2776527365 >> 172.21.15.31:0/2776527365 conn(0x7f3130076bd0 msgr2=0x7f31300fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:01.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.519+0000 7f3138129640 1 -- 172.21.15.31:0/2776527365 shutdown_connections 2024-04-25T02:20:01.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.519+0000 7f3138129640 1 -- 172.21.15.31:0/2776527365 wait complete. 2024-04-25T02:20:01.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.519+0000 7f3138129640 1 Processor -- start 2024-04-25T02:20:01.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.519+0000 7f3138129640 1 -- start start 2024-04-25T02:20:01.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3138129640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3130100d70 0x7f3130197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3138129640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130101730 0x7f3130198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3138129640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3130106660 0x7f313019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3138129640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f31301108e0 con 0x7f3130100d70 2024-04-25T02:20:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3138129640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3130110760 con 0x7f3130101730 2024-04-25T02:20:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3138129640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3130110a60 con 0x7f3130106660 2024-04-25T02:20:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3135ec5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3130100d70 0x7f3130197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3135ec5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3130100d70 0x7f3130197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33362/0 (socket says 172.21.15.31:33362) 2024-04-25T02:20:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3135ec5640 1 -- 172.21.15.31:0/2688730950 learned_addr learned my addr 172.21.15.31:0/2688730950 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f31366c6640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3130106660 0x7f313019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f31356c4640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130101730 0x7f3130198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3135ec5640 1 -- 172.21.15.31:0/2688730950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3130106660 msgr2=0x7f313019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3135ec5640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3130106660 0x7f313019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3135ec5640 1 -- 172.21.15.31:0/2688730950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130101730 msgr2=0x7f3130198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3135ec5640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130101730 0x7f3130198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3135ec5640 1 -- 172.21.15.31:0/2688730950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f313019cf20 con 0x7f3130100d70 2024-04-25T02:20:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3135ec5640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3130100d70 0x7f3130197f30 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f312c00d770 tx=0x7f312c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.520+0000 7f3126ffd640 1 -- 172.21.15.31:0/2688730950 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c017070 con 0x7f3130100d70 2024-04-25T02:20:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.521+0000 7f3138129640 1 -- 172.21.15.31:0/2688730950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f313019d210 con 0x7f3130100d70 2024-04-25T02:20:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.521+0000 7f3126ffd640 1 -- 172.21.15.31:0/2688730950 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f312c0054d0 con 0x7f3130100d70 2024-04-25T02:20:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.521+0000 7f3126ffd640 1 -- 172.21.15.31:0/2688730950 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c004cf0 con 0x7f3130100d70 2024-04-25T02:20:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.521+0000 7f3138129640 1 -- 172.21.15.31:0/2688730950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31301a4aa0 con 0x7f3130100d70 2024-04-25T02:20:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.522+0000 7f3138129640 1 -- 172.21.15.31:0/2688730950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31300689b0 con 0x7f3130100d70 2024-04-25T02:20:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.523+0000 7f3126ffd640 1 -- 172.21.15.31:0/2688730950 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f312c004e90 con 0x7f3130100d70 2024-04-25T02:20:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.524+0000 7f3126ffd640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f311c0a2d80 0x7f311c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.524+0000 7f31356c4640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f311c0a2d80 0x7f311c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.524+0000 7f3126ffd640 1 -- 172.21.15.31:0/2688730950 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f312c0cf1c0 con 0x7f3130100d70 2024-04-25T02:20:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.524+0000 7f31356c4640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f311c0a2d80 0x7f311c0a5240 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3130100d40 tx=0x7f3120005a20 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:01.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.527+0000 7f3126ffd640 1 -- 172.21.15.31:0/2688730950 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f312c09be20 con 0x7f3130100d70 2024-04-25T02:20:01.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.715+0000 7f3138129640 1 -- 172.21.15.31:0/2688730950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f31301a4cd0 con 0x7f3130100d70 2024-04-25T02:20:01.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.715+0000 7f3126ffd640 1 -- 172.21.15.31:0/2688730950 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) ==== 93+0+10 (secure 0 0 0) 0x7f312c09b720 con 0x7f3130100d70 2024-04-25T02:20:01.719 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T02:20:01.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 -- 172.21.15.31:0/2688730950 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f311c0a2d80 msgr2=0x7f311c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f311c0a2d80 0x7f311c0a5240 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3130100d40 tx=0x7f3120005a20 comp rx=0 tx=0).stop 2024-04-25T02:20:01.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 -- 172.21.15.31:0/2688730950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3130100d70 msgr2=0x7f3130197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3130100d70 0x7f3130197f30 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f312c00d770 tx=0x7f312c00dc30 comp rx=0 tx=0).stop 2024-04-25T02:20:01.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 -- 172.21.15.31:0/2688730950 shutdown_connections 2024-04-25T02:20:01.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f311c0a2d80 0x7f311c0a5240 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3130106660 0x7f313019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130101730 0x7f3130198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 --2- 172.21.15.31:0/2688730950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3130100d70 0x7f3130197f30 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 -- 172.21.15.31:0/2688730950 >> 172.21.15.31:0/2688730950 conn(0x7f3130076bd0 msgr2=0x7f3130104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:01.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 -- 172.21.15.31:0/2688730950 shutdown_connections 2024-04-25T02:20:01.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.717+0000 7f3138129640 1 -- 172.21.15.31:0/2688730950 wait complete. 2024-04-25T02:20:01.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.833+0000 7f3771ef5640 1 -- 172.21.15.31:0/3917634986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f376c100fe0 msgr2=0x7f376c10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.833+0000 7f3771ef5640 1 --2- 172.21.15.31:0/3917634986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f376c100fe0 0x7f376c10d4f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f3764009cd0 tx=0x7f376402a8e0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 -- 172.21.15.31:0/3917634986 shutdown_connections 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 --2- 172.21.15.31:0/3917634986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f376c100fe0 0x7f376c10d4f0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 --2- 172.21.15.31:0/3917634986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c100620 0x7f376c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 --2- 172.21.15.31:0/3917634986 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f376c106630 0x7f376c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 -- 172.21.15.31:0/3917634986 >> 172.21.15.31:0/3917634986 conn(0x7f376c0776a0 msgr2=0x7f376c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 -- 172.21.15.31:0/3917634986 shutdown_connections 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 -- 172.21.15.31:0/3917634986 wait complete. 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 Processor -- start 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 -- start start 2024-04-25T02:20:01.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c100620 0x7f376c197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f376c100fe0 0x7f376c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f376c106630 0x7f376c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f376c1108b0 con 0x7f376c106630 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f376c110730 con 0x7f376c100620 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f3771ef5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f376c110a30 con 0x7f376c100fe0 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.834+0000 7f376bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f376c106630 0x7f376c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f376bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f376c106630 0x7f376c19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33374/0 (socket says 172.21.15.31:33374) 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f376bfff640 1 -- 172.21.15.31:0/3804912860 learned_addr learned my addr 172.21.15.31:0/3804912860 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f376b7fe640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c100620 0x7f376c197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f376affd640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f376c100fe0 0x7f376c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f376bfff640 1 -- 172.21.15.31:0/3804912860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f376c100fe0 msgr2=0x7f376c1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f376bfff640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f376c100fe0 0x7f376c1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f376bfff640 1 -- 172.21.15.31:0/3804912860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c100620 msgr2=0x7f376c197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:01.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f376bfff640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c100620 0x7f376c197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:01.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f376bfff640 1 -- 172.21.15.31:0/3804912860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c19ce50 con 0x7f376c106630 2024-04-25T02:20:01.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f376b7fe640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c100620 0x7f376c197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:20:01.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f376bfff640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f376c106630 0x7f376c19c720 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3764033040 tx=0x7f3764003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:01.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f3768ff9640 1 -- 172.21.15.31:0/3804912860 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764004040 con 0x7f376c106630 2024-04-25T02:20:01.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f3768ff9640 1 -- 172.21.15.31:0/3804912860 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3764004690 con 0x7f376c106630 2024-04-25T02:20:01.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f3771ef5640 1 -- 172.21.15.31:0/3804912860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c19d0e0 con 0x7f376c106630 2024-04-25T02:20:01.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f3768ff9640 1 -- 172.21.15.31:0/3804912860 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764030d80 con 0x7f376c106630 2024-04-25T02:20:01.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.835+0000 7f3771ef5640 1 -- 172.21.15.31:0/3804912860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c1a49d0 con 0x7f376c106630 2024-04-25T02:20:01.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.836+0000 7f3771ef5640 1 -- 172.21.15.31:0/3804912860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c0689b0 con 0x7f376c106630 2024-04-25T02:20:01.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.837+0000 7f3768ff9640 1 -- 172.21.15.31:0/3804912860 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f37640041e0 con 0x7f376c106630 2024-04-25T02:20:01.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.838+0000 7f3768ff9640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37540a2d00 0x7f37540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:01.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.838+0000 7f376b7fe640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37540a2d00 0x7f37540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:01.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.838+0000 7f3768ff9640 1 -- 172.21.15.31:0/3804912860 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8484+0+0 (secure 0 0 0) 0x7f376402e030 con 0x7f376c106630 2024-04-25T02:20:01.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.838+0000 7f376b7fe640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37540a2d00 0x7f37540a51c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f375c00af30 tx=0x7f375c00a2d0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:01.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:01.841+0000 7f3768ff9640 1 -- 172.21.15.31:0/3804912860 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f37640bd2c0 con 0x7f376c106630 2024-04-25T02:20:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.026+0000 7f3771ef5640 1 -- 172.21.15.31:0/3804912860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f376c109c20 con 0x7f376c106630 2024-04-25T02:20:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.026+0000 7f3768ff9640 1 -- 172.21.15.31:0/3804912860 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v127) ==== 104+0+11 (secure 0 0 0) 0x7f37640bcbc0 con 0x7f376c106630 2024-04-25T02:20:02.031 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T02:20:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.028+0000 7f3771ef5640 1 -- 172.21.15.31:0/3804912860 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37540a2d00 msgr2=0x7f37540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.028+0000 7f3771ef5640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37540a2d00 0x7f37540a51c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f375c00af30 tx=0x7f375c00a2d0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.028+0000 7f3771ef5640 1 -- 172.21.15.31:0/3804912860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f376c106630 msgr2=0x7f376c19c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.028+0000 7f3771ef5640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f376c106630 0x7f376c19c720 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3764033040 tx=0x7f3764003970 comp rx=0 tx=0).stop 2024-04-25T02:20:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.028+0000 7f3771ef5640 1 -- 172.21.15.31:0/3804912860 shutdown_connections 2024-04-25T02:20:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.028+0000 7f3771ef5640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f37540a2d00 0x7f37540a51c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.028+0000 7f3771ef5640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f376c106630 0x7f376c19c720 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.028+0000 7f3771ef5640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f376c100fe0 0x7f376c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.028+0000 7f3771ef5640 1 --2- 172.21.15.31:0/3804912860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f376c100620 0x7f376c197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.028+0000 7f3771ef5640 1 -- 172.21.15.31:0/3804912860 >> 172.21.15.31:0/3804912860 conn(0x7f376c0776a0 msgr2=0x7f376c06a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.029+0000 7f3771ef5640 1 -- 172.21.15.31:0/3804912860 shutdown_connections 2024-04-25T02:20:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.029+0000 7f3771ef5640 1 -- 172.21.15.31:0/3804912860 wait complete. 2024-04-25T02:20:02.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T02:20:02.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.167+0000 7feda210e640 1 -- 172.21.15.31:0/403945659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed9c1015e0 msgr2=0x7fed9c101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:02.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.167+0000 7feda210e640 1 --2- 172.21.15.31:0/403945659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed9c1015e0 0x7fed9c101a60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009860 tx=0x7fed8c02a360 comp rx=0 tx=0).stop 2024-04-25T02:20:02.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.167+0000 7feda210e640 1 -- 172.21.15.31:0/403945659 shutdown_connections 2024-04-25T02:20:02.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.167+0000 7feda210e640 1 --2- 172.21.15.31:0/403945659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed9c101fa0 0x7fed9c10e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.167+0000 7feda210e640 1 --2- 172.21.15.31:0/403945659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed9c1015e0 0x7fed9c101a60 secure :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009860 tx=0x7fed8c02a360 comp rx=0 tx=0).stop 2024-04-25T02:20:02.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.167+0000 7feda210e640 1 --2- 172.21.15.31:0/403945659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed9c107680 0x7fed9c107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.167+0000 7feda210e640 1 -- 172.21.15.31:0/403945659 >> 172.21.15.31:0/403945659 conn(0x7fed9c0fd110 msgr2=0x7fed9c0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.169+0000 7feda210e640 1 -- 172.21.15.31:0/403945659 shutdown_connections 2024-04-25T02:20:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.169+0000 7feda210e640 1 -- 172.21.15.31:0/403945659 wait complete. 2024-04-25T02:20:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.170+0000 7feda210e640 1 Processor -- start 2024-04-25T02:20:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.170+0000 7feda210e640 1 -- start start 2024-04-25T02:20:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.170+0000 7feda210e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed9c101fa0 0x7fed9c10dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.170+0000 7feda210e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed9c107680 0x7fed9c108460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.170+0000 7feda210e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed9c1089a0 0x7fed9c108e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.170+0000 7feda210e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fed9c06cfc0 con 0x7fed9c1089a0 2024-04-25T02:20:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.170+0000 7feda210e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fed9c06ce40 con 0x7fed9c107680 2024-04-25T02:20:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.170+0000 7feda210e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fed9c06d140 con 0x7fed9c101fa0 2024-04-25T02:20:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.171+0000 7feda090b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed9c107680 0x7fed9c108460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.171+0000 7feda090b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed9c107680 0x7fed9c108460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.31:32996/0 (socket says 172.21.15.31:32996) 2024-04-25T02:20:02.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.171+0000 7feda090b640 1 -- 172.21.15.31:0/2233282953 learned_addr learned my addr 172.21.15.31:0/2233282953 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:02.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.172+0000 7feda090b640 1 -- 172.21.15.31:0/2233282953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed9c101fa0 msgr2=0x7fed9c10dfd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T02:20:02.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.172+0000 7feda090b640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed9c101fa0 0x7fed9c10dfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.172+0000 7feda090b640 1 -- 172.21.15.31:0/2233282953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed9c1089a0 msgr2=0x7fed9c108e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:02.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.172+0000 7feda090b640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed9c1089a0 0x7fed9c108e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.172+0000 7feda090b640 1 -- 172.21.15.31:0/2233282953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed9c10f0a0 con 0x7fed9c107680 2024-04-25T02:20:02.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.172+0000 7feda090b640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed9c107680 0x7fed9c108460 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fed8c035040 tx=0x7fed8c032e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:02.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.173+0000 7fed927fc640 1 -- 172.21.15.31:0/2233282953 <== mon.2 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8c042070 con 0x7fed9c107680 2024-04-25T02:20:02.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.173+0000 7fed927fc640 1 -- 172.21.15.31:0/2233282953 <== mon.2 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed8c0054e0 con 0x7fed9c107680 2024-04-25T02:20:02.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.173+0000 7fed927fc640 1 -- 172.21.15.31:0/2233282953 <== mon.2 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8c037040 con 0x7fed9c107680 2024-04-25T02:20:02.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.173+0000 7feda210e640 1 -- 172.21.15.31:0/2233282953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed9c10f330 con 0x7fed9c107680 2024-04-25T02:20:02.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.173+0000 7feda210e640 1 -- 172.21.15.31:0/2233282953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed9c10f7a0 con 0x7fed9c107680 2024-04-25T02:20:02.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.175+0000 7feda210e640 1 -- 172.21.15.31:0/2233282953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed9c102d20 con 0x7fed9c107680 2024-04-25T02:20:02.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.178+0000 7fed927fc640 1 -- 172.21.15.31:0/2233282953 <== mon.2 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fed8c045070 con 0x7fed9c107680 2024-04-25T02:20:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.179+0000 7fed927fc640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fed880a2f00 0x7fed880a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.179+0000 7fed927fc640 1 -- 172.21.15.31:0/2233282953 <== mon.2 v2:172.21.15.40:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8528+0+0 (secure 0 0 0) 0x7fed8c0eea20 con 0x7fed9c107680 2024-04-25T02:20:02.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.180+0000 7feda110c640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fed880a2f00 0x7fed880a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.181+0000 7feda110c640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fed880a2f00 0x7fed880a53c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fed9c102720 tx=0x7fed9801a270 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:02.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.187+0000 7fed927fc640 1 -- 172.21.15.31:0/2233282953 <== mon.2 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fed8c0bb810 con 0x7fed9c107680 2024-04-25T02:20:02.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:02 smithi040 ceph-mon[32418]: from='client.18174 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:20:02.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:02 smithi040 ceph-mon[32418]: pgmap v806: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 8.4 MiB/s wr, 385 op/s 2024-04-25T02:20:02.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:02 smithi040 ceph-mon[32418]: from='client.18180 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:20:02.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:02 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1686442138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:20:02.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:02 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2688730950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:20:02.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:02 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3804912860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:20:02.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:02 smithi134 ceph-mon[33098]: from='client.18174 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:20:02.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:02 smithi134 ceph-mon[33098]: pgmap v806: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 8.4 MiB/s wr, 385 op/s 2024-04-25T02:20:02.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:02 smithi134 ceph-mon[33098]: from='client.18180 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:20:02.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:02 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1686442138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:20:02.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:02 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2688730950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:20:02.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:02 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3804912860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:20:02.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.477+0000 7feda210e640 1 -- 172.21.15.31:0/2233282953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fed9c10fac0 con 0x7fed9c107680 2024-04-25T02:20:02.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.478+0000 7fed927fc640 1 -- 172.21.15.31:0/2233282953 <== mon.2 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) ==== 100+0+11 (secure 0 0 0) 0x7fed8c0bb110 con 0x7fed9c107680 2024-04-25T02:20:02.482 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 -- 172.21.15.31:0/2233282953 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fed880a2f00 msgr2=0x7fed880a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fed880a2f00 0x7fed880a53c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fed9c102720 tx=0x7fed9801a270 comp rx=0 tx=0).stop 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 -- 172.21.15.31:0/2233282953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed9c107680 msgr2=0x7fed9c108460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed9c107680 0x7fed9c108460 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fed8c035040 tx=0x7fed8c032e10 comp rx=0 tx=0).stop 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 -- 172.21.15.31:0/2233282953 shutdown_connections 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fed880a2f00 0x7fed880a53c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed9c1089a0 0x7fed9c108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed9c107680 0x7fed9c108460 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 --2- 172.21.15.31:0/2233282953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed9c101fa0 0x7fed9c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 -- 172.21.15.31:0/2233282953 >> 172.21.15.31:0/2233282953 conn(0x7fed9c0fd110 msgr2=0x7fed9c06f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 -- 172.21.15.31:0/2233282953 shutdown_connections 2024-04-25T02:20:02.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.480+0000 7feda210e640 1 -- 172.21.15.31:0/2233282953 wait complete. 2024-04-25T02:20:02.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:02 smithi031 ceph-mon[29204]: from='client.18174 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:20:02.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:02 smithi031 ceph-mon[29204]: pgmap v806: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 8.4 MiB/s wr, 385 op/s 2024-04-25T02:20:02.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:02 smithi031 ceph-mon[29204]: from='client.18180 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:20:02.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:02 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1686442138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:20:02.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:02 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2688730950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:20:02.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:02 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3804912860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:20:02.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T02:20:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.593+0000 7fa943129640 1 -- 172.21.15.31:0/1327778159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93c101740 msgr2=0x7fa93c101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.593+0000 7fa943129640 1 --2- 172.21.15.31:0/1327778159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93c101740 0x7fa93c101bc0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fa928009930 tx=0x7fa92802a420 comp rx=0 tx=0).stop 2024-04-25T02:20:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.593+0000 7fa943129640 1 -- 172.21.15.31:0/1327778159 shutdown_connections 2024-04-25T02:20:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.593+0000 7fa943129640 1 --2- 172.21.15.31:0/1327778159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa93c102100 0x7fa93c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.593+0000 7fa943129640 1 --2- 172.21.15.31:0/1327778159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93c101740 0x7fa93c101bc0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.593+0000 7fa943129640 1 --2- 172.21.15.31:0/1327778159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa93c107750 0x7fa93c107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.593+0000 7fa943129640 1 -- 172.21.15.31:0/1327778159 >> 172.21.15.31:0/1327778159 conn(0x7fa93c078bd0 msgr2=0x7fa93c078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.593+0000 7fa943129640 1 -- 172.21.15.31:0/1327778159 shutdown_connections 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.593+0000 7fa943129640 1 -- 172.21.15.31:0/1327778159 wait complete. 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa943129640 1 Processor -- start 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa943129640 1 -- start start 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa943129640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa93c101740 0x7fa93c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa943129640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93c102100 0x7fa93c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa943129640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa93c107750 0x7fa93c19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa943129640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c111a10 con 0x7fa93c102100 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa943129640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c111890 con 0x7fa93c107750 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa943129640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c111b90 con 0x7fa93c101740 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa93bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93c102100 0x7fa93c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa93bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93c102100 0x7fa93c19a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33434/0 (socket says 172.21.15.31:33434) 2024-04-25T02:20:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa93bfff640 1 -- 172.21.15.31:0/790074474 learned_addr learned my addr 172.21.15.31:0/790074474 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa940ec5640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa93c101740 0x7fa93c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.594+0000 7fa9416c6640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa93c107750 0x7fa93c19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa93bfff640 1 -- 172.21.15.31:0/790074474 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa93c101740 msgr2=0x7fa93c19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa93bfff640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa93c101740 0x7fa93c19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa93bfff640 1 -- 172.21.15.31:0/790074474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa93c107750 msgr2=0x7fa93c19ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa93bfff640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa93c107750 0x7fa93c19ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa93bfff640 1 -- 172.21.15.31:0/790074474 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa93c19f140 con 0x7fa93c102100 2024-04-25T02:20:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa9416c6640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa93c107750 0x7fa93c19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:20:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa93bfff640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93c102100 0x7fa93c19a6e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fa928004770 tx=0x7fa9280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa939ffb640 1 -- 172.21.15.31:0/790074474 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92803f070 con 0x7fa93c102100 2024-04-25T02:20:02.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa939ffb640 1 -- 172.21.15.31:0/790074474 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa92803b460 con 0x7fa93c102100 2024-04-25T02:20:02.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa939ffb640 1 -- 172.21.15.31:0/790074474 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa928035070 con 0x7fa93c102100 2024-04-25T02:20:02.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa943129640 1 -- 172.21.15.31:0/790074474 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa93c19f3d0 con 0x7fa93c102100 2024-04-25T02:20:02.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.595+0000 7fa943129640 1 -- 172.21.15.31:0/790074474 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa93c1a6cc0 con 0x7fa93c102100 2024-04-25T02:20:02.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.596+0000 7fa943129640 1 -- 172.21.15.31:0/790074474 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa93c0689b0 con 0x7fa93c102100 2024-04-25T02:20:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.597+0000 7fa939ffb640 1 -- 172.21.15.31:0/790074474 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa92803b610 con 0x7fa93c102100 2024-04-25T02:20:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.598+0000 7fa939ffb640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa92c0a2d80 0x7fa92c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.598+0000 7fa940ec5640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa92c0a2d80 0x7fa92c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.598+0000 7fa939ffb640 1 -- 172.21.15.31:0/790074474 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8528+0+0 (secure 0 0 0) 0x7fa9280efaf0 con 0x7fa93c102100 2024-04-25T02:20:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.598+0000 7fa940ec5640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa92c0a2d80 0x7fa92c0a5240 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa930005d90 tx=0x7fa930005d20 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:02.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.601+0000 7fa939ffb640 1 -- 172.21.15.31:0/790074474 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa9280bc6d0 con 0x7fa93c102100 2024-04-25T02:20:02.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.837+0000 7fa943129640 1 -- 172.21.15.31:0/790074474 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fa93c1a6f80 con 0x7fa93c102100 2024-04-25T02:20:02.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.846+0000 7fa939ffb640 1 -- 172.21.15.31:0/790074474 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+740900 (secure 0 0 0) 0x7fa9280bbfd0 con 0x7fa93c102100 2024-04-25T02:20:02.861 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T02:20:02.861 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 26, 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: "active_gid": 15351, 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: "active_name": "x", 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addrs": { 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1834919992 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-04-25T02:20:02.862 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1834919992 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addr": "172.21.15.31:6801/1834919992", 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: "active_change": "2024-04-25T01:54:28.579486+0000", 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: "available": true, 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: "standbys": [ 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 25003, 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "z", 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_features": 4540701547738038271, 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-04-25T02:20:02.863 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T02:20:02.864 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:02.865 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:02.866 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:02.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:02.868 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.869 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.870 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-04-25T02:20:02.871 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.872 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.873 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.874 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.875 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.876 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.877 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-04-25T02:20:02.879 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:02.880 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-04-25T02:20:02.881 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.882 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.883 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.884 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.885 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:02.888 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:02.889 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-04-25T02:20:02.890 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.893 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.895 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:02.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.897 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.898 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.899 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.900 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.903 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-04-25T02:20:02.904 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-04-25T02:20:02.905 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-04-25T02:20:02.906 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.907 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.909 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.910 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-04-25T02:20:02.911 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-04-25T02:20:02.912 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.913 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.914 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.915 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.916 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T02:20:02.917 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T02:20:02.918 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.919 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.920 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba": { 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba", 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB container image", 2024-04-25T02:20:02.921 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-04-25T02:20:02.922 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.923 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T02:20:02.924 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T02:20:02.925 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-04-25T02:20:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.928 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.929 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.930 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.932 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:02.933 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:02.934 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.935 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.936 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:02.937 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:02.938 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T02:20:02.939 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T02:20:02.940 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-04-25T02:20:02.941 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-04-25T02:20:02.942 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:02.943 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.944 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.945 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.948 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.950 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-04-25T02:20:02.951 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.952 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.953 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.954 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T02:20:02.955 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.956 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:02.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.959 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:02.962 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.964 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.965 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T02:20:02.966 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T02:20:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.969 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.970 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.971 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.972 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.973 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.974 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.975 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.976 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.977 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.978 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.979 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.980 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.981 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.982 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.983 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.986 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.987 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.988 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.990 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.991 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T02:20:02.992 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T02:20:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.994 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T02:20:02.995 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T02:20:02.996 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T02:20:02.997 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.998 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T02:20:02.999 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.000 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T02:20:03.001 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T02:20:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.003 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.004 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.005 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.006 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.007 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.008 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.009 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.010 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.011 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.012 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.013 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.014 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.015 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.016 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.017 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.018 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T02:20:03.020 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.021 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.023 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.024 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.026 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.028 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.029 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.030 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.031 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.033 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.034 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-04-25T02:20:03.035 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.036 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.037 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.039 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.040 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.041 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.042 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.043 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.044 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.045 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.046 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.047 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-04-25T02:20:03.048 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-04-25T02:20:03.049 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-04-25T02:20:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.051 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.052 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.053 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.054 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.055 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.056 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-04-25T02:20:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.058 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.059 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.061 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.062 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.064 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.065 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.066 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.068 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.069 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.070 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.071 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.072 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.073 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.074 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.075 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.076 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.077 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.078 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.079 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.080 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.081 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.082 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.083 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.084 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.085 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-04-25T02:20:03.086 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.087 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.089 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.090 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.092 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.093 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.094 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.095 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.096 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.097 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.098 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.099 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.100 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.101 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.102 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.103 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.104 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.105 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.106 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.107 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.108 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.109 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.110 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.111 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.112 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.113 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.114 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.115 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.116 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.117 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.118 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-04-25T02:20:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.120 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.121 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.122 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.123 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.124 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.125 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.126 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.127 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.128 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.129 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:03.130 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T02:20:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.132 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.134 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.135 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.136 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.137 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.140 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.142 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.143 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.144 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.145 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.147 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.148 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.150 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T02:20:03.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.152 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.153 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.154 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.155 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.156 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.157 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.158 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.159 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.160 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.161 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.162 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.163 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.164 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.165 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.166 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.167 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.168 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.169 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.170 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.171 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.173 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.174 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.175 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.176 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.178 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.179 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.181 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.182 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.183 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.185 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.186 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.187 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.188 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-04-25T02:20:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.190 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-04-25T02:20:03.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.192 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.193 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.194 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.195 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.196 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.197 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.198 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.199 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.200 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.201 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.203 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.204 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.205 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.206 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.207 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.208 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.209 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.210 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.211 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.214 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.216 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.217 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.218 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-04-25T02:20:03.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:03.220 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.221 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.223 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.224 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.225 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.226 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.227 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.228 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.229 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.230 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.231 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.232 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.233 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.234 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.235 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.238 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.239 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.240 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.241 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.242 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.243 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.244 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.245 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.246 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.247 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.248 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.250 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.251 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-25T02:20:03.252 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.253 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.254 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.256 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.257 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.258 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.259 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.260 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.261 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-04-25T02:20:03.262 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.264 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.265 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.268 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.269 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.270 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.271 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.272 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 25067, 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "y", 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_features": 4540701547738038271, 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.273 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.274 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.275 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.276 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.277 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.278 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-04-25T02:20:03.279 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.280 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.281 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.282 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.283 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.284 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.285 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.286 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.287 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.288 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.290 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.291 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-04-25T02:20:03.292 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.293 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:03.294 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.295 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.296 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.297 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.298 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.299 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.300 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.301 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-04-25T02:20:03.302 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-04-25T02:20:03.304 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:03.305 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.306 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T02:20:03.307 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.308 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.309 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T02:20:03.310 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.311 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.313 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.315 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.316 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.317 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.318 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-04-25T02:20:03.319 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.320 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.321 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.322 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.323 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.325 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.326 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-04-25T02:20:03.327 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-04-25T02:20:03.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.329 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba": { 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba", 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.330 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB container image", 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.331 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.332 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.333 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.334 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-04-25T02:20:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.336 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.337 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.338 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.339 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.340 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.341 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.342 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.343 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.344 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.345 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.346 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.348 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.349 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.352 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T02:20:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.354 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.358 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.362 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.364 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.365 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.366 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-04-25T02:20:03.367 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.369 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.370 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.371 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.372 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.373 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T02:20:03.374 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.376 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.377 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.378 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.379 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.380 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T02:20:03.382 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.383 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.384 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.385 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.386 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.387 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.388 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.390 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.391 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T02:20:03.392 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.393 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T02:20:03.394 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T02:20:03.395 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.396 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.397 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T02:20:03.398 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.399 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.400 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.401 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.402 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.403 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.404 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.405 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.406 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.407 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.408 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.411 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.412 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.413 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.414 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.415 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.416 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.417 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.418 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.421 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.422 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.423 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.424 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.425 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.427 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.428 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T02:20:03.429 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T02:20:03.430 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.431 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T02:20:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.434 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.436 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.437 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.438 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.440 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.441 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.442 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.443 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.444 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.445 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.446 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.447 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.450 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.451 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.451 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.451 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.451 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.451 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.451 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.451 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.453 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.453 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.453 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.453 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.453 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.453 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.453 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.453 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.453 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.453 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.454 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.455 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.456 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.457 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-04-25T02:20:03.458 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.460 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.463 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.465 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.466 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-04-25T02:20:03.467 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-04-25T02:20:03.468 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.469 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.470 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.471 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.472 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.473 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.475 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.476 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.477 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.479 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.480 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.481 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.482 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.483 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.484 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.485 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.486 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.487 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.488 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.489 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.490 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T02:20:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.493 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.494 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.495 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.496 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.497 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.498 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.499 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.500 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-04-25T02:20:03.500 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.500 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.500 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.500 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T02:20:03.500 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.500 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.500 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.500 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.502 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.503 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.503 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.503 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-04-25T02:20:03.503 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.503 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.503 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.503 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.503 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.503 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.504 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.505 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.507 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-04-25T02:20:03.508 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.509 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.510 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.511 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.513 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.514 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.517 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.518 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.519 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.521 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.522 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.523 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.524 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T02:20:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.527 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.528 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.529 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.530 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.531 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.532 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.533 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.534 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.535 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.536 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.537 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.539 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.540 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.541 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.542 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.543 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.544 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-04-25T02:20:03.545 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.546 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.547 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.548 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.549 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.550 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.551 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.552 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.553 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-04-25T02:20:03.554 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.557 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.558 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-04-25T02:20:03.561 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T02:20:03.562 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-04-25T02:20:03.563 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T02:20:03.564 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.565 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.566 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.567 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.568 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.570 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.571 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.573 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.574 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.575 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.576 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.577 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.578 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.579 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.580 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.581 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.583 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.584 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.585 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.586 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.587 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.588 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.589 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.590 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.591 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.592 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.593 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.594 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.596 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.597 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.598 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.599 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.601 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.603 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-04-25T02:20:03.604 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.605 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.606 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.607 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.608 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.609 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.610 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.611 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.612 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.613 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.614 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.616 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.617 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.618 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.619 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.620 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.621 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-04-25T02:20:03.622 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.623 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.624 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.625 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.626 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.627 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.628 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.629 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.630 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.631 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.633 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.634 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.635 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.636 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.637 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.638 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.639 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.640 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.641 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.648 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.652 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.653 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.654 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.655 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.656 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.657 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.658 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.659 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-04-25T02:20:03.660 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-04-25T02:20:03.660 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.661 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.661 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.661 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.661 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.662 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.663 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.665 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.666 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.667 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.668 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.669 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.670 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.671 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.672 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.673 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.674 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-04-25T02:20:03.676 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.677 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.679 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.680 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.682 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.683 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.685 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.686 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.687 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.688 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.689 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: "modules": [ 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: "dashboard", 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: "iostat", 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: "nfs", 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: "restful", 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: "snap_schedule" 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.690 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.691 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.692 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.693 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.694 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.695 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.696 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.697 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.698 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.699 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-04-25T02:20:03.700 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-04-25T02:20:03.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-04-25T02:20:03.702 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-04-25T02:20:03.704 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.705 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.706 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.707 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.708 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.710 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.711 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.712 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.713 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.714 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.716 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.718 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.720 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.721 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.722 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.724 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.725 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.726 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.727 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.728 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T02:20:03.729 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-04-25T02:20:03.730 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.731 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.732 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.733 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T02:20:03.734 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T02:20:03.735 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.736 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T02:20:03.737 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T02:20:03.738 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.739 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-04-25T02:20:03.740 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.741 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-04-25T02:20:03.742 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.744 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-04-25T02:20:03.745 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-04-25T02:20:03.746 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba": { 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba", 2024-04-25T02:20:03.747 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB container image", 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.748 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.749 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.750 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.751 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.752 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-04-25T02:20:03.753 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.756 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-04-25T02:20:03.757 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-04-25T02:20:03.758 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.759 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.760 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.762 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.763 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.764 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T02:20:03.765 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T02:20:03.766 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.767 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T02:20:03.769 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.771 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.772 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-04-25T02:20:03.773 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-04-25T02:20:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-04-25T02:20:03.776 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.777 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.778 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.779 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-04-25T02:20:03.780 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.781 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.782 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.783 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.784 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.785 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.786 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.787 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.788 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.789 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.790 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.791 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.793 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-04-25T02:20:03.794 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T02:20:03.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.796 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.797 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.798 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.799 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.800 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.801 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.802 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.803 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.804 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.805 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.807 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T02:20:03.808 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.809 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.810 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.811 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.813 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.814 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.815 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.816 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.817 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.819 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.820 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T02:20:03.823 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T02:20:03.824 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T02:20:03.825 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T02:20:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.827 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.828 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.829 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.830 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.831 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.832 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.833 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.834 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.835 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.836 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.837 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.838 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.839 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.840 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.841 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.842 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.843 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.844 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.846 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T02:20:03.847 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.848 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.849 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.850 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.851 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.852 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.853 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.854 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.855 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.856 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T02:20:03.857 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.858 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.859 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.860 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.862 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.864 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.865 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-04-25T02:20:03.865 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.865 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.865 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.865 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.865 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.865 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.865 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.866 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.867 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.868 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.869 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.870 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.870 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.870 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.870 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.870 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.870 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-04-25T02:20:03.870 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-04-25T02:20:03.870 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.870 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.870 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.871 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-04-25T02:20:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.874 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.875 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.878 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.879 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.880 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.881 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-04-25T02:20:03.882 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-04-25T02:20:03.883 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.884 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.885 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.886 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.887 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.888 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.889 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.890 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.891 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.893 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.894 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.897 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.898 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.899 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-04-25T02:20:03.900 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.901 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.902 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.903 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.904 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.906 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.909 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.910 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.912 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-04-25T02:20:03.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.914 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.915 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.916 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.918 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.919 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.920 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.921 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.922 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.923 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.924 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.925 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.926 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.927 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.928 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.929 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.930 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.931 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.932 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.933 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.934 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.935 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.936 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.937 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.938 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T02:20:03.939 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.941 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.942 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.944 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.945 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.946 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.947 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.948 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.949 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.950 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.951 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.952 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.953 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.954 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.955 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.957 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.958 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.961 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.962 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.963 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.964 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-04-25T02:20:03.965 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T02:20:03.966 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.967 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.968 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.970 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.971 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.972 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.974 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-04-25T02:20:03.975 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.976 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.977 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.978 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.979 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.980 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.981 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.982 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.984 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.985 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.986 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.987 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.989 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.990 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.991 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.992 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:03.995 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.996 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.997 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:03.998 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.000 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.002 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.003 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-04-25T02:20:04.004 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.005 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.006 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.008 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-04-25T02:20:04.009 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.010 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.011 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.012 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:04.013 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.014 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.016 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-04-25T02:20:04.017 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-04-25T02:20:04.018 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-04-25T02:20:04.019 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.020 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.021 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.022 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.023 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.024 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.025 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.026 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-04-25T02:20:04.027 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-04-25T02:20:04.028 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:04.029 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.030 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.031 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.033 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.034 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.038 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.039 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:04.040 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.041 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.042 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.043 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:04.044 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.046 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.047 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.048 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.049 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.050 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.051 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.053 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.054 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T02:20:04.055 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.056 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.059 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T02:20:04.060 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:04.061 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:04.062 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.064 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.065 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.066 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.068 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:04.068 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:04.068 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.068 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.068 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.068 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:04.068 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.068 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.068 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.068 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.069 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.069 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.069 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.069 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.069 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.069 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.069 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.069 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.069 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.070 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.074 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:04.075 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.076 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.077 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.079 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.080 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.081 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.082 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.083 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T02:20:04.084 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.085 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.086 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.087 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.088 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.089 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.090 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.091 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T02:20:04.092 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.094 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.095 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.097 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.098 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "services": { 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "dashboard": "https://172.21.15.31:8443/" 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "always_on_modules": { 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "octopus": [ 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-25T02:20:04.099 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "pacific": [ 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: "quincy": [ 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "reef": [ 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-25T02:20:04.101 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "squid": [ 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-25T02:20:04.102 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: "last_failure_osd_epoch": 90, 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: "active_clients": [ 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3989263264 2024-04-25T02:20:04.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "libcephsqlite", 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 732274836 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.104 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 619469846 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2141103607 2024-04-25T02:20:04.105 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 660991758 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-25T02:20:04.106 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T02:20:04.107 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T02:20:04.107 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T02:20:04.107 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-25T02:20:04.107 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2095904896 2024-04-25T02:20:04.107 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.107 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T02:20:04.107 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:04.107 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T02:20:04.107 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T02:20:04.108 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:03 smithi134 ceph-mon[33098]: osdmap e128: 12 total, 12 up, 12 in 2024-04-25T02:20:04.108 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:03 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:04.108 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:03 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2233282953' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:20:04.109 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:03 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/790074474' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T02:20:04.109 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:03 smithi040 ceph-mon[32418]: osdmap e128: 12 total, 12 up, 12 in 2024-04-25T02:20:04.109 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:03 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:04.109 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:03 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2233282953' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:20:04.109 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:03 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/790074474' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 -- 172.21.15.31:0/790074474 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa92c0a2d80 msgr2=0x7fa92c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa92c0a2d80 0x7fa92c0a5240 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa930005d90 tx=0x7fa930005d20 comp rx=0 tx=0).stop 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 -- 172.21.15.31:0/790074474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93c102100 msgr2=0x7fa93c19a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93c102100 0x7fa93c19a6e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fa928004770 tx=0x7fa9280042f0 comp rx=0 tx=0).stop 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 -- 172.21.15.31:0/790074474 shutdown_connections 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fa92c0a2d80 0x7fa92c0a5240 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa93c107750 0x7fa93c19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93c102100 0x7fa93c19a6e0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 --2- 172.21.15.31:0/790074474 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa93c101740 0x7fa93c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 -- 172.21.15.31:0/790074474 >> 172.21.15.31:0/790074474 conn(0x7fa93c078bd0 msgr2=0x7fa93c0fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 -- 172.21.15.31:0/790074474 shutdown_connections 2024-04-25T02:20:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:02.851+0000 7fa943129640 1 -- 172.21.15.31:0/790074474 wait complete. 2024-04-25T02:20:04.116 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T02:20:04.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T02:20:04.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:03 smithi031 ceph-mon[29204]: osdmap e128: 12 total, 12 up, 12 in 2024-04-25T02:20:04.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:03 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:04.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:03 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2233282953' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:20:04.117 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:03 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/790074474' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T02:20:04.381 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T02:20:04.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:04 smithi040 ceph-mon[32418]: pgmap v808: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.7 MiB/s wr, 329 op/s 2024-04-25T02:20:04.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:04 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:04.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:04 smithi134 ceph-mon[33098]: pgmap v808: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.7 MiB/s wr, 329 op/s 2024-04-25T02:20:04.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:04 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:04.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:04 smithi031 ceph-mon[29204]: pgmap v808: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.7 MiB/s wr, 329 op/s 2024-04-25T02:20:04.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:04 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:06.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:06 smithi031 ceph-mon[29204]: pgmap v809: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-25T02:20:06.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:06 smithi031 ceph-mon[29204]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:06.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:06 smithi040 ceph-mon[32418]: pgmap v809: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-25T02:20:06.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:06 smithi040 ceph-mon[32418]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:06.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:06 smithi134 ceph-mon[33098]: pgmap v809: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-25T02:20:06.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:06 smithi134 ceph-mon[33098]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T02:20:06.525 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T02:20:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 3221, 2024-04-25T02:20:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 3223, 2024-04-25T02:20:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 93235585, 2024-04-25T02:20:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 1024279, 2024-04-25T02:20:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 8, 2024-04-25T02:20:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 4, 2024-04-25T02:20:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.321362486, 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.078342029, 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.134537814, 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.085824181, 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 3221, 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.080460780, 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000024980 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 2472, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2431, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 156400274, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 899864, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 7, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 5, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.333633129, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.063076572, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.201415185, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.064387284, 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 2430, 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.065185921, 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000026825 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1891632, 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 282240 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 603, 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 937, 2024-04-25T02:20:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 92433960, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 1043992, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 10, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 6, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.115795429, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.029168853, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.064775470, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.015171005, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 935, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.029062406, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000031082 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 854464 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 1, 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-MDSRank": { 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 9539, 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.223092389, 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000023387 2024-04-25T02:20:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-PurgeQueue": { 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "request": 0, 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "reply": 0, 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "reply_latency": { 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_reply": 0, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "forward": 0, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_complete": 0, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_keys": 0, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_commit": 0, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_split": 0, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_merge": 0, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes": 11626, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_top": 58, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_bottom": 11568, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pin_tail": 0, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pinned": 11543, 2024-04-25T02:20:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_expired": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_with_caps": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "subtrees": 1, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_hit": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_forward": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_discover": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_dir_fetch": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_remote_ino": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_lock": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "load_cent": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "q": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "exported": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "exported_inodes": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "imported": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "imported_inodes": 0, 2024-04-25T02:20:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_dir_fetch": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_backtrace_fetch": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_peer_discover": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rfiles": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rbytes": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rsnaps": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_set_tag": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_file_inodes": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_inode_file_caps": 0, 2024-04-25T02:20:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_grant": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps_dirty": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_cap_release": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "process_request_cap_release": 0 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache": { 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update_receipt": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_try_discover": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_send_discover": 0, 2024-04-25T02:20:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_handle_discover": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_delayed": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_created": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_enqueued": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_reintegrated": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_migrated": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_processing": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_started": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_completed": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:20:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_exportdir": 0, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_flush": 0, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragmentdir": 0, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragstats": 0, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_inodestats": 0 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log": { 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "evlrg": 0, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "evadd": 0, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "evex": 0, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "evtrm": 0, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "ev": 62722, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "evexg": 0, 2024-04-25T02:20:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "evexd": 0, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "segadd": 0, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "segex": 0, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "segtrm": 0, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "seg": 100, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "segmjr": 7, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "segexg": 0, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "segexd": 0, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "expos": 4194304, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "wrpos": 342755392, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "rdpos": 342752093, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "jlat": { 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "replayed": 62722 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mem": { 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 1603, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "ino+": 11444, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "ino-": 9841, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "dir": 59, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "dir+": 79, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "dir-": 20, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "dn": 11626, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "dn+": 11790, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "dn-": 164, 2024-04-25T02:20:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "cap": 0, 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "cap+": 0, 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "cap-": 0, 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "rss": 69112, 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "heap": 215420 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_server": { 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_client_request": 0, 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_server_request": 0, 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_request": 0, 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_session": 0, 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_peer_request": 0, 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "req_create_latency": { 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getattr_latency": { 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getfilelock_latency": { 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "req_link_latency": { 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookup_latency": { 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookuphash_latency": { 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupino_latency": { 2024-04-25T02:20:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupname_latency": { 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupparent_latency": { 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupsnap_latency": { 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lssnap_latency": { 2024-04-25T02:20:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mkdir_latency": { 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mknod_latency": { 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mksnap_latency": { 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "req_open_latency": { 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "req_readdir_latency": { 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rename_latency": { 2024-04-25T02:20:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "req_renamesnap_latency": { 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "req_snapdiff_latency": { 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmdir_latency": { 2024-04-25T02:20:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmsnap_latency": { 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmxattr_latency": { 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setattr_latency": { 2024-04-25T02:20:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setdirlayout_latency": { 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setfilelock_latency": { 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setlayout_latency": { 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setxattr_latency": { 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "req_symlink_latency": { 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "req_unlink_latency": { 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getvxattr_latency": { 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessions": { 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "session_count": 2, 2024-04-25T02:20:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "session_add": 4, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "session_remove": 2, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_open": 3, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_stale": 0, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "total_load": 0, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "average_load": 0, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "avg_session_uptime": 3213, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "mdthresh_evicted": 0 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 240305, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 83, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 8056, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 168, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 11913926, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 54176, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter": { 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "op_active": 0, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "op_laggy": 0, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send": 5759, 2024-04-25T02:20:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send_bytes": 21, 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "op_resend": 3, 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "op_reply": 5759, 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "op_latency": { 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 5759, 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 4.252933737, 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000738484 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "op_inflight": 0, 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "oplen_avg": { 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 5759, 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 5760 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "op": 5759, 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "op_r": 5759, 2024-04-25T02:20:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "op_w": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "op_rmw": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "op_pg": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_stat": 3334, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_create": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_read": 2423, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_write": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writefull": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writesame": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_append": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_zero": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_truncate": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_delete": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_mapext": 0, 2024-04-25T02:20:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_sparse_read": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_clonerange": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_getxattr": 1, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_setxattr": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_rmxattr": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_call": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_watch": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_notify": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_other": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_active": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_send": 0, 2024-04-25T02:20:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_resend": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_ping": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_active": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_send": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_resend": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_active": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_send": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_resend": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_active": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_send": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_resend": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "command_active": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "command_send": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "command_resend": 0, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 128, 2024-04-25T02:20:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "map_full": 0, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "map_inc": 41, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_sessions": 11, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_open": 13, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_close": 2, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_laggy": 0, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_wr": 0, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_rd": 2, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_del": 0 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "oft": { 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_objs": 0, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_updates": 0, 2024-04-25T02:20:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_removes": 0 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "osd-slow-ops": { 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_ops_count": 0 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "purge_queue": { 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops": 0, 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing": 0, 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_high_water": 0, 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed_ops": 0, 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed": 0, 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_item_in_journal": 0 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 6292, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 341473523, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 6292, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 6292, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 341473523, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_bytes": { 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 5759, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 338574337, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 5759, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 338574337, 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_ops": { 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 1024, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 5759, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 5759, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 5759, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 5759, 2024-04-25T02:20:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T02:20:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 0, 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 0, 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle-0x55ad7abd66e0": { 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T02:20:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 0, 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 0, 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:06.568 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T02:20:07.532 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-25T02:20:07.533 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-25T02:20:07.533 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-04-25T02:20:07.533 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-04-25T02:20:07.533 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-04-25T02:20:07.533 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:20:07.533 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:20:07.533 DEBUG:tasks.check_counter:Getting stats from d 2024-04-25T02:20:07.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-25T02:20:07.793 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T02:20:08.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:08 smithi031 ceph-mon[29204]: pgmap v810: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-25T02:20:08.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:08 smithi040 ceph-mon[32418]: pgmap v810: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-25T02:20:08.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:08 smithi134 ceph-mon[33098]: pgmap v810: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-25T02:20:09.870 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T02:20:09.870 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:09.870 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 4, 2024-04-25T02:20:09.870 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 4, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 2992, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 448, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.001646904, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000038642, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.000550112, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 3, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000727607, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000242535 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:09.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 3, 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 344, 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 480, 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 293776, 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 6, 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-04-25T02:20:09.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.021361968, 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.015591603, 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.002306310, 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 341, 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.023867979, 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000069994 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-25T02:20:09.873 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 293776 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 459, 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 448, 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1648784, 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 277392, 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.067201948, 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.022437839, 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.043233138, 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.009057402, 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:09.874 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 447, 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.013497571, 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000030195 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1648784, 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 277392 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-25T02:20:09.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:09.876 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:09.877 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 148714, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 30, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-25T02:20:09.878 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:09.879 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 462, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1583945, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 462, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T02:20:09.880 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 462, 2024-04-25T02:20:09.881 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1583945, 2024-04-25T02:20:09.881 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:09.881 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:09.881 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:09.881 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:09.881 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:09.881 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:09.881 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T02:20:10.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:10 smithi031 ceph-mon[29204]: pgmap v811: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-25T02:20:10.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:10 smithi031 ceph-mon[29204]: osdmap e129: 12 total, 12 up, 12 in 2024-04-25T02:20:10.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:10 smithi040 ceph-mon[32418]: pgmap v811: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-25T02:20:10.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:10 smithi040 ceph-mon[32418]: osdmap e129: 12 total, 12 up, 12 in 2024-04-25T02:20:10.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:10 smithi134 ceph-mon[33098]: pgmap v811: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-25T02:20:10.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:10 smithi134 ceph-mon[33098]: osdmap e129: 12 total, 12 up, 12 in 2024-04-25T02:20:10.619 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-25T02:20:10.619 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-25T02:20:10.619 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-25T02:20:10.619 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-25T02:20:10.619 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-25T02:20:10.619 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-25T02:20:10.620 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-25T02:20:10.620 DEBUG:tasks.check_counter:Getting stats from g 2024-04-25T02:20:10.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-25T02:20:10.875 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T02:20:12.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:12 smithi040 ceph-mon[32418]: pgmap v813: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 147 KiB/s wr, 58 op/s 2024-04-25T02:20:12.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:12 smithi134 ceph-mon[33098]: pgmap v813: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 147 KiB/s wr, 58 op/s 2024-04-25T02:20:12.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:12 smithi031 ceph-mon[29204]: pgmap v813: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 147 KiB/s wr, 58 op/s 2024-04-25T02:20:12.907 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T02:20:12.907 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:12.907 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 2895, 2024-04-25T02:20:12.907 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2854, 2024-04-25T02:20:12.907 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 103297334, 2024-04-25T02:20:12.907 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 1034812, 2024-04-25T02:20:12.907 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 7, 2024-04-25T02:20:12.907 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-04-25T02:20:12.907 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.345606470, 2024-04-25T02:20:12.907 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.097853315, 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.166076163, 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.074015591, 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 2853, 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.085356279, 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000029918 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1892640, 2024-04-25T02:20:12.908 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 283696 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 2530, 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2867, 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 105505719, 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 1657835, 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 10, 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 7, 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.252957462, 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.058480995, 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.121351780, 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.057217136, 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:12.909 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 2864, 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.074386314, 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000025972 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 854192 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 888, 2024-04-25T02:20:12.910 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 888, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 132674892, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 281572, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 8, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 5, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.158721933, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.018278271, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.112593593, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.021421069, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 887, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.024552670, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000027680 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.911 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 1, 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-MDSRank": { 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-04-25T02:20:12.912 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 9684, 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.211372766, 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000021827 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-PurgeQueue": { 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.913 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "request": 0, 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "reply": 0, 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "reply_latency": { 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_reply": 0, 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "forward": 0, 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_complete": 0, 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_keys": 0, 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_commit": 0, 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_split": 0, 2024-04-25T02:20:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_merge": 0, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes": 11624, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_top": 47, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_bottom": 11577, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pin_tail": 0, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pinned": 11556, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_expired": 0, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_with_caps": 0, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": 0, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "subtrees": 1, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse": 0, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_hit": 0, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_forward": 0, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_discover": 0, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_dir_fetch": 0, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_remote_ino": 0, 2024-04-25T02:20:12.915 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_lock": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "load_cent": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "q": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "exported": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "exported_inodes": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "imported": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "imported_inodes": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_dir_fetch": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_backtrace_fetch": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_peer_discover": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rfiles": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rbytes": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rsnaps": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T02:20:12.916 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_set_tag": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_file_inodes": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_inode_file_caps": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_grant": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps_dirty": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_cap_release": 0, 2024-04-25T02:20:12.917 INFO:teuthology.orchestra.run.smithi031.stdout: "process_request_cap_release": 0 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache": { 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update": 0, 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update_receipt": 0, 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_try_discover": 0, 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_send_discover": 0, 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_handle_discover": 0, 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays": 0, 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_delayed": 0, 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_created": 0, 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_enqueued": 0, 2024-04-25T02:20:12.918 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_reintegrated": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_migrated": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_processing": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_started": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_completed": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_exportdir": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_flush": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragmentdir": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragstats": 0, 2024-04-25T02:20:12.919 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_inodestats": 0 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log": { 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "evlrg": 0, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "evadd": 0, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "evex": 0, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "evtrm": 0, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "ev": 66487, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "evexg": 0, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "evexd": 0, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "segadd": 0, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "segex": 0, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "segtrm": 0, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "seg": 103, 2024-04-25T02:20:12.920 INFO:teuthology.orchestra.run.smithi031.stdout: "segmjr": 7, 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "segexg": 0, 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "segexd": 0, 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "expos": 4194304, 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "wrpos": 342161039, 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "rdpos": 342153275, 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "jlat": { 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "replayed": 66487 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mem": { 2024-04-25T02:20:12.921 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 1613, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "ino+": 11675, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "ino-": 10062, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "dir": 59, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "dir+": 90, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "dir-": 31, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "dn": 11624, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "dn+": 12016, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "dn-": 392, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "cap": 0, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "cap+": 0, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "cap-": 0, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "rss": 73216, 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: "heap": 215420 2024-04-25T02:20:12.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_server": { 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_client_request": 0, 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_server_request": 0, 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_request": 0, 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_session": 0, 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_peer_request": 0, 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "req_create_latency": { 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getattr_latency": { 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.923 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getfilelock_latency": { 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: "req_link_latency": { 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookup_latency": { 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.924 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookuphash_latency": { 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupino_latency": { 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupname_latency": { 2024-04-25T02:20:12.925 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupparent_latency": { 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupsnap_latency": { 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.926 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lssnap_latency": { 2024-04-25T02:20:12.927 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.927 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.927 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mkdir_latency": { 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mknod_latency": { 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mksnap_latency": { 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.928 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: "req_open_latency": { 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: "req_readdir_latency": { 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rename_latency": { 2024-04-25T02:20:12.929 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: "req_renamesnap_latency": { 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: "req_snapdiff_latency": { 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmdir_latency": { 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmsnap_latency": { 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmxattr_latency": { 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setattr_latency": { 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setdirlayout_latency": { 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setfilelock_latency": { 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.932 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setlayout_latency": { 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setxattr_latency": { 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "req_symlink_latency": { 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.933 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: "req_unlink_latency": { 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getvxattr_latency": { 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:12.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessions": { 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "session_count": 1, 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "session_add": 1, 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "session_remove": 0, 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_open": 1, 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_stale": 0, 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "total_load": 0, 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "average_load": 0, 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "avg_session_uptime": 2660, 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "mdthresh_evicted": 0 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:20:12.935 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:12.937 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 240265, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 82, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:12.938 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 7936, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 165, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 11932422, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 54239, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter": { 2024-04-25T02:20:12.939 INFO:teuthology.orchestra.run.smithi031.stdout: "op_active": 0, 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "op_laggy": 0, 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send": 5773, 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send_bytes": 21, 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "op_resend": 3, 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "op_reply": 5773, 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "op_latency": { 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 5773, 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 3.956938354, 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000685421 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "op_inflight": 0, 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "oplen_avg": { 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 5773, 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 5774 2024-04-25T02:20:12.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "op": 5773, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "op_r": 5773, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "op_w": 0, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "op_rmw": 0, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "op_pg": 0, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_stat": 3341, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_create": 0, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_read": 2430, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_write": 0, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writefull": 0, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writesame": 0, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_append": 0, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_zero": 0, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_truncate": 0, 2024-04-25T02:20:12.941 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_delete": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_mapext": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_sparse_read": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_clonerange": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_getxattr": 1, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_setxattr": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_rmxattr": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_call": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_watch": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_notify": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:20:12.942 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_other": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_active": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_send": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_resend": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_ping": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_active": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_send": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_resend": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_active": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_send": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_resend": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_active": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_send": 0, 2024-04-25T02:20:12.943 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_resend": 0, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "command_active": 0, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "command_send": 0, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "command_resend": 0, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 129, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "map_full": 0, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "map_inc": 42, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_sessions": 11, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_open": 13, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_close": 2, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_laggy": 0, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_wr": 0, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_rd": 2, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_del": 0 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.944 INFO:teuthology.orchestra.run.smithi031.stdout: "oft": { 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_objs": 0, 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_updates": 0, 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_removes": 0 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "osd-slow-ops": { 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_ops_count": 0 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "purge_queue": { 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops": 0, 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing": 0, 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_high_water": 0, 2024-04-25T02:20:12.945 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed_ops": 0, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed": 0, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_item_in_journal": 0 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 6309, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 340879973, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 6309, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T02:20:12.946 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 6309, 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 340879973, 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_bytes": { 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T02:20:12.947 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 5773, 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 337976411, 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 5773, 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 337976411, 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_ops": { 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 1024, 2024-04-25T02:20:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 5773, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 5773, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 5773, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 5773, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.949 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 0, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 0, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.950 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle-0x5639a55706e0": { 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 0, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 0, 2024-04-25T02:20:12.951 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:12.952 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:12.952 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T02:20:13.581 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-25T02:20:13.581 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-25T02:20:13.581 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-04-25T02:20:13.581 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-04-25T02:20:13.581 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-04-25T02:20:13.581 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:20:13.581 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:20:13.581 DEBUG:tasks.check_counter:Getting stats from j 2024-04-25T02:20:13.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-25T02:20:13.841 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T02:20:14.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:14 smithi031 ceph-mon[29204]: pgmap v814: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 130 KiB/s wr, 51 op/s 2024-04-25T02:20:14.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:14 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:20:14.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:14 smithi040 ceph-mon[32418]: pgmap v814: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 130 KiB/s wr, 51 op/s 2024-04-25T02:20:14.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:14 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:20:14.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:14 smithi134 ceph-mon[33098]: pgmap v814: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 130 KiB/s wr, 51 op/s 2024-04-25T02:20:14.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:14 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:20:15.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:15 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:20:15.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:15 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:20:15.413 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:15 smithi040 ceph-mon[32418]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:20:15.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:15 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:20:15.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:15 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:20:15.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:15 smithi031 ceph-mon[29204]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:20:15.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:15 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:20:15.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:15 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:20:15.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:15 smithi134 ceph-mon[33098]: from='mgr.15351 172.21.15.31:0/678725656' entity='mgr.x' 2024-04-25T02:20:15.704 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 0, 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2, 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 0, 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 0, 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.001185339, 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.000180719, 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:20:15.705 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:15.706 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 462, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 451, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 1649600, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 279312, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.066310210, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.023545173, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.040921754, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.009087167, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 450, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.014064268, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000031253 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:15.707 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 1649600, 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 279312 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 7, 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 348, 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 3472, 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 294576, 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 6, 2024-04-25T02:20:15.708 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.023603481, 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.017137300, 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.002875034, 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 346, 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.015103563, 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000043651 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:15.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 294576 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:15.710 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:15.711 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:15.712 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 148731, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 31, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:15.713 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T02:20:15.714 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 465, 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 1584335, 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 465, 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 465, 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1584335, 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T02:20:15.715 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T02:20:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:15.716 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T02:20:15.716 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T02:20:16.398 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-25T02:20:16.399 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-25T02:20:16.399 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-25T02:20:16.399 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-25T02:20:16.399 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-25T02:20:16.399 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-25T02:20:16.399 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-25T02:20:16.399 DEBUG:tasks.check_counter:Getting stats from b 2024-04-25T02:20:16.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-25T02:20:16.662 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:16 smithi040 ceph-mon[32418]: pgmap v815: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-25T02:20:16.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:16 smithi134 ceph-mon[33098]: pgmap v815: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-25T02:20:16.733 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T02:20:16.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:16 smithi031 ceph-mon[29204]: pgmap v815: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-25T02:20:18.642 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:18 smithi040 ceph-mon[32418]: pgmap v816: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-25T02:20:18.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:18 smithi134 ceph-mon[33098]: pgmap v816: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-25T02:20:18.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:18 smithi031 ceph-mon[29204]: pgmap v816: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-25T02:20:19.022 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-25T02:20:19.022 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 415464, 2024-04-25T02:20:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 466763, 2024-04-25T02:20:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 147525277, 2024-04-25T02:20:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 219831060, 2024-04-25T02:20:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 10, 2024-04-25T02:20:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-04-25T02:20:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 21.787633589, 2024-04-25T02:20:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 8.708331656, 2024-04-25T02:20:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 15.099148889, 2024-04-25T02:20:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.482828184, 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 466765, 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 10.341230701, 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000022155 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 852242, 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.127642230, 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000149 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1930976, 2024-04-25T02:20:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 950192 2024-04-25T02:20:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1289120, 2024-04-25T02:20:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 1324006, 2024-04-25T02:20:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 381627098, 2024-04-25T02:20:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 920725437, 2024-04-25T02:20:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 17, 2024-04-25T02:20:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 7, 2024-04-25T02:20:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 64.266523165, 2024-04-25T02:20:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 26.745253278, 2024-04-25T02:20:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 45.994299765, 2024-04-25T02:20:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.928536544, 2024-04-25T02:20:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1324008, 2024-04-25T02:20:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 23.263524238, 2024-04-25T02:20:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000017570 2024-04-25T02:20:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2545165, 2024-04-25T02:20:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.347430639, 2024-04-25T02:20:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000136 2024-04-25T02:20:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 821296, 2024-04-25T02:20:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 413840 2024-04-25T02:20:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 190413, 2024-04-25T02:20:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 231385, 2024-04-25T02:20:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 56954905, 2024-04-25T02:20:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 186666211, 2024-04-25T02:20:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 270, 2024-04-25T02:20:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 8, 2024-04-25T02:20:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 11.953228463, 2024-04-25T02:20:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 4.909746895, 2024-04-25T02:20:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 7.337463853, 2024-04-25T02:20:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.601936106, 2024-04-25T02:20:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 231376, 2024-04-25T02:20:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 5.977377384, 2024-04-25T02:20:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000025834 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 386943, 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.066161665, 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000170 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 139216, 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 31968 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-04-25T02:20:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-25T02:20:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-04-25T02:20:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-25T02:20:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-25T02:20:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 82114, 2024-04-25T02:20:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 51.700285375, 2024-04-25T02:20:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000629615 2024-04-25T02:20:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 9551, 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.720587469, 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000075446 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 1000733, 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 955539, 2024-04-25T02:20:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-04-25T02:20:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 955539, 2024-04-25T02:20:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 827.495078614, 2024-04-25T02:20:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000865998 2024-04-25T02:20:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-04-25T02:20:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 45194, 2024-04-25T02:20:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 27, 2024-04-25T02:20:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 1, 2024-04-25T02:20:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 2355, 2024-04-25T02:20:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 2, 2024-04-25T02:20:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 1978, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 606, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 1362, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 10, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 1105, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 2876, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 3, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 3, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 6, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 1942164, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 1916704, 2024-04-25T02:20:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 11400, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 231, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 12, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 286, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 164, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 56, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 2153, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 72, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 2038, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 74, 2024-04-25T02:20:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 80, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 21358422, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 7, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 74, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 44, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 4, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 40, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 30, 2024-04-25T02:20:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 17, 2024-04-25T02:20:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 39325, 2024-04-25T02:20:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 11839, 2024-04-25T02:20:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:20:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:20:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:20:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 47629, 2024-04-25T02:20:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 6552, 2024-04-25T02:20:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 603, 2024-04-25T02:20:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 14083 2024-04-25T02:20:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 481, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 358, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 231, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 358, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 10193, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 33, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 3731, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 3698, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-04-25T02:20:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 4, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 63, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 2, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-04-25T02:20:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 60716, 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 60190, 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 60190, 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 526, 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 659, 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 205, 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 203, 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 203, 2024-04-25T02:20:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 2, 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 1, 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 2, 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 362484650, 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 366118250, 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 41335, 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 136.886487954, 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.003311636 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1642, 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 7655, 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 6013, 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 96, 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 525, 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 429, 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 1978, 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 13808, 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 11830, 2024-04-25T02:20:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 3, 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 4589, 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 4586, 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 82504, 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 231804 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 1048997, 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 30417, 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 1000733, 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 450, 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 60400, 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-04-25T02:20:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 4321, 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 15.632164083, 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.003617719 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 896917, 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 388.404635559, 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000433044 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 36120, 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 226.915369892, 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.006282263 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-04-25T02:20:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-04-25T02:20:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 26, 2024-04-25T02:20:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.007668295, 2024-04-25T02:20:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000294934 2024-04-25T02:20:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-04-25T02:20:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 119, 2024-04-25T02:20:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.704753364, 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.005922297 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 26, 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 10.292431281, 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.395862741 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-04-25T02:20:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2561, 2024-04-25T02:20:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 4.776007635, 2024-04-25T02:20:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001864899 2024-04-25T02:20:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-04-25T02:20:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 10932, 2024-04-25T02:20:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 43.358474644, 2024-04-25T02:20:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.003966197 2024-04-25T02:20:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-04-25T02:20:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 725, 2024-04-25T02:20:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 124.488061321, 2024-04-25T02:20:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.171707670 2024-04-25T02:20:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-04-25T02:20:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-04-25T02:20:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 44, 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.325864592, 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.007406013 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 19, 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 2.298281556, 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.120962187 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-04-25T02:20:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 4, 2024-04-25T02:20:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.020231690, 2024-04-25T02:20:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.005057922 2024-04-25T02:20:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-04-25T02:20:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-04-25T02:20:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 20, 2024-04-25T02:20:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.009222249, 2024-04-25T02:20:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000461112 2024-04-25T02:20:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-04-25T02:20:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-04-25T02:20:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 7, 2024-04-25T02:20:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.020658810, 2024-04-25T02:20:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002951258 2024-04-25T02:20:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-04-25T02:20:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-04-25T02:20:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3698, 2024-04-25T02:20:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 10.241253643, 2024-04-25T02:20:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002769403 2024-04-25T02:20:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:20:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:20:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-04-25T02:20:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-04-25T02:20:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 2, 2024-04-25T02:20:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 5, 2024-04-25T02:20:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 3, 2024-04-25T02:20:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 2, 2024-04-25T02:20:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-04-25T02:20:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 56560, 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 28280, 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 1532, 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 240, 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 240, 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 591127, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 192, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 88, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 1, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 7936, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 165, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 5348932, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 24067, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-04-25T02:20:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-04-25T02:20:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-04-25T02:20:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 53744, 2024-04-25T02:20:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 376248187, 2024-04-25T02:20:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 37, 2024-04-25T02:20:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 53744, 2024-04-25T02:20:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-04-25T02:20:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 53744, 2024-04-25T02:20:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 275.926631593, 2024-04-25T02:20:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.005134091 2024-04-25T02:20:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 53744, 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 63309 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 53744, 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 1065, 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 52679, 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 2219, 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 1640, 2024-04-25T02:20:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 959, 2024-04-25T02:20:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 42290, 2024-04-25T02:20:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 984, 2024-04-25T02:20:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-04-25T02:20:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-04-25T02:20:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-04-25T02:20:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 2, 2024-04-25T02:20:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 4381, 2024-04-25T02:20:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-04-25T02:20:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-04-25T02:20:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-04-25T02:20:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 106, 2024-04-25T02:20:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 3280, 2024-04-25T02:20:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:20:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-04-25T02:20:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:20:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-04-25T02:20:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-04-25T02:20:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-04-25T02:20:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:20:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-04-25T02:20:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-04-25T02:20:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-04-25T02:20:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-04-25T02:20:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-04-25T02:20:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-04-25T02:20:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-04-25T02:20:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-04-25T02:20:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 129, 2024-04-25T02:20:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-04-25T02:20:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 43, 2024-04-25T02:20:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 12, 2024-04-25T02:20:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 12, 2024-04-25T02:20:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-04-25T02:20:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-04-25T02:20:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 6199, 2024-04-25T02:20:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 56, 2024-04-25T02:20:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 1193 2024-04-25T02:20:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-04-25T02:20:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2024-04-25T02:20:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:20:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 3 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 6, 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 6, 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 3698, 2024-04-25T02:20:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 3698, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 1894993, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 415423885, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 1894993, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 1894993, 2024-04-25T02:20:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 415423885, 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 53744, 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 376625097, 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 53744, 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 376625097, 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 53744, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 53744, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 53744, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 53744, 2024-04-25T02:20:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 3698, 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 377346, 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 3698, 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 955, 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 377346, 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x5599e9c5a500": { 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 60716, 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 361923946, 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 60716, 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 41335, 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 361923946, 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:19.079 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-25T02:20:19.922 INFO:tasks.check_counter:Daemon mds.b mds.exported=56 2024-04-25T02:20:19.922 INFO:tasks.check_counter:Daemon mds.b mds.imported=72 2024-04-25T02:20:19.922 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=481 2024-04-25T02:20:19.922 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=358 2024-04-25T02:20:19.922 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=7 2024-04-25T02:20:19.923 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=26 2024-04-25T02:20:19.923 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=19 2024-04-25T02:20:19.923 DEBUG:tasks.check_counter:Getting stats from e 2024-04-25T02:20:19.923 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-25T02:20:20.175 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T02:20:20.612 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:20 smithi040 ceph-mon[32418]: pgmap v817: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-25T02:20:20.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:20 smithi134 ceph-mon[33098]: pgmap v817: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-25T02:20:20.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:20 smithi031 ceph-mon[29204]: pgmap v817: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-25T02:20:22.460 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-25T02:20:22.460 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:22.460 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2743, 2024-04-25T02:20:22.460 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2745, 2024-04-25T02:20:22.460 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 122440231, 2024-04-25T02:20:22.460 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 872284, 2024-04-25T02:20:22.460 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 8, 2024-04-25T02:20:22.460 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-04-25T02:20:22.460 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.250982509, 2024-04-25T02:20:22.460 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.063442370, 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.108636214, 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.061624274, 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2743, 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.065988742, 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000024057 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:20:22.461 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1788, 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 1763, 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 115223121, 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 658689, 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 6, 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.201924951, 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.040966398, 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.114753481, 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.037155855, 2024-04-25T02:20:22.462 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1762, 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.044131067, 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000025046 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 443296, 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 222336 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:22.463 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1078, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 1340, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 96747147, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 898552, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 6, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.145543262, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.026671744, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.087001073, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.023402559, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1339, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.035579914, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000026572 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.464 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 557328 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-25T02:20:22.465 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 8893, 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.258562540, 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000029074 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:22.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 0, 2024-04-25T02:20:22.467 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 5436, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 2988, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 2448, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 1168, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 0, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 0, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 0, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 2, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 0, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 0, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2024-04-25T02:20:22.468 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 80, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 21358422, 2024-04-25T02:20:22.469 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 7, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:20:22.470 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:20:22.471 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-04-25T02:20:22.472 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 0, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 52859, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 0, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-04-25T02:20:22.473 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 2, 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 96, 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 362484650, 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 366150188, 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 366146637, 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 52859 2024-04-25T02:20:22.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1637, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 6930, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 5293, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 78, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 178, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 100, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 5436, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 7591, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 2155, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 0, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 0, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 0, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 61736, 2024-04-25T02:20:22.475 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 215396 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 0, 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-04-25T02:20:22.476 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-04-25T02:20:22.477 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.480 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-04-25T02:20:22.480 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.480 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.480 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.480 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.482 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.485 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.486 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-04-25T02:20:22.487 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 2, 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 2, 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 0, 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 0, 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 2655, 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:20:22.488 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:22.489 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:22.490 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 239678, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 77, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-25T02:20:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 7936, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 165, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 7513444, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 31819, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 5188, 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 26, 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 4, 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 5188, 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 5188, 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 4.000951379, 2024-04-25T02:20:22.493 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000771193 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 5188, 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 5189 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 5188, 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 5188, 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 0, 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 2601, 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 0, 2024-04-25T02:20:22.494 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 2584, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 2, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-04-25T02:20:22.495 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-04-25T02:20:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 129, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 38, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 11, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 11, 2024-04-25T02:20:22.497 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 0, 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 2, 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 0 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 0, 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-04-25T02:20:22.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 5605, 2024-04-25T02:20:22.499 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 333881904, 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 5605, 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 5605, 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 333881904, 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:22.500 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 5188, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 332536557, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 5188, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 332536557, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.501 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 5188, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 5188, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 5188, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 5188, 2024-04-25T02:20:22.502 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:22.503 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x56199bb6a6e0": { 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:22.504 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:22.505 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-25T02:20:22.505 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:22 smithi040 ceph-mon[32418]: pgmap v818: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-04-25T02:20:22.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:22 smithi134 ceph-mon[33098]: pgmap v818: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-04-25T02:20:22.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:22 smithi031 ceph-mon[29204]: pgmap v818: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-04-25T02:20:23.201 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-25T02:20:23.201 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-25T02:20:23.201 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-25T02:20:23.201 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-04-25T02:20:23.202 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=7 2024-04-25T02:20:23.202 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:20:23.202 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:20:23.202 DEBUG:tasks.check_counter:Getting stats from h 2024-04-25T02:20:23.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-25T02:20:23.452 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T02:20:24.608 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:24 smithi040 ceph-mon[32418]: pgmap v819: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-25T02:20:24.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:24 smithi134 ceph-mon[33098]: pgmap v819: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-25T02:20:24.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:24 smithi031 ceph-mon[29204]: pgmap v819: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-25T02:20:25.430 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-25T02:20:25.430 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:25.430 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 685652, 2024-04-25T02:20:25.430 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 737121, 2024-04-25T02:20:25.430 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 230321220, 2024-04-25T02:20:25.430 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 549363409, 2024-04-25T02:20:25.430 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 20, 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 35.422282040, 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 14.313199317, 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 25.873838390, 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.437679512, 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 737121, 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 12.557464637, 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000017035 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1374292, 2024-04-25T02:20:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.206588027, 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000150 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1922400, 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 313168 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 146434, 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 148879, 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 46611422, 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 272248628, 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 16, 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 7, 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 8.343378339, 2024-04-25T02:20:25.432 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 3.243567213, 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 4.948339765, 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.831843644, 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 148876, 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 3.060596992, 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000020558 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 243153, 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.040152132, 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000165 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 4160, 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 135184 2024-04-25T02:20:25.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 366314, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 350305, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 112911730, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 227473596, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 11, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 7, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 21.059916393, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 9.291835021, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 14.167714366, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.650946293, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 350303, 2024-04-25T02:20:25.434 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 6.241739523, 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000017818 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 680763, 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.100826318, 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000148 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 646512 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-25T02:20:25.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 62142, 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 47.582351891, 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000765703 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 30913, 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 2.287679127, 2024-04-25T02:20:25.436 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000074003 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 459984, 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 457521, 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 457521, 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 352.297544221, 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000770013 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 2463, 2024-04-25T02:20:25.437 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 10, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 51, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 5, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 1961, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 268, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 1663, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 30, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 1578, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 823, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 1, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 1, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 6, 2024-04-25T02:20:25.438 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 903447, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 831458, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 2459, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 375, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 5, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 509, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 15, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 449, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 8, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 357, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 39, 2024-04-25T02:20:25.439 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 80, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 21358422, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 7, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 39, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 39, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 29274, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 42233, 2024-04-25T02:20:25.440 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 50372, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 15470, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 1193, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 51854 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 2178, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 4592, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 2373, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 4844, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 2597, 2024-04-25T02:20:25.441 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 20, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 14431, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 14411, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:20:25.442 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 15, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 5, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 103782, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 25605, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 24581, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 79201, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 1024, 2024-04-25T02:20:25.443 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 159, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 31, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 30, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 129, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 7, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 1, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 34324314, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 452693011, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 30535, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 78.237617200, 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002562227 2024-04-25T02:20:25.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1672, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 16704, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 15032, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 79, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 2655, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 2576, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 1975, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 25227, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 23252, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 1, 2024-04-25T02:20:25.445 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 14529, 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 14528, 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 91644, 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 223612 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 535414, 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 644, 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 459984, 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 1012, 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 13438, 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 14275, 2024-04-25T02:20:25.446 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 35.048816626, 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002455258 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 361970, 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 141.023970421, 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000389601 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.447 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 21101, 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 12.991080492, 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000615661 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.450 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 8687, 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 6.865042065, 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000790266 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 33862, 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 88.946810674, 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002626744 2024-04-25T02:20:25.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2915, 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 30.829840902, 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.010576274 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 20, 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.185385648, 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.009269282 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.453 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-04-25T02:20:25.454 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 280, 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.112871693, 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000403113 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-04-25T02:20:25.455 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 14411, 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 36.293725700, 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002518473 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.456 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 1, 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 4, 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 3, 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 1, 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 52277, 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 52277, 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 1558, 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.457 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:25.458 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:25.459 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 469203, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 167, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-25T02:20:25.460 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 7936, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 165, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 5360987, 2024-04-25T02:20:25.461 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 22404, 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 52316, 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 451250133, 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 28, 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 52316, 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-04-25T02:20:25.462 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 52316, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 131.818391338, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002519657 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 52316, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 54912 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 52316, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 2291, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 50025, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 0, 2024-04-25T02:20:25.463 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 21, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 2242, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 32777, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 1262, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 5, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 14536, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 49, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 42, 2024-04-25T02:20:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-04-25T02:20:25.465 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 129, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-04-25T02:20:25.466 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 42, 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 12, 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 12, 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 2670, 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 20, 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 1288 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-04-25T02:20:25.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 1 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 6, 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 6, 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 14411, 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 14411, 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:25.468 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 1198396, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 281955904, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 1198396, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 1198396, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 281955904, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.469 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 52316, 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 452715780, 2024-04-25T02:20:25.470 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 52316, 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 452715780, 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:25.471 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 52316, 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 52316, 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 52316, 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 52316, 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:25.472 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 14411, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1465807, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 14411, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2242, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1465807, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.473 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x55d73e4fc500": { 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 103782, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 448498707, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 103782, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 30535, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 448498707, 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:25.474 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:25.475 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-25T02:20:26.234 INFO:tasks.check_counter:Daemon mds.h mds.exported=15 2024-04-25T02:20:26.234 INFO:tasks.check_counter:Daemon mds.h mds.imported=8 2024-04-25T02:20:26.234 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=2178 2024-04-25T02:20:26.234 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=4592 2024-04-25T02:20:26.234 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=7 2024-04-25T02:20:26.234 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:20:26.234 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:20:26.234 DEBUG:tasks.check_counter:Getting stats from k 2024-04-25T02:20:26.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-25T02:20:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T02:20:26.608 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:26 smithi040 ceph-mon[32418]: pgmap v820: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2024-04-25T02:20:26.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:26 smithi134 ceph-mon[33098]: pgmap v820: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2024-04-25T02:20:26.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:26 smithi031 ceph-mon[29204]: pgmap v820: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2024-04-25T02:20:28.662 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:28 smithi040 ceph-mon[32418]: pgmap v821: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T02:20:28.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:28 smithi134 ceph-mon[33098]: pgmap v821: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T02:20:28.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:28 smithi031 ceph-mon[29204]: pgmap v821: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T02:20:28.750 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1747, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 1749, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 184716975, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 555610, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 9, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.215822100, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.036931582, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.128797972, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.038417313, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1747, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.041738357, 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000023891 2024-04-25T02:20:28.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1991, 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2336, 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 176664676, 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 1405579, 2024-04-25T02:20:28.752 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 9, 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 7, 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.259477609, 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.046253166, 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.151016716, 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.046568787, 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2333, 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.065717064, 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000028168 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 773072 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 3231, 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 3188, 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 90797134, 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 1144056, 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.324981393, 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.082779699, 2024-04-25T02:20:28.754 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.154418453, 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.078242561, 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3188, 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.092636615, 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000029057 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1898352, 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 290544 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.755 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 10866, 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.231486283, 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000021303 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-25T02:20:28.756 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-04-25T02:20:28.757 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 0, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 16367, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 2186, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 14181, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 13244, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 0, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 0, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 0, 2024-04-25T02:20:28.758 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 1, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-04-25T02:20:28.759 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2024-04-25T02:20:28.760 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-04-25T02:20:28.761 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-04-25T02:20:28.762 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 0, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 103782, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 0, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-04-25T02:20:28.763 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 129, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 10, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 34324314, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 452693011, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 452689712, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 103782 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 1661, 2024-04-25T02:20:28.764 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 16512, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 14851, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 59, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 99, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 40, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 16367, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 16835, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 468, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 0, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 0, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 0, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 75804, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 215420 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2024-04-25T02:20:28.765 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 0, 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-04-25T02:20:28.766 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.767 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.768 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-04-25T02:20:28.769 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.769 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.769 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.769 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-04-25T02:20:28.769 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.769 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.769 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.770 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.771 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 1, 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 4, 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 3, 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 3, 2024-04-25T02:20:28.776 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 4867, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:28.777 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:28.778 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 243418, 2024-04-25T02:20:28.779 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 91, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 7936, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 165, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-25T02:20:28.780 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 15416190, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 73357, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 6420, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 21, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 3, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 6420, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 6420, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 4.740942002, 2024-04-25T02:20:28.781 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000738464 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 6420, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 6421 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 6420, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 6420, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 0, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 3396, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 0, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 3022, 2024-04-25T02:20:28.782 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 1, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-04-25T02:20:28.783 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-04-25T02:20:28.784 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 129, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 42, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 11, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 12, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 1, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-04-25T02:20:28.785 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 0, 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 2, 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 0 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 0, 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-04-25T02:20:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 6965, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 451521296, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 6965, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 6965, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 451521296, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:28.788 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 6420, 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 448511367, 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 6420, 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 448511367, 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:28.789 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 6420, 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 6420, 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 6420, 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 6420, 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:28.790 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x55e7a53686e0": { 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-25T02:20:28.791 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-25T02:20:28.792 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-25T02:20:28.793 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:28.793 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-25T02:20:28.793 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-25T02:20:29.589 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-25T02:20:29.589 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-25T02:20:29.589 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-04-25T02:20:29.589 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-04-25T02:20:29.590 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-04-25T02:20:29.590 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:20:29.590 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:20:29.590 DEBUG:tasks.check_counter:Getting stats from c 2024-04-25T02:20:29.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-25T02:20:29.869 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T02:20:30.662 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:30 smithi040 ceph-mon[32418]: pgmap v822: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T02:20:30.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:30 smithi134 ceph-mon[33098]: pgmap v822: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T02:20:30.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:30 smithi031 ceph-mon[29204]: pgmap v822: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T02:20:31.965 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-25T02:20:31.965 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:31.965 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 734128, 2024-04-25T02:20:31.965 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 763397, 2024-04-25T02:20:31.965 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 249465399, 2024-04-25T02:20:31.965 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 498966496, 2024-04-25T02:20:31.965 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 16, 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 3, 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 38.452316164, 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 16.447269843, 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 27.514544696, 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.349714951, 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 763398, 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 18.653693329, 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000024435 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:31.966 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 1459533, 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.228271267, 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000156 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 129069, 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 129595, 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 42323757, 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 193653982, 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 18, 2024-04-25T02:20:31.967 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 6, 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 8.977145007, 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 4.445163978, 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 5.161759182, 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.374999835, 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 129592, 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 3.479927088, 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000026852 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 234327, 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.038310862, 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000163 2024-04-25T02:20:31.968 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 776928 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 225632, 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 184469, 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 68154592, 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 164039578, 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 16, 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 7, 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 12.173380070, 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 4.831715804, 2024-04-25T02:20:31.969 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 8.810531566, 2024-04-25T02:20:31.970 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.445018444, 2024-04-25T02:20:31.970 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:31.970 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 184466, 2024-04-25T02:20:31.970 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 4.474547625, 2024-04-25T02:20:31.970 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000024256 2024-04-25T02:20:31.970 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.970 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:31.970 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 386508, 2024-04-25T02:20:31.970 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.065485373, 2024-04-25T02:20:31.970 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000169 2024-04-25T02:20:31.970 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 1919424, 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 324400 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: "cct": { 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: "total_workers": 1, 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: "unhealthy_workers": 0 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: "finisher-MDSRank": { 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: "queue_len": 0, 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: "complete_latency": { 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 31919, 2024-04-25T02:20:31.971 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 38.517950187, 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.001206740 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: "finisher-PurgeQueue": { 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: "queue_len": 0, 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: "complete_latency": { 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 19706, 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 1.881618739, 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000095484 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mds": { 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "request": 420111, 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "reply": 418127, 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "reply_latency": { 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 418127, 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 310.033722061, 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000741482 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "slow_reply": 0, 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "forward": 1984, 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_fetch_complete": 6, 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_fetch_keys": 0, 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_commit": 43, 2024-04-25T02:20:31.973 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_split": 4, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_merge": 0, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes": 1872, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_top": 251, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_bottom": 1588, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_pin_tail": 33, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_pinned": 1513, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_expired": 379, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_with_caps": 1, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "caps": 1, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "subtrees": 6, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse": 813405, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_hit": 775416, 2024-04-25T02:20:31.974 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_forward": 1981, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_discover": 392, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_dir_fetch": 3, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_remote_ino": 0, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_lock": 483, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "load_cent": 0, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "q": 0, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "exported": 4, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "exported_inodes": 146, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "imported": 4, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "imported_inodes": 317, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_dir_fetch": 0, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_backtrace_fetch": 0, 2024-04-25T02:20:31.975 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_peer_discover": 0, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rfiles": 80, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rbytes": 21358422, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rsnaps": 7, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_set_tag": 0, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_file_inodes": 0, 2024-04-25T02:20:31.976 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_inode_file_caps": 0, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_revoke": 21504, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_grant": 27869, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_caps": 36376, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_caps_dirty": 11149, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_cap_release": 672, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "process_request_cap_release": 34809 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cache": { 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_update": 333, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_update_receipt": 6035, 2024-04-25T02:20:31.977 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_try_discover": 3072, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_send_discover": 6463, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_handle_discover": 411, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays": 20, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays_delayed": 0, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_created": 9748, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_enqueued": 9728, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_reintegrated": 0, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_migrated": 0, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_processing": 0, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:20:31.978 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_started": 0, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_completed": 0, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_exportdir": 4, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_flush": 0, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_fragmentdir": 4, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_fragstats": 0, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_inodestats": 0 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log": { 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "evlrg": 0, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "evadd": 62725, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "evex": 0, 2024-04-25T02:20:31.979 INFO:teuthology.orchestra.run.smithi134.stdout: "evtrm": 0, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "ev": 62725, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "evexg": 0, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "evexd": 0, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "segadd": 100, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "segex": 0, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "segtrm": 0, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "seg": 100, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "segmjr": 7, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "segexg": 0, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "segexd": 0, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "expos": 4194304, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "wrpos": 342799670, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "rdpos": 0, 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "jlat": { 2024-04-25T02:20:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 15985, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 58.762594754, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003676108 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "replayed": 0 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_mem": { 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "ino": 1581, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "ino+": 11546, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "ino-": 9965, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "dir": 73, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "dir+": 3457, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "dir-": 3384, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "dn": 1872, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "dn+": 17699, 2024-04-25T02:20:31.981 INFO:teuthology.orchestra.run.smithi134.stdout: "dn-": 15827, 2024-04-25T02:20:31.982 INFO:teuthology.orchestra.run.smithi134.stdout: "cap": 1, 2024-04-25T02:20:31.982 INFO:teuthology.orchestra.run.smithi134.stdout: "cap+": 9753, 2024-04-25T02:20:31.982 INFO:teuthology.orchestra.run.smithi134.stdout: "cap-": 9752, 2024-04-25T02:20:31.982 INFO:teuthology.orchestra.run.smithi134.stdout: "rss": 83392, 2024-04-25T02:20:31.982 INFO:teuthology.orchestra.run.smithi134.stdout: "heap": 223612 2024-04-25T02:20:31.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.982 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_server": { 2024-04-25T02:20:31.982 INFO:teuthology.orchestra.run.smithi134.stdout: "dispatch_client_request": 463810, 2024-04-25T02:20:31.982 INFO:teuthology.orchestra.run.smithi134.stdout: "dispatch_server_request": 23, 2024-04-25T02:20:31.982 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_request": 420111, 2024-04-25T02:20:31.982 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_session": 868, 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_peer_request": 2756, 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "req_create_latency": { 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 9589, 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 28.863195349, 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003010031 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getattr_latency": { 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 351744, 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 148.646341972, 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000422598 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getfilelock_latency": { 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.983 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: "req_link_latency": { 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookup_latency": { 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 14727, 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 10.311928382, 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000700205 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookuphash_latency": { 2024-04-25T02:20:31.984 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupino_latency": { 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupname_latency": { 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.985 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupparent_latency": { 2024-04-25T02:20:31.986 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.986 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.986 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.986 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.986 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupsnap_latency": { 2024-04-25T02:20:31.986 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.986 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.987 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.987 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.987 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lssnap_latency": { 2024-04-25T02:20:31.987 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.987 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.987 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.987 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.987 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mkdir_latency": { 2024-04-25T02:20:31.987 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.988 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.988 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.988 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.988 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mknod_latency": { 2024-04-25T02:20:31.988 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.988 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.988 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.988 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.988 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mksnap_latency": { 2024-04-25T02:20:31.988 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: "req_open_latency": { 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 5896, 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 6.045877211, 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.001025420 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: "req_readdir_latency": { 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 24243, 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 80.650049620, 2024-04-25T02:20:31.989 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003326735 2024-04-25T02:20:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rename_latency": { 2024-04-25T02:20:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 1908, 2024-04-25T02:20:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 7.278309879, 2024-04-25T02:20:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003814627 2024-04-25T02:20:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "req_renamesnap_latency": { 2024-04-25T02:20:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "req_snapdiff_latency": { 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmdir_latency": { 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 20, 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.157527779, 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.007876388 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmsnap_latency": { 2024-04-25T02:20:31.992 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.992 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.992 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.992 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmxattr_latency": { 2024-04-25T02:20:31.992 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.992 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.992 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.992 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setattr_latency": { 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setdirlayout_latency": { 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setfilelock_latency": { 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 272, 2024-04-25T02:20:31.993 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.115566304, 2024-04-25T02:20:31.994 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000424876 2024-04-25T02:20:31.994 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.994 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setlayout_latency": { 2024-04-25T02:20:31.994 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.994 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.994 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.994 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.994 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setxattr_latency": { 2024-04-25T02:20:31.994 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.994 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.994 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "req_symlink_latency": { 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "req_unlink_latency": { 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 9728, 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 27.964925565, 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.002874683 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getvxattr_latency": { 2024-04-25T02:20:31.995 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_sessions": { 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: "session_count": 2, 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: "session_add": 4, 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: "session_remove": 2, 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: "sessions_open": 2, 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: "sessions_stale": 0, 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: "total_load": 50683, 2024-04-25T02:20:31.996 INFO:teuthology.orchestra.run.smithi134.stdout: "average_load": 25341, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "avg_session_uptime": 1543, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "mdthresh_evicted": 0 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "mempool": { 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "bloom_filter_items": 0, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:31.997 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_items": 0, 2024-04-25T02:20:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_anon_bytes": 484222, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_anon_items": 166, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_meta_items": 0, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_bytes": 0, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_items": 0, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_bytes": 7936, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_items": 165, 2024-04-25T02:20:32.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "pgmap_items": 0, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_co_bytes": 5055151, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_co_items": 21749, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_1_items": 0, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_2_items": 0 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter": { 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "op_active": 0, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "op_laggy": 0, 2024-04-25T02:20:32.001 INFO:teuthology.orchestra.run.smithi134.stdout: "op_send": 29808, 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "op_send_bytes": 340500751, 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "op_resend": 33, 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "op_reply": 29808, 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "op_latency": { 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 29808, 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 100.213739318, 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003361974 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "op_inflight": 0, 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "oplen_avg": { 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 29808, 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 31240 2024-04-25T02:20:32.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "op": 29808, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "op_r": 1275, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "op_w": 28533, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "op_rmw": 0, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "op_pg": 0, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_stat": 0, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_create": 10, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_read": 1269, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_write": 17254, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_writefull": 691, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_writesame": 0, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_append": 0, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_zero": 0, 2024-04-25T02:20:32.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_truncate": 4, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_delete": 9820, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_mapext": 0, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_sparse_read": 0, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_clonerange": 0, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_getxattr": 6, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_setxattr": 20, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_rmxattr": 0, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_call": 0, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_watch": 0, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_notify": 0, 2024-04-25T02:20:32.004 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_pgls": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_other": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_active": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_send": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_resend": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_ping": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_active": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_send": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_resend": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_active": 0, 2024-04-25T02:20:32.005 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_send": 0, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_resend": 0, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_active": 0, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_send": 0, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_resend": 0, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "command_active": 0, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "command_send": 0, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "command_resend": 0, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "map_epoch": 129, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "map_full": 0, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "map_inc": 42, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_sessions": 12, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_session_open": 15, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_session_close": 3, 2024-04-25T02:20:32.006 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_laggy": 0, 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_wr": 1465, 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_rd": 12, 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_del": 689 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: "oft": { 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_objs": 1, 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_updates": 0, 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_removes": 0 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: "osd-slow-ops": { 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: "slow_ops_count": 0 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:32.007 INFO:teuthology.orchestra.run.smithi134.stdout: "purge_queue": { 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_ops": 0, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_ops_high_water": 6, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing": 0, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_high_water": 6, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executed_ops": 9728, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executed": 9728, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_item_in_journal": 0 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 104857600, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 1088825, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 261916069, 2024-04-25T02:20:32.008 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 1088825, 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 1088825, 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 261916069, 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-objecter_bytes": { 2024-04-25T02:20:32.009 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 104857600, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 0, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 0, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 29808, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 341491791, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 29808, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 341491791, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:32.010 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-objecter_ops": { 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 1024, 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 0, 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 0, 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 29808, 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 29808, 2024-04-25T02:20:32.011 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 29808, 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 29808, 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 3758096384, 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 9728, 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 991136, 2024-04-25T02:20:32.012 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 9728, 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 1269, 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 991136, 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-write_buf_throttle-0x562d1b800500": { 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 3758096384, 2024-04-25T02:20:32.013 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 62725, 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 338605366, 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 62725, 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 15985, 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 338605366, 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:32.014 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:32.015 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-25T02:20:32.662 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:32 smithi040 ceph-mon[32418]: pgmap v823: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T02:20:32.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:32 smithi134 ceph-mon[33098]: pgmap v823: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T02:20:32.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:32 smithi031 ceph-mon[29204]: pgmap v823: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T02:20:32.828 INFO:tasks.check_counter:Daemon mds.c mds.exported=4 2024-04-25T02:20:32.828 INFO:tasks.check_counter:Daemon mds.c mds.imported=4 2024-04-25T02:20:32.829 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=333 2024-04-25T02:20:32.829 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=6035 2024-04-25T02:20:32.829 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=7 2024-04-25T02:20:32.829 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:20:32.829 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:20:32.829 DEBUG:tasks.check_counter:Getting stats from f 2024-04-25T02:20:32.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-25T02:20:33.079 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T02:20:34.662 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:34 smithi040 ceph-mon[32418]: pgmap v824: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T02:20:34.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:34 smithi134 ceph-mon[33098]: pgmap v824: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T02:20:34.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:34 smithi031 ceph-mon[29204]: pgmap v824: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T02:20:35.211 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-25T02:20:35.211 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:35.211 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 352686, 2024-04-25T02:20:35.211 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 309934, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 111399215, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 198257800, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 11, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 5, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 17.399568746, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 6.843525573, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 13.069211451, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.400097877, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 309932, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 7.543267015, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000024338 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:35.212 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 635666, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.103374071, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000162 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 613440, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 644751, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 208006323, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 484189453, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 15, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 7, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 33.522445985, 2024-04-25T02:20:35.213 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 14.480247121, 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 23.681107125, 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.382576190, 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 644750, 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 14.760036963, 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000022892 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 1220535, 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.198922065, 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000162 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.214 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 1907264, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 294304 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 129653, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 129814, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 42659621, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 179205453, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 19, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 6, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 9.149724495, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 4.575073109, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 5.220340867, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.360109353, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 129812, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 3.664781607, 2024-04-25T02:20:35.215 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000028231 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 234369, 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.041138757, 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000175 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 778368 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: "cct": { 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: "total_workers": 1, 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: "unhealthy_workers": 0 2024-04-25T02:20:35.216 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "finisher-MDSRank": { 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "queue_len": 0, 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "complete_latency": { 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 34008, 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 39.666803043, 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.001166396 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "finisher-PurgeQueue": { 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "queue_len": 0, 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "complete_latency": { 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 19742, 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 1.920762150, 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000097293 2024-04-25T02:20:35.217 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: "mds": { 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: "request": 421090, 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: "reply": 419265, 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: "reply_latency": { 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 419265, 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 319.837890161, 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000762853 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: "slow_reply": 0, 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: "forward": 1825, 2024-04-25T02:20:35.218 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_fetch_complete": 13, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_fetch_keys": 0, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_commit": 51, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_split": 5, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_merge": 0, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes": 1891, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_top": 253, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_bottom": 1602, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_pin_tail": 36, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_pinned": 1528, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_expired": 664, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_with_caps": 0, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "caps": 0, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "subtrees": 6, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse": 815316, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_hit": 777674, 2024-04-25T02:20:35.219 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_forward": 1825, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_discover": 332, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_dir_fetch": 9, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_remote_ino": 0, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_lock": 488, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "load_cent": 0, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "q": 0, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "exported": 14, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "exported_inodes": 377, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "imported": 7, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "imported_inodes": 312, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_dir_fetch": 0, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_backtrace_fetch": 0, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_peer_discover": 0, 2024-04-25T02:20:35.220 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rfiles": 4, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rbytes": 234, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rsnaps": 7, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_set_tag": 0, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_file_inodes": 0, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_inode_file_caps": 0, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_revoke": 21299, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_grant": 28234, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:20:35.221 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_flushsnap_ack": 5, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_caps": 37605, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_caps_dirty": 12238, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_cap_release": 682, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "process_request_cap_release": 35425 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cache": { 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_update": 1244, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_update_receipt": 5417, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_try_discover": 2763, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_send_discover": 5710, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_handle_discover": 1465, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays": 20, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays_delayed": 0, 2024-04-25T02:20:35.222 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_created": 9736, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_enqueued": 9716, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_reintegrated": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_migrated": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_processing": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_started": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_completed": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_exportdir": 14, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_flush": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_fragmentdir": 5, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_fragstats": 0, 2024-04-25T02:20:35.223 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_inodestats": 0 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log": { 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "evlrg": 0, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "evadd": 66487, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "evex": 0, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "evtrm": 0, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "ev": 66487, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "evexg": 0, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "evexd": 0, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "segadd": 103, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "segex": 0, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "segtrm": 0, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "seg": 103, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "segmjr": 7, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "segexg": 0, 2024-04-25T02:20:35.224 INFO:teuthology.orchestra.run.smithi134.stdout: "segexd": 0, 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "expos": 4194304, 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "wrpos": 342161039, 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "rdpos": 0, 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "jlat": { 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 17063, 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 62.589664211, 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003668151 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "replayed": 0 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_mem": { 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "ino": 1600, 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "ino+": 11808, 2024-04-25T02:20:35.225 INFO:teuthology.orchestra.run.smithi134.stdout: "ino-": 10208, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "dir": 80, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "dir+": 3087, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "dir-": 3007, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "dn": 1891, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "dn+": 17532, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "dn-": 15641, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "cap": 0, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "cap+": 9767, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "cap-": 9767, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "rss": 82792, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "heap": 223612 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_server": { 2024-04-25T02:20:35.226 INFO:teuthology.orchestra.run.smithi134.stdout: "dispatch_client_request": 466447, 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "dispatch_server_request": 373, 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_request": 421090, 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_session": 1039, 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_peer_request": 3453, 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "req_create_latency": { 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 9567, 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 29.108775971, 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003042623 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getattr_latency": { 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 350214, 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 149.033203323, 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000425548 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.227 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getfilelock_latency": { 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "req_link_latency": { 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookup_latency": { 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 15330, 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 13.042021220, 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000850751 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.228 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookuphash_latency": { 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupino_latency": { 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupname_latency": { 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.230 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupparent_latency": { 2024-04-25T02:20:35.230 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.230 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.230 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.230 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupsnap_latency": { 2024-04-25T02:20:35.230 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.230 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lssnap_latency": { 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mkdir_latency": { 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mknod_latency": { 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mksnap_latency": { 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "req_open_latency": { 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 6459, 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 6.272423259, 2024-04-25T02:20:35.232 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000971113 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "req_readdir_latency": { 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 25439, 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 82.373804067, 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003238091 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rename_latency": { 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 2136, 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 9.725759702, 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.004553258 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "req_renamesnap_latency": { 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.233 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: "req_snapdiff_latency": { 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmdir_latency": { 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 20, 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.137746076, 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.006887303 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmsnap_latency": { 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.234 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmxattr_latency": { 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setattr_latency": { 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setdirlayout_latency": { 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.235 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setfilelock_latency": { 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 384, 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.192270723, 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000500705 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setlayout_latency": { 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setxattr_latency": { 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.236 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: "req_symlink_latency": { 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: "req_unlink_latency": { 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 9716, 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 29.951885820, 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003082738 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:20:35.237 INFO:teuthology.orchestra.run.smithi134.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getvxattr_latency": { 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_sessions": { 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: "session_count": 1, 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: "session_add": 1, 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: "session_remove": 0, 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: "sessions_open": 1, 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: "sessions_stale": 0, 2024-04-25T02:20:35.238 INFO:teuthology.orchestra.run.smithi134.stdout: "total_load": 50644, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "average_load": 50644, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "avg_session_uptime": 1567, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "mdthresh_evicted": 0 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "mempool": { 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "bloom_filter_items": 0, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:35.239 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:35.240 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_items": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_anon_bytes": 441494, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_anon_items": 144, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:20:35.241 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_meta_items": 0, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_bytes": 0, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_items": 0, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_bytes": 7936, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_items": 165, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "pgmap_items": 0, 2024-04-25T02:20:35.242 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_co_bytes": 5077795, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_co_items": 21427, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_1_items": 0, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_2_items": 0 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter": { 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "op_active": 0, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "op_laggy": 0, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "op_send": 30915, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "op_send_bytes": 339995912, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "op_resend": 33, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "op_reply": 30915, 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "op_latency": { 2024-04-25T02:20:35.243 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 30915, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 106.413599883, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003442134 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "op_inflight": 0, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "oplen_avg": { 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 30915, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 32396 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "op": 30915, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "op_r": 1292, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "op_w": 29623, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "op_rmw": 0, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "op_pg": 0, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_stat": 0, 2024-04-25T02:20:35.244 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_create": 10, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_read": 1279, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_write": 18342, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_writefull": 693, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_writesame": 0, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_append": 0, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_zero": 0, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_truncate": 5, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_delete": 9808, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_mapext": 0, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_sparse_read": 0, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_clonerange": 0, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_getxattr": 13, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_setxattr": 20, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_rmxattr": 0, 2024-04-25T02:20:35.245 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_call": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_watch": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_notify": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_pgls": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_other": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_active": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_send": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_resend": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_ping": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_active": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_send": 0, 2024-04-25T02:20:35.246 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_resend": 0, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_active": 0, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_send": 0, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_resend": 0, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_active": 0, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_send": 0, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_resend": 0, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "command_active": 0, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "command_send": 0, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "command_resend": 0, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "map_epoch": 129, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "map_full": 0, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "map_inc": 42, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_sessions": 12, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_session_open": 14, 2024-04-25T02:20:35.247 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_session_close": 2, 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_laggy": 0, 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_wr": 1497, 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_rd": 26, 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_del": 703 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "oft": { 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_objs": 1, 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_updates": 0, 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_removes": 0 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "osd-slow-ops": { 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "slow_ops_count": 0 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.248 INFO:teuthology.orchestra.run.smithi134.stdout: "purge_queue": { 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_ops": 0, 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_ops_high_water": 6, 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing": 0, 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_high_water": 6, 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executed_ops": 9716, 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executed": 9716, 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_item_in_journal": 0 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 104857600, 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 1095775, 2024-04-25T02:20:35.249 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 263413402, 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 1095775, 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 1095775, 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 263413402, 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.250 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-objecter_bytes": { 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 104857600, 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 0, 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 0, 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 30915, 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 340986876, 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 30915, 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 340986876, 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:35.251 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-objecter_ops": { 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 1024, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 0, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 0, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 30915, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 30915, 2024-04-25T02:20:35.252 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 30915, 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 30915, 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 3758096384, 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 9716, 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 991172, 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:35.253 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 9716, 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 1279, 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 991172, 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-write_buf_throttle-0x56135b576500": { 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 3758096384, 2024-04-25T02:20:35.254 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 66487, 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 337966735, 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 66487, 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 17063, 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 337966735, 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:35.255 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:35.256 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-25T02:20:35.946 INFO:tasks.check_counter:Daemon mds.f mds.exported=14 2024-04-25T02:20:35.946 INFO:tasks.check_counter:Daemon mds.f mds.imported=7 2024-04-25T02:20:35.946 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=1244 2024-04-25T02:20:35.946 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=5417 2024-04-25T02:20:35.946 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=7 2024-04-25T02:20:35.946 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:20:35.947 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:20:35.947 DEBUG:tasks.check_counter:Getting stats from i 2024-04-25T02:20:35.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-25T02:20:36.186 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T02:20:36.588 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:36 smithi134 ceph-mon[33098]: pgmap v825: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.6 KiB/s wr, 14 op/s 2024-04-25T02:20:36.663 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:36 smithi040 ceph-mon[32418]: pgmap v825: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.6 KiB/s wr, 14 op/s 2024-04-25T02:20:36.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:36 smithi031 ceph-mon[29204]: pgmap v825: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.6 KiB/s wr, 14 op/s 2024-04-25T02:20:37.907 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-25T02:20:37.907 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 249634, 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 239053, 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 74123139, 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 190039145, 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 13, 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 5, 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 14.268425369, 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 5.747067776, 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 9.975588515, 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.474379728, 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 239053, 2024-04-25T02:20:37.908 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 5.781351755, 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000024184 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 459053, 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.086980009, 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000189 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 1929488, 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 962384 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 669752, 2024-04-25T02:20:37.909 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 746703, 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 204184165, 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 676725248, 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 24, 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 7, 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 36.735767851, 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 15.084447454, 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 26.294693252, 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.630307281, 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 746700, 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 17.404445234, 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000023308 2024-04-25T02:20:37.910 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 1360637, 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.237490768, 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000174 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 128480 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 285899, 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 311206, 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 90439262, 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 364662984, 2024-04-25T02:20:37.911 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 9, 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 7, 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 19.107429619, 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 9.284646161, 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 11.335561351, 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.778444767, 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 311204, 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 8.132350878, 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000026131 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 546588, 2024-04-25T02:20:37.912 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.093269160, 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000170 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: "cct": { 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: "total_workers": 1, 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: "unhealthy_workers": 0 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: "finisher-MDSRank": { 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: "queue_len": 0, 2024-04-25T02:20:37.913 INFO:teuthology.orchestra.run.smithi134.stdout: "complete_latency": { 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 62697, 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 49.708455873, 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000792836 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: "finisher-PurgeQueue": { 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: "queue_len": 0, 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: "complete_latency": { 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 28429, 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 1.993704122, 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000070129 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:37.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "mds": { 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "request": 451968, 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "reply": 450729, 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "reply_latency": { 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 450729, 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 394.714016211, 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000875723 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "slow_reply": 0, 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "forward": 1239, 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_fetch_complete": 13, 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_fetch_keys": 0, 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_commit": 100, 2024-04-25T02:20:37.915 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_split": 10, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_merge": 5, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes": 2505, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_top": 681, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_bottom": 1766, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_pin_tail": 58, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_pinned": 1531, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_expired": 1323, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_with_caps": 2, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "caps": 2, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "subtrees": 6, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse": 863805, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_hit": 797947, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_forward": 1209, 2024-04-25T02:20:37.916 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_discover": 302, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_dir_fetch": 3, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_remote_ino": 0, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_lock": 458, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "load_cent": 0, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "q": 0, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "exported": 39, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "exported_inodes": 1066, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "imported": 37, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "imported_inodes": 1167, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_dir_fetch": 0, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_backtrace_fetch": 31, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_peer_discover": 0, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rfiles": 4, 2024-04-25T02:20:37.917 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rbytes": 234, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rsnaps": 7, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_backtrace_fetch": 31, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_set_tag": 0, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_file_inodes": 31, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_inode_file_caps": 1067, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_revoke": 26357, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_grant": 41394, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:20:37.918 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_flushsnap_ack": 1, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_caps": 53589, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_caps_dirty": 20894, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_cap_release": 1139, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "process_request_cap_release": 47964 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cache": { 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_update": 4819, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_update_receipt": 2303, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_try_discover": 1211, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_send_discover": 2537, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_handle_discover": 5246, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays": 20, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays_delayed": 0, 2024-04-25T02:20:37.919 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_created": 13158, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_enqueued": 13138, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_reintegrated": 0, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_migrated": 0, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_processing": 0, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_started": 0, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_completed": 0, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_exportdir": 41, 2024-04-25T02:20:37.920 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_flush": 0, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_fragmentdir": 15, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_fragstats": 0, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_inodestats": 0 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log": { 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "evlrg": 0, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "evadd": 121684, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "evex": 39174, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "evtrm": 36871, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "ev": 84813, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "evexg": 0, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "evexd": 2303, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "segadd": 174, 2024-04-25T02:20:37.921 INFO:teuthology.orchestra.run.smithi134.stdout: "segex": 46, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "segtrm": 43, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "seg": 131, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "segmjr": 7, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "segexg": 0, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "segexd": 3, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "expos": 44565852, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "wrpos": 524460745, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "rdpos": 0, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "jlat": { 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 30932, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 88.933902067, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.002875142 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.922 INFO:teuthology.orchestra.run.smithi134.stdout: "replayed": 0 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_mem": { 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "ino": 2213, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "ino+": 16483, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "ino-": 14270, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "dir": 109, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "dir+": 1504, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "dir-": 1395, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "dn": 2505, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "dn+": 24927, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "dn-": 22422, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "cap": 2, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "cap+": 13172, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "cap-": 13170, 2024-04-25T02:20:37.923 INFO:teuthology.orchestra.run.smithi134.stdout: "rss": 85628, 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: "heap": 223612 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_server": { 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: "dispatch_client_request": 529753, 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: "dispatch_server_request": 354, 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_request": 451968, 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_session": 1398, 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_peer_request": 11941, 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: "req_create_latency": { 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 12943, 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 40.324018219, 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003115507 2024-04-25T02:20:37.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getattr_latency": { 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 337529, 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 153.699582969, 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000455367 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getfilelock_latency": { 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "req_link_latency": { 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.926 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookup_latency": { 2024-04-25T02:20:37.926 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 19659, 2024-04-25T02:20:37.926 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 21.226948979, 2024-04-25T02:20:37.926 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.001079757 2024-04-25T02:20:37.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.926 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookuphash_latency": { 2024-04-25T02:20:37.926 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.926 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.926 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.927 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupino_latency": { 2024-04-25T02:20:37.927 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.927 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.927 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.927 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.927 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupname_latency": { 2024-04-25T02:20:37.927 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.927 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.927 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.927 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.927 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupparent_latency": { 2024-04-25T02:20:37.928 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.928 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.928 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.928 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupsnap_latency": { 2024-04-25T02:20:37.928 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lssnap_latency": { 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mkdir_latency": { 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.929 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mknod_latency": { 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mksnap_latency": { 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: "req_open_latency": { 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 9540, 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 8.040137539, 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000842781 2024-04-25T02:20:37.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "req_readdir_latency": { 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 54274, 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 105.930996189, 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.001951781 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rename_latency": { 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 2865, 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 17.945227848, 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.006263604 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "req_renamesnap_latency": { 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "req_snapdiff_latency": { 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmdir_latency": { 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 20, 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.216654043, 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.010832702 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmsnap_latency": { 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmxattr_latency": { 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setattr_latency": { 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 97, 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.457646232, 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.004718002 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setdirlayout_latency": { 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.933 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setfilelock_latency": { 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 664, 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.391566814, 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000589709 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setlayout_latency": { 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setxattr_latency": { 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.934 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "req_symlink_latency": { 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "req_unlink_latency": { 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 13138, 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 46.481237379, 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.003537923 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getvxattr_latency": { 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.935 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_sessions": { 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: "session_count": 2, 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: "session_add": 3, 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: "session_remove": 1, 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: "sessions_open": 2, 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: "sessions_stale": 0, 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: "total_load": 51086, 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: "average_load": 25543, 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: "avg_session_uptime": 1554, 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: "mdthresh_evicted": 0 2024-04-25T02:20:37.936 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "mempool": { 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bloom_filter_items": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:37.937 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:37.938 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_items": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_anon_bytes": 492284, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_anon_items": 176, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_meta_items": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_bytes": 0, 2024-04-25T02:20:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_items": 0, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_bytes": 7936, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_items": 165, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "pgmap_items": 0, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_co_bytes": 6864675, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_co_items": 27559, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:37.940 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_1_items": 0, 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_2_items": 0 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter": { 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "op_active": 0, 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "op_laggy": 0, 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "op_send": 51262, 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "op_send_bytes": 523400881, 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "op_resend": 33, 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "op_reply": 51262, 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "op_latency": { 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 51262, 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 140.237815133, 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.002735707 2024-04-25T02:20:37.941 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "op_inflight": 0, 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "oplen_avg": { 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 51262, 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 53716 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "op": 51262, 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "op_r": 2204, 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "op_w": 49058, 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "op_rmw": 0, 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "op_pg": 0, 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_stat": 2, 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_create": 21, 2024-04-25T02:20:37.942 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_read": 2160, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_write": 33092, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_writefull": 1203, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_writesame": 0, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_append": 0, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_zero": 0, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_truncate": 8, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_delete": 13295, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_mapext": 0, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_sparse_read": 0, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_clonerange": 0, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_getxattr": 35, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_setxattr": 42, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:20:37.943 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_rmxattr": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_call": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_watch": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_notify": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_pgls": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_other": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_active": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_send": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_resend": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_ping": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_active": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_send": 0, 2024-04-25T02:20:37.944 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_resend": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_active": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_send": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_resend": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_active": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_send": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_resend": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "command_active": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "command_send": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "command_resend": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "map_epoch": 129, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "map_full": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "map_inc": 42, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_sessions": 12, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_session_open": 12, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_session_close": 0, 2024-04-25T02:20:37.945 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_laggy": 0, 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_wr": 2617, 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_rd": 26, 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_del": 1215 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: "oft": { 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_objs": 1, 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_updates": 0, 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_removes": 0 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: "osd-slow-ops": { 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: "slow_ops_count": 0 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.946 INFO:teuthology.orchestra.run.smithi134.stdout: "purge_queue": { 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_ops": 0, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_ops_high_water": 6, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing": 0, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_high_water": 6, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executed_ops": 13138, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executed": 13138, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_item_in_journal": 0 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 104857600, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 1205281, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 260237981, 2024-04-25T02:20:37.947 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 1205281, 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 1205281, 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 260237981, 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-objecter_bytes": { 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:37.948 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 104857600, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 0, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 0, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 51262, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 524739619, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 51262, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 524739619, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.949 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-objecter_ops": { 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 1024, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 0, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 0, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 51262, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 51262, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 51262, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 51262, 2024-04-25T02:20:37.950 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 3758096384, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 13138, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 1338946, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 13138, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:37.951 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 2160, 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 1338946, 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-write_buf_throttle-0x558d0b55a500": { 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 3758096384, 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 121684, 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 520266441, 2024-04-25T02:20:37.952 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 121684, 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 30932, 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 520266441, 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:37.953 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-25T02:20:38.662 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:38 smithi040 ceph-mon[32418]: pgmap v826: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T02:20:38.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:38 smithi134 ceph-mon[33098]: pgmap v826: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T02:20:38.735 INFO:tasks.check_counter:Daemon mds.i mds.exported=39 2024-04-25T02:20:38.735 INFO:tasks.check_counter:Daemon mds.i mds.imported=37 2024-04-25T02:20:38.736 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=4819 2024-04-25T02:20:38.736 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=2303 2024-04-25T02:20:38.736 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=7 2024-04-25T02:20:38.736 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:20:38.736 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:20:38.736 DEBUG:tasks.check_counter:Getting stats from l 2024-04-25T02:20:38.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-25T02:20:38.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:38 smithi031 ceph-mon[29204]: pgmap v826: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T02:20:38.972 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T02:20:40.662 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:40 smithi040 ceph-mon[32418]: pgmap v827: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T02:20:40.676 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:40 smithi134 ceph-mon[33098]: pgmap v827: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T02:20:40.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:40 smithi031 ceph-mon[29204]: pgmap v827: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T02:20:40.801 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-25T02:20:40.801 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:20:40.801 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 3397, 2024-04-25T02:20:40.801 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 3356, 2024-04-25T02:20:40.801 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 174725511, 2024-04-25T02:20:40.801 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 1198634, 2024-04-25T02:20:40.801 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 7, 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 3, 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 0.398948373, 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 0.086986350, 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 0.212988916, 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.087526482, 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 3355, 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.103932847, 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000030978 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:40.802 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 1896176, 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 292016 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 1528, 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 1875, 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 159954486, 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 1264746, 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 9, 2024-04-25T02:20:40.803 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 7, 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 0.215338387, 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 0.040951069, 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 0.125557156, 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.036269069, 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 1873, 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.057512720, 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000030706 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.804 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 780688 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_messages": 2143, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages": 2145, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_bytes": 189293049, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_bytes": 681481, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_created_connections": 7, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_active_connections": 5, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_total_time": 0.223423884, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_send_time": 0.045691426, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_recv_time": 0.120560483, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_running_fast_dispatch_time": 0.046104773, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 2143, 2024-04-25T02:20:40.805 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.055805074, 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000026040 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: "cct": { 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: "total_workers": 1, 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: "unhealthy_workers": 0 2024-04-25T02:20:40.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: "finisher-MDSRank": { 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: "queue_len": 0, 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: "complete_latency": { 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 10929, 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.245392396, 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000022453 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: "finisher-PurgeQueue": { 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: "queue_len": 0, 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: "complete_latency": { 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.807 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: "mds": { 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: "request": 0, 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: "reply": 0, 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: "reply_latency": { 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: "slow_reply": 0, 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: "forward": 0, 2024-04-25T02:20:40.808 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_fetch_complete": 0, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_fetch_keys": 0, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_commit": 0, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_split": 0, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_merge": 0, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes": 15693, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_top": 2172, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_bottom": 13521, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_pin_tail": 0, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_pinned": 12590, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_expired": 0, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "inodes_with_caps": 0, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "caps": 0, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "subtrees": 1, 2024-04-25T02:20:40.809 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_hit": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_forward": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_discover": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_dir_fetch": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_remote_ino": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "traverse_lock": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "load_cent": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "q": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "exported": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "exported_inodes": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "imported": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "imported_inodes": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_dir_fetch": 0, 2024-04-25T02:20:40.810 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_backtrace_fetch": 0, 2024-04-25T02:20:40.811 INFO:teuthology.orchestra.run.smithi134.stdout: "openino_peer_discover": 0, 2024-04-25T02:20:40.811 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rfiles": 0, 2024-04-25T02:20:40.811 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rbytes": 0, 2024-04-25T02:20:40.811 INFO:teuthology.orchestra.run.smithi134.stdout: "root_rsnaps": 0, 2024-04-25T02:20:40.811 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T02:20:40.811 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_set_tag": 0, 2024-04-25T02:20:40.811 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:20:40.811 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:20:40.811 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:20:40.811 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:20:40.811 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_file_inodes": 0, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_inode_file_caps": 0, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_grant": 0, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_caps": 0, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_caps_dirty": 0, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_cap_release": 0, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "process_request_cap_release": 0 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.812 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cache": { 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_update": 0, 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_update_receipt": 0, 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_try_discover": 0, 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_send_discover": 0, 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "dir_handle_discover": 0, 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays": 0, 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays_delayed": 0, 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_created": 0, 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_enqueued": 0, 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_reintegrated": 0, 2024-04-25T02:20:40.813 INFO:teuthology.orchestra.run.smithi134.stdout: "strays_migrated": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_processing": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_started": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_completed": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_exportdir": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_flush": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_fragmentdir": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_fragstats": 0, 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: "ireq_inodestats": 0 2024-04-25T02:20:40.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log": { 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "evlrg": 0, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "evadd": 0, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "evex": 0, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "evtrm": 0, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "ev": 121684, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "evexg": 0, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "evexd": 0, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "segadd": 0, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "segex": 0, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "segtrm": 0, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "seg": 131, 2024-04-25T02:20:40.815 INFO:teuthology.orchestra.run.smithi134.stdout: "segmjr": 11, 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: "segexg": 0, 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: "segexd": 0, 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: "expos": 44565852, 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: "wrpos": 524460745, 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: "rdpos": 524446832, 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: "jlat": { 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: "replayed": 121684 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.816 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_mem": { 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "ino": 2260, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "ino+": 16351, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "ino-": 14091, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "dir": 96, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "dir+": 179, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "dir-": 83, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "dn": 15693, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "dn+": 16802, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "dn-": 1109, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "cap": 0, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "cap+": 0, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "cap-": 0, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "rss": 76764, 2024-04-25T02:20:40.817 INFO:teuthology.orchestra.run.smithi134.stdout: "heap": 215420 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_server": { 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "dispatch_client_request": 0, 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "dispatch_server_request": 0, 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_request": 0, 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_client_session": 0, 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "handle_peer_request": 0, 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "req_create_latency": { 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getattr_latency": { 2024-04-25T02:20:40.818 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getfilelock_latency": { 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "req_link_latency": { 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookup_latency": { 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.819 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookuphash_latency": { 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupino_latency": { 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupname_latency": { 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.820 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupparent_latency": { 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lookupsnap_latency": { 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: "req_lssnap_latency": { 2024-04-25T02:20:40.821 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.822 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.822 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.822 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mkdir_latency": { 2024-04-25T02:20:40.822 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.822 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mknod_latency": { 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: "req_mksnap_latency": { 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: "req_open_latency": { 2024-04-25T02:20:40.823 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: "req_readdir_latency": { 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rename_latency": { 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.824 INFO:teuthology.orchestra.run.smithi134.stdout: "req_renamesnap_latency": { 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: "req_snapdiff_latency": { 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmdir_latency": { 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmsnap_latency": { 2024-04-25T02:20:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: "req_rmxattr_latency": { 2024-04-25T02:20:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setattr_latency": { 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setdirlayout_latency": { 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setfilelock_latency": { 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setlayout_latency": { 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "req_setxattr_latency": { 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "req_symlink_latency": { 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.828 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: "req_unlink_latency": { 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: "req_getvxattr_latency": { 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:40.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_sessions": { 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "session_count": 2, 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "session_add": 3, 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "session_remove": 1, 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "sessions_open": 3, 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "sessions_stale": 0, 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "total_load": 0, 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "average_load": 0, 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "avg_session_uptime": 3214, 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "mdthresh_evicted": 0 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "mempool": { 2024-04-25T02:20:40.830 INFO:teuthology.orchestra.run.smithi134.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bloom_filter_items": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:20:40.831 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_items": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_blob_items": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_txc_items": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_writing_items": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_bytes": 0, 2024-04-25T02:20:40.832 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_items": 0, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_anon_bytes": 241810, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_anon_items": 86, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "buffer_meta_items": 0, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_bytes": 0, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_items": 0, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mapbl_items": 0, 2024-04-25T02:20:40.833 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pglog_items": 0, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_bytes": 7936, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_items": 165, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "pgmap_bytes": 0, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "pgmap_items": 0, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_co_bytes": 16281830, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_co_items": 71742, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_1_bytes": 0, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_1_items": 0, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_2_bytes": 0, 2024-04-25T02:20:40.834 INFO:teuthology.orchestra.run.smithi134.stdout: "unittest_2_items": 0 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter": { 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "op_active": 0, 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "op_laggy": 0, 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "op_send": 6516, 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "op_send_bytes": 21, 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "op_resend": 3, 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "op_reply": 6516, 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "op_latency": { 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 6516, 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 4.718892336, 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000724200 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.835 INFO:teuthology.orchestra.run.smithi134.stdout: "op_inflight": 0, 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "oplen_avg": { 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 6516, 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 6517 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "op": 6516, 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "op_r": 6516, 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "op_w": 0, 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "op_rmw": 0, 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "op_pg": 0, 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_stat": 3523, 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_create": 0, 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_read": 2991, 2024-04-25T02:20:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_write": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_writefull": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_writesame": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_append": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_zero": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_truncate": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_delete": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_mapext": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_sparse_read": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_clonerange": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_getxattr": 1, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_setxattr": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_rmxattr": 0, 2024-04-25T02:20:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_call": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_watch": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_notify": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_pgls": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "osdop_other": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_active": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_send": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_resend": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "linger_ping": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_active": 0, 2024-04-25T02:20:40.838 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_send": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "poolop_resend": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_active": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_send": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "poolstat_resend": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_active": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_send": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "statfs_resend": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "command_active": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "command_send": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "command_resend": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "map_epoch": 129, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "map_full": 0, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "map_inc": 42, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_sessions": 11, 2024-04-25T02:20:40.839 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_session_open": 12, 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_session_close": 1, 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_laggy": 0, 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_wr": 0, 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_rd": 2, 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_del": 0 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "oft": { 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_objs": 0, 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_updates": 0, 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "omap_total_removes": 0 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "osd-slow-ops": { 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "slow_ops_count": 0 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.840 INFO:teuthology.orchestra.run.smithi134.stdout: "purge_queue": { 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_ops": 0, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing": 0, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executing_high_water": 0, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executed_ops": 0, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_executed": 0, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "pq_item_in_journal": 0 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 104857600, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 7064, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 523306491, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:40.841 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 7064, 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 7064, 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 523306491, 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-objecter_bytes": { 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 104857600, 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:40.842 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 0, 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 0, 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 6516, 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 520277518, 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 6516, 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 520277518, 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.843 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-objecter_ops": { 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 1024, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 0, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 0, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 6516, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 6516, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 6516, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 6516, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.844 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 3758096384, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 0, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 0, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 0, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 0, 2024-04-25T02:20:40.845 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: "throttle-write_buf_throttle-0x55eb4160c6e0": { 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: "val": 0, 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: "max": 3758096384, 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: "get_started": 0, 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: "get": 0, 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: "get_sum": 0, 2024-04-25T02:20:40.846 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_fail": 0, 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout: "get_or_fail_success": 0, 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout: "take": 0, 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout: "take_sum": 0, 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout: "put": 0, 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout: "put_sum": 0, 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout: "wait": { 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout: "avgcount": 0, 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout: "sum": 0.000000000, 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout: "avgtime": 0.000000000 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-25T02:20:40.847 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-25T02:20:41.576 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-25T02:20:41.577 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-25T02:20:41.577 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-04-25T02:20:41.577 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-04-25T02:20:41.577 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-04-25T02:20:41.577 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:20:41.577 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:20:41.578 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-25T02:20:41.594 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-25T02:20:41.594 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T02:20:41.594 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T02:20:41.624 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T02:20:41.624 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T02:20:41.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T02:20:41.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.840+0000 7f76e151b640 1 -- 172.21.15.31:0/1098909248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76dc106650 msgr2=0x7f76dc106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:41.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.840+0000 7f76e151b640 1 --2- 172.21.15.31:0/1098909248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76dc106650 0x7f76dc106a30 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009930 tx=0x7f76cc02a420 comp rx=0 tx=0).stop 2024-04-25T02:20:41.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.840+0000 7f76e151b640 1 -- 172.21.15.31:0/1098909248 shutdown_connections 2024-04-25T02:20:41.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.840+0000 7f76e151b640 1 --2- 172.21.15.31:0/1098909248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76dc101050 0x7f76dc10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:41.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.840+0000 7f76e151b640 1 --2- 172.21.15.31:0/1098909248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76dc100690 0x7f76dc100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:41.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.840+0000 7f76e151b640 1 --2- 172.21.15.31:0/1098909248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76dc106650 0x7f76dc106a30 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:41.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.840+0000 7f76e151b640 1 -- 172.21.15.31:0/1098909248 >> 172.21.15.31:0/1098909248 conn(0x7f76dc0777a0 msgr2=0x7f76dc077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:41.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.841+0000 7f76e151b640 1 -- 172.21.15.31:0/1098909248 shutdown_connections 2024-04-25T02:20:41.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.841+0000 7f76e151b640 1 -- 172.21.15.31:0/1098909248 wait complete. 2024-04-25T02:20:41.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.841+0000 7f76e151b640 1 Processor -- start 2024-04-25T02:20:41.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.841+0000 7f76e151b640 1 -- start start 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.841+0000 7f76e151b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76dc100690 0x7f76dc198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.841+0000 7f76e151b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76dc101050 0x7f76dc199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.841+0000 7f76e151b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76dc106650 0x7f76dc1a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.841+0000 7f76e151b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f76dc110910 con 0x7f76dc101050 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.841+0000 7f76e151b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f76dc110790 con 0x7f76dc106650 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.841+0000 7f76e151b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f76dc110a90 con 0x7f76dc100690 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76dc101050 0x7f76dc199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76dc101050 0x7f76dc199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40802/0 (socket says 172.21.15.31:40802) 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d3fff640 1 -- 172.21.15.31:0/1856520536 learned_addr learned my addr 172.21.15.31:0/1856520536 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76db7fe640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76dc106650 0x7f76dc1a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76daffd640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76dc100690 0x7f76dc198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d3fff640 1 -- 172.21.15.31:0/1856520536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76dc100690 msgr2=0x7f76dc198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d3fff640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76dc100690 0x7f76dc198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d3fff640 1 -- 172.21.15.31:0/1856520536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76dc106650 msgr2=0x7f76dc1a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d3fff640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76dc106650 0x7f76dc1a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:41.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d3fff640 1 -- 172.21.15.31:0/1856520536 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76dc1a43b0 con 0x7f76dc101050 2024-04-25T02:20:41.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76daffd640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76dc100690 0x7f76dc198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:20:41.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d3fff640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76dc101050 0x7f76dc199330 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f76c400e9f0 tx=0x7f76c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:41.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d8ff9640 1 -- 172.21.15.31:0/1856520536 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c400cd80 con 0x7f76dc101050 2024-04-25T02:20:41.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76e151b640 1 -- 172.21.15.31:0/1856520536 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76dc1a4580 con 0x7f76dc101050 2024-04-25T02:20:41.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d8ff9640 1 -- 172.21.15.31:0/1856520536 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76c40108e0 con 0x7f76dc101050 2024-04-25T02:20:41.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76d8ff9640 1 -- 172.21.15.31:0/1856520536 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4018920 con 0x7f76dc101050 2024-04-25T02:20:41.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.842+0000 7f76e151b640 1 -- 172.21.15.31:0/1856520536 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76dc1a4aa0 con 0x7f76dc101050 2024-04-25T02:20:41.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.843+0000 7f76e151b640 1 -- 172.21.15.31:0/1856520536 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76dc0689b0 con 0x7f76dc101050 2024-04-25T02:20:41.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.844+0000 7f76d8ff9640 1 -- 172.21.15.31:0/1856520536 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f76c4010a90 con 0x7f76dc101050 2024-04-25T02:20:41.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.845+0000 7f76d8ff9640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f76c80a2d00 0x7f76c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:41.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.845+0000 7f76daffd640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f76c80a2d00 0x7f76c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:41.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.845+0000 7f76d8ff9640 1 -- 172.21.15.31:0/1856520536 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8484+0+0 (secure 0 0 0) 0x7f76c40cf7b0 con 0x7f76dc101050 2024-04-25T02:20:41.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.846+0000 7f76daffd640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f76c80a2d00 0x7f76c80a51c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f76cc000c00 tx=0x7f76cc009610 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:41.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:41.850+0000 7f76d8ff9640 1 -- 172.21.15.31:0/1856520536 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f76c409c3f0 con 0x7f76dc101050 2024-04-25T02:20:42.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.040+0000 7f76e151b640 1 -- 172.21.15.31:0/1856520536 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76dc108bb0 con 0x7f76dc101050 2024-04-25T02:20:42.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.042+0000 7f76d8ff9640 1 -- 172.21.15.31:0/1856520536 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+22949 (secure 0 0 0) 0x7f76c409bcf0 con 0x7f76dc101050 2024-04-25T02:20:42.046 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T02:20:42.046 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":129,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T02:20:09.040890+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":128,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3045817976":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6800/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1358173279":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/686937010":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/3913527371":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":33,"length":1},{"begin":46,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T02:20:42.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 -- 172.21.15.31:0/1856520536 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f76c80a2d00 msgr2=0x7f76c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f76c80a2d00 0x7f76c80a51c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f76cc000c00 tx=0x7f76cc009610 comp rx=0 tx=0).stop 2024-04-25T02:20:42.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 -- 172.21.15.31:0/1856520536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76dc101050 msgr2=0x7f76dc199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76dc101050 0x7f76dc199330 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f76c400e9f0 tx=0x7f76c400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 -- 172.21.15.31:0/1856520536 shutdown_connections 2024-04-25T02:20:42.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f76c80a2d00 0x7f76c80a51c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76dc106650 0x7f76dc1a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76dc101050 0x7f76dc199330 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 --2- 172.21.15.31:0/1856520536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76dc100690 0x7f76dc198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 -- 172.21.15.31:0/1856520536 >> 172.21.15.31:0/1856520536 conn(0x7f76dc0777a0 msgr2=0x7f76dc103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:42.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 -- 172.21.15.31:0/1856520536 shutdown_connections 2024-04-25T02:20:42.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.044+0000 7f76e151b640 1 -- 172.21.15.31:0/1856520536 wait complete. 2024-04-25T02:20:42.059 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:23.432566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:24.255373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 46, 'snap_epoch': 128, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T02:20:42.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T02:20:42.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.161+0000 7f0e33ff9640 1 -- 172.21.15.31:0/18893320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e2c1020d0 msgr2=0x7f0e2c10e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.161+0000 7f0e33ff9640 1 --2- 172.21.15.31:0/18893320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e2c1020d0 0x7f0e2c10e5e0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009cd0 tx=0x7f0e2402a8e0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.161+0000 7f0e33ff9640 1 -- 172.21.15.31:0/18893320 shutdown_connections 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.161+0000 7f0e33ff9640 1 --2- 172.21.15.31:0/18893320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e2c1020d0 0x7f0e2c10e5e0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.161+0000 7f0e33ff9640 1 --2- 172.21.15.31:0/18893320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e2c101710 0x7f0e2c101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.161+0000 7f0e33ff9640 1 --2- 172.21.15.31:0/18893320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c107720 0x7f0e2c107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.161+0000 7f0e33ff9640 1 -- 172.21.15.31:0/18893320 >> 172.21.15.31:0/18893320 conn(0x7f0e2c078b90 msgr2=0x7f0e2c078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.161+0000 7f0e33ff9640 1 -- 172.21.15.31:0/18893320 shutdown_connections 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.161+0000 7f0e33ff9640 1 -- 172.21.15.31:0/18893320 wait complete. 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e33ff9640 1 Processor -- start 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e33ff9640 1 -- start start 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e33ff9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e2c101710 0x7f0e2c197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e33ff9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c1020d0 0x7f0e2c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e33ff9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e2c107720 0x7f0e2c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e33ff9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c1119a0 con 0x7f0e2c107720 2024-04-25T02:20:42.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e33ff9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c111820 con 0x7f0e2c1020d0 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e33ff9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c111b20 con 0x7f0e2c101710 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e32596640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e2c107720 0x7f0e2c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e32596640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e2c107720 0x7f0e2c19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40828/0 (socket says 172.21.15.31:40828) 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e32596640 1 -- 172.21.15.31:0/2889409111 learned_addr learned my addr 172.21.15.31:0/2889409111 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e31d95640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e2c101710 0x7f0e2c197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e31594640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c1020d0 0x7f0e2c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e32596640 1 -- 172.21.15.31:0/2889409111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e2c101710 msgr2=0x7f0e2c197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e32596640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e2c101710 0x7f0e2c197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e32596640 1 -- 172.21.15.31:0/2889409111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c1020d0 msgr2=0x7f0e2c1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e32596640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c1020d0 0x7f0e2c1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e32596640 1 -- 172.21.15.31:0/2889409111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2c19ce50 con 0x7f0e2c107720 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e31594640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c1020d0 0x7f0e2c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.162+0000 7f0e31d95640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e2c101710 0x7f0e2c197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.163+0000 7f0e32596640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e2c107720 0x7f0e2c19c720 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005cc0 tx=0x7f0e24003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:42.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.163+0000 7f0e22ffd640 1 -- 172.21.15.31:0/2889409111 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e24004450 con 0x7f0e2c107720 2024-04-25T02:20:42.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.163+0000 7f0e22ffd640 1 -- 172.21.15.31:0/2889409111 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e24031c90 con 0x7f0e2c107720 2024-04-25T02:20:42.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.163+0000 7f0e33ff9640 1 -- 172.21.15.31:0/2889409111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e2c19d0e0 con 0x7f0e2c107720 2024-04-25T02:20:42.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.163+0000 7f0e22ffd640 1 -- 172.21.15.31:0/2889409111 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2403b400 con 0x7f0e2c107720 2024-04-25T02:20:42.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.163+0000 7f0e33ff9640 1 -- 172.21.15.31:0/2889409111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e2c1a49d0 con 0x7f0e2c107720 2024-04-25T02:20:42.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.164+0000 7f0e33ff9640 1 -- 172.21.15.31:0/2889409111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e2c0689b0 con 0x7f0e2c107720 2024-04-25T02:20:42.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.165+0000 7f0e22ffd640 1 -- 172.21.15.31:0/2889409111 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0e24004680 con 0x7f0e2c107720 2024-04-25T02:20:42.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.166+0000 7f0e22ffd640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0e180a2d00 0x7f0e180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.166+0000 7f0e31d95640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0e180a2d00 0x7f0e180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.166+0000 7f0e22ffd640 1 -- 172.21.15.31:0/2889409111 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8484+0+0 (secure 0 0 0) 0x7f0e2402e030 con 0x7f0e2c107720 2024-04-25T02:20:42.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.166+0000 7f0e31d95640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0e180a2d00 0x7f0e180a51c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0e28005e10 tx=0x7f0e28005da0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:42.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.169+0000 7f0e22ffd640 1 -- 172.21.15.31:0/2889409111 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0e240bd860 con 0x7f0e2c107720 2024-04-25T02:20:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.354+0000 7f0e33ff9640 1 -- 172.21.15.31:0/2889409111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0e2c109d00 con 0x7f0e2c107720 2024-04-25T02:20:42.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.354+0000 7f0e22ffd640 1 -- 172.21.15.31:0/2889409111 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) ==== 93+0+10 (secure 0 0 0) 0x7f0e240bd160 con 0x7f0e2c107720 2024-04-25T02:20:42.359 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T02:20:42.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 -- 172.21.15.31:0/2889409111 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0e180a2d00 msgr2=0x7f0e180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0e180a2d00 0x7f0e180a51c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0e28005e10 tx=0x7f0e28005da0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 -- 172.21.15.31:0/2889409111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e2c107720 msgr2=0x7f0e2c19c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e2c107720 0x7f0e2c19c720 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005cc0 tx=0x7f0e24003c20 comp rx=0 tx=0).stop 2024-04-25T02:20:42.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 -- 172.21.15.31:0/2889409111 shutdown_connections 2024-04-25T02:20:42.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f0e180a2d00 0x7f0e180a51c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e2c107720 0x7f0e2c19c720 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2c1020d0 0x7f0e2c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 --2- 172.21.15.31:0/2889409111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e2c101710 0x7f0e2c197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 -- 172.21.15.31:0/2889409111 >> 172.21.15.31:0/2889409111 conn(0x7f0e2c078b90 msgr2=0x7f0e2c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:42.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 -- 172.21.15.31:0/2889409111 shutdown_connections 2024-04-25T02:20:42.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.356+0000 7f0e33ff9640 1 -- 172.21.15.31:0/2889409111 wait complete. 2024-04-25T02:20:42.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T02:20:42.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.474+0000 7f5ce63a5640 1 -- 172.21.15.31:0/3324373332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce0100610 msgr2=0x7f5ce0100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.474+0000 7f5ce63a5640 1 --2- 172.21.15.31:0/3324373332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce0100610 0x7f5ce0100a90 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009930 tx=0x7f5cd402a420 comp rx=0 tx=0).stop 2024-04-25T02:20:42.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.474+0000 7f5ce63a5640 1 -- 172.21.15.31:0/3324373332 shutdown_connections 2024-04-25T02:20:42.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.474+0000 7f5ce63a5640 1 --2- 172.21.15.31:0/3324373332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce0100fd0 0x7f5ce010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.474+0000 7f5ce63a5640 1 --2- 172.21.15.31:0/3324373332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce0100610 0x7f5ce0100a90 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.474+0000 7f5ce63a5640 1 --2- 172.21.15.31:0/3324373332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce01066b0 0x7f5ce0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.474+0000 7f5ce63a5640 1 -- 172.21.15.31:0/3324373332 >> 172.21.15.31:0/3324373332 conn(0x7f5ce0077740 msgr2=0x7f5ce0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.474+0000 7f5ce63a5640 1 -- 172.21.15.31:0/3324373332 shutdown_connections 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.474+0000 7f5ce63a5640 1 -- 172.21.15.31:0/3324373332 wait complete. 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.474+0000 7f5ce63a5640 1 Processor -- start 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5ce63a5640 1 -- start start 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5ce63a5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce0100610 0x7f5ce0197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5ce63a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce0100fd0 0x7f5ce0198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5ce63a5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce01066b0 0x7f5ce019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5ce63a5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce01108e0 con 0x7f5ce0100fd0 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5ce63a5640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce0110760 con 0x7f5ce01066b0 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5ce63a5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce0110a60 con 0x7f5ce0100610 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5cdf7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce0100fd0 0x7f5ce0198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5cdf7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce0100fd0 0x7f5ce0198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40848/0 (socket says 172.21.15.31:40848) 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5cdf7fe640 1 -- 172.21.15.31:0/619240517 learned_addr learned my addr 172.21.15.31:0/619240517 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5ce4942640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce01066b0 0x7f5ce019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5cdffff640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce0100610 0x7f5ce0197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5cdf7fe640 1 -- 172.21.15.31:0/619240517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce0100610 msgr2=0x7f5ce0197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5cdf7fe640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce0100610 0x7f5ce0197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5cdf7fe640 1 -- 172.21.15.31:0/619240517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce01066b0 msgr2=0x7f5ce019c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5cdf7fe640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce01066b0 0x7f5ce019c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5cdf7fe640 1 -- 172.21.15.31:0/619240517 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce019ce10 con 0x7f5ce0100fd0 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.475+0000 7f5ce4942640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce01066b0 0x7f5ce019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.476+0000 7f5cdf7fe640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce0100fd0 0x7f5ce0198360 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4004770 tx=0x7f5cd40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.476+0000 7f5cdd7fa640 1 -- 172.21.15.31:0/619240517 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd403f070 con 0x7f5ce0100fd0 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.476+0000 7f5cdd7fa640 1 -- 172.21.15.31:0/619240517 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cd403b460 con 0x7f5ce0100fd0 2024-04-25T02:20:42.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.476+0000 7f5cdd7fa640 1 -- 172.21.15.31:0/619240517 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4035070 con 0x7f5ce0100fd0 2024-04-25T02:20:42.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.476+0000 7f5ce63a5640 1 -- 172.21.15.31:0/619240517 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce019d0a0 con 0x7f5ce0100fd0 2024-04-25T02:20:42.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.476+0000 7f5ce63a5640 1 -- 172.21.15.31:0/619240517 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce01a4990 con 0x7f5ce0100fd0 2024-04-25T02:20:42.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.477+0000 7f5ce63a5640 1 -- 172.21.15.31:0/619240517 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce0068560 con 0x7f5ce0100fd0 2024-04-25T02:20:42.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.478+0000 7f5cdd7fa640 1 -- 172.21.15.31:0/619240517 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5cd403b610 con 0x7f5ce0100fd0 2024-04-25T02:20:42.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.479+0000 7f5cdd7fa640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cc80a2d00 0x7f5cc80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.479+0000 7f5cdd7fa640 1 -- 172.21.15.31:0/619240517 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8484+0+0 (secure 0 0 0) 0x7f5cd40efb30 con 0x7f5ce0100fd0 2024-04-25T02:20:42.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.479+0000 7f5cdffff640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cc80a2d00 0x7f5cc80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.479+0000 7f5cdffff640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cc80a2d00 0x7f5cc80a51c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5cd00058c0 tx=0x7f5cd0005850 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:42.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.482+0000 7f5cdd7fa640 1 -- 172.21.15.31:0/619240517 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5cd40bc770 con 0x7f5ce0100fd0 2024-04-25T02:20:42.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.667+0000 7f5ce63a5640 1 -- 172.21.15.31:0/619240517 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5ce01a4bf0 con 0x7f5ce0100fd0 2024-04-25T02:20:42.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.667+0000 7f5cdd7fa640 1 -- 172.21.15.31:0/619240517 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) ==== 104+0+11 (secure 0 0 0) 0x7f5cd40bc070 con 0x7f5ce0100fd0 2024-04-25T02:20:42.671 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T02:20:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.669+0000 7f5ce63a5640 1 -- 172.21.15.31:0/619240517 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cc80a2d00 msgr2=0x7f5cc80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.669+0000 7f5ce63a5640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cc80a2d00 0x7f5cc80a51c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5cd00058c0 tx=0x7f5cd0005850 comp rx=0 tx=0).stop 2024-04-25T02:20:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.669+0000 7f5ce63a5640 1 -- 172.21.15.31:0/619240517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce0100fd0 msgr2=0x7f5ce0198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.669+0000 7f5ce63a5640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce0100fd0 0x7f5ce0198360 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4004770 tx=0x7f5cd40042f0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.670+0000 7f5ce63a5640 1 -- 172.21.15.31:0/619240517 shutdown_connections 2024-04-25T02:20:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.670+0000 7f5ce63a5640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f5cc80a2d00 0x7f5cc80a51c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.670+0000 7f5ce63a5640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ce01066b0 0x7f5ce019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.670+0000 7f5ce63a5640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ce0100fd0 0x7f5ce0198360 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.670+0000 7f5ce63a5640 1 --2- 172.21.15.31:0/619240517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce0100610 0x7f5ce0197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.670+0000 7f5ce63a5640 1 -- 172.21.15.31:0/619240517 >> 172.21.15.31:0/619240517 conn(0x7f5ce0077740 msgr2=0x7f5ce0068b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.670+0000 7f5ce63a5640 1 -- 172.21.15.31:0/619240517 shutdown_connections 2024-04-25T02:20:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.670+0000 7f5ce63a5640 1 -- 172.21.15.31:0/619240517 wait complete. 2024-04-25T02:20:42.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:42 smithi031 ceph-mon[29204]: pgmap v828: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T02:20:42.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:42 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/1856520536' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:20:42.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:42 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2889409111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:20:42.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T02:20:42.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:42 smithi134 ceph-mon[33098]: pgmap v828: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T02:20:42.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:42 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/1856520536' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:20:42.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:42 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2889409111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:20:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.786+0000 7f6c16efc640 1 -- 172.21.15.31:0/1332645156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c10106630 msgr2=0x7f6c10106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.786+0000 7f6c16efc640 1 --2- 172.21.15.31:0/1332645156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c10106630 0x7f6c10106a10 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009930 tx=0x7f6c0402a420 comp rx=0 tx=0).stop 2024-04-25T02:20:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.786+0000 7f6c16efc640 1 -- 172.21.15.31:0/1332645156 shutdown_connections 2024-04-25T02:20:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.786+0000 7f6c16efc640 1 --2- 172.21.15.31:0/1332645156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10100fe0 0x7f6c1010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.786+0000 7f6c16efc640 1 --2- 172.21.15.31:0/1332645156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c10100620 0x7f6c10100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.786+0000 7f6c16efc640 1 --2- 172.21.15.31:0/1332645156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c10106630 0x7f6c10106a10 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.786+0000 7f6c16efc640 1 -- 172.21.15.31:0/1332645156 >> 172.21.15.31:0/1332645156 conn(0x7f6c100776a0 msgr2=0x7f6c10077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.786+0000 7f6c16efc640 1 -- 172.21.15.31:0/1332645156 shutdown_connections 2024-04-25T02:20:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.786+0000 7f6c16efc640 1 -- 172.21.15.31:0/1332645156 wait complete. 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c16efc640 1 Processor -- start 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c16efc640 1 -- start start 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c16efc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c10100620 0x7f6c10198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c16efc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c10100fe0 0x7f6c10198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c16efc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10106630 0x7f6c1019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c16efc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c101108f0 con 0x7f6c10100fe0 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c16efc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6c10110770 con 0x7f6c10100620 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c16efc640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6c10110a70 con 0x7f6c10106630 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c0ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c10100fe0 0x7f6c10198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c0ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c10100fe0 0x7f6c10198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55198/0 (socket says 172.21.15.31:55198) 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c0ffff640 1 -- 172.21.15.31:0/4208680707 learned_addr learned my addr 172.21.15.31:0/4208680707 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c14c98640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c10100620 0x7f6c10198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c15499640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10106630 0x7f6c1019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c0ffff640 1 -- 172.21.15.31:0/4208680707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10106630 msgr2=0x7f6c1019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c0ffff640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10106630 0x7f6c1019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c0ffff640 1 -- 172.21.15.31:0/4208680707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c10100620 msgr2=0x7f6c10198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c0ffff640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c10100620 0x7f6c10198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c0ffff640 1 -- 172.21.15.31:0/4208680707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1019cfc0 con 0x7f6c10100fe0 2024-04-25T02:20:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.787+0000 7f6c15499640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10106630 0x7f6c1019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:20:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.788+0000 7f6c0ffff640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c10100fe0 0x7f6c10198560 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00c890 tx=0x7f6bfc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.788+0000 7f6c0dffb640 1 -- 172.21.15.31:0/4208680707 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc004220 con 0x7f6c10100fe0 2024-04-25T02:20:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.788+0000 7f6c0dffb640 1 -- 172.21.15.31:0/4208680707 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6bfc0043c0 con 0x7f6c10100fe0 2024-04-25T02:20:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.788+0000 7f6c16efc640 1 -- 172.21.15.31:0/4208680707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1019d2b0 con 0x7f6c10100fe0 2024-04-25T02:20:42.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.788+0000 7f6c0dffb640 1 -- 172.21.15.31:0/4208680707 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc004a00 con 0x7f6c10100fe0 2024-04-25T02:20:42.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.788+0000 7f6c16efc640 1 -- 172.21.15.31:0/4208680707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c101a4b40 con 0x7f6c10100fe0 2024-04-25T02:20:42.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.789+0000 7f6c16efc640 1 -- 172.21.15.31:0/4208680707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c100689b0 con 0x7f6c10100fe0 2024-04-25T02:20:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.790+0000 7f6c0dffb640 1 -- 172.21.15.31:0/4208680707 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6bfc004ba0 con 0x7f6c10100fe0 2024-04-25T02:20:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.790+0000 7f6c0dffb640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6c000a2d00 0x7f6c000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.791+0000 7f6c0dffb640 1 -- 172.21.15.31:0/4208680707 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8484+0+0 (secure 0 0 0) 0x7f6bfc0cf150 con 0x7f6c10100fe0 2024-04-25T02:20:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.791+0000 7f6c14c98640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6c000a2d00 0x7f6c000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.791+0000 7f6c14c98640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6c000a2d00 0x7f6c000a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6c04006010 tx=0x7f6c04000bc0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:42.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.795+0000 7f6c0dffb640 1 -- 172.21.15.31:0/4208680707 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6bfc014030 con 0x7f6c10100fe0 2024-04-25T02:20:42.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:42 smithi040 ceph-mon[32418]: pgmap v828: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T02:20:42.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:42 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/1856520536' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:20:42.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:42 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2889409111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:20:42.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.978+0000 7f6c16efc640 1 -- 172.21.15.31:0/4208680707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6c10109c20 con 0x7f6c10100fe0 2024-04-25T02:20:42.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.978+0000 7f6c0dffb640 1 -- 172.21.15.31:0/4208680707 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) ==== 100+0+11 (secure 0 0 0) 0x7f6bfc09bd30 con 0x7f6c10100fe0 2024-04-25T02:20:42.982 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T02:20:42.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 -- 172.21.15.31:0/4208680707 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6c000a2d00 msgr2=0x7f6c000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6c000a2d00 0x7f6c000a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6c04006010 tx=0x7f6c04000bc0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 -- 172.21.15.31:0/4208680707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c10100fe0 msgr2=0x7f6c10198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:42.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c10100fe0 0x7f6c10198560 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00c890 tx=0x7f6bfc00cd50 comp rx=0 tx=0).stop 2024-04-25T02:20:42.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 -- 172.21.15.31:0/4208680707 shutdown_connections 2024-04-25T02:20:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f6c000a2d00 0x7f6c000a51c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10106630 0x7f6c1019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c10100fe0 0x7f6c10198560 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 --2- 172.21.15.31:0/4208680707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c10100620 0x7f6c10198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 -- 172.21.15.31:0/4208680707 >> 172.21.15.31:0/4208680707 conn(0x7f6c100776a0 msgr2=0x7f6c1006a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 -- 172.21.15.31:0/4208680707 shutdown_connections 2024-04-25T02:20:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:42.980+0000 7f6c16efc640 1 -- 172.21.15.31:0/4208680707 wait complete. 2024-04-25T02:20:42.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T02:20:43.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.095+0000 7f0347a3f640 1 -- 172.21.15.31:0/2115335070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0340100fd0 msgr2=0x7f034010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.095+0000 7f0347a3f640 1 --2- 172.21.15.31:0/2115335070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0340100fd0 0x7f034010d4e0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f0338009cd0 tx=0x7f033802a8e0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.095+0000 7f0347a3f640 1 -- 172.21.15.31:0/2115335070 shutdown_connections 2024-04-25T02:20:43.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.095+0000 7f0347a3f640 1 --2- 172.21.15.31:0/2115335070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0340100fd0 0x7f034010d4e0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.095+0000 7f0347a3f640 1 --2- 172.21.15.31:0/2115335070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0340100610 0x7f0340100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.095+0000 7f0347a3f640 1 --2- 172.21.15.31:0/2115335070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03401066b0 0x7f0340106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.095+0000 7f0347a3f640 1 -- 172.21.15.31:0/2115335070 >> 172.21.15.31:0/2115335070 conn(0x7f0340077740 msgr2=0x7f0340077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:43.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.095+0000 7f0347a3f640 1 -- 172.21.15.31:0/2115335070 shutdown_connections 2024-04-25T02:20:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.095+0000 7f0347a3f640 1 -- 172.21.15.31:0/2115335070 wait complete. 2024-04-25T02:20:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.095+0000 7f0347a3f640 1 Processor -- start 2024-04-25T02:20:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.095+0000 7f0347a3f640 1 -- start start 2024-04-25T02:20:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f0347a3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0340100610 0x7f0340197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f0347a3f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0340100fd0 0x7f0340198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f0347a3f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03401066b0 0x7f034019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f0347a3f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f03401108a0 con 0x7f0340100610 2024-04-25T02:20:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f0347a3f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0340110720 con 0x7f0340100fd0 2024-04-25T02:20:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f0347a3f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0340110a20 con 0x7f03401066b0 2024-04-25T02:20:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f03457db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0340100610 0x7f0340197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f03457db640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0340100610 0x7f0340197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55220/0 (socket says 172.21.15.31:55220) 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f03457db640 1 -- 172.21.15.31:0/3401981831 learned_addr learned my addr 172.21.15.31:0/3401981831 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f0344fda640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0340100fd0 0x7f0340198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f0345fdc640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03401066b0 0x7f034019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f03457db640 1 -- 172.21.15.31:0/3401981831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03401066b0 msgr2=0x7f034019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f03457db640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03401066b0 0x7f034019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f03457db640 1 -- 172.21.15.31:0/3401981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0340100fd0 msgr2=0x7f0340198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f03457db640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0340100fd0 0x7f0340198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f03457db640 1 -- 172.21.15.31:0/3401981831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034019ce30 con 0x7f0340100610 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f0345fdc640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03401066b0 0x7f034019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f03457db640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0340100610 0x7f0340197e40 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f033c00d770 tx=0x7f033c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.096+0000 7f03367fc640 1 -- 172.21.15.31:0/3401981831 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c017070 con 0x7f0340100610 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.097+0000 7f03367fc640 1 -- 172.21.15.31:0/3401981831 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f033c009c30 con 0x7f0340100610 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.097+0000 7f0347a3f640 1 -- 172.21.15.31:0/3401981831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034019d120 con 0x7f0340100610 2024-04-25T02:20:43.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.097+0000 7f03367fc640 1 -- 172.21.15.31:0/3401981831 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c003d00 con 0x7f0340100610 2024-04-25T02:20:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.097+0000 7f0347a3f640 1 -- 172.21.15.31:0/3401981831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03401a49b0 con 0x7f0340100610 2024-04-25T02:20:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.098+0000 7f0347a3f640 1 -- 172.21.15.31:0/3401981831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0340068560 con 0x7f0340100610 2024-04-25T02:20:43.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.099+0000 7f03367fc640 1 -- 172.21.15.31:0/3401981831 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f033c003ea0 con 0x7f0340100610 2024-04-25T02:20:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.099+0000 7f03367fc640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f032c0a2d00 0x7f032c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.100+0000 7f0344fda640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f032c0a2d00 0x7f032c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.100+0000 7f03367fc640 1 -- 172.21.15.31:0/3401981831 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8484+0+0 (secure 0 0 0) 0x7f033c0cf0e0 con 0x7f0340100610 2024-04-25T02:20:43.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.100+0000 7f0344fda640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f032c0a2d00 0x7f032c0a51c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f03401055a0 tx=0x7f033000a2d0 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:43.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.104+0000 7f03367fc640 1 -- 172.21.15.31:0/3401981831 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f033c09bdc0 con 0x7f0340100610 2024-04-25T02:20:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.301+0000 7f0347a3f640 1 -- 172.21.15.31:0/3401981831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f03401a4c20 con 0x7f0340100610 2024-04-25T02:20:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.302+0000 7f03367fc640 1 -- 172.21.15.31:0/3401981831 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v129) ==== 81+0+943 (secure 0 0 0) 0x7f033c009de0 con 0x7f0340100610 2024-04-25T02:20:43.306 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3045817976 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:43.306 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/2306032998 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:43.306 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/2306032998 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:43.306 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/363866598 2024-04-26T01:45:23.647161+0000 2024-04-25T02:20:43.306 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/2433894874 2024-04-26T01:45:23.647161+0000 2024-04-25T02:20:43.306 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1358173279 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:43.306 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/3170545029 2024-04-26T01:45:23.647161+0000 2024-04-25T02:20:43.306 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/3170545029 2024-04-26T01:45:23.647161+0000 2024-04-25T02:20:43.306 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/686937010 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:43.306 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/196728475 2024-04-26T01:44:56.414005+0000 2024-04-25T02:20:43.307 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3913527371 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:43.307 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/2254757453 2024-04-26T01:44:56.414005+0000 2024-04-25T02:20:43.307 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/3153405418 2024-04-26T01:44:56.414005+0000 2024-04-25T02:20:43.307 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/3153405418 2024-04-26T01:44:56.414005+0000 2024-04-25T02:20:43.307 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1967729027 2024-04-26T01:45:23.647161+0000 2024-04-25T02:20:43.307 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1543547115 2024-04-26T01:44:56.414005+0000 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:listed 16 entries 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.303+0000 7f0347a3f640 1 -- 172.21.15.31:0/3401981831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f032c0a2d00 msgr2=0x7f032c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.303+0000 7f0347a3f640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f032c0a2d00 0x7f032c0a51c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f03401055a0 tx=0x7f033000a2d0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.303+0000 7f0347a3f640 1 -- 172.21.15.31:0/3401981831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0340100610 msgr2=0x7f0340197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.303+0000 7f0347a3f640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0340100610 0x7f0340197e40 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f033c00d770 tx=0x7f033c00dc30 comp rx=0 tx=0).stop 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.303+0000 7f0347a3f640 1 -- 172.21.15.31:0/3401981831 shutdown_connections 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.303+0000 7f0347a3f640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f032c0a2d00 0x7f032c0a51c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.303+0000 7f0347a3f640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03401066b0 0x7f034019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.303+0000 7f0347a3f640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0340100fd0 0x7f0340198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.303+0000 7f0347a3f640 1 --2- 172.21.15.31:0/3401981831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0340100610 0x7f0340197e40 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.303+0000 7f0347a3f640 1 -- 172.21.15.31:0/3401981831 >> 172.21.15.31:0/3401981831 conn(0x7f0340077740 msgr2=0x7f03400ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.304+0000 7f0347a3f640 1 -- 172.21.15.31:0/3401981831 shutdown_connections 2024-04-25T02:20:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.304+0000 7f0347a3f640 1 -- 172.21.15.31:0/3401981831 wait complete. 2024-04-25T02:20:43.318 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T02:20:43.318 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T02:20:43.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T02:20:43.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.477+0000 7fb77ab5a640 1 -- 172.21.15.31:0/1791773144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb774100620 msgr2=0x7fb774100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.477+0000 7fb77ab5a640 1 --2- 172.21.15.31:0/1791773144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb774100620 0x7fb774100aa0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb764009930 tx=0x7fb76402a420 comp rx=0 tx=0).stop 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.477+0000 7fb77ab5a640 1 -- 172.21.15.31:0/1791773144 shutdown_connections 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.477+0000 7fb77ab5a640 1 --2- 172.21.15.31:0/1791773144 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb774100fe0 0x7fb77410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.477+0000 7fb77ab5a640 1 --2- 172.21.15.31:0/1791773144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb774100620 0x7fb774100aa0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.477+0000 7fb77ab5a640 1 --2- 172.21.15.31:0/1791773144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb774106630 0x7fb774106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb77ab5a640 1 -- 172.21.15.31:0/1791773144 >> 172.21.15.31:0/1791773144 conn(0x7fb774077740 msgr2=0x7fb774077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb77ab5a640 1 -- 172.21.15.31:0/1791773144 shutdown_connections 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb77ab5a640 1 -- 172.21.15.31:0/1791773144 wait complete. 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb77ab5a640 1 Processor -- start 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb77ab5a640 1 -- start start 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb77ab5a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb774100620 0x7fb774197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb77ab5a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb774100fe0 0x7fb774198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb77ab5a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb774106630 0x7fb77419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb77ab5a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb7741108b0 con 0x7fb774106630 2024-04-25T02:20:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb77ab5a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb774110730 con 0x7fb774100620 2024-04-25T02:20:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb77ab5a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb774110a30 con 0x7fb774100fe0 2024-04-25T02:20:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.478+0000 7fb7790f7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb774106630 0x7fb77419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb7790f7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb774106630 0x7fb77419c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55242/0 (socket says 172.21.15.31:55242) 2024-04-25T02:20:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb7790f7640 1 -- 172.21.15.31:0/2612358987 learned_addr learned my addr 172.21.15.31:0/2612358987 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb7788f6640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb774100620 0x7fb774197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb7790f7640 1 -- 172.21.15.31:0/2612358987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb774100fe0 msgr2=0x7fb774198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb7790f7640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb774100fe0 0x7fb774198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb7790f7640 1 -- 172.21.15.31:0/2612358987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb774100620 msgr2=0x7fb774197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb7790f7640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb774100620 0x7fb774197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb7790f7640 1 -- 172.21.15.31:0/2612358987 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77419cf30 con 0x7fb774106630 2024-04-25T02:20:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb7790f7640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb774106630 0x7fb77419c800 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00cad0 tx=0x7fb76c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb769ffb640 1 -- 172.21.15.31:0/2612358987 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c00ae20 con 0x7fb774106630 2024-04-25T02:20:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb769ffb640 1 -- 172.21.15.31:0/2612358987 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb76c005020 con 0x7fb774106630 2024-04-25T02:20:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb77ab5a640 1 -- 172.21.15.31:0/2612358987 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb77419d220 con 0x7fb774106630 2024-04-25T02:20:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb769ffb640 1 -- 172.21.15.31:0/2612358987 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c00f6b0 con 0x7fb774106630 2024-04-25T02:20:43.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.479+0000 7fb77ab5a640 1 -- 172.21.15.31:0/2612358987 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7741a4ab0 con 0x7fb774106630 2024-04-25T02:20:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.480+0000 7fb77ab5a640 1 -- 172.21.15.31:0/2612358987 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb774068560 con 0x7fb774106630 2024-04-25T02:20:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.481+0000 7fb769ffb640 1 -- 172.21.15.31:0/2612358987 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb76c0051d0 con 0x7fb774106630 2024-04-25T02:20:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.482+0000 7fb769ffb640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb75c0a2d80 0x7fb75c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.482+0000 7fb769ffb640 1 -- 172.21.15.31:0/2612358987 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8484+0+0 (secure 0 0 0) 0x7fb76c0cf430 con 0x7fb774106630 2024-04-25T02:20:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.482+0000 7fb7788f6640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb75c0a2d80 0x7fb75c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:43.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.483+0000 7fb7788f6640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb75c0a2d80 0x7fb75c0a5240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb770005d00 tx=0x7fb77000b410 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.486+0000 7fb769ffb640 1 -- 172.21.15.31:0/2612358987 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb76c014030 con 0x7fb774106630 2024-04-25T02:20:43.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.671+0000 7fb77ab5a640 1 -- 172.21.15.31:0/2612358987 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb774109630 con 0x7fb774106630 2024-04-25T02:20:43.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.673+0000 7fb769ffb640 1 -- 172.21.15.31:0/2612358987 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+22949 (secure 0 0 0) 0x7fb76c09c070 con 0x7fb774106630 2024-04-25T02:20:43.677 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T02:20:43.677 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":129,"fsid":"36afbea4-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:44:19.899313+0000","modified":"2024-04-25T02:20:09.040890+0000","last_up_change":"2024-04-25T01:51:37.971884+0000","last_in_change":"2024-04-25T01:51:18.029576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:50.171342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:23.432566+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:24.255373+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":128,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b99beda-ae8b-4e4b-8569-1c3b5f44654b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6803","nonce":268630401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6805","nonce":268630401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6809","nonce":268630401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":268630401},{"type":"v1","addr":"172.21.15.31:6807","nonce":268630401}]},"public_addr":"172.21.15.31:6803/268630401","cluster_addr":"172.21.15.31:6805/268630401","heartbeat_back_addr":"172.21.15.31:6809/268630401","heartbeat_front_addr":"172.21.15.31:6807/268630401","state":["exists","up"]},{"osd":1,"uuid":"285db63c-f115-4517-894f-01b97208ba8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6801","nonce":3593667525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6803","nonce":3593667525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6807","nonce":3593667525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3593667525},{"type":"v1","addr":"172.21.15.40:6805","nonce":3593667525}]},"public_addr":"172.21.15.40:6801/3593667525","cluster_addr":"172.21.15.40:6803/3593667525","heartbeat_back_addr":"172.21.15.40:6807/3593667525","heartbeat_front_addr":"172.21.15.40:6805/3593667525","state":["exists","up"]},{"osd":2,"uuid":"af88e288-e53a-46a6-93a0-576c53c16b71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6801","nonce":414932802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6803","nonce":414932802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6807","nonce":414932802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":414932802},{"type":"v1","addr":"172.21.15.134:6805","nonce":414932802}]},"public_addr":"172.21.15.134:6801/414932802","cluster_addr":"172.21.15.134:6803/414932802","heartbeat_back_addr":"172.21.15.134:6807/414932802","heartbeat_front_addr":"172.21.15.134:6805/414932802","state":["exists","up"]},{"osd":3,"uuid":"bc257cfc-3310-4d0f-bb39-ec5aed0f18f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6811","nonce":513563424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6813","nonce":513563424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6817","nonce":513563424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":513563424},{"type":"v1","addr":"172.21.15.31:6815","nonce":513563424}]},"public_addr":"172.21.15.31:6811/513563424","cluster_addr":"172.21.15.31:6813/513563424","heartbeat_back_addr":"172.21.15.31:6817/513563424","heartbeat_front_addr":"172.21.15.31:6815/513563424","state":["exists","up"]},{"osd":4,"uuid":"0dd91e15-4315-40e0-a93a-aa3e7d5ebed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6809","nonce":3237694093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6811","nonce":3237694093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6815","nonce":3237694093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3237694093},{"type":"v1","addr":"172.21.15.40:6813","nonce":3237694093}]},"public_addr":"172.21.15.40:6809/3237694093","cluster_addr":"172.21.15.40:6811/3237694093","heartbeat_back_addr":"172.21.15.40:6815/3237694093","heartbeat_front_addr":"172.21.15.40:6813/3237694093","state":["exists","up"]},{"osd":5,"uuid":"e992053c-b5dd-41cc-a72f-c371eadc508d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6809","nonce":1004304767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6811","nonce":1004304767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6815","nonce":1004304767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1004304767},{"type":"v1","addr":"172.21.15.134:6813","nonce":1004304767}]},"public_addr":"172.21.15.134:6809/1004304767","cluster_addr":"172.21.15.134:6811/1004304767","heartbeat_back_addr":"172.21.15.134:6815/1004304767","heartbeat_front_addr":"172.21.15.134:6813/1004304767","state":["exists","up"]},{"osd":6,"uuid":"1c03259a-375f-482d-be6e-410eed785c7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6819","nonce":2233839542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6821","nonce":2233839542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6825","nonce":2233839542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2233839542},{"type":"v1","addr":"172.21.15.31:6823","nonce":2233839542}]},"public_addr":"172.21.15.31:6819/2233839542","cluster_addr":"172.21.15.31:6821/2233839542","heartbeat_back_addr":"172.21.15.31:6825/2233839542","heartbeat_front_addr":"172.21.15.31:6823/2233839542","state":["exists","up"]},{"osd":7,"uuid":"d408419f-02da-4563-ae2a-e6b436cc4f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6817","nonce":1477675732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6819","nonce":1477675732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6823","nonce":1477675732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1477675732},{"type":"v1","addr":"172.21.15.40:6821","nonce":1477675732}]},"public_addr":"172.21.15.40:6817/1477675732","cluster_addr":"172.21.15.40:6819/1477675732","heartbeat_back_addr":"172.21.15.40:6823/1477675732","heartbeat_front_addr":"172.21.15.40:6821/1477675732","state":["exists","up"]},{"osd":8,"uuid":"71eaf5df-507d-4ea3-b5e6-fbc65d839f10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6817","nonce":3295606048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6819","nonce":3295606048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6823","nonce":3295606048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3295606048},{"type":"v1","addr":"172.21.15.134:6821","nonce":3295606048}]},"public_addr":"172.21.15.134:6817/3295606048","cluster_addr":"172.21.15.134:6819/3295606048","heartbeat_back_addr":"172.21.15.134:6823/3295606048","heartbeat_front_addr":"172.21.15.134:6821/3295606048","state":["exists","up"]},{"osd":9,"uuid":"eda04a7b-4e18-470d-9678-e21ffeb1f7ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6827","nonce":238962088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6829","nonce":238962088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6833","nonce":238962088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":238962088},{"type":"v1","addr":"172.21.15.31:6831","nonce":238962088}]},"public_addr":"172.21.15.31:6827/238962088","cluster_addr":"172.21.15.31:6829/238962088","heartbeat_back_addr":"172.21.15.31:6833/238962088","heartbeat_front_addr":"172.21.15.31:6831/238962088","state":["exists","up"]},{"osd":10,"uuid":"16a79c3b-ecee-4bc6-bd47-abd3a8d8e5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6825","nonce":2339369197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6827","nonce":2339369197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6831","nonce":2339369197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2339369197},{"type":"v1","addr":"172.21.15.40:6829","nonce":2339369197}]},"public_addr":"172.21.15.40:6825/2339369197","cluster_addr":"172.21.15.40:6827/2339369197","heartbeat_back_addr":"172.21.15.40:6831/2339369197","heartbeat_front_addr":"172.21.15.40:6829/2339369197","state":["exists","up"]},{"osd":11,"uuid":"5113a81e-7097-4d97-8998-b59224bb6b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6825","nonce":897924774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6827","nonce":897924774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6831","nonce":897924774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":897924774},{"type":"v1","addr":"172.21.15.134:6829","nonce":897924774}]},"public_addr":"172.21.15.134:6825/897924774","cluster_addr":"172.21.15.134:6827/897924774","heartbeat_back_addr":"172.21.15.134:6831/897924774","heartbeat_front_addr":"172.21.15.134:6829/897924774","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:58.296565+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:22.955677+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:46.923428+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:11.428141+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:36.786488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:01.520509+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:26.615501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:52.867927+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:17.780542+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:43.274298+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:10.140106+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:36.352161+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3045817976":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6800/2306032998":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/363866598":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/2433894874":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1358173279":"2024-04-26T01:54:28.579184+0000","172.21.15.31:6801/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:6800/3170545029":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/686937010":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/196728475":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/3913527371":"2024-04-26T01:54:28.579184+0000","172.21.15.31:0/2254757453":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6800/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:6801/3153405418":"2024-04-26T01:44:56.414005+0000","172.21.15.31:0/1967729027":"2024-04-26T01:45:23.647161+0000","172.21.15.31:0/1543547115":"2024-04-26T01:44:56.414005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":33,"length":1},{"begin":46,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T02:20:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 -- 172.21.15.31:0/2612358987 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb75c0a2d80 msgr2=0x7fb75c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb75c0a2d80 0x7fb75c0a5240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb770005d00 tx=0x7fb77000b410 comp rx=0 tx=0).stop 2024-04-25T02:20:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 -- 172.21.15.31:0/2612358987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb774106630 msgr2=0x7fb77419c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb774106630 0x7fb77419c800 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00cad0 tx=0x7fb76c00cf90 comp rx=0 tx=0).stop 2024-04-25T02:20:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 -- 172.21.15.31:0/2612358987 shutdown_connections 2024-04-25T02:20:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fb75c0a2d80 0x7fb75c0a5240 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb774106630 0x7fb77419c800 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb774100fe0 0x7fb774198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 --2- 172.21.15.31:0/2612358987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb774100620 0x7fb774197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 -- 172.21.15.31:0/2612358987 >> 172.21.15.31:0/2612358987 conn(0x7fb774077740 msgr2=0x7fb774068cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 -- 172.21.15.31:0/2612358987 shutdown_connections 2024-04-25T02:20:43.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.675+0000 7fb77ab5a640 1 -- 172.21.15.31:0/2612358987 wait complete. 2024-04-25T02:20:43.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:43 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/619240517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:20:43.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:43 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/4208680707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:20:43.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:43 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3401981831' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T02:20:43.690 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:50.171342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:23.432566+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:24.255373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 46, 'snap_epoch': 128, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T02:20:43.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T02:20:43.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:43 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/619240517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:20:43.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:43 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/4208680707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:20:43.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:43 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3401981831' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T02:20:43.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.791+0000 7ff88751b640 1 -- 172.21.15.31:0/773512566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff880101330 msgr2=0x7ff88010d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.791+0000 7ff88751b640 1 --2- 172.21.15.31:0/773512566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff880101330 0x7ff88010d460 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7ff878009ca0 tx=0x7ff87802a8c0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.791+0000 7ff88751b640 1 -- 172.21.15.31:0/773512566 shutdown_connections 2024-04-25T02:20:43.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.791+0000 7ff88751b640 1 --2- 172.21.15.31:0/773512566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff880101330 0x7ff88010d460 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.791+0000 7ff88751b640 1 --2- 172.21.15.31:0/773512566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff880100990 0x7ff880100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.791+0000 7ff88751b640 1 --2- 172.21.15.31:0/773512566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880106690 0x7ff880106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.791+0000 7ff88751b640 1 -- 172.21.15.31:0/773512566 >> 172.21.15.31:0/773512566 conn(0x7ff880076440 msgr2=0x7ff880076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.791+0000 7ff88751b640 1 -- 172.21.15.31:0/773512566 shutdown_connections 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.791+0000 7ff88751b640 1 -- 172.21.15.31:0/773512566 wait complete. 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff88751b640 1 Processor -- start 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff88751b640 1 -- start start 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff88751b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff880100990 0x7ff880197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff88751b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880101330 0x7ff8801982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff88751b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff880106690 0x7ff88019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff88751b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff880110890 con 0x7ff880100990 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff88751b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff880110710 con 0x7ff880106690 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff88751b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff880110a10 con 0x7ff880101330 2024-04-25T02:20:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff8852b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff880100990 0x7ff880197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff8852b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff880100990 0x7ff880197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55256/0 (socket says 172.21.15.31:55256) 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff8852b7640 1 -- 172.21.15.31:0/735896184 learned_addr learned my addr 172.21.15.31:0/735896184 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff885ab8640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff880106690 0x7ff88019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff884ab6640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880101330 0x7ff8801982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff8852b7640 1 -- 172.21.15.31:0/735896184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880101330 msgr2=0x7ff8801982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff8852b7640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880101330 0x7ff8801982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff8852b7640 1 -- 172.21.15.31:0/735896184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff880106690 msgr2=0x7ff88019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff8852b7640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff880106690 0x7ff88019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff8852b7640 1 -- 172.21.15.31:0/735896184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88019cd90 con 0x7ff880100990 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff884ab6640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880101330 0x7ff8801982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff885ab8640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff880106690 0x7ff88019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.792+0000 7ff8852b7640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff880100990 0x7ff880197da0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00c890 tx=0x7ff87c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.793+0000 7ff8767fc640 1 -- 172.21.15.31:0/735896184 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c004220 con 0x7ff880100990 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.793+0000 7ff88751b640 1 -- 172.21.15.31:0/735896184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88019d080 con 0x7ff880100990 2024-04-25T02:20:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.793+0000 7ff8767fc640 1 -- 172.21.15.31:0/735896184 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff87c0043c0 con 0x7ff880100990 2024-04-25T02:20:43.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.793+0000 7ff8767fc640 1 -- 172.21.15.31:0/735896184 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c004a00 con 0x7ff880100990 2024-04-25T02:20:43.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.793+0000 7ff88751b640 1 -- 172.21.15.31:0/735896184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8801a4910 con 0x7ff880100990 2024-04-25T02:20:43.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.794+0000 7ff88751b640 1 -- 172.21.15.31:0/735896184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff880101d80 con 0x7ff880100990 2024-04-25T02:20:43.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.795+0000 7ff8767fc640 1 -- 172.21.15.31:0/735896184 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff87c004ba0 con 0x7ff880100990 2024-04-25T02:20:43.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.795+0000 7ff8767fc640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff86c0a2d00 0x7ff86c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:43.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.795+0000 7ff8767fc640 1 -- 172.21.15.31:0/735896184 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8484+0+0 (secure 0 0 0) 0x7ff87c0cf1a0 con 0x7ff880100990 2024-04-25T02:20:43.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.795+0000 7ff884ab6640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff86c0a2d00 0x7ff86c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:43.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.796+0000 7ff884ab6640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff86c0a2d00 0x7ff86c0a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff8801003b0 tx=0x7ff870005d90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:43.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.800+0000 7ff8767fc640 1 -- 172.21.15.31:0/735896184 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff87c014030 con 0x7ff880100990 2024-04-25T02:20:43.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:43 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/619240517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:20:43.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:43 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/4208680707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:20:43.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:43 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3401981831' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T02:20:43.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.984+0000 7ff88751b640 1 -- 172.21.15.31:0/735896184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff8801a4c50 con 0x7ff880100990 2024-04-25T02:20:43.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.984+0000 7ff8767fc640 1 -- 172.21.15.31:0/735896184 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) ==== 93+0+10 (secure 0 0 0) 0x7ff87c09bde0 con 0x7ff880100990 2024-04-25T02:20:43.988 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T02:20:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 -- 172.21.15.31:0/735896184 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff86c0a2d00 msgr2=0x7ff86c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff86c0a2d00 0x7ff86c0a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff8801003b0 tx=0x7ff870005d90 comp rx=0 tx=0).stop 2024-04-25T02:20:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 -- 172.21.15.31:0/735896184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff880100990 msgr2=0x7ff880197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff880100990 0x7ff880197da0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00c890 tx=0x7ff87c00cd50 comp rx=0 tx=0).stop 2024-04-25T02:20:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 -- 172.21.15.31:0/735896184 shutdown_connections 2024-04-25T02:20:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7ff86c0a2d00 0x7ff86c0a51c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff880106690 0x7ff88019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880101330 0x7ff8801982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 --2- 172.21.15.31:0/735896184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff880100990 0x7ff880197da0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 -- 172.21.15.31:0/735896184 >> 172.21.15.31:0/735896184 conn(0x7ff880076440 msgr2=0x7ff880069f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 -- 172.21.15.31:0/735896184 shutdown_connections 2024-04-25T02:20:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:43.986+0000 7ff88751b640 1 -- 172.21.15.31:0/735896184 wait complete. 2024-04-25T02:20:44.001 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T02:20:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.102+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/1129498369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e4100540 msgr2=0x7fe5e4100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.102+0000 7fe5e9cdf640 1 --2- 172.21.15.31:0/1129498369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e4100540 0x7fe5e4100920 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4009900 tx=0x7fe5d402a410 comp rx=0 tx=0).stop 2024-04-25T02:20:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.102+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/1129498369 shutdown_connections 2024-04-25T02:20:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.102+0000 7fe5e9cdf640 1 --2- 172.21.15.31:0/1129498369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e410e0f0 0x7fe5e41104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.102+0000 7fe5e9cdf640 1 --2- 172.21.15.31:0/1129498369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e4100e60 0x7fe5e410dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.102+0000 7fe5e9cdf640 1 --2- 172.21.15.31:0/1129498369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e4100540 0x7fe5e4100920 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.102+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/1129498369 >> 172.21.15.31:0/1129498369 conn(0x7fe5e4077740 msgr2=0x7fe5e4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.103+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/1129498369 shutdown_connections 2024-04-25T02:20:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.103+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/1129498369 wait complete. 2024-04-25T02:20:44.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.103+0000 7fe5e9cdf640 1 Processor -- start 2024-04-25T02:20:44.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.103+0000 7fe5e9cdf640 1 -- start start 2024-04-25T02:20:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.103+0000 7fe5e9cdf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e4100540 0x7fe5e419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.103+0000 7fe5e9cdf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e4100e60 0x7fe5e419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.103+0000 7fe5e9cdf640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e410e0f0 0x7fe5e419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.103+0000 7fe5e9cdf640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e4113bd0 con 0x7fe5e4100540 2024-04-25T02:20:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.103+0000 7fe5e9cdf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e4113a50 con 0x7fe5e4100e60 2024-04-25T02:20:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.103+0000 7fe5e9cdf640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e4113d50 con 0x7fe5e410e0f0 2024-04-25T02:20:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e4100540 0x7fe5e419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e4100540 0x7fe5e419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55270/0 (socket says 172.21.15.31:55270) 2024-04-25T02:20:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e37fe640 1 -- 172.21.15.31:0/3207003974 learned_addr learned my addr 172.21.15.31:0/3207003974 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e2ffd640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e4100e60 0x7fe5e419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e3fff640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e410e0f0 0x7fe5e419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e37fe640 1 -- 172.21.15.31:0/3207003974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e410e0f0 msgr2=0x7fe5e419e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e37fe640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e410e0f0 0x7fe5e419e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e37fe640 1 -- 172.21.15.31:0/3207003974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e4100e60 msgr2=0x7fe5e419a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e37fe640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e4100e60 0x7fe5e419a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e37fe640 1 -- 172.21.15.31:0/3207003974 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e419f100 con 0x7fe5e4100540 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e3fff640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e410e0f0 0x7fe5e419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e37fe640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e4100540 0x7fe5e419a110 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4005940 tx=0x7fe5d40040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e0ff9640 1 -- 172.21.15.31:0/3207003974 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d403f030 con 0x7fe5e4100540 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/3207003974 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e419f390 con 0x7fe5e4100540 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e0ff9640 1 -- 172.21.15.31:0/3207003974 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5d40044c0 con 0x7fe5e4100540 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e0ff9640 1 -- 172.21.15.31:0/3207003974 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4035070 con 0x7fe5e4100540 2024-04-25T02:20:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.104+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/3207003974 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e41a6c80 con 0x7fe5e4100540 2024-04-25T02:20:44.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.105+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/3207003974 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5e4068560 con 0x7fe5e4100540 2024-04-25T02:20:44.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.106+0000 7fe5e0ff9640 1 -- 172.21.15.31:0/3207003974 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe5d4042030 con 0x7fe5e4100540 2024-04-25T02:20:44.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.107+0000 7fe5e0ff9640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe5cc0a2d00 0x7fe5cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.107+0000 7fe5e0ff9640 1 -- 172.21.15.31:0/3207003974 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8484+0+0 (secure 0 0 0) 0x7fe5d40ef7e0 con 0x7fe5e4100540 2024-04-25T02:20:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.107+0000 7fe5e2ffd640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe5cc0a2d00 0x7fe5cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.108+0000 7fe5e2ffd640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe5cc0a2d00 0x7fe5cc0a51c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe5e41000b0 tx=0x7fe5d8005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:44.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.112+0000 7fe5e0ff9640 1 -- 172.21.15.31:0/3207003974 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe5d40bc420 con 0x7fe5e4100540 2024-04-25T02:20:44.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.296+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/3207003974 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe5e4109280 con 0x7fe5e4100540 2024-04-25T02:20:44.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.296+0000 7fe5e0ff9640 1 -- 172.21.15.31:0/3207003974 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) ==== 104+0+11 (secure 0 0 0) 0x7fe5d40bbd20 con 0x7fe5e4100540 2024-04-25T02:20:44.300 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.298+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/3207003974 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe5cc0a2d00 msgr2=0x7fe5cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.298+0000 7fe5e9cdf640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe5cc0a2d00 0x7fe5cc0a51c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe5e41000b0 tx=0x7fe5d8005f90 comp rx=0 tx=0).stop 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.298+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/3207003974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e4100540 msgr2=0x7fe5e419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.298+0000 7fe5e9cdf640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e4100540 0x7fe5e419a110 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4005940 tx=0x7fe5d40040c0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.299+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/3207003974 shutdown_connections 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.299+0000 7fe5e9cdf640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fe5cc0a2d00 0x7fe5cc0a51c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.299+0000 7fe5e9cdf640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5e410e0f0 0x7fe5e419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.299+0000 7fe5e9cdf640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e4100e60 0x7fe5e419a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.299+0000 7fe5e9cdf640 1 --2- 172.21.15.31:0/3207003974 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5e4100540 0x7fe5e419a110 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.299+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/3207003974 >> 172.21.15.31:0/3207003974 conn(0x7fe5e4077740 msgr2=0x7fe5e410f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.299+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/3207003974 shutdown_connections 2024-04-25T02:20:44.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.299+0000 7fe5e9cdf640 1 -- 172.21.15.31:0/3207003974 wait complete. 2024-04-25T02:20:44.314 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T02:20:44.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.415+0000 7f02dcf9b640 1 -- 172.21.15.31:0/2195712671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8100620 msgr2=0x7f02d8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.415+0000 7f02dcf9b640 1 --2- 172.21.15.31:0/2195712671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8100620 0x7f02d8100aa0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009930 tx=0x7f02c002a420 comp rx=0 tx=0).stop 2024-04-25T02:20:44.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.416+0000 7f02dcf9b640 1 -- 172.21.15.31:0/2195712671 shutdown_connections 2024-04-25T02:20:44.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.416+0000 7f02dcf9b640 1 --2- 172.21.15.31:0/2195712671 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d8100fe0 0x7f02d810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.416+0000 7f02dcf9b640 1 --2- 172.21.15.31:0/2195712671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8100620 0x7f02d8100aa0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.416+0000 7f02dcf9b640 1 --2- 172.21.15.31:0/2195712671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02d8106630 0x7f02d8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.416+0000 7f02dcf9b640 1 -- 172.21.15.31:0/2195712671 >> 172.21.15.31:0/2195712671 conn(0x7f02d80776a0 msgr2=0x7f02d8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:44.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.416+0000 7f02dcf9b640 1 -- 172.21.15.31:0/2195712671 shutdown_connections 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.416+0000 7f02dcf9b640 1 -- 172.21.15.31:0/2195712671 wait complete. 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.416+0000 7f02dcf9b640 1 Processor -- start 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.416+0000 7f02dcf9b640 1 -- start start 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02dcf9b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d8100620 0x7f02d80fd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02dcf9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8100fe0 0x7f02d80fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02dcf9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02d8106630 0x7f02d80fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02dcf9b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f02d81108f0 con 0x7f02d8100fe0 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02dcf9b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f02d8110770 con 0x7f02d8106630 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02dcf9b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f02d8110a70 con 0x7f02d8100620 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02cffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8100fe0 0x7f02d80fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02cffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8100fe0 0x7f02d80fdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55298/0 (socket says 172.21.15.31:55298) 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02cffff640 1 -- 172.21.15.31:0/3522297281 learned_addr learned my addr 172.21.15.31:0/3522297281 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02d659c640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d8100620 0x7f02d80fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02d6d9d640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02d8106630 0x7f02d80fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02cffff640 1 -- 172.21.15.31:0/3522297281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d8100620 msgr2=0x7f02d80fd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02cffff640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d8100620 0x7f02d80fd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02cffff640 1 -- 172.21.15.31:0/3522297281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02d8106630 msgr2=0x7f02d80fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02cffff640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02d8106630 0x7f02d80fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02cffff640 1 -- 172.21.15.31:0/3522297281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d81a63c0 con 0x7f02d8100fe0 2024-04-25T02:20:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.417+0000 7f02d659c640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d8100620 0x7f02d80fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:20:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.418+0000 7f02cffff640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8100fe0 0x7f02d80fdb90 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f02c0005940 tx=0x7f02c00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:44.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.420+0000 7f02cf7fe640 1 -- 172.21.15.31:0/3522297281 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c003f030 con 0x7f02d8100fe0 2024-04-25T02:20:44.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.420+0000 7f02cf7fe640 1 -- 172.21.15.31:0/3522297281 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02c00318a0 con 0x7f02d8100fe0 2024-04-25T02:20:44.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.420+0000 7f02dcf9b640 1 -- 172.21.15.31:0/3522297281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02d81a6650 con 0x7f02d8100fe0 2024-04-25T02:20:44.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.420+0000 7f02cf7fe640 1 -- 172.21.15.31:0/3522297281 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0035070 con 0x7f02d8100fe0 2024-04-25T02:20:44.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.420+0000 7f02dcf9b640 1 -- 172.21.15.31:0/3522297281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02d81a6b90 con 0x7f02d8100fe0 2024-04-25T02:20:44.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.421+0000 7f02dcf9b640 1 -- 172.21.15.31:0/3522297281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02d80689b0 con 0x7f02d8100fe0 2024-04-25T02:20:44.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.422+0000 7f02cf7fe640 1 -- 172.21.15.31:0/3522297281 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f02c0042030 con 0x7f02d8100fe0 2024-04-25T02:20:44.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.423+0000 7f02cf7fe640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02c40a2d80 0x7f02c40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.423+0000 7f02cf7fe640 1 -- 172.21.15.31:0/3522297281 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8484+0+0 (secure 0 0 0) 0x7f02c00efac0 con 0x7f02d8100fe0 2024-04-25T02:20:44.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.423+0000 7f02d659c640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02c40a2d80 0x7f02c40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.423+0000 7f02d659c640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02c40a2d80 0x7f02c40a5240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f02c8009f40 tx=0x7f02c8005f90 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:44.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.427+0000 7f02cf7fe640 1 -- 172.21.15.31:0/3522297281 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f02c00bc0b0 con 0x7f02d8100fe0 2024-04-25T02:20:44.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.611+0000 7f02dcf9b640 1 -- 172.21.15.31:0/3522297281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f02d8109ba0 con 0x7f02d8100fe0 2024-04-25T02:20:44.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.612+0000 7f02cf7fe640 1 -- 172.21.15.31:0/3522297281 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) ==== 100+0+11 (secure 0 0 0) 0x7f02c00c03b0 con 0x7f02d8100fe0 2024-04-25T02:20:44.616 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T02:20:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 -- 172.21.15.31:0/3522297281 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02c40a2d80 msgr2=0x7f02c40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02c40a2d80 0x7f02c40a5240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f02c8009f40 tx=0x7f02c8005f90 comp rx=0 tx=0).stop 2024-04-25T02:20:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 -- 172.21.15.31:0/3522297281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8100fe0 msgr2=0x7f02d80fdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8100fe0 0x7f02d80fdb90 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f02c0005940 tx=0x7f02c00042f0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 -- 172.21.15.31:0/3522297281 shutdown_connections 2024-04-25T02:20:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7f02c40a2d80 0x7f02c40a5240 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02d8106630 0x7f02d80fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d8100fe0 0x7f02d80fdb90 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 --2- 172.21.15.31:0/3522297281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02d8100620 0x7f02d80fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 -- 172.21.15.31:0/3522297281 >> 172.21.15.31:0/3522297281 conn(0x7f02d80776a0 msgr2=0x7f02d806a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 -- 172.21.15.31:0/3522297281 shutdown_connections 2024-04-25T02:20:44.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.614+0000 7f02dcf9b640 1 -- 172.21.15.31:0/3522297281 wait complete. 2024-04-25T02:20:44.627 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:44 smithi031 ceph-mon[29204]: pgmap v829: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T02:20:44.628 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:44 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2612358987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:20:44.628 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:44 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/735896184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:20:44.628 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:44 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3207003974' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:20:44.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T02:20:44.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:44 smithi134 ceph-mon[33098]: pgmap v829: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T02:20:44.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:44 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2612358987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:20:44.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:44 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/735896184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:20:44.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:44 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3207003974' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:20:44.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.728+0000 7fc9fda7d640 1 -- 172.21.15.31:0/874261515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81020d0 msgr2=0x7fc9f810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.728+0000 7fc9fda7d640 1 --2- 172.21.15.31:0/874261515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81020d0 0x7fc9f810e5e0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009cd0 tx=0x7fc9f002a8e0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.728+0000 7fc9fda7d640 1 -- 172.21.15.31:0/874261515 shutdown_connections 2024-04-25T02:20:44.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.728+0000 7fc9fda7d640 1 --2- 172.21.15.31:0/874261515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81020d0 0x7fc9f810e5e0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.728+0000 7fc9fda7d640 1 --2- 172.21.15.31:0/874261515 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9f8101710 0x7fc9f8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.728+0000 7fc9fda7d640 1 --2- 172.21.15.31:0/874261515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9f8107720 0x7fc9f8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.728+0000 7fc9fda7d640 1 -- 172.21.15.31:0/874261515 >> 172.21.15.31:0/874261515 conn(0x7fc9f80fd180 msgr2=0x7fc9f80ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.728+0000 7fc9fda7d640 1 -- 172.21.15.31:0/874261515 shutdown_connections 2024-04-25T02:20:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.728+0000 7fc9fda7d640 1 -- 172.21.15.31:0/874261515 wait complete. 2024-04-25T02:20:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.728+0000 7fc9fda7d640 1 Processor -- start 2024-04-25T02:20:44.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9fda7d640 1 -- start start 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9fda7d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9f8101710 0x7fc9f8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9fda7d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81020d0 0x7fc9f81984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9fda7d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9f8107720 0x7fc9f819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9fda7d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f81119e0 con 0x7fc9f81020d0 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9fda7d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f8111860 con 0x7fc9f8101710 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9fda7d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f8111b60 con 0x7fc9f8107720 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81020d0 0x7fc9f81984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81020d0 0x7fc9f81984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55318/0 (socket says 172.21.15.31:55318) 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f6ffd640 1 -- 172.21.15.31:0/2605074263 learned_addr learned my addr 172.21.15.31:0/2605074263 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f77fe640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9f8101710 0x7fc9f8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f7fff640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9f8107720 0x7fc9f819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f6ffd640 1 -- 172.21.15.31:0/2605074263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9f8107720 msgr2=0x7fc9f819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f6ffd640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9f8107720 0x7fc9f819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f6ffd640 1 -- 172.21.15.31:0/2605074263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9f8101710 msgr2=0x7fc9f8197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f6ffd640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9f8101710 0x7fc9f8197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f6ffd640 1 -- 172.21.15.31:0/2605074263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f819cf50 con 0x7fc9f81020d0 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f7fff640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9f8107720 0x7fc9f819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.729+0000 7fc9f77fe640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9f8101710 0x7fc9f8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.730+0000 7fc9f6ffd640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81020d0 0x7fc9f81984a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00d770 tx=0x7fc9ec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.730+0000 7fc9f4ff9640 1 -- 172.21.15.31:0/2605074263 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec017070 con 0x7fc9f81020d0 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.730+0000 7fc9f4ff9640 1 -- 172.21.15.31:0/2605074263 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9ec0054d0 con 0x7fc9f81020d0 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.730+0000 7fc9fda7d640 1 -- 172.21.15.31:0/2605074263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f819d240 con 0x7fc9f81020d0 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.730+0000 7fc9f4ff9640 1 -- 172.21.15.31:0/2605074263 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec004cf0 con 0x7fc9f81020d0 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.730+0000 7fc9fda7d640 1 -- 172.21.15.31:0/2605074263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f81a4ad0 con 0x7fc9f81020d0 2024-04-25T02:20:44.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.731+0000 7fc9fda7d640 1 -- 172.21.15.31:0/2605074263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f80689b0 con 0x7fc9f81020d0 2024-04-25T02:20:44.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.732+0000 7fc9f4ff9640 1 -- 172.21.15.31:0/2605074263 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc9ec004e90 con 0x7fc9f81020d0 2024-04-25T02:20:44.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.733+0000 7fc9f4ff9640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9e00a2d00 0x7fc9e00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:20:44.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.733+0000 7fc9f77fe640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9e00a2d00 0x7fc9e00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:20:44.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.733+0000 7fc9f4ff9640 1 -- 172.21.15.31:0/2605074263 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8484+0+0 (secure 0 0 0) 0x7fc9ec0cf390 con 0x7fc9f81020d0 2024-04-25T02:20:44.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.733+0000 7fc9f77fe640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9e00a2d00 0x7fc9e00a51c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8005fd0 tx=0x7fc9e8005850 comp rx=0 tx=0).ready entity=mgr.15351 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:20:44.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.737+0000 7fc9f4ff9640 1 -- 172.21.15.31:0/2605074263 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc9ec09bf70 con 0x7fc9f81020d0 2024-04-25T02:20:44.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:44 smithi040 ceph-mon[32418]: pgmap v829: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T02:20:44.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:44 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2612358987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:20:44.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:44 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/735896184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:20:44.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:44 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3207003974' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:20:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.937+0000 7fc9fda7d640 1 -- 172.21.15.31:0/2605074263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc9f81992b0 con 0x7fc9f81020d0 2024-04-25T02:20:44.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.937+0000 7fc9f4ff9640 1 -- 172.21.15.31:0/2605074263 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v129) ==== 81+0+943 (secure 0 0 0) 0x7fc9f81992b0 con 0x7fc9f81020d0 2024-04-25T02:20:44.941 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3045817976 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:44.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/2306032998 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:44.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/2306032998 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:44.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/363866598 2024-04-26T01:45:23.647161+0000 2024-04-25T02:20:44.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/2433894874 2024-04-26T01:45:23.647161+0000 2024-04-25T02:20:44.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1358173279 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:44.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/3170545029 2024-04-26T01:45:23.647161+0000 2024-04-25T02:20:44.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/3170545029 2024-04-26T01:45:23.647161+0000 2024-04-25T02:20:44.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/686937010 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:44.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/196728475 2024-04-26T01:44:56.414005+0000 2024-04-25T02:20:44.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3913527371 2024-04-26T01:54:28.579184+0000 2024-04-25T02:20:44.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/2254757453 2024-04-26T01:44:56.414005+0000 2024-04-25T02:20:44.943 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/3153405418 2024-04-26T01:44:56.414005+0000 2024-04-25T02:20:44.943 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/3153405418 2024-04-26T01:44:56.414005+0000 2024-04-25T02:20:44.943 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1967729027 2024-04-26T01:45:23.647161+0000 2024-04-25T02:20:44.943 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/1543547115 2024-04-26T01:44:56.414005+0000 2024-04-25T02:20:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:listed 16 entries 2024-04-25T02:20:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.939+0000 7fc9fda7d640 1 -- 172.21.15.31:0/2605074263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9e00a2d00 msgr2=0x7fc9e00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.939+0000 7fc9fda7d640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9e00a2d00 0x7fc9e00a51c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8005fd0 tx=0x7fc9e8005850 comp rx=0 tx=0).stop 2024-04-25T02:20:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.939+0000 7fc9fda7d640 1 -- 172.21.15.31:0/2605074263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81020d0 msgr2=0x7fc9f81984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:20:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.939+0000 7fc9fda7d640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81020d0 0x7fc9f81984a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00d770 tx=0x7fc9ec00dc30 comp rx=0 tx=0).stop 2024-04-25T02:20:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.939+0000 7fc9fda7d640 1 -- 172.21.15.31:0/2605074263 shutdown_connections 2024-04-25T02:20:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.939+0000 7fc9fda7d640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.31:6800/1834919992,v1:172.21.15.31:6801/1834919992] conn(0x7fc9e00a2d00 0x7fc9e00a51c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.939+0000 7fc9fda7d640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9f8107720 0x7fc9f819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.939+0000 7fc9fda7d640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f81020d0 0x7fc9f81984a0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.939+0000 7fc9fda7d640 1 --2- 172.21.15.31:0/2605074263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9f8101710 0x7fc9f8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:20:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.939+0000 7fc9fda7d640 1 -- 172.21.15.31:0/2605074263 >> 172.21.15.31:0/2605074263 conn(0x7fc9f80fd180 msgr2=0x7fc9f80fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:20:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.940+0000 7fc9fda7d640 1 -- 172.21.15.31:0/2605074263 shutdown_connections 2024-04-25T02:20:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T02:20:44.940+0000 7fc9fda7d640 1 -- 172.21.15.31:0/2605074263 wait complete. 2024-04-25T02:20:44.954 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi031.front.sepia.ceph.com... 2024-04-25T02:20:44.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:20:44.954 DEBUG:teuthology.orchestra.run.smithi031:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-25T02:20:44.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi031.stderr:ceph-fuse[81038]: fuse finished with error 0 and tester_r 0 2024-04-25T02:20:44.985 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T02:20:45.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:45 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/3522297281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:20:45.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:45 smithi134 ceph-mon[33098]: from='client.? 172.21.15.31:0/2605074263' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T02:20:45.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:45 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/3522297281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:20:45.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:45 smithi031 ceph-mon[29204]: from='client.? 172.21.15.31:0/2605074263' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T02:20:45.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:45 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/3522297281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:20:45.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:45 smithi040 ceph-mon[32418]: from='client.? 172.21.15.31:0/2605074263' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T02:20:46.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:46 smithi134 ceph-mon[33098]: pgmap v830: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2024-04-25T02:20:46.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:46 smithi031 ceph-mon[29204]: pgmap v830: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2024-04-25T02:20:46.913 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:46 smithi040 ceph-mon[32418]: pgmap v830: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2024-04-25T02:20:47.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:47 smithi031 ceph-mon[29204]: pgmap v831: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-25T02:20:47.912 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:47 smithi040 ceph-mon[32418]: pgmap v831: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-25T02:20:47.942 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:47 smithi134 ceph-mon[33098]: pgmap v831: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-25T02:20:50.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:50 smithi031 ceph-mon[29204]: pgmap v832: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-25T02:20:50.412 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:50 smithi040 ceph-mon[32418]: pgmap v832: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-25T02:20:50.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:50 smithi134 ceph-mon[33098]: pgmap v832: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-25T02:20:51.087 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T02:20:51.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:20:51.087 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T02:20:51.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:20:51.124 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-04-25T02:20:51.190 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T02:20:51.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:20:51.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T02:20:51.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:20:51.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-04-25T02:20:51.366 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T02:20:51.377 INFO:tasks.cephadm:Teardown begin 2024-04-25T02:20:51.377 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/cephadm.py", line 2240, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-04-25T02:20:51.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:20:51.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:20:51.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:20:51.467 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T02:20:51.467 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T02:20:51.568 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T02:20:51.582 DEBUG:teuthology.orchestra.run.smithi134:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T02:20:51.600 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T02:20:51.600 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T02:20:51.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.a 2024-04-25T02:20:51.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:51 smithi031 systemd[1]: Stopping Ceph mon.a for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:20:52.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:51 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a[29200]: 2024-04-25T02:20:51.931+0000 7fc41bddf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:20:52.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:51 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a[29200]: 2024-04-25T02:20:51.931+0000 7fc41bddf700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T02:20:52.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:52 smithi031 podman[100975]: 2024-04-25 02:20:52.084619993 +0000 UTC m=+0.281867237 container died 9b79660ad9d9b4bb16bd84f0ee4d8b683d2ae6bb527324878cb032828214574b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-04-25T02:20:52.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:52 smithi031 podman[100975]: 2024-04-25 02:20:52.490404148 +0000 UTC m=+0.687651390 container cleanup 9b79660ad9d9b4bb16bd84f0ee4d8b683d2ae6bb527324878cb032828214574b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T02:20:52.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:52 smithi031 bash[100975]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a 2024-04-25T02:20:53.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 02:20:52 smithi031 podman[101055]: 2024-04-25 02:20:52.758812988 +0000 UTC m=+0.672876821 container remove 9b79660ad9d9b4bb16bd84f0ee4d8b683d2ae6bb527324878cb032828214574b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:20:53.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T02:20:53.272 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:20:53.273 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T02:20:53.273 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T02:20:53.273 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.b 2024-04-25T02:20:53.662 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:53 smithi040 systemd[1]: Stopping Ceph mon.b for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:20:53.996 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:53 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-b[32414]: 2024-04-25T02:20:53.657+0000 7f1e2d40f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:20:53.997 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:53 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-b[32414]: 2024-04-25T02:20:53.657+0000 7f1e2d40f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T02:20:53.997 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:53 smithi040 podman[73017]: 2024-04-25 02:20:53.730301807 +0000 UTC m=+0.200417165 container died 6695eaeb0133542ceea77d2d5bea3d67affcc8a5eb58c4268e162fcb25e95dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T02:20:54.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:54 smithi040 podman[73017]: 2024-04-25 02:20:54.083934706 +0000 UTC m=+0.554050070 container cleanup 6695eaeb0133542ceea77d2d5bea3d67affcc8a5eb58c4268e162fcb25e95dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-25T02:20:54.405 INFO:journalctl@ceph.mon.b.smithi040.stdout:Apr 25 02:20:54 smithi040 bash[73017]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-b 2024-04-25T02:20:54.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T02:20:54.564 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:20:54.564 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T02:20:54.565 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T02:20:54.565 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.c 2024-04-25T02:20:54.911 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:54 smithi134 systemd[1]: Stopping Ceph mon.c for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:20:55.192 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:54 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-c[33079]: 2024-04-25T02:20:54.907+0000 7f61d1030700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:20:55.192 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:54 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-c[33079]: 2024-04-25T02:20:54.907+0000 7f61d1030700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T02:20:55.192 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:54 smithi134 podman[73053]: 2024-04-25 02:20:54.978575417 +0000 UTC m=+0.186469843 container died 5ff0d2aaefdcc7ddce3ad6e537b07d8c39fdb1a6f1b422578070b4fe6462e01e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T02:20:55.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:55 smithi134 podman[73053]: 2024-04-25 02:20:55.457806755 +0000 UTC m=+0.665701174 container cleanup 5ff0d2aaefdcc7ddce3ad6e537b07d8c39fdb1a6f1b422578070b4fe6462e01e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T02:20:55.692 INFO:journalctl@ceph.mon.c.smithi134.stdout:Apr 25 02:20:55 smithi134 bash[73053]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mon-c 2024-04-25T02:20:55.899 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T02:20:55.945 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:20:55.945 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T02:20:55.946 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T02:20:55.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.x 2024-04-25T02:20:56.398 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 02:20:55 smithi031 systemd[1]: Stopping Ceph mgr.x for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:20:56.991 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 02:20:56 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x[29434]: 2024-04-25T02:20:56.697+0000 7f7dee786700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:20:56.991 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 02:20:56 smithi031 podman[103832]: 2024-04-25 02:20:56.728718821 +0000 UTC m=+0.160156370 container died 8a3e25b0c23cd91357918bafd5644422cde5dd1ca8c3a51c8f6badd470ce5958 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-25T02:20:57.429 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 02:20:57 smithi031 podman[103832]: 2024-04-25 02:20:57.149250081 +0000 UTC m=+0.580687653 container cleanup 8a3e25b0c23cd91357918bafd5644422cde5dd1ca8c3a51c8f6badd470ce5958 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T02:20:57.429 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 02:20:57 smithi031 bash[103832]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-x 2024-04-25T02:20:57.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T02:20:57.616 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:20:57.616 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T02:20:57.616 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T02:20:57.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.y 2024-04-25T02:20:57.912 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 02:20:57 smithi040 systemd[1]: Stopping Ceph mgr.y for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:20:58.243 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 02:20:57 smithi040 podman[73140]: 2024-04-25 02:20:57.951358667 +0000 UTC m=+0.142769859 container died b66e1f6b463d956a929cca7bc5ff887448b026a370000fbca5bd8a5dba184246 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:20:58.602 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 02:20:58 smithi040 podman[73140]: 2024-04-25 02:20:58.339545123 +0000 UTC m=+0.530956307 container cleanup b66e1f6b463d956a929cca7bc5ff887448b026a370000fbca5bd8a5dba184246 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T02:20:58.602 INFO:journalctl@ceph.mgr.y.smithi040.stdout:Apr 25 02:20:58 smithi040 bash[73140]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-y 2024-04-25T02:20:58.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T02:20:58.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:20:58.813 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T02:20:58.813 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T02:20:58.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.z 2024-04-25T02:20:59.144 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 02:20:58 smithi134 systemd[1]: Stopping Ceph mgr.z for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:20:59.404 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 02:20:59 smithi134 podman[73175]: 2024-04-25 02:20:59.141253713 +0000 UTC m=+0.142255995 container died 9bee8c413a8c21ec1e3e6ec280a99124f5c3b1069d9b11af7b2091398ea42d08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-25T02:20:59.692 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 02:20:59 smithi134 podman[73175]: 2024-04-25 02:20:59.487784314 +0000 UTC m=+0.488786592 container cleanup 9bee8c413a8c21ec1e3e6ec280a99124f5c3b1069d9b11af7b2091398ea42d08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-d2c45d9) 2024-04-25T02:20:59.692 INFO:journalctl@ceph.mgr.z.smithi134.stdout:Apr 25 02:20:59 smithi134 bash[73175]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mgr-z 2024-04-25T02:20:59.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T02:20:59.935 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:20:59.936 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T02:20:59.936 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T02:20:59.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.0 2024-04-25T02:21:00.241 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:20:59 smithi031 systemd[1]: Stopping Ceph osd.0 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:21:00.741 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0[43534]: 2024-04-25T02:21:00.253+0000 7f48278b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:21:00.741 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0[43534]: 2024-04-25T02:21:00.253+0000 7f48278b8700 -1 osd.0 129 *** Got signal Terminated *** 2024-04-25T02:21:00.741 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0[43534]: 2024-04-25T02:21:00.253+0000 7f48278b8700 -1 osd.0 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:21:06.175 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:05 smithi031 podman[103952]: 2024-04-25 02:21:05.86351745 +0000 UTC m=+5.735369327 container died 385638b09d1c67de337b9a292e08f2a1a555696e04ddbef9acde60212ecad5e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:21:06.491 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:06 smithi031 podman[103952]: 2024-04-25 02:21:06.263119067 +0000 UTC m=+6.134970981 container cleanup 385638b09d1c67de337b9a292e08f2a1a555696e04ddbef9acde60212ecad5e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-d2c45d9) 2024-04-25T02:21:06.491 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:06 smithi031 bash[103952]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0 2024-04-25T02:21:06.767 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:06 smithi031 podman[103972]: 2024-04-25 02:21:06.514805258 +0000 UTC m=+0.645729153 container remove 385638b09d1c67de337b9a292e08f2a1a555696e04ddbef9acde60212ecad5e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:21:07.116 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:06 smithi031 podman[104064]: 2024-04-25 02:21:06.756832745 +0000 UTC m=+0.019276161 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:21:07.116 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:06 smithi031 podman[104064]: 2024-04-25 02:21:06.883027623 +0000 UTC m=+0.145471015 container create 181b50fadaa0e319aee1ca025eef96b9a2b31835bc8bb12e639ff91b66aea779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T02:21:07.117 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:07 smithi031 podman[104064]: 2024-04-25 02:21:07.113404973 +0000 UTC m=+0.375848365 container init 181b50fadaa0e319aee1ca025eef96b9a2b31835bc8bb12e639ff91b66aea779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:21:07.434 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:07 smithi031 podman[104064]: 2024-04-25 02:21:07.116642879 +0000 UTC m=+0.379086274 container start 181b50fadaa0e319aee1ca025eef96b9a2b31835bc8bb12e639ff91b66aea779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T02:21:07.434 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:07 smithi031 podman[104064]: 2024-04-25 02:21:07.227879285 +0000 UTC m=+0.490322692 container attach 181b50fadaa0e319aee1ca025eef96b9a2b31835bc8bb12e639ff91b66aea779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:21:07.741 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 02:21:07 smithi031 podman[104064]: 2024-04-25 02:21:07.431595351 +0000 UTC m=+0.694038742 container died 181b50fadaa0e319aee1ca025eef96b9a2b31835bc8bb12e639ff91b66aea779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T02:21:08.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T02:21:08.154 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:21:08.154 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T02:21:08.155 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T02:21:08.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.1 2024-04-25T02:21:08.482 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:08 smithi040 systemd[1]: Stopping Ceph osd.1 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:21:08.912 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:08 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1[37010]: 2024-04-25T02:21:08.477+0000 7ffaecc2f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:21:08.912 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:08 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1[37010]: 2024-04-25T02:21:08.477+0000 7ffaecc2f700 -1 osd.1 129 *** Got signal Terminated *** 2024-04-25T02:21:08.913 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:08 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1[37010]: 2024-04-25T02:21:08.477+0000 7ffaecc2f700 -1 osd.1 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:21:14.341 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:14 smithi040 podman[73265]: 2024-04-25 02:21:14.029743833 +0000 UTC m=+5.677860303 container died b712b4be767eb7116e41858a496c08e1086db07e019ed3a9ecb64f10c70bbd43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9) 2024-04-25T02:21:14.662 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:14 smithi040 podman[73265]: 2024-04-25 02:21:14.429026002 +0000 UTC m=+6.077142465 container cleanup b712b4be767eb7116e41858a496c08e1086db07e019ed3a9ecb64f10c70bbd43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T02:21:14.663 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:14 smithi040 bash[73265]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1 2024-04-25T02:21:15.016 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:14 smithi040 podman[73284]: 2024-04-25 02:21:14.688848329 +0000 UTC m=+0.658424025 container remove b712b4be767eb7116e41858a496c08e1086db07e019ed3a9ecb64f10c70bbd43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-25T02:21:15.304 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:15 smithi040 podman[73379]: 2024-04-25 02:21:14.916710498 +0000 UTC m=+0.018407380 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:21:15.304 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:15 smithi040 podman[73379]: 2024-04-25 02:21:15.031883756 +0000 UTC m=+0.133580614 container create df9f692dd6faf79055a999b49b9cf2851e7ac4f461a6447d57a4a91b3bfe1e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T02:21:15.304 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:15 smithi040 podman[73379]: 2024-04-25 02:21:15.301343109 +0000 UTC m=+0.403039965 container init df9f692dd6faf79055a999b49b9cf2851e7ac4f461a6447d57a4a91b3bfe1e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:21:15.662 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:15 smithi040 podman[73379]: 2024-04-25 02:21:15.304895749 +0000 UTC m=+0.406592605 container start df9f692dd6faf79055a999b49b9cf2851e7ac4f461a6447d57a4a91b3bfe1e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:21:15.662 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:15 smithi040 podman[73379]: 2024-04-25 02:21:15.401474796 +0000 UTC m=+0.503171663 container attach df9f692dd6faf79055a999b49b9cf2851e7ac4f461a6447d57a4a91b3bfe1e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:21:16.163 INFO:journalctl@ceph.osd.1.smithi040.stdout:Apr 25 02:21:15 smithi040 podman[73379]: 2024-04-25 02:21:15.825086027 +0000 UTC m=+0.926782894 container died df9f692dd6faf79055a999b49b9cf2851e7ac4f461a6447d57a4a91b3bfe1e16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T02:21:16.616 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T02:21:16.660 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:21:16.660 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T02:21:16.660 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T02:21:16.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.2 2024-04-25T02:21:16.942 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:16 smithi134 systemd[1]: Stopping Ceph osd.2 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:21:17.209 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:16 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2[38239]: 2024-04-25T02:21:16.974+0000 7f6540997700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:21:17.210 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:16 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2[38239]: 2024-04-25T02:21:16.974+0000 7f6540997700 -1 osd.2 129 *** Got signal Terminated *** 2024-04-25T02:21:17.210 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:16 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2[38239]: 2024-04-25T02:21:16.974+0000 7f6540997700 -1 osd.2 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:21:22.730 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:22 smithi134 podman[73299]: 2024-04-25 02:21:22.458203237 +0000 UTC m=+5.605242897 container died 33a3e491f4f74f6b0044c2963a85ed06e07415e65984416286dbdcd574df2f1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T02:21:23.105 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:22 smithi134 podman[73299]: 2024-04-25 02:21:22.825427232 +0000 UTC m=+5.972466892 container cleanup 33a3e491f4f74f6b0044c2963a85ed06e07415e65984416286dbdcd574df2f1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T02:21:23.105 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:22 smithi134 bash[73299]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2 2024-04-25T02:21:23.420 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:23 smithi134 podman[73328]: 2024-04-25 02:21:23.102111555 +0000 UTC m=+0.643421536 container remove 33a3e491f4f74f6b0044c2963a85ed06e07415e65984416286dbdcd574df2f1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:21:23.692 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:23 smithi134 podman[73421]: 2024-04-25 02:21:23.320384192 +0000 UTC m=+0.018496326 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:21:23.692 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:23 smithi134 podman[73421]: 2024-04-25 02:21:23.452982839 +0000 UTC m=+0.151094948 container create 317771ceb36126bf89956fe3cd0ab1c2a15d5faeabc4cd903286d4c1431b12a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T02:21:24.192 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:23 smithi134 podman[73421]: 2024-04-25 02:21:23.714056697 +0000 UTC m=+0.412168804 container init 317771ceb36126bf89956fe3cd0ab1c2a15d5faeabc4cd903286d4c1431b12a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:21:24.192 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:23 smithi134 podman[73421]: 2024-04-25 02:21:23.717246234 +0000 UTC m=+0.415358341 container start 317771ceb36126bf89956fe3cd0ab1c2a15d5faeabc4cd903286d4c1431b12a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T02:21:24.192 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:23 smithi134 podman[73421]: 2024-04-25 02:21:23.830736066 +0000 UTC m=+0.528848192 container attach 317771ceb36126bf89956fe3cd0ab1c2a15d5faeabc4cd903286d4c1431b12a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, ceph=True, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-25T02:21:24.604 INFO:journalctl@ceph.osd.2.smithi134.stdout:Apr 25 02:21:24 smithi134 podman[73421]: 2024-04-25 02:21:24.253299783 +0000 UTC m=+0.951411895 container died 317771ceb36126bf89956fe3cd0ab1c2a15d5faeabc4cd903286d4c1431b12a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T02:21:24.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T02:21:24.956 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:21:24.956 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T02:21:24.956 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T02:21:24.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.3 2024-04-25T02:21:25.241 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:25 smithi031 systemd[1]: Stopping Ceph osd.3 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:21:25.740 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:25 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3[50065]: 2024-04-25T02:21:25.285+0000 7fee106ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:21:25.741 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:25 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3[50065]: 2024-04-25T02:21:25.285+0000 7fee106ff700 -1 osd.3 129 *** Got signal Terminated *** 2024-04-25T02:21:25.741 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:25 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3[50065]: 2024-04-25T02:21:25.285+0000 7fee106ff700 -1 osd.3 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:21:26.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:26.543+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:27.692 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:27 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:27.568+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:27.692 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:27 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:27.436+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:27.741 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:27 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3[50065]: 2024-04-25T02:21:27.423+0000 7fee0ccf3700 -1 osd.3 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.588481+0000 front 2024-04-25T02:21:00.588506+0000 (oldest deadline 2024-04-25T02:21:26.488591+0000) 2024-04-25T02:21:28.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:27 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:27.954+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:28.692 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:28 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:28.589+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:28.692 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:28 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:28.658+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:28.692 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:28 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:28.418+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:28.740 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:28 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3[50065]: 2024-04-25T02:21:28.425+0000 7fee0ccf3700 -1 osd.3 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.588481+0000 front 2024-04-25T02:21:00.588506+0000 (oldest deadline 2024-04-25T02:21:26.488591+0000) 2024-04-25T02:21:29.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:28 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:28.962+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:29.680 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:29 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:29.397+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:29.681 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:29 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:29.615+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:29.741 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:29 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3[50065]: 2024-04-25T02:21:29.421+0000 7fee0ccf3700 -1 osd.3 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.588481+0000 front 2024-04-25T02:21:00.588506+0000 (oldest deadline 2024-04-25T02:21:26.488591+0000) 2024-04-25T02:21:29.741 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:29 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:29.648+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:29.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:29 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:29.677+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:30.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:29 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:29.928+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:30.692 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:30.566+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:30.692 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:30.634+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:30.692 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:30.404+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:30.872 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:30 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:30.636+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:30.872 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:30 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:30.500+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:31.172 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:30 smithi031 podman[104183]: 2024-04-25 02:21:30.869363819 +0000 UTC m=+5.705682848 container died cd83febc388ce72e36829a29ad1c8836c4ab5741469076489a72e28e2bccda61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T02:21:31.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:30 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:30.911+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:31.464 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:31 smithi031 podman[104183]: 2024-04-25 02:21:31.260000269 +0000 UTC m=+6.096319297 container cleanup cd83febc388ce72e36829a29ad1c8836c4ab5741469076489a72e28e2bccda61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T02:21:31.464 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:31 smithi031 bash[104183]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3 2024-04-25T02:21:31.740 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:31 smithi031 podman[104200]: 2024-04-25 02:21:31.528267684 +0000 UTC m=+0.648904462 container remove cd83febc388ce72e36829a29ad1c8836c4ab5741469076489a72e28e2bccda61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9) 2024-04-25T02:21:31.741 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:31 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:31.650+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:31.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:31 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:31.460+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:31.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:31.555+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:31.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:31.616+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:31.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:31.450+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:32.168 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:31 smithi031 podman[104289]: 2024-04-25 02:21:31.749732055 +0000 UTC m=+0.017793760 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:21:32.168 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:31 smithi031 podman[104289]: 2024-04-25 02:21:31.854330402 +0000 UTC m=+0.122392134 container create 4cfab137bce46abac98de975a6d9f93c6845a17d9b7eae4c1685b68aaa3e60c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:21:32.168 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:32 smithi031 podman[104289]: 2024-04-25 02:21:32.065436619 +0000 UTC m=+0.333498306 container init 4cfab137bce46abac98de975a6d9f93c6845a17d9b7eae4c1685b68aaa3e60c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:21:32.168 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:32 smithi031 podman[104289]: 2024-04-25 02:21:32.068546051 +0000 UTC m=+0.336607738 container start 4cfab137bce46abac98de975a6d9f93c6845a17d9b7eae4c1685b68aaa3e60c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T02:21:32.168 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:32 smithi031 podman[104289]: 2024-04-25 02:21:32.165483504 +0000 UTC m=+0.433545200 container attach 4cfab137bce46abac98de975a6d9f93c6845a17d9b7eae4c1685b68aaa3e60c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T02:21:32.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:31 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:31.946+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:32.429 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 02:21:32 smithi031 podman[104289]: 2024-04-25 02:21:32.371581612 +0000 UTC m=+0.639643290 container died 4cfab137bce46abac98de975a6d9f93c6845a17d9b7eae4c1685b68aaa3e60c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5) 2024-04-25T02:21:32.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:32 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:32.425+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:32.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:32 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:32.656+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:32.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:32 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:32.496+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:32.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:32 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:32.581+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:32.989 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:32 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:32.696+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:33.019 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T02:21:33.061 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:21:33.061 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T02:21:33.061 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T02:21:33.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.4 2024-04-25T02:21:33.380 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:33 smithi040 systemd[1]: Stopping Ceph osd.4 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:21:33.380 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:32 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:32.920+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:33.654 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:33 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:33.405+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:33.662 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:33 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4[43546]: 2024-04-25T02:21:33.376+0000 7f8d4db02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:21:33.662 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:33 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4[43546]: 2024-04-25T02:21:33.376+0000 7f8d4db02700 -1 osd.4 129 *** Got signal Terminated *** 2024-04-25T02:21:33.663 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:33 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4[43546]: 2024-04-25T02:21:33.376+0000 7f8d4db02700 -1 osd.4 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:21:33.943 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:33 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:33.619+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:33.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:33 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:33.684+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:33.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:33 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:33.526+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:33.990 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:33 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:33.649+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:34.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:33 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:33.956+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:34.607 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:34.559+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:34.607 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:34.559+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:34.607 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:34.604+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:34.699 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:34 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:34.413+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:34.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:34.679+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:34.990 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:34 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:34.695+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:35.162 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:35 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:35.041+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:35.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:34 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:34.907+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:35.705 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:35 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:35.420+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:35.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:35 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:35.645+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:35.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:35 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:35.668+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:35.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:35 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:35.571+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:35.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:35 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:35.571+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:35.990 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:35 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:35.701+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:35.991 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:35 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:35.701+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:36.162 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:36 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:36.000+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:36.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:35 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:35.905+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:36.662 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:36 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4[43546]: 2024-04-25T02:21:36.205+0000 7f8d4a0f6700 -1 osd.4 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.606361+0000 front 2024-04-25T02:21:10.606391+0000 (oldest deadline 2024-04-25T02:21:35.906559+0000) 2024-04-25T02:21:36.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:36 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:36.458+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:36.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:36 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:36.458+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:36.741 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:36 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:36.685+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:36.741 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:36 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:36.685+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:36.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:36 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:36.673+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:36.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:36 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:36.619+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:36.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:36 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:36.619+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.298508+0000 front 2024-04-25T02:21:11.298605+0000 (oldest deadline 2024-04-25T02:21:36.598586+0000) 2024-04-25T02:21:36.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:36 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:36.571+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:36.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:36 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:36.571+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:37.162 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:37 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:37.046+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:37.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:36 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:36.903+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:37.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:36 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:36.903+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:37.663 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:37 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4[43546]: 2024-04-25T02:21:37.178+0000 7f8d4a0f6700 -1 osd.4 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.606361+0000 front 2024-04-25T02:21:10.606391+0000 (oldest deadline 2024-04-25T02:21:35.906559+0000) 2024-04-25T02:21:37.676 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:37 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:37.420+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:37.676 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:37 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:37.420+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:37.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:37 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:37.649+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:37.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:37 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:37.615+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:37.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:37 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:37.615+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.298508+0000 front 2024-04-25T02:21:11.298605+0000 (oldest deadline 2024-04-25T02:21:36.598586+0000) 2024-04-25T02:21:37.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:37 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:37.612+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:37.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:37 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:37.612+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:37.990 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:37 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:37.672+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:37.991 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:37 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:37.672+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:38.412 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:38 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4[43546]: 2024-04-25T02:21:38.168+0000 7f8d4a0f6700 -1 osd.4 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.606361+0000 front 2024-04-25T02:21:10.606391+0000 (oldest deadline 2024-04-25T02:21:35.906559+0000) 2024-04-25T02:21:38.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:38 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:38.057+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:38.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:37 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:37.953+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:38.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:37 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:37.953+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:38.695 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:38 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:38.393+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:38.695 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:38 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:38.393+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:38.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:38 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:38.583+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:38.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:38 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:38.583+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:38.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:38 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:38.665+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:38.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:38 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:38.656+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:38.943 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:38 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:38.656+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.298508+0000 front 2024-04-25T02:21:11.298605+0000 (oldest deadline 2024-04-25T02:21:36.598586+0000) 2024-04-25T02:21:38.990 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:38 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:38.691+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:38.990 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:38 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:38.691+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:39.125 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:38 smithi040 podman[73497]: 2024-04-25 02:21:38.809781118 +0000 UTC m=+5.551458468 container died efa15a4fb0d5e99763ceecad2be0351702531f6ed3ba52bd3f95f97e35bb9874 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:21:39.125 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:39 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:39.052+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:39.125 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:38 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:38.931+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:39.125 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:38 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:38.931+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:39.413 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:39 smithi040 podman[73497]: 2024-04-25 02:21:39.22957027 +0000 UTC m=+5.971247651 container cleanup efa15a4fb0d5e99763ceecad2be0351702531f6ed3ba52bd3f95f97e35bb9874 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T02:21:39.413 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:39 smithi040 bash[73497]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4 2024-04-25T02:21:39.708 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:39 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:39.380+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:39.708 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:39 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:39.380+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:39.887 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:39 smithi040 podman[73514]: 2024-04-25 02:21:39.464249676 +0000 UTC m=+0.651571355 container remove efa15a4fb0d5e99763ceecad2be0351702531f6ed3ba52bd3f95f97e35bb9874 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9) 2024-04-25T02:21:39.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:39 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:39.619+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:39.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:39 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:39.619+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:39.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:39 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:39.673+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:39.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:39 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:39.673+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.298508+0000 front 2024-04-25T02:21:11.298605+0000 (oldest deadline 2024-04-25T02:21:36.598586+0000) 2024-04-25T02:21:39.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:39 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:39.707+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:39.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:39 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:39.707+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:39.990 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:39 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:39.713+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:39.990 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:39 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:39.713+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:40.162 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:40 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:40.023+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:40.163 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:39 smithi040 podman[73604]: 2024-04-25 02:21:39.787082272 +0000 UTC m=+0.018576186 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:21:40.163 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:39 smithi040 podman[73604]: 2024-04-25 02:21:39.915176737 +0000 UTC m=+0.146670630 container create 589ac1ba7e97a1a7bb1a54fe7a2462a95b7ad7f1c992ddc96937ba6dc52ba52f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T02:21:40.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:39 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:39.918+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:40.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:39 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:39.918+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:40.480 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:40 smithi040 podman[73604]: 2024-04-25 02:21:40.168169435 +0000 UTC m=+0.399663343 container init 589ac1ba7e97a1a7bb1a54fe7a2462a95b7ad7f1c992ddc96937ba6dc52ba52f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T02:21:40.480 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:40 smithi040 podman[73604]: 2024-04-25 02:21:40.171391633 +0000 UTC m=+0.402885528 container start 589ac1ba7e97a1a7bb1a54fe7a2462a95b7ad7f1c992ddc96937ba6dc52ba52f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:21:40.480 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:40 smithi040 podman[73604]: 2024-04-25 02:21:40.368335216 +0000 UTC m=+0.599829117 container attach 589ac1ba7e97a1a7bb1a54fe7a2462a95b7ad7f1c992ddc96937ba6dc52ba52f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T02:21:40.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:40 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:40.383+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:40.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:40 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:40.383+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:40.765 INFO:journalctl@ceph.osd.4.smithi040.stdout:Apr 25 02:21:40 smithi040 podman[73604]: 2024-04-25 02:21:40.477744347 +0000 UTC m=+0.709238248 container died 589ac1ba7e97a1a7bb1a54fe7a2462a95b7ad7f1c992ddc96937ba6dc52ba52f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-25T02:21:40.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:40 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:40.658+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:40.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:40 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:40.658+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:40.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:40 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:40.705+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:40.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:40 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:40.705+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.298508+0000 front 2024-04-25T02:21:11.298605+0000 (oldest deadline 2024-04-25T02:21:36.598586+0000) 2024-04-25T02:21:40.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:40 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:40.750+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:40.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:40 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:40.750+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:41.092 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:40 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:40.990+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:41.093 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:40 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:40.936+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:41.093 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:40 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:40.936+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:41.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T02:21:41.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:21:41.163 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T02:21:41.163 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T02:21:41.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.5 2024-04-25T02:21:41.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:40 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:40.759+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:41.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:40 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:40.759+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:41.442 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:41 smithi134 systemd[1]: Stopping Ceph osd.5 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:21:41.731 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:41 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:41.345+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:41.732 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:41 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:41.345+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:41.793 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:41 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:41.614+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:41.793 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:41 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:41.614+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:41.793 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:41 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:41.479+0000 7fe311275700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:21:41.793 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:41 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:41.479+0000 7fe311275700 -1 osd.5 129 *** Got signal Terminated *** 2024-04-25T02:21:41.793 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:41 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:41.479+0000 7fe311275700 -1 osd.5 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:21:41.794 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:41 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:41.661+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:41.794 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:41 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:41.661+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.298508+0000 front 2024-04-25T02:21:11.298605+0000 (oldest deadline 2024-04-25T02:21:36.598586+0000) 2024-04-25T02:21:41.990 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:41 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:41.727+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:41.991 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:41 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:41.727+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:42.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:41 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:41.789+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:42.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:41 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:41.789+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:42.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:41 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:41.942+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:42.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:41 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:41.969+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:42.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:41 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:41.969+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:42.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:42 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:42.373+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:42.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:42 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:42.373+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:42.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:42 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:42.814+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:42.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:42 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:42.814+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:42.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:42 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:42.689+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:42.943 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:42 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:42.689+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.298508+0000 front 2024-04-25T02:21:11.298605+0000 (oldest deadline 2024-04-25T02:21:36.598586+0000) 2024-04-25T02:21:42.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:42 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:42.591+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:42.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:42 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:42.591+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:43.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:42 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:42.737+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:43.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:42 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:42.737+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:43.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:42 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:42.948+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:43.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:42 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:42.989+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:43.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:42 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:42.989+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:43.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:42 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:42.989+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:43.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:43 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:43.351+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:43.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:43 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:43.351+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:43.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:43 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:43.652+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:43.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:43 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:43.652+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.298508+0000 front 2024-04-25T02:21:11.298605+0000 (oldest deadline 2024-04-25T02:21:36.598586+0000) 2024-04-25T02:21:43.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:43 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:43.767+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:43.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:43 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:43.767+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:43.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:43 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:43.603+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:43.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:43 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:43.603+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:44.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:43 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:43.751+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:44.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:43 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:43.751+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:44.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:43 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:43.751+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:17.760687+0000 front 2024-04-25T02:21:17.760768+0000 (oldest deadline 2024-04-25T02:21:43.060871+0000) 2024-04-25T02:21:44.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:43 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:43.967+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:44.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:44 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:44.012+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:44.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:44 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:44.012+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:44.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:44 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:44.012+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:44.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:44 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:44.383+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:44.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:44 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:44.383+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:44.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:44 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:44.810+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:44.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:44 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:44.810+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:44.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:44 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:44.632+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:44.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:44 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:44.632+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:44.943 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:44 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:44.645+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:44.943 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:44 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:44.645+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.298508+0000 front 2024-04-25T02:21:11.298605+0000 (oldest deadline 2024-04-25T02:21:36.598586+0000) 2024-04-25T02:21:44.943 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:44 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:44.645+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:20.500979+0000 front 2024-04-25T02:21:20.500996+0000 (oldest deadline 2024-04-25T02:21:44.601126+0000) 2024-04-25T02:21:45.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:44 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:44.797+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:45.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:44 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:44.797+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:45.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:44 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:44.797+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:17.760687+0000 front 2024-04-25T02:21:17.760768+0000 (oldest deadline 2024-04-25T02:21:43.060871+0000) 2024-04-25T02:21:45.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:45 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:45.005+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:45.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:45 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:45.037+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:45.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:45 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:45.037+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:45.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:45 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:45.037+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:45.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:45 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:45.414+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:45.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:45 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:45.414+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:45.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:45 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:45.669+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:45.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:45 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:45.669+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:45.943 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:45 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:45.644+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:00.295730+0000 front 2024-04-25T02:21:00.295744+0000 (oldest deadline 2024-04-25T02:21:26.195841+0000) 2024-04-25T02:21:45.943 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:45 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:45.644+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.298508+0000 front 2024-04-25T02:21:11.298605+0000 (oldest deadline 2024-04-25T02:21:36.598586+0000) 2024-04-25T02:21:45.943 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:45 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5[44744]: 2024-04-25T02:21:45.644+0000 7fe309861700 -1 osd.5 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:20.500979+0000 front 2024-04-25T02:21:20.500996+0000 (oldest deadline 2024-04-25T02:21:44.601126+0000) 2024-04-25T02:21:45.944 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:45 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:45.805+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:45.944 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:45 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:45.805+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:45.944 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:45 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:45.805+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:46.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:45 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:45.837+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:46.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:45 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:45.837+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:46.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:45 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:45.837+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:17.760687+0000 front 2024-04-25T02:21:17.760768+0000 (oldest deadline 2024-04-25T02:21:43.060871+0000) 2024-04-25T02:21:46.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:46 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:46.019+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:46.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:46 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:46.060+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:46.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:46 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:46.060+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:46.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:46 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:46.060+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:46.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:46 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:46.452+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:46.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:46 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:46.452+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:46.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:46 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:46.803+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:46.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:46 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:46.803+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:46.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:46 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:46.803+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:46.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:46 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:46.654+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:46.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:46 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:46.654+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:46.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:46 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:46.654+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:47.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:46 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:46.805+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:47.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:46 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:46.805+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:47.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:46 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:46.805+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:17.760687+0000 front 2024-04-25T02:21:17.760768+0000 (oldest deadline 2024-04-25T02:21:43.060871+0000) 2024-04-25T02:21:47.357 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:47 smithi134 podman[73530]: 2024-04-25 02:21:47.070742251 +0000 UTC m=+5.714021942 container died c11954d35b557391c0d636900828af954fb676e28ded4724d693daecd26c8c32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-04-25T02:21:47.357 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:47 smithi134 podman[73530]: 2024-04-25 02:21:47.354347028 +0000 UTC m=+5.997626708 container cleanup c11954d35b557391c0d636900828af954fb676e28ded4724d693daecd26c8c32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T02:21:47.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:47 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:47.032+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:47.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:47 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:47.051+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:47.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:47 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:47.051+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:47.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:47 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:47.051+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:47.641 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:47 smithi134 bash[73530]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5 2024-04-25T02:21:47.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:47 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:47.410+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:47.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:47 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:47.410+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:47.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:47 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:47.410+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:47.942 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:47 smithi134 podman[73548]: 2024-04-25 02:21:47.638777589 +0000 UTC m=+0.566872393 container remove c11954d35b557391c0d636900828af954fb676e28ded4724d693daecd26c8c32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:21:47.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:47 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:47.798+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:47.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:47 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:47.798+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:47.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:47 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:47.798+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:47.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:47 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:47.651+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:47.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:47 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:47.651+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:47.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:47 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:47.651+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:48.221 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:47 smithi134 podman[73637]: 2024-04-25 02:21:47.8664078 +0000 UTC m=+0.017885764 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:21:48.221 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:47 smithi134 podman[73637]: 2024-04-25 02:21:47.981779703 +0000 UTC m=+0.133257639 container create 7cfb2ee14dae500be50757481b7e61ec45684986d234ff386181410b4c5f1067 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T02:21:48.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:47 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:47.820+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:48.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:47 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:47.820+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:48.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:47 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:47.820+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:17.760687+0000 front 2024-04-25T02:21:17.760768+0000 (oldest deadline 2024-04-25T02:21:43.060871+0000) 2024-04-25T02:21:48.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:48 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:48.081+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:48.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:48 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:48.081+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:48.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:48 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:48.016+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:48.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:48 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:48.016+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:48.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:48 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:48.016+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:48.536 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:48 smithi134 podman[73637]: 2024-04-25 02:21:48.218212324 +0000 UTC m=+0.369690264 container init 7cfb2ee14dae500be50757481b7e61ec45684986d234ff386181410b4c5f1067 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T02:21:48.536 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:48 smithi134 podman[73637]: 2024-04-25 02:21:48.222186996 +0000 UTC m=+0.373664934 container start 7cfb2ee14dae500be50757481b7e61ec45684986d234ff386181410b4c5f1067 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:21:48.536 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:48 smithi134 podman[73637]: 2024-04-25 02:21:48.309324307 +0000 UTC m=+0.460802245 container attach 7cfb2ee14dae500be50757481b7e61ec45684986d234ff386181410b4c5f1067 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T02:21:48.536 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 25 02:21:48 smithi134 podman[73637]: 2024-04-25 02:21:48.53288592 +0000 UTC m=+0.684363867 container died 7cfb2ee14dae500be50757481b7e61ec45684986d234ff386181410b4c5f1067 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS) 2024-04-25T02:21:48.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:48 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:48.385+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:48.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:48 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:48.385+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:48.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:48 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:48.385+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:48.817 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:48 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:48.668+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:48.817 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:48 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:48.668+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:48.817 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:48 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:48.668+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:49.132 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:48 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:48.813+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:49.132 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:48 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:48.813+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:49.132 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:48 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:48.813+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:49.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T02:21:49.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:21:49.192 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T02:21:49.193 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T02:21:49.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.6 2024-04-25T02:21:49.221 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:48 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:48.794+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:49.221 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:48 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:48.794+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:49.221 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:48 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:48.794+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:17.760687+0000 front 2024-04-25T02:21:17.760768+0000 (oldest deadline 2024-04-25T02:21:43.060871+0000) 2024-04-25T02:21:49.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:49 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:49.073+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:49.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:49 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:49.073+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:49.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:49 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:49.017+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:49.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:49 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:49.017+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:49.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:49 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:49.017+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:49.491 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:49 smithi031 systemd[1]: Stopping Ceph osd.6 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:21:49.491 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:49.354+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:49.491 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:49.355+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:49.491 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:49.355+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:49.817 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:49.530+0000 7f25692a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:21:49.817 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:49.530+0000 7f25692a1700 -1 osd.6 129 *** Got signal Terminated *** 2024-04-25T02:21:49.818 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:49.530+0000 7f25692a1700 -1 osd.6 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:21:49.818 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:49.813+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:50.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:49 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:49.812+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:50.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:49 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:49.812+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:50.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:49 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:49.812+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:50.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:49 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:49.701+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:50.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:49 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:49.701+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:50.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:49 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:49.701+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:50.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:49.813+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:50.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:49 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:49.813+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:17.760687+0000 front 2024-04-25T02:21:17.760768+0000 (oldest deadline 2024-04-25T02:21:43.060871+0000) 2024-04-25T02:21:50.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:50 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:50.031+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:50.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:50 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:50.031+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:50.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:50 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:50.014+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:50.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:50 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:50.014+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:50.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:50 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:50.014+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:50.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:50 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:50.382+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:50.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:50 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:50.382+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:50.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:50 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:50.382+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:51.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:50 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:50.770+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:51.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:50 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:50.770+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:51.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:50 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:50.770+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:51.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:50 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:50.725+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:51.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:50 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:50.725+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:51.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:50 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:50.725+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:51.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:50 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:50.791+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:51.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:50 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:50.791+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:51.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:50 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:50.791+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:17.760687+0000 front 2024-04-25T02:21:17.760768+0000 (oldest deadline 2024-04-25T02:21:43.060871+0000) 2024-04-25T02:21:51.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:50 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:50.984+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:51.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:50 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:50.984+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:51.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:50 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:50.984+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:51.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:50 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:50.983+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:51.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:50 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:50.983+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:51.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:51 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:51.372+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:51.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:51 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:51.372+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:51.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:51 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:51.372+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:52.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:51 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:51.760+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:52.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:51 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:51.760+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:52.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:51 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:51.760+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:52.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:51 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:51.691+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:52.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:51 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:51.691+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:52.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:51 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:51.691+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:52.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:51 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:51.770+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:52.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:51 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:51.770+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:52.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:51 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:51.770+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:17.760687+0000 front 2024-04-25T02:21:17.760768+0000 (oldest deadline 2024-04-25T02:21:43.060871+0000) 2024-04-25T02:21:52.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:51 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:51.950+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:52.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:51 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:51.950+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:52.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:51 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:51.969+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:52.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:51 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:51.969+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:52.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:51 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:51.969+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:52.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:52.407+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:52.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:52.407+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:52.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:52.407+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:52.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:52.407+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:21:53.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:52 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:52.734+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:53.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:52 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:52.734+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:53.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:52 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:52.734+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:53.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:52 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:52.723+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:53.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:52 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:52.723+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:53.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:52 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:52.723+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:53.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:52 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:52.723+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:21:53.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:52.755+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:53.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:52.755+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:53.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:52.755+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:17.760687+0000 front 2024-04-25T02:21:17.760768+0000 (oldest deadline 2024-04-25T02:21:43.060871+0000) 2024-04-25T02:21:53.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:52 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:52.956+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:53.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:52 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:52.956+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:53.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:52 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:52.990+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:53.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:52 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:52.990+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:53.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:52 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:52.990+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:53.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:52 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:52.990+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:21:53.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:53 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:53.370+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:53.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:53 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:53.370+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:53.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:53 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:53.370+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:53.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:53 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:53.370+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:21:54.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:53 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:53.734+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:54.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:53 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:53.734+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:54.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:53 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:53.734+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:54.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:53 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:53.758+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:54.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:53 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:53.758+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:54.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:53 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:53.758+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:54.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:53 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:53.758+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:21:54.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:53 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:53.779+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.958465+0000 front 2024-04-25T02:21:04.958139+0000 (oldest deadline 2024-04-25T02:21:29.058330+0000) 2024-04-25T02:21:54.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:53 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:53.779+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.359838+0000 front 2024-04-25T02:21:11.359645+0000 (oldest deadline 2024-04-25T02:21:34.859681+0000) 2024-04-25T02:21:54.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:53 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:53.779+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:17.760687+0000 front 2024-04-25T02:21:17.760768+0000 (oldest deadline 2024-04-25T02:21:43.060871+0000) 2024-04-25T02:21:54.241 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:53 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6[56596]: 2024-04-25T02:21:53.779+0000 7f256188d700 -1 osd.6 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.262147+0000 front 2024-04-25T02:21:28.262232+0000 (oldest deadline 2024-04-25T02:21:52.962234+0000) 2024-04-25T02:21:54.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:53 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:53.954+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:54.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:53 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:53.954+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:54.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:53 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:53.994+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:54.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:53 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:53.994+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:54.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:53 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:53.994+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:54.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:53 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:53.994+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:21:54.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:54 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:54.407+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:54.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:54 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:54.407+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:54.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:54 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:54.407+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:54.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:54 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:54.407+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:21:55.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:54 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:54.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:55.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:54 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:54.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:55.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:54 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:54.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:55.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:54 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:54.758+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:55.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:54 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:54.758+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:55.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:54 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:54.758+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:55.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:54 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:54.758+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:21:55.240 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:54 smithi031 podman[104407]: 2024-04-25 02:21:54.986052748 +0000 UTC m=+5.591148607 container died 88d17f63c892c34e5f875dbc6032e931e3944909fd368d6255eb3f462ea8b226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:21:55.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:54 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:54.989+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:55.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:54 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:54.989+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:55.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:54 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:54.989+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.940846+0000 front 2024-04-25T02:21:29.941066+0000 (oldest deadline 2024-04-25T02:21:54.640942+0000) 2024-04-25T02:21:55.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:55 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:55.014+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:55.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:55 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:55.014+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:55.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:55 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:55.014+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:55.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:55 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:55.014+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:21:55.735 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:55 smithi031 podman[104407]: 2024-04-25 02:21:55.489350104 +0000 UTC m=+6.094446022 container cleanup 88d17f63c892c34e5f875dbc6032e931e3944909fd368d6255eb3f462ea8b226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:21:55.736 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:55 smithi031 bash[104407]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6 2024-04-25T02:21:55.736 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:55 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:55.450+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:55.736 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:55 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:55.450+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:55.736 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:55 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:55.450+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:55.736 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:55 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:55.450+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:21:55.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:55 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:55.655+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:55.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:55 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:55.655+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:55.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:55 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:55.655+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:55.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:55 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:55.786+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:55.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:55 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:55.786+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:55.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:55 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:55.786+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:55.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:55 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:55.786+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:21:55.990 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:55 smithi031 podman[104425]: 2024-04-25 02:21:55.732513444 +0000 UTC m=+0.744580852 container remove 88d17f63c892c34e5f875dbc6032e931e3944909fd368d6255eb3f462ea8b226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-25T02:21:56.412 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:55 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:55.974+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:56.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:55 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:55.974+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:56.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:55 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:55.974+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.940846+0000 front 2024-04-25T02:21:29.941066+0000 (oldest deadline 2024-04-25T02:21:54.640942+0000) 2024-04-25T02:21:56.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:55 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:55.981+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:56.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:55 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:55.981+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:56.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:55 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:55.981+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:56.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:55 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:55.981+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:21:56.439 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:56 smithi031 podman[104520]: 2024-04-25 02:21:56.183357076 +0000 UTC m=+0.112166877 container create 5d7c1a924eca59f9bcf4eafb51f67b335898e46514a30f3109676e85f32bcd25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-25T02:21:56.439 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:56 smithi031 podman[104520]: 2024-04-25 02:21:56.08956161 +0000 UTC m=+0.018371434 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:21:56.439 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:56 smithi031 podman[104520]: 2024-04-25 02:21:56.436118036 +0000 UTC m=+0.364927846 container init 5d7c1a924eca59f9bcf4eafb51f67b335898e46514a30f3109676e85f32bcd25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:21:56.740 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:56 smithi031 podman[104520]: 2024-04-25 02:21:56.439456751 +0000 UTC m=+0.368266554 container start 5d7c1a924eca59f9bcf4eafb51f67b335898e46514a30f3109676e85f32bcd25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, ceph=True) 2024-04-25T02:21:56.741 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:56 smithi031 podman[104520]: 2024-04-25 02:21:56.536314645 +0000 UTC m=+0.465124455 container attach 5d7c1a924eca59f9bcf4eafb51f67b335898e46514a30f3109676e85f32bcd25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, ceph=True) 2024-04-25T02:21:56.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:56 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:56.459+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:56.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:56 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:56.459+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:56.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:56 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:56.459+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:56.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:56 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:56.459+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:21:57.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:56 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:56.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:57.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:56 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:56.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:57.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:56 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:56.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:57.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:56 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:56.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:21:57.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:56 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:56.769+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:57.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:56 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:56.769+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:57.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:56 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:56.769+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:57.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:56 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:56.769+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:21:57.209 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 02:21:56 smithi031 podman[104520]: 2024-04-25 02:21:56.738671002 +0000 UTC m=+0.667480812 container died 5d7c1a924eca59f9bcf4eafb51f67b335898e46514a30f3109676e85f32bcd25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:21:57.314 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:56 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:56.992+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:57.314 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:56 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:56.992+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:57.314 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:56 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:56.992+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.940846+0000 front 2024-04-25T02:21:29.941066+0000 (oldest deadline 2024-04-25T02:21:54.640942+0000) 2024-04-25T02:21:57.314 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:56 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:56.934+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:57.314 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:56 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:56.934+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:57.314 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:56 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:56.934+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:57.315 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:56 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:56.934+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:21:57.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T02:21:57.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:21:57.603 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T02:21:57.604 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T02:21:57.604 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.7 2024-04-25T02:21:57.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:57 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:57.474+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:57.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:57 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:57.474+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:57.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:57 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:57.474+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:57.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:57 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:57.474+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:21:57.909 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:57 smithi040 systemd[1]: Stopping Ceph osd.7 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:21:57.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:57 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:57.663+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:57.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:57 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:57.663+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:57.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:57 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:57.663+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:57.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:57 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:57.663+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:21:57.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:57 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:57.765+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:57.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:57 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:57.765+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:57.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:57 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:57.765+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:57.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:57 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:57.765+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:21:58.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:57 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:57.909+0000 7f2362739700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:21:58.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:57 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:57.909+0000 7f2362739700 -1 osd.7 129 *** Got signal Terminated *** 2024-04-25T02:21:58.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:57 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:57.909+0000 7f2362739700 -1 osd.7 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:21:58.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:57 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:57.969+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:58.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:57 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:57.969+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:58.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:57 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:57.969+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.940846+0000 front 2024-04-25T02:21:29.941066+0000 (oldest deadline 2024-04-25T02:21:54.640942+0000) 2024-04-25T02:21:58.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:57 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:57.958+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:58.164 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:57 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:57.958+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:58.164 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:57 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:57.958+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:58.164 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:57 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:57.958+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:21:58.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:58 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:58.480+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:58.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:58 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:58.480+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:58.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:58 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:58.480+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:58.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:58 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:58.480+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:21:58.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:58 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:58.613+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:58.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:58 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:58.613+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:58.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:58 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:58.613+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:58.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:58 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:58.613+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:21:58.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:58 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:58.757+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:58.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:58 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:58.757+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:58.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:58 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:58.757+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:58.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:58 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:58.757+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:21:59.162 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:58 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:58.935+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:21:59.162 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:58 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:58.935+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:21:59.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:58 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:58.935+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.940846+0000 front 2024-04-25T02:21:29.941066+0000 (oldest deadline 2024-04-25T02:21:54.640942+0000) 2024-04-25T02:21:59.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:59 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:59.002+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:21:59.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:59 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:59.002+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:21:59.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:59 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:59.002+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:21:59.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:21:59 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:21:59.002+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:21:59.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:59 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:59.475+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:21:59.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:59 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:59.475+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:21:59.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:59 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:59.475+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:21:59.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:21:59 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:21:59.475+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:21:59.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:59 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:59.663+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:21:59.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:59 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:59.663+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:21:59.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:59 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:59.663+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:21:59.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:21:59 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:21:59.663+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:21:59.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:59 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:59.800+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:21:59.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:59 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:59.800+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:21:59.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:59 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:59.800+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:21:59.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:21:59 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:21:59.800+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:00.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:00 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:00.023+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:00.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:00 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:00.023+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:00.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:00 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:00.023+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:00.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:00 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:00.023+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:00.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:59 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:59.931+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:22:00.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:59 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:59.931+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:22:00.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:59 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:59.931+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.940846+0000 front 2024-04-25T02:21:29.941066+0000 (oldest deadline 2024-04-25T02:21:54.640942+0000) 2024-04-25T02:22:00.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:21:59 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:21:59.931+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.442434+0000 front 2024-04-25T02:21:37.442520+0000 (oldest deadline 2024-04-25T02:21:59.142651+0000) 2024-04-25T02:22:00.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:00 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:00.681+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:22:00.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:00 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:00.681+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:22:00.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:00 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:00.681+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:22:00.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:00 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:00.681+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:22:00.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:00 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:00.847+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:00.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:00 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:00.847+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:00.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:00 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:00.847+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:00.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:00 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:00.847+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:00.990 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:00.501+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:00.990 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:00.501+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:00.990 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:00.501+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:00.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:00 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:00.501+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:01.162 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:01 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:01.053+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:01.162 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:01 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:01.053+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:01.162 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:01 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:01.053+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:01.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:01 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:01.053+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:01.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:01 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:01.053+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:01.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:00 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:22:00.893+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:22:01.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:00 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:22:00.893+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:22:01.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:00 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:22:00.893+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.940846+0000 front 2024-04-25T02:21:29.941066+0000 (oldest deadline 2024-04-25T02:21:54.640942+0000) 2024-04-25T02:22:01.164 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:00 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:22:00.893+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.442434+0000 front 2024-04-25T02:21:37.442520+0000 (oldest deadline 2024-04-25T02:21:59.142651+0000) 2024-04-25T02:22:01.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:01 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:01.666+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:22:01.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:01 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:01.666+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:22:01.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:01 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:01.666+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:22:01.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:01 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:01.666+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:22:01.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:01 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:01.666+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.791877+0000 front 2024-04-25T02:21:36.791759+0000 (oldest deadline 2024-04-25T02:22:01.491579+0000) 2024-04-25T02:22:01.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:01 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:01.851+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:01.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:01 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:01.851+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:01.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:01 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:01.851+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:01.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:01 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:01.851+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:01.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:01 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:01.851+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:01.990 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:01.486+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:01.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:01.486+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:01.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:01.486+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:01.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:01 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:01.486+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:02.162 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:02 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:02.096+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:02.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:02 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:02.096+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:02.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:02 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:02.096+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:02.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:02 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:02.096+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:02.163 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:02 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:02.096+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:02.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:01 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:22:01.895+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:10.836337+0000 front 2024-04-25T02:21:10.836306+0000 (oldest deadline 2024-04-25T02:21:34.237100+0000) 2024-04-25T02:22:02.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:01 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:22:01.895+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.139550+0000 front 2024-04-25T02:21:21.139504+0000 (oldest deadline 2024-04-25T02:21:47.039643+0000) 2024-04-25T02:22:02.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:01 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:22:01.895+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.940846+0000 front 2024-04-25T02:21:29.941066+0000 (oldest deadline 2024-04-25T02:21:54.640942+0000) 2024-04-25T02:22:02.163 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:01 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7[50549]: 2024-04-25T02:22:01.895+0000 7f235ad25700 -1 osd.7 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.442434+0000 front 2024-04-25T02:21:37.442520+0000 (oldest deadline 2024-04-25T02:21:59.142651+0000) 2024-04-25T02:22:02.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:02.644+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:22:02.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:02.644+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:22:02.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:02.644+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:22:02.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:02.644+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:22:02.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:02.644+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.791877+0000 front 2024-04-25T02:21:36.791759+0000 (oldest deadline 2024-04-25T02:22:01.491579+0000) 2024-04-25T02:22:02.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:02.833+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:02.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:02.833+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:02.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:02.833+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:02.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:02.833+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:02.943 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:02 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:02.833+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:02.990 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:02 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:02.533+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:02.990 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:02 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:02.533+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:02.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:02 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:02.533+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:02.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:02 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:02.533+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:03.413 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:03 smithi040 podman[73709]: 2024-04-25 02:22:03.342238004 +0000 UTC m=+5.550754803 container died 1dbabda55a08a8bd43a88a6c80077e34a62864c5a8293bde056e44a0297b1270 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T02:22:03.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:03 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:03.115+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:03.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:03 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:03.115+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:03.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:03 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:03.115+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:03.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:03 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:03.115+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:03.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:03 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:03.115+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:03.912 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:03 smithi040 podman[73709]: 2024-04-25 02:22:03.728114871 +0000 UTC m=+5.936631669 container cleanup 1dbabda55a08a8bd43a88a6c80077e34a62864c5a8293bde056e44a0297b1270 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T02:22:03.912 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:03 smithi040 bash[73709]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7 2024-04-25T02:22:03.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:03 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:03.875+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:03.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:03 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:03.875+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:03.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:03 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:03.875+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:03.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:03 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:03.875+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:03.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:03 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:03.875+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:03.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:03 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:03.683+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:22:03.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:03 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:03.683+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:22:03.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:03 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:03.683+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:22:03.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:03 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:03.683+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:22:03.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:03 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:03.683+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.791877+0000 front 2024-04-25T02:21:36.791759+0000 (oldest deadline 2024-04-25T02:22:01.491579+0000) 2024-04-25T02:22:03.990 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:03 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:03.517+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:03.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:03 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:03.517+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:03.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:03 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:03.517+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:03.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:03 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:03.517+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:04.294 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:03 smithi040 podman[73737]: 2024-04-25 02:22:03.988874987 +0000 UTC m=+0.639266903 container remove 1dbabda55a08a8bd43a88a6c80077e34a62864c5a8293bde056e44a0297b1270 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-d2c45d9) 2024-04-25T02:22:04.295 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:04 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:04.130+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:04.295 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:04 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:04.130+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:04.295 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:04 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:04.130+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:04.295 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:04 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:04.130+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:04.295 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:04 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:04.130+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:04.563 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:04 smithi040 podman[73829]: 2024-04-25 02:22:04.194383435 +0000 UTC m=+0.018108641 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:22:04.563 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:04 smithi040 podman[73829]: 2024-04-25 02:22:04.306292811 +0000 UTC m=+0.130018001 container create 16a6e008c4de4643b85ae5399a9a873567f8b4e085d04f1ea52b5bf2a87e83d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T02:22:04.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:04 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:04.470+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:04.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:04 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:04.470+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:04.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:04 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:04.470+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:04.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:04 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:04.470+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:04.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:04 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:04.470+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:04.867 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:04 smithi040 podman[73829]: 2024-04-25 02:22:04.560231612 +0000 UTC m=+0.383956819 container init 16a6e008c4de4643b85ae5399a9a873567f8b4e085d04f1ea52b5bf2a87e83d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:22:04.868 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:04 smithi040 podman[73829]: 2024-04-25 02:22:04.564124127 +0000 UTC m=+0.387849321 container start 16a6e008c4de4643b85ae5399a9a873567f8b4e085d04f1ea52b5bf2a87e83d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9) 2024-04-25T02:22:04.868 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:04 smithi040 podman[73829]: 2024-04-25 02:22:04.668286069 +0000 UTC m=+0.492011262 container attach 16a6e008c4de4643b85ae5399a9a873567f8b4e085d04f1ea52b5bf2a87e83d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T02:22:04.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:04 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:04.904+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:04.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:04 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:04.904+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:04.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:04 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:04.904+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:04.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:04 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:04.904+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:04.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:04 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:04.904+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:04.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:04 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:04.678+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:22:04.942 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:04 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:04.678+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:22:04.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:04 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:04.678+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:22:04.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:04 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:04.678+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:22:04.943 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:04 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:04.678+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.791877+0000 front 2024-04-25T02:21:36.791759+0000 (oldest deadline 2024-04-25T02:22:01.491579+0000) 2024-04-25T02:22:05.162 INFO:journalctl@ceph.osd.7.smithi040.stdout:Apr 25 02:22:04 smithi040 podman[73829]: 2024-04-25 02:22:04.865123642 +0000 UTC m=+0.688848836 container died 16a6e008c4de4643b85ae5399a9a873567f8b4e085d04f1ea52b5bf2a87e83d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T02:22:05.642 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:05 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:05.171+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:05.642 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:05 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:05.171+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:05.642 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:05 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:05.171+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:05.642 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:05 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:05.171+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:05.642 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:05 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:05.171+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:05.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T02:22:05.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:22:05.708 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T02:22:05.708 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T02:22:05.709 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.8 2024-04-25T02:22:05.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:05 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:05.499+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:05.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:05 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:05.499+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:05.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:05 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:05.499+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:05.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:05 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:05.499+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:05.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:05 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:05.499+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:06.010 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:05 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:05.879+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:06.010 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:05 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:05.879+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:05 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:05.879+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:05 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:05.879+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:05 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:05.879+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:05 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:05.693+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:05 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:05.693+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:05 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:05.693+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:05 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:05.693+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:05 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:05.693+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.791877+0000 front 2024-04-25T02:21:36.791759+0000 (oldest deadline 2024-04-25T02:22:01.491579+0000) 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:05 smithi134 systemd[1]: Stopping Ceph osd.8 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:06.006+0000 7f7772e5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:06.006+0000 7f7772e5e700 -1 osd.8 129 *** Got signal Terminated *** 2024-04-25T02:22:06.011 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:06.006+0000 7f7772e5e700 -1 osd.8 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:22:06.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:06 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:06.134+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:06.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:06 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:06.134+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:06.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:06 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:06.134+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:06.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:06 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:06.134+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:06.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:06 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:06.134+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:06.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:06 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:06.467+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:06.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:06 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:06.467+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:06.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:06 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:06.467+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:06.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:06 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:06.467+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:06.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:06 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:06.467+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:07.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:06.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:22:07.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:06.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:22:07.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:06.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:22:07.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:06.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:22:07.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:06.697+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.791877+0000 front 2024-04-25T02:21:36.791759+0000 (oldest deadline 2024-04-25T02:22:01.491579+0000) 2024-04-25T02:22:07.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:06.854+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:07.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:06.854+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:07.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:06.854+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:07.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:06.854+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:07.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:06 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:06.854+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:07.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:07 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:07.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:07.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:07 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:07.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:07.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:07 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:07.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:07.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:07 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:07.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:07.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:07 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:07.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:07.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:07 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:07.419+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:07.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:07 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:07.419+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:07.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:07 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:07.419+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:07.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:07 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:07.419+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:07.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:07 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:07.419+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:08.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:07 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:07.733+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:22:08.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:07 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:07.733+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:22:08.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:07 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:07.733+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:22:08.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:07 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:07.733+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:22:08.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:07 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:07.733+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.791877+0000 front 2024-04-25T02:21:36.791759+0000 (oldest deadline 2024-04-25T02:22:01.491579+0000) 2024-04-25T02:22:08.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:07 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:07.855+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:08.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:07 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:07.855+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:08.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:07 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:07.855+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:08.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:07 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:07.855+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:08.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:07 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:07.855+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:08.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:08 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:08.205+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:08.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:08 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:08.205+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:08.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:08 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:08.205+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:08.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:08 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:08.205+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:08.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:08 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:08.205+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:08.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:08 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:08.440+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:08.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:08 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:08.440+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:08.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:08 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:08.440+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:08.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:08 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:08.440+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:08.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:08 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:08.440+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:08.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:08 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:08.440+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.846239+0000 front 2024-04-25T02:21:43.846126+0000 (oldest deadline 2024-04-25T02:22:07.946136+0000) 2024-04-25T02:22:09.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:08 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:08.814+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:09.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:08 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:08.814+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:09.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:08 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:08.814+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:09.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:08 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:08.814+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:09.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:08 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:08.814+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:09.193 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:08 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:08.702+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:22:09.193 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:08 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:08.702+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:22:09.193 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:08 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:08.702+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:22:09.193 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:08 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:08.702+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:22:09.193 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:08 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:08.702+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.791877+0000 front 2024-04-25T02:21:36.791759+0000 (oldest deadline 2024-04-25T02:22:01.491579+0000) 2024-04-25T02:22:09.193 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:08 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:08.702+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:42.592457+0000 front 2024-04-25T02:21:42.592484+0000 (oldest deadline 2024-04-25T02:22:07.793040+0000) 2024-04-25T02:22:09.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:09 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:09.193+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:09.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:09 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:09.193+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:09.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:09 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:09.193+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:09.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:09 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:09.193+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:09.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:09 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:09.193+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:09.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:09 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:09.193+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:09.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:09 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:09.443+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:09.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:09 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:09.443+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:09.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:09 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:09.443+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:09.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:09 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:09.443+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:09.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:09 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:09.443+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:09.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:09 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:09.443+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.846239+0000 front 2024-04-25T02:21:43.846126+0000 (oldest deadline 2024-04-25T02:22:07.946136+0000) 2024-04-25T02:22:10.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:09.746+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:22:10.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:09.746+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:22:10.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:09.746+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:22:10.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:09.746+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:22:10.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:09.746+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.791877+0000 front 2024-04-25T02:21:36.791759+0000 (oldest deadline 2024-04-25T02:22:01.491579+0000) 2024-04-25T02:22:10.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:09.746+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:42.592457+0000 front 2024-04-25T02:21:42.592484+0000 (oldest deadline 2024-04-25T02:22:07.793040+0000) 2024-04-25T02:22:10.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:09.848+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:10.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:09.848+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:10.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:09.848+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:10.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:09.848+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:10.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:09.848+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:10.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:09 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:09.848+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:10.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:10 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:10.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:10.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:10 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:10.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:10.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:10 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:10.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:10.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:10 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:10.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:10.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:10 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:10.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:10.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:10 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:10.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:10.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:10 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:10.439+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:10.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:10 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:10.439+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:10.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:10 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:10.439+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:10.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:10 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:10.439+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:10.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:10 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:10.439+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:10.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:10 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:10.439+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.846239+0000 front 2024-04-25T02:21:43.846126+0000 (oldest deadline 2024-04-25T02:22:07.946136+0000) 2024-04-25T02:22:11.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:10.745+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:04.785536+0000 front 2024-04-25T02:21:04.785491+0000 (oldest deadline 2024-04-25T02:21:28.285667+0000) 2024-04-25T02:22:11.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:10.745+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.687709+0000 front 2024-04-25T02:21:11.687632+0000 (oldest deadline 2024-04-25T02:21:38.688252+0000) 2024-04-25T02:22:11.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:10.745+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.789352+0000 front 2024-04-25T02:21:19.789215+0000 (oldest deadline 2024-04-25T02:21:45.689376+0000) 2024-04-25T02:22:11.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:10.745+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.790487+0000 front 2024-04-25T02:21:29.790666+0000 (oldest deadline 2024-04-25T02:21:55.690412+0000) 2024-04-25T02:22:11.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:10.745+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.791877+0000 front 2024-04-25T02:21:36.791759+0000 (oldest deadline 2024-04-25T02:22:01.491579+0000) 2024-04-25T02:22:11.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8[51739]: 2024-04-25T02:22:10.745+0000 7f776b44a700 -1 osd.8 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:42.592457+0000 front 2024-04-25T02:21:42.592484+0000 (oldest deadline 2024-04-25T02:22:07.793040+0000) 2024-04-25T02:22:11.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:10.820+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:11.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:10.820+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:11.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:10.820+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:11.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:10.820+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:11.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:10.820+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:11.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:10 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:10.820+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:11.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:11 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:11.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:11.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:11 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:11.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:11.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:11 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:11.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:11.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:11 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:11.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:11.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:11 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:11.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:11.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:11 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:11.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:11.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:11 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:11.413+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:11.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:11 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:11.413+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:11.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:11 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:11.413+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:11.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:11 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:11.413+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:11.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:11 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:11.413+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:11.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:11 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:11.413+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.846239+0000 front 2024-04-25T02:21:43.846126+0000 (oldest deadline 2024-04-25T02:22:07.946136+0000) 2024-04-25T02:22:11.815 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:11 smithi134 podman[73743]: 2024-04-25 02:22:11.517908273 +0000 UTC m=+5.619642450 container died 4b5931644b549dfb90265f040fce53e35693087b77961136a38c455698d3cc6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T02:22:12.186 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:11 smithi134 podman[73743]: 2024-04-25 02:22:11.915761878 +0000 UTC m=+6.017496064 container cleanup 4b5931644b549dfb90265f040fce53e35693087b77961136a38c455698d3cc6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T02:22:12.186 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:11 smithi134 bash[73743]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8 2024-04-25T02:22:12.187 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:11 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:11.812+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:12.187 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:11 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:11.812+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:12.187 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:11 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:11.812+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:12.187 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:11 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:11.812+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:12.187 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:11 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:11.812+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:12.187 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:11 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:11.812+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:12.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:12 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:12.107+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:12.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:12 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:12.107+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:12.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:12 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:12.107+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:12.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:12 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:12.107+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:12.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:12 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:12.107+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:12.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:12 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:12.107+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:12.441 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:12 smithi134 podman[73760]: 2024-04-25 02:22:12.183704371 +0000 UTC m=+0.665159321 container remove 4b5931644b549dfb90265f040fce53e35693087b77961136a38c455698d3cc6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T02:22:12.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:12 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:12.463+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:12.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:12 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:12.463+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:12.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:12 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:12.463+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:12.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:12 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:12.463+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:12.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:12 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:12.463+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:12.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:12 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:12.463+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.846239+0000 front 2024-04-25T02:21:43.846126+0000 (oldest deadline 2024-04-25T02:22:07.946136+0000) 2024-04-25T02:22:12.823 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:12 smithi134 podman[73855]: 2024-04-25 02:22:12.484477201 +0000 UTC m=+0.102698507 container create 70993a3616e272524b9ee67166068f881df2cf7cf525427c41125123ef3b1336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS) 2024-04-25T02:22:12.823 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:12 smithi134 podman[73855]: 2024-04-25 02:22:12.399519744 +0000 UTC m=+0.017741068 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:22:12.823 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:12 smithi134 podman[73855]: 2024-04-25 02:22:12.72944243 +0000 UTC m=+0.347663739 container init 70993a3616e272524b9ee67166068f881df2cf7cf525427c41125123ef3b1336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:22:12.823 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:12 smithi134 podman[73855]: 2024-04-25 02:22:12.732924096 +0000 UTC m=+0.351145403 container start 70993a3616e272524b9ee67166068f881df2cf7cf525427c41125123ef3b1336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:22:12.823 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:12 smithi134 podman[73855]: 2024-04-25 02:22:12.82043804 +0000 UTC m=+0.438659347 container attach 70993a3616e272524b9ee67166068f881df2cf7cf525427c41125123ef3b1336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T02:22:13.192 INFO:journalctl@ceph.osd.8.smithi134.stdout:Apr 25 02:22:13 smithi134 podman[73855]: 2024-04-25 02:22:13.027203422 +0000 UTC m=+0.645424732 container died 70993a3616e272524b9ee67166068f881df2cf7cf525427c41125123ef3b1336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-25T02:22:13.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:12 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:12.854+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:13.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:12 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:12.854+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:13.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:12 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:12.854+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:13.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:12 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:12.854+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:13.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:12 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:12.854+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:13.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:12 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:12.854+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:13.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:13 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:13.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:13.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:13 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:13.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:13.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:13 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:13.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:13.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:13 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:13.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:13.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:13 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:13.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:13.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:13 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:13.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:13.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T02:22:13.701 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:22:13.702 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T02:22:13.702 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T02:22:13.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.9 2024-04-25T02:22:13.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:13 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:13.827+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:13.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:13 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:13.827+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:13.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:13 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:13.827+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:13.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:13 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:13.827+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:13.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:13 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:13.827+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:13.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:13 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:13.827+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:13.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:13 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:13.504+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:13.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:13 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:13.504+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:13.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:13 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:13.504+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:13.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:13 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:13.504+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:13.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:13 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:13.504+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:13.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:13 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:13.504+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.846239+0000 front 2024-04-25T02:21:43.846126+0000 (oldest deadline 2024-04-25T02:22:07.946136+0000) 2024-04-25T02:22:13.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:13 smithi031 systemd[1]: Stopping Ceph osd.9 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:22:14.490 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:14 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:14.015+0000 7fc41d964700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:22:14.491 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:14 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:14.015+0000 7fc41d964700 -1 osd.9 129 *** Got signal Terminated *** 2024-04-25T02:22:14.491 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:14 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:14.015+0000 7fc41d964700 -1 osd.9 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:22:14.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:14 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:14.173+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:14.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:14 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:14.173+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:14.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:14 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:14.173+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:14.664 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:14 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:14.173+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:14.664 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:14 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:14.173+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:14.664 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:14 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:14.173+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:14.990 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:14 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:14.538+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:14.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:14 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:14.538+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:14.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:14 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:14.538+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:14.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:14 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:14.538+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:14.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:14 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:14.538+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:14.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:14 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:14.538+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.846239+0000 front 2024-04-25T02:21:43.846126+0000 (oldest deadline 2024-04-25T02:22:07.946136+0000) 2024-04-25T02:22:15.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:14 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:14.818+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:15.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:14 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:14.818+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:15.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:14 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:14.818+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:15.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:14 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:14.818+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:15.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:14 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:14.818+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:15.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:14 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:14.818+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:15.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:15 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:15.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:15.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:15 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:15.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:15.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:15 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:15.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:15.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:15 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:15.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:15.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:15 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:15.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:15.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:15 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:15.154+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:15.990 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:15 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:15.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:15.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:15 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:15.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:15.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:15 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:15.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:15.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:15 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:15.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:15.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:15 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:15.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:15.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:15 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:15.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.846239+0000 front 2024-04-25T02:21:43.846126+0000 (oldest deadline 2024-04-25T02:22:07.946136+0000) 2024-04-25T02:22:16.094 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:15 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:15.786+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:16.094 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:15 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:15.786+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:16.094 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:15 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:15.786+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:16.094 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:15 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:15.786+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:16.094 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:15 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:15.786+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:16.094 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:15 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:15.786+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:16.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:16 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:16.116+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:16.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:16 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:16.116+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:16.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:16 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:16.116+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:16.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:16 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:16.116+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:16.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:16 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:16.116+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:16.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:16 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:16.116+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:16.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:16 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:16.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:16.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:16 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:16.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:16.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:16 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:16.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:16.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:16 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:16.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:16.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:16 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:16.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:16.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:16 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:16.553+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.846239+0000 front 2024-04-25T02:21:43.846126+0000 (oldest deadline 2024-04-25T02:22:07.946136+0000) 2024-04-25T02:22:17.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:16 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:16.750+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:17.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:16 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:16.750+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:17.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:16 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:16.750+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:17.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:16 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:16.750+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:17.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:16 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:16.750+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:17.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:16 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:16.750+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:17.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:17 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:17.079+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:17.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:17 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:17.079+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:17.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:17 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:17.079+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:17.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:17 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:17.079+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:17.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:17 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:17.079+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:17.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:17 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:17.079+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:17.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:17 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:17.571+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:17.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:17 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:17.571+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:17.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:17 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:17.571+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:17.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:17 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:17.571+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:17.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:17 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:17.571+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:17.992 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:17 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:17.571+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.846239+0000 front 2024-04-25T02:21:43.846126+0000 (oldest deadline 2024-04-25T02:22:07.946136+0000) 2024-04-25T02:22:18.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:17 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:17.771+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:18.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:17 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:17.772+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:18.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:17 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:17.772+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:18.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:17 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:17.772+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:18.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:17 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:17.772+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:18.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:17 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:17.772+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:18.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:18 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:18.075+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:18.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:18 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:18.075+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:18.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:18 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:18.075+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:18.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:18 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:18.075+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:18.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:18 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:18.075+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:18.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:18 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:18.075+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:18.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:18 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:18.075+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T02:21:52.284805+0000 front 2024-04-25T02:21:52.284716+0000 (oldest deadline 2024-04-25T02:22:17.584475+0000) 2024-04-25T02:22:18.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:18 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:18.617+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:02.339327+0000 front 2024-04-25T02:21:02.339460+0000 (oldest deadline 2024-04-25T02:21:29.940038+0000) 2024-04-25T02:22:18.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:18 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:18.617+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:12.241356+0000 front 2024-04-25T02:21:12.241199+0000 (oldest deadline 2024-04-25T02:21:36.341306+0000) 2024-04-25T02:22:18.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:18 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:18.617+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.642336+0000 front 2024-04-25T02:21:21.642244+0000 (oldest deadline 2024-04-25T02:21:46.942364+0000) 2024-04-25T02:22:18.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:18 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:18.617+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:29.843494+0000 front 2024-04-25T02:21:29.843476+0000 (oldest deadline 2024-04-25T02:21:52.143694+0000) 2024-04-25T02:22:18.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:18 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:18.617+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:37.945628+0000 front 2024-04-25T02:21:37.945563+0000 (oldest deadline 2024-04-25T02:22:03.845616+0000) 2024-04-25T02:22:18.991 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:18 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9[63602]: 2024-04-25T02:22:18.617+0000 7fc415f50700 -1 osd.9 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.846239+0000 front 2024-04-25T02:21:43.846126+0000 (oldest deadline 2024-04-25T02:22:07.946136+0000) 2024-04-25T02:22:19.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:18 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:18.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:19.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:18 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:18.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:19.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:18 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:18.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:19.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:18 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:18.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:19.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:18 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:18.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:19.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:18 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:18.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:19.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:19 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:19.117+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:19.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:19 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:19.117+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:19.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:19 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:19.117+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:19.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:19 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:19.117+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:19.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:19 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:19.117+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:19.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:19 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:19.117+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:19.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:19 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:19.117+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T02:21:52.284805+0000 front 2024-04-25T02:21:52.284716+0000 (oldest deadline 2024-04-25T02:22:17.584475+0000) 2024-04-25T02:22:19.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:19 smithi031 podman[104636]: 2024-04-25 02:22:19.397380479 +0000 UTC m=+5.494588074 container died 38ae8224f902b326faa1e39b2675126f8296a5dd3bd3cfa02c9d41378d09c1e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:22:20.146 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:19 smithi031 podman[104636]: 2024-04-25 02:22:19.90067768 +0000 UTC m=+5.997885275 container cleanup 38ae8224f902b326faa1e39b2675126f8296a5dd3bd3cfa02c9d41378d09c1e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T02:22:20.146 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:19 smithi031 bash[104636]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9 2024-04-25T02:22:20.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:19 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:19.763+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:20.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:19 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:19.763+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:20.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:19 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:19.763+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:20.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:19 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:19.763+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:20.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:19 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:19.763+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:20.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:19 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:19.763+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:20.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:20 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:20.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:20.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:20 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:20.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:20.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:20 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:20.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:20.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:20 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:20.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:20.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:20 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:20.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:20.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:20 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:20.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:20.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:20 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:20.131+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T02:21:52.284805+0000 front 2024-04-25T02:21:52.284716+0000 (oldest deadline 2024-04-25T02:22:17.584475+0000) 2024-04-25T02:22:20.469 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:20 smithi031 podman[104654]: 2024-04-25 02:22:20.143760723 +0000 UTC m=+0.743770181 container remove 38ae8224f902b326faa1e39b2675126f8296a5dd3bd3cfa02c9d41378d09c1e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T02:22:20.740 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:20 smithi031 podman[104747]: 2024-04-25 02:22:20.369139649 +0000 UTC m=+0.018656355 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:22:20.741 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:20 smithi031 podman[104747]: 2024-04-25 02:22:20.486362447 +0000 UTC m=+0.135879142 container create 1303b441dd08c3b367ef1a24be46df57084d3fe9942a9d873fcba2e61d97b56a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:22:21.159 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:20 smithi031 podman[104747]: 2024-04-25 02:22:20.847338424 +0000 UTC m=+0.496855121 container init 1303b441dd08c3b367ef1a24be46df57084d3fe9942a9d873fcba2e61d97b56a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:22:21.159 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:20 smithi031 podman[104747]: 2024-04-25 02:22:20.850732822 +0000 UTC m=+0.500249519 container start 1303b441dd08c3b367ef1a24be46df57084d3fe9942a9d873fcba2e61d97b56a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T02:22:21.159 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:20 smithi031 podman[104747]: 2024-04-25 02:22:20.947683309 +0000 UTC m=+0.597200014 container attach 1303b441dd08c3b367ef1a24be46df57084d3fe9942a9d873fcba2e61d97b56a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-d2c45d9) 2024-04-25T02:22:21.191 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:20 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:20.722+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:21.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:20 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:20.722+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:21.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:20 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:20.722+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:21.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:20 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:20.722+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:21.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:20 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:20.722+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:21.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:20 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:20.722+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:21.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:21.158+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:21.412 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:21.158+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:21.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:21.158+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:21.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:21.158+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:21.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:21.158+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:21.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:21.158+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:21.413 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:21 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:21.158+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T02:21:52.284805+0000 front 2024-04-25T02:21:52.284716+0000 (oldest deadline 2024-04-25T02:22:17.584475+0000) 2024-04-25T02:22:21.446 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 02:22:21 smithi031 podman[104747]: 2024-04-25 02:22:21.156629535 +0000 UTC m=+0.806146233 container died 1303b441dd08c3b367ef1a24be46df57084d3fe9942a9d873fcba2e61d97b56a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-25T02:22:21.930 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T02:22:21.969 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:22:21.969 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T02:22:21.970 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T02:22:21.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.10 2024-04-25T02:22:22.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:21.757+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:22.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:21.757+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:22.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:21.757+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:22.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:21.757+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:22.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:21.757+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:22.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:21.757+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:22.287 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:22 smithi040 systemd[1]: Stopping Ceph osd.10 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:22:22.287 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:22 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:22.150+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:22.287 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:22 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:22.150+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:22.287 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:22 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:22.150+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:22.287 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:22 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:22.150+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:22.287 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:22 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:22.150+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:22.288 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:22 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:22.150+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:22.288 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:22 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:22.150+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T02:21:52.284805+0000 front 2024-04-25T02:21:52.284716+0000 (oldest deadline 2024-04-25T02:22:17.584475+0000) 2024-04-25T02:22:22.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:22 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:22.282+0000 7fda95a65700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:22:22.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:22 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:22.282+0000 7fda95a65700 -1 osd.10 129 *** Got signal Terminated *** 2024-04-25T02:22:22.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:22 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:22.282+0000 7fda95a65700 -1 osd.10 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:22:23.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:22 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:22.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:23.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:22 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:22.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:23.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:22 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:22.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:23.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:22 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:22.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:23.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:22 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:22.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:23.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:22 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:22.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:23.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:23 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:23.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:23.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:23 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:23.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:23.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:23 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:23.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:23.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:23 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:23.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:23.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:23 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:23.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:23.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:23 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:23.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:23.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:23 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:23.159+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T02:21:52.284805+0000 front 2024-04-25T02:21:52.284716+0000 (oldest deadline 2024-04-25T02:22:17.584475+0000) 2024-04-25T02:22:24.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:23 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:23.719+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:24.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:23 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:23.719+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:24.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:23 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:23.719+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:24.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:23 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:23.719+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:24.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:23 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:23.719+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:24.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:23 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:23.719+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:24.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:24.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:24.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:24.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:24.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:24.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:24.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:24.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:24.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:24.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:24.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:24.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:24.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:24 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:24.168+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T02:21:52.284805+0000 front 2024-04-25T02:21:52.284716+0000 (oldest deadline 2024-04-25T02:22:17.584475+0000) 2024-04-25T02:22:25.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:24 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:24.765+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:25.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:24 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:24.765+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:25.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:24 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:24.765+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:25.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:24 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:24.765+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:25.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:24 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:24.765+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:25.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:24 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:24.765+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:25.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:25 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:25.187+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:25.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:25 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:25.187+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:25.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:25 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:25.187+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:25.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:25 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:25.187+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:25.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:25 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:25.187+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:25.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:25 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:25.187+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:25.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:25 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:25.187+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T02:21:52.284805+0000 front 2024-04-25T02:21:52.284716+0000 (oldest deadline 2024-04-25T02:22:17.584475+0000) 2024-04-25T02:22:26.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:25 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:25.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:26.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:25 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:25.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:26.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:25 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:25.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:26.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:25 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:25.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:26.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:25 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:25.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:26.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:25 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:25.726+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:26.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:26.198+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:26.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:26.198+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:26.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:26.198+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:26.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:26.198+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:26.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:26.198+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:26.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:26.198+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:26.663 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:26 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:26.198+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T02:21:52.284805+0000 front 2024-04-25T02:21:52.284716+0000 (oldest deadline 2024-04-25T02:22:17.584475+0000) 2024-04-25T02:22:26.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:26.680+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:26.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:26.680+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:26.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:26.680+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:26.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:26.680+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:26.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:26.680+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:26.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:26.680+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:26.942 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:26.680+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-25T02:22:00.891311+0000 front 2024-04-25T02:22:00.891392+0000 (oldest deadline 2024-04-25T02:22:26.191352+0000) 2024-04-25T02:22:27.566 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:27 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:27.213+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:01.972517+0000 front 2024-04-25T02:21:01.972727+0000 (oldest deadline 2024-04-25T02:21:27.272599+0000) 2024-04-25T02:22:27.566 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:27 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:27.213+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.973794+0000 front 2024-04-25T02:21:11.973846+0000 (oldest deadline 2024-04-25T02:21:36.074012+0000) 2024-04-25T02:22:27.566 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:27 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:27.213+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:21.077371+0000 front 2024-04-25T02:21:21.077343+0000 (oldest deadline 2024-04-25T02:21:42.777430+0000) 2024-04-25T02:22:27.566 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:27 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:27.213+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:28.579306+0000 front 2024-04-25T02:21:28.579374+0000 (oldest deadline 2024-04-25T02:21:52.679397+0000) 2024-04-25T02:22:27.567 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:27 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:27.213+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:32.679779+0000 front 2024-04-25T02:21:32.679806+0000 (oldest deadline 2024-04-25T02:22:00.280487+0000) 2024-04-25T02:22:27.567 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:27 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:27.213+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:43.082231+0000 front 2024-04-25T02:21:43.082189+0000 (oldest deadline 2024-04-25T02:22:08.382251+0000) 2024-04-25T02:22:27.567 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:27 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10[57177]: 2024-04-25T02:22:27.213+0000 7fda8e051700 -1 osd.10 129 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T02:21:52.284805+0000 front 2024-04-25T02:21:52.284716+0000 (oldest deadline 2024-04-25T02:22:17.584475+0000) 2024-04-25T02:22:27.871 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:27 smithi040 podman[73932]: 2024-04-25 02:22:27.720380442 +0000 UTC m=+5.545357099 container died baa6d104319909f83a0687d395e7ff8126ecf64e5571472fa4aec435d7ce5ade (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T02:22:28.162 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:28 smithi040 podman[73932]: 2024-04-25 02:22:28.075498597 +0000 UTC m=+5.900475253 container cleanup baa6d104319909f83a0687d395e7ff8126ecf64e5571472fa4aec435d7ce5ade (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T02:22:28.162 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:28 smithi040 bash[73932]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10 2024-04-25T02:22:28.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:27 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:27.693+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:28.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:27 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:27.693+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:28.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:27 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:27.693+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:28.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:27 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:27.693+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:28.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:27 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:27.693+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:28.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:27 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:27.693+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:28.193 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:27 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:27.693+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-25T02:22:00.891311+0000 front 2024-04-25T02:22:00.891392+0000 (oldest deadline 2024-04-25T02:22:26.191352+0000) 2024-04-25T02:22:28.662 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:28 smithi040 podman[73952]: 2024-04-25 02:22:28.343256811 +0000 UTC m=+0.623924740 container remove baa6d104319909f83a0687d395e7ff8126ecf64e5571472fa4aec435d7ce5ade (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T02:22:29.008 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:28 smithi040 podman[74053]: 2024-04-25 02:22:28.603315685 +0000 UTC m=+0.018784619 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:22:29.008 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:28 smithi040 podman[74053]: 2024-04-25 02:22:28.752789739 +0000 UTC m=+0.168258660 container create 5bdcc885ed70e4932001e0267761dffd1b5d190ebaee5129f8acb72a83ecf434 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-25T02:22:29.009 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:29 smithi040 podman[74053]: 2024-04-25 02:22:29.005739933 +0000 UTC m=+0.421208869 container init 5bdcc885ed70e4932001e0267761dffd1b5d190ebaee5129f8acb72a83ecf434 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:22:29.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:28 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:28.700+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:29.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:28 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:28.700+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:29.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:28 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:28.700+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:29.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:28 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:28.700+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:29.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:28 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:28.700+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:29.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:28 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:28.700+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:29.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:28 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:28.700+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-25T02:22:00.891311+0000 front 2024-04-25T02:22:00.891392+0000 (oldest deadline 2024-04-25T02:22:26.191352+0000) 2024-04-25T02:22:29.310 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:29 smithi040 podman[74053]: 2024-04-25 02:22:29.009408553 +0000 UTC m=+0.424877477 container start 5bdcc885ed70e4932001e0267761dffd1b5d190ebaee5129f8acb72a83ecf434 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T02:22:29.310 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:29 smithi040 podman[74053]: 2024-04-25 02:22:29.089032685 +0000 UTC m=+0.504501617 container attach 5bdcc885ed70e4932001e0267761dffd1b5d190ebaee5129f8acb72a83ecf434 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T02:22:29.603 INFO:journalctl@ceph.osd.10.smithi040.stdout:Apr 25 02:22:29 smithi040 podman[74053]: 2024-04-25 02:22:29.307155498 +0000 UTC m=+0.722624414 container died 5bdcc885ed70e4932001e0267761dffd1b5d190ebaee5129f8acb72a83ecf434 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5) 2024-04-25T02:22:30.025 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T02:22:30.071 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:22:30.071 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T02:22:30.071 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T02:22:30.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.11 2024-04-25T02:22:30.098 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:29 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:29.728+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:30.098 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:29 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:29.728+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:30.099 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:29 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:29.728+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:30.099 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:29 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:29.728+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:30.099 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:29 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:29.728+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:30.099 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:29 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:29.728+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:30.099 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:29 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:29.728+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-25T02:22:00.891311+0000 front 2024-04-25T02:22:00.891392+0000 (oldest deadline 2024-04-25T02:22:26.191352+0000) 2024-04-25T02:22:30.380 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:30 smithi134 systemd[1]: Stopping Ceph osd.11 for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:22:30.691 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:30.376+0000 7f932fb50700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:22:30.692 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:30.376+0000 7f932fb50700 -1 osd.11 129 *** Got signal Terminated *** 2024-04-25T02:22:30.692 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:30.376+0000 7f932fb50700 -1 osd.11 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:22:31.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:30.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:31.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:30.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:31.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:30.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:31.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:30.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:31.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:30.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:31.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:30.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:31.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:30 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:30.727+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-25T02:22:00.891311+0000 front 2024-04-25T02:22:00.891392+0000 (oldest deadline 2024-04-25T02:22:26.191352+0000) 2024-04-25T02:22:32.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:31.735+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:32.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:31.735+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:32.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:31.735+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:32.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:31.735+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:32.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:31.735+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:32.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:31.735+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:32.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:31.735+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-25T02:22:00.891311+0000 front 2024-04-25T02:22:00.891392+0000 (oldest deadline 2024-04-25T02:22:26.191352+0000) 2024-04-25T02:22:33.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:32 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:32.709+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:33.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:32 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:32.709+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:33.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:32 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:32.709+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:33.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:32 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:32.709+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:33.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:32 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:32.709+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:33.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:32 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:32.709+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:33.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:32 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:32.709+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-25T02:22:00.891311+0000 front 2024-04-25T02:22:00.891392+0000 (oldest deadline 2024-04-25T02:22:26.191352+0000) 2024-04-25T02:22:33.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:32 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:32.709+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6820 osd.8 since back 2024-04-25T02:22:10.792598+0000 front 2024-04-25T02:22:10.792631+0000 (oldest deadline 2024-04-25T02:22:31.892662+0000) 2024-04-25T02:22:34.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:33 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:33.721+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:34.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:33 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:33.721+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:34.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:33 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:33.721+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:34.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:33 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:33.721+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:34.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:33 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:33.721+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:34.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:33 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:33.721+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:34.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:33 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:33.721+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-25T02:22:00.891311+0000 front 2024-04-25T02:22:00.891392+0000 (oldest deadline 2024-04-25T02:22:26.191352+0000) 2024-04-25T02:22:34.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:33 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:33.721+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6820 osd.8 since back 2024-04-25T02:22:10.792598+0000 front 2024-04-25T02:22:10.792631+0000 (oldest deadline 2024-04-25T02:22:31.892662+0000) 2024-04-25T02:22:35.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:34.764+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T02:21:03.880778+0000 front 2024-04-25T02:21:03.880803+0000 (oldest deadline 2024-04-25T02:21:26.681469+0000) 2024-04-25T02:22:35.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:34.764+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6804 osd.1 since back 2024-04-25T02:21:11.883369+0000 front 2024-04-25T02:21:11.883113+0000 (oldest deadline 2024-04-25T02:21:34.183359+0000) 2024-04-25T02:22:35.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:34.764+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6804 osd.2 since back 2024-04-25T02:21:19.484568+0000 front 2024-04-25T02:21:19.484591+0000 (oldest deadline 2024-04-25T02:21:46.485397+0000) 2024-04-25T02:22:35.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:34.764+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T02:21:26.986758+0000 front 2024-04-25T02:21:26.986876+0000 (oldest deadline 2024-04-25T02:21:52.286736+0000) 2024-04-25T02:22:35.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:34.764+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6812 osd.4 since back 2024-04-25T02:21:36.387963+0000 front 2024-04-25T02:21:36.387933+0000 (oldest deadline 2024-04-25T02:22:01.687779+0000) 2024-04-25T02:22:35.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:34.764+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6812 osd.5 since back 2024-04-25T02:21:45.689097+0000 front 2024-04-25T02:21:45.689164+0000 (oldest deadline 2024-04-25T02:22:09.189216+0000) 2024-04-25T02:22:35.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:34.764+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.40:6820 osd.7 since back 2024-04-25T02:22:00.891311+0000 front 2024-04-25T02:22:00.891392+0000 (oldest deadline 2024-04-25T02:22:26.191352+0000) 2024-04-25T02:22:35.192 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:34 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11[58339]: 2024-04-25T02:22:34.764+0000 7f932c144700 -1 osd.11 129 heartbeat_check: no reply from 172.21.15.134:6820 osd.8 since back 2024-04-25T02:22:10.792598+0000 front 2024-04-25T02:22:10.792631+0000 (oldest deadline 2024-04-25T02:22:31.892662+0000) 2024-04-25T02:22:36.162 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:35 smithi134 podman[73972]: 2024-04-25 02:22:35.879088313 +0000 UTC m=+5.617360204 container died 70d4659b9b1e1671c71094b17a857d546c09d5e4f9d96d7216f82610093037d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T02:22:36.442 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:36 smithi134 podman[73972]: 2024-04-25 02:22:36.159518358 +0000 UTC m=+5.897790249 container cleanup 70d4659b9b1e1671c71094b17a857d546c09d5e4f9d96d7216f82610093037d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T02:22:36.442 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:36 smithi134 bash[73972]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11 2024-04-25T02:22:36.760 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:36 smithi134 podman[73989]: 2024-04-25 02:22:36.444029983 +0000 UTC m=+0.560719286 container remove 70d4659b9b1e1671c71094b17a857d546c09d5e4f9d96d7216f82610093037d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-04-25T02:22:37.075 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:36 smithi134 podman[74076]: 2024-04-25 02:22:36.66007023 +0000 UTC m=+0.017851509 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:22:37.075 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:36 smithi134 podman[74076]: 2024-04-25 02:22:36.778328934 +0000 UTC m=+0.136110206 container create 128f3df48c32550e09a336acf0441e1387252005dd7a889484fd12ad9a1423cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T02:22:37.075 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:36 smithi134 podman[74076]: 2024-04-25 02:22:36.98060477 +0000 UTC m=+0.338386039 container init 128f3df48c32550e09a336acf0441e1387252005dd7a889484fd12ad9a1423cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T02:22:37.075 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:36 smithi134 podman[74076]: 2024-04-25 02:22:36.983674862 +0000 UTC m=+0.341456138 container start 128f3df48c32550e09a336acf0441e1387252005dd7a889484fd12ad9a1423cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T02:22:37.075 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:37 smithi134 podman[74076]: 2024-04-25 02:22:37.072010258 +0000 UTC m=+0.429791527 container attach 128f3df48c32550e09a336acf0441e1387252005dd7a889484fd12ad9a1423cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:22:37.442 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:37 smithi134 podman[74076]: 2024-04-25 02:22:37.269313153 +0000 UTC m=+0.627094426 container died 128f3df48c32550e09a336acf0441e1387252005dd7a889484fd12ad9a1423cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:22:37.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T02:22:37.864 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:37 smithi134 podman[74102]: 2024-04-25 02:22:37.775626236 +0000 UTC m=+0.499104454 container remove 128f3df48c32550e09a336acf0441e1387252005dd7a889484fd12ad9a1423cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T02:22:37.864 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:37 smithi134 systemd[1]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.11.service: Deactivated successfully. 2024-04-25T02:22:37.864 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:37 smithi134 systemd[1]: Stopped Ceph osd.11 for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T02:22:37.864 INFO:journalctl@ceph.osd.11.smithi134.stdout:Apr 25 02:22:37 smithi134 systemd[1]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968@osd.11.service: Consumed 2min 5.823s CPU time. 2024-04-25T02:22:37.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:22:37.879 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T02:22:37.879 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T02:22:37.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.a 2024-04-25T02:22:38.206 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 02:22:37 smithi031 systemd[1]: Stopping Ceph mds.a for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:22:38.490 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 02:22:38 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-a[70428]: 2024-04-25T02:22:38.202+0000 7fe91c717700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:22:38.490 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 02:22:38 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-a[70428]: 2024-04-25T02:22:38.202+0000 7fe91c717700 -1 mds.a *** got signal Terminated *** 2024-04-25T02:22:42.606 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 02:22:42 smithi031 podman[104864]: 2024-04-25 02:22:42.298734907 +0000 UTC m=+4.222034113 container died 375c20a58b875ed22b1c42712145e178424bb8818b1fdc231631114a246b172e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-25T02:22:42.990 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 02:22:42 smithi031 podman[104864]: 2024-04-25 02:22:42.702896324 +0000 UTC m=+4.626195527 container cleanup 375c20a58b875ed22b1c42712145e178424bb8818b1fdc231631114a246b172e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T02:22:42.990 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 02:22:42 smithi031 bash[104864]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-a 2024-04-25T02:22:43.276 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 02:22:43 smithi031 podman[104878]: 2024-04-25 02:22:43.004021833 +0000 UTC m=+0.697193214 container remove 375c20a58b875ed22b1c42712145e178424bb8818b1fdc231631114a246b172e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T02:22:43.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T02:22:43.328 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:22:43.328 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T02:22:43.328 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T02:22:43.328 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.d 2024-04-25T02:22:43.660 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 02:22:43 smithi031 systemd[1]: Stopping Ceph mds.d for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:22:43.990 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 02:22:43 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-d[69472]: 2024-04-25T02:22:43.656+0000 7fdb03351700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:22:43.990 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 02:22:43 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-d[69472]: 2024-04-25T02:22:43.656+0000 7fdb03351700 -1 mds.d *** got signal Terminated *** 2024-04-25T02:22:46.703 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 02:22:46 smithi031 podman[104984]: 2024-04-25 02:22:46.417670662 +0000 UTC m=+2.865771676 container died cd5a56bb55672a38d5d3dbc5bc22c786b43f791da7907d4a7cf8c90db13d32f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T02:22:46.990 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 02:22:46 smithi031 podman[104984]: 2024-04-25 02:22:46.773458538 +0000 UTC m=+3.221559560 container cleanup cd5a56bb55672a38d5d3dbc5bc22c786b43f791da7907d4a7cf8c90db13d32f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:22:46.990 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 02:22:46 smithi031 bash[104984]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-d 2024-04-25T02:22:47.203 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T02:22:47.235 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:22:47.235 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T02:22:47.236 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T02:22:47.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.g 2024-04-25T02:22:47.490 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 02:22:47 smithi031 systemd[1]: Stopping Ceph mds.g for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:22:47.990 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 02:22:47 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-g[70079]: 2024-04-25T02:22:47.576+0000 7f877eb05700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:22:47.991 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 02:22:47 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-g[70079]: 2024-04-25T02:22:47.576+0000 7f877eb05700 -1 mds.g *** got signal Terminated *** 2024-04-25T02:22:52.056 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 02:22:51 smithi031 podman[105104]: 2024-04-25 02:22:51.784247099 +0000 UTC m=+4.330318823 container died 5c78e932ce5f7b5a093f68a8aacd2c75d5b59836ed19e1e7f433f32a5fbe2bbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-g, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:22:52.431 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 02:22:52 smithi031 podman[105104]: 2024-04-25 02:22:52.144115727 +0000 UTC m=+4.690187405 container cleanup 5c78e932ce5f7b5a093f68a8aacd2c75d5b59836ed19e1e7f433f32a5fbe2bbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:22:52.431 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 02:22:52 smithi031 bash[105104]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-g 2024-04-25T02:22:52.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T02:22:52.614 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:22:52.614 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T02:22:52.615 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T02:22:52.615 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.j 2024-04-25T02:22:52.990 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 02:22:52 smithi031 systemd[1]: Stopping Ceph mds.j for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:22:52.991 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 02:22:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-j[69083]: 2024-04-25T02:22:52.914+0000 7fd935379700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:22:52.991 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 02:22:52 smithi031 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-j[69083]: 2024-04-25T02:22:52.914+0000 7fd935379700 -1 mds.j *** got signal Terminated *** 2024-04-25T02:22:55.609 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 02:22:55 smithi031 podman[105225]: 2024-04-25 02:22:55.299950902 +0000 UTC m=+2.469063275 container died 14c1d80366404aefe0ebd26f3b210bc51733c571408b934a9bb597b1c31e2420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5) 2024-04-25T02:22:55.609 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 02:22:55 smithi031 podman[105225]: 2024-04-25 02:22:55.606173585 +0000 UTC m=+2.775285955 container cleanup 14c1d80366404aefe0ebd26f3b210bc51733c571408b934a9bb597b1c31e2420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T02:22:55.990 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 02:22:55 smithi031 bash[105225]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-j 2024-04-25T02:22:55.990 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 02:22:55 smithi031 podman[105239]: 2024-04-25 02:22:55.807036279 +0000 UTC m=+0.506850282 container remove 14c1d80366404aefe0ebd26f3b210bc51733c571408b934a9bb597b1c31e2420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:22:56.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T02:22:56.086 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:22:56.086 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T02:22:56.086 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T02:22:56.086 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.b 2024-04-25T02:22:56.389 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 02:22:56 smithi040 systemd[1]: Stopping Ceph mds.b for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:22:56.662 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 02:22:56 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b[62588]: 2024-04-25T02:22:56.385+0000 7fa30c302700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:22:56.662 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 02:22:56 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b[62588]: 2024-04-25T02:22:56.385+0000 7fa30c302700 -1 mds.b *** got signal Terminated *** 2024-04-25T02:23:02.306 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 02:23:02 smithi040 podman[74160]: 2024-04-25 02:23:02.026403042 +0000 UTC m=+5.745663876 container died ac5fc400b968e0a3b1a9c024bf3a8fe321921ab87a59de26f8344ffd6a6fc677 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:23:02.662 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 02:23:02 smithi040 podman[74160]: 2024-04-25 02:23:02.410794757 +0000 UTC m=+6.130055599 container cleanup ac5fc400b968e0a3b1a9c024bf3a8fe321921ab87a59de26f8344ffd6a6fc677 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, ceph=True) 2024-04-25T02:23:02.662 INFO:journalctl@ceph.mds.b.smithi040.stdout:Apr 25 02:23:02 smithi040 bash[74160]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-b 2024-04-25T02:23:02.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T02:23:02.909 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:23:02.909 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T02:23:02.909 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T02:23:02.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.e 2024-04-25T02:23:03.243 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 02:23:02 smithi040 systemd[1]: Stopping Ceph mds.e for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:23:03.531 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 02:23:03 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e[61957]: 2024-04-25T02:23:03.239+0000 7f084e9dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:23:03.531 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 02:23:03 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e[61957]: 2024-04-25T02:23:03.239+0000 7f084e9dc700 -1 mds.e *** got signal Terminated *** 2024-04-25T02:23:05.242 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 02:23:04 smithi040 podman[74291]: 2024-04-25 02:23:04.936099813 +0000 UTC m=+1.802863026 container died 71f06a6cc78c7224382cfddcd23992c06b762ce66dbde46f45ab75d4faf1279e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:23:05.242 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 02:23:05 smithi040 podman[74291]: 2024-04-25 02:23:05.239913459 +0000 UTC m=+2.106676665 container cleanup 71f06a6cc78c7224382cfddcd23992c06b762ce66dbde46f45ab75d4faf1279e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-25T02:23:05.493 INFO:journalctl@ceph.mds.e.smithi040.stdout:Apr 25 02:23:05 smithi040 bash[74291]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-e 2024-04-25T02:23:05.712 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T02:23:05.746 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:23:05.746 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T02:23:05.746 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T02:23:05.746 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.h 2024-04-25T02:23:06.072 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 02:23:05 smithi040 systemd[1]: Stopping Ceph mds.h for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:23:06.412 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 02:23:06 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-h[62279]: 2024-04-25T02:23:06.068+0000 7f9cb66d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:23:06.412 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 02:23:06 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-h[62279]: 2024-04-25T02:23:06.068+0000 7f9cb66d3700 -1 mds.h *** got signal Terminated *** 2024-04-25T02:23:11.412 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 02:23:11 smithi040 podman[74412]: 2024-04-25 02:23:11.057952273 +0000 UTC m=+5.089523724 container died 46c3665ae4d9b93d04aa269e8c5910fe722bcbac69775d7c34b1f7ac7b404ec8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-h, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T02:23:11.788 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 02:23:11 smithi040 podman[74412]: 2024-04-25 02:23:11.518144545 +0000 UTC m=+5.549715995 container cleanup 46c3665ae4d9b93d04aa269e8c5910fe722bcbac69775d7c34b1f7ac7b404ec8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:23:11.789 INFO:journalctl@ceph.mds.h.smithi040.stdout:Apr 25 02:23:11 smithi040 bash[74412]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-h 2024-04-25T02:23:11.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T02:23:11.954 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:23:11.955 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T02:23:11.955 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T02:23:11.955 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.k 2024-04-25T02:23:12.162 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 02:23:12 smithi040 systemd[1]: Stopping Ceph mds.k for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:23:12.662 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 02:23:12 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-k[61639]: 2024-04-25T02:23:12.271+0000 7ff489a68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:23:12.662 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 02:23:12 smithi040 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-k[61639]: 2024-04-25T02:23:12.271+0000 7ff489a68700 -1 mds.k *** got signal Terminated *** 2024-04-25T02:23:16.642 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 02:23:16 smithi040 podman[74533]: 2024-04-25 02:23:16.386204958 +0000 UTC m=+4.216374290 container died 7a908d565424fa5a70c93e51bcdca2ef19dcd78b624ba1eb214436f35e1bed18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T02:23:16.912 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 02:23:16 smithi040 podman[74533]: 2024-04-25 02:23:16.721777545 +0000 UTC m=+4.551946869 container cleanup 7a908d565424fa5a70c93e51bcdca2ef19dcd78b624ba1eb214436f35e1bed18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9) 2024-04-25T02:23:16.912 INFO:journalctl@ceph.mds.k.smithi040.stdout:Apr 25 02:23:16 smithi040 bash[74533]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-k 2024-04-25T02:23:17.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T02:23:17.200 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:23:17.200 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T02:23:17.200 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T02:23:17.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.c 2024-04-25T02:23:17.501 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 02:23:17 smithi134 systemd[1]: Stopping Ceph mds.c for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:23:17.942 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 02:23:17 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-c[63676]: 2024-04-25T02:23:17.497+0000 7fd83abcb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:23:17.942 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 02:23:17 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-c[63676]: 2024-04-25T02:23:17.497+0000 7fd83abcb700 -1 mds.c *** got signal Terminated *** 2024-04-25T02:23:20.411 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 02:23:20 smithi134 podman[74181]: 2024-04-25 02:23:20.111224395 +0000 UTC m=+2.719161905 container died 1498fd4b1901db8a5d0ea6e3b350e3459981b789b4dbd065db2384637fcdb18c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9) 2024-04-25T02:23:20.691 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 02:23:20 smithi134 podman[74181]: 2024-04-25 02:23:20.514779337 +0000 UTC m=+3.122716847 container cleanup 1498fd4b1901db8a5d0ea6e3b350e3459981b789b4dbd065db2384637fcdb18c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T02:23:20.692 INFO:journalctl@ceph.mds.c.smithi134.stdout:Apr 25 02:23:20 smithi134 bash[74181]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-c 2024-04-25T02:23:20.826 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T02:23:20.862 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:23:20.862 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T02:23:20.862 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T02:23:20.862 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.f 2024-04-25T02:23:21.188 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 02:23:20 smithi134 systemd[1]: Stopping Ceph mds.f for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:23:21.441 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 02:23:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-f[63368]: 2024-04-25T02:23:21.184+0000 7f18a455b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:23:21.441 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 02:23:21 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-f[63368]: 2024-04-25T02:23:21.184+0000 7f18a455b700 -1 mds.f *** got signal Terminated *** 2024-04-25T02:23:25.404 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 02:23:25 smithi134 podman[74315]: 2024-04-25 02:23:25.076514006 +0000 UTC m=+3.995166695 container died f69574034f9fde8b9b2c5cf717e66fa3fbbb2ea3589bc16fb76f61dc835e875a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9) 2024-04-25T02:23:25.404 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 02:23:25 smithi134 podman[74315]: 2024-04-25 02:23:25.401848337 +0000 UTC m=+4.320501018 container cleanup f69574034f9fde8b9b2c5cf717e66fa3fbbb2ea3589bc16fb76f61dc835e875a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T02:23:25.691 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 02:23:25 smithi134 bash[74315]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-f 2024-04-25T02:23:25.691 INFO:journalctl@ceph.mds.f.smithi134.stdout:Apr 25 02:23:25 smithi134 podman[74329]: 2024-04-25 02:23:25.586082564 +0000 UTC m=+0.504393475 container remove f69574034f9fde8b9b2c5cf717e66fa3fbbb2ea3589bc16fb76f61dc835e875a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9) 2024-04-25T02:23:25.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T02:23:25.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:23:25.848 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T02:23:25.848 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T02:23:25.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.i 2024-04-25T02:23:26.166 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 02:23:25 smithi134 systemd[1]: Stopping Ceph mds.i for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:23:26.442 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 02:23:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-i[63998]: 2024-04-25T02:23:26.162+0000 7fd1781bd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:23:26.442 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 02:23:26 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-i[63998]: 2024-04-25T02:23:26.162+0000 7fd1781bd700 -1 mds.i *** got signal Terminated *** 2024-04-25T02:23:30.191 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 02:23:30 smithi134 podman[74440]: 2024-04-25 02:23:30.004485967 +0000 UTC m=+3.946009927 container died 4aaa4ddc19e624a140786bbfdd92b5d5d05996ee26c609d312f4719cb10abb93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-25T02:23:30.633 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 02:23:30 smithi134 podman[74440]: 2024-04-25 02:23:30.446085043 +0000 UTC m=+4.387609013 container cleanup 4aaa4ddc19e624a140786bbfdd92b5d5d05996ee26c609d312f4719cb10abb93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T02:23:30.633 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 02:23:30 smithi134 bash[74440]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-i 2024-04-25T02:23:30.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T02:23:30.889 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 02:23:30 smithi134 podman[74454]: 2024-04-25 02:23:30.630300459 +0000 UTC m=+0.624961730 container remove 4aaa4ddc19e624a140786bbfdd92b5d5d05996ee26c609d312f4719cb10abb93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-i, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:23:30.889 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 02:23:30 smithi134 systemd[1]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.i.service: Deactivated successfully. 2024-04-25T02:23:30.889 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 02:23:30 smithi134 systemd[1]: Stopped Ceph mds.i for 36afbea4-02a5-11ef-bc93-c7b262605968. 2024-04-25T02:23:30.889 INFO:journalctl@ceph.mds.i.smithi134.stdout:Apr 25 02:23:30 smithi134 systemd[1]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.i.service: Consumed 12min 11.642s CPU time. 2024-04-25T02:23:30.900 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:23:30.900 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T02:23:30.900 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T02:23:30.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.l 2024-04-25T02:23:31.191 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 02:23:30 smithi134 systemd[1]: Stopping Ceph mds.l for 36afbea4-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:23:31.691 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 02:23:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-l[63057]: 2024-04-25T02:23:31.198+0000 7f00e9cdd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:23:31.691 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 02:23:31 smithi134 ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-l[63057]: 2024-04-25T02:23:31.198+0000 7f00e9cdd700 -1 mds.l *** got signal Terminated *** 2024-04-25T02:23:33.870 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 02:23:33 smithi134 podman[74559]: 2024-04-25 02:23:33.555537598 +0000 UTC m=+2.435934004 container died 7dcfa65053c5e5b3bb73a4e3b91d84edae8163e4cc6ccf1f27be2ef1d78bb86f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5) 2024-04-25T02:23:34.186 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 02:23:33 smithi134 podman[74559]: 2024-04-25 02:23:33.94105699 +0000 UTC m=+2.821453386 container cleanup 7dcfa65053c5e5b3bb73a4e3b91d84edae8163e4cc6ccf1f27be2ef1d78bb86f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-l, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T02:23:34.186 INFO:journalctl@ceph.mds.l.smithi134.stdout:Apr 25 02:23:33 smithi134 bash[74559]: ceph-36afbea4-02a5-11ef-bc93-c7b262605968-mds-l 2024-04-25T02:23:34.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36afbea4-02a5-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T02:23:34.333 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:23:34.333 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T02:23:34.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T02:23:34.534 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T02:23:37.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T02:23:37.425 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T02:23:40.050 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T02:23:40.240 INFO:teuthology.orchestra.run.smithi134.stdout:Deleting cluster with fsid: 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T02:23:42.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:23:42.701 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:23:42.734 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:23:42.761 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T02:23:42.763 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342/remote/smithi031/crash 2024-04-25T02:23:42.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/crash -- . 2024-04-25T02:23:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:tar: /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T02:23:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Error is not recoverable: exiting now 2024-04-25T02:23:42.803 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342/remote/smithi040/crash 2024-04-25T02:23:42.804 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/crash -- . 2024-04-25T02:23:42.840 INFO:teuthology.orchestra.run.smithi040.stderr:tar: /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T02:23:42.841 INFO:teuthology.orchestra.run.smithi040.stderr:tar: Error is not recoverable: exiting now 2024-04-25T02:23:42.843 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342/remote/smithi134/crash 2024-04-25T02:23:42.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/crash -- . 2024-04-25T02:23:42.878 INFO:teuthology.orchestra.run.smithi134.stderr:tar: /var/lib/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T02:23:42.878 INFO:teuthology.orchestra.run.smithi134.stderr:tar: Error is not recoverable: exiting now 2024-04-25T02:23:42.879 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T02:23:42.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-25T02:23:42.987 INFO:tasks.cephadm:Compressing logs... 2024-04-25T02:23:42.987 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T02:23:42.990 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T02:23:42.992 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T02:23:43.016 INFO:teuthology.orchestra.run.smithi134.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T02:23:43.016 INFO:teuthology.orchestra.run.smithi134.stderr:: No such file or directory 2024-04-25T02:23:43.016 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.log 2024-04-25T02:23:43.016 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T02:23:43.016 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T02:23:43.017 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T02:23:43.017 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T02:23:43.018 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T02:23:43.019 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T02:23:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose --/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.c.log: /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T02:23:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T02:23:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T02:23:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T02:23:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T02:23:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T02:23:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T02:23:43.021 INFO:teuthology.orchestra.run.smithi040.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T02:23:43.021 INFO:teuthology.orchestra.run.smithi040.stderr:: No such file or directory 2024-04-25T02:23:43.021 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.log 2024-04-25T02:23:43.022 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T02:23:43.022 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T02:23:43.022 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T02:23:43.028 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T02:23:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T02:23:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T02:23:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T02:23:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T02:23:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T02:23:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T02:23:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T02:23:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log: /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T02:23:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T02:23:43.030 INFO:teuthology.orchestra.run.smithi031.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T02:23:43.030 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T02:23:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T02:23:43.032 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.d.log: 87.4% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T02:23:43.178 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.k.log: 91.0% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T02:23:43.199 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.l.log: 84.4% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T02:23:43.236 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.6% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T02:23:43.241 INFO:teuthology.orchestra.run.smithi134.stderr: 91.0% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T02:23:43.249 INFO:teuthology.orchestra.run.smithi031.stderr: 84.5% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T02:23:43.273 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T02:23:43.296 INFO:teuthology.orchestra.run.smithi031.stderr: 90.2% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T02:23:43.297 INFO:teuthology.orchestra.run.smithi031.stderr: 91.2% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T02:23:43.317 INFO:teuthology.orchestra.run.smithi040.stderr: 84.5% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T02:23:43.403 INFO:teuthology.orchestra.run.smithi031.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T02:23:43.404 INFO:teuthology.orchestra.run.smithi040.stderr: 87.4% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.log.gz 92.1% 2024-04-25T02:23:43.405 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T02:23:43.434 INFO:teuthology.orchestra.run.smithi040.stderr: 92.7% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T02:23:43.453 INFO:teuthology.orchestra.run.smithi134.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T02:23:43.503 INFO:teuthology.orchestra.run.smithi031.stderr: 94.7% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T02:23:43.520 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T02:23:43.557 INFO:teuthology.orchestra.run.smithi134.stderr: 90.4% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T02:23:43.667 INFO:teuthology.orchestra.run.smithi040.stderr: 90.3% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T02:23:51.303 INFO:teuthology.orchestra.run.smithi031.stderr: 90.3% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T02:24:03.407 INFO:teuthology.orchestra.run.smithi134.stderr: 92.0% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T02:24:03.418 INFO:teuthology.orchestra.run.smithi031.stderr: 91.0% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T02:24:09.857 INFO:teuthology.orchestra.run.smithi040.stderr: 91.9% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T02:24:55.057 INFO:teuthology.orchestra.run.smithi031.stderr: 94.9% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T02:25:16.244 INFO:teuthology.orchestra.run.smithi031.stderr: 94.9% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T02:25:33.563 INFO:teuthology.orchestra.run.smithi040.stderr: 95.0% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T02:25:37.166 INFO:teuthology.orchestra.run.smithi040.stderr: 95.1% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T02:25:53.172 INFO:teuthology.orchestra.run.smithi040.stderr: 90.8% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T02:26:00.880 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T02:26:08.431 INFO:teuthology.orchestra.run.smithi040.stderr: 91.3% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T02:27:43.449 INFO:teuthology.orchestra.run.smithi031.stderr: 90.7% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T02:27:47.827 INFO:teuthology.orchestra.run.smithi134.stderr: 94.9% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T02:27:52.175 INFO:teuthology.orchestra.run.smithi031.stderr: 91.2% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T02:28:33.513 INFO:teuthology.orchestra.run.smithi031.stderr: 90.7% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T02:28:39.935 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T02:28:56.491 INFO:teuthology.orchestra.run.smithi031.stderr: 90.8% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T02:29:48.363 INFO:teuthology.orchestra.run.smithi134.stderr: 91.1% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T02:30:11.003 INFO:teuthology.orchestra.run.smithi134.stderr: 90.5% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T02:30:53.201 INFO:teuthology.orchestra.run.smithi134.stderr: 90.6% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T02:32:19.053 INFO:teuthology.orchestra.run.smithi134.stderr: 90.5% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T02:36:06.329 INFO:teuthology.orchestra.run.smithi031.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-25T02:36:06.331 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-04-25T02:36:06.331 INFO:teuthology.orchestra.run.smithi031.stderr:real 12m23.327s 2024-04-25T02:36:06.331 INFO:teuthology.orchestra.run.smithi031.stderr:user 8m25.766s 2024-04-25T02:36:06.331 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m20.113s 2024-04-25T02:43:18.179 INFO:teuthology.orchestra.run.smithi134.stderr: 94.1% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T02:43:31.035 INFO:teuthology.orchestra.run.smithi134.stderr: 94.1% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T02:45:17.405 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T02:45:17.445 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-25T02:45:17.445 INFO:teuthology.orchestra.run.smithi134.stderr:real 21m34.438s 2024-04-25T02:45:17.446 INFO:teuthology.orchestra.run.smithi134.stderr:user 8m58.056s 2024-04-25T02:45:17.446 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m25.850s 2024-04-25T02:46:34.836 INFO:teuthology.orchestra.run.smithi040.stderr: 94.0% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T02:48:50.593 INFO:teuthology.orchestra.run.smithi040.stderr: 95.2% -- replaced with /var/log/ceph/36afbea4-02a5-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T02:48:50.611 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-25T02:48:50.611 INFO:teuthology.orchestra.run.smithi040.stderr:real 25m7.603s 2024-04-25T02:48:50.611 INFO:teuthology.orchestra.run.smithi040.stderr:user 9m4.182s 2024-04-25T02:48:50.611 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m26.713s 2024-04-25T02:48:50.612 INFO:tasks.cephadm:Archiving logs... 2024-04-25T02:48:50.612 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342/remote/smithi031/log 2024-04-25T02:48:50.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T02:50:12.615 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342/remote/smithi040/log 2024-04-25T02:50:12.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T02:51:25.098 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342/remote/smithi134/log 2024-04-25T02:51:25.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T02:52:46.473 INFO:tasks.cephadm:Removing cluster... 2024-04-25T02:52:46.473 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 --force 2024-04-25T02:52:46.858 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T02:52:47.180 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 --force 2024-04-25T02:52:47.753 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T02:52:48.430 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36afbea4-02a5-11ef-bc93-c7b262605968 --force 2024-04-25T02:52:49.115 INFO:teuthology.orchestra.run.smithi134.stdout:Deleting cluster with fsid: 36afbea4-02a5-11ef-bc93-c7b262605968 2024-04-25T02:52:49.917 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T02:52:49.918 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T02:52:49.937 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T02:52:49.956 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T02:52:49.972 INFO:tasks.cephadm:Teardown complete 2024-04-25T02:52:49.973 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T02:52:49.985 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-04-25T02:52:49.986 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T02:52:49.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T02:52:49.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T02:52:50.000 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T02:52:50.130 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T02:52:50.141 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T02:52:50.142 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T02:52:50.144 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T02:52:50.147 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T02:52:50.159 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-04-25T02:52:50.160 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-04-25T02:52:50.162 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-25T02:52:50.198 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T02:52:50.198 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-04-25T02:52:50.198 INFO:teuthology.orchestra.run.smithi031.stdout:^- hv01.front.sepia.ceph.com 2 6 377 24 +1540us[+1538us] +/- 579ms 2024-04-25T02:52:50.198 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -258us[ -261us] +/- 80ms 2024-04-25T02:52:50.198 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +276us[ +276us] +/- 88ms 2024-04-25T02:52:50.198 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T02:52:50.201 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T02:52:50.201 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-25T02:52:50.201 INFO:teuthology.orchestra.run.smithi040.stdout:^- hv01.front.sepia.ceph.com 2 6 377 22 +1548us[+1539us] +/- 579ms 2024-04-25T02:52:50.201 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -262us[ -272us] +/- 80ms 2024-04-25T02:52:50.201 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +288us[ +279us] +/- 88ms 2024-04-25T02:52:50.201 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T02:52:50.209 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T02:52:50.209 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-04-25T02:52:50.210 INFO:teuthology.orchestra.run.smithi134.stdout:^- hv01.front.sepia.ceph.com 2 6 377 20 +1539us[+1539us] +/- 579ms 2024-04-25T02:52:50.210 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -243us[ -207us] +/- 80ms 2024-04-25T02:52:50.210 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 +283us[ +319us] +/- 88ms 2024-04-25T02:52:50.210 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T02:52:50.220 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T02:52:50.230 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T02:52:50.231 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T02:52:50.257 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T02:52:50.405 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T02:52:50.517 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T02:52:50.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T02:52:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:52:50.696 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T02:52:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:52:50.728 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T02:52:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:52:50.754 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T02:52:50.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T02:52:50.808 INFO:teuthology.task.internal:Duration was 4512.846967 seconds 2024-04-25T02:52:50.808 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T02:52:50.840 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T02:52:50.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T02:52:50.843 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T02:52:50.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T02:52:50.892 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T02:52:50.904 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T02:52:50.905 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T02:52:51.274 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T02:52:51.274 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T02:52:51.275 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T02:52:51.304 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-04-25T02:52:51.304 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T02:52:51.336 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-04-25T02:52:51.337 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T02:52:51.364 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T02:52:51.364 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T02:52:51.367 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T02:52:51.380 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T02:52:51.432 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T02:52:51.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T02:52:51.438 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T02:52:51.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T02:52:51.869 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T02:52:51.882 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T02:52:51.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T02:52:51.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T02:52:51.943 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T02:52:51.975 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T02:52:51.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:52:51.990 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:52:51.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:52:52.024 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-04-25T02:52:52.027 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-04-25T02:52:52.045 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-04-25T02:52:52.074 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:52:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:52:52.101 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:52:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:52:52.116 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:52:52.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:52:52.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T02:52:52.143 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T02:52:52.144 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342/remote/smithi031 2024-04-25T02:52:52.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T02:52:52.218 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342/remote/smithi040 2024-04-25T02:52:52.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T02:52:52.286 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342/remote/smithi134 2024-04-25T02:52:52.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T02:52:52.420 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T02:52:52.421 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T02:52:52.424 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T02:52:52.427 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T02:52:52.474 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T02:52:52.509 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T02:52:52.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T02:52:52.573 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T02:52:52.573 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T02:52:52.577 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T02:52:52.580 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T02:52:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 02:52 /home/ubuntu/cephtest 2024-04-25T02:52:52.594 INFO:teuthology.orchestra.run.smithi040.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 02:52 /home/ubuntu/cephtest 2024-04-25T02:52:52.595 INFO:teuthology.orchestra.run.smithi134.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 02:52 /home/ubuntu/cephtest 2024-04-25T02:52:52.596 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T02:52:52.702 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T02:52:52.960 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-04-25T02:52:52.960 INFO:teuthology.nuke:Checking targets against current locks 2024-04-25T02:52:52.988 DEBUG:teuthology.nuke:shortname: smithi031 2024-04-25T02:52:52.988 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T02:52:53.011 DEBUG:teuthology.nuke:shortname: smithi040 2024-04-25T02:52:53.011 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T02:52:53.016 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:32:08.826590', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF82udbYFdRf9tjtMmjIylP+lAgqNC2Zzy5PaIYR33d1VbMz9xJLhyD3NL2ihqyfcEzpbnkGGJpvbOPd63OyZ1s='} 2024-04-25T02:52:53.036 DEBUG:teuthology.nuke:shortname: smithi134 2024-04-25T02:52:53.037 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T02:52:53.041 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:32:08.828142', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/v/iguqqCAkM4GT3humQA9fhNSYy03YKicJINuDvyy/svrxqMFd2Y7uQEVPThOrhWY1ZlTrZXJX6n3ZBOSIm8='} 2024-04-25T02:52:53.062 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672342', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:32:08.829383', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAk8R7bNpgxgHx1j/zCAi+KwGKg9dU4qWHeaksTyaq0rSn/Zf/+1RAsa8NZ6Coutd4YHp+6krinPeNiN23lKOck='} 2024-04-25T02:52:53.074 INFO:teuthology.orchestra.console.smithi031:Power off 2024-04-25T02:52:53.074 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T02:52:53.098 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2024-04-25T02:52:53.098 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:52:53.113 INFO:teuthology.orchestra.console.smithi040:Power off 2024-04-25T02:52:53.113 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T02:52:53.126 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-04-25T02:52:53.140 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-04-25T02:52:53.140 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:52:53.165 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-25T02:52:53.180 INFO:teuthology.orchestra.console.smithi134:Power off 2024-04-25T02:52:53.181 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T02:52:53.203 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-04-25T02:52:53.203 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:52:53.226 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-04-25T02:52:57.128 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:52:57.167 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:52:57.227 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:52:57.239 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2024-04-25T02:52:57.276 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-25T02:52:57.339 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-04-25T02:53:01.244 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:53:01.277 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:53:01.340 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:53:01.356 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2024-04-25T02:53:01.387 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-04-25T02:53:01.450 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-04-25T02:53:01.456 INFO:teuthology.orchestra.console.smithi031:Power off completed 2024-04-25T02:53:01.551 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-04-25T02:53:05.390 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:53:05.500 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-04-25T02:53:05.601 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-04-25T02:53:05.706 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 4512.846966981888 failure_reason: 'error during scrub thrashing: rank damage found: {''backtrace''}' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-04-25T02:53:05.706 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T02:53:05.778 INFO:teuthology.run:FAIL